Artificial intelligenceWords. Links. Meaning.

TRANSFORMERS — understand attention

Context
transforms words.

A visual dive
at the center of attention.

Start reading ↓
01A central idea:
connect the context.
Before the equations

A word never travels
alone.

In “she sat on the river bank”, the context points to the edge of a river. In “she opened an account at the bank”, it points to a financial institution. For a model, that context has to become a computation.

This course follows an autoregressive Language Transformer: it transforms a sequence into contextualized representations, then proposes the next token. It's a variant of architecture, not the definition of all Transformers. [1]

01

Represent

The words become
coordinates.

The text is divided into tokens : words, fragments or punctuation marks. A learned table associates each token with a vector, its embedding. We can think of these vectors as points in a many-dimensional space. [1]

Position also matters: “the cat chases the dog” and “the dog chases the cat” contain the same words. Position information allows the network to distinguish their order. In the original Transformer, it is added to the embeddings. [4]

01 / From text to numbersTry it yourself
cat
1.20.8-0.30.5

A token
four coordinates.

Choose a word. Vectors are invented and limited to four dimensions for reading. The two “the” here share the same embedding; their position will distinguish them. Word-by-word division is simplified.
To remember

The initial vector identifies a token. The following layers will modify it depending on what surrounds it. The coordinates of the demonstration are a reading aid, not a dictionary of linguistic properties.

From the isolated word
Meaning is constructed
in relationships.
to the word in context.
02

Connect

What should we
pay attention to?

Attention produces an update for each position. Three learned projections of the same state play three roles: the query Q is compared to keys K ; the weights obtained are used to mix the values V. These roles are a way of reading the calculation, without assuming an intention of the model. [2]

The dot product measures the compatibility of a query with a key. We divide the scores by √dₖ to check their scale, then softmax transforms them into positive weights whose sum is 1. The output is a weighted sum of the values. [2]

Q

The query

What this position is looking for.

Q = XWQ
K

The key

What it can correspond to.

K = XWK
V

The value

The information to be transmitted.

V = XWV
Attention(Q, K, V)= softmax(QKᵀ / √dₖ + M) V

Each line corresponds to a query. M is 0 on authorized positions and −∞ on hidden positions.

02 / The attention laboratoryLive calculation

Choose the token that receives the information.

9.2 %72.2 %18.6 %maskedmaskedmaskedsleeps
q = [1.8 ; 0.2]Output Σ αᵢvᵢ = [0.932 ; -0.066]
View scores and weights
Normalised weights for “ sleeps ”, position 3
Tokenq · k / √2Weight α
The · 10.2699.2 %
cat · 22.33372.2 %
sleeps · 30.97618.6 %
on · 4−∞0.0 %
the · 5−∞0.0 %
mat · 6−∞0.0 %
Small two-dimensional model, with Q, K and V fixed by hand. The weights are actually calculated by softmax; they do not describe a trained model. Turn off the mask to allow future positions and compare the two heads.

The future remains hidden.

To predict the next token, a position must not access the following ones. The causal mask blocks these links before softmax: their weight becomes zero. In the laboratory, select “sleeps” then remove the mask to see the following three positions. [2]

Several heads, several blends.

Each head uses its own projections. Their outputs are concatenated and then projected together. Compare the heads in the demonstration: the same token receives two different mixes. In a trained model, we do not impose a precise linguistic function on each head. [2]

03

Assemble

A layer connects.
The next one refines.

Tokens + positions
One block · pre-normalized variant
Standardization
Multi-head attention
+ residual connection
Standardization
Feed-forward network
+ residual connection
↻ Repeat on multiple blocks
Normalization → projection → softmax
Simplified path of a pre-normalized autoregressive decoder. The place of standardization varies depending on the architecture.

Connect, then transform.

Attention exchanges information between positions. The network feed-forward then applies a nonlinear transformation to each position, with the same parameters. Residual connections add the output of a subblock to its input; normalizations control the scale of activations. [4]

As the blocks progress, the representations evolve. At the output, a projection provides a score per vocabulary token. A softmax gives the distribution of the next continuation. [1]

The Welch Labs look

Drawing the matrices and tracking their dimensions helps you see where the information flows and what it costs in memory. His exploration of DeepSeek's latent attention extends this reading to the cache of keys and values. [3]

04

Continue

And now,
which token?

The network produces possibilities. Systematically choosing the most probable results in greedy decoding; sampling allows other continuations. The retained token joins the context, and the process begins again. [1]

Vary the choice.

In our example, “sleeps” has the highest score. At low temperatures, its probability dominates. Raise the slider: less favoured candidates become more present. The ranking remains the same, because the scores are fixed.

Make several samples at the same temperature. The word obtained can change while the probabilities are unchanged. The formula for this module is pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T), with T strictly positive.

03 / Several possible continuationsTry it yourself

The cat …

sleeps54.8 %
plays22.3 %
eats12.2 %
watches6.7 %
runs4.1 %
Five candidates with fixed, illustrative scores. Temperature changes their distribution, then the button samples a token. A low temperature concentrates the choices; a higher one spreads them out. This module does not run an LLM.
And learning?

When training a causal language model, predictions are compared to the actual next tokens, then backpropagation adjusts the parameters to reduce the error. During ordinary generation, these parameters remain fixed: it is the context that lengthens. [1]

What remains

Represent.
Connect. Predict.

To reread a Transform diagram, follow three questions: what representation goes into the block, what positions can be swapped, and what transformation produces the output? The matrices then become a path that can be followed.

Return to the laboratory

To go further

Ideas are shared.

An original course, inspired by the visual pedagogy of 3Blue1Brown and Welch Labs. The diagrams on this page are created for the portfolio.

  1. [1] 3Blue1BrownTransformers, the tech behind LLMs · Chapter 5
  2. [2] 3Blue1BrownAttention in transformers, step-by-step · Chapter 6
  3. [3] Welch LabsHow DeepSeek Rewrote the Transformer [MLA]
  4. [4] Vaswani et al. · 2017Attention Is All You Need · Founding article