TRANSFORMERS — understand attention
transforms words.
A visual dive
at the center of attention.
connect the context.
A word never travels
alone.
In “she sat on the river bank”, the context points to the edge of a river. In “she opened an account at the bank”, it points to a financial institution. For a model, that context has to become a computation.
This course follows an autoregressive Language Transformer: it transforms a sequence into contextualized representations, then proposes the next token. It's a variant of architecture, not the definition of all Transformers. [1]
Represent
The words become
coordinates.
The text is divided into tokens : words, fragments or punctuation marks. A learned table associates each token with a vector, its embedding. We can think of these vectors as points in a many-dimensional space. [1]
Position also matters: “the cat chases the dog” and “the dog chases the cat” contain the same words. Position information allows the network to distinguish their order. In the original Transformer, it is added to the embeddings. [4]
The initial vector identifies a token. The following layers will modify it depending on what surrounds it. The coordinates of the demonstration are a reading aid, not a dictionary of linguistic properties.
in relationships.
Connect
What should we
pay attention to?
Attention produces an update for each position. Three learned projections of the same state play three roles: the query Q is compared to keys K ; the weights obtained are used to mix the values V. These roles are a way of reading the calculation, without assuming an intention of the model. [2]
The dot product measures the compatibility of a query with a key. We divide the scores by √dₖ to check their scale, then softmax transforms them into positive weights whose sum is 1. The output is a weighted sum of the values. [2]
The query
What this position is looking for.
Q = XWQThe key
What it can correspond to.
K = XWKThe value
The information to be transmitted.
V = XWVEach line corresponds to a query. M is 0 on authorized positions and −∞ on hidden positions.
Choose the token that receives the information.
View scores and weights
| Token | q · k / √2 | Weight α |
|---|---|---|
| The · 1 | 0.269 | 9.2 % |
| cat · 2 | 2.333 | 72.2 % |
| sleeps · 3 | 0.976 | 18.6 % |
| on · 4 | −∞ | 0.0 % |
| the · 5 | −∞ | 0.0 % |
| mat · 6 | −∞ | 0.0 % |
The future remains hidden.
To predict the next token, a position must not access the following ones. The causal mask blocks these links before softmax: their weight becomes zero. In the laboratory, select “sleeps” then remove the mask to see the following three positions. [2]
Several heads, several blends.
Each head uses its own projections. Their outputs are concatenated and then projected together. Compare the heads in the demonstration: the same token receives two different mixes. In a trained model, we do not impose a precise linguistic function on each head. [2]
Assemble
A layer connects.
The next one refines.
Connect, then transform.
Attention exchanges information between positions. The network feed-forward then applies a nonlinear transformation to each position, with the same parameters. Residual connections add the output of a subblock to its input; normalizations control the scale of activations. [4]
As the blocks progress, the representations evolve. At the output, a projection provides a score per vocabulary token. A softmax gives the distribution of the next continuation. [1]
Drawing the matrices and tracking their dimensions helps you see where the information flows and what it costs in memory. His exploration of DeepSeek's latent attention extends this reading to the cache of keys and values. [3]
Continue
And now,
which token?
The network produces possibilities. Systematically choosing the most probable results in greedy decoding; sampling allows other continuations. The retained token joins the context, and the process begins again. [1]
Vary the choice.
In our example, “sleeps” has the highest score. At low temperatures, its probability dominates. Raise the slider: less favoured candidates become more present. The ranking remains the same, because the scores are fixed.
Make several samples at the same temperature. The word obtained can change while the probabilities are unchanged. The formula for this module is pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T), with T strictly positive.
The cat …
When training a causal language model, predictions are compared to the actual next tokens, then backpropagation adjusts the parameters to reduce the error. During ordinary generation, these parameters remain fixed: it is the context that lengthens. [1]
What remains
Represent.
Connect. Predict.
To reread a Transform diagram, follow three questions: what representation goes into the block, what positions can be swapped, and what transformation produces the output? The matrices then become a path that can be followed.
Return to the laboratoryTo go further
Ideas are shared.
An original course, inspired by the visual pedagogy of 3Blue1Brown and Welch Labs. The diagrams on this page are created for the portfolio.