Attention & the Transformer — how AI thinks

01 · The core trick

Click "it" — watch the model decide what it means

lets every token look at every other token and pull in what's relevant. Click any token below to make it the query; brightness shows where its attention goes. Then switch between the two "tired / wide" sentences and watch the same word flip its referent.

Query token: "it" · brighter = more attention

illustrative

What you're seeing

"it was too tired" — only an animal can be tired, so the model resolves "it" to "animal".

02 · Depth builds understanding

A token's journey through the stack

A is the same block stacked 30–100 times. Scrub through the layers and watch the vector for "it" transform from "generic pronoun" to "the tired animal".

LayerL0

What changed for "it" at layer 0

"it" enters as a generic pronoun vector — no idea yet what it refers to. Each block: attention (mix between tokens) → MLP (process within token) → add back to the residual stream.

03 · Under the hood

Queries, keys, values

A library analogy: each token writes a question card (query), wears a name tag (key), and carries contents (value). Questions are matched against name tags; the best matches hand over their contents.

1 · Project

Each token's vector is multiplied by three learned matrices → Query, Key, Value. Shapes: [n, d_model] → [n, d_head] each.

2 · Score

QKᵀ: every query dotted with every key → an n×n score matrix. Divide by √d_head so softmax doesn't saturate.

3 · Softmax

Each row becomes a probability distribution — 'how much should token i listen to each token j'. Causal mask hides the future.

4 · Mix

Multiply scores by V: each token's output is a weighted average of value vectors. Concat heads, project back to d_model.

Before transformers

RNNs read text one token at a time, squeezing everything into a single running summary — by word 500, word 3 had effectively dissolved. Attention connects any two tokens in one step, no matter the distance. That one change is why 2017's "Attention Is All You Need" renamed the field.

Knowledge check

Did it stick?

1/3In "…because it was too wide", how does the model know 'it' means the street?

2/3Why does doubling context length roughly quadruple attention cost?

3/3What's the role of the Value vectors?

Loading module…

Common questions this module answers

What does attention do in a transformer?

For every token, attention computes how relevant every other token is, then mixes their information together. It is how 'it' figures out whether it refers to 'the animal' or 'the street' — by attending to the right earlier token.

Why is attention quadratic — O(n²)?

Every token compares itself against every other token, so n tokens require n × n comparisons. Doubling context length quadruples attention compute, which is why long context windows are expensive.

What are queries, keys and values?

Each token emits a query ('what am I looking for?'), a key ('what do I contain?') and a value ('what do I pass along?'). Attention scores are query–key dot products; the output is a score-weighted sum of values.