Skip to content
ML Visualization

Attention & Transformers

Deep LearningAdvanced~11 min

Attention & Transformers — Attention lets each token in a sequence build its representation as a weighted blend of the other tokens, where the weights come from the similarity of learned query and key vectors. It is the core mechanism of the transformer, the architecture behind modern large language models.

Attention is the idea that made large language models possible. Give it a handful of words and watch each one decide how much to draw from every other — the weights light up as arcs and a heatmap, computed live. Then crank the temperature and watch attention flatten until the model can no longer tell which words matter.

Loading attention…

The idea in plain words

Attention answers a simple question for every word in a sequence: which of the other words should I pay attention to? Each word compares itself against the others and builds a new representation by blending in the ones it finds most relevant. In the phrase “the animal didn’t cross the street because it was tired,” attention is what lets it look back to animal rather than street.

Mechanically, every token produces three vectors — a query, a key, and a value. A token’s query is compared with every key by a dot product; the scores are turned into weights with a softmax, and the token’s output is the weighted sum of all the values. Stack this operation into many layers and heads and you get the transformer — the architecture behind modern large language models. Here the tokens are the word embeddings you already met.

Now, the math

Scaled dot-product attention is one compact formula:

Attention(Q,K,V)=softmax ⁣(QK⊤d)V\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V
Q,K,VQ, K, V
the query, key, and value matrices — one row per token.
QK⊤QK^\top
all pairwise similarity scores between queries and keys.
d\sqrt{d}
a scaling factor that keeps the scores from growing with dimension.
softmax\mathrm{softmax}
turns each row of scores into attention weights that sum to one.
▸ Show the derivation

Dividing by d\sqrt{d} matters: in high dimensions dot products grow large, which would push the softmax into a near one-hot spike and starve the gradients. The scaling keeps attention smooth and trainable. This demo uses the token embeddings themselves as Q, K, and V (self-similarity attention), so a word attends to the words closest in meaning; real transformers learn separate projections for Q, K, and V, and run several attention “heads” in parallel.

Now Break It

Try this: Raise the temperature and the softmax flattens — every word attends to every other word equally, so attention carries no information.

Control: Temperature slider (raise it high)

What happens: Attention has flattened! At high temperature the softmax becomes uniform — every word attends to every word equally, and the layer can no longer focus on what is relevant.

Where attention & transformers is used

Attention is the engine of the transformer, and transformers now dominate machine learning. They power large language models like the GPT and Claude families, machine translation, code assistants, and text summarization; the same mechanism drives vision transformers for image recognition, protein-structure models like AlphaFold, speech recognition, and multimodal systems that tie images and text together. Attention replaced recurrent networks for most sequence tasks because it looks at all positions at once, so it both captures long-range relationships and trains efficiently on parallel hardware.

A frequent misconception is that attention “understands” language. It only computes weighted averages based on learned similarity; the apparent understanding emerges from stacking many layers over enormous data. Another pitfall is forgetting the cost: because every token attends to every other, computation grows with the square of the sequence length, which is why long-context models need special efficiency tricks. Finally, self-attention alone is order-blind — it treats a sentence as a bag of tokens — so transformers add positional information separately; without it, “dog bites man” and “man bites dog” would look identical.

Frequently asked questions

What is attention in a neural network?
Attention is a mechanism that lets a model, when processing one element of a sequence, weigh how much to draw from every other element. Each token forms a weighted combination of the others based on learned relevance scores. This lets the model relate distant words, such as a pronoun and the noun it refers to.
What are queries, keys, and values?
For each token the model computes three vectors. The query represents what this token is looking for, the key represents what a token offers, and the value is the information it passes on. Comparing a query to all keys produces attention weights, and those weights combine the values into the token's output.
What is self-attention?
Self-attention is attention applied within a single sequence, where the queries, keys, and values all come from the same set of tokens. Every token attends to every token in the same sentence, including itself. It is the core operation repeated throughout a transformer.
Why divide by the square root of d in attention?
The dot products that score query-key pairs grow larger as the vector dimension increases. Without scaling, large scores push the softmax toward a near one-hot distribution with tiny gradients, making training unstable. Dividing by the square root of the dimension keeps the scores in a reasonable range so attention stays smooth and learnable.
What is a transformer?
A transformer is a neural network architecture built by stacking attention layers together with feed-forward layers, normalization, and positional encodings. It processes all positions in parallel rather than one at a time, which made it far more scalable than recurrent networks. Transformers are the foundation of modern large language models.
How is attention different from an RNN?
A recurrent network processes a sequence step by step, carrying a hidden state forward, which makes long-range dependencies hard to learn and hard to parallelize. Attention instead compares all positions directly in one operation, capturing long-range relationships easily and training efficiently on parallel hardware. This is a major reason transformers replaced RNNs for most language tasks.
← Back to all visualizationsContinue on the Learning Path →

Written & reviewed by the ML Visualization team · Last updated .