Attention
Attention lets a model look at every other word when processing one word, and decide how much each one matters. It is the core idea behind transformers and modern language models.
Click any word to make it the query. The lines fan out to every other word, and their thickness shows how much attention the query word pays to each.
Click any word in the bottom row to make it the query. Line thickness shows how much attention “sat” pays to each other word.
How it works
Attention lets each word gather information from every other word. For every word the model builds three vectors: a query (what am I looking for), a key (what do I offer), and a value (what I will pass on if chosen). To process one word, its query is compared against every word's key by a dot product, giving a raw score for each pair. Those scores are turned into weights that add up to one using softmax — the fan of lines you see, where thickness is the weight. Each word's value is then blended together in those proportions to produce the word's new, context-aware representation. This all happens for every word at once, in parallel, which is why transformers are so much faster to train than the step-by-step RNNs. Real models run several attention patterns side by side, called multi-head attention, so different heads can track different kinds of relationships — one might follow grammar, another might link a pronoun to what it refers to.
Check yourself
When a word is the query, why is its line to itself the strongest but hidden here?
Go deeper (free): Jay Alammar — The Illustrated Transformer ↗