Positional Encoding
Attention compares every word with every other word at once — but that comparison has no built-in sense of order. Without help, “the dog bit the man” and “the man bit the dog” are the same bag of words. The fix: before attention runs, a position signal is added to each token embedding, making word #1 mathematically different from word #5.
Toggle positions off and shuffle the sentence: the model's view collapses to an unordered bag — both orders identical. Turn positions back on and each slot carries a unique signal, so order survives. Real models use sinusoidal patterns or learned vectors; recent LLMs mostly use RoPE, which rotates the query and key vectors by an angle that grows with position.
What the model effectively sees: “the dog bit the man” — order preserved, meaning intact.
✓ Position signals make every slot unique, so order carries meaning.
Check yourself
If attention already connects every word to every other, why does the model still need to be told each word's position?
Go deeper (free): Transformer positional encoding, explained ↗