Sequence Memory
Before transformers, sequences like sentences were handled by recurrent neural networks, or RNNs. An RNN reads one item at a time and keeps a single memory that it updates at every step.
Step through the sentence one word at a time. A single memory carries everything read so far. By the end it still holds onto France, enough to predict the word French.
The network reads one word at a time, updating a single memory that carries everything so far. By the end it still remembers “France” — enough to predict “French”.
How it works
An RNN processes a sequence step by step. At each step it takes the current item and its own previous memory, called the hidden state, and combines them to produce an updated memory. That memory is passed to the next step, so information can flow forward through the sequence. This is what let RNNs handle language, speech, and time series. But there is a catch: with a single memory being overwritten each step, details from early on tend to fade by the time the network reaches the end — the long-range memory problem. Improved versions called LSTM and GRU added gates to hold onto important information for longer, but the core limitation remained, and processing had to happen in order, one step at a time, which is slow. Transformers replaced RNNs by using attention to let every word see every other word at once, no memory bottleneck and no forced sequential order.
Check yourself
What might happen to the memory of an early word in a very long sentence?