Sequence Memory
Before transformers, sequences like sentences were handled by recurrent neural networks, or RNNs. An RNN reads one item at a time and keeps a single memory that it updates at every step.
Step through the sentence one word at a time. A single memory carries everything read so far. By the end it still holds onto France, enough to predict the word French.
The network reads one word at a time, updating a single memory that carries everything so far. By the end it still remembers “France” — enough to predict “French”.
How it works
An RNN processes a sequence step by step. At each step it takes the current item and its own previous memory, called the hidden state, and combines them to produce an updated memory. That memory is passed to the next step, so information can flow forward through the sequence. This is what let RNNs handle language, speech, and time series. But there is a catch: with a single memory being overwritten each step, details from early on tend to fade by the time the network reaches the end — the long-range memory problem. Improved versions called LSTM and GRU added gates to hold onto important information for longer, but the core limitation remained, and processing had to happen in order, one step at a time, which is slow. Transformers replaced RNNs by using attention to let every word see every other word at once, no memory bottleneck and no forced sequential order.
Variants & real-world flavors
Vanilla RNNs suffer worst from fading memory, so practice moved to gated variants. LSTM (1997) keeps a separate cell state guarded by forget, input, and output gates — the network learns what to keep, what to overwrite, and what to reveal. GRU simplifies to two gates with nearly the same power and less compute. Bidirectional RNNs run the sequence both ways so each position sees past and future context — fine for analysis, impossible for generation. Seq2seq stacked an encoder RNN and a decoder RNN for translation, and its central weakness — squeezing a whole sentence through one fixed vector — is exactly what the attention mechanism was invented to fix, before attention outgrew RNNs entirely.
Check yourself
What might happen to the memory of an early word in a very long sentence?
Go deeper (free): Karpathy — The Unreasonable Effectiveness of RNNs ↗