State Space Models
Attention is powerful but expensive: every token looks at every other token, so cost grows with the square of the sequence length. State space models are a newer approach that grows only linearly.
Instead of comparing all pairs, a state space model keeps a small running summary of everything seen so far and updates it one token at a time, like a smarter version of the RNN. Doubling the sequence roughly doubles the work rather than quadrupling it, which makes very long inputs cheap. Current models often mix a few attention layers with these, keeping quality while cutting cost.
every token attends to every other (15 links)
Check yourself
For a long document, why does the state space path stay affordable while pure attention balloons?
Go deeper (free): The Annotated S4 — state space models ↗