KV Cache & Inference
Generation is one token at a time, and attention needs every earlier token's key and value vectors at every step. Recomputing them from scratch for each new token would make step N cost N times the work — quadratic waste, since those vectors never change. The KV cache fixes it: store each token's keys and values the first time they are computed, and each new token only computes its own, then reads the rest from memory. The trade is speed for GPU memory — the cache grows with every token, which is the real reason long contexts are expensive.
Watch tokens generate. With the cache on, past tokens sit stored and each step does one column of work; turn it off and every step recomputes the entire past — compare the total work counter.
Check yourself
The KV cache makes generation much faster. Why does a very long conversation still slow down and cost more, even with the cache on?
Go deeper (free): Hugging Face — KV caching explained ↗