Normalization
Each layer of a deep network slightly rescales the numbers passing through it. Slightly — but compounding over 50 or 100 layers, slightly-bigger explodes to infinity and slightly-smaller vanishes to zero. Either way, training dies. Normalization layers are the thermostat: after each layer, rescale activations back to a standard range (mean 0, size 1).
Set the per-layer gain to 1.3× — harmless-sounding, yet watch the bars blow up within ten layers. Drop it below 1 and they fade to nothing. Now toggle LayerNorm on: every layer's output is re-centred and re-scaled, so depth stops compounding and the gain slider barely matters. You met LayerNorm inside the transformer block — this is the failure it prevents; without it, 100-block LLMs could not train at all.
Small per-layer drift compounds exponentially with depth. Push the gain to either end, or turn LayerNorm on.
Check yourself
Why does a tiny per-layer effect become catastrophic only when networks get DEEP?
Go deeper (free): Layer Normalization — original paper ↗