Scaling Laws
The strangest fact of the LLM era: model quality improves as a smooth, predictable POWER LAW in compute, model size, and data. Plot loss against compute on log-log axes and you get a straight line spanning many orders of magnitude — which lets labs forecast a billion-dollar model's performance from cheap small-scale runs.
Raise the compute budget and slide down the law — each 10× of compute buys a steady drop in loss. Then break the allocation: pour everything into model size and you get an undertrained giant that falls OFF the curve; the Chinchilla result showed several early famous models made exactly this mistake, beaten by smaller models trained on more data. Scaling laws are why the industry keeps building bigger — and why data supply has become the real bottleneck.
loss: 4.52 — on the power law
On a log-log plot, loss vs compute is a straight line across many orders of magnitude — so labs can PREDICT a giant model's performance from small cheap runs, before spending millions.
Check yourself
Why would a lab trust a straight line fitted on small models to predict a model 1000× larger — and what could break that trust?
Go deeper (free): Epoch AI — Chinchilla scaling explained ↗