Distillation
Quantization shrinks a model's numbers; distillation shrinks the model itself. A large TEACHER model labels data for a small STUDENT — but instead of just its final answer, the student trains on the teacher's full probability distribution over every option. Those soft targets are called dark knowledge.
Toggle the training signal. A hard label gives one bit: husky. The teacher's soft probabilities reveal its whole worldview — husky 62%, wolf 31%, cat 6%, car 1% — so every training example teaches the student how classes RELATE, not just which one wins. That density of signal is why a 1B student can capture a surprising share of a 70B teacher's ability.
input: a husky photo
teacher (70B)
A hard label says only “husky, 100%”. Correct, but it hides everything the teacher knows about how classes relate. The student must rediscover all of that alone.
This is how small fast models inherit most of a giant's skill — the pattern behind many efficient models on phones and laptops.
Check yourself
Why does a student learn faster from “62% husky, 31% wolf” than from “husky” — what extra information is in the wrong answers?
Go deeper (free): Distilling the Knowledge in a Neural Network (paper) ↗