Mixture of Experts
Making a model smarter usually means more parameters, but running every parameter for every token is expensive. Mixture of Experts keeps the model huge while only using a slice of it each time.
Instead of one big feed-forward block, the layer holds many smaller experts. A small gating network looks at each token and routes it to just a couple of them; the rest stay dark. So a model can hold hundreds of billions of parameters (large capacity) yet only compute a few billion per token (cheap to run). This sparsity is why most current open-weight models are MoE.
token: def → experts 0, 3
Check yourself
The model has six experts but only two light up per token. How does that give big-model quality at small-model cost?
Go deeper (free): Hugging Face — Mixture of Experts explained ↗