Curriculum & Data Mixes
Before any training run, someone decides what the model reads: how much web text, how much code, how many books and papers. That ratio — the data mix — shapes the model's abilities as much as its size does. More code lifts programming and even reasoning; more books lift long-form coherence; but the budget is fixed, so boosting one starves another. Labs also order the data as a curriculum, commonly saving the highest-quality sources for late in training when the model retains them best. Mix ratios are among the most guarded secrets in AI.
Drag the three sliders and watch the skill bars respond. Try an all-code model, then an all-books model, then hunt for a balanced recipe — notice you can never max everything at once.
The mix is a budget: boosting one skill starves another at the same total data size. Real labs treat these ratios as closely guarded recipes, and often reorder the data too — saving the highest-quality sources for the end of training, when the model retains them best.
Check yourself
Adding more code data improves coding, unsurprisingly — but why does it also tend to improve reasoning on non-code tasks?
Go deeper (free): The Pile — what goes into training data (paper) ↗