LoRA
Full fine-tuning updates every weight of a huge model — billions of parameters, needing serious GPUs. LoRA (Low-Rank Adaptation) freezes the original weights entirely and trains only a tiny adapter beside each big matrix: two thin matrices whose product is added to the frozen one. The change a fine-tune needs is usually simple enough to fit in that thin sliver.
The frozen square is one weight matrix; the thin strips are the adapter. Press Train — only the strips learn. Slide the rank up and watch the trainable fraction: even rank 64 touches well under 5% of the parameters, and rank 8 often suffices. This is why a hobbyist can fine-tune a large model on one consumer GPU, and why one base model can serve many tasks by swapping small adapter files.
trainable: 0.07M of 17M params = 0.39%
The frozen giant never changes; only the thin strips learn. At inference their product is added to W — or merged in, costing nothing extra.
Variants & real-world flavors
QLoRA is the combination that democratised fine-tuning: freeze the base model in 4-bit and train LoRA adapters on top — a 70B model becomes tunable on a single consumer GPU. DoRA decomposes each weight update into magnitude and direction, closing most of the small quality gap to full fine-tuning. The older adapter family inserts tiny trainable layers between frozen ones instead of adding low-rank deltas; prefix and prompt tuning go lighter still, training only a handful of virtual tokens prepended to the input. The shared economics: one frozen base, many small task-specific attachments — which is why a single server can hot-swap dozens of LoRAs per request while storing the big weights once.
Check yourself
After training, the adapter can be merged into the frozen matrix by simple addition. Why does that make LoRA free at inference time?
Go deeper (free): Hugging Face — LoRA conceptual guide ↗