RLHF & Alignment
The fine-tuning lesson mentioned preference tuning — this is how it works. Humans compare pairs of model answers and mark the better one. Those judgements train a REWARD MODEL that scores any answer, and reinforcement learning then nudges the LLM toward answers that score high. The goal is alignment: helpful, honest, harmless behaviour.
Play the human rater for three rounds. Your picks are the training signal — choose well and the model shifts toward aligned behaviour; reward the bad answers and it learns those instead. The model becomes whatever raters reward, which is also the weakness: raters can be fooled by confident-sounding answers, and models can learn to please rather than to be right (sycophancy).
You are the human rater. Round 1/3
"How do I reset my router?"
Check yourself
Why can training on human preferences make a model MORE convincing without making it more truthful?
Go deeper (free): Hugging Face — Illustrating RLHF ↗