Hallucination
A model can state something false with the exact same confidence it uses for true facts. This is hallucination, and it happens because the model predicts plausible text, not verified truth.
At its core the model picks the most likely next tokens. When a fact is common in training data, the likely continuation is also the correct one. When the answer is rare, missing, or fictional, there is no true continuation to reach for, so it produces something that merely looks right: a plausible name, a well-formatted citation, a confident guess. It has no built-in sense of not knowing. Retrieval and tools are the usual fixes.
well-represented in training → confident and correct
Check yourself
The model answers Paris and the invented citation with equal confidence. Why can it not tell which one it actually knows?
Go deeper (free): Lilian Weng — Extrinsic hallucinations in LLMs ↗