A model can tell you something with complete confidence that turns out to be completely wrong. People call it hallucination, and it stems from the same trick that makes the model useful in the first place: it predicts words, it doesn't remember facts. Knowing that one fact changes how you use the thing.
About 2 min of reading
A model can tell you something with full confidence that is simply wrong, because it predicts words and never remembers facts. I show a confident lie, count it as made up, and label which answers deserve a check before I trust them.
I print a plausible fact the model states, then note that it landed on the truth by pattern alone and has no memory to back it up.
Before you run it, say out loud what it will print. Then open the answer.
I run a claim against a trusted source and print that it was made up, showing the kind of confident nonsense a model produces when nothing exists to check against.
Before you run it, say out loud what it will print. Then open the answer.
I label a draft and a date differently so it is clear that ideas are safe to trust while anything with a fact must be verified first.
Before you run it, say out loud what it will print. Then open the answer.
Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.
Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.
An LLM doesn't look facts up in a database. It predicts text that looks correct, and sometimes that text is wrong even when it sounds certain. Confidence is not evidence.
For dates, numbers, or advice that someone will act on, always verify against a reliable source. Trust the model for ideas and drafts, never for verified facts. A model is a great starting point and a terrible one for the final word.
Prompt: "Who wrote Romeo and Juliet?"
Model: "Shakespeare wrote it in 1597."That's right here, but the model has no memory at all, it only predicts words. It landed on a true answer by pattern, the same way it could land on a confident lie a moment later. Truth and fabrication come from the same source.
You: "Name the first president of Australia."
Model: "Sir Harold Blinker Sr., elected 1847."There's no fact to check against, so the model generates something that just sounds right. That's **hallucination*: and the worst ones sound completely convincing, which is why you can't trust a model on the things that actually matter.
Ideas, drafts, explanations: trust the model.
Dates, numbers, medical advice: verify it.Use the model for creative work and rough drafts, where a good first attempt is the whole point. For anything that matters, a date, a figure, advice someone will act on, confirm the fact with a reliable source before you do anything.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.