Like students guessing on a tough exam, AI chatbots often bluff when they don’t know the answer.
The result? Plausible sounding but completely false statements – what researchers call hallucinations – that can mislead users and undermine trust.
Despite steady progress in AI, these hallucinations remain stubbornly present in even the most advanced systems, including GPT-5.
Now, a new paper from OpenAI argues that hallucinations are not strange side effects of machine intelligence, but predictable statistical errors built into how large language models (LLMs) are trained – and, crucially, how they’re tested.
Fixing them, the researchers say, will require a rethink of the benchmarks that drive AI development.
Hallucinations Baked In
At their core, language models are probability machines. They don’t “know” truth from falsehood the way humans do.
Instead, they predict which words are most likely to follow others based on patterns in their training data.
For example, when asked about the title of paper co-author Adam Tauman Kalai’s Ph.D. dissertation, a “widely used chatbot” confidently gave three different answers. All wrong.
The researchers then asked about his birthday, and got three more answers. Also all wrong.
OpenAI formalises this problem with what it calls the Is-It-Valid (IIV) test. In essence, it reduces text generation to a binary classification problem: is a given string valid or invalid?
The maths shows that if a model struggles with this classification, it will necessarily produce hallucinations during generation.
“The model sees only positive examples of fluent language and must approximate the overall distribution,” the researchers explained.
“All base models will err on inherently unlearnable facts. For each person there are 364 times more incorrect birthday claims than correct ones.”
For “arbitrary facts” with no learnable patterns, the error rate bottoms out at a stubbornly high level.
In short: hallucinations aren’t bugs, they’re essentially baked into the statistical foundations of language modelling.
Why Hallucinations Persist After Training
The paper argues that post-training – the fine-tuning process where models are adjusted with human feedback – often makes hallucinations worse because of how success is measured.
Much like students on a test, leaving a question unanswered guarantees failure, whereas guessing opens a chance to earn points.
Language models face the same incentives when evaluated on accuracy-based benchmarks. Saying “I don’t know” is penalised as much as being wrong, while guessing might look correct.
OpenAI calls this an “epidemic of penalising uncertainty.” Imagine two models:
- Model A only answers when it’s confident, abstaining when uncertain.
- Model B always gives an answer, even when guessing.
On today’s benchmarks, Model B will outperform Model A – not because it’s more accurate overall, but because the scoring system rewards boldness over caution.
