NinA AI

Blog · AI & Technology

Why AI Language Models Hallucinate: A Plausible-Sounding Problem

Olaf Lemmens, Founder NinA AI Agency · September 13, 2025 · 8 min read

An early morning in Amsterdam. While drinking my coffee, I quickly check if ChatGPT can help me with a new idea. The answer seems perfect. Until I realize things in it aren't right.

And that happens more often than you think.

I recently gave a workshop and someone told me about their own experience. A researcher who shared a study with CoPilot, the question? How many respondents is this?

50, said CoPilot confidently. The correct answer.

But then he asked again. "are you really, really sure??"

"You're right, it's 49."

AI lied.

The OpenAI paper I read this week shows why that happens. And more importantly: what you can do about it.

TL;DR:

  • •Language models hallucinate because they'd rather guess than say "I don't know."
  • •The cause lies in how they're trained and tested.
  • •You can reduce hallucinations with smart prompts.

What is a hallucination, really?

A hallucination is a plausible-sounding untruth. The paper models this soberly: see a model as a classifier that must determine whether a possible output is "valid" or "wrong." If it can't perfectly learn that separation, you get invalid (but believable) sentences back. That's not mysticism, but basic statistics.

GPT-4 calibration histograms before and after reinforcement learning

AI's guessing game

OpenAI compares language models to students during a multiple-choice exam. If you don't know the answer, you have three options:

  1. 1Leave it blank
  2. 2Say "I don't know"
  3. 3Just guess

What do most students do? Guess. Because there's no penalty for wrong answers, only points for correct ones. Language models work the same way: the training system rewards them for always saying something, even if it's nonsense.

Where training goes wrong

Even with a perfectly error-free dataset, errors still occur. Why? Because statistically, the model can never know everything.

🎲 Arbitrary facts

Things that appear only once in training (like obscure birthdays). Here a model must guess.

🤖 Poor models

Older systems like trigram models structurally failed at simple tasks. Modern models are smarter, but still fallible.

🗑️ Garbage in, garbage out

If there are errors in the training data, the model happily adopts them.

Why the problem persists

Most benchmarks evaluate answers binary: right or wrong. There's no reward for doubt or caution. So the model keeps bluffing.

That means: as long as evaluation is set up wrong, models will keep hallucinating.

What you should do about it

I often notice clients see ChatGPT as a "truth machine." But it's not. It's a generator of plausible sentences. That sounds less sexy, but is more honest.

Tips to reduce hallucinations in your prompts:

  1. 1Explicitly ask for "I don't know": add "if you're not sure, say: I don't know."
  2. 2Demand evidence: ask for the source or step-by-step reasoning.
  3. 3Compare multiple runs: ask the same question three times and check consistency.
  4. 4Use retrieval (RAG): connect the model to your own reliable data.

Useful prompt templates

🎯 Confidence threshold + penalty

"Only answer if you're >75% sure. A wrong answer weighs more heavily than 'I don't know'. If in doubt, explicitly say: 'I don't know' and explain what you need to know."

📎 Demand evidence & limit scope

"Limit yourself to [source/dataset/scope]. Name your source or evidence in 1 line. If evidence is missing within this scope, say 'I don't know within this scope'."

📋 Answer format with 'IDK' option

"Answer in this JSON schema: {'answer': string, 'certainty': 'high|low|idk', 'reasoning': string, 'what_am_i_missing': string}. Use 'idk' if you're <75% sure."

❓ Verify when in doubt

"If you're <75% sure: ask max 2 clarification questions or provide a checklist of what you need to verify. Don't answer yet."

🔒 Obscure facts? Make guessing expensive

"For fact questions about a person/unique record: only answer with 'I don't know' unless you can cite the exact match. No guess-dates or -titles."

Conclusion

Hallucinations aren't an AI glitch — they're a direct consequence of how we build and evaluate models. We'll never completely eliminate them, but we can be smarter about how we use AI.

"Anyone who uses AI as if it's always right will eventually get burned. Anyone who sees AI as a smart but sometimes overconfident colleague will get maximum benefit from it."

So I'm curious: how do you deal with hallucinations in your prompts?