An AI hallucination is when a language model states something false or unsupported with confidence: a citation that does not exist, a wrong date, an API flag that was never shipped. It happens because models generate likely-sounding text rather than looking facts up, and it cannot be fully eliminated today, but grounding, citations, permission to say "I don't know" and automated verification reduce it a lot.
NIST's Generative AI Profile (NIST AI 600-1, July 2024) uses the term confabulation and defines it as "The production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived."
This guide covers what counts as a hallucination, what the research says causes it, and the mitigations that vendors document.
What counts as an AI hallucination?
Anthropic's guide to reducing hallucinations describes it as when a model generates "text that is factually incorrect or inconsistent with the given context." NIST's fuller description lists several forms:
- False facts stated as true.
- Output that diverges from the prompt or input, such as a summary that adds a claim the document never made.
- Self-contradiction, where the model contradicts what it said earlier in the same context.
- Confabulated logic or citations: NIST notes that "LLMs sometimes provide logical steps for how they arrived at an answer even when the answer itself is incorrect."
For developers, the common versions are invented functions, packages, config keys or CLI flags, and confident explanations of code the model did not actually read.
NIST also flags why this matters: risk arises "when users believe false content," often because the response sounds confident. The tone gives no signal that the content is wrong.
Why do LLMs hallucinate?
NIST gives the short answer: "Confabulations are a natural result of the way generative models are designed: they generate outputs that approximate the statistical distribution of their training data; for example, LLMs predict the next token or word in a sentence or phrase." Statistical prediction can produce accurate text, and it can also produce text that is fluent and wrong.
A September 2025 OpenAI paper, Why Language Models Hallucinate (Kalai, Nachum, Vempala and Zhang), goes further. Its abstract says: "language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty."
The paper splits the cause into two stages.
Pretraining. The authors argue that hallucinations "need not be mysterious" and "originate simply as errors in binary classification." Some facts follow no learnable pattern; a person's birthday cannot be inferred from anything else. For these "arbitrary facts," the paper derives that the hallucination rate after pretraining should be at least the share of such facts that appear only once in the training data. Its example: "if 20% of birthday facts appear exactly once in the pretraining data, then one expects base models to hallucinate on at least 20% of birthday facts." The paper argues that training objectives would produce errors even if the training data were error-free.
Post-training and evaluation. Training after pretraining often aims to reduce hallucinations, so why do they persist? The paper's answer is that most benchmarks grade answers as simply right or wrong. "Under binary grading, abstaining is strictly sub-optimal." A model that guesses scores better than one that says "I don't know," so models are tuned to be good test-takers.
The paper illustrates this with its own lead author: asked for Adam Tauman Kalai's birthday "if you know," one open model gave three different wrong dates across three attempts.
Can hallucinations be fixed in the model itself?
Partly, and the OpenAI paper proposes one direction. Rather than adding more hallucination benchmarks, it argues for changing the scoring of the mainstream benchmarks that dominate leaderboards, so wrong answers cost more than abstaining. It suggests stating an explicit confidence target in each evaluation prompt, for example: "Answer only if you are >t confident, since mistakes are penalized t/(1−t) points, while correct answers receive 1 point, and an answer of 'I don't know' receives 0 points."
That is a change for model builders. As an app developer, you cannot retrain the model, but you can apply the same idea: make "I don't know" an acceptable answer in your prompts and your evals.
How do you reduce hallucinations with prompts?
Anthropic's guide lists three basic strategies:
- Allow the model to say "I don't know." "Explicitly give Claude permission to admit uncertainty. This technique can drastically reduce false information." Give it an exact phrase to use, such as "I don't have enough information to confidently assess this."
- Use direct quotes for factual grounding. For long documents (over 20,000 tokens), ask the model to extract word-for-word quotes first, then answer only from those quotes.
- Verify with citations. Have the model cite a quote for every claim, then check each claim and retract any it cannot support.
And four advanced ones:
- Chain-of-thought verification: review the model's reasoning when an answer looks wrong.
- Best-of-N verification: run the same prompt several times and compare. "Inconsistencies across outputs could indicate hallucinations."
- Iterative refinement: feed the output back and ask the model to verify or expand its statements.
- External knowledge restriction: instruct the model to use only the provided documents, not its general knowledge.
The guide closes with a caveat worth repeating: "although these techniques significantly reduce hallucinations, they don't eliminate them entirely. Always validate critical information, especially for high-stakes decisions."
How does grounding reduce hallucinations?
Grounding means giving the model the source material it should answer from, instead of relying on what it memorized. Three common forms:
| Approach | What it does | Documented example |
|---|---|---|
| Retrieval (RAG) | Fetches relevant passages from your documents into the prompt | Anthropic's glossary says RAG improves "the factual accuracy of the generated text" |
| Web search grounding | Lets the model search the live web and cite results | Google's Grounding with Google Search: "Reduce model hallucinations by basing responses on real-world information." |
| Structured citations | Returns pointers to the exact passages behind each claim | Anthropic's Citations feature |
Anthropic's Citations docs say that because the API extracts cited_text directly, "citations are guaranteed to contain valid pointers to the provided documents." The pointer is guaranteed valid; whether the passage actually supports the claim still needs checking.
Grounding moves the problem rather than ending it. If retrieval returns the wrong passage, the model may answer confidently from it. Our guides to RAG and embeddings cover how to improve retrieval, and fine-tuning vs RAG explains why retrieval, not training, is the usual fix for missing facts.
How do coding agents guard against hallucinated code?
Code has an advantage over prose: you can run it. A made-up function fails the type checker, a wrong import fails the build, and a broken edge case fails a test.
Anthropic's Claude Code best practices lead with this: "Give Claude a check it can run: tests, a build, a screenshot to compare." The same page names the failure it prevents, a plausible-looking implementation that misses edge cases, and gives a blunt rule: "If you can't verify it, don't ship it."
Practical habits:
- Ask the agent to run tests, the type checker and the linter before it reports success, and to show the output.
- Before installing a package an agent suggests, confirm it exists on the official registry and is the one you expect.
- Point the agent at real docs (a URL, an MCP server or a local file) rather than letting it recall API details.
- Review diffs, not summaries. A summary can describe work that was not done.
A related but different risk is prompt injection, where instructions hidden in a web page or issue mislead the agent on purpose. See prompt injection in coding agents.
How should teams manage hallucination risk?
NIST's profile frames hallucination as a risk to manage, not a bug to close. Its suggested actions include:
- "Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."
- "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments."
- "Establish and maintain policies and procedures to record and track GAI system reported errors, near-misses, and negative impacts."
In practice that means an evaluation set with questions whose answers you know, including questions the model should decline. Score wrong answers as worse than abstentions, as the OpenAI paper suggests. Keep a human in the loop for high-stakes uses; NIST says confabulation risk is especially important to monitor in "applications involving consequential decision making."
Lowering temperature is not a fix. It makes output more conservative and repeatable, but a model can be consistently wrong.
Where should grounding prompts live?
The prompts that reduce hallucinations, such as "quote first, then answer" or "cite a source for each claim or remove it," work best when the whole team uses the same tested version. Kipt keeps prompts like these as Markdown in a private GitHub repo you own, with fill-in blanks for the document and the question, so they are a Cmd+K search away instead of retyped from memory.
What should you read next?
Read what is RAG to ground answers in your own documents, then what are embeddings to improve the retrieval step. For the agent side, prompt injection in coding agents covers the deliberate version of the same problem.