Debug a hallucination

Mental model: an unsupported claim is a traceable failure of evidence availability, selection, synthesis, citation, or abstention policy—not automatically a model defect.

Mechanism: claim → evidence → stage diagnosis

Use this playbook when an AI answer contains unsupported, contradicted, fabricated, or overconfident claims.

Inputs

• User request, prompt, retrieved context, answer, citations, model version, and trace.
• Expected answer or source of truth.
• Any user feedback or incident report.

Procedure

1. Split the answer into factual claims.
2. Mark each claim as supported, contradicted, unsupported, or unverifiable against the available evidence.
3. Check whether the needed evidence was retrieved.
4. If evidence was missing, inspect query rewriting, chunking, indexing, permissions, and recall.
5. If evidence was present, inspect context ordering, prompt instructions, citation rules, and answer synthesis.
6. Check whether the UI or prompt forced an answer when abstention was appropriate.
7. Add the case to the eval set with expected evidence and groundedness criteria.
8. Choose one fix and re-run the relevant RAG or product eval suite.

Diagnosis table

Pitfall

Do not label every hallucination as "model problem". Many are architecture problems: missing evidence, unsafe product pressure, weak citation checks, or stale retrieval.

Executable claim ledger

claims = {"annual revenue doubled": "unsupported", "launch date": "supported"}
assert "unsupported" in claims.values()
print("requires abstention or retrieval repair")

Run with python3; expected output names the required next action. Do not release a prompt-only fix until retrieval and citation regressions pass.

Sources

• RAG — retriever-generator formulation.
• RAGAS — component-level groundedness and retrieval metrics.

Connects to: hallucination detection · why LLMs hallucinate · RAG failure modes