Evaluate RAG answer quality

Mental model: evaluate retrieval before generation. A fluent generator cannot ground an answer in evidence it never received.

Mechanism: fixture → component scores → root-cause experiment

Use this playbook when a RAG prototype produces plausible answers and needs a disciplined quality check before deeper tuning or release.

Inputs

• A representative set of user questions.
• Retrieved chunks for each question.
• Generated answers, citations, latency, token counts, and model version.
• Expected answer or grading rubric for each question.

Procedure

1. Pick 20-50 questions from real usage or realistic workflows.
2. Mark the source documents or chunks that should support each answer.
3. Run the current system and log query, retrieved chunks, answer, citations, latency, and cost.
4. Score retrieval first: needed evidence present, ranking quality, and context noise.
5. Score generation second: answer relevance, groundedness, citation support, and refusal behavior.
6. Label each failure as missing evidence, wrong evidence, bad synthesis, weak citation, unsafe answer, or product mismatch.
7. Group failures by root cause and choose one system change to test next.
8. Add recurring failures to the regression suite.

Output

Pitfall

Do not tune the prompt before checking retrieval. If the right evidence is missing from the context, the generator is being asked to solve the wrong problem.

Connects to: evaluating RAG systems · retriever vs generator eval · grounding and citations

Executable failure label

retrieved, supports_claim = False, False
print("retrieval failure" if not retrieved else "synthesis failure" if not supports_claim else "pass")

Run with python3; expected output is retrieval failure. Require evidence presence, citation support, answer quality, latency, and cost before promoting a change.

Sources

• RAGAS — retrieval and generation component metrics.
• BEIR — heterogeneous retrieval evaluation.
• RAG — system formulation.