Esta nota todavía no está traducida, así que se muestra la fuente en inglés.
Systematic error analysis
Error analysis is where evals become engineering direction. Aggregate scores tell you whether quality changed; error analysis tells you what to fix next.
The loop
- Collect failures from eval runs, production traces, human review, and support.
- Label each failure with task, slice, root cause, severity, and fix path.
- Count failures by bucket and product impact.
- Prioritize the biggest high-impact buckets.
- Add representative cases to the regression suite.
- Re-run after each fix and compare against the baseline.
Useful failure buckets
| Bucket | Typical fix |
|---|---|
| Missing context | retrieval, chunking, query rewriting |
| Misread context | prompt, rubric, model choice |
| Unsupported claim | grounding check, abstention, citation rules |
| Bad tool call | tool schema, examples, permission design |
| Format failure | structured output schema, parser, retry |
| Unsafe answer | guardrail, policy prompt, human review |
| Cost or latency spike | routing, caching, model choice |
Slice the errors
- By user intent and task type.
- By document type, language, length, and freshness.
- By model, prompt version, retriever version, and tool version.
- By customer segment, risk level, or traffic source.
Pitfall
Do not let every failure become a one-off prompt patch. If failures cluster, fix the system component responsible. If they do not cluster, improve coverage and keep watching.
Connects to: ML error analysis · eval sets · tracing