Esta nota todavía no está traducida, así que se muestra la fuente en inglés.
Human evaluation
Human evaluation is still the reference point for subjective, high-stakes, or ambiguous AI behavior. But "ask people what they think" is not an eval; it is a survey unless the review process is designed.
What humans are best for
- Calibrating LLM judges against trusted labels.
- Reviewing high-impact decisions and safety failures.
- Judging tone, usefulness, nuance, and domain appropriateness.
- Discovering new failure modes that the rubric did not include.
- Resolving cases where the evidence is ambiguous.
Review design
| Element | Good practice |
|---|---|
| Rubric | small number of criteria with examples |
| Instructions | define pass, fail, partial, and abstain |
| Sampling | mix random production samples with targeted edge cases |
| Calibration | reviewers grade the same seed set and discuss disagreements |
| Quality control | measure inter-rater agreement and audit outliers |
Avoiding noisy labels
- Hide model identity when comparing models.
- Randomize answer order in pairwise comparisons.
- Keep the reviewer focused on one criterion at a time.
- Capture rationale for failures, not only a score.
- Track reviewer drift over time.
Pitfall
Human eval is expensive, so teams often under-specify it. That wastes the expense. Ten carefully calibrated reviews are more useful than a hundred inconsistent opinions.
Connects to: LLM-as-judge · human-in-the-loop production · task-specific evals