Feedback data and active learning

Mechanism: uncertainty/error → selected example → label → evaluated update

scores = [0.51, 0.92, 0.49]
print(min(scores, key=lambda s: abs(s-.5)))

Run with python3; expected output selects the most uncertain example. Feedback is not ground truth: log source, incentives, reviewer agreement, exposure bias, and label policy; assess whether sampling amplifies existing representation gaps.

Sources

• Active Learning Literature Survey — active-learning foundations.
• Data Cascades in High-Stakes AI — feedback-loop risks.

Production feedback is one of the best sources of new data, but only if it is captured, triaged, labeled, and fed back deliberately. Otherwise it becomes noisy telemetry.

Feedback sources

• User corrections, thumbs, edits, ratings, and comments.
• Human review decisions and escalation outcomes.
• Support tickets and incident reports.
• Low-confidence predictions and abstentions.
• Disagreement between model, judge, and human label.
• Drift or slice failures found in monitoring.

Active learning loop

1. Identify examples where additional labels would most improve the system.
2. Prioritize by uncertainty, product impact, risk, and slice coverage.
3. Send selected examples to human review with clear guidelines.
4. Add labels to the dataset with provenance and reviewer metadata.
5. Update train or eval sets according to purpose.
6. Re-run the relevant evals and monitor for regressions.

Feedback hygiene

Pitfall

Not all feedback should become training data. Some belongs in evals, product design, documentation, or policy changes instead.

Connects to: feedback loops · human-in-the-loop production · systematic error analysis