Product metrics for AI
Mental model: a model metric becomes a product metric only when it predicts a user outcome under the real workflow. Instrument the chain: eligible user → request → model/tool trajectory → user action → verified outcome → harm, correction, or support cost. Optimize the narrowest metric that safely moves that outcome.
Mechanism: outcome hypothesis → leading indicators → release decision
State one outcome hypothesis, such as “grounded answer reduces support reopens.” Pair it with leading indicators—evidence coverage, successful completion, abstention correctness, p95 latency, cost per successful task—and guardrails: unsafe action, appeal, over-reliance, and disparate error rates. A change ships only when outcome and guardrails clear their thresholds.
completed, eligible, unsafe, cost = 82, 100, 1, 18.0
print("success", completed/eligible, "unsafe_rate", unsafe/eligible, "cost_per_success", cost/completed)
Run with python3; expected output separates success, harm, and unit economics. An engagement increase alone may mean users are confused, entertained, or trapped in retries.
| Layer | Example metric | Decision it supports |
|---|---|---|
| User outcome | task completed or correction avoided | does the feature help? |
| Behavior | evidence cited, tool success, abstention | why did it help or fail? |
| Experience | p95 time-to-useful-result, edit rate | can people use it? |
| Operations | cost/success, retry rate, incident rate | can it scale safely? |
| Trust | appeal, override, complaint, opt-out | should authority widen? |
Failure modes and decision rule
Do not optimize clicks, tokens, or thumbs-up without a task-level counterfactual. Segment metrics by workflow and affected group, retain traces needed to investigate, and treat a safety or trust regression as a release blocker even when the aggregate outcome improves. Revisit metrics when the product boundary, model, or user behavior changes.
Exercises
- Define one outcome metric and two guardrails for a drafting assistant.
- Add a retry rate to the artifact and compare request cost with successful-task cost.
Connects to: product evals · tradeoffs · feedback loops · recourse
Sources
- Google Rules of ML — product and system measurement sequencing.
- The ML Test Score — production-readiness evidence.
- NIST AI RMF — outcome, risk, and monitoring framing.