Product metrics for AI

Mental model: a model metric becomes a product metric only when it predicts a user outcome under the real workflow. Instrument the chain: eligible user → request → model/tool trajectory → user action → verified outcome → harm, correction, or support cost. Optimize the narrowest metric that safely moves that outcome.

Mechanism: outcome hypothesis → leading indicators → release decision

State one outcome hypothesis, such as “grounded answer reduces support reopens.” Pair it with leading indicators—evidence coverage, successful completion, abstention correctness, p95 latency, cost per successful task—and guardrails: unsafe action, appeal, over-reliance, and disparate error rates. A change ships only when outcome and guardrails clear their thresholds.

completed, eligible, unsafe, cost = 82, 100, 1, 18.0
print("success", completed/eligible, "unsafe_rate", unsafe/eligible, "cost_per_success", cost/completed)

Run with python3; expected output separates success, harm, and unit economics. An engagement increase alone may mean users are confused, entertained, or trapped in retries.

Failure modes and decision rule

Do not optimize clicks, tokens, or thumbs-up without a task-level counterfactual. Segment metrics by workflow and affected group, retain traces needed to investigate, and treat a safety or trust regression as a release blocker even when the aggregate outcome improves. Revisit metrics when the product boundary, model, or user behavior changes.

Exercises

1. Define one outcome metric and two guardrails for a drafting assistant.
2. Add a retry rate to the artifact and compare request cost with successful-task cost.

Connects to: product evals · tradeoffs · feedback loops · recourse

Sources

• Google Rules of ML — product and system measurement sequencing.
• The ML Test Score — production-readiness evidence.
• NIST AI RMF — outcome, risk, and monitoring framing.