Stand up LLM observability

Mental model: a trace is the join key between user outcome, model behavior, tool actions, cost, and safety controls. If an incident cannot be replayed with protected evidence, it cannot be reliably debugged.

Mechanism: trace → protected evidence → alert → eval fixture

Use this playbook before the first serious release of an LLM feature. If you cannot see prompts, context, outputs, tool calls, cost, latency, and feedback, you cannot debug or evaluate the system reliably.

Inputs

• Product workflows and model call sites.
• Privacy and retention policy.
• Eval and incident-review needs.
• Cost and latency budget.

Procedure

1. Assign a trace ID to every user request and downstream model call.
2. Log model, prompt version, input metadata, retrieved context IDs, output, tool calls, errors, latency, token counts, and cost.
3. Redact or mask sensitive fields before storage where required.
4. Capture user feedback, human review labels, and support escalations against the trace ID.
5. Add dashboards for volume, error rate, cost, latency, guardrail decisions, and quality signals.
6. Make traces replayable in staging for prompt, retrieval, and model changes.
7. Define retention windows and access rules for traces.
8. Add alerting for cost spikes, tool errors, safety failures, and quality drops.

Minimum dashboard

Pitfall

Observability that stores everything forever creates a privacy problem. Trace what you need, protect it, and set retention deliberately.

Connects to: LLM observability and tracing · feedback loops · privacy governance

Executable trace contract

trace = {"id":"t1", "prompt":"p7", "model":"m2", "latency_ms":420, "cost":.003, "outcome":"pass"}
assert all(trace[k] is not None for k in trace)
print("trace is replayable")

Run with python3; expected output is trace is replayable. Redact before storage, enforce retention, and retain access audit logs.

Sources

• OpenTelemetry specification — trace and span semantics.
• NIST AI RMF — measurement and monitoring context.