Evaluation & Observability

The other always-active branch answers a single question: does this actor actually help? Without trajectories, metrics, and traces you're automating on vibes. This is where my observability instinct from app work — Langfuse, OpenTelemetry — gets translated to the agent world, so a workflow earns automation instead of being assumed to work.

Evaluate before you automate. A workflow that can't show a success metric, a cost number, and a trajectory you can read isn't ready to run on cron. Watch for skill regressions: a self-refined skill can quietly get worse.

Planned notes

• Trajectories: reading what the agent actually did
• Logs and traces for an agent run
• Per-workflow success metrics
• Skill regressions: catching quiet degradation
• Cost and token accounting
• Translating Langfuse / OTel instinct to agents
• Evaluating a workflow before automating it
• Offline eval vs production monitoring
• What "good" looks like per actor
• A lightweight scorecard per workflow

Core sources

• Hermes — Session Storage (trajectories / logs) — https://hermes-agent.nousresearch.com/docs/developer-guide/session-storage
• Hermes — Agent Loop (what to trace) — https://hermes-agent.nousresearch.com/docs/developer-guide/agent-loop
• Agentic Systems — evaluation — TODO (see SOURCES.md).
• Langfuse / OpenTelemetry — observability tooling to adapt — TODO confirm canonical URLs (see SOURCES.md).

Connects to: Native Orchestration · Skills & Procedural Memory · Income Workflows · Agent Security & Ops