Agent Observability: Monitoring Decisions, Not Requests

Summary

Agent observability captures the sequence of decisions an agent makes between model calls, not just request-level metrics, and is distinct from monitoring (health) and evaluation (quality). Unlike deterministic database systems, an agent can succeed at every step yet fail overall—for example, looping forever on a failed fix while spans report success. The primary artifact is a trace, a parent-child tree of spans covering agent invocations, model calls, tool executions, and memory reads and writes. OpenTelemetry GenAI conventions define span types like create_agent and execute_tool, but the standards are still in development and change frequently; vendor support includes Datadog, Honeycomb, and frameworks like LangChain and CrewAI. A reasoning trace layer on top of operational telemetry explains why an agent chose that path; in multi-agent systems the trace must span the full coordination chain, not one agent’s output.

Takeaways

1. 1

   Agent observability isolates three distinct jobs: monitoring (system health), observability (reconstructing what the agent did and why), and evaluation (judging quality).

2. 2

   An agent can succeed at every individual step yet fail overall, as in a loop that never converges while individual spans report success; traditional request-level monitoring cannot detect this failure class.

3. 3

   OpenTelemetry GenAI semantic conventions define agent span types (create_agent, execute_tool) and are supported by vendors like Datadog and frameworks like LangChain; spans must carry attributes beyond name and duration to enable cross-trace queries.

FAQ