Observability for Agentic Systems

Traditional observability relies on a simple premise: when software fails, it crashes, throws an exception, or returns a non-200 status code. You set up alerts for error rates, monitor memory leaks, and parse stack traces to find the broken line of code.

Agentic systems invalidate this entire model.

When a multi-agent system fails, it rarely crashes. The infrastructure remains healthy. The API returns a 200 OK. The memory usage is stable. Yet, the output is completely wrong because an agent misunderstood a tool response, entered an infinite loop, or hallucinated a fact. The system is green, but the business logic is broken.

Monitoring agent observability requires moving past simple infrastructure metrics. You need to understand the internal reasoning path of autonomous actors. This requires a specialized three-layer stack: structured logs, distributed traces, and continuous evaluations.

Why Traditional Monitoring Breaks Down

In a standard microservice architecture, execution paths are deterministic. Code follows explicit conditional branches. If a service fails, the stack trace points directly to the unhandled exception.

Agents operate via probabilistic execution. You provide a goal, a set of tools, and a system prompt. The LLM decides the execution path dynamically at runtime. A system might call three tools for one user request and five completely different tools for an identical request the next day.

Because the execution path is dynamic, a failure is rarely a syntax error. It is a reasoning error. Common failure modes include:

• ·Tool Misuse: Passing incorrectly formatted arguments to a database tool despite valid schema definitions.
• ·State Drift: Losing the original user intent across a long sequence of tool steps.
• ·Infinite Loops: Two agents passing a task back and forth without making progress, consuming tokens without throwing errors.

To diagnose these issues, you cannot just look at the final output. You must inspect the intermediate steps. If you only monitor the entry and exit points of your system, you are operating a black box.

Layer 1: Structured Logs Without the Bloat

Logs are your baseline telemetry, but logging for agents requires strict discipline. The most common mistake is dumping raw prompt and completion strings into standard log aggregators. This practice causes immediate problems. It bloats storage costs, exposes personally identifiable information (PII), and makes text search practically impossible.

Instead, treat logs as structured metadata records. Every step in an agent loop must emit a structured log containing specific execution variables:

• ·Model Identifier: The exact model version used, including pinpointed deployment tags.
• ·Token Metrics: Explicit breakdown of input, output, and cached tokens to track cost efficiency.
• ·Latency: Execution time isolated to the model call itself, separate from network overhead.
• ·Tool Invocations: The name of the tool called and a hash or sanitized version of the arguments.

{
  "timestamp": "2026-09-25T14:32:01.084Z",
  "agent_id": "customer-router-v2",
  "step": 3,
  "model": "gpt-4o-2024-05-13",
  "tokens": {
    "input": 1420,
    "output": 180,
    "cached": 512
  },
  "latency_ms": 840,
  "tool_call": {
    "name": "fetch_user_account",
    "status": "success"
  }
}

Keep raw text out of your primary log streams. If you must retain raw prompts for debugging, route them to an isolated, short-retention object store, referenced only by a unique step ID in your structured log.

Layer 2: Distributed Traces and Agent Handoffs

Logs tell you what happened at a specific moment. Traces tell you the story of how a request moved through your system. For multi-agent architectures, distributed tracing is the most critical layer of the stack.

When a user submits a request, that request might trigger a coordinator agent. The coordinator calls a search tool, evaluates the results, and hands the task to a specialist writer agent. The writer agent then calls a formatting tool.

To debug this sequence, you must visualize it as a tree of parent-child relationships. Distributed tracing represents each operation as a span. A span has a start time, a duration, and metadata attributes.

[User Request: Resolve Invoice Discrepancy] (Root Span)
 ├── [Coordinator Agent: Analyze Intent] (Child Span)
 │    └── [Tool Call: fetch_invoice_data] (Child Span)
 └── [Specialist Agent: Draft Response] (Child Span)
      └── [Tool Call: format_currency] (Child Span)

By implementing the W3C TraceContext standard, you can propagate trace identifiers across network boundaries and agent handoffs. If the coordinator agent invokes the writer agent via a message queue, the trace ID passes through the message metadata.

This allows you to track execution across independent agent runtimes. If an agent enters an infinite loop, the trace visualization makes it immediately obvious: you will see a repetitive, deeply nested pattern of identical spans. This visibility is exactly how you identify where to implement circuit breakers for agent loops to prevent runaway token spend.

Layer 3: Evaluations as Runtime Assertions

Traces show you the execution path, but they cannot tell you if the quality of the output was acceptable. That is the role of evaluations.

In traditional software, you write unit tests with assertions: assert response.status_code == 200. In agentic systems, you cannot assert text quality with a simple equality check. You must use a mix of deterministic checks and model-based evaluations.

Do not relegate evaluations entirely to post-hoc offline analysis. To run a reliable production system, you must integrate evaluations directly into your runtime pipeline and your CI/CD workflows. This approach forms the foundation of evaluation-driven development.

Deterministic Checks

Before using expensive LLM-as-judge evaluations, run fast, deterministic assertions on agent outputs:

• ·Schema Validation: Ensure tool arguments or final outputs match your expected JSON schema exactly.
• ·Regex Constraints: Verify that required identifiers, tracking codes, or formatting rules are present.
• ·Guardrail Filtering: Check for banned phrases or structural anomalies before passing data to the next step.

LLM-as-Judge Evaluations

For semantic quality, use a separate, highly capable model to evaluate the production agent's performance. The evaluation model must have a narrow, specific prompt and a structured scoring rubric. Do not ask the judge model: "Is this response good?" Instead, ask specific, binary questions:

• ·Faithfulness: Does the response contain facts not present in the retrieved context documents?
• ·Goal Alignment: Did the agent actually answer the specific question the user asked?
• ·Safety: Did the agent expose internal system prompts or unapproved system details?

Run these evaluations on a sampled percentage of production traffic to monitor drift. More importantly, run them in your CI/CD pipeline against a golden dataset of historical user interactions before deploying any change to an agent prompt or system configuration.

Navigating the Tooling Landscape

You do not need to build this infrastructure from scratch. The modern observability ecosystem relies on open standards and specialized analysis layers.

1. ·Instrumentation Standard (OpenTelemetry): Use OpenTelemetry (OTel) to instrument your application code. OTel is vendor-agnostic and widely supported. It provides the SDKs needed to create spans, manage context propagation, and export telemetry data.
2. ·Observability Layer (Langfuse, Arize, Braintrust): Route your OTel telemetry to a specialized LLM observability platform. These tools ingest your traces and format them specifically for AI workflows, displaying prompt versions, token costs, and step-by-step agent trajectories side-by-side.

By decoupling your instrumentation (OpenTelemetry) from your storage and analysis platform, you avoid vendor lock-in. If you decide to switch from one observability platform to another, you do not have to rewrite a single line of application code.

Implementation Strategy

Do not try to build the entire three-layer stack in a single deployment. Approach agent observability systematically:

1. ·Start with Traces: Instrument your core agent loops and tool calls with OpenTelemetry spans. Getting visibility into the execution tree provides the highest immediate return on investment.
2. ·Add Evals in CI: Build a dataset of 50 core user scenarios. Set up an automated evaluation script using an LLM-as-judge to run against this dataset on every pull request. This prevents regressions before they hit production.
3. ·Implement Log Sampling Last: Once your traces and CI evaluations are stable, configure structured metadata logging for production traffic. Apply sampling filters so you only store full trace details for anomalies, errors, or a small percentage of successful runs.

Instrument a single agent loop with OpenTelemetry spans this week. Map out its tool calls as child spans, look at the execution tree, and see exactly what your agent is doing before you ship it to users.