AI Observability Is More Than a Token Bill

When a traditional service fails, we usually inspect status codes, latency, and stack traces. Those metrics still matter when an AI application fails, but they are often not enough to answer the most important question:

Why did the model make this choice?

The answer does not live in a single log entry. It lives across the entire path from the user’s goal and context assembly to model decisions, tool execution, and the final result.

First, distinguish four kinds of failure

If every failure is recorded as “the model performed poorly,” the team cannot take specific action. At minimum, we should distinguish:

1. Input failure

The user’s goal is incomplete, or the system did not receive the context required to finish the task. Upgrading the model rarely helps here.

2. Reasoning failure

The context is correct, but the model chooses the wrong path, misses a constraint, or reaches a conclusion too early.

3. Tool failure

The model makes the right choice, but execution fails because of permissions, networking, parameters, or external state.

4. Delivery failure

The internal work is complete, but the final response does not explain the result, limitations, or next step clearly.

These four problems need different owners: product, prompt design, platform engineering, or interaction design. The first value of observability is returning a problem to the right person.

Every run needs a shared trace

Each user request should have a stable trace_id that connects model calls and tool execution:

type AgentEvent = {
  traceId: string
  spanId: string
  parentSpanId?: string
  type: 'model' | 'tool' | 'handoff' | 'result'
  startedAt: string
  durationMs?: number
  status: 'running' | 'ok' | 'error'
}

A tool call is the external expression of a model decision. Recording the tool output without the reason for the call loses the decision context. Saving the prompt without the tool result makes it impossible to know whether execution matched the plan.

Observation does not mean unlimited collection. Raw prompts may contain personal information, credentials, or business data. A safer approach is to:

• Record structured metadata by default.
• Redact content at the field level.
• Give debugging samples a short retention period.
• Let users and businesses explicitly opt out of training or analysis.
• Never store secrets or complete identity data in logs.

Quality metrics must stay close to the task

Tokens, latency, and model error rates are platform metrics. They cannot tell us whether the user received the result they needed.

More meaningful metrics for a code-repair agent might include:

“Useful tool calls” matter more than raw call count. One accurate search can replace ten speculative reads.

Turn evaluation into an everyday feedback loop

Offline benchmarks can compare model versions, but they cannot cover your real workflow. A sustainable evaluation set comes from high-value production samples:

1. Collect runs that are clearly successful or clearly unsuccessful.
2. Remove private information while preserving the decision structure.
3. Label the type of failure instead of assigning only one overall score.
4. Rerun the cases when prompts, models, or tools change.
5. Compare result quality, cost, and latency together.

An evaluation set does not need to be large at the beginning. Twenty cases with clear boundaries and stable reproduction usually guide engineering better than two thousand samples with uncertain origins.

Observability for people

The final layer of observation is not on a dashboard; it is in the product interface.

When an agent is reading files, waiting for approval, or deploying a release, the user needs to know which stage it has reached. A good status update does not dump internal reasoning. It explains:

• What is being completed now.
• What has already been confirmed.
• Whether the user needs to take action.
• Which risks remain open.

This visibility directly changes how reliable the system feels. A task that takes tens of seconds but shows clear progress often inspires more confidence than an opaque result that appears suddenly after ten seconds.

Explainable does not mean exposing everything

An AI system does not need to show every internal token to the user, and it should not copy sensitive context into logs. What it needs is a sufficiently clear causal chain:

User goal
  → which context was used
  → which tool was selected
  → what status the tool returned
  → what result was ultimately delivered

When that chain can be queried, compared, and reproduced, a team can turn “occasionally brilliant” into “reliably trustworthy.”