What is Agent observability? Traces, tools, and task outcomes

Follow an AI agent through a task, trace model calls and tool retries, and measure task success separately from execution status.

Best practices
Illustration of several robotic arms working together

A meeting-room assistant says it found an accessible room in the east wing. The request returned HTTP 200. Both model calls finished. The room it suggested was in the west wing.

When you open the trace, the first calendar lookup timed out and the retry returned a valid result. That explains the retry. It takes a separate check against the original request and the room catalogue to explain why the answer failed.

Agent observability is the practice of following an agent’s observable actions through a task and checking what the task achieved. It connects model calls, tool attempts and application state to the outcome. It can show which tool the agent called and what status it returned, provided you instrumented those steps. It cannot recover hidden model reasoning or fill in missing spans.

Read the run from the outside in

Start with the agent invocation. Under it, look for model calls, tool executions, retrieval and state changes at the points your application can observe. On the meeting-room run, the relevant sequence looks like this:

Recorded step What it tells you
Initial model call The assistant prepared a calendar lookup.
Calendar lookup, attempt 1 The tool timed out.
Calendar lookup, attempt 2 The retry succeeded and returned availability.
Final model call The assistant produced a reply.
Task check The suggested room failed the east-wing requirement.

These are example observations from a hypothetical run. The task check is an application or evaluation result, separate from the status of the model and tool spans. One failed attempt out of two describes the tool. One failed task describes what the user got.

In a single trace, use parent–child relationships and timestamps to read the operations you captured, including parallel work. A conversation can contain multiple traces. If an application resumes a task in a new trace, keep an application-owned run ID to join the two parts. When you have the originating span context and an explicit causal relationship, a Span Link can express that relationship too.

Put each field on the span that owns it

Operation A useful starting point
Agent invocation Agent or workflow name and version, duration, and a run ID. Record the conversation ID if the application supplies one and collection is permitted.
Model call Requested model and response model when available, duration, and reported token usage. Leave unavailable usage unknown.
Tool execution Stable tool name, duration, outcome and error type on failure. Record retry attempts separately if you need to count them.
Retrieval or state transition A bounded source or stage name, result count or transition outcome, and a run identifier where needed to join records.

The OpenTelemetry GenAI conventions for agent entrypoints, model inference and tool execution are in Development. They describe operations such as invoke_agent and execute_tool. Check your framework and instrumentation version for the spans and attributes they actually emit. A general HTTP integration may show a slow outbound call while missing the tool name and the retry.

Begin with metadata. Prompts, messages, tool arguments and results can contain personal data or secrets, even after summarization. Treat content capture as an explicit opt-in. Define who can see the data, remove what the investigation does not require before export, and set a retention limit. Recorded inputs and outputs can help explain a run, but the same trace is no guarantee that the run can be replayed deterministically.

Use a trace to investigate and an outcome to measure

Pick a task with a verifiable result before instrumenting more flows. For the room assistant, check the room ID against an approved test catalogue and the user’s constraints. Keep the result and evaluation-rule version alongside the run ID. That gives you a signal that stays meaningful even when you change the model or tool.

Once the first trace looks right, add dashboards for end-to-end task duration, tool attempt failures, retries and task success. Define each denominator. A run that retries twice contributes three tool attempts and one task. Group by bounded labels such as workflow version or known tool name. Keep individual run and session IDs in access-controlled trace or log queries, where they can help explain an outlier without generating a new time series for every run.

Open the same run in Guance

If the application already emits OpenTelemetry spans, send its application traces through DataKit and open the run in APM. Start at the tool timeout, follow the retry and check which model call produced the reply. For a supported framework such as AgentScope, use the separate Agent Monitoring setup. The Agent Explorer can then show collected model and tool calls by trace or session.

Bring the same meeting-room test into that workflow. The tool timeline helps explain the delay. The catalogue check tells you whether the assistant solved the task. Guance also documents output evaluation with rules, model-based judgments and manual annotation for eligible data. Use an application check when correctness depends on a system record the evaluator cannot see.

Further reading

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting