What is LLM observability? Latency, token usage, and evaluation

Measure LLM latency and token usage, estimate cost without confusing it with billing, and evaluate outputs separately from traces.

Best practices
Illustration of several robotic arms working together

A support assistant returns HTTP 200, but the answer cites an outdated return policy. The model was reachable. The request may even have been fast. To see why the answer was wrong, you need the model call and retrieval path, plus an evaluation that checks the answer against the current policy.

LLM observability connects those pieces. It measures model calls, timing, errors, and token usage, then ties that evidence to an independently assessed outcome. Application traces help explain how the answer was produced. Evaluation tells you whether it met the task.

Measure the part of the wait you can see

Consider a synthetic streamed request. The client receives a metadata-only event at 180 ms, the first chunk with new model output at 240 ms, and the first visible text at 320 ms. The model call finishes at 2.4 s. The application finishes at 4.2 s. The first event, first output, and full response describe different points in the request.

Time to first token (TTFT) needs an observable token boundary and a stated start point. If your client sees chunks, label the metric by what you measure. OpenTelemetry's current GenAI client inference convention measures time to the first qualifying decoded output chunk. An empty or metadata-only event does not count. A qualifying chunk can contain reasoning or tool-call arguments before any user-visible text. The convention recommends omitting the metric for non-streaming calls and requires omitting it when no qualifying chunk arrives. GenAI client inference conventions

Track the model call and end-to-end application duration separately. Retrieval, retries, tool calls, and rendering may account for the rest of the wait. Compare percentiles for the same operation and streaming mode, and use the service's SLO to set alert thresholds.

Keep token usage separate from the bill

Start with the usage the provider actually returns. Keep the model, service tier, input, output, and any cache categories the API reports. Some payloads include cached tokens within total input. Others report uncached input, cache reads, and cache writes as separate buckets. Check the response contract before adding the fields. Mark unavailable usage as unknown and distinguish any local estimate from provider-reported usage. OpenAI prompt caching · Claude prompt caching

Suppose a synthetic response reports 10,000 input tokens, including 6,000 cached, and 2,000 output tokens. At illustrative rates of $2, $0.50, and $8 per million uncached input, cached input, and output tokens respectively, plus $0.01 for a tool call, the estimate is $0.037. The rates are invented for this example. Real pricing can also include tool calls, fixed token blocks, capacity or flat subscription charges. Reconcile estimates against the invoice for the actual model, tier, account, and period. Sampled traces are useful for investigation. They leave out calls and should not serve as the sole usage ledger. OpenAI API pricing · Amazon Bedrock provisioned throughput

Evaluate the answer alongside the trace

In the support example, a trace can show which document was retrieved, which prompt version was used, and whether the model timed out. A test with a current reference answer can tell you whether the reply used the right policy. In production, task completion and user feedback help identify cases to inspect. Feedback is self-selected, so a thumbs-down rate is not an accuracy rate.

For regression checks, keep an evaluation set and a versioned rubric. Use deterministic checks for fields and formats when they are enough. When you add an LLM judge, calibrate it against representative human-reviewed cases before treating its score as a release signal. Compare prompt or model versions on the same set and separately track the production cohort.

Start with metadata and add content deliberately

The GenAI inference spans, metrics, and events cited here were marked Development in the dedicated OpenTelemetry repository on November 10, 2026. Check what your instrumentation actually emits before adopting a dashboard's field names or units. Record the model, timing, status, and relevant versions first. Keep request identifiers in trace context where needed, rather than turning them into metric labels. OpenTelemetry GenAI repository · GenAI event conventions

Leave prompt, response, and tool payload capture off by default. If a narrow debugging case warrants it, enable the relevant signal explicitly, remove sensitive fields before export, and restrict access and retention. Sampling limits volume. It does not remove secrets from the samples you keep. GenAI content capture settings

Guance LLM Monitoring supports a Langfuse integration and the documented LangChain OpenTelemetry integration. Start with a request whose expected result you know. Check the model, duration, and token usage actually reported, and inspect the call hierarchy if tool spans are present. Use APM for ordinary application traces. Relating a tool span to a downstream request requires instrumentation and trace context across the boundary. Evaluate the answer separately and reconcile estimated cost with billing records. Set up Guance LLM Monitoring · Guance APM

Build a RAG application with Python · What is Agent observability? (Simplified Chinese)

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting