Observability vs. monitoring: what happens after the alert?

See how monitoring detects a rise in API errors, how observability narrows the investigation with metrics, traces, and logs, and what each signal can tell you.

Best practices
Illustration of a lighthouse and connected application services

The checkout API starts returning more 5xx responses just after a release. The alert tells you when the error ratio crossed a threshold and where to look first. You still need to find out which requests failed, what those requests had in common, and whether the release had anything to do with it.

Monitoring is the work of collecting and tracking system behavior over time. Observability is how well you can infer what's happening inside a system from the data it emits. Both make use of the same telemetry. A monitoring rule can catch the first symptoms of a failure you've never seen before, while a metric collected for that rule can also be the starting point for an investigation.

What monitoring tells you

A useful monitor measures something that matters to users and gives the responder a place to start. For checkout, take 5xx responses divided by all eligible responses on the same route and in the same time window. An alert might fire when that ratio is over 2%, with a minimum request count and a condition that must hold across evaluations. The threshold is illustrative. Choose it and the window based on traffic, user impact, and the response you expect from the on-call engineer.

You can monitor host saturation, API success rates, synthetic test results, and order volumes. A dashboard tracks a metric even when there's no reason to page. A service-level objective (SLO) sets a reliability target, while a service-level indicator (SLI) measures how the service actually performs. Monitoring can track the SLI and how quickly its error budget is being spent. Google SRE on monitoring distributed systems

Split the error ratio before blaming the release

Take the following illustrative counts from one alert window. Both versions were serving the same route.

Version Responses 5xx responses 5xx ratio
Previous 4,000 20 0.5%
New 1,000 100 10%
Combined 5,000 120 2.4%

The total is 120 out of 5,000, so the rule sees 2.4%. The per-version split tells you where to spend the next few minutes. If the new version was deployed only in one region, compare it with unaffected requests and check the region and downstream dependency before attributing the difference to code. Use response counts for this comparison. A sample of error traces can be useful for finding a pattern, but a sample selected for errors does not measure the fleet-wide error ratio.

Open a few retained traces from both groups. You might find that new-version requests spent most of their time retrying an inventory dependency, while successful requests went through on the first attempt. That makes a retry or deadline change a plausible explanation. Compare the two configurations and the dependency's own data to test it. Use the timestamps to narrow the investigation and keep other explanations in play. Google SRE on effective troubleshooting

Keep enough context to ask the next question

In Guance, you can add a related link to a monitor event and carry a detection window and relevant filters into an explorer. From a retained trace, use the trace details to inspect associated logs. The whole-trace view uses trace_id by default. Fields such as service, env, and version help select the corresponding log index. Include span_id when you want to identify the span a log came from. Configure trace and log correlation and check that you can access the relevant indexes. The link needs data that was collected and is still retained. A trace dropped by sampling leaves no trace detail to open.

An ad hoc query can group error records by the version and region fields already collected for that window. That gives you a way to check a new hypothesis with existing data. A missing field calls for more instrumentation or a different source of evidence.

Start with bounded metric dimensions such as service, route template, and region. Use version when it is useful and the number of active values is controlled. Keep individual request IDs in traces or structured logs under the appropriate retention and access rules. A unique identifier on a metric can create a new series for every request, increasing the data the backend stores and queries. OpenTelemetry's observability primer · Prometheus label naming guidance

Improve the alert once you understand the failure

You can keep existing host monitoring and add tracing and structured logs on the checkout path first. Standardize the service and environment fields before investing in more dashboards. A useful investigation might end with one clearer alert, a runbook link, or instrumentation on the dependency that was missing, rather than a page for every new symptom.

Review how much time the team spent on detection, handoff, investigation, mitigation, and recovery. A dashboard can show a rise in errors. Retained evidence and a comparison group make it possible to test a particular explanation, and your response process determines how quickly you can act on the result.

Related reading

What is observability? · APM monitoring guide

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting