Metrics, monitoring, and alerting: an SRE guide

Understand what metrics measure, how monitoring turns them into a useful view of a system, and which alerts deserve an immediate response. Includes collection, dashboards, and Guance monitor configuration.

Best practices
Abstract illustration of monitoring data and application services

A CPU reading of 80% does not tell an on-call engineer whether to act. The workload, the recent trend, and what users are experiencing change that judgment. Metrics provide the measurements; monitoring makes them useful over time; alerting brings conditions that need a response to someone's attention. Understanding those roles helps you collect data for incident investigation and capacity planning, and decide when to involve a person.

What are metrics?

A metric is a numerical measurement of system behavior or state, or an aggregation of those measurements. It might record CPU utilization at a particular moment, count requests over an interval, or describe a distribution of request durations. Metrics reduce a large amount of activity to data that can inform a diagnosis or decision. The OpenTelemetry metrics data model includes several representations, rather than treating every metric as a single instantaneous value.

Three points matter in practice:

  • Metrics need a time dimension. A single reading says little. Repeated observations reveal patterns, changes, and trends. CPU at 80% right now is less informative than a week of readings showing it reaches 95% every day at 2 p.m.
  • Metrics need context. An 80% CPU reading may be normal for a batch worker and a warning sign for a latency-sensitive API gateway. Compare it with the workload, historical behavior, and service objectives.
  • Metrics should support a decision. Useful measurements help you decide whether to investigate, scale a service, change code, or leave a healthy system alone.

Which metrics should you collect?

A system has several layers, and each exposes a different kind of failure. Plan collection across those layers so an application problem does not leave you guessing about the infrastructure beneath it.

1. Host metrics

Start with the health of individual machines: CPU utilization, memory, disk space and I/O, process state, network throughput, and network errors. Node Exporter, Telegraf, and Guance's DataKit can collect host data. Check which collectors are enabled and which metrics actually arrive before adding measurements for your workload.

2. Application metrics

Application metrics describe what the software is doing: request rate, error rate, response time and latency distributions, queue depth and processing time, and time spent calling dependencies.

These measurements can come from instrumentation or existing integrations. Prometheus client libraries and OpenTelemetry SDKs let an application expose metrics from its code. An application performance monitoring (APM) integration can also generate request traces automatically for supported operations. Automatic instrumentation still requires installation and configuration, and its coverage depends on the language, runtime, framework, and version. Custom business operations may need explicit instrumentation. OpenTelemetry zero-code instrumentation describes the available approaches.

3. Infrastructure metrics

Collect data from the components supporting the application: containers and Kubernetes clusters, database connections, slow queries and replication lag, cache hit rates, load balancers, and message backlogs. Exporters or collection plugins often provide this data, but their presence does not mean every useful measurement is enabled by default.

4. External dependency metrics

Payment gateways, messaging providers, and cloud APIs can affect your users even when your own services are healthy. Monitor availability, errors, request latency, and quota consumption. The goal is to establish which side of the dependency boundary needs attention, rather than spending an incident investigating the wrong system.

What is monitoring?

Monitoring continuously collects, processes, displays, and analyzes system data. That data can include metrics, logs, and external availability checks. A typical metrics monitoring path has four stages:

  1. Collection. Pull or receive data from hosts, applications, middleware, and external dependencies.
  2. Storage. Use a time-series storage system sized for the actual number of series, ingestion rate, and query range. Unbounded label growth remains a capacity problem even with a time-series database.
  3. Visualization. Turn measurements into views an engineer can scan quickly. A useful ops dashboard makes its service, environment, and time range explicit. Start with request errors and latency, then drill into the relevant instances or dependencies. Putting every available chart on one screen makes that path harder to follow.
  4. Analysis. Compare behavior with a baseline, identify unusual patterns, and examine relationships. Software can narrow the search, but two metrics changing together does not establish causation.

What is alerting?

Monitoring gives people a way to inspect a system. Alerting brings a condition to their attention when it needs a response. Getting that distinction right takes work: too few alerts leave gaps, while too many make important notifications easy to miss.

Some principles help:

  • Reserve pages for conditions that need action now. A problem that can wait may belong in a ticket. A self-healing fluctuation that requires no intervention, or a purely informational event, usually belongs on a dashboard or in an event record.
  • Base urgent notifications primarily on user-visible symptoms. “Request errors exceed 2%” is an example, not a universal threshold; the service objective, request volume, and evaluation window still matter. High CPU alone usually does not justify waking someone when service is unaffected. An actionable warning that a disk will soon fill can still be valuable. Google's SRE monitoring guidance discusses both symptoms and approaching saturation.
  • Give actionable alerts a runbook. Include the relevant queries, initial checks, and escalation path. A runbook helps the responder make a judgment; it cannot make that judgment for them.
  • Route by urgency. A team might use a phone call for P1, instant messaging for P2, and a ticket for P3. Define those levels and channels explicitly rather than treating the names as a universal standard.

Alert fatigue has an operational cost. A team flooded with notifications it cannot act on will eventually start ignoring them, including the one that matters. For more detail, see Preventing alert fatigue (Chinese).

Configure the detection and the investigation context together. In Guance monitors, the rule type determines the available detection frequency, lookback window, and thresholds. Configure the notification policy, then include relevant log or trace links in the event content. Their filters and time range need to match the detected condition, and the underlying data must already be collected. Before handing the rule to the on-call team, use a controlled failure to check triggering, recovery, and the scope of the linked queries.

Where monitoring reaches its limits

Microservices, containers, and serverless workloads replace a small set of familiar servers with many short-lived instances. That changes how monitoring needs to be organized:

  • Instances disappear quickly. A container may live for only minutes. Aggregate by stable dimensions such as service and workload as well as inspecting individual machines. Host metrics remain useful for resource saturation and underlying failures.
  • Cardinality grows with label combinations. Pod, route, and status-code labels can produce many series. Each distinct combination actually observed can create a separate series. Unbounded values such as user IDs and raw URLs containing dynamic parameters need particular care. Prometheus's label guidance explains this relationship.
  • Requests cross service boundaries. A request may pass through a dozen services. A metric from one component rarely explains the whole reason it was slow.

This is why metrics are often used alongside logs and distributed traces. Metrics reveal changes in aggregate behavior. Logs record individual events. Traces show the calls a request made and where it spent time. Correlating them can narrow an investigation, but the result still depends on collection coverage, sampling, aligned timestamps, and consistent identifiers. Having all three signals does not guarantee a complete explanation.

Putting the pieces to work

Metrics, monitoring, and alerting describe a sequence of work: measure system behavior, keep track of it, and involve a person when action is needed. Start by asking what a failure would look like to a user. Then decide which measurements and notifications would expose it. That produces a more useful system than collecting data first and looking for a reason to alert on it later.

Frequently asked questions

Should metrics, logs, or tracing come first?

Start with the gap that most limits your current decisions. If request metrics already exist but slow requests remain unexplained, tracing may be the next useful addition. If basic error records are missing, fix logging first. Cost depends on volume, cardinality, sampling, processing, and retention; metrics are not invariably cheapest, nor traces invariably most expensive. Guance provides collection, query, and correlation paths for these signals, but they still need configuration and consistent fields. A shared platform does not imply identical physical storage or retention policies.

How should I choose alert thresholds?

Collect a baseline that covers the relevant business cycle. A week or two can be a starting point, but may miss month-end processing or holiday traffic. Set thresholds with the service objective, duration, and request volume in mind, then adjust them as you learn from false positives and missed incidents. Dynamic thresholds or change detection may help when fixed thresholds do not fit, provided their training data and operating conditions fit the workload.

How long should monitoring data be retained?

Plan around its use. For example, you might evaluate 7–14 days of high-resolution data for troubleshooting and 3–12 months of aggregated data for capacity planning. Those are starting points, not required periods; seasonal behavior may need a longer history. Check compliance obligations separately. Retention and aggregation should reflect the data's purpose, the platform's capabilities, and the actual cost.

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting