The four golden signals of SRE
Measure latency, traffic, errors, and saturation with consistent Node.js instrumentation and PromQL. Understand percentile limits, error-budget burn rates, and what resource utilization leaves out.
A service can have a reassuring average response time while a few requests wait for seconds. It can also report fewer errors simply because traffic stopped reaching it. The four golden signals help you read those changes together: latency, traffic, errors, and saturation.
Google’s SRE book uses these signals to prioritize monitoring for user-facing services. They provide a useful starting point, rather than a complete health model for every system. A data pipeline still needs freshness and correctness checks; a scheduled job needs a way to tell whether it finished on time.
Site reliability engineering applies software engineering to running services. A site reliability engineer also works on automation, capacity, and incident response. Choosing these four signals is one part of that work.
Signal one: latency
Latency measures how long a request takes to be served.
Two distinctions matter:
- Measure successful and failed requests separately. A service can reject a request immediately or spend 30 seconds waiting before returning a timeout. Combining these responses with successful work can make the latency chart misleading. Keep the failure distribution visible too.
- Look at the distribution as well as the average. P50, P90, and P99 are points in the distribution of observed request durations. They do not describe percentages of users, and a percentile says nothing about a customer’s commercial importance. Higher percentiles help expose the tail, but no single percentile catches every outlier.
Here is an instrumentation fragment for Node.js and Express, using the prom-client 15.1.3 API:
import { Histogram } from 'prom-client';
const requestLatency = new Histogram({
name: 'http_request_duration_seconds',
help: 'Completed HTTP response duration in seconds',
labelNames: ['service', 'method', 'route', 'status_code'],
buckets: [0.01, 0.05, 0.1, 0.5, 1, 2.5, 5, 10]
});
const knownRoutes = new Set(['/orders', '/orders/:id']);
const knownMethods = new Set(['GET', 'POST', 'PUT', 'PATCH', 'DELETE', 'HEAD', 'OPTIONS']);
function latencyMiddleware(req, res, next) {
const start = process.hrtime.bigint();
res.once('finish', () => {
const route = req.route?.path;
requestLatency.observe(
{
service: 'checkout',
method: knownMethods.has(req.method) ? req.method : 'OTHER',
route: typeof route === 'string' && knownRoutes.has(route) ? route : '__other__',
status_code: String(res.statusCode)
},
Number(process.hrtime.bigint() - start) / 1e9
);
});
next();
}
Register this middleware before the application’s business routes. The example assumes /orders and /orders/:id are mounted directly on the app. With nested routers, explicitly map the full route template. A raw URL containing an order ID would create a separate label value for every order; the allowlist keeps unmatched routes in __other__ instead.
The application also needs a metrics endpoint that exposes its registry, and Prometheus must scrape it. If requests are spread across Node.js workers, arrange collection or aggregation across those processes rather than assuming one worker’s registry represents the whole service.
This measures elapsed time from entering the middleware until the response emits finish. That event means the response has been handed to the underlying system; it does not prove the client received it. An aborted response may never emit finish, so track aborts separately—for example, check res.writableFinished when close fires. The traffic and error queries below deliberately use the same completed-response population. This is not a client round-trip measurement.
For a successful-response P99, the following query treats HTTP 2xx as success. Adjust that definition if the application accepts other statuses, and query failures separately:
histogram_quantile(0.99,
sum by (service, route, le) (rate(http_request_duration_seconds_bucket{status_code=~"2.."}[5m])))
For a classic histogram, histogram_quantile estimates the percentile by interpolating within a bucket. Put useful bucket boundaries near the latency thresholds you care about. The result is not an exact sort of individual request durations, and a small request count makes the tail particularly difficult to interpret. Read it alongside observation counts and slow-request records.
Signal two: traffic
Traffic measures demand on the system. The unit follows the work:
- Web service: HTTP requests per second.
- Streaming service: bandwidth or concurrent sessions.
- Database: queries per second or active connections.
- Queue consumer: messages processed per second.
Traffic gives the other signals context. A 1% failure rate at 10 requests per second represents a different number of failures from 1% at 10,000 requests per second. Its trend also informs capacity planning.
Using the histogram above, the completed-response rate is:
sum by (service) (rate(http_request_duration_seconds_count[5m]))
_count is the cumulative observation count exported by that histogram. The query does not depend on a separate, unshown request counter. Requests arriving at the ingress, unfinished requests, and open connections require their own measurements.
Signal three: errors
Errors measure the rate or fraction of requests that fail. A 500 response is an explicit failure; a 200 response containing the wrong result can be a failure too. Whether a slow response counts as a bad event depends on the service’s agreed correctness and latency objectives. Neither the status code nor slowness alone defines every SLI.
The example below calculates the fraction of completed responses returning HTTP 5xx, grouped by service:
# HTTP 5xx as a fraction of completed responses, per service
(
sum by (service) (rate(http_request_duration_seconds_count{status_code=~"5.."}[5m]))
or on (service)
(0 * sum by (service) (rate(http_request_duration_seconds_count[5m])))
)
/
(sum by (service) (rate(http_request_duration_seconds_count[5m])) > 0)
This is a status-based example. Wrong content and business failures need their own detection and counters. The or branch supplies a zero for a missing 5xx series only where a total series exists. The denominator’s > 0 filters out zero-traffic windows. Missing collection data still needs its own alert; it is not evidence of an error-free service.
For an SLO of 99.9% successful eligible requests, the burn rate is the bad-event fraction divided by 1 - 0.999, provided both counts use the same eligibility and success definitions. A 1% bad-event fraction therefore gives a burn rate of 10: the budget is being consumed at ten times its allowed average rate.
Signal four: saturation
Saturation describes pressure on constrained resources and work waiting to be served. Queueing and waiting expose capacity pressure; headroom and trends help show how soon that pressure may become a service problem.
Look at CPU run queues, waits for a connection from a pool, message backlog, and memory reclaim or paging pressure. CPU usage, memory occupancy, disk capacity, and pool occupancy provide related utilization or capacity information. They do not, by themselves, measure how much work is waiting. There is no general rule that 90% CPU usage doubles latency; the point at which performance deteriorates depends on the resource and workload.
The following query requires Linux host memory metrics collected from node_exporter, with labels matched to the intended host. These metrics do not come from the Node.js histogram above. The result is not process heap usage or usage against a container’s cgroup limit; it is a host memory-use percentage to support capacity analysis, rather than a complete saturation measure:
# Memory-use percentage: a supporting capacity signal
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
How RED and USE relate
- RED looks at request rate, errors, and duration. These overlap with the first three golden signals, while the golden signals also emphasize saturation. This is a correspondence between their concerns, not a strict equation.
- USE checks utilization, saturation, and errors for each resource. It helps investigate resource bottlenecks, complementing the service-level view.
- In practice, use golden signals or RED at the service level and USE at the resource level. Keep the service and resource views connected in the dashboard.
Turn the signals into actionable alerts
- Latency: a P99 threshold can support diagnosis or an alert, but does not directly measure error-budget consumption. For a request-based latency SLO, count eligible requests completed within the target duration against all eligible requests, then calculate the bad-event fraction. Account for volume and how long the condition lasts.
- Errors: calculate burn rate from the SLI’s actual bad events. Combine long and short windows to distinguish an ongoing problem from a spike that has recovered. Low-volume services need particular care; burn-rate alerts are not automatically faster or quieter.
- Saturation: choose thresholds from observed queueing, time to exhaustion, and the available response. A capacity issue with time to act can create a ticket; a current service impact requiring immediate action may need a page. 80% is not a universal boundary.
- Traffic: a drop can mean ingress failure, ordinary demand changes, or broken collection. Compare it with expected demand, ingress availability, and collection health so the alert has a clear response.
Follow one service across the four signals
In Guance, collected traces can show errors and request durations, while host and container collection provides the corresponding resource data. If traces are sampled, the retained records and their aggregates do not automatically represent all traffic. Before using service ratios for an SLO, check where the metrics were produced and which sampling or filtering decisions affected them. Connection-pool waits and application queues may also need custom metrics.
For a known slow request, open trace details to see which calls took time, then examine resources and queues in the same interval. Align service and resource identifiers, the time window, retention, and access permissions. Log correlation additionally needs consistent trace IDs and the appropriate index mapping. Placing charts on one dashboard does not establish a cause.
Frequently asked questions
Are four signals enough for troubleshooting?
They prioritize what to watch and help narrow the symptoms. Four numbers cannot identify a resource bottleneck or root cause on their own. Follow them into logs, traces, resource measurements, and application internals.
Why use percentiles instead of the average?
Suppose 99 of 100 requests take 10 ms and one takes 10 seconds. The mean is 109.9 ms, which hides the severity of that single request. P99 can still miss it: with the nearest-rank definition, the 99th observation is 10 ms. Other algorithms and histogram estimates depend on interpolation. Combine percentiles with the fraction above a threshold, the sample count, and individual slow requests. One request is not necessarily one user.
How do the signals apply to batch processing?
Use item or batch duration for latency, processing rate for traffic, failed-item fraction for errors, and backlog plus worker occupancy for capacity pressure. Also track result correctness, freshness, and deadlines. A pipeline can process quickly while operating on data that arrived too late.
Sources: Google SRE: monitoring distributed systems, Prometheus histograms and quantiles, Google SRE: alerting on SLOs, prom-client 15.1.3, Guance monitor rules, and Guance application performance monitoring (Chinese product page). Technical references checked November 5, 2026.
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
