RED and USE metrics: from slow requests to resource pressure
Use RED to measure request rate, errors, and duration. Use USE to check resource utilization, saturation, and errors, with Prometheus examples.
Checkout slows down during a sale. The service is still completing about as many requests per second as before, but its slowest requests now take several seconds. The database hosts also look busy. Before adding capacity, find out whether checkout requests are actually waiting on the resource that looks busy.
RED measures the service through request rate, errors, and duration. USE measures each resource through utilization, saturation, and errors. Start with RED to see what changed for callers. Then follow the affected request path and use USE to check the resources on that path. Tom Wilkie’s RED method · Brendan Gregg’s USE method
Use RED to find the affected requests
Start with the checkout service. A stable request rate can hide changes in traffic mix, such as a shift from reads to payment attempts. Split by a bounded route label such as /orders/{id} when the service-level view is too broad. Use the same environment, time window, and definition of a request across all three panels.
sum by (service) (rate(http_requests_total[5m]))
Errors can be a rate in failures per second or a percentage of all requests. This example counts HTTP 5xx responses. For a customer-facing SLO, decide separately how to treat HTTP 4xx, requests that time out before receiving a response, and failed payments returned with HTTP 200. Those cases require signals that actually record their outcomes.
100 * (
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
or on (service) (0 * sum by (service) (rate(http_requests_total[5m])))
) / (sum by (service) (rate(http_requests_total[5m])) > 0)
The zero fallback applies to a service with measured traffic and reliable status labels. If the numerator is missing because status telemetry broke, the result can look healthy. Watch scrape health and instrument the failure paths. The denominator filter leaves the ratio absent when traffic is zero or unavailable. Show volume with the percentage so that one failed request out of two does not look like a broad incident.
Read the tail with the traffic volume
Ninety-nine requests at 20 ms and one at 3 seconds average less than 50 ms. A classic Prometheus histogram lets you see the distribution through its bucket counters. Apply rate() before summing to handle resets per series, and retain le so histogram_quantile() knows the bucket boundaries.
histogram_quantile(0.99,
sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))
This query estimates P99 in seconds for each service. Combine replicas only when they use compatible bucket boundaries. Read the estimate with the number of observations and the bucket layout. If you need the share that finished within 300 ms, instrument a bucket at 0.3 and divide its rate by the matching count rate for the same request population. PromQL functions · Prometheus histogram guidance
Use USE to look for work that had to wait
Check each resource the slow path actually touches. Utilization describes how busy it is, saturation describes work that cannot be served immediately, and errors record failures at that layer.
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU | Non-idle time by host or core | Runnable work waiting for CPU, or CPU pressure stall time | Hardware faults where the platform exposes them |
| Block device | Time with I/O in progress | In-flight I/O read alongside completion latency and the device baseline | I/O errors from the device or operating system |
| Database connection pool | Checked-out connections relative to the pool limit | Waiters or time to acquire a connection | Acquire timeouts and pool errors |
This Node Exporter query averages Linux CPU time across logical CPUs on each instance. Non-idle time includes modes such as iowait and steal. Look at the modes, individual cores, and container throttling before concluding that the application used all its available CPU. Linux load average also counts tasks in uninterruptible sleep, so it is broader than a count of CPU waiters. Linux /proc documentation
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
For disks, the rate of weighted I/O time estimates how many requests were in flight on average per device. It includes requests being serviced. Parallel storage can sustain several concurrent requests without being overloaded, so compare the value with completion latency and normal behavior for that device.
rate(node_disk_io_time_weighted_seconds_total[5m])
Receive errors and receive drops are separate counters. A rise in drops can point to buffer pressure or policy as well as network trouble. This example reports receive errors per second per interface.
rate(node_network_receive_errs_total[5m])
Put the request and resource views on the same timeline
If checkout P99 rose at 10:02 and only one database host shows sustained CPU pressure, check which requests reached that host. A slow database span can reflect pool acquisition, lock waits, disk I/O, or CPU contention depending on what it measures. Use traces, database wait metrics, and logs to distinguish them. Host utilization alone cannot tell you where the time went.
In Guance, the DataKit Prom collector can scrape the application and Node Exporter endpoints. Use consistent service, environment, and instance labels to align the charts. OpenTelemetry traces can show the database operation on sampled requests once the relevant services are instrumented. A service dashboard should put rate, errors, and tail latency together, with a drill-down into the resource instance the request used.
Google’s four golden signals cover latency, traffic, errors, and service saturation. USE applies the saturation question to each resource and adds utilization and resource errors. For a batch worker, the same service-side idea can use items processed per second, failed items per second or as a proportion, and per-run duration. Keep the denominator and unit visible. Google SRE on the four golden signals · Prometheus best practices
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
