Monitor Node.js with Prometheus
Scrape Express runtime and HTTP metrics, query request rate, 5xx ratio and P95 with PromQL, and configure alerts that account for low traffic.
If checkout starts returning 503s while CPU and heap usage still look normal, you need to know how many requests failed and whether completed responses slowed down too. With the Node.js Prometheus client, a single /metrics endpoint can expose both runtime metrics and the counters and histograms you add to answer those questions.
This walkthrough follows one Express route from instrumentation to a Prometheus scrape and two alert rules. The checkout responses are synthetic.
Give Prometheus a target to scrape
Prometheus polls the application on a schedule. Each metric name and label set identifies a time series, and each successful scrape supplies new samples. Prometheus evaluates PromQL and alert rules, Alertmanager routes notifications, and Grafana can visualize the data.
The project previously shipped as prom-client. This example uses its successor package @prometheus-io/client 0.16.1 on Node.js 22. That package version supports Node.js 22, 24, and 26 or later. Install the dependencies in a scratch project.
npm install express@5.3.0 @prometheus-io/client@0.16.1
Save the example as checkout.cjs and run node checkout.cjs. The loopback listener is for local testing. In a deployment, let the scraper reach /metrics over a controlled network path and restrict who else can reach it.
const express = require('express');
const client = require('@prometheus-io/client');
const app = express();
client.collectDefaultMetrics();
const requests = new client.Counter({
name: 'http_requests_total',
help: 'Completed checkout HTTP responses',
labelNames: ['method', 'route', 'status'],
});
const inFlight = new client.Gauge({
name: 'http_requests_in_flight',
help: 'Checkout HTTP responses still in progress',
});
const duration = new client.Histogram({
name: 'request_duration_seconds',
help: 'Completed checkout response duration in seconds',
labelNames: ['route'],
buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
});
app.get('/metrics', async (req, res) => {
res.set('Content-Type', client.register.contentType);
res.end(await client.register.metrics());
});
const checkoutRoute = '/checkout';
for (const status of ['204', '503']) {
requests.inc({ method: 'POST', route: checkoutRoute, status }, 0);
}
app.post(checkoutRoute, (req, res) => {
const route = checkoutRoute;
const stopTimer = duration.startTimer({ route });
inFlight.inc();
res.once('finish', () => {
requests.inc({ method: req.method, route, status: String(res.statusCode) });
stopTimer();
});
res.once('close', () => inFlight.dec());
res.status(req.query.fail === '1' ? 503 : 204).end();
});
app.listen(3000, '127.0.0.1');
curl -i -X POST 'http://127.0.0.1:3000/checkout'
curl -i -X POST 'http://127.0.0.1:3000/checkout?fail=1'
curl -fsS http://127.0.0.1:3000/metrics
The two requests return 204 and 503. The query parameter simulates failure and is for the example only. A subsequent scrape shows both status codes on http_requests_total and two completed observations in request_duration_seconds_count. The runtime metrics cover signals such as event loop lag, garbage collection, heap usage, and CPU, with some metrics specific to the host platform.
Match the metric to the question
A counter records how many responses completed, broken down by method, route and status. It resets when the process restarts. Read the final status on the response's finish event. That event marks the handoff to the operating system, so the histogram estimates server-side time from the route handler and excludes client network time.
A gauge tracks responses still in progress. It rises when the handler starts and falls on close, including when the connection ends early. An interrupted response that never reaches finish contributes to neither the completed counter nor the latency histogram. If disconnects matter to your service, count them separately.
A classic histogram records counts in cumulative time buckets, which Prometheus can combine across replicas with matching bucket boundaries. Put boundaries near the latency thresholds you care about. A summary can calculate quantiles in the application, which are useful per process. Averaging the P95 reported by several processes will not give the P95 across all requests.
Keep label values bounded. With a route like /orders/:id, store the route template and keep the order ID and raw query string out of metric labels. Every distinct label set adds a series. A histogram multiplies that series cost by the number of buckets plus its sum and count.
Read request rate, error ratio, and P95
Save the following as prometheus.yml, put the alert file from the next section next to it, and start Prometheus with prometheus --config.file=prometheus.yml on the same host as the example.
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- checkout-alerts.yml
scrape_configs:
- job_name: checkout-node
static_configs:
- targets: ["127.0.0.1:3000"]
The first query gives completed responses per second by route. The second returns the percentage of responses with a 5xx status across the checkout job. The third estimates P95 in seconds per route.
sum by (route) (rate(http_requests_total{job="checkout-node"}[5m]))
100 * (
sum(rate(http_requests_total{job="checkout-node",status=~"5.."}[5m]))
or 0 * sum(rate(http_requests_total{job="checkout-node"}[5m]))
) / (sum(rate(http_requests_total{job="checkout-node"}[5m])) > 0)
histogram_quantile(0.95, sum by (le, route) (
rate(request_duration_seconds_bucket{job="checkout-node"}[5m])
))
Apply rate() to each counter series before aggregation so Prometheus can handle process restarts. In the ratio, or 0 * ... supplies a zero numerator if there is measurable traffic but no usable 5xx rate. Newly seen series need enough samples for a rate. The example initializes the two known status counters so the scraper can observe their zero baseline before the first error. With zero traffic or no usable total, the denominator filter leaves the ratio empty. Check up separately to tell a quiet service from a failed scrape.
Alert on sustained problems
The rules in checkout-alerts.yml use a rolling five-minute query window, and require the condition to remain true at rule evaluations for five minutes before firing. The minimum request rate protects these example thresholds from extremely small samples. Adjust it and the thresholds to the service's traffic and SLO. Configure an Alertmanager target in Prometheus to route the resulting notifications.
groups:
- name: checkout
rules:
- record: checkout:http_requests_per_second:rate5m
expr: sum(rate(http_requests_total{job="checkout-node"}[5m]))
- alert: CheckoutHighErrorRatio
expr: |
(
sum(rate(http_requests_total{job="checkout-node",status=~"5.."}[5m]))
or 0 * checkout:http_requests_per_second:rate5m
) / checkout:http_requests_per_second:rate5m > 0.01
and checkout:http_requests_per_second:rate5m > 0.2
for: 5m
- alert: CheckoutHighP95
expr: |
histogram_quantile(0.95, sum by (le) (
rate(request_duration_seconds_bucket{job="checkout-node"}[5m])
)) > 2
and checkout:http_requests_per_second:rate5m > 0.2
for: 5m
Use the same endpoint with Guance
DataKit's Prom collector can scrape the existing /metrics endpoint and send the selected metrics to Guance. Set the target, any required authentication, and the scrape interval. Check types, units, and label mappings before creating an error-rate or latency monitor. A slow-request alert can take you to traces once you have instrumented Node.js APM. To jump from a span to logs, configure trace and span ID injection and align the service, environment and version fields with the tracing setup.
- Configure DataKit's Prom collector
- Instrument Node.js with dd-trace
- Correlate logs with traces
- Configure monitors
Questions that come up in deployment
How do I scrape replicas or Node.js workers?
Give the scraper a target for each replica. A single load-balanced endpoint can return data from a different process on each scrape. If you use Node.js cluster, expose workers separately or use the client's AggregatorRegistry to aggregate them. On Kubernetes, an installed Prometheus Operator can discover endpoints through a selected ServiceMonitor, with matching selectors and ports.
What about a CronJob that exits between scrapes?
For a service-level batch job independent of a particular machine, consider sending its result or last success time to Pushgateway. Prometheus or DataKit can scrape that endpoint. Pushgateway retains groups until they are deleted, so define cleanup when jobs go away. For machine-bound jobs, look at the Node Exporter textfile collector. If you need a record per run, logs or traces may fit better.
How much does this instrumentation cost?
Measure it under your own traffic. The number of metrics, distinct label sets, histogram buckets and scrape frequency affect collection overhead and payload size. Labels such as user IDs also grow the number of series the backend has to store and query.
References
- Prometheus client for Node.js and package changelog
- Prometheus histograms and summaries and instrumentation practices
- Prometheus alerting rules and when to use Pushgateway
- Node.js HTTP response events and Prometheus Operator setup
Related reading
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
