Monitor Linux hosts with Prometheus and Node Exporter

Install Node Exporter as a systemd service, configure Prometheus scraping, read CPU, memory, disk and network metrics, and publish reliable textfile metrics.

Best practices
Illustration of Linux host metrics collection and monitoring

Node Exporter gives Prometheus a view of the Linux host: CPU time, available memory, filesystems, disk activity and network traffic. It reads operating system and hardware statistics and exposes them in Prometheus format. The exact set depends on the system and the collectors you enable.

Getting an HTTP response from the exporter is the first step. The more useful work is making sure it runs reliably, that Prometheus scrapes the intended host, and that a query means what its dashboard label says. A load average is not CPU utilization; a timestamp written by cron is not evidence that a backup succeeded. This guide covers the installation and those details together.

Install Node Exporter as a service

The following example targets Linux amd64 with systemd and pins Node Exporter to v1.8.2 to make the commands explicit. Treat it as a fixed example, not as the current release recommendation. Choose an appropriate release and architecture from the official releases, and verify its published checksum. An ARM64 host needs a different archive.

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xzf node_exporter-1.8.2.linux-amd64.tar.gz
cd node_exporter-1.8.2.linux-amd64
sudo install -m 0755 node_exporter /usr/local/bin/node_exporter
sudo useradd --system --user-group --no-create-home --shell /usr/sbin/nologin node_exporter

The account command assumes a Linux distribution that supports these useradd options. Skip account creation if the service account already exists, and check the location of nologin on your distribution. Keep one owner for the process: starting ./node_exporter & and then starting systemd leaves two processes competing for the same port.

Save this unit as /etc/systemd/system/node_exporter.service. The binary is installed at the path used by ExecStart, and the dedicated account runs the service:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

Reload the unit definitions, enable the service at boot and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
systemctl status node_exporter --no-pager
curl --fail --silent --show-error http://127.0.0.1:9100/metrics

Metrics in the response show that the local endpoint is serving data. Check reachability from the Prometheus host as well. Permissions can cause individual collectors to fail even when /metrics returns HTTP 200, so inspect the service logs and node_scrape_collector_success. Prometheus installation guide · systemd service configuration

Configure Prometheus to scrape the hosts

Merge this job into the existing scrape_configs list in your Prometheus configuration. Do not add a second top-level scrape_configs key. Replace server1 and server2 with hostnames or addresses that Prometheus can resolve and reach:

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['server1:9100', 'server2:9100']

Reload Prometheus using the mechanism supported by your deployment, check the Targets page, then query up{job="node"}. A value of 1 means the most recent scrape succeeded; it does not establish that the business service is healthy. All PromQL examples below use this job="node" scope. Configuration and reload behavior

Keep the collectors your monitoring uses

Node Exporter enables many collectors by default. Whether you need zfs or infiniband depends on the host. Disabling unused collectors can remove collection work and output series, but it also removes metrics that dashboards and alerts may depend on.

This command enables only a selected set. With the systemd setup above, add the flags to the existing ExecStart, reload the unit and restart the service; do not launch a second exporter:

node_exporter --collector.disable-defaults \
  --collector.cpu --collector.meminfo --collector.diskstats \
  --collector.filesystem --collector.netdev --collector.loadavg

This list omits textfile. If you use the custom metrics later in this guide, add --collector.textfile and its directory flag. Check the deployed version's --help, collector duration and dashboard dependencies before removing collectors. Collector options

Read the host metrics correctly

CPU

# Non-idle CPU time (%), keeping job and instance separate
100 - avg by (job, instance) (
  rate(node_cpu_seconds_total{job="node",mode="idle"}[5m])
) * 100

# One-minute load average per logical CPU
node_load1{job="node"}
  / on (job, instance)
count by (job, instance) (node_cpu_seconds_total{job="node",mode="idle"})

Linux load average includes tasks in uninterruptible sleep as well as runnable work. Disk waits can therefore raise load without producing the same increase in CPU utilization.

The load query explicitly matches on job, instance. Aggregation on the right removes other labels, so relying on default matching can leave the two sides with no match. Both CPU examples assume that job, instance uniquely identifies a host. In a multi-cluster store, include cluster or other required identity labels in both by and on, or narrow the selectors. Otherwise, aggregation can combine hosts from different clusters. PromQL vector matching

Memory

# Estimated memory usage based on available memory (%)
(1 - node_memory_MemAvailable_bytes{job="node"}
  / node_memory_MemTotal_bytes{job="node"}) * 100

Use MemAvailable rather than treating low MemFree as memory pressure. Linux uses otherwise idle memory for caches. MemAvailable estimates how much memory can be used by new applications without swapping; its presence still depends on the fields exposed by the kernel. Linux meminfo fields

Filesystems and disk I/O

# Space unavailable to ordinary users (%); exclude tmpfs/overlay, read-only and zero-size filesystems
(100 * (1 -
  node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|overlay"}
  / node_filesystem_size_bytes{job="node",fstype!~"tmpfs|overlay"}
))
and (node_filesystem_size_bytes{job="node",fstype!~"tmpfs|overlay"} > 0)
and (node_filesystem_readonly{job="node",fstype!~"tmpfs|overlay"} == 0)

# Reported I/O busy-time fraction; multiply by 100 for a percentage
rate(node_disk_io_time_seconds_total{job="node"}[5m])

avail is the space available to an unprivileged user. Subtracting it from total capacity includes both used space and reserved space unavailable to that user. It is not simply the fraction of physically used bytes. The query excludes zero-capacity and read-only filesystems. Monitor read-only status separately, so a filesystem that unexpectedly becomes read-only does not disappear from attention along with this chart. Filesystem metric implementation

Disk busy time is not a universal saturation measure. Storage can process requests in parallel, so a value near 1 does not by itself prove that the device has reached its performance limit. Read it alongside latency, queueing and throughput. Linux I/O accounting

Network

rate(node_network_receive_bytes_total{job="node",device="eth0"}[5m]) * 8   # Inbound bps
rate(node_network_transmit_bytes_total{job="node",device="eth0"}[5m]) * 8  # Outbound bps

eth0 is an example. Use the actual device label, which might be ens5 or enp1s0. Understand the relationship between physical interfaces, bonds, bridges and virtual interfaces before summing them; the same traffic can appear at more than one layer. These expressions return a traffic rate, not utilization as a percentage of link capacity.

Export batch-job results with the textfile collector

The textfile collector exposes metrics tied to a host, such as a cron job's result, the last successful backup time or remaining certificate validity. Files must contain valid Prometheus text-format metrics and end in .prom. Sample timestamps are not supported. A Unix time used as a gauge value is different from an extra timestamp appended to a sample. Textfile collector rules

Create a directory that the exporter account can traverse and read. In this example, root or an appropriately authorized backup process writes the files; the directory does not need to be writable by everyone:

sudo install -d -o root -g node_exporter -m 0750 /var/lib/node_exporter/textfile

Invoke this script only from a hook after the real backup job succeeds. The script does not perform or verify a backup. Replace /backups with the actual directory. GNU Linux du -sb reports apparent bytes, which is a different measure from compressed object size or allocated disk blocks.

#!/bin/bash
# /usr/local/bin/backup-metric.sh
set -euo pipefail

OUT_DIR=/var/lib/node_exporter/textfile
BACKUP_DIR=/backups
OUT="$OUT_DIR/backup.prom"

# If measuring the directory fails, preserve the previous success record.
du_output=$(du -sb -- "$BACKUP_DIR")
backup_size=${du_output%%$'\t'*}
[[ "$backup_size" =~ ^[0-9]+$ ]] || exit 1
completed_at=$(date +%s)

# The temporary file shares the directory and does not end in .prom.
tmp=$(mktemp "$OUT_DIR/backup.prom.XXXXXX")
trap 'rm -f -- "$tmp"' EXIT
cat > "$tmp" <<METRICS
# TYPE backup_last_success_timestamp gauge
backup_last_success_timestamp $completed_at
# TYPE backup_size_bytes gauge
backup_size_bytes $backup_size
METRICS
chmod 0644 "$tmp"
mv -f -- "$tmp" "$OUT"
trap - EXIT

Save the script, make it executable and call it after the existing backup job confirms success. Running it independently on a cron schedule would report the script's execution time as a backup success. Keeping the previous success timestamp on failure makes it possible to detect an overdue backup. Errors reading the backup directory or writing the metric file should also surface as job failures.

Add --collector.textfile.directory=/var/lib/node_exporter/textfile to Node Exporter. If using --collector.disable-defaults, also add --collector.textfile. For the systemd service, change ExecStart, reload the unit and restart it. Check the custom metrics and node_textfile_scrape_error. Renaming within one directory avoids exposing a half-written file; it does not guarantee persistence through power loss or validate the backup's contents.

Common deployment questions

Should I use Node Exporter or a monitoring Agent such as DataKit?

If you already operate Prometheus scraping and queries, Node Exporter fits that setup. To send the same host metrics to Guance, configure DataKit's prom collector with the exporter's http://host-address:9100/metrics endpoint. The existing exporter can stay in place.

Check the resulting fields, tags and measurement names. With an empty measurement_name, DataKit's default mapping splits the metric name at the first underscore into a measurement and a field; keep_exist_metric_name controls preservation of the original field name. Tags may also be renamed by configuration. Connecting the endpoint does not automatically migrate existing PromQL, dashboards or alerts. They need to be checked against the actual mapping.

Does a containerized Node Exporter monitor the container or the host?

That depends on the paths, mounts and namespaces each collector uses. It is inaccurate to say that every default metric necessarily describes the container. The official container example combines host PID and network namespaces, a read-only host-root mount and --path.rootfs. Separately mounted /proc and /sys paths must match --path.procfs and --path.sysfs. Path flags do not replace namespace configuration, and host PID or networking does not automatically expose every host mount.

For Node Exporter on Kubernetes, a DaemonSet commonly runs one exporter on each eligible node, with the necessary host access and mounts. Prometheus still needs discovery and scraping configuration for those endpoints. A DaemonSet alone does not create a scrape job. Node Exporter supplies node operating-system metrics; it does not automatically provide per-Pod resource breakdowns or application-internal metrics. Kubernetes service discovery in Prometheus

Should port 9100 be publicly accessible?

Keep it off the public internet. Restrict access with firewalls or security groups to the actual collectors, such as Prometheus or DataKit. Node Exporter v1.8.2 supports exporter-toolkit TLS and basic-auth settings through --web.config.file. Configure certificates or bcrypt password hashes and the corresponding scrape-client settings; use TLS with basic authentication. Authentication supplements network restrictions. The endpoint can still reveal kernel versions, mount points and other host details. Web configuration

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting