Centralized logging with Rsyslog: forwarding, queues, and recovery

Rsyslog can collect logs from several hosts onto a central receiver, where they can be stored, searched, or forwarded to a backend. A separate copy helps when an application host fails and makes cross-host investigations easier. Those benefits depend on delivery and durable storage: a message still waiting in a sender's memory has not become a protected central copy.

This guide configures a receiver, forwarding, an action queue, transport security, rotation, and a DataKit destination. The examples target Linux installations of Rsyslog 8. Package names, service users, firewall tooling, and available modules vary by distribution; validate the combined configuration on the installed version before restarting the service.

Give remote inputs their own ruleset

Create a receiver configuration such as /etc/rsyslog.d/10-server.conf. Check that the modules are not already loaded elsewhere:

module(load="imtcp")
module(load="imudp")

template(name="RemoteHostFile" type="string"
  string="/var/log/remote/%hostname:::secpath-replace%/%programname:::secpath-replace%.log")

ruleset(name="remote") {
  action(type="omfile" dynaFile="RemoteHostFile"
         createDirs="on" dirCreateMode="0750" fileCreateMode="0640")
  stop
}

input(type="imtcp" port="514" ruleset="remote")
# Enable UDP only if your senders require it.
input(type="imudp" port="514" ruleset="remote")

The listener's ruleset identifies remote traffic explicitly. Testing only whether $fromhost-ip differs from 127.0.0.1 can misclassify local or IPv6 traffic and is not an authentication boundary. Unrelated local inputs continue through their configured rulesets.

The dynamic path groups files by the message's hostname and program name. These fields can be supplied by senders, so secure path substitution prevents them from becoming arbitrary directory paths. Sanitization does not authenticate the hostname or prevent an allowed sender from generating excessive unique filenames. Restrict senders and watch file count, open-file limits, and disk usage.

Create /var/log/remote with ownership and permissions that let the actual Rsyslog service user create its subdirectories. Account for SELinux or AppArmor policy if enabled. Permit only the required sender networks to the chosen TCP or UDP port; avoid opening a plain syslog listener to the public Internet.

Validate and then restart on a systemd host:

sudo rsyslogd -N1
sudo systemctl restart rsyslog
sudo systemctl status rsyslog

Use the equivalent service commands on other systems. A successful configuration check does not test network reachability or runtime write permissions. Send a distinctive test message from an allowed client and confirm the expected file and event contents.

Forward from a client

The traditional selectors remain useful for understanding existing configurations. Choose one of these alternatives:

# UDP: one @
*.* @192.168.1.10:514

# TCP: two @ characters
*.* @@192.168.1.10:514

Do not enable both unless two copies are intentional. To forward authentication facilities only, a selector such as auth,authpriv.* @@192.168.1.10:514 narrows the stream.

For new configurations, an explicit omfwd action makes transport and queue settings easier to review. The next example replaces the simple TCP rule; adding it alongside that rule would duplicate delivery.

Add a bounded action queue

Provision a writable spool directory for the service. Configure its work directory once, reusing an existing setting if the distribution already supplies one. Then add a forwarding action:

global(workDirectory="/var/spool/rsyslog")

action(
  type="omfwd"
  target="192.168.1.10"
  port="514"
  protocol="tcp"
  template="RSYSLOG_ForwardFormat"
  action.resumeRetryCount="-1"
  queue.type="LinkedList"
  queue.filename="central-forward"
  queue.maxDiskSpace="1g"
  queue.saveOnShutdown="on"
)

These queue settings belong to this action. With legacy directives, placing queue settings after a forwarding rule can apply them to a subsequent action instead of the intended one. Keeping them in the action block avoids that ordering trap. See the forwarding module.

A disk-assisted linked-list queue starts in memory and can spill to disk. saveOnShutdown saves its remaining in-memory backlog during an orderly shutdown; it does not protect that memory against power loss, SIGKILL, or an out-of-memory kill. resumeRetryCount="-1" permits indefinite retries, while disk space and queue capacity remain finite. Consult the queue parameters when deciding whether a pure disk queue and stronger checkpoint/sync settings are necessary.

Size the spool for the expected outage duration and arrival rate, with space for overhead and other workloads. Monitor queue growth, suspended actions, discards, and disk capacity. Recovery also needs spare forwarding and receiver capacity: matching the incoming rate only prevents further growth.

Test an unavailable receiver, recovery, graceful sender restart, and an exhausted queue. Count unique test event IDs at the destination so missing or duplicate records are visible.

Choose transport by its failure behavior

RELP is useful when application-level acknowledgement matters, but it is not a blanket promise of no loss and no duplicates across all failures. Install the distribution's RELP module package on both systems. For an alternative receiver on port 2514, reuse the remote ruleset above:

module(load="imrelp")
input(type="imrelp" port="2514" ruleset="remote")

On the sender, replace the TCP forwarding action with an omrelp action, including a uniquely named queue:

module(load="omrelp")
action(type="omrelp" target="192.168.1.10" port="2514"
       template="RSYSLOG_ForwardFormat"
       action.resumeRetryCount="-1"
       queue.type="LinkedList" queue.filename="relp-forward"
       queue.maxDiskSpace="1g" queue.saveOnShutdown="on")

This example is unencrypted. Configure TLS separately before using it across an untrusted network. The RELP output reference describes its own TLS options; they are not interchangeable with the TCP stream-driver options.

Authenticate and encrypt the connection

For TCP TLS, Rsyslog can use the gtls driver with the appropriate TLS package, often named rsyslog-gnutls. Provision a trusted CA and the required certificates and keys, enable encrypted mode, and use name-based peer verification such as x509/name with permitted peer names. The certificate identity must match the configured expectation.

Both ends must agree on encryption, authentication, port, and framing. Follow the official TLS server and TLS client examples for the installed version. Test a valid peer, an untrusted certificate, a wrong name, and certificate renewal. A firewall allowlist limits reachability; it does not replace encryption or cryptographic identity verification, even on an internal network.

Rotate central files and verify reopening

A logrotate policy can manage the receiver's files. On a systemd installation, an example /etc/logrotate.d/remote-logs is:

/var/log/remote/*/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        /usr/bin/systemctl kill --kill-who=main --signal=HUP rsyslog.service
    endscript
}

Confirm the service name and executable path, or use the distribution's existing Rsyslog rotation helper. The HUP signal tells Rsyslog to close open files so later writes reopen them. delaycompress leaves the newest rotated file uncompressed until the next cycle. Verify permissions when the new active file is created.

rotate 14 keeps fourteen rotated files; it is not an unconditional fourteen-day guarantee. Actual rotation runs, empty files, size policies, and missed schedules affect coverage. Test a rotation while events arrive and confirm both Rsyslog and any file collector follow the intended files. Avoid assuming copytruncate is lossless.

Forward to DataKit for centralized analysis

You can keep Rsyslog as the network receiver and use DataKit for collection and forwarding to Guance. Choose either file collection from /var/log/remote or a configured socket destination so the same intended event is not ingested twice.

The Guance syslog guide describes enabling TCP or UDP sockets in DataKit's logging collector. Its example TCP listener uses port 9540, not an assumed universal syslog port. Configure an address and port appropriate to the deployment, restrict access, and verify the collector's actual configuration file location. Then forward to that real reachable address:

*.* @@192.168.1.20:9540

Use an action queue for this hop as well if buffering is required. If forwarding from the central receiver's dedicated ruleset, place the forwarding action before its stop; a rule in the default ruleset will not automatically see those remote events. Prevent forwarding loops.

Set source and service fields, then inspect the actual incoming message and parser output. Confirm hostname, timestamp, severity, and message extraction before adding Pipeline transformations; do not parse the same header twice. For no-data alerts, distinguish a host that legitimately has no new events from a broken forwarding path.

Choose retention and indexes for the required queries. If forwarding older data to external storage, test both delivery and later retrieval. A local receipt or a successful search today does not establish the full archive lifecycle.

Common questions

Why do timestamps appear out of order?

Check sender clocks, timezone handling, queue delays, and parser assumptions. Traditional syslog timestamps can omit the year; RSYSLOG_ForwardFormat includes higher-precision timestamps with the year. Configure NTP or chrony and test both new events and replayed backlog.

Can one UDP receiver handle ten thousand clients?

Client count alone cannot answer that. Measure message rate, size, bursts, parsing cost, kernel drops, and output throughput. On Linux, inspect net.core.rmem_max and the imudp rcvbufSize setting when investigating receive-buffer drops. Rsyslog documents rcvbufSize and its operating-system limits. Buffer tuning can help but does not make UDP reliable. Partition load across receivers or choose a connection-oriented protocol when the failure model calls for it.

How long should centralized logs be retained?

Set a policy for the applicable jurisdiction, contract, audit purpose, and investigation needs. There is no universal six-month requirement for every deployment. Document the hot-search and archive periods separately and test deletion and retrieval.

Does this approach fit containers?

It can collect host syslog or journald-derived events. Application stdout may be better collected through the container runtime's existing path. Avoid routing it through host syslog merely to centralize it if that creates an unnecessary duplicate or failure point.

Related reading

• Rsyslog configuration fundamentals (Chinese)
• Filebeat vs. Logstash (Chinese)
• Fluentd vs. Fluent Bit (Chinese)