Fluentd explained: log collection, routing, and plugins
Follow a log through Fluentd, configure tag matching and file buffering, and check what an HTTP or Forward destination can confirm.
A Fluentd configuration can pass validation and still send logs to the wrong place. Move <match **> above a rule for Nginx access logs and the broad rule takes those events first. When a destination changes, the order of the rules matters as much as the endpoint.
Fluentd is a CNCF graduated project and an extensible log collector written primarily in Ruby. This guide follows an Nginx event from the input to the output, then shows what changes when the destination switches from a local file to HTTP. The examples follow the Fluentd 1.x and receiver documentation. CNCF project · Fluentd source
Fluentd at a glance
- Route by tag. A
<source>emits events, matching<filter>rules process them, and the first matching<match>selects the output. - Check plugin requirements. Fluentd ships core plugins. External plugins need a compatible Fluentd and Ruby environment and, where applicable, a supported destination.
- Budget for interruptions. A file buffer can retain pending chunks across a restart if the disk data survives. Outages still consume disk space and queue capacity.
- Verify a destination change with known events. Check the payload, field types, event time, and retry behavior before moving production traffic.
Route an Nginx access log by tag
<source>
@type tail
path /var/log/nginx/access.log
pos_file /var/log/fluentd/nginx.pos
tag access.nginx
<parse>
@type nginx
</parse>
</source>
<filter access.nginx>
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
env "production"
</record>
</filter>
<match access.**>
@type file
path /var/log/aggregated/nginx
</match>
The input reads the Nginx access file and emits events tagged access.nginx. The filter adds the collector's hostname and an environment field. The output rule access.** then sends the events to files. In a routing scope, Fluentd processes matching filters in order until the first matching output. Put specific outputs before broad patterns, and put the filters they need before them. To send an event to several destinations, configure an explicit fan-out such as out_copy. Configuration and match order
pos_file records the input's read position. With that state preserved, Fluentd can resume reading after a restart. When a file is already present at startup and has no saved position, the default is to start at the end. Set read_from_head true if you intend to read existing lines. Give each tail input its own position file and make sure the service account can read the logs and write to the position and output directories. A saved position says how far the collector read. Check the destination separately to confirm delivery. Tail input
The built-in nginx parser uses the timestamp it extracts as the event time. Check its pattern against your actual log_format and adjust the parser if the format differs. The file output groups events into time slices by default, so a new output file can appear after a delay. Nginx parser · File output
Choose plugins for each stage
| Stage | Common plugins |
|---|---|
| Input | in_tail reads files, in_http receives HTTP requests, in_syslog accepts syslog over UDP or TCP, and in_exec reads output from an external command. |
| Filter | record_transformer edits fields, grep filters records by pattern, and parser parses a string field in an existing record. The external fluent-plugin-geoip enriches records with IP geolocation data. |
| Output | out_file writes files, out_http sends HTTP requests, and out_forward sends events over Forward. The external fluent-plugin-opensearch registers the opensearch output. |
The parser filter receives events that have already been split. Assemble a multiline stack trace at an input that supports multiline parsing if it needs to arrive as one event. Install external plugins in the Ruby environment that actually runs Fluentd and check their dependencies. Parser filter · Plugin management · GeoIP plugin · OpenSearch plugin
Buffer for the outage you can absorb
Replace the earlier file output with this block to send the same tag pattern to an HTTP receiver. The example sends one JSON record per line and requires Fluentd 1.10.4 or later for the explicit json_array setting. Match the format and response behavior to your receiver before using the block.
<match access.**>
@type http
endpoint https://logs.example.com/ingest
json_array false
retryable_response_codes [503]
<format>
@type json
</format>
<buffer>
@type file
path /var/log/fluentd/buffer/http
flush_mode interval
flush_interval 5s
total_limit_size 512m
overflow_action throw_exception
retry_max_interval 30s
retry_forever true
</buffer>
</match>
A file buffer keeps pending chunks on disk. retry_forever removes the time and attempt limits for retryable failures. In this example HTTP 503 is explicitly retryable. A 400 caused by a malformed payload generally calls for a format fix. A persistent disk helps a restart recover pending chunks, but an outage still fills the queue. 512m caps this buffer instance and you also need filesystem headroom. The actual backlog window depends on the incoming rate and queue limits. HTTP output · Buffer settings
With Forward, out_forward can distribute traffic across weighted receivers and fail over when one is unavailable. Replication needs a separate fan-out rule. The default sends without waiting for an acknowledgment. Setting require_ack_response true makes the sender wait for an ACK from in_forward, enabling at-least-once delivery on that hop. If the receiver accepts a chunk but the ACK goes missing, a retry can create a duplicate. Use downstream checks to establish when records are stored or indexed. Forward output
Where Fluent Bit fits
Fluentd is useful when a pipeline depends on particular plugins or transformation rules. Fluent Bit is designed for resource-constrained collection and can run as a Kubernetes DaemonSet. One option is to collect at the node with Fluent Bit and forward logs to a Fluentd aggregation tier. If the required filtering and destination are available at the node, direct output can simplify the pipeline. Compare actual CPU and memory use with the intended plugins, buffer configuration, and event volume. Fluent Bit project comparison · Forward interoperability
Send existing Fluentd logs to another receiver
Keep the inputs and filters while you check the output contract. Guance DataKit's Log Streaming collector exposes /v1/write/logstreaming. Enable it for your DataKit deployment and point the Fluentd HTTP output at the reachable endpoint, for example with ?source=nginx&service=web. The default text mode can receive newline-delimited JSON from json_array false. Use a Pipeline or the collector's json_as_fields setting when the JSON keys need to become fields. The separately documented type=influxdb path requires InfluxDB line protocol. DataKit Log Streaming
Use known sample events with stable identifiers to compare what each receiver can query. Fluentd's tag and event time are separate from the record, so add the values you need as explicit fields before ordinary JSON serialization and map them at the destination. Check event time, multiline records, field types, and duplicates. A small, isolated receiver interruption can also show whether the buffer grows, drains after recovery, or resends records. Switch the production route after those observations agree with the expected behavior.
Common questions
How is a tag different from a record field?
The tag is routing metadata, and the fields make up the record. To query by tag at a destination that reads JSON fields, use record_transformer to put ${tag} into an explicit field. Record transformer
What happens when the buffer directory fills?
A configured buffer limit and a full filesystem are separate failure conditions. For overflow_action, choose throw_exception to raise an error, block to wait for space, or drop_oldest_chunk to discard the oldest chunk. Disk write failures still need disk capacity management. When in_tail sees BufferOverflowError, it pauses reading and updating its position file until the error clears. If rotated files disappear during that pause, the unread events may be lost. Buffer limits · Tail input overflow
What limits Fluentd throughput?
Look at parsing and filter CPU, file buffer I/O, and the rate at which receivers can accept chunks. A fixed events-per-second ceiling would ignore the actual workload. You can distribute inputs across instances or use more workers where the plugins support it. Compare with the same event size, parsing rules, load and acknowledgment requirements. Multiple workers
How do configuration changes take effect?
Use fluentd --dry-run -c fluent.conf in the deployment's runtime environment to check configuration and plugin loading. Linux installations from fluent-package usually use /etc/fluent/fluentd.conf. The older td-agent package has reached end of life. systemctl reload fluentd depends on the installed unit's reload support. On Unix, SIGHUP sent to the supervisor reloads the configuration by restarting workers. Check the plugins and buffers first, since failed memory-buffer flushes during that restart are not retried. Config paths · Command options · Signal behavior
Related reading
- Vector architecture and configuration (Simplified Chinese)
- Fluent Bit explained
- Fluentd vs. Fluent Bit (Simplified Chinese)
- Choosing a log collector
Documentation checked November 6, 2026.
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
