Fluent Bit explained: Tail, log parsers, and SQL stream processing
Configure Fluent Bit inputs, filters and outputs; register log parsers, collect Kubernetes metadata, track Tail offsets, and load a windowed SQL stream-processing task.
Fluent Bit is a lightweight log and telemetry processor in the Fluentd ecosystem, implemented primarily in C. It can run on hosts, Kubernetes nodes, and edge devices. Its small-footprint design does not put a fixed ceiling on memory: open files, parsing, metadata caches, and queued data all consume resources. Build options and plugins also determine its dependencies.
This guide follows the Fluent Bit 3.2 documentation for classic configuration and SQL syntax. That is an explicit example scope, not a recommendation that 3.2 is the latest release. The aim is to connect the configuration to what happens to a record: how it is read, parsed, routed, and retained when an output falls behind.
Key points
- Lightweight still needs a budget. Measure the effect of throughput, file count, filters, and backlog before sizing a collector on every node.
- Classic configuration separates stages.
[INPUT],[FILTER], and[OUTPUT]define collection, processing, and delivery. YAML is also supported; check the deployed version's options. - Kubernetes enrichment has prerequisites. Tags and filenames identify the workload; cached metadata and configured API Server or Kubelet access supply additional fields.
1. Follow a record through the configuration
Save this as /etc/fluent-bit/fluent-bit.conf. Section 3 defines the referenced parser file. The directories must exist; the process needs to read the configuration and source logs and write its database under /var/lib/fluent-bit/.
[SERVICE]
Flush 1
Log_Level info
Parsers_File /etc/fluent-bit/parsers-app.conf
[INPUT]
Name tail
Path /var/log/app/*.log
Tag app.*
Parser app_json
DB /var/lib/fluent-bit/tail-app.db
Mem_Buf_Limit 10MB
Refresh_Interval 5
[FILTER]
Name record_modifier
Match app.*
Record hostname ${HOSTNAME}
Record env production
[OUTPUT]
Name http
Match app.*
Host logs.example.com
Port 443
URI /ingest
Format json
tls On
The input reads records and assigns tags. Filters and outputs select those records through Match. For Tail, the * in Tag app.* expands to the absolute filename with slashes replaced by dots. It is not the same literal tag on every record. Refresh_Interval controls discovery of files, while Flush controls the engine's flush cadence. A one-second flush setting is not an end-to-end delivery guarantee. The environment must supply ${HOSTNAME} for that substitution to work. Tail configuration
Before pointing the HTTP output at a different backend, check the endpoint, authentication, payload format, and response semantics against a known record. Changing Host and URI does not complete a migration. This example uses memory buffering. Reaching Mem_Buf_Limit can pause ingestion, and the Tail database stores read offsets rather than a durable queue of unsent records. Buffering and storage
For files you want to collect into Guance, another option is DataKit's log collector. Configure accessible logfiles paths, then set the source, Pipeline processing, and event-time mapping for the application. This is a separate collection path: it does not reuse Fluent Bit's Tail database, parser definitions, or SQL tasks. A switch still needs an explicit reading position, field mapping, and handling of overlapping collection.
2. Choose the plugins the records need
| Stage | Common plugins |
|---|---|
| Input | tail for files, systemd for journal records, syslog, http, opentelemetry |
| Filter | grep for inclusion/exclusion, record_modifier for added fields, modify for conditional changes, kubernetes for metadata, lua for scripts, nest for record structure |
| Output | http, file, forward for destinations such as Fluentd, kafka, es/opensearch, s3 |
Availability depends on the version, platform, and build. A package need not contain every plugin in the table.
The Kubernetes filter identifies the Pod, namespace, and container from information such as the tag. It then uses cached data or configured access to the API Server or Kubelet to obtain additional metadata. It does not simply retrieve every field from the container runtime.
For /var/log/containers/*.log, align the host-log mounts, tag, Kube_Tag_Prefix, and Docker/CRI parsing. API access also depends on service-account credentials, RBAC, and network access. The ordinary app.* file input above is not a complete Kubernetes configuration.
Metadata enrichment and application-message parsing are separate steps. Merge_Log On can parse JSON in the log field, or use the configured parser, and merge the result. The default Keep_Log behavior preserves the original field, so its presence does not prove parsing failed. Kubernetes filter and Merge_Log
3. Register the log parser before referencing it
A log parser needs a definition loaded at startup. Put these definitions in /etc/fluent-bit/parsers-app.conf, which the first example loads with Parsers_File:
[PARSER]
Name app_json
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S%z
Time_Keep On
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>\S+) \S+ \S+ \[(?<time>[^\]]+)\] "(?<method>\S+) (?<path>\S+) \S+" (?<code>\d{3})(?:\s|$)
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
Time_Keep On
Types code:integer
The Tail input uses Parser app_json for one JSON object per line:
{"time":"2026-11-05T10:00:00+0800","level":"error","pod_name":"checkout-7d8f","message":"upstream timeout"}
Here, the +0800 offset matches %z. If the source uses a different timestamp format, change the format as well as the key. Time_Keep On keeps the original time field after it is parsed.
For Nginx access logs, configure the relevant input to use Parser nginx. Its timestamp format matches a value such as 05/Nov/2026:10:00:00 +0800. The pattern captures the request path separately from the HTTP protocol and converts code to an integer.
Fluent Bit uses Onigmo in Ruby mode for regular expressions, including named captures. A pattern written for a different regex dialect may need changes; multiline stack traces need multiline configuration as well. JSON is a parser format. A parser named json exists only if that definition has been loaded—it is not a universally available zero-configuration name. Parser registration · Regex and time parsing
4. Give SQL aggregation an explicit source and window
The stream processor can filter records and aggregate them as they pass through Fluent Bit. This query counts error records by a top-level pod_name in five-second tumbling windows:
SELECT pod_name, COUNT(*) AS errors
FROM TAG:'app.*'
WINDOW TUMBLING (5 SECOND)
WHERE level = 'error'
GROUP BY pod_name;
TAG:'app.*' selects records by tag pattern. STREAM:name selects an input stream, whose default name looks like tail.0; an input Alias can name it explicitly. A tag wildcard cannot be substituted as STREAM:app.*.
The window matters too. Without one, aggregation applies to the current buffer of records rather than a stable five-second interval.
Save this task as /etc/fluent-bit/streams.conf:
[STREAM_TASK]
Name pod_errors
Exec SELECT pod_name, COUNT(*) AS errors FROM TAG:'app.*' WINDOW TUMBLING (5 SECOND) WHERE level = 'error' GROUP BY pod_name;
Add Streams_File /etc/fluent-bit/streams.conf to the existing [SERVICE] section, then load the main configuration, for example with fluent-bit -c /etc/fluent-bit/fluent-bit.conf. Do not create a second service section.
The input must already contain the top-level pod_name and level shown in the JSON example. Tail does not create Pod metadata by itself, and the Kubernetes filter's nested fields are not automatically equivalent to that top-level key.
A standalone SELECT writes its result to standard output. It does not replace the original output or reduce the raw logs being shipped. To put derived records back into the pipeline, use CREATE STREAM ... WITH (tag='new-tag') AS SELECT ... and route that tag explicitly. Keep derived records from matching their own task again. Each window counts the records this instance actually processed; cross-node aggregation and backend alerts remain separate work. SQL and windows · Stream names and tags
Common questions
How are Fluent Bit and Fluentd related?
They belong to the same ecosystem but are separate processors, not two generations that must be adopted in sequence. Fluent Bit is primarily C; Fluentd uses Ruby and C. Running Fluent Bit on nodes and Fluentd centrally over the Forward protocol is a common arrangement. Either can also act as a forwarder or aggregator, or run independently, depending on the plugins and deployment. Official project relationship
How should I debug a configuration?
Switch the output to stdout to inspect the records and tags, then use Log_Level debug to inspect parser loading, parsing, and routing. A log field may come from the input, be a legitimate application field, or remain because of Keep_Log. Compare a known sample's expected fields, types, and event time instead of treating that field as proof of failure.
Can Tail collect a record twice?
Yes. DB stores tracked-file information and offsets, but recovery still depends on persisting that state and handling file identity, rotation, truncation, and the reading position. At startup, a discovered file without a saved offset is read from its end by default; use Read_from_Head deliberately if existing content should be read. Rotate_Wait controls how long to continue watching a rotated file, while Ignore_Older excludes files by age. Neither is a deduplication switch.
An offset does not acknowledge backend storage. Lost memory buffers, rotation or deletion during an input pause, and output retries or retry limits can lead to gaps or duplicates. Filesystem buffering improves recovery, but a Tail database alone cannot provide exactly-once delivery. Check source-file retention, buffering, retries, and receiver semantics together. Tail state and rotation · Output retries
When is a Lua filter useful?
Use Lua when built-in filters cannot express a record transformation, such as field redaction or a calculation. It still operates through the filter callback interface and the available runtime, with type and timestamp-precision considerations. It cannot do literally anything. Slow or blocking logic affects processing, and exceptions or field changes need sample-based checks. Prefer a simple built-in filter when it can express the rule. Lua callbacks and limits
Continue reading
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
