Choosing a log collector: Vector, Fluent Bit, or the OpenTelemetry Collector

Reading files on a node, cleaning up complex events, and forwarding OpenTelemetry data are different jobs. Fluent Bit, Vector, and the OpenTelemetry Collector overlap, but their value depends on which job the team needs to simplify. Vector is worth considering for complex transformations. Fluent Bit is a natural candidate for established node-level file collection. A team already standardizing logs, metrics, and traces on OpenTelemetry may get more value from the Collector.

The failure path can matter more than the length of the configuration. Parsing JSON, removing a token, and converting a duration to a number takes only a few lines. Send an event with duration_ms="n/a", however, and the result deserves a closer look.

The test recorded for this article used Vector 0.59.0 and the following VRL program:

. = parse_json!(.message)
del(.token)
.duration_ms = to_int!(.duration_ms)

For the valid sample, the output contained a numeric duration and no token. For the conversion failure, the default remap configuration forwarded the original event. Its message still contained the original token. Executing del earlier in the program did not sanitize the original event used by the failure path.

Vector makes that behavior configurable. The team has to decide whether failed events should be preserved, isolated, or discarded. That decision affects the rest of the pipeline, so implementation effort cannot be estimated only from the rules needed for valid input.

Test scope: the original validation, dated October 9, 2026, ran three synthetic samples through Vector 0.59.0. The other product comparisons use official documentation; this is not a throughput benchmark of seven collectors. The sample token was a deliberately constructed noncredential string. The saved validation artifacts are linked below.

Handle transformation failures explicitly

By default, a runtime error in Vector's remap transform sends the original, unmodified event downstream. With drop_on_error=true and reroute_dropped=true, the valid event in the recorded test followed the normal output. The conversion failure and malformed JSON event went to normalize.dropped with error annotations. That output still contained the original events. Routing them elsewhere did not redact them. See the Vector remap reference.

Preserving the original can be a deliberate choice for diagnostic logs whose parser will be repaired later. If sensitive fields must be removed before data leaves the node, the failure path needs its own policy: restricted quarantine, a separate sanitization step, or an explicitly approved discard policy with counters. Enabling drop_on_error without handling the relevant output can replace a disclosure problem with silent loss.

VRL is useful when multiple input formats, derived fields, and error routes need to be maintained together. Rules can be reviewed and versioned. That also means someone must maintain representative invalid samples and regression checks. A format change requires reviewing both the successful output and the exception path.

Fluent Bit has parsers and filters, and the Collector's filelog receiver and processors also support transformations. If existing rules are simple and maintainable, another expression language may not justify a migration. Vector becomes more compelling when organizing and testing the rules is itself a recurring maintenance problem. See Fluent Bit parsers and the filelog receiver.

Separate file checkpoints from delivery queues

Fluent Bit's Tail input can use a SQLite db file to track files and read positions. That state can help it resume after a restart. It is not a queue containing every event that has been read but not yet delivered. Whether outgoing data is buffered on disk depends on the storage configuration and output behavior.

The OpenTelemetry filelog receiver can use a storage extension to persist reading positions. Surviving an exporter restart may also require a persistent sending queue, supported and configured for that exporter. Defining a component in a configuration file is not enough: the receiver, exporter, and service must reference it correctly, and the selected Collector distribution must include it. See the Tail input reference and Collector resiliency guidance.

The distinction explains a common restart failure. A checkpoint tells the collector where to read next. A delivery queue holds events already read but not yet delivered. If the checkpoint advances while those events remain only in memory, a restart can lose them without rereading the file. An apparently clean resume may conceal a gap.

Vector's disk buffers and end-to-end acknowledgments also need to be understood as part of a particular component chain. A durable buffer can accept delivery responsibility before the final analytics backend has persisted the event. Durability on one node is also different from replication across nodes. See Vector's guarantees and end-to-end acknowledgments.

A local disk queue may be enough to ride out a brief network interruption. Recovery after permanent node loss, or independent replay to several backends, is a different architectural requirement. Changing collectors alone may not meet it; a replicated replay layer may be necessary.

Test long lines and file rotation

Tail maintains a per-file buffer that can grow to a configured limit. A line exceeding that limit can cause the file to stop being monitored. Enabling skip_long_lines allows collection to continue past the oversized line, at the cost of discarding that line. Neither behavior is automatically acceptable for long stack traces, request payloads, or audit records.

The number of active files also affects memory use. At an initial allocation of 32 KiB per file, 1,000 active files account for about 31.25 MiB before chunks, parsers, output queues, and process overhead. A memory measurement from a single file does not describe a busy container node.

Rotation requires a separate test. Rename-based rotation and copytruncate have different failure windows. With copytruncate, data can change between the copy and truncation. A path pattern that matches both the current file and its rotated copy may also produce duplicates. A working Filebeat deployment may already embody useful knowledge about these behaviors. That knowledge is part of the migration cost, even if another collector has a smaller idle footprint. The Fluent Bit Tail documentation describes its file handling.

Fluent Bit can be a good fit for container file collection when the team already understands its plugins and rotation behavior. Vector can handle more complex transformations at an aggregation layer. Both can coexist, provided ownership of each field transformation is clear. Uniformity is less valuable than a pipeline the team can explain and recover.

Consider the whole telemetry pipeline

When collection also includes traces and metrics, the OpenTelemetry Collector can consolidate receiving, processing, and exporting those signals. Comparing only the memory used to collect logs misses the cost of maintaining separate ingestion services.

A shared pipeline still needs correct mapping. service.name on a resource is not necessarily interchangeable with an attribute of the same name on an individual log record. A trace ID embedded in a message may not automatically populate the backend's correlation field. Establish those mappings and test the destination's interpretation.

The Collector distribution also determines which components are available. File collection requires a build that includes filelog. A team already managing OpenTelemetry versions and configuration can reuse that work. Introducing that operating model just to read a few local files may offer less benefit.

Size the buffer for an outage and its recovery

Start with the rate at the buffer itself. If events enter a node's queue at an average of 2 MiB/s, one hour of downstream unavailability requires about 7.03 GiB of payload capacity. Across 100 nodes, that is roughly 703 GiB. Serialization overhead, free-space margin, and ongoing traffic during recovery are additional.

Do not mix uncompressed file bytes with compressed network traffic when calculating this number. The buffer may hold a representation that differs from both.

A full buffer needs a defined policy. Fluent Bit documents that when an output reaches storage.total_limit_size, its oldest queued chunk can be discarded to make room for newer data. This is bounded buffering, not indefinite retention. See Fluent Bit buffering.

After the destination recovers, outgoing throughput must exceed new incoming traffic for the backlog to shrink. More disk space only extends the waiting period if the destination can never catch up. Budget for both outage storage and additional recovery throughput. When the backend is the bottleneck, replacing the collector is unlikely to solve it.

Decide whether an existing collector should stay

Stable rules and plugins are operational assets. Replacing Filebeat, Logstash, Fluentd, or Rsyslog means validating the protocols, rotation, buffering, and recovery behavior they currently provide. A migration is easier to justify when it removes meaningful maintenance, resource use, or duplicated collection. A tidier architecture diagram alone is a weak reason. References: Filebeat, Logstash, Fluentd buffering, and Rsyslog.

Where DataKit can reduce the number of components

If Guance is the destination, DataKit's file and container collection and Pipeline processing may cover responsibilities currently assigned to separate services. Handling routine parsing, timestamps, and multiline records there can remove a processing service and the work needed to keep its configuration in sync. Start with the DataKit logging guide and log Pipeline guide.

The original article series includes a six-sample Pipeline validation covering numeric conversion, event time, and status normalization. It does not test DataKit file rotation or crash recovery. Those results support conclusions about processing only.

If a stable Vector deployment also performs complex transformations, multiple-destination routing, or a specialized protocol function, keeping it can be cheaper than rewriting everything. Validate the destination protocol and field mapping, then retain the responsibilities it handles well. Where DataKit already meets the ordinary collection and processing requirements, consolidating those components may reduce deployment and maintenance work.

The savings become credible once the design names the layer that disappears and the team responsible for each layer that remains.

The saved Vector inputs, configurations, and results and six-sample Guance Pipeline validation include Chinese explanatory notes. They do not cover throughput, file rotation, power loss, or permanent node failure. Those topics remain selection and acceptance-test criteria, rather than measured results from this test.