Reliable log delivery: buffers, acknowledgments, and consumer offsets

A log pipeline can include Kafka, receive HTTP 200 from the analytics backend, and still lose a record. After a message reaches the queue, a consumer must read it, send it downstream, and confirm that the backend accepted it. Acknowledging too early at any of those steps can remove the opportunity for a normal retry.

The apparent loss may have several causes. Only part of a batch may have succeeded. The consumer may have committed its offset before delivery completed. The record may even be stored with an incorrectly parsed event timestamp, outside the query window. More broker replicas address only a small part of this problem.

Assign each component a specific job: collector buffering for short interruptions, a message layer for independent replicas and replay, and a backend for storage and analysis. Each additional layer adds operating work. Its value depends on whether it covers a failure the business needs to survive.

Published by Guance. Reviewed October 14, 2026. The OpenObserve examples refer to the small tests recorded for the original article on October 9. The message sequences below are illustrative; no end-to-end Kafka, Redpanda, or Guance fault-injection test is claimed.

Define acceptable loss and recovery

A team may accept losing unsent debug logs when a node is permanently destroyed. Business errors and audit records may need stronger custody and replay. Applying the strictest policy to everything increases replication, retention, and recovery costs; treating everything as best effort may fail the requirements of critical records.

Loss is only one requirement. Specify how quickly records must become searchable after recovery, how long they must remain replayable, and how much duplication is acceptable. At-least-once delivery normally permits redelivery. Occasional duplicates may be tolerable during troubleshooting, while billing or exact event counts require a separate deduplication mechanism.

For a fifteen-minute network interruption, a collector's persistent buffer may be enough. Surviving permanent collector-node loss requires an independent durable copy made before that loss. No downstream platform can recover a record that never left the application. Source-file retention, collector buffering, and message retention must together cover the recovery target.

Identify what each acknowledgment proves

Records move from application files to collector buffers, possibly through a message queue, and into an analytics backend. An acknowledgment can allow the sender to discard its copy. If it arrives before final delivery, the receiver must take responsibility for the remaining work.

Vector distinguishes disk buffering from end-to-end acknowledgments; a persistent buffer can take custody of events for later delivery. A Kafka or Redpanda producer acknowledgment covers the relevant broker write, subject to its configuration. It does not establish that an HTTP analytics backend has processed the record. See Vector delivery guarantees, Kafka delivery semantics, and Redpanda producer configuration.

An HTTP status is also different from a per-event result. In the recorded OpenObserve test, the accepted event-age window was set to 24 hours. Sending an event from two days earlier returned HTTP 200 with successful=0, failed=1 in the response body. A consumer that commits on the status code alone will not recover that record simply because its queue is reliable.

Queries need the same scrutiny. A separate numeric-conversion test returned HTTP 200 alongside is_partial=true and function_error. The request completed, but the result was incomplete. Both ingestion clients and query clients must interpret application-level responses. The original response evidence and event-age test contain Chinese annotations.

Check how the output plugin handles partial failure. An existing plugin may already provide the necessary behavior. A custom delivery service makes error classification, retries, and progress tracking your team's responsibility. Cheaper backend resources can therefore come with additional software maintenance.

Commit the contiguous completed offset

Consider this illustrative sequence within one Kafka partition:

A committed offset identifies the next record to consume. The safe contiguous commit here is 101. Committing 103 because record 102 succeeded would resume after the unfinished record on restart.

Once record 101 succeeds, the consumer can advance to 103. Until then, a restart may deliver 102 again, creating a duplicate. Concurrency increases throughput while also increasing the state needed to distinguish completed records from gaps.

Partial batch success, out-of-order completion, and consumer-group rebalances all affect progress. Serial processing within each partition simplifies this logic at moderate volume. Parallel processing requires coordinated commits, retry state, and recovery behavior. Adding consumers does not remove those responsibilities.

Kafka transactions provide atomicity within their supported Kafka read/write scope. They do not automatically put an external HTTP write and a Kafka offset commit into one transaction. If the backend write succeeds but the commit fails, a restart can repeat the write. The receiving system needs verified idempotency or event deduplication for that case. Kafka transaction and delivery design

Handle a successful write with a lost response

Suppose the backend accepts an entire batch, then the connection breaks before the response reaches the consumer. The consumer cannot distinguish success from failure and may retry. Preventing both loss and duplication requires more than a retry setting.

An idempotent producer handles duplicate sends within the relevant protocol guarantees. It does not deduplicate an application that constructs a new message for the same business event. Nor does a backend field named _id establish uniqueness: the recorded OpenObserve Bulk test submitted two create operations with the same _id and retrieved two records. Verify the receiver's behavior instead of inferring it from a field name.

Generate a stable event_id before fan-out when possible, and preserve it across dual writes, retries, and replay. A hash of the message body can incorrectly merge two legitimate occurrences of the same log message. For files, an identifier may need both the file's rotation generation and the record position. Event IDs identify records; offsets track consumption progress.

Deduplication has an operating cost. Its state must remain available for the replay interval it is meant to cover, and cleaning or querying that state consumes resources. Troubleshooting may tolerate occasional duplicate rows. Billing and precise event statistics may require deduplication before aggregation. Decide based on the use of the data.

Give rejected records a durable destination

An invalid record or one outside the backend's accepted time window may fail every retry. Mixing it indefinitely with transient network failures or rate limits can block partition progress and grow the backlog.

A quarantine destination needs its own custody rules. Preserve the original record, source position, failure reason, and processing version, with controlled access and a way to replay corrected data. If the consumer advances after quarantine, that destination assumes responsibility for the record. An in-memory counter or an error message about the failure is not a replacement for the record itself.

In the original Vector test, the failed-transform route retained the original input, including a deliberately added token field. Quarantined events must therefore not be assumed sanitized. Access control, sensitive-data handling, and repair work belong in the cost of keeping the main flow moving. Vector remap error handling

Without a destination for permanent failures, extra disk only stores a larger collection of unprocessable events. Define bounded retries, durable quarantine, and a repair process before sizing the queue for recoverable backlog.

Match the architecture to the failure boundary

The strongest recovery evidence is a reconciliation of known event IDs after an interruption, with every duplicate explained. Equal totals can hide missing records offset by duplicates. Also measure recovery time: logs that eventually appear days later may be useless during an incident.

What remains after moving to a managed backend

With Guance SaaS, the provider operates the analytics storage and query backend. Your team still owns the application-to-collector path, local buffering, network access, and sending behavior. The DataKit log processing guide describes the collection and processing stages to validate.

If direct delivery meets the outage and recovery requirements, a separate Kafka deployment may add little. If an existing queue also serves archival, multiple destinations, or replay, retain those capabilities. Moving backend operations to a provider changes who runs the analytics service; the value of the message layer still comes from the jobs it performs.