Check Fluent Bit’s internal errors and delivery health

Check Fluent Bit's own logs and its runtime metrics together. Startup logs reveal configuration or plugin failures; counters show whether a running process is actually delivering records. A live process alone does not establish a healthy pipeline.

Increase diagnostic logging temporarily

Set the service log level in the configuration:

[SERVICE]
    Log_Level debug

Inspect the process output through your service manager or container runtime. Debug logging can expose plugin initialization and retry details, but it can also produce substantial output. Return to the normal level after capturing the relevant failure.

If your deployment uses an environment variable such as FLB_LOG_LEVEL, make the mapping explicit:

[SERVICE]
    Log_Level ${FLB_LOG_LEVEL}

Set that variable before starting the process. Do not assume an arbitrarily named environment variable overrides the configuration unless the image or launcher documents that behavior.

Enable the monitoring endpoint

For diagnosis from the same host, bind to loopback:

[SERVICE]
    HTTP_Server On
    HTTP_Listen 127.0.0.1
    HTTP_Port 2020

Combine these settings with the existing [SERVICE] section. Query the metrics:

curl -fsS http://127.0.0.1:2020/api/v1/metrics | jq .
curl -fsS http://127.0.0.1:2020/api/v1/metrics/prometheus

Compare input records with successful output records, retries, and errors over time. Buffering can create a temporary gap. An increase in retries_failed means chunks have exhausted their allowed retries; it is more serious than a retry that later succeeds. Several output counters count chunks, not individual log lines, so read their documented units before estimating data loss.

Use health checks with care

Enable the health evaluator alongside the HTTP server:

[SERVICE]
    Health_Check On
    HC_Errors_Count 5
    HC_Retry_Failure_Count 5
    HC_Period 60

/api/v1/health reports health based on output-error and failed-retry thresholds over the configured window. It is not a complete test of every input, parser, or destination.

A Kubernetes liveness probe that restarts Fluent Bit for an external destination outage can create restart loops and lose in-memory buffers. Consider readiness, alerts, and persistent buffering separately. A kubelet HTTP probe also needs a reachable Pod interface; a loopback-only listener is intended for local access or an in-container check, not direct probing via the Pod IP.

Interpret the failure in context

Restrict access to monitoring endpoints; they expose operational information. Reference: Fluent Bit monitoring and configuration variables.