Vector log collection: VRL, routing, and disk buffers
Follow an Nginx access log through a Vector data pipeline, use VRL to control fields, and see when disk buffers and HTTP retries can preserve or duplicate events.
Nginx is still writing access logs, but the chart for 502 responses goes flat after a configuration change. The first thing to check is the record leaving the collector. If the parser stopped extracting status, the log can still reach storage while the query that counts 502s sees nothing.
Vector is a Rust-based observability data pipeline. Its sources receive events, transforms change them, and sinks send them on. This guide follows a file log from the source through Vector Remap Language (VRL) to the output, then looks at what happens when the receiver falls behind. For metrics and traces, check the signal types supported by each component. Vector’s OpenTelemetry source can receive traces, but its trace support is experimental. OpenTelemetry source
Follow an access log through the pipeline
Save this example as vector.toml. Create ./vector-data in the working directory and give the Vector process write access to it. The process also needs read access to the Nginx log. Run vector validate vector.toml and then vector --config vector.toml. With read_from = "end", append a new test line after Vector starts watching the file. A previously checkpointed file resumes from its saved position.
# vector.toml
data_dir = "./vector-data"
[sources.nginx_access]
type = "file"
include = ["/var/log/nginx/access.log"]
read_from = "end"
[transforms.parse_nginx]
type = "remap"
inputs = ["nginx_access"]
source = '''
fields, err = parse_regex(.message, r'^(?<ip>\S+) \S+ \S+ \[(?<time>[^\]]+)\] "(?<method>[A-Z]+) (?<path>[^" ]+)[^"]*" (?<status>\d{3}) (?<size>\d+|-)')
if err == null {
. |= fields
.status = to_int!(.status)
if .size == "-" {
del(.size)
} else {
.size = to_int!(.size)
}
} else {
.parse_error = true
}
'''
[sinks.console]
type = "console"
inputs = ["parse_nginx"]
encoding.codec = "json"
The inputs lists define the graph. The file source adds the raw line as message; parse_nginx extracts fields and converts status and known response sizes to integers. A dash in the size column leaves size absent, so it stays distinct from zero bytes. The example keeps the original Nginx time as text. If a downstream time chart needs event time, parse it and map it to the field the receiver actually uses. The console sink lets you inspect that output before connecting to a backend.
192.0.2.10 - - [09/Nov/2026:10:00:00 +0800] "GET /checkout HTTP/1.1" 502 128
This 502 should appear with path = "/checkout", numeric status = 502 and size = 128. A line in an unexpected format goes through with parse_error = true and its original message. Count those events separately. If access logs can contain credentials or personal information, use synthetic records on the console and decide where unparsed events may go before enabling an external sink. File source
Use VRL to control what leaves Vector
Consider a separate source where message contains JSON with both a log level and a password. Deleting the password only from a parsed object can leave it in the raw string. If the downstream route needs just the normalized level and status, set the entire output event to those allowed fields.
parsed = parse_json!(.message)
level = downcase(string!(parsed.level))
if !includes(["trace", "debug", "info", "warn", "error", "fatal"], level) {
level = "unknown"
}
status = if includes(["error", "fatal"], level) { "error" } else { level }
. = {"level": level, "status": status}
Here parse_json! and string! stop the VRL program when the input is invalid. The remap transform controls what then happens to the event. For this redaction route, set drop_on_error = true and drop_on_abort = true on that transform and monitor its discarded-event count. A remap can otherwise forward the unchanged event when processing fails. This example restricts level values to six known strings and maps any other string to unknown. It discards all other fields. Add fields such as an event timestamp or correlation ID to the allowlist only after checking their types and contents. Keep the Nginx parser and this JSON rule on separate inputs. Remap transform · VRL errors
Budget for a slow receiver
Add the HTTP sink below to the first configuration. Replace the example URI with your receiver and check its authentication and expected framing. Both sinks will receive events from parse_nginx; remove the console sink after local validation if it is unnecessary.
[sinks.platform]
type = "http"
inputs = ["parse_nginx"]
uri = "https://logs.example.com/"
encoding.codec = "json"
buffer.type = "disk"
buffer.max_size = 5368709120
buffer.when_full = "block"
This sink can queue up to 5 GiB on disk. Keep the Vector data directory on persistent, writable storage. A process restart can recover buffered data that was synchronized to disk. That capacity is finite. With block, a full buffer pushes back on the source. A file can rotate out before Vector catches up, and a disk fault or crash before synchronization can still cost data.
The receiver matters too. An HTTP 429 or 503 can be retried; a 400 usually means you need to fix the payload or endpoint. If the receiver writes a batch but its response never reaches Vector, retrying can create duplicates. Plan capacity using the byte rate at the buffer and the longest outage you expect to absorb. Then check whether the receiver has enough spare throughput to clear the queue while traffic continues. HTTP sink · Buffering model
Put the transformations where you can operate them
A Vector Agent can tail files on each host or node and send directly to a backend. An Aggregator can accept events from many Agents and centralize shared transformations and routing. Centralization reduces the number of places a rule must change. It also concentrates network traffic and backpressure. Measure event size, transform cost, disk usage and sink throughput using your own workload before choosing where to put the work. Deployment roles
When moving file collection to Guance, start with a representative sample collected by DataKit. Compare the event time and numeric field types in Log Explorer, including a multiline stack trace if your application emits one. Reimplement any required VRL logic as Pipeline rules. A valid VRL script does not run unchanged in Pipeline. If you keep Vector and only change the destination, verify the receiving interface and the shape of the HTTP body before redirecting the whole stream. DataKit log collector · Guance log processing
Common configuration questions
Can a source send directly to a sink? Yes, when the sink accepts the source's event type and the receiver understands the payload. Vector accepts TOML, YAML and JSON. Validate the specific file your deployment loads.
Does a checkpoint mean the receiver has the event? A file checkpoint records a read position. It is separate from durable storage at the destination. Check the source and sink acknowledgement support, and keep enough rotated files to cover lag at the source. End-to-end acknowledgements
How should I compare Vector with Fluent Bit? Try the transformation you need on a sample and check how each collector behaves when the output slows down. If your existing Fluent Bit filters and filesystem buffering handle that workload, switching collectors brings a migration cost. If field-level transformations are hard to express and maintain, put the VRL version beside the current configuration and compare what each one emits. Fluent Bit buffering
Further reading
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
