What is log aggregation? From collection to searchable logs

Learn how log aggregation collects records across services, preserves event time, handles delivery failures, and routes logs by retention.

Best practices
Illustration of logs, metrics, and traces

At 10:02, checkout starts returning errors. One service logs the upstream timeout to stdout, the gateway writes its response to a file, and the database emits a slow-query record. If the three records are still on separate machines, you have to find each one and work out whether their clocks and request IDs line up.

Log aggregation collects records from distributed sources and makes them searchable in a shared destination. Collection gets an event off its host. Aggregation also gives operators a common way to find related records. Log management covers the rest of the lifecycle, including access, retention, and deletion. In this example, start with the checkout path rather than trying to onboard every host.

Start with the records you need to join

Map the applications, gateway, database, container runtime, hosts, and cloud services that can explain a failed checkout. For each source, note where the record is written, which collector can read it, and how quickly that copy rotates away. Decide where it can be stored and who may query it before wiring the collector to a destination.

Use stable service and environment fields to find the right deployment. When the applications propagate and record the same request or trace ID, you can narrow the search to a single request. Otherwise, comparing records in the same time window can reveal a pattern, but time proximity alone cannot join them reliably.

Pick a collection path for each source

Source Collection path What to check
Application or database files DataKit file collector The service account can read the file and its replacement after rotation
Kubernetes stdout and stderr DataKit container collector The runtime log path is accessible and collection scope includes the intended pods
Remote sender DataKit socket input or HTTP Log Streaming Sender and receiver agree on protocol and payload. Set up the inputs separately
Files inside a pod DataKit logfwd sidecar The app and sidecar share the log volume. Inject pod metadata when you need it

For a host file, copy the sample to a logging.conf file under DataKit's conf.d and adapt the paths and tags. If service is omitted, the file collector uses source as its value. Restart DataKit, write a test record, and verify that the collector's account can read it.

[[inputs.logging]]
  logfiles = ["/var/log/checkout/*.log"]
  source = "checkout-file"
  service = "checkout"

A matching path does not decide where reading starts. DataKit 1.5.5 and later prefer a valid saved offset. Without one, it considers from_beginning, the size threshold for reading a newly discovered file from the start, and then reading from the end. Verify the result with a file that contains both existing and newly appended records, then repeat the test after rotation. Treat a historical import as separate work. DataKit log collector · Log Streaming · logfwd sidecar

Keep event time, severity, and record boundaries intact

Suppose the checkout service writes a JSON event like this.

{
  "timestamp": "2026-11-09T10:02:14+08:00",
  "level": "ERROR",
  "request_id": "checkout-017",
  "message": "inventory request timed out"
}

If the collector falls back to its current time, an older event read from a rotated file may appear in the wrong search window. In a Guance Pipeline, map the source timestamp to time, parse it with its time zone, and map the severity to status. Check a delayed sample and a malformed one. For DataKit's file input, the default status is info when nothing else sets it. A text stack trace needs a separate boundary check. In the file input, multiline_match identifies the first line of each event when multiline processing is enabled.

Redact before data leaves the host when the destination cannot receive raw secrets. With a local Pipeline, removing an extracted field with drop_key() can still leave its value in the original message. Check both fields and raw text. Use drop() only when the whole event can be discarded. Review exclusion rules on a representative slice of traffic so that losing routine noise doesn't also hide audit records. Guance Pipeline execution · Pipeline functions · Blacklist rules

Work out what happens when the destination is slow

The collector can run while the destination is unreachable. Watch upload retries, time lag between event and arrival, and available queue space as well as the collector's own status. DataKit's disk WAL has a size limit. Depending on configuration, a full queue can discard newly collected data or block collection for the selected data category. Size the queue and the source-file retention for the outage you need to survive, then measure whether the pipeline can drain while new logs keep arriving.

UDP provides no delivery acknowledgment. With a sender that retries after an ambiguous failure, a record may be delivered twice. If missing or duplicate audit events are material, keep a stable event identifier where you can and reconcile what was sent against what arrived. Check that a fallback copy is still present after rotation before relying on it for recovery. DataKit WAL configuration

Set retention around how logs will be used

A useful destination lets you search by service, environment, and time without changing where applications write. In Guance, a workspace is the destination for the collected data. Once additional regular log indexes are enabled, matching proceeds in configured order and writes to the first match. Put a narrow audit rule above a broad production rule if both would accept the same event. Use retention and access requirements to decide which records need separate treatment.

An index used during an incident and an archive used for an annual audit have different access patterns. For eligible data, Data Forwarding can send records to supported object storage. Confirm which rule selects them and practice retrieving a known record from the archive. When choosing between a self-managed and a hosted backend, compare the cost of keeping the same data for the same time, including parsing, storage, access, and the work of restoring it. Guance log indexes · Data Forwarding

Once the checkout events arrive in the right time window with useful fields, the same records can support log-based trends and alerts. Log monitoring covers that next step. For policies that apply across the whole lifecycle, see log management in Simplified Chinese.

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting