Microservices logging best practices: follow a request across services
Standardize microservices logs, preserve event time, avoid duplicate collection, and correlate logs with OpenTelemetry traces and Guance.
A checkout request returns a timeout. The gateway logs it as an error, but the payment service finishes the charge a fraction of a second later. A delay in the gateway’s log stream lets the payment record reach storage first. Sorting by arrival time gives you the order of delivery. To understand the request, you need the time each event occurred and a way to find the records that belong to it.
Good microservices logging makes those two things reliable. Give teams a shared event format, collect each stream once, and put the active trace context on the records that need it. Then decide what to keep and who can read it.
Keep event time and collection time separate
Agree on a few fields before choosing a backend. Give the event a timestamp with an explicit offset, such as UTC in RFC 3339 format. Keep a stable service identity, environment and release version, and use a short, consistent event name. Agree on the severity mapping so a warning from one framework can be filtered alongside warnings from another. Synchronize host clocks and account for skew when comparing events across services.
Here is one application record for a synthetic request. It is a JSON line emitted by the application. A logger can fill in the active span’s IDs when it writes the record.
{"event_time":"2026-10-01T12:00:00.425Z","service":"payments","env":"production","version":"1.8.4","severity":"INFO","event":"payment.completed","event_id":"evt_payment_01","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7"}
OpenTelemetry’s log data model distinguishes the event’s Timestamp from ObservedTimestamp, the time the collection system observed the event. It also separates Body, severity, the emitting Resource and optional trace context. If you convert JSON logs to OpenTelemetry records, make the mapping explicit. For example, map the emitting service and deployment fields to Resource attributes and preserve the original severity text alongside a normalized severity number when available. The JSON keys above describe an application schema. Use OTLP’s own representation when exporting. OpenTelemetry Logs Data Model
With Guance, use a log Pipeline to parse the event time into time and map severity to status. Check the stored record with a sample whose time, zone and severity you already know. If a parser falls back to a collection timestamp, a delayed record may appear outside the incident window.
Collect each stream once
Start with the path each service already uses. On Kubernetes, a node-level collector can read the container runtime’s logs for application stdout and stderr. A sidecar is a deployment pattern. It can read an application file and send it onward or stream it to its own stdout. If both paths pick up the same event, the backend can receive two copies. Decide which path owns it before turning on a second collector. Kubernetes logging architecture
Centralize the streams and watch for delayed or repeated delivery. If an exporter loses the response to a batch that was already accepted, a retry can deliver it again. If you need deduplication, generate a stable event ID at the source and apply an explicit policy in the consumer or backend. A trace ID identifies the trace and may occur on many distinct log records. OTLP duplicate-data limitations
For a Guance workspace with multiple log indexes, start with the retention and access needs of each class of log. Set the rules for regular indexes accordingly. Logs routed by those rules are written to the first matching regular index, so validate an event that matches two rules before reordering them. An index rule controls placement. Review the corresponding access controls separately.
Put the active trace context on the record
A request ID from a gateway helps when you need an application-level reference. A trace ID identifies a distributed trace, and a span ID identifies an operation within it. Keep the two kinds of IDs in separate fields when you use both.
At an inbound boundary, extract the incoming trace context. On outbound calls, inject the context into the supported carrier. When the application logs, read the active span context and include its trace_id and, when available, span_id. For a non-OTLP JSON log using OpenTelemetry’s naming convention, the IDs are lowercase hexadecimal strings, with 32 characters for the trace ID and 16 for the span ID. Leave trace fields absent on records outside a trace. A collector cannot reconstruct missing application context simply by receiving the record. Trace context in non-OTLP log formats · OpenTelemetry context propagation
Follow the request from either side
Return to the checkout example. Find the trace for the relevant service and time window, then follow the failed span. Search the logs by the same trace ID, and use the span ID to narrow to the payment attempt if the field is available. If you begin with the gateway’s log, use its trace ID to look up the trace. The gateway timeout and the later payment result can both be valid, so inspect the event timestamps and the span relationships before inferring what happened.
In Guance, open Related Logs in the trace details. The documented default correlation field is trace_id. The trace’s service, env and version can determine which log indexes are selected through index mapping. If a known record is absent, query its index directly and compare the stored fields with the trace before changing the application instrumentation. Both the logs and trace must have been collected and still be available. Trace sampling can leave a searchable log whose trace is missing.
Protect the logs you plan to keep
Define the fields applications may log. Keep passwords, access tokens and raw payment details out of the record. For fields you must keep, redact or mask them before they reach a destination that should not receive them. Test normal, malformed and multiline input. A parsed field can look clean while the original body still contains the value. Restrict read and export permissions, and audit changes to them. In Guance, the index audit view records operations on the index. OWASP logging guidance · Redact sensitive data in the Collector
Set retention by investigation needs, applicable requirements and cost. Measure the lookup window your responders actually use, and test retrieval before moving records to an archive. Security audit logs and short-lived diagnostics can have different lifecycles.
A service mesh can produce its own proxy telemetry and help propagate context, depending on the instrumentation. Application events such as “charge submitted” still belong in the service’s logs. Validate one request across the gateway and payment service first. Check the event time, stored IDs and selected index at each hop, then extend the same approach to other paths.
Further reading · Log monitoring
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
