12 logging best practices for useful, manageable production logs

Useful logs explain an event with enough context to investigate it, without exposing sensitive data or overwhelming the application and its operators. That requires three decisions: what to record, how to represent it, and how to manage it after collection.

Logging everything is a poor substitute for those decisions. Important events can disappear into repetitive output while ingestion, storage, and search costs grow. These twelve practices provide a starting point for reviewing an application's logging design.

Twelve practices at a glance

Start with consistent levels, a structured schema, and reliable collection. Treat sampling and retention as design decisions, rather than waiting for an oversized bill or a full disk.

1. Define the purpose before adding a log statement

Identify the application outcome you need to understand. Is the event evidence that an order completed, an explanation for an authentication failure, or a record of an administrative change? Decide whether a log, metric, or trace best answers the question.

The initial design will miss some useful context. Review real incidents and routine queries regularly, remove repetitive events, and add fields that were missing during diagnosis. The goal of an error event is to support a repair: record the operation, relevant identifiers, failure details, and enough context to connect earlier events. Do not interpret “more context” as permission to dump entire requests or database rows.

2. Use levels consistently

Define severity conventions that match your logging framework and incident process:

• INFO: an expected event with operational or business value.
• WARN: an unexpected condition that may need attention even though work continues.
• ERROR: an operation failed and needs investigation or explicit handling.
• FATAL: a failure that prevents the process or a critical subsystem from continuing, where the framework supports this level.
• DEBUG and TRACE: detailed diagnostic output, usually disabled during normal production operation.

These names are not a universal contract. Some libraries exit when a fatal method is called; others only write an event. An error can be handled or unhandled. Document those distinctions instead of inferring control flow from a severity name.

INFO is a common production threshold. For investigations, support scoped verbosity changes where practical, restrict who can make them, and set a rollback time. Changing an environment variable normally requires a process restart unless the application explicitly reloads it; a setting in one process does not automatically reach other workers.

In Guance, map application severity to the log status field. Parse legacy text with Pipeline as needed, then inspect the result. Preserve an HTTP response code in a separate field so that 500 is not confused with a logging level.

3. Emit structured events

Prefer a stable machine-readable format, commonly one JSON object per line. Keep numeric durations numeric, represent timestamps consistently with a time zone or documented epoch unit, and use the same field names across services.

Choose a framework with structured output. Where available, configure infrastructure components too: PostgreSQL supports jsonlog from version 15, and Nginx supports JSON escaping through log_format ... escape=json. Escaping alone does not design a valid JSON record; the surrounding template and value types still matter. For systems you cannot change, parse the existing output in the collection pipeline and track parse failures.

Text remains useful inside the message field. The benefit of structured fields is that filters and aggregations no longer depend on extracting every fact from that text.

4. Include enough context to explain the event

This event states an outcome but gives an operator little to work with:

{"timestamp":"2026-08-25T14:52:43Z","level":"INFO","message":"Login failed"}

A more useful event identifies the operation, reason, service, and request:

{
  "timestamp": "2026-08-25T14:52:43Z",
  "level": "INFO",
  "message": "Login failed",
  "reason": "invalid_credentials",
  "user_id": "u_12345",
  "source_ip": "192.0.2.25",
  "attempt_num": 3,
  "request_id": "req_456",
  "service": "user-auth"
}

Choose authentication-event severity through your security policy; a single rejected login and a sustained attack need not trigger the same response. User identifiers and IP addresses may themselves be sensitive, so record them only for a defined purpose with appropriate access and retention.

A request ID is not automatically a distributed trace ID. If you add trace_id, obtain the real value from the active tracing context. Do not populate it with an arbitrary string that cannot resolve to a collected trace. The OWASP logging guidance offers a useful framework for deciding which event attributes to keep and which to exclude.

5. Sample deliberately

Sampling can reduce repetitive, high-volume output by retaining a subset—for example, one in every five events from a defined category. Implement it in the application or collection path according to which costs you need to reduce. Dropping an event after it has already been ingested may not reduce upstream transport or ingestion charges.

Sampling loses information by design. Preserve required audit records, rare failures, and representative error details; do not assume a sampling algorithm automatically protects errors. During an error storm, rate limiting may protect service availability, but count and report discarded events so that quiet logs do not imply recovery.

Record the policy or sample rate where downstream analysis needs it. A sampled count is not the total count, and a deterministic “every fifth event” rule can bias periodic workloads. Use unsampled counters for rates and budgets that require complete totals.

6. Write a request completion event

A canonical request log line summarizes the facts known at completion: method, route, outcome, duration, service, identity where justified, and request correlation. Add database time or query counts when they answer a real operational question.

{
  "method": "POST",
  "path": "/user/login",
  "request_id": "req_98765",
  "http_status_code": 500,
  "user_id": "u_789",
  "service": "auth",
  "duration_ms": 320,
  "db_time_ms": 120
}

Prefer a normalized route when raw paths contain identifiers, and do not include credentials or full request bodies. Emit the summary on handled success and failure paths. A crash or abrupt termination can still prevent it from being written, so it does not replace exception events or distributed traces. It gives an investigation a starting point without requiring the reader to reconstruct every request from scattered messages.

7. Centralize collection and retain source context

When an operation crosses hosts and services, central search makes related events easier to find. Collect files or container output through an appropriate shipper, then preserve service, host, environment, and source identifiers. Guance's DataKit log collector provides configuration for file collection and processing.

Verify collection using a known event from each target source. Check timestamp handling, permissions, file rotation, duplicates, and outages. Centralization does not itself guarantee complete delivery; a collector and its destination need capacity, retry, and loss-monitoring policies too.

8. Set retention by purpose

Recent troubleshooting data, operational history, and audit evidence may need different lifetimes. Define retention per category and access pattern. Keep frequently queried data available for investigation, and use an appropriate archive where longer storage is required.

Volume and duration affect cost, but the bill may also depend on indexing, query scans, archives, retrieval, and contractual minimums. Use the platform's actual units rather than treating “ingestion × retention” as a universal pricing formula. Guance log indexes support separating matching data and storage policies; verify the applicable configuration before relying on it.

Local file rotation is a separate responsibility. Bound disk use on application hosts and coordinate rotated-file handling with the shipper. Platform retention cannot prevent an application's disk from filling.

9. Protect access, transport, and storage

Logs can expose database details, personal information, security decisions, and operational secrets. Use encrypted transport and appropriate storage encryption, restrict access by role and workload, and audit access and administrative changes.

Include exports, archives, backups, and alert notifications in that policy. A tightly controlled search interface does not protect a broadly shared export. Test that permissions isolate the data they are intended to isolate.

10. Keep sensitive values out of logs

Prevent passwords, tokens, session cookies, and unnecessary personal data from entering the pipeline. Use an allowlist of safe fields when serializing objects. Go's slog.LogValuer, for example, can control the logged representation of a type.

Add redaction in the processing pipeline as another layer, then inspect actual post-processing samples regularly. That layer cannot undo an earlier write to an application file or console. Test nested fields, free-text error messages, URLs, and unusual failure paths, where a structured-field redaction rule may not apply. Tokenization or masking may reduce exposure while still leaving data linkable to a person.

11. Measure the performance and failure cost

Logging consumes CPU for formatting and serialization, memory for buffering, and I/O capacity for output. Measure representative messages and peak traffic; an empty-message microbenchmark does not describe an application's real logging cost.

Use lazy evaluation for expensive debug fields where supported. Async output can reduce work on the request path, but the queue still needs a bounded-capacity policy. Decide whether a full queue blocks, drops, or spills to disk, and what happens on shutdown or a destination failure. Sampling, smaller payloads, and carefully chosen fields can be more effective than changing libraries alone.

12. Use logs alongside metrics and traces

Metrics are usually the better foundation for continuous request rates, error ratios, latency distributions, and alert thresholds. Logs explain particular events; traces connect timed work across service boundaries.

Logs can also power trend analysis, log-based metrics, and anomaly detection. The important limitation is coverage: an aggregate built from filtered or sampled events may not represent all requests. For service-level objectives and high-confidence rates, prefer purpose-built counters and histograms with known collection semantics, then use logs to investigate the contributing failures.

Frequently asked questions

Which practices should we implement first?

Start with useful severity conventions, structured fields, and collection you have verified end to end. Define the purpose and data-handling rules at the same time. Then tune retention and sampling using actual query needs and traffic volumes.

Can sampling discard an important error?

Yes. Retaining all ERROR and FATAL events reduces one risk but does not preserve the preceding context, and overload can still cause loss elsewhere. Test the policy against incidents, preserve required evidence, and expose drop counts.

Are rotation and retention the same thing?

Rotation manages local files and their size or age. Retention controls how long collected data remains in a platform or archive. Configure and verify both.

Further reading

Consult the OpenTelemetry logs data model, PostgreSQL logging documentation, Nginx log module, and Go slog reference for implementation details.

The related guides to log formatting and reducing logging costs are currently in Chinese.