10 Python logging practices for production services

Python's logging module is flexible enough to support a large application. That flexibility also makes it easy for different modules to emit incompatible formats, duplicate exceptions, or quietly expose sensitive data.

A useful logging standard settles a few practical questions: who configures output, which events deserve attention, which fields identify the request, and what happens when a destination is slow or a file fills up. These ten practices provide a starting point for a production service and its Guance collection pipeline.

1. Give each module a named logger

Module-level calls such as logging.info(...) write through the root logger. They are convenient for small scripts, but named loggers make it easier to identify the source of a record and change verbosity for one part of an application.

import logging

logger = logging.getLogger(__name__)

def process_order(order_id):
    logger.info("Processing order %s", order_id)

__name__ supplies a hierarchical name such as shop.services.order. You can enable DEBUG for shop.services while keeping the rest of the application at INFO.

A named logger can still propagate to a handler on the root logger. Naming and output configuration solve different problems; you do not need a separate handler in every module.

2. Configure output at application startup

Configure handlers, formatters, and levels once at the application entry point. dictConfig keeps that setup explicit:

# logging_config.py
from logging.config import dictConfig

def setup_logging():
    dictConfig({
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "default": {
                "format": "[%(asctime)s] %(levelname)s %(name)s: %(message)s"
            }
        },
        "handlers": {
            "console": {
                "class": "logging.StreamHandler",
                "stream": "ext://sys.stdout",
                "formatter": "default"
            }
        },
        "root": {"level": "INFO", "handlers": ["console"]},
    })

Reusable library code should obtain a named logger and leave application-wide configuration to its caller. It should not call basicConfig or install output handlers as an import side effect. A library may attach a NullHandler to avoid unsolicited fallback output while remaining configurable by the application.

The Python logging cookbook describes how logger hierarchies, handlers, and library configuration fit together.

3. Agree on what each severity means

Severity is useful only if the team applies it consistently:

Avoid treating every successful request as equally important, or every retry as an error. Decide which events need to be retained and which should trigger action.

When normalizing records in Guance, map log severity into the standard status field. Keep an HTTP status code in its own field; 500 and error describe different aspects of a request.

4. Defer message formatting, not expensive computation

Pass arguments separately when using standard-library logging:

# Both forms format the message before the level check.
logger.debug("Processing user %s order %s" % (user, order))
logger.debug(f"Processing user {user} order {order}")

# Logging can defer interpolation until the record is formatted.
logger.debug("Processing user %s order %s", user, order)

This avoids unnecessary string interpolation when DEBUG is disabled. It does not defer Python's evaluation of arguments. An expensive function call still runs in logger.debug("State: %s", build_state()). Guard that work explicitly:

if logger.isEnabledFor(logging.DEBUG):
    logger.debug("State: %s", build_state())

Write messages that identify the operation and outcome. A stable message such as Payment callback processed, with separate order_id and amount fields, is easier to investigate than Done. Structured fields also avoid extracting identifiers from free-form text later.

5. Capture exceptions at the layer that handles them

An error message without a traceback often leaves out the information needed to locate the failure:

try:
    result = charge(user, amount)
except PaymentError:
    logger.exception("Payment failed user=%s", user.id)
    raise

logger.exception() records an ERROR with the current exception information. It belongs inside an exception handler. logger.error(..., exc_info=True) provides the same exception behavior.

Decide which layer logs a failure. Logging and re-raising at every layer produces several records for one exception and can inflate alert counts. The fragment above is appropriate only when this layer owns that diagnostic record.

For otherwise uncaught exceptions, sys.excepthook can help in a command-line process. It is not a universal solution for threads, asynchronous tasks, or web frameworks; use the relevant runtime or framework exception hooks too.

6. Emit fields that can be queried directly

JSON gives collectors a reliable structure to parse. With python-json-logger installed, a formatter entry can look like this:

"formatters": {
    "json": {
        "()": "pythonjsonlogger.json.JsonFormatter",
        "format": "%(asctime)s %(levelname)s %(name)s %(message)s",
    }
}

Select json on the handler as well. This is a configuration fragment, not a complete replacement for the earlier dictionary. Consult the Python JSON Logger quick start for your installed version; older configurations may use the legacy jsonlogger module path.

Add request and business context as fields rather than embedding everything in message. A collector still needs a parsing configuration: JSON output does not guarantee DataKit will extract every nested field or infer your intended timestamp and severity mappings.

7. Use timestamps with an explicit timezone

The default asctime value resembles 2026-08-25 14:30:22,123. It contains no timezone, which makes correlation across regions ambiguous.

Use an ISO 8601 timestamp with an explicit offset, such as 2026-08-25T14:30:22.123+08:00, or UTC. Setting a formatter's time converter to UTC changes the clock basis; you must still make the output format identify that timezone. Verify the event time after ingestion with a known sample.

Use wall-clock time for event timestamps and a monotonic clock for elapsed durations. They answer different questions.

8. Keep sensitive values out of the record

Logging a secret once can copy it into local files, buffers, search indexes, exports, and backups. Prevent that at the point of emission wherever possible:

• Do not log passwords, API keys, authentication tokens, full payment card numbers, or government identifiers.
• Record only the portion of personal data required for the task, such as a masked phone number.
• Apply application-side filtering to the actual structured fields and exception output, not just the final message string.
• Add collection-side redaction or sensitive-data controls as a second layer, and test where those controls run relative to buffering and storage.

Regex replacement alone will not reliably catch every secret, especially in nested JSON or exception-local values. Test representative samples and inspect the raw collected record as well as the search view.

For Guance, choose the appropriate Pipeline or sensitive-data processing capability for your deployment. Confirm its supported fields and processing stage before treating it as a pre-storage guarantee.

9. Bound local file growth

For a host-based, single-writer setup, rotate files by size or time:

"handlers": {
    "file": {
        "class": "logging.handlers.RotatingFileHandler",
        "filename": "logs/app.log",
        "maxBytes": 20 * 1024 * 1024,
        "backupCount": 10,
        "encoding": "utf-8",
    }
}

This retains the active file plus up to ten backups, each with a nominal 20 MiB threshold. Create the directory, select this handler on the appropriate logger, and set a formatter as required. TimedRotatingFileHandler is the time-based alternative.

Do not point several worker processes at the same standard rotating handler destination and assume rotation is coordinated. Use a single writer or external collection. In containers, write to stdout/stderr and configure runtime rotation and collection explicitly; defaults may not provide the retention or disk bounds you need.

10. Centralize investigation and retention

Once a service has several instances, local files alone are an awkward investigation interface. Central collection lets you filter by service and host, compare a failure across instances, and retain records after an instance disappears.

In Guance, configure DataKit log collection and validate the parsed output before building alerts. Use the Log Explorer to search and group events, then create monitors around actionable failure patterns. Route notifications to the team that owns the service.

Use index and retention settings to distinguish records that need longer availability from verbose diagnostics with a shorter useful life. Evaluate archival separately from interactive search requirements. For log-to-trace navigation, instrument tracing and include the correct trace_id in the log; central storage does not create correlation identifiers for you.

Frequently asked questions

When should a project adopt a logging standard?

At the start, when naming loggers and configuring one output path costs little. You can refine the schema as the application grows, but retrofitting inconsistent field names across many services is expensive.

How should development and production differ?

Keep logging calls the same and vary configuration. Development may use readable console output at DEBUG; production may use JSON at INFO, sent to stdout or a managed file. Make verbosity changes reversible and avoid enabling verbose logging indefinitely during an incident.

How do I quiet a third-party library?

Set its named logger's level. For example, a dictConfig loggers entry of "urllib3": {"level": "WARNING"} reduces routine messages while preserving warnings and errors. Check propagation and handler levels if the result is unexpected.

Does file rotation conflict with the twelve-factor logging model?

The deployment models differ. A twelve-factor application emits an event stream to stdout and lets its environment route and retain it. File rotation applies to applications that deliberately manage local file output. Either can feed Guance, but responsibility for disk usage and retention must be explicit.

Related reading

• Flask logging
• Python logging libraries compared
• Logging best practices (Chinese)
• Handling sensitive data in logs (Chinese)