Python logging: configuration, JSON output, and exception handling

Python's standard logging module is the common foundation for application and library logs. A few explicit choices—logger names, handler ownership, formatting, and exception handling—make those logs much easier to operate once several modules and worker processes are involved.

This guide starts with the default behavior, builds a centralized configuration, and adds JSON output and tracebacks. It also covers file rotation and the collection checks needed before building alerts in Guance.

Start with a named logger

In a fresh Python process, this code prints only the warning:

import logging

logging.debug("Debug details")
logging.info("Application started")
logging.warning("Retry limit approaching")

WARNING:root:Retry limit approaching

The root logger defaults to WARNING. The module-level convenience functions use that root logger and call basicConfig() when the root has no handlers. They are useful for small scripts, but named loggers are easier to manage in an application with multiple modules:

logger = logging.getLogger(__name__)

A name such as myapp.services.orders follows the module hierarchy. You can then change the level for one part of the application without increasing verbosity everywhere. Creating a named logger does not itself configure a destination or enable INFO messages. Python logging tutorial

Give loggers, handlers, and formatters separate jobs

A logger is the interface application code calls. A handler sends records to an output destination. A formatter determines how each record is represented. Filters can make additional decisions or enrich records.

This standalone setup sends debug-and-higher records to stdout and error-and-higher records to a file:

import sys
import logging

logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)
logger.propagate = False

stdout_handler = logging.StreamHandler(stream=sys.stdout)
stdout_handler.setLevel(logging.DEBUG)

error_handler = logging.FileHandler("error.log", encoding="utf-8")
error_handler.setLevel(logging.ERROR)

formatter = logging.Formatter(
    "%(asctime)s | %(levelname)s | %(name)s | "
    "%(filename)s:%(lineno)s | %(message)s"
)
stdout_handler.setFormatter(formatter)
error_handler.setFormatter(formatter)
logger.addHandler(stdout_handler)
logger.addHandler(error_handler)

Run setup once per process. Repeatedly adding handlers can duplicate output.

The originating logger's effective level decides whether a record is created; handler thresholds decide which destinations receive it. Propagation sends an accepted record to ancestor handlers without reapplying each ancestor logger's level. That distinction explains why changing the root logger's level does not always filter records from a child with its own explicit level.

Here, propagate=False keeps the myapp handlers from also sending the same record to root handlers. In the centralized configuration below, the root owns the output handler instead, so ordinary child loggers should propagate to it.

Include the calling file and line number

The formatter above uses %(filename)s:%(lineno)s to identify the logging call. Use %(pathname)s when the full source path is needed. If a helper wraps the call, Python 3.8 and later support stacklevel to report the helper's caller:

def log_order_created(order_id):
    logger.info("Order created: %s", order_id, stacklevel=2)

log_order_created("ORD-123")

Here, the line number refers to log_order_created("ORD-123"), rather than the logger.info() line inside the helper. Deeper wrappers need the corresponding stack depth. These LogRecord fields identify where logging occurred; an exception traceback separately records where an error propagated.

Centralize configuration with dictConfig

For a larger application, keep logging setup in one place and load it at startup:

from logging.config import dictConfig

dictConfig({
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "default": {
            "format": "%(asctime)s | %(levelname)s | %(name)s | %(message)s",
        },
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "stream": "ext://sys.stdout",
            "formatter": "default",
        },
    },
    "root": {"level": "INFO", "handlers": ["console"]},
})

Use this as an alternative to the manual handler setup, rather than stacking both configurations. Application modules can simply call logging.getLogger(__name__) and let records propagate to the root handler.

disable_existing_loggers=False avoids disabling preexisting non-root loggers that are not explicitly named in the dictionary. It is especially useful when a framework or library creates loggers before application configuration runs. dictConfig reference

Produce JSON without losing context fields

Structured output lets a collector index order_id, severity, and exception details as separate fields. Python does not ship a JSON-specific logging formatter, but a custom formatter can use the standard json module. For a maintained implementation with field handling, install python-json-logger:

pip install python-json-logger

Current versions expose the formatter from pythonjsonlogger.json:

from pythonjsonlogger.json import JsonFormatter

json_formatter = JsonFormatter(
    "%(name)s %(asctime)s %(levelname)s %(filename)s %(lineno)d %(message)s",
    rename_fields={"levelname": "level", "asctime": "time"},
)
stdout_handler.setFormatter(json_formatter)

This snippet uses the stdout_handler from the manual setup. With dictConfig, configure the formatter through its "()" factory instead. Pin the package version and use the import path documented for that version. Python JSON Logger quick start

A typical record looks like:

{"name":"myapp","time":"2026-08-25 14:39:03,265","level":"INFO","filename":"main.py","lineno":31,"message":"Server listening on port 8080"}

That default asctime representation does not contain a time-zone offset. Agree on a timezone or configure an explicit timestamp format before combining logs from multiple hosts. Structured syntax alone does not make an ambiguous timestamp unambiguous.

Attach business context with extra:

logger.info(
    "Order created",
    extra={"order_id": "ORD-123", "user_id": "USR-9"},
)

The JSON formatter can include those attributes as fields. Avoid names that collide with built-in LogRecord attributes, such as message, name, or levelname; a collision can raise KeyError. Apply your privacy policy to user identifiers and other context rather than logging every available field.

JSON is a useful default for centralized ingestion. Human-readable text remains reasonable when it suits the reader and collection pipeline; production logging does not require JSON by definition.

Preserve exception tracebacks

An error message without its traceback often omits the detail needed to fix the failure. Inside an exception handler, use logger.exception():

try:
    1 / 0
except ZeroDivisionError:
    logger.exception("Calculation failed")

This is equivalent to an error-level call with exc_info=True. With a JSON formatter, confirm that the traceback appears in an exception field and remains part of one event.

For uncaught exceptions in a simple command-line application's main thread, you can install sys.excepthook:

import sys

def handle_exception(exc_type, exc_value, exc_tb):
    if issubclass(exc_type, KeyboardInterrupt):
        sys.__excepthook__(exc_type, exc_value, exc_tb)
        return
    logger.critical(
        "Uncaught exception",
        exc_info=(exc_type, exc_value, exc_tb),
    )

sys.excepthook = handle_exception

The hook logs the exception; the interpreter's uncaught-exception path is what ends the program. It is not a universal handler for every execution context. Threads have threading.excepthook, asyncio has event-loop exception handling, and web frameworks often catch exceptions before they reach this hook. Use the appropriate boundary for the runtime, and avoid logging one failure repeatedly at every layer. sys.excepthook

Rotate files without competing writers

A plain FileHandler does not rotate files. For a single writer, use RotatingFileHandler for size-based rotation or TimedRotatingFileHandler for time-based rotation:

from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler(
    "app.log",
    maxBytes=10_000_000,
    backupCount=5,
    encoding="utf-8",
)
handler.setFormatter(json_formatter)

Attach it to the logger that owns file output if you choose this destination. The example limits each active file to roughly 10 MB before rollover and retains five backups; it does not make concurrent multiprocess writes safe.

Several worker processes rotating the same file can race. Route records through one listener process, use separate files, or write to stdout and let the runtime and collector manage them. For external rotation on Unix, consider WatchedFileHandler so the process notices a renamed-and-recreated file. Simply adding logrotate while keeping an ordinary FileHandler can leave the application writing to an old file descriptor. copytruncate has its own loss window and should be assessed accordingly. Python logging cookbook and handler reference

Check ingestion before configuring alerts

Emit one ordinary JSON event and one event with a traceback. Configure the DataKit log collector for the application's file or container output, then inspect both events in Guance.

Check the timestamp, severity field, context attributes, and complete exception. Different formatters use different names—for example, level, levelname, or severity—so map the fields in Pipeline according to the actual output. Verify an error query before creating a log-detection rule and notification policy.

If the application is traced, include the active request's trace_id and verify the link to that trace. A JSON formatter does not generate a distributed trace or infer its identifier.

Frequently asked questions

Why are my INFO messages missing?

Check the logger's effective level and the receiving handler's threshold. Both must allow the record. Configure INFO or DEBUG explicitly where needed, and ensure a handler receives the record.

What is the difference between getLogger(name) and getLogger("myapp")?

__name__ gives each module its place in the package hierarchy, such as myapp.services.orders. A fixed name intentionally puts calls under the same logger. Module names usually make per-component tuning easier; a fixed application name is useful when a shared logger is intentional.

Why does each message appear twice?

A common cause is a handler on the application logger plus another on an ancestor, with propagation enabled. Either let the ancestor own the handler or set propagate=False on the logger that owns its output. Also check whether startup code has added the same kind of handler repeatedly.

Should I use logging, Loguru, or structlog?

Use the standard API for libraries and for compatibility with existing Python logging configuration. Loguru offers a more compact application-facing API. structlog focuses on structured events and context binding and can integrate with standard logging. Choose based on the application's event model and operational requirements rather than assuming one is a drop-in replacement for every workflow.

Related reading

• Loguru in practice (Chinese)
• Python logging best practices (Chinese)
• Comparing Python logging libraries (Chinese)