Write structured JSON logs in Python

Emit one JSON object per log record so a collector can read named fields instead of guessing their boundaries from text. You can use python-json-logger, the framework-oriented json-logging package, or a small formatter with an explicitly defined schema.

Use python-json-logger

Install it into the interpreter that runs your application:

python -m pip install python-json-logger

For current versions with the pythonjsonlogger.json module:

import logging
from pythonjsonlogger.json import JsonFormatter

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter(
    "%(asctime)s %(name)s %(levelname)s %(message)s"
))
logger = logging.getLogger("app_json")
logger.setLevel(logging.INFO)
logger.propagate = False
logger.addHandler(handler)
logger.info("Login completed", extra={"user_id": 42, "ip": "192.0.2.10"})

Configure the handler once at startup. The older from pythonjsonlogger import jsonlogger form belongs to older examples; match the import to the package version you install. Extra fields can become JSON fields, but names must not overwrite reserved LogRecord attributes. Include only fields you intend to retain.

Use json-logging

python -m pip install json-logging

import logging
import sys
import json_logging

json_logging.init_non_web(enable_json=True)
logger = logging.getLogger("json_example")
logger.setLevel(logging.INFO)
logger.propagate = False
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.info("Application started")

Initialization alone is not a reason to assume INFO records will appear: configure the logger's level and output handler as well. The package also provides framework-specific initialization such as init_flask() and init_sanic(); use the setup documented for the framework and package version in your application.

Define a small schema yourself

import json
import logging
from datetime import datetime, timezone

class JsonFormatter(logging.Formatter):
    def format(self, record):
        entry = {
            "time": datetime.fromtimestamp(
                record.created, timezone.utc
            ).isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        if record.exc_info:
            entry["exception"] = self.formatException(record.exc_info)
        return json.dumps(entry, ensure_ascii=False)

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger = logging.getLogger("custom_json")
logger.setLevel(logging.INFO)
logger.propagate = False
logger.addHandler(handler)
logger.info("Application started")

This records the event's creation timestamp rather than the later moment when a handler formats it. It deliberately includes only the four named fields and an optional exception; arbitrary extra attributes are not automatically copied. Add an explicit field allowlist if you need more context. JSON serialization escapes embedded newlines, so a traceback remains within one JSON record.

Validate collection and migration

For Guance, the DataKit log collector documentation describes json_as_fields for extracting top-level JSON fields. Test a representative record and verify timestamp interpretation, numeric types, level mapping, and any nested-field processing you require.

When migrating text logs, keep the old parsing rules for services still emitting the old format. Compare equivalent sample events before changing queries or dashboards.

FAQ

How do I include the call site? Add selected attributes such as record.filename, record.lineno, and record.funcName to your custom schema.

Is serialization overhead negligible? It depends on event rate, record size, exception formatting, and the output sink. Measure your workload. Libraries that support orjson can offer another serializer, but compatibility and measured benefit should guide the choice.

References

• nhairs.github.io documentation
• github.com documentation
• docs.python.org documentation
• docs.guance.com documentation