Change log levels at runtime without restarting the application

Use a protected endpoint, Unix signals, or configuration reload to enable DEBUG on a specific worker and restore its log level after troubleshooting.

Best practices
Illustration of logs, metrics, and traces

Suppose two payment requests time out on the same worker. The normal INFO logs identify the affected requests, but they omit the upstream response timing you need. Enable DEBUG for the payments logger on that worker, capture another occurrence, and restore the previous level. The useful change is a small window on a known target. A fleet-wide switch generates more data and can make the affected requests harder to find.

You can adjust a log level without restarting the process through an authenticated control endpoint, a signal handler, or a logging framework that reloads configuration. Each path has a different scope. An HTTP request routed to one worker changes that worker's memory. A Unix signal reaches a process. A file edit takes effect only where the running application watches and successfully reloads that file.

Give an internal endpoint a deadline

With Django and Loguru, keep the normal application sinks configured at INFO. Register one additional sink when the worker starts. It only accepts DEBUG events marked with the payments component, and a deadline in the worker controls when those events pass. This keeps existing sinks and formatting in place.

import json
import logging
import sys
import time
from threading import Lock

from django.contrib.auth.decorators import permission_required
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_protect
from django.views.decorators.http import require_POST
from loguru import logger

audit = logging.getLogger("ops.log_level_audit")
lock = Lock()
debug_until = 0.0
payments_log = logger.bind(component="payments")

def debug_filter(record):
    if (record["level"].name != "DEBUG"
            or record["extra"].get("component") != "payments"):
        return False
    with lock:
        return time.monotonic() < debug_until

logger.add(sys.stderr, level="DEBUG", filter=debug_filter)

@require_POST
@csrf_protect
@permission_required("ops.change_loglevel", raise_exception=True)
def change_log_level(request):
    global debug_until
    try:
        payload = json.loads(request.body)
    except (ValueError, UnicodeError):
        return JsonResponse({"error": "Invalid JSON"}, status=400)
    level = payload.get("log_level") if isinstance(payload, dict) else None
    if level not in ("INFO", "DEBUG"):
        return JsonResponse({"error": "Expected INFO or DEBUG"}, status=400)

    with lock:
        if level == "INFO":
            debug_until = 0.0
        elif time.monotonic() >= debug_until:
            debug_until = time.monotonic() + 300
    audit.warning("actor=%s component=payments requested=%s", request.user.pk, level)
    return JsonResponse({"requested": level, "scope": "current worker"})

The payments module writes through payments_log. In your Django project, define ops.change_loglevel and grant it to the operations role. The endpoint expects authenticated POST requests with CSRF protection for the session flow. Send {"log_level":"DEBUG"} to open a five-minute window or {"log_level":"INFO"} to close it early. Repeated requests during the same window leave the deadline in place. Wire the audit logger to a separate durable destination and check that change records arrive there.

In a multi-worker deployment, an ordinary load-balanced request may reach a different worker each time. Use a control plane that targets each worker or distributes an expiring setting. Record which instances applied it and check their output. Django permissions · Django CSRF protection · Loguru handler recipes

Send a signal to a known process

A signal can be useful when a small Unix service already has a managed operator path. Node.js reserves SIGUSR1 to start the debugger. In this example the process handles SIGUSR2 to enable DEBUG on one Pino child logger, and SIGHUP restores its previous level. The timeout provides a second way to restore it.

const pino = require("pino");

const appLog = pino({ level: "info" });
const paymentsLog = appLog.child({ component: "payments" });
const baseline = paymentsLog.level;
let expiry;

function restoreLevel() {
  if (!expiry) return;
  clearTimeout(expiry);
  expiry = undefined;
  paymentsLog.level = baseline;
  appLog.info({ levelRestored: baseline }, "payments debug disabled");
}

process.on("SIGUSR2", () => {
  if (expiry) return;
  paymentsLog.level = "debug";
  expiry = setTimeout(restoreLevel, 300_000);
  expiry.unref();
  appLog.info("payments debug enabled for 5 minutes");
});
process.on("SIGHUP", restoreLevel);

Once the process has installed these handlers, use kill -s USR2 <pid> on the intended target. Use kill -s HUP <pid> to restore early. Check that the application and its supervisor have no other use for either signal. A successful kill confirms signal delivery was requested, so verify the level on the process and retain an operator-side record of who targeted it. Downstream transport filters can still discard a DEBUG event the logger emitted. Node.js signal events · Pino logger level

Reload the file the framework actually watches

With Logback, set scan="true" and an explicit unit such as scanPeriod="10 seconds". Adjust a named logger such as com.example.payments to limit the affected package. Config scanning and Spring Boot's Logback configuration extensions cannot be used together. With Log4j 2, monitorInterval="10" polls every ten seconds, and 0 disables polling. These settings describe when a framework checks for a change. Confirm that each instance can see the changed file and successfully loaded it. Restore the previous configuration after the investigation. Logback configuration · Spring Boot Logback extensions · Log4j 2 configuration

Check that the right logs reached the destination

After enabling DEBUG, find one new record for the selected service and instance. If DataKit is collecting the output, confirm that the Pipeline maps the incoming severity to status and preserves the event timestamp. An application-level change can work while the downstream filter still drops the event or the parser assigns the wrong severity. Also watch the change in bytes and check that debugging fields do not expose credentials or request payloads. After restoration, verify that new DEBUG events stop for the target. DataKit log collection

Keep the change time, target, and expiry with the investigation record. For the shape of the debug events themselves, see log formatting best practices.

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting