JSON logging: structure events for reliable queries

JSON logging records each event as a structured JSON object. Instead of extracting a status code or user ID from a changing sentence, a log platform can read a named field and use it in searches and aggregations. A consistent schema makes that useful; valid JSON alone does not guarantee consistent meaning.

New applications can emit JSON directly, infrastructure components may offer JSON output, and a parsing pipeline can extract fields from existing text logs. Before sending any of these to a shared platform, agree on event time, severity, field types, correlation identifiers, and retention.

Compare text with structured fields

An Apache-style access record might look like this:

127.0.0.1 - alice [25/Aug/2026:11:26:42 +0800] "GET /api/users HTTP/1.1" 200 3477

The corresponding information can be represented as JSON:

{
  "ip": "127.0.0.1",
  "user_id": "alice",
  "time": "2026-08-25T11:26:42+08:00",
  "method": "GET",
  "path": "/api/users",
  "protocol": "HTTP/1.1",
  "status_code": 200,
  "response_bytes": 3477
}

This uses status_code for the HTTP status so it cannot be confused with a platform's log-severity field. The example is formatted across lines for reading. For file or stdout collection, emit each complete event on one physical line unless the collector is explicitly configured for a different framing format. Escape embedded newlines through the JSON serializer; do not concatenate raw strings into a JSON template.

JSON adds repeated keys and punctuation, but it makes field boundaries explicit. You still need to handle malformed records, incompatible schemas, and unexpected values.

What JSON helps with

Developers can recognize key-value records, while tools can parse them without matching the surrounding prose. Adding an optional field is usually easier to accommodate than changing a positional text format. However, renaming a field, changing its type, or removing a required field can still break downstream consumers. Treat the log schema as an interface.

Structured context also makes investigation more useful. Service, version, request identifiers, route, and duration can narrow a search or connect related records. Only collect user identifiers when justified, and apply access and retention controls. JSON does not itself provide correlation, privacy protection, or observability coverage.

Account for the costs

Raw JSON is less comfortable to scan in a terminal. Use a development renderer or a tool such as jq when inspecting it; keep production output machine-readable and consistently framed. If development uses another format, test the actual production JSON path too.

Repeated keys increase bytes before compression. Serialization and parsing also consume CPU. Their importance depends on event size, rate, storage pricing, and the logger's behavior. Measure representative workloads instead of assuming the overhead is always negligible or that compression makes volume irrelevant.

Three ways to adopt JSON logging

Emit JSON from the application

A structured logging library avoids manual quoting and lets callers attach typed fields. With Python and structlog, install structlog and configure processors before creating application loggers:

import structlog

structlog.configure(processors=[
    structlog.processors.TimeStamper(fmt="iso", utc=True, key="time"),
    structlog.processors.add_log_level,
    structlog.processors.JSONRenderer(),
])

logger = structlog.get_logger()
logger.info("Image upload completed", filename="cover.jpg", size_bytes=2382)

A representative event, with a timestamp that will differ on each run, is:

{"filename":"cover.jpg","size_bytes":2382,"event":"Image upload completed","time":"2026-08-25T04:07:41Z","level":"info"}

key="time" is deliberate: structlog's TimeStamper otherwise uses timestamp. add_log_level produces level; map that to the receiving platform's severity field if needed. See the structlog processor reference.

Enable a component's JSON output

PostgreSQL 15 and later can write JSON logs through the logging collector:

# postgresql.conf
logging_collector = on
log_destination = 'jsonlog'

Enabling logging_collector requires a server restart. If it is already enabled, changing log_destination can be reloaded. Follow your database change process, confirm the effective settings, and inspect current_logfiles to locate the active JSON output. Adding jsonlog does not automatically enable every statement or event category; the other PostgreSQL logging settings still decide what is recorded. See PostgreSQL logging configuration.

Other components have different capabilities. Nginx can use an appropriately escaped JSON access-log format; MySQL's JSON error logging depends on its logging components and configuration. Check the specific product, version, and log type rather than assuming a universal JSON switch.

Parse existing text in the collection path

When application changes are impractical, collect the existing records and extract structured fields through Pipeline. This preserves a gradual migration path for older services. Test rules against representative records and retain a way to inspect parse failures; a text parser can still break when the source format changes. Start with DataKit log collection and the Pipeline quick start.

Make the fields useful in Guance

1. Agree on names and types

Choose one name such as user_id instead of alternating among uid, userID, and user_id. Keep types stable too: a numeric duration should not become a string in another service. Normalize third-party field names in Pipeline where necessary, keeping the original data available when it helps investigation.

2. Parse event time and severity explicitly

For collected file logs, configure JSON parsing and map event time to time and severity to status. Guance's log collection documentation describes its handling of these fields, including current-time fallback when an event timestamp is unavailable and unknown severity when none is extracted.

Verify timezone interpretation, timestamp units, and severity mappings with sample events. A historical record stamped with ingestion time can fall into the wrong search window. An HTTP response code such as 401 belongs in a separate field, not in log severity.

3. Include units in numeric field names

Prefer duration_ms to an ambiguous duration. Apply the same convention to byte counts and other measurements. If you normalize a value to another unit, update both the number and its field name.

4. Carry the context needed for investigation

Useful context may include a stable route, service version, elapsed time, and the current trace identifier. For example:

{
  "time": "2026-08-25T15:45:12+08:00",
  "status": "warn",
  "message": "User login failed",
  "user_id": "u_123456",
  "trace_id": "5b8aa5a2d2c872e88be903eca78912ab",
  "request": {"path": "/login", "status_code": 401, "duration_ms": 200},
  "service": "auth-service"
}

The trace ID is an illustrative nonzero 32-character hexadecimal value. In an application, obtain the real ID from the active tracing context and preserve the representation expected by the chosen integration; do not generate an unrelated ID for each log event or copy this value. Keep secrets, tokens, and unnecessary personal data out of the record.

Guance's trace details documentation describes associated-log lookup and index mapping. When logs span several indexes, configure the mapping using appropriate service, env, and version values, then confirm an actual trace can find its logs.

5. Set retention by purpose

Use matching rules to route native logs into suitable indexes, then configure the available storage policies for each. Debugging records and audit records can have different retention needs. Confirm the plan and workspace options, retrieval requirements, and audit obligations before filtering or shortening retention. JSON formatting by itself does not reduce those costs. See Guance log indexes.

Common questions

Will JSON logging affect application performance?

Yes, it adds serialization work and output bytes. Whether that matters requires measurement at the expected event rate and payload size. Pino and Zerolog are options for Node.js and Go, respectively, but library selection does not replace a workload-specific check of buffering, backpressure, and output cost.

Does JSON still need Pipeline parsing in Guance?

It depends on the collector and incoming field format. A file record may need JSON extraction, event-time conversion, severity mapping, and normalization. Inspect the output of any existing parser before adding another one; repeated parsing can be unnecessary or incorrect.

What should we do with existing text logs?

Keep collecting them and introduce tested parsing rules gradually. New applications can emit structured events directly while older services retain their current output. Both can support consistent queries once their important fields are normalized.

Related reading

• Logfmt explained (Chinese)
• Structured logging explained (Chinese)

The structlog example was run locally and its output parsed as JSON. PostgreSQL restart/reload behavior and Guance ingestion, trace correlation, and retention settings were reviewed from documentation, not exercised on a live deployment.