Parse Nginx access logs for request analysis

Start by identifying the active log_format. Nginx's predefined combined format contains the client address, timestamp, request line, status, response-body bytes, referrer, and user agent. Custom configurations can reorder fields or add upstream timing, so a parser must match the actual records.

For example:

192.168.1.1 - - [15/Jan/2024:10:30:45 +0800] "GET /api/users HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0 ..."

Use shell tools for a quick inspection

For ordinary combined records with conventional request lines:

# Most frequent client addresses.
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Counts by status.
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Most frequent request targets, including their query strings.
awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -10

# Only HTTP 5xx statuses.
awk '$9 >= 500 && $9 < 600 {print $1, $7, $9}' access.log

# Requests logged in each second; sort so equal timestamps are adjacent.
awk '{print substr($4, 2)}' access.log | sort | uniq -c

The last command counts completion-log timestamps, not necessarily request arrival times. It retains date, hour, minute, and second, assuming one timezone. Fixed-position parsing is a quick diagnostic, not a robust parser for malformed requests or an arbitrary custom format.

Parse records with Python

This example extracts the full quoted request first and reports records it cannot parse:

import re

pattern = re.compile(
    r'^(\S+) \S+ \S+ \[([^\]]+)\] "([^"\n]*)" '
    r'(\d{3}) (\d+|-) "([^"\n]*)" "([^"\n]*)"\s*$'
)
unmatched = 0
with open('/var/log/nginx/access.log', encoding='utf-8', errors='replace') as logs:
    for line in logs:
        match = pattern.match(line)
        if not match:
            unmatched += 1
            continue
        ip, timestamp, request, status, size, referer, user_agent = match.groups()
        print(f'{ip} {request} -> {status}')
print(f'Unmatched records: {unmatched}')

This targets the shown format and tolerates - for an unavailable size. Nginx's default log escaping may leave escaped byte sequences in field values; the regex extracts them without claiming to decode them. Preserve failed records for parser development rather than silently counting them as successful parses.

Prefer structured output for new pipelines

If you control the format, log_format ... escape=json plus a JSON parser avoids whitespace-position assumptions. Represent potentially missing values as strings or explicitly normalize them so a - value cannot break JSON.

Add $request_time or $upstream_response_time to the format if needed; a collector cannot recover fields Nginx never emitted. When using the Guance Nginx integration, test its parser with your actual custom records and verify field types and timestamps.

FAQ

How can I select a time window?

Parse the timestamp rather than relying on an awk range beginning and ending on exact matching lines. Those boundary lines may not exist. For a fixed sample day, after extracting timestamp in the Python loop:

from datetime import datetime

start = datetime.fromisoformat('2024-01-15T10:00:00+08:00')
end = datetime.fromisoformat('2024-01-15T11:00:00+08:00')
when = datetime.strptime(timestamp, '%d/%b/%Y:%H:%M:%S %z')
if start <= when < end:
    print(request, status)

The month parser expects the locale's month abbreviations to match the log; configure an English-compatible locale for these sample records.

What about multi-gigabyte logs?

Stream records rather than loading the file into memory. Narrow the files or time partition first, and use an indexed log system for repeated large queries. Sorting can itself require substantial temporary disk space.

How do request_time and upstream_response_time differ?

The former covers Nginx's overall request processing; the latter records upstream response timing and may contain multiple values for retries or redirects. Their difference alone does not prove a client or network bottleneck. Consider buffering, request upload, retries, and response delivery together.

References: access-log formats, upstream timing variables.