Parse Apache access logs for analysis

Apache access logs follow the configured LogFormat; Combined Log Format is common, but is not a universal default. Use command-line tools for a quick investigation, a script for repeatable processing, and a log platform when you need ongoing search and aggregation.

Identify the format

A Combined-format entry looks like this:

192.168.1.1 - - [15/Jan/2024:10:30:45 +0800] "GET /api/users HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0 ..."

Its fields are the remote host or IP, ident value, authenticated user, timestamp, request line, status, response body size, Referer, and User-Agent. A dash represents an unavailable field; %b also uses - when the response size is zero. The timestamp includes the UTC offset.

Inspect logs from the command line

These shortcuts assume the sample field layout and an ordinary three-part request line:

# Extract the remote address and request target.
awk '{print $1, $7}' access.log

# Find the ten most frequent remote addresses.
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Count responses by status code.
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Find 4xx and 5xx responses.
awk '$9 >= 400 && $9 < 600 {print $1, $7, $9}' access.log

# Count requests by date and hour.
awk -F'[][]' '{print $2}' access.log | cut -d: -f1-2 | sort | uniq -c

Whitespace-based fields are unsuitable for arbitrary custom formats or malformed request lines. They also do not treat quoted User-Agent strings as a single field.

Parse records with Python

The following extracts the core fields from ordinary Combined-format requests. It deliberately leaves Referer and User-Agent unparsed:

import re

pattern = re.compile(
    r'(\S+) \S+ \S+ \[([^\]]+)\] "(\S+) (\S+) \S+" (\d{3}) (\d+|-)'
)

with open('access.log', encoding='utf-8', errors='replace') as f:
    for line in f:
        match = pattern.match(line)
        if match:
            ip, time_str, method, url, status, size = match.groups()
            size_bytes = 0 if size == '-' else int(size)
            print(f"{ip} {method} {url} -> {status} ({size_bytes} bytes)")

Test parsing against ordinary requests, missing Referers, escaped quotes, User-Agents containing spaces, and malformed request lines. Count unmatched lines rather than silently treating them as successfully processed. For complete field extraction, use a parser that understands Apache's escaping rules.

When sending logs to Guance, test representative records in Pipeline and check that status and response size have the intended numeric types before applying the rule to collected data.

Frequently asked questions

What if processing a large file is slow?

Measure the actual workload. Filtering early can reduce later processing, although an extra grep also adds another pass through the pipeline. For parallel processing, split only at record boundaries and combine counts correctly.

What if the log does not use Combined format?

Inspect LogFormat and the CustomLog directive that selects it. Adjust both field extraction and tests to the actual format.

How do I select a time interval?

For this specific hour, match the timestamp field:

awk '$4 ~ /^\[15\/Jan\/2024:10:/' access.log

An awk range ending at the first 11: record includes that endpoint and is not an exact half-open time filter. For arbitrary intervals or mixed offsets, parse timestamps and compare normalized date-time values.

See Apache's log format reference.