Log sampling: What to keep when you cut volume
Compare random log sampling, time-window rate limits, and trace ID hashing. Keep required records and measure the cost and evidence that remain.
An upstream outage can produce a thousand near-identical error logs. Keeping each one with a 10% probability leaves about a hundred in expectation. If only one entry carries the new error code, it may disappear from the sample. That is the trade-off in log sampling. You choose which events can lose detail and protect the ones you may need to investigate.
Log sampling keeps a subset of eligible records. The method determines whether individual events, windows, or entire trace IDs get a chance to pass. Where you apply it determines which downstream work you avoid.
Log sampling at a glance
- Random sampling makes a probability-based decision for each eligible event. A time-window limit favors early arrivals. Hash sampling makes a repeatable decision from an identifier.
- Identify the records that must be retained before you choose a rate or a sampling point.
- Compare the input, drops by reason, and output. Check that known errors and request context still reach the destination.
What sampling can save
- Storage. Fewer written records generally need less space. The bill still depends on the units your backend charges for and how long it keeps each record.
- Query work. A query may finish sooner when it scans fewer records. Its filters, index layout, and aggregation still matter.
- Downstream processing and transfer. A decision in the application can prevent a log write. A collector can reduce work after that point, once the application has already produced the record.
- Investigation time. Repetitive access and health-check logs can be sampled while keeping named errors and business events easy to find.
- Capacity at peak. Sampling or rate limiting can make bursts of eligible records easier for the downstream pipeline to handle. Monitor the source with independent metrics so a logging limit cannot hide the burst.
Three ways to reduce log volume
1. Random sampling
Make a random decision for each eligible event. The two Zerolog examples are logger initializers to place inside a function. Both use github.com/rs/zerolog and Go's os package, and the rate-limit example also uses time. Only INFO goes through the configured sampler. Send only eligible INFO records to these loggers and route mandatory records separately.
randomLog := zerolog.New(os.Stdout).With().Timestamp().Logger().
Sample(zerolog.LevelSampler{InfoSampler: zerolog.RandomSampler(5)})
An INFO event has about a one-in-five chance of passing. A sequence of five events can produce zero, one, or several retained entries. Zerolog's BasicSampler{N: 5} instead selects at a fixed counter interval. That systematic selection can favor certain events if the workload repeats on a similar cycle. Zerolog v1.35.1 sampler source
2. Time-window rate limiting
A rate limit reduces a burst by admitting a fixed number before it begins dropping. This Zerolog sampler instance admits the first three INFO events in a one-second period. With no NextSampler, it drops the subsequent eligible events in that period. Each instance counts its own events, and traffic can pass on both sides of a window boundary. Concurrent events at rollover can also make the count vary. Zerolog v1.35.1 sampler source
limitedLog := zerolog.New(os.Stdout).With().Timestamp().Logger().
Sample(zerolog.LevelSampler{
InfoSampler: &zerolog.BurstSampler{Period: time.Second, Burst: 3},
})
Early arrivals can use up the allowance before an important INFO event arrives. Zap's NewSamplerWithOptions groups by level and message, keeping the first N entries in a tick and then keeping entries at the configured interval. Its documented implementation favors speed, so the counts can vary slightly under load. Zap sampler documentation
3. Trace ID hash sampling
Propagate the same trace_id to the services that participate. Give them the same byte representation, hash, and threshold. The same ID then leads to the same decision wherever the rule runs. This Go helper hashes the decoded 16-byte identifier and keeps about 10% of valid IDs. It keeps an eligible event when the ID is missing, has anything other than 32 hex characters, or is all zeros. Count those fallbacks separately so a context propagation problem is visible. W3C Trace Context
import (
"crypto/sha256"
"encoding/binary"
"encoding/hex"
)
func keepTraceLog(traceID string) bool {
var id [16]byte
if len(traceID) != hex.EncodedLen(len(id)) {
return true
}
if _, err := hex.Decode(id[:], []byte(traceID)); err != nil || id == ([16]byte{}) {
return true
}
digest := sha256.Sum256(id[:])
bucket := binary.BigEndian.Uint64(digest[:8]) % 100
return bucket < 10
}
Apply the helper only to log classes allowed to be sampled. A shared decision keeps related events together where each service emits and ships them successfully. Missing propagation, absent log writes, and downstream drops can still leave gaps. Mandatory records can also appear for IDs the hash did not select. The rate applies to trace IDs, so traces with more log entries can move the retained share of records away from 10%.
Where to apply the rule
| Layer | Mechanism | Effect |
|---|---|---|
| Application | Logger sampler or per-request decision | Can avoid writing the record and can use request context directly. |
| Collector | Sampler or filter in a supported pipeline | The application has already written the record. The savings begin after the rule runs. |
A filter removes records that match a known exclusion. A sampler selects from a class you still want to observe. Count filtered, throttled, and sampled records separately so the sample rate has a defined denominator.
With Guance, a DataKit blacklist can drop matching logs before they reach the workspace. If you configure a local Filter in datakit.conf, the workspace blacklist does not apply to that DataKit instance. Log indexes route records that were submitted and can apply different retention periods. Use an application or a collector with a supported rule to perform probability- or trace-ID-based sampling. DataKit blacklist · Log indexes
Safeguards that go with sampling
- Keep mandatory events outside the rule. Audit, security, and required business records need a defined retention path. A necessary business event can have INFO severity, so level alone is an unsafe exclusion rule.
- Measure the input, drops, and output. Track the eligible input, discard reasons, and downstream writes. Use independent counters taken before sampling for request and error rates. A lower sampled log count can mean the limiter is dropping more, even while failures rise.
Takeaway
Choose the method after defining what must reach the destination. Use random sampling for a view of routine events, rate limiting to absorb a burst, and a shared hash decision when related records need to travel together. Check the rule with known events before applying it to live traffic, then adjust using measured volume and investigation needs.
Frequently asked questions
Can sampling make an incident harder to investigate?
Yes. A rare error can be missed, and trace-linked logs can still have gaps. Protect mandatory records and use known incidents or controlled events to check what reaches the destination.
What sample rate should I start with?
Measure volume and billing by log class. Start with one high-volume class that is eligible for sampling and check how the retained logs support investigation before tuning the rate. Keep important INFO records in a separate rule.
How are filters and sampling related?
A blacklist discards matching records. Random or hash sampling keeps a subset of eligible records. A rate limit selects by timing and arrival order. Count those drops separately.
Further reading: Reduce logging costs · Logging best practices
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
