How to reduce logging costs in seven steps

Measure log ingestion and storage costs, remove low-value records, set sampling and retention, and monitor billable usage without losing the evidence your team needs.

Best practices
Illustration of log collection, routing, and storage

Start reducing logging costs with two questions. Which line on the bill is growing, and which records would you need to reconstruct the next incident? A burst of health-check traffic may drive up ingestion, while a rarely used archive can accumulate storage charges quietly. Those problems call for different changes.

The seven steps below start at the meter and work back through what the application emits, what the collector keeps, and how long the platform can query it. The product examples follow the published Guance documentation. The volume calculation uses hypothetical traffic.

1. Find the meter behind the bill

Charge What to check
Ingestion Is the meter based on uncompressed GB, compressed GB, or the number of records? At which point in the pipeline does it count?
Indexing and retention Does the plan charge by indexed entries and retention tier, by storage volume over time, or a combination?
Queries and archive retrieval Check the scanned bytes, reads, or rehydrated volume that the provider bills. Check object storage requests and cross-region transfer separately.
Operating the pipeline Include queues such as Kafka, transfer, collector capacity, and the engineering time to keep the path running.

Suppose a service sends 100 GB of uncompressed logs a day for 30 days. That is 3,000 GB. If you remove records accounting for 20% of those bytes before the meter, the same measure falls to 2,400 GB, a difference of 600 GB. The bill depends on the applicable price tier, commitments, and other meters. A 20% cut in event count could remove a very different share of bytes.

In Guance, Commercial Plan standard log storage defaults to entry-based metering. When the workspace switches to write-volume billing, the meter uses the volume before compression. The Enterprise Plan meters newly ingested compressed volume and stored volume separately. Match the comparison to your workspace and contract.

2. Stop generating low-value logs

Successful health checks are a good place to look. If they serve no audit, liveness, or debugging purpose, keep the availability metric and stop emitting the redundant access records. Keep the requests that help explain failures and slowdowns, along with audit and business events that need individual records.

Request volume and error rate can often come from metrics collected before any log sampling. Group them by a bounded label such as a route template. An ID for every user or request would turn a cheap counter into a high-cardinality metric. Raise log levels for the modules where routine output adds little, and check whether the application supports a temporary runtime change before relying on it.

If you use Guance's workspace Blacklist, DataKit applies matching rules before upload. If a collector has a local filter in datakit.conf, workspace Blacklist rules do not apply to that collector. Review the actual matched records and the monitors that depend on them. Anything you still need in an archive needs a collection path that preserves it.

3. Sample routine events and keep exceptions explicit

Apply sampling to a defined class of frequent, low-value events. Routine INFO records may tolerate a lower sampling rate. Keep required audit records and critical business events outside the sample and favor errors. Even a well-chosen sample can miss a rare sequence, so use independent counters or measurements taken before sampling for totals and error rates.

When several services log one request, a shared hash rule on the propagated request ID makes their keep/drop decision consistent for that stream. The same key, algorithm, and threshold must reach each service. A later error cannot recover upstream records already discarded, so plan the exception path before sampling.

4. Set retention around how people query

Measure how far back responders actually look. A short searchable window may cover routine diagnosis, while occasional failures or audits need records spanning a longer period. Set the windows by data class, then verify whether the archive can return the fields, time range, and access controls you need.

Guance ordinary log indexes match rules in order and use the first match. Put the narrow audit rule above a broad production rule. Check where representative events land before changing their retention. Storage policy changes can leave existing data billed under the old policy until cleanup, so allow for the transition when comparing bills.

Guance Data Forwarding can send matching reported data to destinations such as object storage. Verify forwarding, destination storage, and retrieval costs separately from the searchable index. Once a collector drops an event, the platform has no copy to forward.

5. Limit repeated error records and count every occurrence

During an outage, thousands of copies of one stack trace can crowd out a new error. Group duplicates by service and a stable error type, then keep the first or a few full examples per time window. Record occurrence and suppression counts independently so responders can see the scale and progression of the incident. Give new error types and required audit records their own path.

An application logger or a stateful collector can enforce a window when it supports the required counters. A static filter only matches or drops a record. Check the scope of the limit across instances so the team's estimate reflects where the rule runs.

6. Alert on volume before invoice review

Monitor both what enters the collector and what reaches the searchable store. If the store's event count falls while the collector's suppressed count rises, the system could still be in an error storm. Compare bytes per event and metered usage with the invoice to see whether the change affects the cost.

In Guance, configure Log Detection with the appropriate index, source, service dimensions, and evaluation window, then set a threshold and notification policy. This counts matching queryable events. Track collector input bytes and drops separately when you need the pre-filter volume. Start with common causes such as lingering DEBUG output, a noisy new service, or repeated errors.

7. Compare platforms with the same workload

If the bill still exceeds the budget, compare the platforms using the same traffic, sampling, retention, and query requirements. Include indexed and retained volume, archive retrieval, network transfer, and the cost of operating collectors. Check whether allowances apply to that workload and whether each platform can retain different classes separately.

Migration also involves shipping protocols, parsers, queries, alerts, dashboards, and historical logs. Estimate which pieces move directly and which need rebuilding before treating a lower unit price as a lower total cost.

Keep the cost review tied to incidents

Measure first, remove low-value records, then adjust sampling and retention. Watch the meters after each change and keep enough evidence to answer the questions your responders and auditors ask. Repeat the review as traffic and query patterns change.

Frequently asked questions

Will reducing logs hide an incident?
Start with duplicates and records whose operational purpose is already covered by metrics. Preserve required audit events, useful errors, and critical business records. Consistent request-level sampling helps keep related entries together where the same rule is applied. Collection gaps and later filters can still remove individual records.

How long should logs be retained?
Use the time ranges people actually search, plus the applicable business and audit requirements. Check both how much the archive keeps and how long a retrieval takes. In Guance, review the available retention options for each ordinary index in your workspace.

How do we make a case for collecting less?
Rank sources by the billed meter and inspect samples of the largest low-value sources. Estimate the change under your contract, then check the searches and alerts that use the affected data. A count of the ten noisiest sources only tells part of the story if the bill is based on bytes.

Further reading. Log sampling (Simplified Chinese) · Logging best practices

Guance docs. Logs · Log Index · Monitoring

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting