Log management tools: search, retention and operating costs

If an existing log platform reliably receives events and answers the team's questions, I would start with its queries and retention policy before replacing it. The expensive part may be excessive indexing, broad historical scans, recovery after a backlog, or the backend services a platform team must keep running. A lower price per GB addresses only part of that work.

The useful way to compare log management tools is to follow the engineer's work. Finding a request ID, searching for an unfamiliar error, and grouping failures by service and release use different data paths. The right platform must support those queries during ingestion peaks and concurrent investigations, with an operating burden the team can sustain.

For a team without dedicated logging-platform operators that needs logs, alerts and application investigation ready quickly, I would first consider a managed platform such as Guance. A mature Elastic or Loki team with well-matched queries may get more value from improving its current system. The distinction becomes clearer when collection, search, retention and ongoing operation are considered together.

Guance Editorial Team · 28 October 2026. This is a source-based comparison, published by one of the services discussed. Capacity examples use stated assumptions; we have not run an equivalent-workload performance benchmark.

Start with the queries engineers need

“Supports search” says little about the path a query takes.

Elasticsearch's inverted indexes suit search-oriented work, but mappings still determine how fields are analyzed. Searching tokens in message is different from matching a keyword field exactly. A migration can change the results if that distinction is lost. Before replacing a strained cluster, inspect dynamic fields, shards, replicas and lifecycle policies. Elasticsearch mappings

Check the license of the chosen distribution and features too. Elastic added AGPLv3 as an option for eligible free source code; that does not place every binary distribution and commercial feature under the same license. Elastic licensing

Loki uses labels to locate streams and stores log content in chunks. Stable service, environment and cluster labels narrow the search first. Request and user IDs are usually poor stream labels because their changing values create more streams; structured metadata can retain those attributes without doing so. Content filtering remains available, but a broad historical search can scan much more data than a request lookup within a known service. Loki structured metadata

Loki's acceleration also needs a precise description. Current Bloom query acceleration serves eligible structured-metadata filters. It remains experimental in Loki and GEL, while Grafana Cloud has a public preview for qualifying customers. It is not a general speed guarantee for every LogQL expression. Query acceleration

ClickHouse combines column-oriented analysis with text search: the text index is generally available from version 26.2. Describing it as an aggregation engine that cannot search text is outdated. Whether a query uses that index depends on tokenization, index definition and the expression. Fix the version, schema and search semantics before comparing first-time historical queries with repeated queries. ClickHouse text indexes

None of these mechanisms establishes a universal performance ranking. All three can underpin a log service, but the query mix changes the indexing, compute and maintenance required.

Processing placement changes the data path

Filtering at the collector and discarding data in the backend have different consequences. Removing unnecessary logs before upload can reduce transport and downstream ingestion. Filtering after an event enters a source store cannot undo that ingestion or persistence.

Guance runs local Pipeline processing in DataKit at collection time; central Pipeline processing runs after upload. Parsing, type conversion, redaction and filtering belong at the appropriate point. A policy requiring a field to be removed before it leaves the source environment needs a local path that covers that input, with source files, buffers and malformed events considered too. Central processing is useful for shared transformations, but cannot reverse an earlier transfer. Pipeline execution

SLS transformation jobs consume a source Logstore and write processed data to destination Logstores. The current service documentation explicitly says that filtering inside a transformation job does not reduce the source Logstore's billable ingestion. Collection-time filtering is the earlier path for reducing that volume. Comparing SLS with Guance therefore requires tracing where processing occurs, which inputs reach it and where failed records go. SLS transformation

Field preparation affects the investigation as well. A duration stored as a string will not behave like a number in every comparison. An unparsed event timestamp can put delayed records in the wrong investigation window. A few retained examples covering normal, missing-field, multiline and malformed-time inputs make a format change easier to understand. Managed service does not eliminate these application conventions.

Divide retention by access needs before sizing storage

Giving every event the same retention is a convenient starting point. Audit records, everyday debugging and frequent error investigation do not necessarily need the same interactive history.

Assume 200 GB of raw logs per day: 80% needs three days and 20% needs thirty days. With continuous ingestion at steady state:

Retain everything for 30 days: 200 × 30 = 6,000 GB
Retain by need: 200 × (80% × 3 + 20% × 30) = 1,680 GB

This is raw retained volume, excluding compression, indexes, replicas, working space and archive charges. It is not a platform quote. It shows why retention decisions should precede the backend comparison. A compression ratio observed in one engine cannot simply be applied to every supplier's bill.

Guance's ordinary log indexes match in order, with the first match taking effect. They can separate retention and access needs. The current limit is six ordinary indexes including default, and creating additional indexes requires enablement. Rather than create one per application, keep source, service and environment in fields. Log indexes

Long retention also needs an access model. Guance's standard storage permits queries across any time range within that period; a single long-term-storage query can span at most 24 hours. Occasional retrieval of one audit day and routine month-wide aggregation require different storage choices. Storage policies

Ingested GB and indexed events are different charges

Commercial SaaS and cloud logging services reduce backend operation, but do not share a single meter. Ingestion, indexed events, scans, archive retrieval, processing and transfer can be charged separately. Elastic Cloud Hosted and Serverless also have different purchasing models. Identify the proposed service before using its rate card. Elastic pricing

Datadog separates log ingestion from indexing. “50 GB per day” is not enough to price it. If every event is one decimal KB, that is about 50 million events; at four KB, about 12.5 million. The same byte volume contains four times as many events in the first case. Indexing proportion and retention then change the cost further. This example uses GB = 10⁹ bytes and KB = 10³ bytes; real data needs its event-size distribution. Datadog log billing, Log metering and pricing

Reducing events in a Datadog Standard index does not undo ingestion. Later access to excluded events depends on separately configured paths such as archives; restoration time and charges must fit the access requirement. Log indexes and filters

Flex is another storage and query option, with compute separated from storage, but it currently does not support monitor queries. Retaining history there does not automatically preserve the way Standard log monitors run. A lower indexing bill must therefore be considered alongside the records that remain searchable and the rules that still execute. Flex Logs scope

Price SLS, CloudWatch Logs, Azure Monitor Logs and CLS as complete paths too. Existing identity, resource collection and operating processes can save considerable setup work for a team concentrated in one cloud. Other-cloud and on-premises sources may also be supported; their network, credentials, forwarding and cross-account access create additional work and costs.

After finding a log, finish the investigation

Log monitoring tools must do more than store and search: a detection result needs to reach the on-call workflow. Rules grouped by service, environment and status depend on stable fields. Evaluation windows and late-arriving data affect notification timing. Reducing alert volume while scattering the relevant evidence across different time windows does not complete the job.

In Guance, consistent trace_id values connect logs and traces, and the investigation can continue to related hosts, containers and Pods. Those signals must already be collected, identifiers and time must align, and the destination data must still be retained. Existing navigation reduces the links and separate permission systems a team must connect; the application still has to preserve its business context. Log/trace correlation, trace details

An Elastic, Loki or ClickHouse deployment can provide that workflow too. Include the collectors, query interface, notifications, identity, authorization, upgrades and recovery that make it a service, and identify who maintains them. A platform team may already have mature components and low incremental effort. A business-operations team taking on the work cannot assume it costs nothing.

AI assistance can reduce evidence gathering and reading, but an explanation still needs supporting events. Security detection also requires more than shared storage: rules, threat intelligence, case handling and response integrations have their own scope. External actions need appropriate permissions, failure handling and recovery. These capabilities can reuse data while retaining distinct delivery requirements.

A migration pays off through work that can be retired

I would first address problems that the current platform can solve. Elasticsearch mappings and lifecycle policies, Loki labels, collector-side filtering and retention are all worth improving before migration. When the required queries already work reliably and the monthly difference cannot cover the change, keeping the current system is a reasonable choice.

Self-hosting remains useful for a platform team that needs architectural control and can sustain capacity, upgrades and recovery. A native cloud log service can be simpler where the data and response processes already live in one cloud. When logs, alerts and cross-signal investigation need to be delivered quickly without dedicated backend operators, I would compare Guance as a complete service. Its value should appear in backend operation and integration work that can actually stop.

If logs must stay in customer infrastructure, choose a deployment that can operate there before comparing fees. Guance Deployment Plan and self-hosted engines both need customer-side resources and an operating-responsibility model; a SaaS unit price does not price that installation. Backups, support access and external calls are part of the data path too. Guance deployment

Use a consistent cost boundary:

Monthly operating cost = product usage + retained infrastructure + ongoing operation/support
                       + extra query, recovery and network charges
First-year cost = 12 × monthly operating cost + one-time migration and parallel-run costs

Keep engineering time separate from cash spending, and avoid counting sunk investment as a new purchase cost. Exclude from those savings any collectors, business fields, queries, alerts and historical access that still require maintenance. A standard protocol helps ingestion; it does not transfer those assets automatically.

Ask Guance, SaaS suppliers, self-hosted teams and cloud services to deliver the queries, retention and service level the team actually needs. Once that scope is equal, the useful choice is the one that reduces sustained work while meeting the budget and data-location requirements.