Moving from ELK to OpenObserve: queries, historical data, and payback

If an ELK deployment is expensive to operate, OpenObserve's storage architecture is worth evaluating. But after several years of production use, the largest part of a migration may be the queries, alerts, and data conventions that engineers rely on every day. Changing the ingestion endpoint is only the beginning.

The existing pipeline may use document IDs to avoid duplicates, mappings to enforce field types, and saved Kibana queries to investigate incidents. A compatible-looking Bulk endpoint can reduce collector changes without preserving those behaviors. Each dependency needs its own acceptance test.

Append-only logs, mostly queried with filters and aggregations, are a more straightforward migration candidate. A team that knows SQL and can operate the backend can usually estimate that work. Extensive document updates, nested queries, or specialized plugins call for a narrower scope: keep the workloads that depend on ELK and price the rest separately.

Validation scope: the source article was checked on October 9, 2026. Its Bulk experiment used OpenObserve OSS 1.0.4; the Elasticsearch comparison is based on official API documentation. No complete ELK migration was performed. The recorded test results are linked at the end.

Check what Bulk compatibility actually preserves

The recorded experiment sent two create actions to OpenObserve's /api/default/_bulk endpoint with the same _id and different event contents:

{"create":{"_index":"bulk_samples","_id":"same-id"}}
{"event_id":"b-001","message":"first"}
{"create":{"_index":"bulk_samples","_id":"same-id"}}
{"event_id":"b-002","message":"second"}

The actual input also included _timestamp, omitted here for readability. The full input is in the validation bundle. OpenObserve returned errors=false and status 200 for both items. A subsequent query returned two records, both retaining _id="same-id".

In Elasticsearch, create is intended to fail with a conflict when a document with that ID already exists in the same index. The OpenObserve result therefore does not preserve that uniqueness constraint. Accepting the NDJSON format is a different guarantee from implementing Elasticsearch's document semantics. Compare the Elasticsearch Bulk API with the OpenObserve Bulk API.

If the pipeline already treats logs as append-only events and keeps a business event ID for reconciliation, the difference may be acceptable. If it repeatedly sends the same document ID and depends on Elasticsearch to prevent duplicates, add an idempotency mechanism or change the consumption workflow. Otherwise duplicate error events can inflate alert counts as well as stored volume.

Apply the same scrutiny to update, delete, index templates, and lifecycle policies. A pipeline using only an append operation is a smaller project than an application treating Elasticsearch as an updatable document store.

Preserve field types and object relationships

Ordinary timestamps, response codes, and durations are usually easy to map. Missing fields, numeric strings, invalid values, and arrays of objects reveal the important differences.

Consider an order containing {sku:A, price:10} and {sku:B, price:100}. A query for “SKU A with a price greater than 50” should not match. If the migrated data is searched as two independent arrays of SKUs and prices, it can incorrectly combine A with 100.

Elasticsearch already distinguishes ordinary object arrays from a nested model. A migration needs negative test cases that expose this distinction; matching field counts is not enough. See the Elasticsearch array documentation.

The source validation did not test every nested representation. Preserving the original model, splitting child objects into separate events, and rewriting the query are different design choices. Splitting events may simplify a query but increase record count, which matters for event-based pricing. A large business-data redesign may not be justified by lower log-storage cost alone.

The duration test provides a more concrete example. With "n/a" mixed into a field, an OpenObserve CAST query returned HTTP 200 while reporting partial results and a conversion error. TRY_CAST allowed the query to proceed, but invalid values did not satisfy the numeric predicate. If the old mapping rejected those records at ingestion while the new system accepts them, add data-quality monitoring and reconcile the populations before comparing success rates.

Migrate the queries that responders need first

Prioritize the searches used during incidents: error lookups, request correlation, latency distributions, and recurring alerts. An infrequently opened historical dashboard can wait. A query used by the on-call rotation cannot disappear on cutover day.

Moving KQL or Query DSL to SQL involves more than syntax. Check case sensitivity for exact matches, whether missing durations enter a denominator, the timezone used for minute buckets, percentile definitions, and the treatment of no-data windows in alerts. When results differ, compare the event set and field values first, then the aggregation, and only then the chart.

Fix a representative event dataset and absolute time range before rewriting queries. Differences can then be traced to a specific expression or record. Migrating a set of roughly twenty frequent queries is a useful planning exercise: it exposes whether the work is mostly syntax changes or a replacement for custom aggregations and plugins. A successful SELECT * LIMIT 100 tells you very little about that effort.

OpenObserve's SQL interface can be a good fit when most queries are straightforward filters and aggregations. A large investment in Kibana objects, complex DSL, or plugins changes the economics. Those assets do not become free to replace because the destination has no license fee.

Choose a historical-data strategy before exporting

An Elasticsearch snapshot belongs to Elasticsearch's restore workflow. It is not a generic data file that OpenObserve can ingest. Moving historical records generally requires a logical export and a write path into the target model. See the Elasticsearch snapshot restore documentation.

For large result sets, repeatedly increasing from is not a good export strategy. Elasticsearch recommends search_after for deep pagination, with a point in time (PIT) when a consistent view is required. Carry forward the complete sort tuple and use the latest returned PIT ID. A timestamp alone is not a safe checkpoint when several records can share it. See pagination guidance.

A PIT is not an indefinitely durable migration checkpoint. After it expires, do not assume a new PIT's internal sort positions can resume the old export. A more recoverable design exports bounded time segments that can be repeated, saves counts, event IDs, and checksums with the files, and marks a batch complete only after target-side acceptance.

Keep original event time separate from export time. Send new writes through an incremental path during the transition and reconcile the overlap between the historical and live streams.

Test the destination's age limits before producing a large export. In the earlier OpenObserve validation, the allowed ingestion age was explicitly set to 24 hours. An event from two days earlier produced HTTP 200, but the response reported zero successful records and one failure. A 30-day retention policy does not imply that the service will accept records from 30 days ago. Discovering that difference after exporting terabytes extends both the backfill work and the period of parallel operation.

There are three practical options:

• Let old data expire in ELK. This avoids backfill when historical access is infrequent and the old cluster can remain available for its retention period.
• Backfill selected records. Move the subset that is regularly queried or required in a shared view.
• Migrate the full history. Justify this when long-range queries must be unified or an early shutdown of the old platform offsets the export and rewrite costs.

Moving history does not make the records more valuable. It buys a different access path and, potentially, an earlier exit from the old system.

Calculate the payback period with transition costs included

Let E be the full monthly cost of continuing with ELK, O the destination's steady-state monthly cost, and M the one-time migration investment. Ignoring growth, simple payback is:

Payback in months = M / (E - O), provided E > O

Include parallel ingestion, retained historical infrastructure, contractual commitments, query rewrites, and additional maintenance in the relevant cost categories. A reduction in the storage bill is not necessarily the net monthly saving.

For example, if the net saving is CNY 10,000 per month and migration, query changes, and transition spending total CNY 240,000, simple payback is 24 months. These are illustrative inputs, not vendor prices. Recalculate if the application architecture will change next year or the existing ELK deployment is about to shrink.

Use that result to set scope. Append-only logs with a clear SQL workload and manageable rewrite effort are a credible OpenObserve candidate. Important ELK-specific queries and a long payback period may favor keeping the current system. If operating the platform itself is the main burden, include a managed option such as Guance in the same cost model.

Guance provides Pipeline processing, log queries, and external indexes for supported data sources. An external index may let some history remain in the original system, reducing export and backfill work while retaining the old system's operating cost. It is not a drop-in implementation of arbitrary Elasticsearch queries or cross-engine joins. Verify connection, field, and query constraints in the Pipeline guide and external-index documentation.

The decision should identify which recurring work disappears, which compatibility obligations remain, and how soon the net saving repays the transition.

The recorded duplicate-ID and query-error results and historical-ingestion-age test include Chinese explanatory notes. Their conclusions are limited to the listed versions and inputs; they do not replace production compatibility testing.