Migrate log platforms with dual writing, reconciliation, and rollback

A log migration can keep writing to two platforms for weeks without getting closer to a safe cutover. Both dashboards show the same daily volume, yet an important service is missing events, alerts behave differently, or nobody knows how to recover the old platform after its writes stop.

Dual writing creates a comparison window. Completing the migration requires stronger evidence: identifiable events arrive correctly, operational queries retain their meaning, alert ownership is clear, and the rollback path can recover the data generated during a failure. Decide those acceptance conditions before deciding how many days to overlap.

Published by Guance. The reconciliation examples use synthetic data; the migration plan has not been exercised against a customer's production environment.

Choose where the two paths separate

If logs already pass through Kafka or Redpanda, independent consumer groups are often a practical migration boundary. Both platforms can read the same retained stream, keep separate offsets, and catch up independently. Two consumers in the same group generally divide partition work; they do not each receive a complete copy. See Kafka's consumer design.

Independent consumption still shares the broker. Extra reads, retention, storage, and catch-up traffic consume capacity. Verify that the stream retains enough data for a destination outage plus recovery and that a replay does not overwhelm the other consumers.

Collector fan-out can be simpler when a shared stream does not already exist. A collector sends each event to the old and new destinations, avoiding a new broker solely for migration. However, two output branches can share process memory, disk, CPU, source acknowledgements, and backpressure. A blocked new destination may affect the existing one.

For example, Vector's end-to-end acknowledgements can account for the processing status of all destination copies. Source support, buffer policy, and sink failure behavior matter. A diagram with two arrows does not establish failure isolation; test a slow or unavailable destination with the exact configuration. See Vector acknowledgement behavior.

Keeping history where it is can be the least disruptive option when there is no forced shutdown date. If a contract or infrastructure retirement requires complete removal, calculate the cost and duration of backfill before choosing that approach. Historical data is often much larger than the short dual-write window.

Reconcile events, not just totals

Suppose a source produces six distinct events. The new destination drops one and stores another twice. Its count is still six, but it contains only five unique source events. A total-count comparison passes while the migration has both loss and duplication.

A small synthetic OpenObserve lab accompanying the original analysis demonstrates this pattern using deliberately controlled input. It is an illustration of a reconciliation failure, not evidence that OpenObserve inherently loses or duplicates events. The supporting lab report is in Chinese.

There is a second blind spot: an upstream collector drops an event before fan-out. Both destinations receive the same incomplete set. Comparing them with each other cannot reveal what neither received. For critical streams, reconcile against an independent source ledger, durable queue positions, or another suitable source-of-truth record. Synthetic probes help detect a broken route, but a few successful probes do not prove complete delivery of all production events.

Assign a stable event ID before fan-out and preserve it across retries. An ID created separately at each destination cannot join the two copies. A hash of message text alone also conflates genuinely distinct events that happen to contain identical messages.

Reconcile closed time buckets by service, environment, and source. Within each bucket, compare:

• Total records and distinct event IDs.
• IDs missing from either destination and unexpected extra IDs.
• Per-ID multiplicity, so duplicates remain visible.
• Required field values and types, including timestamp, severity, trace ID, and numeric measurements.

The required invariants depend on the pipeline. An intentional aggregation or filter changes the expected result and should be documented before counting differences as failures. If production data has no reliable event ID, state the limits of the comparison instead of claiming exact reconciliation from approximate counts.

Separate event time from the cutover position

A log can have several relevant times: when the application created it, when a platform ingested it, and when a query could first find it. Parsing rules, clock skew, buffered delivery, and indexing delay can separate those times substantially.

In the supporting synthetic lab, six records were visible using their ingestion time but none matched the intended event-time window until the source timestamp was correctly mapped to OpenObserve's _timestamp. After the mapping, the same six matched. That difference concerned timestamp interpretation; it was not proof of transport loss.

Agree on timestamp field, unit, timezone, and missing/invalid timestamp behavior in both pipelines. Include delayed events and events near bucket boundaries. A destination that silently substitutes ingestion time can look correct in a broad daily chart while breaking an incident query for a five-minute window.

Use a durable stream position to describe a streaming cutover where one exists: topic, partition, and offset. A timestamp alone is not an exact checkpoint. An event with an old event time may be appended after the cutover offset, and partition offsets cannot be collapsed into one global number.

Before stopping the old consumer path, verify that the new destination has processed through each agreed partition boundary and reconcile the corresponding event set. A recorded consumer offset alone does not prove that the destination stored every event successfully.

Measure how long events take to become searchable, then leave an appropriate settling window before comparing a closed bucket. Recheck buckets after late arrivals. The window should reflect observed behavior and the acceptance objective, rather than an arbitrary delay that hides a slow or unreliable path.

Migrate the queries people depend on

Data arrival is only one part of the migration. Inventory the saved searches, dashboards, alerts, and runbook links used during real incidents. Translate their semantics, then run them against the same known cases.

A simple example is a set of three error events, one without a duration_ms value. SQL-style count(*) returns three rows, while count(duration_ms) returns two non-null values. Replacing one with the other changes an error-count alert even if the surrounding query looks similar. Likewise, test missing fields, case sensitivity, tokenization, regular expressions, numeric coercion, percentile calculations, and time boundaries where relevant.

Run new alerts in shadow mode first: evaluate and record their results without paging the same responders twice. Compare both expected triggers and expected non-triggers, including periods with no data. Before activating notifications, name the platform that owns each alert and disable or redirect the superseded route. A rollback plan must restore notification ownership as well as a dashboard URL.

Check access using ordinary user roles. An administrator seeing all migrated data does not demonstrate that a service team can query only the intended indexes or tenants. Preserve required redaction and field visibility policies when mapping fields.

Finally, inspect log-to-trace links. A log may remain searchable long after its associated trace expires. A valid trace_id and link template cannot recreate expired APM data; align retention with the troubleshooting workflow that actually needs the relationship.

Budget for the overlap and its exit conditions

A successful quiet week does not cover the conditions likely to break migration. Exercise peak traffic, releases, collector restarts, destination throttling, outages, and replay. Record what must pass before ending the overlap and who decides that the evidence is sufficient.

For a hypothetical source generating 500 GB of raw logs per day, sending an extra full copy for 14 days transfers another 7,000 GB, or 7 TB using decimal units. That is additional raw volume sent. It does not establish compressed storage size, indexed volume, network charges, or either vendor's bill.

A useful planning model is:

migration cost = M + D × n + F

Here M is one-time migration work, D is incremental daily cost during overlap, n is overlap days, and F is the old platform's continuing fixed or minimum commitment for the relevant period. Put all terms in the same currency and avoid counting a charge in two terms. Include engineering time, temporary capacity, data transfer, replay, and backfill where they apply.

The model explains why an open-ended overlap becomes expensive even when the destination's nominal ingest rate is attractive. It also explains why immediately copying all history may cost more than leaving it to expire. Get a quote based on your measured data and retention rather than translating raw GB directly into a price.

Make rollback a data-recovery plan

Returning users to the old query page is easy while the old platform still receives complete data. After old writes stop, that page has a growing gap. A useful rollback must either keep the old path warm or replay that gap from a source with sufficient retention.

The replay window must cover detection time, the decision to roll back, repair or reconfiguration, and catch-up. A source retaining six hours cannot recover an outage first discovered 24 hours later. Recovery also requires sufficient throughput to catch up while new traffic continues.

A staged cutover makes these boundaries explicit:

1. Move selected queries and dashboards after their data and semantics pass acceptance.
2. Transfer alert and notification ownership after shadow evaluation.
3. Stop old writes only when the agreed recovery path and checkpoint are ready.
4. Retire the old platform and replay reserve after the rollback window closes and historical obligations are met.

Test recovery from a known checkpoint before relying on it. Preserve the event IDs used for reconciliation, and decide how the destination handles duplicates during replay. Record which step is still reversible through configuration alone and which now requires data restoration.

Plan a migration to Guance

When historical data may remain in its current platform, Guance's external indexes can provide a query path to supported external sources. The documented integrations include SLS Logstore, Elasticsearch, OpenSearch, LogEase, and Volcengine TLS. This does not migrate those records or eliminate the source platform's charges. Confirm supported queries and map time, _docid, and message appropriately; a non-unique document ID can produce misleading detail views. See external index configuration.

For new writes, use the same reconciliation and query acceptance process described above. Keep service identity, event time, severity, and trace identifiers consistent, then verify log-to-trace correlation on an actual retained trace.

If the old platform must disappear on a fixed date, external querying is only a transition. Plan and validate the required historical export and backfill. If history can expire naturally, make the last-write date, remaining retention, and final retirement date explicit so the temporary arrangement has an end.