I stopped a production migration when a 50M-row CDC snapshot blocked writes

Context

We were migrating the data platform to a new architecture: raw MySQL landing layer → Debezium CDC → Google Pub/Sub → specialized serving layers (MongoDB, ClickHouse, BigQuery). The landing layer was the shared CDC boundary for recommendations, segmentation, reporting, and attribution.

We'd already done a store-by-store cutover for 200+ merchants without issues. But the initial rollout of Debezium on the landing layer itself hit a wall.

What went wrong

Debezium uses snapshots to bootstrap replication state when first connecting to a database. For this deployment, the initial snapshot read the entire landing table and streamed rows into our CDC event transport.

The landing table was ~50M rows. When Debezium started:

• The snapshot query hit 60% CPU. MySQL's query planner picked a suboptimal execution path.
• The read lock held longer than expected. Downstream code hitting the landing layer started timing out.
• Production writes began queueing. The database connection pool saturated.

Within minutes, we had requests piling up. Dashboards weren't updating. Customer-facing systems were degrading.

The recovery

1. DetectionSnapshot drove CPU to 60%; writes began queueing.
2. DiagnosisThe ~50M-row snapshot and lock behavior, not downstream consumers, were saturating MySQL.
3. Stop decisionI stopped the Debezium job and killed the snapshot before migration progress outweighed production risk.
4. RecoveryProduction writes recovered within seconds; Datastream bootstrapped the baseline.
5. Permanent changeWe moved to incremental table migration with a rollback path and production-scale snapshot testing.

Immediate: stopped the Debezium job and killed the snapshot. Production writes recovered within seconds.

Short term: we'd built the system with a fallback — Google Cloud's Datastream service was already capturing changes from the same landing layer as a parallel replication path. We pivoted: use Datastream to bootstrap the initial snapshot, then switch tables to Debezium CDC incrementally.

Long term: this dual-replication strategy (Datastream + Debezium) became our production design for several months. It gave us:

• Safe snapshots (Datastream handles them at scale).
• Gradual transition to pure Debezium (one table at a time).
• A rollback path if Debezium had issues on any table.

Why we didn't see it coming

1. Scale assumptions. We'd tested Debezium snapshots on dev tables (10M rows) and they were fast. Production was 5x larger, and performance doesn't scale linearly.

2. Architectural purity. We wanted a single CDC path — Debezium from landing layer CDC. Two tools felt redundant. This desire for elegance made us skip the safety net.

3. Query optimization assumption. We assumed the snapshot query would use an index. MySQL's query planner chose differently under real load.

The lesson

Don't optimize for architectural purity when production is at risk.

We had a perfectly good tool (Datastream) already in place. Using it alongside Debezium wasn't inelegant — it was pragmatic. It meant:

• We never risked production stability for consistency.
• We had a recovery path.
• We could migrate gradually instead of all-or-nothing.
• We bought time to understand Debezium's behavior at our scale.

The best architecture for production is the one that lets you recover quickly. Redundancy — even redundancy that feels "inelegant" — is not waste. It's insurance.

What changed

• We now load-test snapshots at production scale before deploying new CDC approaches.
• We keep backup replication paths for critical data flows. Managed services (Datastream) handle the heavy lifting of snapshots; custom tools (Debezium) handle ongoing CDC.
• We design migrations with rollback paths. If something breaks at production scale, we can revert and try again without data loss.