Zero Downtime Database Migration: The Cassandra Story

Every large system eventually outgrows the database it was born on. The migration itself is rarely the scary part. The scary part is moving data while millions of people keep reading and writing every second, expecting nothing to break. That is what a zero downtime database migration really tests: not your new engine, but your ability to move a moving train onto new tracks without anyone noticing.

This post walks through how teams pull that off, using the well-documented case of Discord moving trillions of messages off Apache Cassandra. It shows why a mature team outgrows a database and how they migrate without downtime. The patterns apply whether you are leaving Cassandra, Postgres, or Oracle.

Why teams outgrow Cassandra

Cassandra is a strong default for write-heavy, always-on workloads. It scales horizontally, has no single point of failure, and uses consistent hashing to spread data across nodes. So why leave it?

The pain shows up at the tail, not the average. Discord's engineering team publicly described latency spikes tied to Cassandra's JVM garbage collector and to hot partitions, where a few very active rows overwhelmed the nodes serving them. As message volume grew from billions to trillions, compaction started fighting with live traffic for the same disks.

A few common triggers push teams off a database they once loved:

• Tail latency that grows faster than average latency, so the ninety-ninth percentile becomes unpredictable.
• Operational cost, where keeping the cluster healthy consumes more time than building features.
• Hot spots that no tuning fully fixes, because the access pattern itself is skewed.

Discord's answer was ScyllaDB, a Cassandra-compatible store written in C++ that avoids garbage collection pauses. Keeping the same data model meant the migration project was infrastructure work, not a full rewrite.

The anatomy of a zero downtime database migration

A zero downtime database migration is not one big switch. It is a sequence of small, reversible steps where the old and new systems run in parallel until you trust the new one. The database migration strategy almost always follows the same five phases.

Phase 1: Dual writes

First, write every new change to both the old and the new database at once. The old store stays the source of truth. The new one is catching up.

# writer.py
async def save_message(msg):
    await cassandra.write(msg)          # source of truth
    try:
        await scylla.write(msg)         # best-effort mirror
    except Exception as e:
        metrics.increment("shadow_write_failed")
        log.warning("scylla write failed", error=e)

Phase 2: Backfill the history

Dual writes only capture data from the moment you turn them on. Everything older still lives only in the old store, so a backfill job streams historical rows into the new database in the background.

This is the slowest phase. Discord built a dedicated data migrator in Rust to move data at high throughput while respecting rate limits, so the data transfer did not starve live traffic. The backfill must be idempotent: you can re-run any chunk safely, because jobs at this scale crash and resume often.

# backfill.py
def backfill_partition(partition_key, checkpoint):
    rows = cassandra.scan(partition_key, after=checkpoint)
    for batch in chunk(rows, size=500):
        scylla.write_batch(batch)        # idempotent upsert
        save_checkpoint(partition_key, batch[-1].id)
        throttle()                       # yield to live traffic

Phase 3: Shadow reads and data validation

Now both source and target databases hold the same data, in theory. Before you trust the new one, you verify. You read from the old store to serve the user, and in parallel you read the same key from the new store and compare. This data validation step is where quiet corruption gets caught.

# reader.py
async def get_message(msg_id):
    real = await cassandra.read(msg_id)  # served to the user
    asyncio.create_task(shadow_check(msg_id, real))
    return real

async def shadow_check(msg_id, expected):
    if await scylla.read(msg_id) != expected:
        metrics.increment("read_mismatch")

Phase 4: The cutover

Once the mismatch rate is effectively zero and stays there, you flip reads to the new database. The safest way is a gradual rollout behind a feature flag: one percent of reads, then ten, then fifty, then a hundred, watching latency in real time at each step.

The old database keeps taking writes during this window. That is your safety net and your rollback plan. If the new store misbehaves at fifty percent, you flip the flag back to zero and instantly serve from the old, proven system again. No data lost, no downtime.

Phase 5: Decommission

After reads have run entirely on the new database through a full traffic cycle, including peak hours, you stop the dual writes and retire the old cluster. Only now, in the post migration cleanup, is the work truly done.

A visual model of the migration

• Dual writes: writes go to old and new, reads come from old, rollback is trivial.
• Backfill: writes go to old and new, reads come from old, rollback is trivial.
• Shadow reads: writes go to old and new, reads come from old with new compared, rollback is trivial.
• Cutover: writes go to old and new, reads ramp to new, rollback is a flag flip.
• Decommission: writes and reads go to new only, rollback is hard.

Rollback stays trivial right up until the final step. That is the entire point. The discipline mirrors any careful monolith to microservices migration, where you keep the old path alive until the new one earns trust.

What Discord actually gained

Discord publicly reported that the ScyllaDB fleet handled their workload on dramatically fewer nodes than the Cassandra cluster it replaced, cutting both operational load and cost. The latency tail flattened because the new engine did not use a garbage-collected runtime.

There was a second lesson. Discord also added a data services layer in front of the store that coalesced concurrent requests for the same hot partition, so a viral message did not become thousands of identical database hits. The access pattern in front of the store matters just as much, a theme that recurs when teams scale a system to billions of requests.

Common mistakes in a zero downtime migration

Even with the right five phases, teams trip over the same roots:

• Skipping data validation. Cutting over because the backfill finished, instead of a verified zero mismatch rate, is how silent data loss reaches production.
• A non-idempotent migration script. If re-running a chunk double counts or corrupts rows, your migration cannot survive the crashes it will hit.
• Cutting over all at once. A big-bang read switch removes graceful rollback. Always ramp with a flag.
• No disaster recovery plan. Keep backups and a tested rollback plan for the whole overlap window.
• Decommissioning too early. Retire the old cluster only after a full peak cycle in the production environment.

When you do not need a full migration

Not every scaling wall needs a database migration strategy this heavy. Before committing a quarter of engineering time, rule out cheaper fixes before large volumes of data get migrated: a read replica, better partition keys, or a cache in front of the existing store. A full migration is the right tool when the engine itself is the bottleneck, not when you simply have not tuned what you run. For the fundamentals, start with our system design guides.

Conclusion

A zero downtime database migration looks like magic from the outside and disciplined bookkeeping from the inside. Dual write so the new store catches up. Backfill the history idempotently. Shadow read until the mismatch rate is zero. Cut over behind a flag so rollback is one click. Decommission only when the new system has earned it. Discord moved trillions of messages off Cassandra with exactly this playbook, and the same five phases will carry your migration too. Explore more walkthroughs on the Levelop blog.

References

1. Discord Engineering, How Discord Stores Trillions of Messages, discord.com/blog.
2. ScyllaDB, Cassandra-compatible NoSQL database documentation, scylladb.com.
3. Martin Fowler, Evolutionary Database Design, martinfowler.com.
4. Levelop, Consistent Hashing Explained, levelop.dev/blog.

Written by Avinash Tyagi, founder of Levelop. For more system design and interview preparation content, visit the Levelop blog.