Hot partitions, shared reads, a live database migration, and the deleted messages that froze it at 99.9999%
One busy Discord channel could slow down conversations that had nothing to do with it.
Discord was storing trillions of messages across 177 database servers. Adding more servers didn't fix the real problem. So the engineers changed how requests reached the database, and then moved every message to a new one. When that move was 99.9999% complete, it stopped. The reason: messages people had already deleted.
This article follows a single message through all of it. If you'd rather watch, here's the animated version:
This is a historical case study. The migration finished in May 2022 and Discord described it in March 2023; the background comes from Discord's January 2017 post. It isn't Discord's architecture today. Diagrams of servers and traffic are illustrative.
Where a message lives
Start with what the app actually asks for. Open a channel and you need its newest messages. Scroll up and you need the ones just before. That's an ordered slice of one channel's history.
So Discord stored messages by an address and a position. The 2017 post describes a schema like this (simplified):
PRIMARY KEY ((channel_id, bucket), message_id)
The part in the inner parentheses is the partition key: the channel plus a time window called a bucket. It decides which machines hold the data. The message_id is a Snowflake, an ID that sorts roughly by creation time, so it gives each message its position inside that group.
Think of a shelf. The address takes you to the right shelf. The position tells you where to start reading.
Why the bucket? A busy channel keeps growing for years, and buckets cut its history into pieces of manageable size (the 2017 post describes 10-day buckets; that's historical, not a current setting). The trade-off is that a quiet channel may need to check several buckets to fill one screen.
But notice what a bucket limits. It limits how much history sits in one place. It does not limit how many people ask for it at once.
Spreading data is not spreading demand
Cassandra, the database Discord used, spreads partitions across machines by hashing their keys, and keeps three copies of each.
Now make one channel wildly popular. Every request for it still goes to the machines that hold its copies.
Would adding more servers help? It helps the cluster, but not this channel. New servers take over some ranges of data from the old ones, which frees up capacity overall. A single partition is never split, though. It moves as one piece, and all of its traffic moves with it.
Every machine can hold the same amount of data and still get wildly different traffic.
Why the quiet channel waits
Inside one of those machines, two jobs compete. Users are reading. And the database is constantly tidying its own files, merging them and throwing away old versions. That's compaction, and it uses the same CPU and disk the users need.
Discord read and wrote with quorum consistency: with three copies, a request waits for two of them to answer. If one copy is slow, that's usually fine, because the other two answer. But traffic for a hot partition concentrates on the replicas that hold it. When enough of those are overloaded, a quorum request can't avoid them, and it waits.
Those same machines also hold lots of quiet channels. So queries that touch them see higher latency too, including a quiet channel that just happened to share them. Discord saw latency cascade from node to node as machines struggled to compact, and the team had to step in by hand to keep nodes healthy.
Stop asking the same question twice
Before buying bigger databases, Discord asked a better question: does every request need its own database read?
Picture a big announcement. Hundreds of people open the channel in the same second, and they all ask for the same messages. A naive system sends hundreds of identical reads.
Discord built a layer of small services in Rust that sit between the app and the database. When the first request for some data arrives, the service starts a read. When an identical request arrives while that read is still running, it doesn't start another. It joins the first one and waits for the same result. That's request coalescing.
It isn't caching. A cache reuses an answer that already finished. Coalescing shares a read that's still in progress.
The clever part is deciding what counts as the same request. Same channel isn't enough: the newest page and last week's history are different questions. And a shared read shares its failures too. If it errors, everyone waiting gets the error. So coalescing only works on truly identical work.
Making the requests meet
There's a catch. Discord runs many copies of these services. If identical requests land on different copies, each copy starts its own read, because it can't see what the others are doing. You've built a meeting point, then sent everyone to different rooms.
So Discord routed by channel, using a consistent hash on the channel ID. Every request for the same channel goes to the same service instance, and identical requests actually meet.
Two separate ideas, working together: routing decides where requests go; coalescing decides which of them can share one read. And this routing is separate from the database's own placement. One picks a service instance. The other picks where the data lives.
A new engine
Fewer duplicate reads helped, but Discord also wanted a database that behaved more predictably under load. They picked ScyllaDB. It speaks the same query language as Cassandra, so the data model could stay the same. Underneath it works very differently:
- It's written in C++ instead of Java, so there's no garbage collector pausing the database to clean up memory.
- It splits each machine by CPU core. Every core owns its own slice of the data and memory and does its own work, and cores message each other only when they have to.
A new engine doesn't make hot partitions impossible, and Discord says they can still happen on ScyllaDB. The fix was the combination: less duplicate work, and a more predictable engine underneath.
Before switching, Discord tested their real queries. One failed: reverse queries, which scan messages in the opposite direction to how they're stored, were too slow. The ScyllaDB team improved it, and that removed the last blocker.
Benchmark the awkward query, not just the easy one.
Moving the floor while people stand on it
Replacing the engine while trillions of messages were still changing was a different challenge. Discord started by writing every new message to both databases, then had to copy everything that already existed.
That sounds simple until things change mid-copy. Someone edits a message you're copying right now. If the old version lands after the new one, the old one must not win. And writing to two databases isn't one step: if one write succeeds and the other fails, "we wrote to both" no longer means "both agree".
Discord's first plan used Spark, and the estimate was three months. So they wrote their own migrator in Rust. It reads the old database in chunks, saves its progress (in SQLite) so a crash doesn't restart everything, and pushes the data across. The new estimate was nine days, and at its peak it moved up to 3.2 million messages per second. Both "three months" and "nine days" were estimates, not measured run times.
Copied is not the same as correct
A new database can hold every row and still give the wrong answer: messages in the wrong order, an old edit, a deleted message that came back, or an empty page.
So Discord checked behaviour, not just row counts. They sent a small share of real reads to both databases and compared the answers. That's strong evidence, but it isn't proof. Treat copy progress, validation and readiness as three separate signals. A green progress bar answers only one of them.
The deleted message problem
When you delete a message in a database with three copies, you can't just erase it. If one copy was offline at that moment, it comes back later still holding the message, and the deleted message rises from the dead.
So the database writes a marker instead: a tombstone, which says "this was deleted". Reads have to step over every tombstone, and tombstones hang around until compaction is allowed to clean them up.
Discord already knew this could hurt. In 2017, one channel with millions of deleted messages took twenty seconds to load. Deleted, from the user's point of view, does not mean gone from the database.
Stuck at 99.9999 percent
Several days into the migration, the progress bar hit 99.9999% and stopped. The last few ranges kept timing out. Inside them were gigantic runs of tombstones the old database had never cleaned up.
So the team compacted that range. Seconds later, the migration was complete.
Never guess the time left from the percentage left.
What changed
From Discord's March 2023 post:
- Database nodes: 177 Cassandra nodes became 72 ScyllaDB nodes.
- Reading message history, p99: 40โ125 ms before, 15 ms after.
- Inserting a message, p99: 5โ70 ms before, a steady 5 ms after.
p99 is the time that 99% of requests finish within. These aren't a controlled benchmark: the hardware changed too (about 4 TB per old node against 9 TB per new one), and so did the application path.
Then came the 2022 World Cup final. Every time Messi or Mbappรฉ scored, a flood of people reacted in the same few seconds, and you can see the whole match in Discord's message graph. It was the kind of crowd that used to overload the database. This time, by Discord's account, it didn't even break a sweat.
Three lessons
- A good partition key doesn't stop a hotspot. It bounds how much data sits in one place, not how many people ask for it.
- A faster engine doesn't stop duplicate work. Route identical requests together, then let them share one read.
- A finished copy doesn't prove the data is correct. Compare real answers, and watch out for what deletes leave behind.
Each layer solves a different problem.
Watch the full animated version: Why One Discord Channel Could Slow Down Everyone (YouTube)
Have you ever had a progress bar stall at 99%? What was hiding in the last 1%? ๐
Sources
- Discord, How Discord Stores Trillions of Messages (March 2023)
- Discord, How Discord Stores Billions of Messages (January 2017)
- Apache Cassandra docs: Dynamo architecture and Tombstones
- ScyllaDB documentation (shard-per-core design)
Follow The Byte Mechanics
How software works, under the hood โ๏ธ





















Top comments (0)