Zero results. Ana, a production coordinator at Netflix Studios, saved a new film called Night Train a minute ago, and search can't find it. The movie database has it. The search index has never heard of it.
Ana is our example, but the gap is one Netflix's engineers wrote about. Most engineers reach for the obvious cure: when you save the film, write it to both places. That cure is exactly how this happens. Netflix's engineers wrote down why it fails, and then they turned down the textbook answer too: one transaction across both databases.
So how do you keep two databases in agreement, when you can't write to both at once? The animated version:
The second write fails
Ana's film is one row, film 8812, in MySQL, the database that owns the truth. Search runs on a second store, Elasticsearch. Netflix's engineers call the store that owns the truth the primary store, and a search index like this a derived store.
The obvious code does a dual write: save the row to MySQL, then send the same film to Elasticsearch.
- The second write can fail: a timeout, a deploy, a node restarting. MySQL says Night Train exists, search says it doesn't, and nothing notices.
- Flip the order and you get the opposite bug: search shows a film the database never saved.
- Even when both writes succeed, two editors renaming the film at the same moment can land in a different order in each store. Now they disagree for good.
The repair job becomes the system
So teams add a repair job that re-reads the database and patches search. Netflix's engineers list the costs: custom work nobody can reuse, stores wrong until it runs, and load on the primary. Netflix's movie search team lived a version of this, polling other services for changes on a timer even when nothing had changed; Netflix says it quickly grew very complex and hard to maintain.
The real problem fits in one sentence: no single write can land in two different databases at once.
The textbook cure doesn't fit
The textbook answer is a distributed transaction: one commit that lands in every store, or in none. The common standard is XA: each store first promises it can commit, and only then does anyone commit.
But if the app dies between the promise and the commit, the stores are left waiting; Netflix's post points out that XA blocks in exactly that case. And Elasticsearch doesn't support XA, or any transaction that spans different kinds of store. A transaction across two stores is only as strong as the weaker one.
The outbox: one database, one transaction
What if the whole transaction stayed inside one database? Add a second table to MySQL, a change log, and write to it in the same transaction as the film. Night Train's row and a note saying "film 8812 was created" commit together or not at all. A worker polls the table, sends each note to Elasticsearch, and removes it once every store has it.
This is the outbox pattern, and it works: if the worker crashes, the note simply waits. But Netflix runs services in many languages, and every one would need the same outbox library behaving the same way. And MySQL can't put a schema change, like adding a column, inside a transaction.
Read the database's own diary
Every serious database already keeps a change log. MySQL keeps one called the binlog, for its replicas; Postgres has its write-ahead log. Read through the replication interface, every committed insert, update and delete comes out, in commit order.
Change data capture (CDC) means reading that log and turning every committed change into an event. Ana's save becomes one event: create, film 8812, Night Train. Her app now writes to MySQL, and only MySQL. A rolled-back transaction is never a committed change, so search can't show a film that was never saved.
Netflix's Delta pipeline
Netflix built this as a platform called Delta:
- The Delta-Connector reads MySQL's log in real time.
- Each change goes onto a Kafka topic in Keystone, Netflix's streaming platform.
- A Delta application on Apache Flink picks up the event and enriches it by calling the deal, talent and vendor services.
- The finished document lands in Elasticsearch. Netflix describes the whole trip as nearly real-time.
The polling system is gone: search updates when something actually changes, and only then.
The log forgets
Transaction logs are trimmed after a while, so a brand-new search index needs a full copy of the table, a dump. Netflix tried Maxwell, SpinalTap, Yelp's MySQL Streamer and Debezium; for dumps, each had at least one deal-breaker. Some stop reading the log until the dump finishes. Others lock the table, and then Ana can't save anything.
DBLog, Netflix's framework, copies the table in chunks sorted by primary key, and fences each chunk with watermarks:
- Write a low watermark: an update to a tiny table of its own, which shows up in the log.
- Select the chunk into memory, then write a high watermark.
- If a row in the chunk was edited in the log between the two marks, drop it from the copy. The copy might be stale; the log can't be.
The log only pauses briefly, and no table is locked. The rule fits in one line: when the copy and the log disagree, the log wins.
Don't lose it in the pipe
Netflix's messaging had been tuned for availability over durability, so a lagging broker could take over as leader and events could be lost. For Delta, Netflix built a dedicated Kafka cluster: unclean leader election off, a replication factor of 3 (up from 2), acks = all with at least 2 in-sync replicas (up from 1), and brokers that reattach their old disk so catch-up drops from hours to minutes.
The gap that always closes
Delta is eventually consistent: for a moment after Ana saves, MySQL knows about Night Train and search doesn't yet. The difference from dual writes is that the gap closes on its own. If a screen must confirm a save instantly, read it from MySQL; search is for finding things. The outbox still earns its place for events the table can't express, like "deal approved", in one service and one language.
Takeaways
- No single write can land in two databases at once, so dual writes drift.
- XA blocks on failure, and Elasticsearch can't join it.
- The outbox works inside one service; CDC works for every service.
- DBLog copies a live table without locks: when the copy and the log disagree, the log wins.
- In an interview: one store owns the truth, and every other store is built from its log.
Watch the full breakdown: https://youtu.be/VNz614qTxAs
Sources: Netflix TechBlog, "Delta: A Data Synchronization and Enrichment Platform" and "DBLog: A Generic Change-Data-Capture Framework" (2019). Ana, Night Train and film 8812 are illustrative.
More breakdowns like this: https://systemdesignlab.in
Top comments (0)