DEV Community: Vinayak Gaur

Why Cloudflare's New Stream Waits a Second on Purpose: Event Streams Explained

Vinayak Gaur — Fri, 23 Oct 2026 14:15:09 +0000

Sale night, 8 p.m. Meera's fraud checker crashes for four minutes, and her sneaker shop keeps taking orders the whole time. Every order in those four minutes was supposed to be checked for fraud. None of them were. The events were simply gone.

Most engineers know the fix: put Kafka in the middle. But Cloudflare has machines in more than 335 cities, and it decided it couldn't run Kafka on them. So it built its own stream, called K2, and K2 makes your writes wait. Up to a second. On purpose.

This post walks through why. The animated version:

Direct calls drop events

Follow one order: order 4471, one pair of sneakers. It has two readers. The fraud checker needs it now; the analytics job wants it for tonight's sales report.

If the checkout calls each reader directly, that only works while both are up and both keep pace. On sale night, orders arrive faster than the fraud checker can work. Then it crashes, and there's nowhere for the event to wait, so it's dropped. Cloudflare's post names this exact trap: producers and consumers must keep up with each other, in volume and in time, or events get dropped.

A log in the middle

The answer is a log: an event stream that accepts every write, keeps it in order, and lets each reader catch up at its own pace. Order 4471 is appended to the end and gets a number, its offset. The fraud checker remembers the last offset it read. If it crashes, the log keeps filling; when it comes back, it starts from its offset, and nothing is lost.

Why Cloudflare couldn't run Kafka

Kafka is the classic way to run that log. It stores it on the local disks of a cluster of servers called brokers, and copies every piece to other brokers in case one dies. That works when brokers are big, long-lived machines on a reliable network.

Cloudflare's edge is the opposite. Its machines are spread across 335+ cities, each job gets a small slice of one, the slice can disappear, and the cities often talk over the public internet. So where do you keep a log that must never lose an event, when no machine you run is permanent?

Let the storage remember

You don't keep it on your machines at all. Cloudflare already ran a service whose whole job is to never lose data: R2, its object storage. Cloudflare says R2 is designed for eleven nines of durability, and it's strongly consistent: once a write succeeds, every reader sees it.

So K2 stores the log in R2. Copying data and agreeing on it, the hardest parts of a distributed log, become the storage layer's problem. K2 didn't solve replication. It handed it to a service that already had. The machines that handle events can come and go; the storage underneath doesn't care.

No append, so write whole files

Object storage has one big gap: you can't append to an object. So K2 never appends. Meera's order lands on a Cloudflare edge service first, which holds it in a memory buffer. The buffer waits a short moment for more orders, then writes everything it collected to R2 as one segment file.

Each event still needs a strictly increasing offset. K2 gets that from R2's atomic operations, writes that happen all at once or not at all, so there's no separate coordination service to run. The checkout then hears back: accepted, or failed with a flag saying whether a retry is worth it.

That short wait is the price. Cloudflare says it adds up to about one second at p99: 99 writes in 100 come back faster than that.

Slower, but whose problem?

Kafka writes to a broker's local disk in milliseconds; K2 waits for a batch, then a trip to object storage. But that second buys no cluster to run, no disks to fill, a bill that grows with the data you send, and cheap long history. For analytics, one second is nothing; for fraud, a second before an order ships is still fine.

Queue or stream?

Cloudflare also runs Queues, built for a different job: a queue tracks, retries and dead-letters every message on its own, while a stream moves batches for high volume, long retention and fan-out. A queue babysits every message. A stream moves the river.

Borrow a batch, don't own it

K2 doesn't push events at readers; each reader asks. The fraud model calls consume for up to 100 records and gets a batch with a 5-minute lease.

Ack when done, and the batch is never sent again.
Nack if something went wrong, and it's redelivered.
Extend the lease if the work is slow.

Replay sale night: the fraud model crashes holding order 4471. It never acknowledges, so once the lease is up another worker takes the batch. Same four-minute crash, and this time every order is still checked.

One stream, many readers

Readers are grouped by subscriptions. Workers inside one subscription share the work, so when sale night gets busy, Meera adds three more fraud workers and they split the orders. Analytics gets a subscription of its own and sees every event at its own speed. In Cloudflare's example, the stream keeps events for seven days, so a reader that starts tomorrow can begin from the earliest one.

Every read is billed

K2 is free during the public beta on Workers Paid. After that, Cloudflare expects to charge $0.04 per GB produced, $0.04 per GB consumed, and $0.02 per GB-month retained. With two subscriptions reading every byte, each gigabyte is paid for three times: once in, twice out. Adding workers splits the bill; adding a subscription adds a whole new consume charge.

Kafka isn't dead, it's local

Today K2 has no message keys, Kafka clients can't connect yet, and every reader polls. Push delivery, an express tier and multi-GB/s streams are on the roadmap.

The interview answer: Kafka keeps its log on local disks, fast but you run the cluster. A log on object storage lets the storage keep it safe, cheap and elastic but slower. If your events need answers in milliseconds, run the disks. If they need to survive anything, cheaply, and a second doesn't hurt, let the bucket remember.

Takeaways

A log in the middle lets producers and consumers stop keeping pace with each other.
K2 hands replication to object storage and pays for it with up to ~1 s of write latency at p99.
No append means batches: an edge buffer, then whole segment files.
Leases with ack, nack and extend mean a crashed reader loses nothing.
Fan-out is the stream's superpower, and on this price sheet, its fastest-growing line.

Watch the full breakdown: https://youtu.be/1Wd0FVKkRpE

Sources: Cloudflare, "Cloudflare K2: event streams on R2" (Micah Wylde, Marc Selwan), https://blog.cloudflare.com/cloudflare-k2-streams/ and InfoQ's coverage. Meera's shop and order 4471 are illustrative.

More breakdowns like this: https://systemdesignlab.in

How Airbnb Taught Computers What a Home Feels Like

Vinayak Gaur — Tue, 20 Oct 2026 14:15:01 +0000

Airbnb has millions of homes: treehouses, castles, city lofts, houseboats. When you look at one, Airbnb has to guess which others you'll like, not by price or city alone, but by feel. Their answer was to turn every home into a list of just 32 numbers. Those numbers lifted clicks on their recommendations by 21%, and now help rank Airbnb's search results in real time.

This post walks through how, from zero. The animated version:

Two doors to a booking

Guests find the home they book through two doors. Search: you type a city and dates, and a model ranks the results using more than a hundred signals. And the Similar Listings carousel on every listing page. Together, search ranking and similar listings drive 99% of Airbnb's booking conversions.

Both need the same thing: a way to say two homes are similar. Same city, same price, entire home versus private room: those are database columns. But guests also care about style, architecture and the feel of the place, and no column captures that. A cosy cabin and a glass penthouse can cost the same, sit in the same city, and appeal to completely different people.

What an embedding is

Imagine a map where every home is a dot, and homes that are alike sit close together. On a flat map each dot is two numbers, an x and a y. Now give the map 32 directions instead of two. Every home becomes a list of 32 numbers: its embedding. You can't picture 32 dimensions, but you can still measure how close two homes are, with cosine similarity.

Search sessions are sentences

Nobody types those numbers in. They're learned, with a trick borrowed from language models: words that keep showing up near each other tend to be related.

Airbnb treats a guest's trail of clicks as a sentence, and each listing ID as a word. A session is one uninterrupted run of clicks; a gap of more than 30 minutes starts a new one. When someone clicks three beach cottages in a row, that's a hint those cottages have something in common.

Learning by nudging

Start every home at a random spot on the map. Slide a window along each session. The home in the middle is the central listing; the homes clicked just before and after it, up to five on each side, are its positive context: pull them closer. Then grab a few homes completely at random and push them away. That's negative sampling.

Airbnb made two changes that fit its business:

The booking is the point. Clicks are cheap; a booking is what the guest actually wanted. In sessions that end in a booking, the booked home stays in the context of every step. They call it global context.
The same-city trap. Guests search within one market, so positives come from one city, but random negatives come from the whole world. The model learns "Paris is not Tokyo" and never learns which Paris flats differ from each other. The fix: also sample negatives from the same market.

Training at Airbnb scale

Airbnb trained this on more than 800 million search click sessions, producing an embedding for 4.5 million active listings. Why 32 numbers and not 300? Every vector has to live in the memory of the search machines for real-time comparisons, and 32 was the sweet spot between quality and RAM. A brand-new listing has no clicks yet, so it borrows: the average of the three geographically closest listings of the same type and price range.

Nobody told the model about location, price or style, but it learned all three. California listings clustered by embedding land in real map areas. Same type and price score higher cosine similarity. And starting at a treehouse, its nearest neighbours are treehouses; castles find castles.

Similar Listings, rebuilt

The carousel became pure geometry: take the home's vector, compare it with every home in the same market that's free on your dates, and keep the 12 closest, a k-nearest-neighbours search with k = 12. In an A/B test, the carousel's click-through rate rose 21%, and 4.9% more guests found the home they booked right there.

Search that learns while you browse

For every user, Airbnb keeps two short lists, updated in real time with Kafka: every listing you clicked in the last two weeks, and every listing you skipped (ranked high, but you clicked something lower down). Each search gives every candidate two new scores, embedding click similarity and embedding skip similarity, both features in the ranking model. High click similarity pushes a home up; high skip similarity pushes it down. It launched in summer 2017.

Takeaways

Learn from behaviour, not labels.
Shape the training around the outcome you care about: the booking.
Keep vectors small enough to use in real time.

The same recipe works for almost any marketplace: products, songs, videos, jobs.

Watch the full animated breakdown: https://youtu.be/-NK9hxCb4Sk

Source: Mihajlo Grbovic, Haibin Cheng, Qing Zhang, Lynn Yang, Phillippe Siclait and Matt Jones, "Listing Embeddings for Similar Listing Recommendations and Real-time Personalization in Search Ranking", The Airbnb Tech Blog (March 2018).

Practice designing systems like this yourself at https://systemdesignlab.in

How Uber Eats Halved Search Time With No Big Fix

Vinayak Gaur — Fri, 16 Oct 2026 14:15:01 +0000

Uber Eats search had gotten so slow that Uber's own engineers called it untenable. Then they cut the wait in half. The surprising part: no single change saved more than about 200 milliseconds. Some saved twenty.

In September 2026, Uber Engineering published every change behind it. This post follows one search through the whole pipeline and pulls out the five ideas behind the fixes, ideas that work on any system you build. The animated version:

Eight p.m., sushi: the six stages of a search

Here's the search we'll follow. It's 8 p.m. Maya opens Uber Eats and types sushi.

Query understanding works out what she means: sushi is a dish, not the name of a restaurant.
Retrieval searches a catalog of a few billion items for every restaurant and roll that could match.
Hydration fetches the facts about each candidate: the signals a ranking model needs to judge it.
Ranking scores the candidates so the best few rise to the top.
Presentation turns the winners into cards, with prices and photos.
Finally, the cards travel back to Maya's phone and draw on screen.

Six stages, one after another, and Maya waits through all of them.

Idea 1: measure what the user sees

The usual answer to "how slow is it?" is back-end response time: how long the server takes to reply. Uber changed the clock. Their metric is above the fold: from the moment Maya taps search to the moment the first screen of results is fully drawn, images included.

That clock counts things the server timer never saw: sending results over the network, building the cards, loading the photos. And it pointed straight at the first fix. The server was building every result, one card at a time, before sending any of them.

Now it sends a small first page and keeps the rest cached on the server for when Maya scrolls, and the cards render at the same time instead of one by one. Together, those two changes made the first screen appear more than 200 ms sooner, the biggest single win.

Idea 2: delete work before you speed it up

Retrieval was running several search strategies side by side and merging what they found. Some matched by meaning, some matched plain words. Uber checked what each one actually added. Several broad word-matching strategies were slow and contributed little the others hadn't already found. Removing them saved about 120 ms, by deleting code. Duplicate chain stores are now removed earlier too, so every later stage carries fewer candidates.

The catalog itself got the same treatment. Many of its few billion items are the same product sold by different stores, and every listing used to be its own item with its own data to fetch. Grouped by the product underneath, the catalog is roughly 100 times smaller, and fetching data once per product cut lookups by more than 100 times, about 50 ms.

Idea 3: don't wait on what you don't need

Hydration used to be one big step: fetch everything about every candidate, then rank. But ranking only needs the signals its model scores. The price of Maya's salmon roll, the promotion, whether it's in stock: those only matter when the card is drawn.

So Uber split hydration in two. Ranking hydration fetches only what scoring needs; presentation hydration fetches the display details at the same time, in parallel. Ranking no longer waits for prices it never reads, which was worth more than 100 ms. Removing false dependencies between steps, where one step waited on another it didn't actually need, saved about 35 ms more.

Idea 4: hedge the tail

Hydration also fans out: one search asks many machines at once, each holding one slice of the data, called a shard. The answer is only as fast as the slowest shard. Most shards answer quickly, but now and then one stalls for a moment, and with enough shards per search, one of them is often stalling. That's tail latency: a rare slow case, multiplied by fan-out, becomes the common case.

Uber's fix is request hedging. If a shard hasn't answered within a time limit, send the same request to another copy and take whichever answer comes back first. It works because the stalls were isolated, one shard at a time, and it saved 40 ms.

It's safe here because hydration only reads. Send a payment twice and you charge twice: writes need idempotency keys.

Idea 5: lay data out the way it's read

Every search runs a small ad auction with hundreds of bids to score. The bids were stored row by row, with the same shared details repeated in every row. But scoring reads one field across every bid, like running down one column of a spreadsheet, so Uber stored the bids as columns and moved campaign and pacing data into the ad service's own memory. Ad selection got about 30 ms faster, bid preparation about 110 ms, and cutting needless format conversions saved 20 ms more.

The plumbing had the same lessons:

Results encoded one after another left most CPU cores idle; encoding in parallel saved more than 50 ms.
Embeddings rounded to five decimal places and packed compactly shrank 46%, and their database queries got twice as fast.
The service mesh capped requests per connection, so busy services queued silently; parallel connections removed up to 53 ms.
In some Go services, garbage collection was eating more than 40% of the CPU. Plain values instead of pointers let data live on the stack and disappear when a function returns.

Uber even built an AI agent loop that profiles live latency, drafts fixes, opens pull requests and benchmarks them. In Uber's words, what it finds is "individually uninspiring but compound meaningfully at scale."

Takeaways

Measure what the user sees, not what your server times.
Delete work before you make it faster.
Draw your request's dependency graph; anything that doesn't need to wait, shouldn't.
Hedge the tail on reads; never on payments.
Lay data out the way it's read, and keep the hot part in memory.

None of those fixes was big on its own. Together, Maya's sushi shows up in half the time.

Watch the full animated breakdown: https://youtu.be/QlzyLS6ll24

Sources: Uber Engineering, "Halving the Time: How Uber Eats Rebuilt Its Search Pipeline" (Sept 2026), https://www.uber.com/us/en/blog/uber-eats-search-pipeline/ · InfoQ, https://www.infoq.com/news/2026/10/uber-eats-search-latency/

Practice designing systems like this yourself at https://systemdesignlab.in

How 5 Servers Agree on One Truth: Raft Consensus Explained

Vinayak Gaur — Tue, 13 Oct 2026 14:15:01 +0000

The server in charge just crashed. In the Raft authors' own tests, with fixed election timers, the other four servers argued for more than ten seconds about who should take over. Then they added five milliseconds of randomness, and the median wait dropped to 287 milliseconds.

Raft is the algorithm that lets a group of machines agree on one truth, even when two of five of them die. It runs inside etcd, the key-value store Kubernetes keeps its cluster data in. This post walks through how it works, following one example the whole way. The animated version:

One copy isn't enough

A ticket shop is selling the last seat of a sold-out concert: seat 7A. If the shop keeps its bookings on one server, life is simple, until that server dies and the shop closes.

So you keep copies on several servers. Now any one of them can die and the others carry on. But copies bring a new problem: Priya books seat 7A on one server, and a second later Leo books the same seat on another. Both servers said yes. The seat is sold twice. The copies have to agree.

Same commands, same order

Raft's answer starts with a log. Every change is written down as a numbered command, like "seat 7A goes to Priya". Each server keeps its own copy of the log and plays the commands into its own copy of the data, one by one. If every server has the same commands in the same order, every server ends up in exactly the same state. The paper calls this a replicated state machine.

The whole job of consensus is to keep those logs identical while servers crash, restart and lose messages. The trick is majorities: a change only counts once more than half the servers have it. With five servers, that's three, so five servers can lose any two and keep working. That's why five is the typical size.

Who's in charge

Raft elects one leader. Every write goes through it, and log entries only flow from the leader to the others. Each server is a leader, a follower or a candidate. Followers are passive: they answer, unless the leader goes quiet.

The leader sends a steady heartbeat to every follower. Time is split into numbered terms, each starting with an election and having at most one leader. A server that sees a higher term catches up at once, and an old leader that sees one steps down.

An election

Every follower runs an election timer, reset by each heartbeat. When server one, the leader for term 4, dies, the heartbeats stop. Server three's timer runs out first. It moves to term 5, becomes a candidate, votes for itself and asks everyone else for their vote.

Each server gets one vote per term, first come, first served. Servers two and four say yes: three of five, a majority. Server three leads term 5 and starts sending heartbeats. Because every server votes once per term and a win needs three votes, two leaders can never win the same term.

Split votes and the random timer

If two servers time out at the same moment, they can split the vote and nobody reaches three. With identical timers, they collide again and again. That's why fixed timeouts took over ten seconds. The fix: each server picks its timeout at random, from a range like 150–300 ms. One server almost always wakes first and wins before anyone else wakes up.

The timeout needs to be much longer than one round of messages (about 0.5 to 20 ms) and much shorter than the months between server failures.

Committing a write

Priya taps buy. Her request goes to the leader (a follower would just redirect her). The leader appends index 7, term 5: 7A → Priya to its log and sends it to all four followers at once in an AppendEntries message. The same message with no entries is the heartbeat.

Servers two and four write it and say yes. With the leader's own copy, that's three of five, so the entry is committed: it survives even if the leader dies a second later. Priya sees her ticket. A slow server five holds nobody up; the leader keeps retrying until it catches up.

When logs disagree

Crashes leave messes: missing entries, or extra entries that were never committed. So every AppendEntries also carries the index and term of the entry just before the new ones. If the follower doesn't have that exact entry, it refuses. The leader steps back one entry and tries again until the logs agree, and the follower replaces everything after that point with the leader's entries. The leader never changes its own log.

Not just anyone can lead

Imagine the leader crashed the instant Priya's booking committed, and server five, which never got it, asks for votes. If it won, Priya's booking would be wiped out. So Raft adds one rule: a server refuses its vote if its own log is more up to date than the candidate's. Compare the last entries: the later term wins; if the terms match, the longer log wins.

Priya's entry is on three servers, and any candidate needs three votes. Two groups of three out of five always share at least one server, and that server refuses to vote for a log without the booking. So every new leader already has every committed entry.

The leader that didn't know

If the network splits with the leader on the small side, the big side elects a new leader for term 6. For a moment two servers think they're leader, but the old one can only reach two of five, so nothing it writes commits. When the network heals, it sees term 6 and steps down, and its uncommitted entry is overwritten. Bad timing can slow Raft down, but it can't make two servers disagree.

What it costs, and where it runs

Every write waits for a round trip to a majority, and each follower writes the entry to disk before it answers. Every write goes through one leader. Six servers survive no more failures than five. If three of five go down, the cluster stops taking writes rather than risk two different answers. And Raft assumes servers crash or go silent; it doesn't defend against a server that lies on purpose.

Raft was designed to be understandable: in a study of 43 students who learned both, 33 scored better on Raft than on Paxos. That clarity helped it spread: etcd, HashiCorp's Consul and TiKV all run on it.

Takeaways

Same commands in the same order give the same state on every server.
A write counts once a majority holds it: 3 of 5 survives two failures; use odd cluster sizes.
Elect one leader on a random timer; one vote per server per term.
Only an up-to-date server can win, so every leader has every committed entry.
Safety never depends on timing.

Watch the full animated breakdown: https://youtu.be/J_qzUMR3JbA

Sources: Diego Ongaro and John Ousterhout, In Search of an Understandable Consensus Algorithm (Extended Version), https://raft.github.io/raft.pdf · https://raft.github.io/ · etcd FAQ · Kubernetes architecture docs · HashiCorp Consul consensus docs.

Practice designing systems like this yourself at https://systemdesignlab.in

How LinkedIn Rebuilt Its Feed: FollowFeed Explained

Vinayak Gaur — Fri, 09 Oct 2026 14:49:59 +0000

Every time you open LinkedIn, your feed is built from scratch. For 400 million members, in about a tenth of a second.

For years that feed ran on a search engine built for something else, until the cost stopped making sense. A team spent twenty months replacing it. The new system, FollowFeed, is five times faster, holds twenty times more data, and runs on half the servers. The trick behind all three numbers is that your feed is never stored at all. The animated version:

A feed is a question

You follow people and companies. Each one produces content: shares, likes, comments. Call each of those streams a timeline: a list of what that entity did, newest first.

Your feed isn't a thing stored somewhere. It's an answer to a question: out of every timeline I follow, what are the best few dozen items right now? LinkedIn's index holds hundreds of millions of timelines. Answering that question, fast, is the whole problem.

The system they outgrew

The old feed ran on Sensei, a generic search platform built on Lucene. Lucene's core trick is the inverted index: a map from every term to the documents containing it. It's fast if you keep it in memory, and Sensei did. That's the catch.

More content meant more memory, which meant more machines, and membership doubled in three years, from 200 million to 400 million. Sensei was also three projects stitched together (Bobo, Norbert and Zoie), and once LinkedIn built a new search engine, Galene, Sensei stopped being anyone's priority. The maintenance bill stayed.

Push or pull

When someone you follow posts, do you push it or pull it?

Push (fan-out-on-write): copy the post into the pre-built feed of every follower, immediately. Reads are trivially cheap, because your feed is already sitting there, finished.
Pull (fan-out-on-read): store the post once, in its author's timeline, and assemble the feed when someone asks.

Push is fine until you count the copies: LinkedIn measured 62 times more data than the pull design. And a pre-built feed is a frozen ranking. Change the relevance model and you have to re-rank everyone's stored feed.

So FollowFeed pulls. Feeds are built at query time, every time.

Take the code to the data

Pulling has a cost: the work now happens while the member is waiting. Personalizing a feed isn't one lookup. The model wants thousands of candidate records, and the naive version ships all of them across the network to a scoring service. The network is the slow part.

So FollowFeed inverts it. Scoring runs on the index nodes, the machines already holding the data. Each node filters and scores its own records, and returns only the handful that survived.

Timelines on disk

If you're not holding everything in memory, you need storage that's fast on disk. FollowFeed uses RocksDB, an embeddable key-value store built on LevelDB and tuned for SSDs. LinkedIn runs Java and RocksDB is C++, so the team wrote the bindings and contributed them back to open source.

A timeline isn't stored as one key per post (that would be one disk lookup per post). It's a linked list of blobs. Each blob holds a chunk of serialized records plus metadata: the timestamp range it covers and how many records are inside. Records use Avro, so each node keeps a schema registry inside RocksDB.

Seven hundred and twenty

Hundreds of millions of timelines don't fit on one machine. The naive choice is one partition per node, but then adding a node means splitting partitions and moving the pieces.

FollowFeed over-partitions instead: the data is cut into 720 partitions, permanently. The number is chosen for its divisors. 720 divides evenly by 40, 45, 60, 80 and 360, so the cluster can be any of those sizes and every node still gets whole partitions. You move partitions; you never split one. Rebalancing becomes bookkeeping instead of surgery.

Two caches

There's a cache in front of RocksDB, and its shape is the interesting part. Some timeline keys map to long lists of records, others to nothing at all. Size a cache by entry count, and a few huge entries blow the memory budget.

So FollowFeed runs two: a fat cache for keys with records, measured in total records (its weight), and a skinny cache for keys with no records, capped by key count. Both are Guava caches, split into sub-caches so threads don't queue on one lock.

One request, end to end

A request arrives at a broker with the entities you follow, your filter criteria and how many records to return. The broker finds the right index nodes through D2, LinkedIn's name service and load balancer (backed by ZooKeeper), and fans the query out in parallel. Each node pulls candidates from cache or RocksDB, filters them with a small SQL-like grammar (privacy, geography, content type), scores and ranks what's left, and returns its best records. The broker merges, drops duplicates, applies diversity filtering, and returns the top few.

Scoring is where the latency budget goes. The team started with a general-purpose scoring library that interprets the model at runtime, then generated Java code for the model instead: 50 microseconds per record at p99, fifteen times faster than where they started.

Keeping it fed, everywhere

Everything that happens on LinkedIn lands on a Kafka stream, but that stream is partitioned by Kafka's rules, not FollowFeed's 720. A service in the middle, the Partitioner, republishes the firehose onto topics matching the index nodes' partition ranges. Adding a replica means starting a node and pointing it at the same topics. Every timeline is replicated to every datacenter, and each datacenter's serving cluster is sized to the read traffic it actually takes.

Fast at the tail

Broker calls are asynchronous, so one slow node can't hold the others hostage, and if a node is slow the broker fires the same request at up to three replicas and takes the first answer. Inside each node: read-copy-update, fine-grained locking, and tuned garbage collection, caches and networking.

What it bought them

Mobile feed queries at p99: about 140 ms, five times faster than Sensei
Page load at p90: 150 ms better
20 times more data in the index, on half the hardware

Takeaways

A feed can be a query, not a table. Pull avoids 62× duplication and frozen rankings.
When the network is the bottleneck, move the computation to the data.
Over-partition with a number that has many divisors, so rebalancing never splits a partition.
Size caches by what actually costs memory, not by entry count.

Watch the full breakdown: https://youtu.be/WlEdcWpXn7M

Source: FollowFeed: LinkedIn's Feed Made Faster and Smarter, LinkedIn Engineering.

Practice designing systems like this yourself at systemdesignlab.in.

Why Your Payment Got Charged Twice: Idempotency Keys Explained

Vinayak Gaur — Tue, 06 Oct 2026 14:15:06 +0000

Maya taps Pay for a $20 ride. Ten seconds later: request timed out. The app retries, and Maya gets charged $40.

The retry went through. So did the first attempt. The first request never failed: it worked perfectly, and the answer just never made it back.

Stripe's fix for this is one extra header on the request, a random string called an idempotency key. This post walks through how it works and how to build it yourself. The animated version:

Three ways to fail

Stripe's engineers start from a blunt fact: the network is unreliable. A call from a client to a server can fail in three places:

Before the request reaches the server. Nothing happened, so a retry is safe.
Halfway through. The server started working and something broke. Maybe the charge ran, maybe it didn't.
After the work is done. The money moved, then the connection dropped before the response got back.

From the client's side, all three look exactly the same: a timeout and no answer.

Retry, or give up?

Both obvious options are wrong. Never retry, and case 1 gives Maya a free ride. Always retry, and case 3 charges her twice. Waiting longer doesn't help you choose: a slow server and a dead connection look identical from the outside.

What you want is a retry that's safe no matter which case happened.

What "idempotent" means

An operation is idempotent if doing it twice has the same effect as doing it once. A GET only reads. DELETE the same thing twice and it's just as deleted. Stripe's article uses a PUT to a DNS provider: "set this record to this value" can be sent twice and the record ends up the same.

A charge isn't like that. A POST /v1/charges with amount=2000 (that's $20 in cents) says make a new charge, not which charge. The server can't tell a retry from a new purchase. Send it twice and you get two charge objects and $40 gone.

The idempotency key

The server needs a name for the attempt. That's the idempotency key: the client makes up a unique string and sends it in an Idempotency-Key header.

Stripe suggests a V4 UUID, or any string random enough that two clients never pick the same one. Keys can be up to 255 characters.
The crucial part: the key is made once, before the first attempt, and every retry of that payment reuses it.

Think of a coat check: show ticket 42 twice and you get the same coat back, not a second coat.

What the server remembers

Each key points to a saved result. Stripe saves the first response for every key: its status code and body.

First time a key arrives: no entry, so the server runs the charge and stores the response against the key.
Second time: the entry is there, so it charges nothing and returns the saved response.

It saves the result whether the request succeeded or failed. If the first attempt returned a 500, the retry gets that same 500. A key doesn't mean "try again from scratch". It means "tell me what happened to this request".

The hard case: crashing halfway

Replay the three failures with a key. Case 1: the server never saw the key, so the retry runs normally. Case 3: the retry finds the key and gets the saved reply. Case 2 is the hard one: the server died partway through, so it has to remember how far it got.

Brandur Leach, who wrote Stripe's article, sketches a Postgres design where each key's row stores a recovery point: started, ride_created, charge_created, finished. A retry picks up from there, and each step between calls to other services commits all at once or not at all.

When his server calls Stripe, it passes its own idempotency key along, so even a crash in the middle of the charge can't charge Maya twice.

Edge cases that bite

Two retries at once. Both look up the key and find no finished result. So the key's row gets a lock: the first request stamps a locked_at time and does the work, and the second gets a 409 Conflict. Nothing is saved or charged, and the client retries later.

Same key, different request. If a bug reuses Maya's key for a $50 charge, Stripe compares it with the original and rejects it. Keys are also scoped per user, and should never be built from personal data like an email address.

How long to remember. Stripe lets keys be pruned once they're at least 24 hours old. For your own system Brandur suggests about 72 hours, so a bad Friday deploy still has its keys through the weekend. A background "reaper" deletes old keys, and a "completer" pushes abandoned requests to the finish.

Back off, and add jitter

Keys make a retry safe. They don't make it polite. If the server goes down and thousands of clients retry instantly, they hammer the thing that needs to recover.

Stripe's answer is exponential backoff: wait 2^n, where n is the number of failures so far. But if an outage hits thousands of clients in the same second, they all back off on the same schedule and come back in synchronized waves: the thundering herd.

The fix is jitter: add a random amount to every wait, and the waves smear into a steady trickle. Stripe's own client libraries can do both for you, and attach an idempotency key to every request they retry.

The whole recipe

Handle failures consistently: clients retry.
Handle failures safely: servers are idempotent, using keys.
Handle failures responsibly: back off exponentially and add jitter.

At Stripe, every POST accepts an idempotency key. GET and DELETE don't need one: they're idempotent by definition.

Watch the full animated breakdown: https://youtu.be/OE_RnizyyEM

Sources: Stripe's Designing robust and predictable APIs with idempotency, the Stripe API docs on idempotent requests, and Brandur Leach's Implementing Stripe-like idempotency keys in Postgres. Maya, the charge ids and timings in the images are illustrative.

Practice designing systems like this yourself at systemdesignlab.in.

How Databases Survive Being Unplugged: Write-Ahead Logs Explained

Vinayak Gaur — Fri, 02 Oct 2026 18:10:43 +0000

By default, Postgres can leave your changed rows out of its table files for five minutes or more. Commit a payment, pull the power cable one second later, and when the server comes back, the payment is still there.

This post walks through how that works: the write-ahead log (WAL). It's the same idea in Postgres, SQLite, MySQL and almost every serious database. If you prefer video, here's the animated version:

The unplug test

We'll use one example throughout. Asha sends Ravi ₹500. To the database that's two changes: subtract 500 from Asha's balance, add 500 to Ravi's.

Those changes happen in memory first, because memory is fast. But memory is volatile. Worse, imagine the power dies after the first change reaches disk but before the second one does: Asha has lost ₹500 and Ravi never got it.

So a database has to promise two things:

All or nothing. A transfer happens completely or not at all.
Durable. Once it says "committed", the transfer survives a crash.

Why not just write the tables?

The obvious fix is to write every changed page into the table files on every commit, and wait for the disk. It's slow and it still isn't safe.

Postgres stores tables in 8 KB pages. Asha's row and Ravi's row live on different pages, and every index on those tables has its own pages too. One small transfer can touch four or five pages scattered across the disk. That's random I/O, on every commit. And if the power dies halfway through that list, some pages are new, some are old, and nothing records which is which.

The one rule: log first, data later

Write-ahead logging replaces all of that with one rule:

Before you change a data file, write down what you're about to change in a log, and make sure the log is on permanent storage.

For our transfer the log gets three short records: asha -500, ravi +500, and a commit record. The moment the commit record is on disk, the transfer is durable. The table pages can stay in memory and be written whenever it's convenient.

Inside the log

In Postgres the log is a series of files in pg_wal, 16 MB each by default. Every record gets a log sequence number (LSN), which is just a byte position in the log, and it only goes up.

Each data page remembers the LSN of the last change applied to it. That gives Postgres one more rule: a dirty page (changed in memory) can't be written to disk until the log is flushed at least up to that page's LSN. The log is always ahead of the data.

Why writing twice is faster

This is the part that surprises people. Writing everything twice (once to the log, later to the tables) beats forcing every table page to disk on every commit.

The table writes were scattered. The log is written sequentially: every new record goes on the end. So at commit time there's only one file to sync, the log. The scattered page writes still happen, but later, in the background, batched, where no user is waiting on them.

fsync: the real promise

Calling write() doesn't mean the data is on disk. The operating system usually just copies it into its page cache. To force it down to storage, the database calls fsync, which blocks until the OS says the data is on the device.

So a commit really means: append the commit record, fsync the log, and only then tell the client "done".

There are more caches below the OS: disk controllers and the drives themselves. The Postgres docs warn that many consumer drives and SSDs keep writes in a cache that doesn't survive a power failure, and some ignore flush commands. The log is only as honest as the disk under it.

Recovery: replay the log

When the unplugged server boots, it runs crash recovery before accepting a single query. It reads a small control file, pg_control, which points at the last checkpoint, then scans the log forward and redoes every change it finds. If a page's LSN is already past a record, that change is skipped, so nothing is applied twice.

Asha's transfer has a commit record on disk, so both halves are replayed and Ravi gets his ₹500.

If the power had died before the commit record reached the log, the changes might even be replayed onto the pages, but Postgres treats the transaction as aborted, so nobody ever sees them. All or nothing.

Checkpoints

If the log grew forever, recovery would replay all of it and the disk would fill up. So Postgres runs checkpoints: flush every dirty page to the table files, then write a checkpoint record. Everything before it is safely in the tables, recovery can start there, and old log files can be recycled.

By default a checkpoint starts every 5 minutes, or sooner if the log is about to pass 1 GB. That's where the "five minutes" at the top comes from. Checkpoint more often and recovery is faster; less often and you do less write work in normal running.

Torn pages

There's one crash the log alone can't fix. An 8 KB page is sixteen 512-byte sector writes. Lose power in the middle and you get a torn page: half new, half old, neither version. The log's small change records can't repair that.

Postgres handles it with full page writes: the first time a page is modified after a checkpoint, the whole page image goes into the log. Recovery restores that clean copy first, then replays the later changes on top.

Group commit

fsync is the expensive step. When a thousand users commit at once, they don't need a thousand of them: their commit records sit next to each other at the end of the log, and one fsync makes all of them durable. The busier the system, the more commits ride on each flush.

Trading durability for speed

Some workloads don't need every commit to wait for the disk. With synchronous_commit = off, the client is told "done" before the log is flushed, and the WAL writer flushes it shortly after. It wakes every 200 ms by default, and the worst-case window of lost commits is three times that, about 600 ms.

A crash can lose those last few commits, but it can't corrupt the database. Recovery still replays a clean, consistent log. You choose per transaction: Asha's payment keeps synchronous commit, a page-view counter can go async.

The same log, everywhere

SQLite has a WAL mode too: changes are appended to a -wal file, a commit is a commit record, readers don't block the writer, and it checkpoints automatically at 1,000 pages (about 4 MB). And in Postgres, the same log powers point-in-time recovery and streaming replicas.

Three takeaways

A commit is durable when its log record is on disk, not when the tables change.
Sequential log writes are what make that cheap.
Checkpoints keep recovery short.

Watch the full animated breakdown: https://youtu.be/mhpTQN13bGI

Sources: the PostgreSQL docs on WAL, reliability, WAL configuration and async commit, and SQLite's WAL page. Balances and LSNs in the images are illustrative.

Practice designing systems like this yourself at systemdesignlab.in.