Sale night, 8 p.m. Meera's fraud checker crashes for four minutes, and her sneaker shop keeps taking orders the whole time. Every order in those four minutes was supposed to be checked for fraud. None of them were. The events were simply gone.
Most engineers know the fix: put Kafka in the middle. But Cloudflare has machines in more than 335 cities, and it decided it couldn't run Kafka on them. So it built its own stream, called K2, and K2 makes your writes wait. Up to a second. On purpose.
This post walks through why. The animated version:
Direct calls drop events
Follow one order: order 4471, one pair of sneakers. It has two readers. The fraud checker needs it now; the analytics job wants it for tonight's sales report.
If the checkout calls each reader directly, that only works while both are up and both keep pace. On sale night, orders arrive faster than the fraud checker can work. Then it crashes, and there's nowhere for the event to wait, so it's dropped. Cloudflare's post names this exact trap: producers and consumers must keep up with each other, in volume and in time, or events get dropped.
A log in the middle
The answer is a log: an event stream that accepts every write, keeps it in order, and lets each reader catch up at its own pace. Order 4471 is appended to the end and gets a number, its offset. The fraud checker remembers the last offset it read. If it crashes, the log keeps filling; when it comes back, it starts from its offset, and nothing is lost.
Why Cloudflare couldn't run Kafka
Kafka is the classic way to run that log. It stores it on the local disks of a cluster of servers called brokers, and copies every piece to other brokers in case one dies. That works when brokers are big, long-lived machines on a reliable network.
Cloudflare's edge is the opposite. Its machines are spread across 335+ cities, each job gets a small slice of one, the slice can disappear, and the cities often talk over the public internet. So where do you keep a log that must never lose an event, when no machine you run is permanent?
Let the storage remember
You don't keep it on your machines at all. Cloudflare already ran a service whose whole job is to never lose data: R2, its object storage. Cloudflare says R2 is designed for eleven nines of durability, and it's strongly consistent: once a write succeeds, every reader sees it.
So K2 stores the log in R2. Copying data and agreeing on it, the hardest parts of a distributed log, become the storage layer's problem. K2 didn't solve replication. It handed it to a service that already had. The machines that handle events can come and go; the storage underneath doesn't care.
No append, so write whole files
Object storage has one big gap: you can't append to an object. So K2 never appends. Meera's order lands on a Cloudflare edge service first, which holds it in a memory buffer. The buffer waits a short moment for more orders, then writes everything it collected to R2 as one segment file.
Each event still needs a strictly increasing offset. K2 gets that from R2's atomic operations, writes that happen all at once or not at all, so there's no separate coordination service to run. The checkout then hears back: accepted, or failed with a flag saying whether a retry is worth it.
That short wait is the price. Cloudflare says it adds up to about one second at p99: 99 writes in 100 come back faster than that.
Slower, but whose problem?
Kafka writes to a broker's local disk in milliseconds; K2 waits for a batch, then a trip to object storage. But that second buys no cluster to run, no disks to fill, a bill that grows with the data you send, and cheap long history. For analytics, one second is nothing; for fraud, a second before an order ships is still fine.
Queue or stream?
Cloudflare also runs Queues, built for a different job: a queue tracks, retries and dead-letters every message on its own, while a stream moves batches for high volume, long retention and fan-out. A queue babysits every message. A stream moves the river.
Borrow a batch, don't own it
K2 doesn't push events at readers; each reader asks. The fraud model calls consume for up to 100 records and gets a batch with a 5-minute lease.
- Ack when done, and the batch is never sent again.
- Nack if something went wrong, and it's redelivered.
- Extend the lease if the work is slow.
Replay sale night: the fraud model crashes holding order 4471. It never acknowledges, so once the lease is up another worker takes the batch. Same four-minute crash, and this time every order is still checked.
One stream, many readers
Readers are grouped by subscriptions. Workers inside one subscription share the work, so when sale night gets busy, Meera adds three more fraud workers and they split the orders. Analytics gets a subscription of its own and sees every event at its own speed. In Cloudflare's example, the stream keeps events for seven days, so a reader that starts tomorrow can begin from the earliest one.
Every read is billed
K2 is free during the public beta on Workers Paid. After that, Cloudflare expects to charge $0.04 per GB produced, $0.04 per GB consumed, and $0.02 per GB-month retained. With two subscriptions reading every byte, each gigabyte is paid for three times: once in, twice out. Adding workers splits the bill; adding a subscription adds a whole new consume charge.
Kafka isn't dead, it's local
Today K2 has no message keys, Kafka clients can't connect yet, and every reader polls. Push delivery, an express tier and multi-GB/s streams are on the roadmap.
The interview answer: Kafka keeps its log on local disks, fast but you run the cluster. A log on object storage lets the storage keep it safe, cheap and elastic but slower. If your events need answers in milliseconds, run the disks. If they need to survive anything, cheaply, and a second doesn't hurt, let the bucket remember.
Takeaways
- A log in the middle lets producers and consumers stop keeping pace with each other.
- K2 hands replication to object storage and pays for it with up to ~1 s of write latency at p99.
- No append means batches: an edge buffer, then whole segment files.
- Leases with ack, nack and extend mean a crashed reader loses nothing.
- Fan-out is the stream's superpower, and on this price sheet, its fastest-growing line.
Watch the full breakdown: https://youtu.be/1Wd0FVKkRpE
Sources: Cloudflare, "Cloudflare K2: event streams on R2" (Micah Wylde, Marc Selwan), https://blog.cloudflare.com/cloudflare-k2-streams/ and InfoQ's coverage. Meera's shop and order 4471 are illustrative.
More breakdowns like this: https://systemdesignlab.in
Top comments (0)