When I first read that Kafka was a "distributed event streaming platform," I nodded, closed the tab, and went back to writing REST endpoints.
It took me a while to realize the problem wasn't Kafka. It was that every explanation started with what Kafka is and none of them started with why it exists. If you already think about systems the way REST teaches you to, most Kafka tutorials feel like they're answering a question you didn't ask.
This post is for engineers who've built REST APIs for years and want to understand Kafka the way it actually clicks — not the API, not the config, but the shift in what a system is supposed to be.
The short version:
REST is about asking. Kafka is about telling.
Once that lands, everything else about Kafka stops feeling arbitrary.
The REST mental model has a hidden assumption
Every REST system assumes one thing that's so obvious it's almost invisible: the caller knows who to ask.
Your mobile app knows to call the orders API. The orders API knows to call the payments API. The payments API knows to call the fraud service. Someone, somewhere, decided who talks to whom, and that decision is baked into the code.
Consider an order being placed. In a typical REST setup:
- The order service saves the order
- Then it calls the inventory service to decrement stock
- Then it calls the email service to send a confirmation
- Then it calls the analytics service to log the event
Four services. Three couplings the order service has to know about. And when the business decides to add loyalty points on every order, you go back and edit the order service — again.
You didn't do anything wrong. This is what REST tells you to do. But look at what's really happening: the order service is responsible for knowing every other service that cares about orders. That responsibility grows every time the business grows.
That's the hidden assumption. In REST, the sender is responsible for knowing all the receivers.
What Kafka actually changes
Kafka flips the direction.
The order service no longer calls anyone. It writes a single fact to a log:
"order 1234 was placed."
And then it's done.
Inventory, email, analytics, loyalty — none of them are called. They read the log on their own schedule, notice a new order, and each one does whatever it does. The order service doesn't know they exist. It doesn't care.
That's the whole idea.
Everything else you'll ever read about Kafka — partitions, offsets, consumer groups, brokers, retention — exists to make that one shift work at scale.
If you take nothing else from this post, take this:
In REST, services call each other. In Kafka, services publish facts and read facts, and no one is called by name.
Five things that stop feeling weird once the shift lands
Once you accept the model, a bunch of things that look strange in Kafka tutorials suddenly make sense. Here are the five that tripped me up longest.
1. Kafka doesn't delete messages after they're read
In a REST world, a response is consumed. Once your client got the JSON back, the server moves on. Message queues work similarly — pop a message off, process it, gone.
Kafka doesn't do that. Messages sit in the log for as long as you tell it to keep them — days, weeks, forever if you want. The reason is baked into the model: a fact doesn't stop being true just because one service noticed it.
If analytics needs to reprocess the last week of orders, the events are still there. If you spin up a new service tomorrow that also cares about orders, it can read every order that ever happened.
The log isn't a queue. It's a record of everything that occurred, and consumers get to decide when and how often they read from it.
2. Consumers track their own position
In REST, the server knows who called it. Requests are tracked, sessions exist, the server is the coordinator.
In Kafka, the log has no idea who's reading. Each consumer is responsible for remembering where it is in the log — its offset. If a consumer crashes and restarts, it reads its last saved offset and picks up from there. If you want a second copy of a service to also process every event, it just starts from offset zero and reads independently.
This is why replay is trivial in Kafka and impossible in REST. Rewinding a REST call doesn't mean anything. Rewinding a Kafka offset means "read those events again."
3. You scale by adding partitions, not by adding servers per endpoint
REST scales by putting a load balancer in front of your service and adding more instances behind it. Each request goes to whichever instance is free.
Kafka scales by splitting the log itself. A topic isn't a single log — it's split into partitions, and consumers coordinate to divide those partitions among themselves. If you have four partitions and four consumers in a group, each consumer handles one partition. Add a fifth consumer and one of them will sit idle, because a partition is only ever read by one consumer in a group at a time.
This is a completely different scaling model, and it means the number of partitions you choose is a design decision that affects throughput for the life of the topic. It's not something you tune later without pain.
4. Order is only guaranteed within a partition
REST doesn't really care about order. Each request is independent, and if you need ordering, you build it in the application layer.
Kafka guarantees order — but only within a single partition. Across partitions, all bets are off. This means partitioning is a design choice you make based on what needs to stay ordered.
If you partition orders by customer ID, all events for one customer stay in order. If you partition by timestamp or randomly, you'll see events for the same customer arrive out of sequence. The moment you choose a partition key, you've decided what "in order" means for that topic.
Get this wrong and you'll ship a system where events arrive in an order that makes no sense to a human, and debugging it will feel like a nightmare until you notice what's happening.
5. There's no "the server returned an error"
In REST, errors have a shape. The server returns a 500, the client sees it, retries or gives up. Error handling lives in the request path.
In Kafka, there's no server responding to your call. There's a log. If a consumer fails to process a message, the failure happens on the consumer side, and the consumer has to decide what to do about it: retry, skip, log, or move the message to a dead-letter topic for someone to look at later.
This moves error handling out of the request path entirely. You stop asking "what should I return when this fails?" and start asking "what should this consumer do when it can't process what it just read?" That's a different question, and it takes a while to get comfortable with.
When REST is still the right answer
None of this makes REST obsolete. It's not supposed to.
REST is still the right choice when:
- The caller needs an answer now — an authorization check, a price lookup, a validation
- You're doing CRUD on a single resource
- The integration is between two systems, not many
- The caller genuinely does know who to ask, and that's not going to change
Kafka is the right choice when:
- Multiple systems care about the same event
- You want to be able to add new consumers later without touching the producer
- Events have value beyond the moment they happen — audit, replay, analytics, machine learning
- The coupling between services is starting to feel like the thing you spend most of your time maintaining
The most honest tell is this: if every new feature means adding another HTTP call from an existing service, you're probably ready for events. That pattern doesn't scale — not technically, but organizationally. Every new integration means editing an old service, which means testing it, which means owning it.
Events break that cycle. New consumer, no producer changes.
The shift, restated
Kafka doesn't replace REST. What it replaces is a specific habit — the habit of every service being responsible for knowing who to notify when something happens.
Once producers only publish facts, and consumers only read what they care about, systems get easier to change. Not easier to build the first time — Kafka has real operational cost, and the mental model takes effort. But easier to change, which is what actually matters after month three of any real system.
The hardest part of learning Kafka isn't the API. It's letting go of the assumption that someone always has to be waiting for the answer.
Once you accept that services can just publish facts into the world and stop caring who reads them, the rest of Kafka is just plumbing.
About the Author
I'm Pankaj Batra, a Software Engineer focused on Flutter, automation, and enterprise integrations.
I write about practical engineering: mobile architecture, workflow automation, APIs, event-driven systems, and lessons from production systems.
Connect
- Portfolio: https://pankajbatra.vercel.app
- LinkedIn: https://linkedin.com/in/pankaj-batra-0a294a205
- GitHub: https://github.com/Pankaj0405
If this was useful, follow for more engineering notes.
Top comments (0)