DEV Community: Pankaj Batra

Kafka for Engineers Who've Only Used REST: What Actually Changes

Pankaj Batra — Sat, 01 Aug 2026 13:04:53 +0000

When I first read that Kafka was a "distributed event streaming platform," I nodded, closed the tab, and went back to writing REST endpoints.

It took me a while to realize the problem wasn't Kafka. It was that every explanation started with what Kafka is and none of them started with why it exists. If you already think about systems the way REST teaches you to, most Kafka tutorials feel like they're answering a question you didn't ask.

This post is for engineers who've built REST APIs for years and want to understand Kafka the way it actually clicks — not the API, not the config, but the shift in what a system is supposed to be.

The short version:

REST is about asking. Kafka is about telling.

Once that lands, everything else about Kafka stops feeling arbitrary.

The REST mental model has a hidden assumption

Every REST system assumes one thing that's so obvious it's almost invisible: the caller knows who to ask.

Your mobile app knows to call the orders API. The orders API knows to call the payments API. The payments API knows to call the fraud service. Someone, somewhere, decided who talks to whom, and that decision is baked into the code.

Consider an order being placed. In a typical REST setup:

The order service saves the order
Then it calls the inventory service to decrement stock
Then it calls the email service to send a confirmation
Then it calls the analytics service to log the event

Four services. Three couplings the order service has to know about. And when the business decides to add loyalty points on every order, you go back and edit the order service — again.

You didn't do anything wrong. This is what REST tells you to do. But look at what's really happening: the order service is responsible for knowing every other service that cares about orders. That responsibility grows every time the business grows.

That's the hidden assumption. In REST, the sender is responsible for knowing all the receivers.

What Kafka actually changes

Kafka flips the direction.

The order service no longer calls anyone. It writes a single fact to a log:

"order 1234 was placed."

And then it's done.

Inventory, email, analytics, loyalty — none of them are called. They read the log on their own schedule, notice a new order, and each one does whatever it does. The order service doesn't know they exist. It doesn't care.

That's the whole idea.

Everything else you'll ever read about Kafka — partitions, offsets, consumer groups, brokers, retention — exists to make that one shift work at scale.

If you take nothing else from this post, take this:

In REST, services call each other. In Kafka, services publish facts and read facts, and no one is called by name.

Five things that stop feeling weird once the shift lands

Once you accept the model, a bunch of things that look strange in Kafka tutorials suddenly make sense. Here are the five that tripped me up longest.

1. Kafka doesn't delete messages after they're read

In a REST world, a response is consumed. Once your client got the JSON back, the server moves on. Message queues work similarly — pop a message off, process it, gone.

Kafka doesn't do that. Messages sit in the log for as long as you tell it to keep them — days, weeks, forever if you want. The reason is baked into the model: a fact doesn't stop being true just because one service noticed it.

If analytics needs to reprocess the last week of orders, the events are still there. If you spin up a new service tomorrow that also cares about orders, it can read every order that ever happened.

The log isn't a queue. It's a record of everything that occurred, and consumers get to decide when and how often they read from it.

2. Consumers track their own position

In REST, the server knows who called it. Requests are tracked, sessions exist, the server is the coordinator.

In Kafka, the log has no idea who's reading. Each consumer is responsible for remembering where it is in the log — its offset. If a consumer crashes and restarts, it reads its last saved offset and picks up from there. If you want a second copy of a service to also process every event, it just starts from offset zero and reads independently.

This is why replay is trivial in Kafka and impossible in REST. Rewinding a REST call doesn't mean anything. Rewinding a Kafka offset means "read those events again."

3. You scale by adding partitions, not by adding servers per endpoint

REST scales by putting a load balancer in front of your service and adding more instances behind it. Each request goes to whichever instance is free.

Kafka scales by splitting the log itself. A topic isn't a single log — it's split into partitions, and consumers coordinate to divide those partitions among themselves. If you have four partitions and four consumers in a group, each consumer handles one partition. Add a fifth consumer and one of them will sit idle, because a partition is only ever read by one consumer in a group at a time.

This is a completely different scaling model, and it means the number of partitions you choose is a design decision that affects throughput for the life of the topic. It's not something you tune later without pain.

4. Order is only guaranteed within a partition

REST doesn't really care about order. Each request is independent, and if you need ordering, you build it in the application layer.

Kafka guarantees order — but only within a single partition. Across partitions, all bets are off. This means partitioning is a design choice you make based on what needs to stay ordered.

If you partition orders by customer ID, all events for one customer stay in order. If you partition by timestamp or randomly, you'll see events for the same customer arrive out of sequence. The moment you choose a partition key, you've decided what "in order" means for that topic.

Get this wrong and you'll ship a system where events arrive in an order that makes no sense to a human, and debugging it will feel like a nightmare until you notice what's happening.

5. There's no "the server returned an error"

In REST, errors have a shape. The server returns a 500, the client sees it, retries or gives up. Error handling lives in the request path.

In Kafka, there's no server responding to your call. There's a log. If a consumer fails to process a message, the failure happens on the consumer side, and the consumer has to decide what to do about it: retry, skip, log, or move the message to a dead-letter topic for someone to look at later.

This moves error handling out of the request path entirely. You stop asking "what should I return when this fails?" and start asking "what should this consumer do when it can't process what it just read?" That's a different question, and it takes a while to get comfortable with.

When REST is still the right answer

None of this makes REST obsolete. It's not supposed to.

REST is still the right choice when:

The caller needs an answer now — an authorization check, a price lookup, a validation
You're doing CRUD on a single resource
The integration is between two systems, not many
The caller genuinely does know who to ask, and that's not going to change

Kafka is the right choice when:

Multiple systems care about the same event
You want to be able to add new consumers later without touching the producer
Events have value beyond the moment they happen — audit, replay, analytics, machine learning
The coupling between services is starting to feel like the thing you spend most of your time maintaining

The most honest tell is this: if every new feature means adding another HTTP call from an existing service, you're probably ready for events. That pattern doesn't scale — not technically, but organizationally. Every new integration means editing an old service, which means testing it, which means owning it.

Events break that cycle. New consumer, no producer changes.

The shift, restated

Kafka doesn't replace REST. What it replaces is a specific habit — the habit of every service being responsible for knowing who to notify when something happens.

Once producers only publish facts, and consumers only read what they care about, systems get easier to change. Not easier to build the first time — Kafka has real operational cost, and the mental model takes effort. But easier to change, which is what actually matters after month three of any real system.

The hardest part of learning Kafka isn't the API. It's letting go of the assumption that someone always has to be waiting for the answer.

Once you accept that services can just publish facts into the world and stop caring who reads them, the rest of Kafka is just plumbing.

About the Author

I'm Pankaj Batra, a Software Engineer focused on Flutter, automation, and enterprise integrations.

I write about practical engineering: mobile architecture, workflow automation, APIs, event-driven systems, and lessons from production systems.

Connect

Portfolio: https://pankajbatra.vercel.app
LinkedIn: https://linkedin.com/in/pankaj-batra-0a294a205
GitHub: https://github.com/Pankaj0405

If this was useful, follow for more engineering notes.

Building a Server-Driven Survey Engine in Flutter Without a WebView

Pankaj Batra — Tue, 21 Jul 2026 12:29:44 +0000

This article was originally published on my Hashnode blog and is shared here for the Dev.to community.
Enterprise apps often need forms that change without an app-store release.

New questions appear. Visibility rules change. Validation logic evolves. Dropdown options come from remote services. Nested sections grow and shrink.

If every one of those rules lives only in Flutter code, the mobile app becomes a second form engine — expensive to build and easy to drift from the web.

This post is about a different approach: server-driven surveys rendered with native Flutter widgets, while the form logic stays outside the UI layer — and without wrapping everything in a WebView.

I won’t walk through proprietary implementation details. I’ll share the design thinking, the trade-offs, and the lessons that mattered most.

The Real Challenge Isn’t UI

Drawing text fields and dropdowns is the easy part.

The hard part is keeping behavior consistent:

Conditional visibility and enablement
Validation and error messaging
Calculated values and expressions
Multi-step navigation
Nested / repeating sections
Remote option lists

On the web, mature form libraries already solve much of this.

On mobile, teams often choose one of three paths:

Rebuild the logic in Dart
Embed a WebView and reuse the web form
Hybrid model — native UI, shared logic engine

We chose the third.

Why Avoid a WebView?

WebViews are fast to ship, but they come with costs:

Heavier memory and less “native” feel
Awkward integration with camera, files, signatures, and platform UX
Harder theming consistency with the rest of the app
Debugging that feels like maintaining two apps

For production mobile UX, native widgets are usually worth the extra design work.

The constraint we accepted:

The UI can be Flutter-native.

The meaning of the form (rules, state, validity) should not be reinvented in Flutter.

The Design Principle That Unlocked Everything

The most important decision was ownership:

One layer owns form logic. Flutter owns presentation and device I/O.

That means Flutter should:

Render what the logic layer says is currently visible
Capture user input
Handle device features (files, camera, signature, etc.)
Apply theming and accessibility

Flutter should not become the place where business rules are reimplemented “just for mobile.”

This single rule prevents a class of bugs where mobile and web disagree about whether a field should appear, whether a value is valid, or what an expression means.

Server-Driven UI, Practically

In this model, the app receives a form definition (JSON) from the backend.

That definition describes:

Structure (pages, panels, questions)
Behavior (conditions, validators, expressions)
Presentation hints (titles, layout flags, locales)

The mobile client’s job is to interpret and render, not to hardcode each form.

That’s what “server-driven” means here:

Product can iterate forms without waiting for a Flutter release for every schema change
Mobile stays aligned with the same logic model used elsewhere
Engineering effort shifts from “build every form” to “build a reliable renderer”

What Made This Hard (Without the Internals)

Hybrid systems fail in the seams — the places where two runtimes meet.

A few categories of problems showed up repeatedly:

1) Nested structures break naive assumptions

Top-level questions behave differently from questions nested inside repeating or dynamic containers.

If your mental model is “look up a field by name and update it,” nested forms will eventually surprise you.

Lesson: design for ownership and context early, not after the first nested bug.

2) Presentation flags are easy to misread

A flag that hides a panel title is not the same as hiding every child title.

Some question types also have different title semantics than ordinary inputs.

Lesson: match the upstream form engine’s meaning before inventing mobile-only rules.

3) Expression language has a strict shape

Operators and functions are not interchangeable.

A valid expression can look “almost right” and still fail because the syntax is interpreted differently than expected.

Lesson: when logic “doesn’t work,” verify expression form before assuming the mobile layer is broken.

4) Performance is a product of strategy, not just widgets

Large forms stress:

startup cost
communication between UI and logic layers
remote option loading on mobile networks

Fetching everything up front can feel simple and then become slow.

Lesson: measure separately — init cost, interaction cost, and network cost are different problems.

Trade-offs We Accepted

What you gain

Native Flutter UX
Stronger parity with an existing form logic ecosystem
Faster iteration on form content from the server
Cleaner separation between UI and rules

What you pay

Bridge complexity between runtimes
More careful debugging across layers
Constraints around what the logic runtime can do directly (especially without browser APIs)
Ongoing discipline so Flutter doesn’t quietly reabsorb business logic

This architecture is not “simpler.”

It is clearer — if the ownership boundaries stay honest.

Principles I’d Reuse on the Next Project

Choose a source of truth for form logic and protect it.
Prefer native rendering when UX quality matters.
Treat nested forms as a first-class design problem.
Don’t optimize everything on day one — instrument first.
Reproduce edge cases in demos (nested sections, remote options, expression rules).
Document decisions, not just features — future you will need the “why.”

Final Thoughts

Building a server-driven survey experience in Flutter is less about clever widgets and more about boundary design.

If Flutter owns too much logic, you rebuild a form engine.

If you lean only on WebView, you sacrifice native quality.

If you split responsibilities cleanly, you get something harder to build — and much easier to evolve.

The goal isn’t to expose every internal mechanism publicly.

The goal is to ship forms that stay correct as the business changes — without turning every schema update into a mobile release.

About the Author

I'm Pankaj Batra, a Software Engineer focused on Flutter, automation, and enterprise integrations.

I write about practical engineering: mobile architecture, workflow automation, APIs, and lessons from production systems.

Connect

Portfolio: https://pankajbatra.vercel.app
LinkedIn: https://linkedin.com/in/pankaj-batra-0a294a205
GitHub: https://github.com/Pankaj0405

If this was useful, follow for more engineering notes.

5 Power Automate Design Mistakes That Slow Down Your Flows (And How to Fix Them)

Pankaj Batra — Mon, 13 Jul 2026 08:53:27 +0000

This article was originally published on my Hashnode blog and is shared here for the Dev.to community.

Power Automate makes it incredibly easy to automate business processes.

With just a few actions, you can connect Microsoft Forms, SharePoint, Outlook, Teams, SQL Server, and hundreds of other services—often without writing a single line of code.

But as workflows grow, something interesting happens.

The automation still works...

...yet it becomes slower, harder to debug, and increasingly difficult to maintain.

In many cases, the issue isn't Power Automate itself.

It's the workflow design.

Over the past few months, while working on enterprise automation solutions, I've noticed the same design patterns appearing repeatedly. Individually they may seem harmless, but together they can significantly impact performance, readability, and long-term maintainability.

In this article, I'll share five common Power Automate design mistakes and practical ways to avoid them.

Why Flow Design Matters

Unlike traditional code, Power Automate workflows are visual.

That makes them incredibly easy to build.

Unfortunately, it also makes it easy to introduce unnecessary actions, nested loops, duplicated logic, and expensive operations without realizing their long-term impact.

As workflows become more complex, these small design decisions compound.

A well-designed workflow provides:

🚀 Faster execution
🐞 Easier debugging
📈 Better scalability
🧹 Simpler maintenance
👥 Easier collaboration with other developers

Let's look at five common mistakes.

1. Using "Apply to each" When You Don't Need It

One of the most common performance issues is unnecessary Apply to each loops.

Power Automate automatically creates these loops whenever an action returns an array, even when you're expecting only a single item.

Many developers simply leave the loop in place.

Example:

Get Items

↓

Apply to each

↓

Condition

↓

Update Item

While this works, every additional loop adds unnecessary processing and makes the flow harder to understand.

Better Approach

If your logic expects a single record, retrieve the first result instead of iterating through the collection.

Get Items

↓

First()

↓

Condition

↓

Update Item

Benefits

Less processing
Cleaner workflows
Faster execution
Easier debugging

2. Retrieving More Data Than Necessary

Another common mistake is retrieving an entire SharePoint list and filtering the results afterward.

Example:

Get Items

↓

5000 Records

↓

Filter Array

↓

Find One Record

Although functional, this forces Power Automate to process far more data than necessary.

Instead, use OData Filter Queries whenever possible.

Example:

Status eq 'Pending'

Title eq 'Vendor A'

This allows SharePoint to perform the filtering before returning the data.

Benefits

Less data transferred
Faster execution
Better scalability

3. Performing Duplicate Checks After Creating Records

Another design pattern I occasionally see looks like this:

Create Item

↓

Search for Duplicate

↓

Delete Duplicate

Although functional, it performs unnecessary work.

A cleaner approach is:

Search Existing Record

↓

Exists?

↓

Yes → Skip

↓

No → Create Item

Benefits

Prevents duplicate records
Reduces unnecessary writes
Improves performance
Keeps data clean

4. Writing Extremely Complex Expressions

Power Automate expressions are incredibly powerful.

Functions such as:

split()
concat()
if()
length()
coalesce()

allow sophisticated transformations.

The mistake is combining everything into one huge expression.

Instead of writing something like:

concat(
 split(...),
 if(...),
 body(...)
)

Break the logic into multiple Compose actions.

Compose – Format Date

↓

Compose – Customer Name

↓

Compose – Final Output

This approach makes workflows significantly easier to understand, debug, and maintain.

5. Never Looking at Flow History

Many developers only check whether the flow succeeded.

Flow History provides much more valuable information.

It can help identify:

Slow-running actions
Retry attempts
Connector latency
Failed expressions
Workflow bottlenecks

Reviewing Flow History regularly often reveals optimization opportunities that aren't obvious while building the workflow.

Best Practices I Follow

Whenever I build a Power Automate workflow, I try to follow a few simple principles.

✅ Keep workflows as linear as possible.

✅ Filter data before retrieving it.

✅ Avoid unnecessary loops.

✅ Break complex logic into smaller Compose actions.

✅ Prevent duplicate records before writing data.

✅ Review Flow History regularly.

Small improvements made early can dramatically improve maintainability as workflows evolve.

Final Thoughts

Power Automate makes automation accessible to everyone.

Designing workflows that remain clean, scalable, and performant as they grow is a different skill.

Most performance improvements don't come from changing connectors.

They come from making better design decisions.

A well-structured workflow is easier to understand, easier to debug, and much more reliable as business requirements evolve.

About the Author

Hi, I'm Pankaj Batra, a Software Engineer focused on building scalable software, automation solutions, and enterprise integrations.

I regularly write about:

⚡ Power Automate & Workflow Automation
📱 Flutter & Mobile Development
🐍 Python & Backend Development
🔗 REST APIs & System Integrations
🏗️ Software Architecture

Connect with me

🌐 Portfolio: https://pankajbatra.vercel.app
💼 LinkedIn: https://linkedin.com/in/pankaj-batra-0a294a205
💻 GitHub: https://github.com/Pankaj0405

If you found this article helpful, consider following me for more practical software engineering content.