TL;DR: We ran Kafka in a payments company for 3 years. Most of our 180 topics were glorified queues. We moved to SNS/SQS in 5 months. Here's the real story, including the parts I got wrong.
I used to think running your own Kafka cluster was a sign of engineering maturity. Like your team had graduated to "real" distributed systems. Then I spent two years operating one inside a payments company with a six-person platform team and realized we hadn't graduated to anything — we'd just volunteered for operational overhead we didn't need.
The moment it became undeniable was a Friday morning call with our acquiring bank's technical team, where I had to explain how a slow disk caused 12,000 duplicate transactions in their settlement file. Me on mute half the time, trying to figure out how to say "our messaging infrastructure isn't operated at the level your business requires" without actually saying that. That's when I decided Kafka had to go.
What I Walked Into
Kafka was already there when I joined the payments platform team. I tried to find out why it was chosen. Got nowhere. There was a Confluence page from 2019, "Event-Driven Architecture — Proposed," authored by someone who'd left a year and a half before I showed up. Linked to a dead Slack channel. The page had four views. I'm pretty sure two were me.
The setup: 9 brokers on r5.2xlarge, running Kafka 2.7 (two minor versions behind because upgrading meant regression-testing consumers and nobody wanted to own that risk). Three t3.medium ZooKeeper nodes. A Schema Registry instance. Somewhere around 170-180 topics, I'd have to check my spreadsheet for the exact count. Retention was 7 days on most, 30 on a handful of others. Never found documentation on why some got 30 days. I suspect whoever set it up just picked different numbers for different topics based on vibes.
Everything else in our stack was AWS-native. ECS, DynamoDB, S3, CloudWatch, IAM. And then Kafka, running on EC2 with its own access model, its own monitoring (JMX exporters into Grafana — I think three people looked at those dashboards regularly, and one of them was me), its own operational playbook that was partly in a wiki and partly in someone's head.
Our consumers were all Java, built on spring-kafka. The framework was fine honestly — spring-kafka is mature and the abstractions are reasonable. The problem was never the client library. It was everything underneath it.
What Broke
Three incidents in five months. I wasn't looking for a migration project. The migration found me.
Settlement duplicates, March. Settlement service consumes auth.completed to build end-of-day files for acquiring banks. Broker-6 in us-east-1c hit EBS I/O latency degradation on a Thursday evening. Not a failure — degradation. Triggered partition reassignment, which triggered consumer group rebalance, which led to roughly 12,000 auth events getting processed twice. Settlement file went out with duplicates. Bank caught it during their morning reconciliation before we did.
The Friday call I mentioned. Wouldn't recommend it.
This is the thing about Kafka's consumer group rebalancing — it's probably the worst operational aspect of the entire system. The protocol is complex, the failure modes are subtle, and when it goes wrong in a payments context, the consequences are financial. I've used Kafka in non-payments contexts where a rebalance causing a few duplicate processings is a non-event. Here it meant calling a bank.
Fraud scoring gap, May. Fraud service subscribes to transaction.initiated, needs to respond within 200ms or auth falls through to default-allow. During a routine broker restart, partition leadership migrated. Consumers took about 90 seconds to reconnect. Transactions in that window went through unscored.
Nothing bad happened. I put "we got lucky" in the incident report because I wanted that phrase on the record for when someone reviewed our risk posture.
Reconciliation false positives, ongoing. Recon service consumed from two topics with different partition counts and consumer group configs. During rebalances it'd occasionally miss events, producing phantom mismatches. Ops team investigated each one manually — 20 to 40 minutes each, happening maybe 15-20 times a week. Priya from SRE brought it up in two consecutive retros. She was right both times and I should have moved faster after the first.
The Audit That Made the Decision
Blocked two weeks. Pulled consumer group configs, measured actual throughput, traced ordering dependencies topic by topic. Built a spreadsheet (Google Sheets, ugly, columns like "topic / peak msg/sec / consumers / ordering needed? / idempotent? / what breaks if delayed 5 min?").
Most of our topics were fan-out notifications or work queues. Webhook delivery, emails, PDFs, audit logging, report generation. They needed reliable delivery. They didn't need a distributed commit log.
Topics that genuinely needed strict ordering: ledger.entry.created (per-account — can't apply a debit before a preceding credit), auth.completed to settlement (per-merchant), dispute.state.changed (per-dispute lifecycle). Three.
I'll be honest though: "genuinely needed" is doing some work in that sentence. I validated ordering requirements against normal operating conditions. I did not exhaustively validate every edge case — month-end batch windows, reconciliation sweeps, the weird quarterly processor settlement that runs differently from daily ones. I'll come back to this because it bit me later.
There was also a whole bunch of topics with zero active consumers. Created during some experiment in 2020, never cleaned up. I deleted those first. Nobody noticed for three weeks, which tells you how much they mattered.
Getting It Approved
Architecture review. I showed the three incidents, showed on-call data (pulled 2.3 Kafka-related pages per week from PagerDuty, averaged over six months), showed the bus factor (two people could debug the cluster — me and one other engineer), showed that managed alternatives already existed inside the AWS ecosystem we were paying for.
Framed it as operational risk, not technology preference. Didn't say "Kafka is bad." Said "we don't have the team to run Kafka at the level payments requires, and we don't need most of what Kafka offers."
Director's concern was execution risk. Fair question. Our compliance lead, Meera, also asked whether SQS met our data residency requirements — single region, data doesn't leave the AWS region boundary. It did. But I hadn't thought to check before she asked, which is slightly embarrassing in retrospect. Payments infra changes always have a compliance dimension and I'd been thinking purely in terms of engineering.
I committed to phased migration, topic-by-topic, zero transaction impact. Approved. PAY-4127. I typed that epic number into Slack probably 200 times over the next five months. Actually looking back, I think we also had a sub-epic for just the monitoring work but I can't remember the number.
Ravi on the fraud team pushed back. He'd built a solid consumer framework: offset management, rebalance listeners, retry logic, graceful shutdown. Months of work, well-tested. I talked to him separately, not in the meeting. Told him the code was good but the reason it needed to exist was the problem. If the managed service provides retries and DLQs natively, a framework compensating for their absence is a signal you're fighting the wrong abstraction.
He disagreed. Said we'd lose flexibility. I said we'd lose complexity that we'd been calling flexibility to justify maintaining it.
He came around about three weeks later when he saw the SQS consumer code — maybe 40 lines, no rebalance handling. The code diff convinced him more than anything I said.
How We Did It
Early mistake I caught: I initially proposed Kinesis for ordered topics. My tech lead pointed out Kinesis has shard management, which is operationally the same class of problem as Kafka partitions. He was right. SQS FIFO queues instead.
Our deploy pipeline ("Piper" internally, not sure that was ever its official name) needed per-topic feature flags for dual-publishing. Spent three days building that:
dual-publish config - per topic toggle
messaging:
dual_publish:
enabled_topics:
- auth.completed
- merchant.updated
disabled_topics:
- ledger.entry.created # not yet, ordered topic
target:
sns_arn_prefix: arn:aws:sns:us-east-1:XXXX:
Unglamorous but it meant we could toggle per topic without redeploying. That investment paid for itself immediately.
January through early March — notifications and async. Webhooks, emails, PDFs, audit fan-out. Idempotent consumers, fire-and-forget semantics. SNS fan-out to SQS subscribers. About a week of dual-publishing per topic, output comparison, cutover. Some of these were trivially easy and I probably over-engineered the verification for the first couple, but I was still building confidence in the process.
DLQ visibility was an immediate win. Failed webhooks (merchant endpoint down) used to cause consumer lag that mixed with healthy messages. Now failed messages land in DLQ, CloudWatch alarm fires, clean messages keep flowing. The ops team's weekly false positive investigations from these topics went to zero. Priya sent me a Slack message: "thank you." No context needed.
March through April — work queues. Report generation, settlement file building, chargeback docs. The settlement builder was the one where I ran shadow mode for 21 days comparing output byte-for-byte before cutting over. Maybe paranoid. Don't care. I wasn't having that Friday call again.
Schema incident hit mid-March. This one still annoys me because I'd specifically written it down during planning — "need to handle schema governance without Schema Registry" — and then somehow convinced myself it wouldn't be urgent because "producers won't push breaking changes during migration." Dumb assumption. A producer team deployed a breaking change to merchant.updated, SQS consumer couldn't deserialize, messages piled into DLQ, merchant notifications down for 40 minutes. The Slack thread was not great.
Four days building a JSON Schema validation layer at consumer ingress. Added a CI check on producer repos that validates against a contract file before deploy. It's not as good as Schema Registry (no automatic compatibility negotiation, no centralized schema evolution tracking) and I'm still not fully happy with it. That's a problem I've deferred, not solved. But it stops the bleeding.
April through mid-July — ordered topics. Ledger FIFO queue, message group ID set to account_id. Load tested ten days at 3x production volume. Two weeks shadow mode. Settlement ordering, group ID per merchant_id. Dispute lifecycle was last — delayed three weeks because Ankur (dispute service owner) was on paternity leave and nobody else could validate ordering correctness end-to-end. We waited rather than risk it. Right call.
Finished mid-July. Planned for June. Month late but no incidents during migration. I'll take it. Though honestly by July I was pretty tired of the whole thing and the last few topics I moved faster than I probably should have. They were low-risk topics but still, the discipline slipped a bit toward the end.
If I did this migration again, I'd start with the hardest topic first, not the easiest. Building confidence by doing easy topics first felt right at the time but it also meant we discovered the real problems (schema governance, ordering edge cases) five months in rather than five weeks in. By then you've built momentum and organizational expectation around a timeline. Discovering fundamental gaps early is better even if it's scarier.
What I Got Wrong
The monitoring gap
Priya told me before we started: "We'll lose JMX visibility before CloudWatch equivalents are ready." I nodded, agreed, and then didn't prioritize building the new monitoring because I was focused on the migration itself. For about two weeks after Phase 1 we were partially blind. I didn't know what "healthy" looked like on SQS for our traffic patterns because I hadn't established baselines.
Burned an unplanned sprint building dashboards and alarms. Should have been sprint one.
The mental model problem
Kafka and SQS fail differently. Kafka: consumer stops, lag builds, you see it, you fix it, consumer catches up. SQS: message becomes visible again after timeout, gets redelivered, eventually lands in DLQ. Objectively better behavior in most cases — but foreign to engineers who'd spent years watching consumer lag as their primary health signal.
One engineer (he reads my LinkedIn so I won't name him) messaged at 11 PM saying we were "losing transactions." We weren't. Messages were in the DLQ exactly as designed. But nobody had explained the new model. That's on me. Ran a brown-bag session two days later. Should have been week one.
The month-end ordering surprise
This is the one I'm least comfortable admitting. Remember when I said I validated ordering requirements against normal operating conditions? In month four post-migration, we discovered that one of our "doesn't need ordering" topics — processor.batch.settlement — actually did need ordering during month-end reconciliation windows. The daily flow was fine unordered. But the monthly processor batch sends a sequence of summary records that our recon system expected in order for a specific aggregation step.
It had never been a problem on Kafka because the single-partition setup happened to preserve ordering. When we moved to SQS standard queue, messages could arrive out of order, and the monthly recon broke.
We retrofitted it to a FIFO queue with group ID set to processor_id. Took a few days. Not catastrophic. But it's the kind of thing that makes me less confident about the other topics I classified as "ordering not required." Maybe they're fine. Maybe there's another month-end or quarter-end edge case waiting. I haven't exhaustively validated every temporal edge condition and I probably should.
What I Gave Up and Still Think About
I'm not going to pretend this was all upside. There are real tradeoffs I'm living with.
Message replay is gone. Kafka lets you reset consumer offsets and replay from any point in the log. We used this maybe twice a year for debugging production issues — reset the fraud consumer back 2 hours, replay events, watch what happened. With SQS, once a message is consumed, it's gone. We now log all events to S3 as a side-channel for replay purposes, but it's not the same. It's batched, delayed, and honestly the tooling around querying those S3 logs is not great. I keep meaning to set up Athena queries for it properly but haven't gotten around to it.
I miss it about once a quarter. Not enough to justify Kafka's operational overhead. But I miss it.
I'll say this for Kafka: the pull-based consumption model and the persistent log are genuinely elegant ideas. Being able to have multiple consumer groups each reading the same stream at their own pace, rewinding independently — nothing else does that as cleanly. We just weren't using it enough to justify the operational cost. But if someone told me they needed those semantics for real, I wouldn't argue against Kafka. I'd argue for a dedicated team to run it.
FIFO throughput ceiling is real. 300 messages per second per message group without batching, 3,000 with batching. We're well under today. But payments companies grow. I modeled the crossover point: if our transaction volume grows roughly 4x from current, the per-merchant settlement FIFO queue will start hitting group-level throughput limits during evening batch windows.
I documented this in the ADR with a trigger condition: "If peak per-merchant throughput exceeds 200 msg/sec sustained, evaluate sharding strategy or alternative." We're at about 50 today. I check it quarterly. It's not a crisis. But it's a ceiling and I'm aware of it.
Exactly-once is different. Kafka's transactional producer gives you true exactly-once within the Kafka ecosystem. SQS FIFO deduplication has a 5-minute window — well, technically it's "exactly-once delivery" within that window but you still need idempotent processing on the consumer side because deduplication only prevents the queue from delivering the same message twice, it doesn't prevent your producer from sending semantically-duplicate messages with different IDs. So really it's at-least-once with a deduplication convenience layer. We handle the rest with idempotency keys at the consumer level. It works. But it's application-level concern that Kafka's transactional model handled at infrastructure level.
A purist would call this a regression. I call it pragmatic for our volume and failure patterns.
Cost at scale is uncertain. We saw 40% cost reduction. But SQS pricing is per-message. Kafka's cost is mostly fixed infrastructure. If our message volume doubles (possible within 18 months based on merchant growth projections), the gap narrows. If it triples, SQS might actually be more expensive. I haven't hit that crossover yet but I've modeled it and it exists. Roughly $14K/month on messaging today vs. the ~$23K we were paying for Kafka infra. If volume triples, SQS would be around $42K. So there's a ceiling on the cost story too.
I'm not worried about it today. I am aware of it.
Fourteen Months Out
Kafka brokers terminated. ZooKeeper gone. The Grafana Kafka dashboard — I actually forgot to decommission that instance for two months after migration. Nobody noticed, which kind of tells you everything about how much value it was providing.
The numbers:
On-call pages from messaging: 2.3/week → something like 0.4. Mostly DLQ alerts that auto-resolve.
Settlement duplicates: zero. The failure mode is architecturally gone.
Recon false positives: zero (after the month-end fix).
Fraud scoring gaps: zero.
Cost: ~40% down at current volume. Crossover point modeled, being tracked.
Capacity: about a third of my week back, same for the other engineer who'd been half-time Kafka admin. That time went to fraud rules and a merchant self-service feature we'd been deprioritizing for a year.
Ravi's consumer framework is archived in the repo. I left a README explaining what it was and why it's no longer needed. Felt right. His rebalance listener code was genuinely clever — I learned things from reading it. The problem it solved just doesn't exist anymore.
What I'd Actually Tell You
I'm not going to say "audit your topics" like it's a revelation. You already know whether your Kafka cluster is well-operated or a liability. You know whether you have the team to run it properly. You know whether you're using it as a log or as a queue.
What I will say: the decision isn't "Kafka vs. SQS." It's "what are we actually paying for — in engineering time, in operational risk, in cognitive overhead — and is that cost justified by our actual usage patterns?"
For us it wasn't. For you it might be. If you've got a dedicated platform team, high-throughput event sourcing, multi-consumer replay requirements, Kafka is genuinely the right tool. Don't rip it out because some guy on LinkedIn (me) told a story about how it went well for his team.
But if you're in the position I was in — small team, payments-grade reliability requirements, Kafka expertise concentrated in two people, entire rest of your stack on AWS — at least do the audit. Look at what you're actually using versus what you're maintaining the capability to theoretically use.
The gap between those two things was embarrassingly large for us. I suspect it might be for you too. But I could be wrong about your situation, and only your spreadsheet will tell you.
Anyway — that's the story. Five months, a bunch of Jira tickets, one schema incident that still annoys me, and a month-end ordering bug that keeps me humble about how thorough my original audit actually was. Make of it what you will.
Before -
After -
Top comments (0)