DEV Community: Basu

Why I Killed Our Kafka Cluster (And What I'd Do Differently)

Basu — Wed, 05 Aug 2026 18:35:00 +0000

TL;DR: We ran Kafka in a payments company for 3 years. Most of our 180 topics were glorified queues. We moved to SNS/SQS in 5 months. Here's the real story, including the parts I got wrong.

I used to think running your own Kafka cluster was a sign of engineering maturity. Like your team had graduated to "real" distributed systems. Then I spent two years operating one inside a payments company with a six-person platform team and realized we hadn't graduated to anything — we'd just volunteered for operational overhead we didn't need.

The moment it became undeniable was a Friday morning call with our acquiring bank's technical team, where I had to explain how a slow disk caused 12,000 duplicate transactions in their settlement file. Me on mute half the time, trying to figure out how to say "our messaging infrastructure isn't operated at the level your business requires" without actually saying that. That's when I decided Kafka had to go.

What I Walked Into
Kafka was already there when I joined the payments platform team. I tried to find out why it was chosen. Got nowhere. There was a Confluence page from 2019, "Event-Driven Architecture — Proposed," authored by someone who'd left a year and a half before I showed up. Linked to a dead Slack channel. The page had four views. I'm pretty sure two were me.

The setup: 9 brokers on r5.2xlarge, running Kafka 2.7 (two minor versions behind because upgrading meant regression-testing consumers and nobody wanted to own that risk). Three t3.medium ZooKeeper nodes. A Schema Registry instance. Somewhere around 170-180 topics, I'd have to check my spreadsheet for the exact count. Retention was 7 days on most, 30 on a handful of others. Never found documentation on why some got 30 days. I suspect whoever set it up just picked different numbers for different topics based on vibes.

Everything else in our stack was AWS-native. ECS, DynamoDB, S3, CloudWatch, IAM. And then Kafka, running on EC2 with its own access model, its own monitoring (JMX exporters into Grafana — I think three people looked at those dashboards regularly, and one of them was me), its own operational playbook that was partly in a wiki and partly in someone's head.

Our consumers were all Java, built on spring-kafka. The framework was fine honestly — spring-kafka is mature and the abstractions are reasonable. The problem was never the client library. It was everything underneath it.

What Broke
Three incidents in five months. I wasn't looking for a migration project. The migration found me.

Settlement duplicates, March. Settlement service consumes auth.completed to build end-of-day files for acquiring banks. Broker-6 in us-east-1c hit EBS I/O latency degradation on a Thursday evening. Not a failure — degradation. Triggered partition reassignment, which triggered consumer group rebalance, which led to roughly 12,000 auth events getting processed twice. Settlement file went out with duplicates. Bank caught it during their morning reconciliation before we did.

The Friday call I mentioned. Wouldn't recommend it.

This is the thing about Kafka's consumer group rebalancing — it's probably the worst operational aspect of the entire system. The protocol is complex, the failure modes are subtle, and when it goes wrong in a payments context, the consequences are financial. I've used Kafka in non-payments contexts where a rebalance causing a few duplicate processings is a non-event. Here it meant calling a bank.

Fraud scoring gap, May. Fraud service subscribes to transaction.initiated, needs to respond within 200ms or auth falls through to default-allow. During a routine broker restart, partition leadership migrated. Consumers took about 90 seconds to reconnect. Transactions in that window went through unscored.

Nothing bad happened. I put "we got lucky" in the incident report because I wanted that phrase on the record for when someone reviewed our risk posture.

Reconciliation false positives, ongoing. Recon service consumed from two topics with different partition counts and consumer group configs. During rebalances it'd occasionally miss events, producing phantom mismatches. Ops team investigated each one manually — 20 to 40 minutes each, happening maybe 15-20 times a week. Priya from SRE brought it up in two consecutive retros. She was right both times and I should have moved faster after the first.

The Audit That Made the Decision
Blocked two weeks. Pulled consumer group configs, measured actual throughput, traced ordering dependencies topic by topic. Built a spreadsheet (Google Sheets, ugly, columns like "topic / peak msg/sec / consumers / ordering needed? / idempotent? / what breaks if delayed 5 min?").

Most of our topics were fan-out notifications or work queues. Webhook delivery, emails, PDFs, audit logging, report generation. They needed reliable delivery. They didn't need a distributed commit log.

Topics that genuinely needed strict ordering: ledger.entry.created (per-account — can't apply a debit before a preceding credit), auth.completed to settlement (per-merchant), dispute.state.changed (per-dispute lifecycle). Three.

I'll be honest though: "genuinely needed" is doing some work in that sentence. I validated ordering requirements against normal operating conditions. I did not exhaustively validate every edge case — month-end batch windows, reconciliation sweeps, the weird quarterly processor settlement that runs differently from daily ones. I'll come back to this because it bit me later.

There was also a whole bunch of topics with zero active consumers. Created during some experiment in 2020, never cleaned up. I deleted those first. Nobody noticed for three weeks, which tells you how much they mattered.

Getting It Approved
Architecture review. I showed the three incidents, showed on-call data (pulled 2.3 Kafka-related pages per week from PagerDuty, averaged over six months), showed the bus factor (two people could debug the cluster — me and one other engineer), showed that managed alternatives already existed inside the AWS ecosystem we were paying for.

Framed it as operational risk, not technology preference. Didn't say "Kafka is bad." Said "we don't have the team to run Kafka at the level payments requires, and we don't need most of what Kafka offers."

Director's concern was execution risk. Fair question. Our compliance lead, Meera, also asked whether SQS met our data residency requirements — single region, data doesn't leave the AWS region boundary. It did. But I hadn't thought to check before she asked, which is slightly embarrassing in retrospect. Payments infra changes always have a compliance dimension and I'd been thinking purely in terms of engineering.

I committed to phased migration, topic-by-topic, zero transaction impact. Approved. PAY-4127. I typed that epic number into Slack probably 200 times over the next five months. Actually looking back, I think we also had a sub-epic for just the monitoring work but I can't remember the number.

Ravi on the fraud team pushed back. He'd built a solid consumer framework: offset management, rebalance listeners, retry logic, graceful shutdown. Months of work, well-tested. I talked to him separately, not in the meeting. Told him the code was good but the reason it needed to exist was the problem. If the managed service provides retries and DLQs natively, a framework compensating for their absence is a signal you're fighting the wrong abstraction.

He disagreed. Said we'd lose flexibility. I said we'd lose complexity that we'd been calling flexibility to justify maintaining it.

He came around about three weeks later when he saw the SQS consumer code — maybe 40 lines, no rebalance handling. The code diff convinced him more than anything I said.

How We Did It
Early mistake I caught: I initially proposed Kinesis for ordered topics. My tech lead pointed out Kinesis has shard management, which is operationally the same class of problem as Kafka partitions. He was right. SQS FIFO queues instead.

Our deploy pipeline ("Piper" internally, not sure that was ever its official name) needed per-topic feature flags for dual-publishing. Spent three days building that:

dual-publish config - per topic toggle

messaging:
dual_publish:
enabled_topics:
- auth.completed
- merchant.updated
disabled_topics:
- ledger.entry.created # not yet, ordered topic
target:
sns_arn_prefix: arn:aws:sns:us-east-1:XXXX:
Unglamorous but it meant we could toggle per topic without redeploying. That investment paid for itself immediately.

January through early March — notifications and async. Webhooks, emails, PDFs, audit fan-out. Idempotent consumers, fire-and-forget semantics. SNS fan-out to SQS subscribers. About a week of dual-publishing per topic, output comparison, cutover. Some of these were trivially easy and I probably over-engineered the verification for the first couple, but I was still building confidence in the process.

DLQ visibility was an immediate win. Failed webhooks (merchant endpoint down) used to cause consumer lag that mixed with healthy messages. Now failed messages land in DLQ, CloudWatch alarm fires, clean messages keep flowing. The ops team's weekly false positive investigations from these topics went to zero. Priya sent me a Slack message: "thank you." No context needed.

March through April — work queues. Report generation, settlement file building, chargeback docs. The settlement builder was the one where I ran shadow mode for 21 days comparing output byte-for-byte before cutting over. Maybe paranoid. Don't care. I wasn't having that Friday call again.

Schema incident hit mid-March. This one still annoys me because I'd specifically written it down during planning — "need to handle schema governance without Schema Registry" — and then somehow convinced myself it wouldn't be urgent because "producers won't push breaking changes during migration." Dumb assumption. A producer team deployed a breaking change to merchant.updated, SQS consumer couldn't deserialize, messages piled into DLQ, merchant notifications down for 40 minutes. The Slack thread was not great.

Four days building a JSON Schema validation layer at consumer ingress. Added a CI check on producer repos that validates against a contract file before deploy. It's not as good as Schema Registry (no automatic compatibility negotiation, no centralized schema evolution tracking) and I'm still not fully happy with it. That's a problem I've deferred, not solved. But it stops the bleeding.

April through mid-July — ordered topics. Ledger FIFO queue, message group ID set to account_id. Load tested ten days at 3x production volume. Two weeks shadow mode. Settlement ordering, group ID per merchant_id. Dispute lifecycle was last — delayed three weeks because Ankur (dispute service owner) was on paternity leave and nobody else could validate ordering correctness end-to-end. We waited rather than risk it. Right call.

Finished mid-July. Planned for June. Month late but no incidents during migration. I'll take it. Though honestly by July I was pretty tired of the whole thing and the last few topics I moved faster than I probably should have. They were low-risk topics but still, the discipline slipped a bit toward the end.

If I did this migration again, I'd start with the hardest topic first, not the easiest. Building confidence by doing easy topics first felt right at the time but it also meant we discovered the real problems (schema governance, ordering edge cases) five months in rather than five weeks in. By then you've built momentum and organizational expectation around a timeline. Discovering fundamental gaps early is better even if it's scarier.

What I Got Wrong
The monitoring gap
Priya told me before we started: "We'll lose JMX visibility before CloudWatch equivalents are ready." I nodded, agreed, and then didn't prioritize building the new monitoring because I was focused on the migration itself. For about two weeks after Phase 1 we were partially blind. I didn't know what "healthy" looked like on SQS for our traffic patterns because I hadn't established baselines.

Burned an unplanned sprint building dashboards and alarms. Should have been sprint one.

The mental model problem
Kafka and SQS fail differently. Kafka: consumer stops, lag builds, you see it, you fix it, consumer catches up. SQS: message becomes visible again after timeout, gets redelivered, eventually lands in DLQ. Objectively better behavior in most cases — but foreign to engineers who'd spent years watching consumer lag as their primary health signal.

One engineer (he reads my LinkedIn so I won't name him) messaged at 11 PM saying we were "losing transactions." We weren't. Messages were in the DLQ exactly as designed. But nobody had explained the new model. That's on me. Ran a brown-bag session two days later. Should have been week one.

The month-end ordering surprise
This is the one I'm least comfortable admitting. Remember when I said I validated ordering requirements against normal operating conditions? In month four post-migration, we discovered that one of our "doesn't need ordering" topics — processor.batch.settlement — actually did need ordering during month-end reconciliation windows. The daily flow was fine unordered. But the monthly processor batch sends a sequence of summary records that our recon system expected in order for a specific aggregation step.

It had never been a problem on Kafka because the single-partition setup happened to preserve ordering. When we moved to SQS standard queue, messages could arrive out of order, and the monthly recon broke.

We retrofitted it to a FIFO queue with group ID set to processor_id. Took a few days. Not catastrophic. But it's the kind of thing that makes me less confident about the other topics I classified as "ordering not required." Maybe they're fine. Maybe there's another month-end or quarter-end edge case waiting. I haven't exhaustively validated every temporal edge condition and I probably should.

What I Gave Up and Still Think About
I'm not going to pretend this was all upside. There are real tradeoffs I'm living with.

Message replay is gone. Kafka lets you reset consumer offsets and replay from any point in the log. We used this maybe twice a year for debugging production issues — reset the fraud consumer back 2 hours, replay events, watch what happened. With SQS, once a message is consumed, it's gone. We now log all events to S3 as a side-channel for replay purposes, but it's not the same. It's batched, delayed, and honestly the tooling around querying those S3 logs is not great. I keep meaning to set up Athena queries for it properly but haven't gotten around to it.

I miss it about once a quarter. Not enough to justify Kafka's operational overhead. But I miss it.

I'll say this for Kafka: the pull-based consumption model and the persistent log are genuinely elegant ideas. Being able to have multiple consumer groups each reading the same stream at their own pace, rewinding independently — nothing else does that as cleanly. We just weren't using it enough to justify the operational cost. But if someone told me they needed those semantics for real, I wouldn't argue against Kafka. I'd argue for a dedicated team to run it.

FIFO throughput ceiling is real. 300 messages per second per message group without batching, 3,000 with batching. We're well under today. But payments companies grow. I modeled the crossover point: if our transaction volume grows roughly 4x from current, the per-merchant settlement FIFO queue will start hitting group-level throughput limits during evening batch windows.

I documented this in the ADR with a trigger condition: "If peak per-merchant throughput exceeds 200 msg/sec sustained, evaluate sharding strategy or alternative." We're at about 50 today. I check it quarterly. It's not a crisis. But it's a ceiling and I'm aware of it.

Exactly-once is different. Kafka's transactional producer gives you true exactly-once within the Kafka ecosystem. SQS FIFO deduplication has a 5-minute window — well, technically it's "exactly-once delivery" within that window but you still need idempotent processing on the consumer side because deduplication only prevents the queue from delivering the same message twice, it doesn't prevent your producer from sending semantically-duplicate messages with different IDs. So really it's at-least-once with a deduplication convenience layer. We handle the rest with idempotency keys at the consumer level. It works. But it's application-level concern that Kafka's transactional model handled at infrastructure level.

A purist would call this a regression. I call it pragmatic for our volume and failure patterns.

Cost at scale is uncertain. We saw 40% cost reduction. But SQS pricing is per-message. Kafka's cost is mostly fixed infrastructure. If our message volume doubles (possible within 18 months based on merchant growth projections), the gap narrows. If it triples, SQS might actually be more expensive. I haven't hit that crossover yet but I've modeled it and it exists. Roughly $14K/month on messaging today vs. the ~$23K we were paying for Kafka infra. If volume triples, SQS would be around $42K. So there's a ceiling on the cost story too.

I'm not worried about it today. I am aware of it.

Fourteen Months Out
Kafka brokers terminated. ZooKeeper gone. The Grafana Kafka dashboard — I actually forgot to decommission that instance for two months after migration. Nobody noticed, which kind of tells you everything about how much value it was providing.

The numbers:

On-call pages from messaging: 2.3/week → something like 0.4. Mostly DLQ alerts that auto-resolve.
Settlement duplicates: zero. The failure mode is architecturally gone.
Recon false positives: zero (after the month-end fix).
Fraud scoring gaps: zero.
Cost: ~40% down at current volume. Crossover point modeled, being tracked.
Capacity: about a third of my week back, same for the other engineer who'd been half-time Kafka admin. That time went to fraud rules and a merchant self-service feature we'd been deprioritizing for a year.
Ravi's consumer framework is archived in the repo. I left a README explaining what it was and why it's no longer needed. Felt right. His rebalance listener code was genuinely clever — I learned things from reading it. The problem it solved just doesn't exist anymore.

What I'd Actually Tell You
I'm not going to say "audit your topics" like it's a revelation. You already know whether your Kafka cluster is well-operated or a liability. You know whether you have the team to run it properly. You know whether you're using it as a log or as a queue.

What I will say: the decision isn't "Kafka vs. SQS." It's "what are we actually paying for — in engineering time, in operational risk, in cognitive overhead — and is that cost justified by our actual usage patterns?"

For us it wasn't. For you it might be. If you've got a dedicated platform team, high-throughput event sourcing, multi-consumer replay requirements, Kafka is genuinely the right tool. Don't rip it out because some guy on LinkedIn (me) told a story about how it went well for his team.

But if you're in the position I was in — small team, payments-grade reliability requirements, Kafka expertise concentrated in two people, entire rest of your stack on AWS — at least do the audit. Look at what you're actually using versus what you're maintaining the capability to theoretically use.

The gap between those two things was embarrassingly large for us. I suspect it might be for you too. But I could be wrong about your situation, and only your spreadsheet will tell you.

Anyway — that's the story. Five months, a bunch of Jira tickets, one schema incident that still annoys me, and a month-end ordering bug that keeps me humble about how thorough my original audit actually was. Make of it what you will.

Before -

After -

Why Almost Every Search Feature Eventually Ends Up Using Elasticsearch

Basu — Wed, 29 Jul 2026 15:30:25 +0000

There's a ticket that every product team gets eventually. It says something like "search is broken" or "why can't users find anything?" or my favorite, "can we make search not suck?"

The search box was built on PostgreSQL. It worked fine for two years. Then the catalog grew, users started expecting Google-level typo tolerance, and the PM asked for faceted filters. Now the LIKE query takes 4 seconds and someone's googling "Elasticsearch vs Solr."

I've been on the team that migrates. It's not fun, but it's predictable. Here's why it happens and what you're actually signing up for.

The Moment Your Database Stops Being Good Enough

Every search feature starts the same way:

SELECT * FROM products WHERE name LIKE '%headset%';

This works. With 10,000 products and a reasonable index, it returns in milliseconds. Nobody complains.

Then the requirements compound:

"Can we add typo tolerance? Users type 'wireles' and get nothing back."
"Can we search across name, description, brand, and category simultaneously?"
"Can we rank results by relevance instead of returning them in insertion order?"
"Can we add filters (price range, brand, in-stock) that work with the text search?"
"Can we show facet counts (42 results in Electronics, 18 in Gaming)?"

Each of these is solvable individually in PostgreSQL. Together, at scale, they compound into a query that no amount of indexing will save. I ran EXPLAIN ANALYZE on our compound search query once it hit a million records — the planner was doing a sequential scan because the trigram GIN index couldn't handle the combined text + filter + sort. Response time: 3.8 seconds. The PM's exact words were "that's not search, that's a loading screen."

PostgreSQL's full-text search (tsvector / tsquery) buys you time. It handles tokenization, stemming, and ranking. It's genuinely good for moderate-scale use cases — maybe up to a few million documents if your queries aren't too complex. But the moment you need fuzzy matching, synonym expansion, language-aware analysis, and relevance tuning together, you've outgrown what a relational database was designed to do.

The progression almost always looks like:

LIKE '%term%'  →  PostgreSQL FTS  →  Elasticsearch

The question isn't whether you'll make this jump. It's when your product requirements force it.

What Elasticsearch Actually Does Differently

Databases scan documents looking for matches. Elasticsearch flips this — it pre-builds a map from term → documents that contain it. So instead of asking "does this document contain 'headset'?" a million times, it asks "which documents contain 'headset'?" once.

Three documents:

Doc 1: "Wireless gaming headset with noise cancellation"
Doc 2: "Gaming keyboard with RGB lighting"  
Doc 3: "Wireless ergonomic mouse"

ES stores:

wireless      → [1, 3]
gaming        → [1, 2]
headset       → [1]
keyboard      → [2]
mouse         → [3]

Search for "wireless gaming headset" = intersect the posting lists: [1,3] ∩ [1,2] ∩ [1] = Doc 1. Done. No scanning. This is why it stays fast at 100 million documents — you're looking up terms in a map, not scanning rows.

The tricky part that took me a while to internalize: all this work happens at write time. When a document is indexed, ES tokenizes it, stems it, lowercases it, and slots every term into the inverted index. Queries are fast because indexing was slow (relatively). You're trading write cost for read speed.

Analyzers — Why Search Feels Smarter Than It Is

The reason "running shoes" matches a document containing "run" isn't magic. It's the analyzer pipeline that runs during indexing:

Tokenization: split text into individual terms
Lowercasing: "Running" → "running"

Stop word removal: drop "with", "the", "a"

Stemming: "running" → "run", "cancellation" → "cancel"

Both the indexed document AND the search query pass through the same analyzer. So when you search for "running," it's stemmed to "run," which matches documents that were also stemmed to "run" at index time.

This is also why Elasticsearch handles typos — Levenshtein distance (edit distance) allows fuzzy matching. "elsticsearch" is 1 edit away from "elasticsearch," so the engine can suggest or match the correct term.

The analyzer is the most underrated configuration in Elasticsearch, and honestly the documentation around custom analyzers is frustratingly sparse for how important they are. A poorly configured analyzer is the #1 reason search results feel bad — I spent two days debugging "why don't users find products by brand" before realizing the analyzer was stripping the brand name as a stop word. Getting it right requires understanding your data: English prose needs stemming, product SKUs need keyword (no analysis), and multi-language catalogs need per-field language analyzers.

Relevance Scoring — Where Databases Fully Give Up

Finding matches is easy. Ranking them is where search gets genuinely hard.

Three documents match "python performance":

Doc A: "Python Performance Optimization Guide"
Doc B: "Python Python Python Performance"
Doc C: "Performance Tuning for Large-Scale Python Applications"

Which one should rank first? ES uses BM25, which basically asks: how often does the term appear in this document, how rare is it across all documents, and how long is the document? A match in a short, focused title beats the same match buried in a 10,000-word page. And keyword stuffing (Doc B) actually gets penalized because BM25 applies diminishing returns to repeated terms.

You can't do this in SQL. Not without pulling every matching row into memory and rescoring in application code — at which point your database is just an expensive SELECT *.

The one thing nobody tells you: BM25 is the default and it works well out of the box, but the moment you need business-specific ranking (boost products that are in-stock, penalize items with bad reviews, promote sponsored listings), you're writing custom function_score queries. Those get complex fast and are genuinely hard to debug when results feel "off." We had a ranking bug once where out-of-stock items ranked first because the boost function was multiplicative on a field that happened to be zero for in-stock products. Took a full day to figure out.

The Architecture: ES Sits Beside Your Database, Not Instead Of It

This trips up newcomers. ES is not your primary datastore. It's a read-optimized copy.

Writes → PostgreSQL (source of truth)
           ↓
         CDC / Events / Dual writes
           ↓
         Elasticsearch (searchable view)
           ↓
Reads ← Search API

Your database handles transactions and correctness. ES handles discovery and ranking. Data flows into ES through CDC (Debezium), event pipelines (Kafka consumers), or direct application writes (simple but you'll eventually hit inconsistency bugs when one write succeeds and the other fails).

One gotcha: ES is near-real-time with a default 1-second refresh interval. Write a document, query immediately — it might not be there yet. For most search UIs nobody notices. But if you have a flow where a user creates something and lands on a page that searches for it, you'll get a bug report within the first week.

How it scales: a single Lucene index gets too big, so ES splits it into shards — each an independent Lucene index on potentially different nodes. Search hits all shards in parallel, a coordinator merges the top results. Horizontal scaling = add nodes, redistribute shards. Underneath, Lucene uses immutable segments (new data → new segment, never modify old ones, merge in background). This is why reads are fast (no locking on immutable data) and crash recovery is clean.

What Queries Actually Look Like

This is the part most "intro to ES" articles skip. Here's what you're actually writing when you use Elasticsearch:

Simple multi-field search:

GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "wireless gaming headset",
      "fields": ["name^3", "description", "brand^2", "category"],
      "type": "best_fields",
      "fuzziness": "AUTO"
    }
  }
}

The ^3 boosts matches in the name field 3x over description. fuzziness: AUTO handles typos. This single query does what would take a complex stored procedure in PostgreSQL — multi-field search, fuzzy matching, field-weighted relevance scoring.

Search with filters and facets:

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        { "multi_match": { "query": "headset", "fields": ["name^3", "description"] } }
      ],
      "filter": [
        { "term": { "in_stock": true } },
        { "range": { "price": { "gte": 50, "lte": 200 } } }
      ]
    }
  },
  "aggs": {
    "brands": { "terms": { "field": "brand.keyword", "size": 10 } },
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 50 },
          { "from": 50, "to": 100 },
          { "from": 100, "to": 200 },
          { "from": 200 }
        ]
      }
    }
  }
}

The bool query separates relevance (must — affects scoring) from filtering (filter — yes/no, no scoring, cached). Aggregations give you facet counts in the same request. This is the "show me wireless headsets, $50-200, in stock, with brand and price facets on the sidebar" query — one round trip.

One thing that tripped me up early: that brand.keyword field. If you try to aggregate on an analyzed text field, ES throws an error or gives garbage results. You need the .keyword sub-field (not analyzed, exact match) for terms aggregations. The mapping needs to define both:

"brand": {
  "type": "text",
  "fields": { "keyword": { "type": "keyword" } }
}

Text for searching, keyword for filtering and aggregating. Once you know this it's obvious, but I lost a few hours to it the first time.

Custom ranking with function_score:

GET /products/_search
{
  "query": {
    "function_score": {
      "query": { "multi_match": { "query": "headset", "fields": ["name", "description"] } },
      "functions": [
        {
          "filter": { "term": { "is_sponsored": true } },
          "weight": 1.5
        },
        {
          "field_value_factor": {
            "field": "review_score",
            "modifier": "log1p",
            "missing": 1
          }
        }
      ],
      "boost_mode": "multiply"
    }
  }
}

This boosts sponsored products by 1.5x and factors in review scores with logarithmic dampening (so a product with 4.8 stars doesn't infinitely dominate over 4.5). This is where relevance tuning lives — and where it gets hard to debug when results feel wrong.

The Alternatives Landscape (2025+)

Elasticsearch isn't the only option anymore. The landscape has shifted, and for some use cases the newer tools are genuinely better:

Typesense — we used this for internal documentation search and it was running in an afternoon. Typo tolerance and relevance work out of the box. No JVM, no cluster management, single binary. Great for product/site search under 100M documents. Falls short on complex aggregations and log analytics — it's not trying to be ES, and that's fine.

Meilisearch — similar positioning to Typesense. Rust-based, developer-friendly, sub-50ms search without tuning. Good until you need sharding or serious multi-tenancy. I'd pick it for a startup's product search without hesitation.

OpenSearch — AWS's fork after Elastic changed the license. Functionally near-identical to ES 7.x, API-compatible. If you're already on AWS and want managed infrastructure (OpenSearch Service) without licensing headaches, this is the pragmatic choice. Migration from ES is mostly a find-and-replace in your client config.

Algolia — fully managed, zero infrastructure. Response times are absurd (single-digit milliseconds). But pricing is per search operation, which gets expensive fast at scale. Good for teams that value DX and speed-to-market over cost efficiency. Less control over relevance tuning than self-hosted ES.

PostgreSQL FTS — don't dismiss it. For under 2-3M documents with straightforward requirements (tokenized search + basic ranking, no fuzzy), it's genuinely sufficient. You avoid an entire operational category. We should've stuck with it longer on one project instead of jumping to ES prematurely.

My take: if your dataset is under 10M documents and your requirements are "text search + filters + typo tolerance" — look at Typesense or Meilisearch first. You'll be in production in a day instead of a week. If you need complex aggregations, log analytics, custom scoring pipelines, or you're north of 100M documents — Elasticsearch (or OpenSearch) still wins. Nothing else handles that combination at that scale.

The Cost

Nobody talks about this in the "why Elasticsearch is amazing" posts:

You're now running two data stores. Two things to monitor, two things that can fail independently, and sync logic between them that will have bugs.
Mapping changes require reindexing. Change an analyzer on a field? Every existing document keeps the old index structure. You need to build a new index from scratch and swap.
At 500M documents, that reindex takes hours. Hope you set up the alias pattern (products_v1 → products_v2 swap) from the beginning. Retrofitting it into a system that assumed a single index name is a bad week.
Shard management is a dark art. Too few shards = can't distribute load. Too many = overhead per shard kills performance. The default of 1 shard is almost never right for production, and the "correct" number depends on your document count, document size, and query patterns.

It's worth the cost if your product genuinely needs good search. Just know what you're signing up for operationally.

When You Don't Need Elasticsearch

Not every search box justifies a cluster:

< 100K documents with simple queries: PostgreSQL FTS handles this fine. Seriously, don't add infrastructure for a problem you don't have.
Exact-match lookups only: if users search by SKU, order ID, or email — that's a database index, not a search engine. A B-tree does this in microseconds.
Strong transactional consistency required: if a write must be visible in search results within the same request, ES's 1-second refresh lag will hurt you. We learned this one the hard way with a status update flow where users would update a record and immediately search for it with the new status — and not find it.
Tiny product catalog: 500 products? Load them into memory on startup, filter in Java. You don't need distributed infrastructure for this.

The decision should be: "have we outgrown what our database can do for this specific access pattern?" If yes, add ES. If you're speculating that you might need it someday — wait until you actually do.

The Progression Every Team Goes Through

Elasticsearch isn't the best database (it isn't one), and it isn't even the search algorithm (Lucene does that part). It's infrastructure that bridges the gap between what users expect from a search box and what a relational database can deliver.

That gap widens with every feature request. And at some point, probably around the third PM ticket about "search quality," you'll start evaluating ES. When you do, plan for the alias-based reindexing pattern from day one, budget for the operational overhead, and don't index more fields than you actually search on. The cluster will thank you.

Further reading:

Elasticsearch: The Definitive Guide (official, somewhat dated but conceptually solid)
Relevant Search — Turnbull & Berryman (the best book on search relevance engineering)
Apache Lucene (the actual engine underneath)
BM25 scoring

Why Elasticsearch Should Not Be Preferred for Writing Data

Basu — Tue, 17 Dec 2024 18:26:18 +0000

Elasticsearch is one of the most popular tools for searching and analyzing data. Its ability to process massive datasets and deliver near real-time search results has made it a cornerstone for applications ranging from e-commerce platforms to monitoring systems. However, when it comes to writing data, Elasticsearch falls short in several critical areas. In this article, we’ll explore the challenges of writing data to Elasticsearch, explain why it’s not ideal for write-heavy use cases, and discuss better alternatives. 📊💾🚫

Understanding Elasticsearch’s Core Strength: Read-Heavy Workloads

Elasticsearch is optimized for fast querying and analytics, powered by its inverted index architecture. This makes it a perfect choice for use cases where:

Data is read and searched frequently.

Queries involve complex filtering, aggregations, or full-text search.

However, Elasticsearch was not built for heavy or continuous data writing. Its architecture and design decisions, while excellent for search, introduce inefficiencies for write-intensive applications. 🧠🔍💡

Challenges of Writing Data to Elasticsearch

High Resource Consumption

Indexing Overhead: Every time data is written to Elasticsearch, it goes through a series of processes:

Analysis: Text fields are broken into tokens using analyzers.

Inverted Index Creation: Tokens are mapped to their locations in documents.

Segment Management: Data is written to immutable segments on disk.

These steps consume significant CPU, memory, and disk resources. 📈🛠️🔄

Write Amplification: Elasticsearch continuously creates and merges segments to optimize queries. These operations amplify disk I/O, leading to slower writes and higher infrastructure costs. 🚀📂📉

Eventual Consistency

Elasticsearch is eventually consistent. This means that after writing data to one node, it may take time for the data to replicate across the cluster and become available for querying.

For applications requiring immediate consistency (e.g., financial transactions or inventory updates), this delay is unacceptable. ⏱️⚠️🔄

Frequent Updates Are Costly

Unlike traditional databases, Elasticsearch doesn’t modify existing documents directly.

Instead, it marks the old document as deleted and writes a new version of the document to a new segment. This process:

Consumes more disk space.

Triggers reindexing and segment merges, further taxing the system. 🗂️🖋️🔄

Limited Transactional Capabilities

Elasticsearch does not support ACID (Atomicity, Consistency, Isolation, Durability) transactions.

Concurrent writes or updates can lead to data conflicts or inconsistencies, making it unsuitable for applications requiring strict transactional guarantees. ❌🔒🔗

Performance Degradation Under Heavy Write Load

Write-heavy workloads can overwhelm Elasticsearch, causing cluster instability. Symptoms include:

High latencies for both writes and reads.

Increased memory usage leading to out-of-memory errors.

Node failures and slower query performance. 🐢🔥💥

Disk Usage Overhead

The inverted index and additional metadata (e.g., for replicas and segments) result in significant disk usage. Frequent writes, updates, and deletes exacerbate this problem, leading to higher storage requirements and costs. 💾📉📛

Example: Writing Data to Elasticsearch

Scenario

Imagine a real-time analytics system for tracking user interactions on a website. The system logs every page view, button click, and transaction as a separate document in Elasticsearch. With millions of interactions recorded daily, the following challenges arise: 🌐📱📊

High Write Throughput:

Each interaction generates a new document.

Elasticsearch’s indexing process struggles to keep up, resulting in slower writes and increased resource usage.

Frequent Updates:

If user interactions need updates (e.g., adding session details), Elasticsearch marks the old document as deleted and writes a new version. This doubles the write effort and bloats disk usage. 📈📝📛

Delayed Availability:

Newly indexed data isn’t immediately available for queries due to the refresh interval (default: 1 second). For real-time applications, this delay is problematic. 🕒⚡🚫

Better Alternatives for Write-Heavy Workloads

If your application involves high write throughput or frequent updates, consider these alternatives: 💡🔄💼

Relational Databases

Examples: MySQL, PostgreSQL

Advantages:

ACID-compliant transactions ensure data consistency.

Optimized for frequent updates and transactional writes.

Use Cases: Financial systems, inventory management, and applications requiring strong consistency.

NoSQL Databases

Examples: MongoDB, Apache Cassandra, DynamoDB

Advantages:

High scalability and fault tolerance.

Efficient for high write throughput.

Use Cases: Real-time analytics, distributed systems, and high-availability applications.

Message Queues or Streaming Systems

Examples: Apache Kafka, Amazon Kinesis

Advantages:

Handle massive write loads efficiently.

Provide durability and scalability for event-driven architectures.

Use Cases: Event logging, real-time data pipelines, and buffering writes before processing.

Time-Series Databases

Examples: InfluxDB, TimescaleDB

Advantages:

Designed for time-series data with high write speeds.

Built-in support for time-based queries and retention policies.

Use Cases: IoT data, monitoring systems, and performance analytics. 📊📆🚀

Optimizing Writes to Elasticsearch (If You Must)

For scenarios where Elasticsearch is necessary for search and analytics but still requires frequent data writes, consider the following optimizations: 🛠️📊✨

Use Bulk API:

Batch multiple write operations into a single request to reduce indexing overhead.

Adjust Refresh Interval:

Increase the refresh interval to delay making new data searchable, reducing resource usage during writes.

Shard Configuration:

Optimize the number of shards and replicas to balance performance and storage.

Pre-process Data:

Use tools like Apache Kafka or AWS Lambda to aggregate and transform data before writing to Elasticsearch.

Monitor and Scale:

Use Elasticsearch’s monitoring tools to identify bottlenecks and scale resources as needed. 📊📈🖥️

Conclusion

Elasticsearch is an exceptional tool for search and analytics, but it is not designed to handle write-heavy workloads efficiently. Its resource-intensive indexing process, eventual consistency model, and limited transactional capabilities make it unsuitable for applications that prioritize high write throughput or frequent updates. 📈🚀❌

Instead, consider using purpose-built databases and systems for writing data, and use Elasticsearch as a secondary layer for search and analytics. This approach ensures better performance, scalability, and cost-efficiency for your application. 💼💡✔️

Have you faced challenges with writing data to Elasticsearch? Share your experiences in the comments! 💬📣🤔

Records in java

Basu — Tue, 26 Mar 2024 05:27:59 +0000

Whenever we talk about boilerplate code in Java, there is always an argument that now IDEs are intelligent enough to generate sources. The problem comes when there is an addition in the fields. Record in Java 14 solves this problem at least for the Data Classes.
A record class in Java automatically has:

A private final field for each component of the record.
A public constructor whose signature matches the record components.
Public read accessor methods for each component of the record, with the same name and type as the component.
Implementations of equals(), hashCode(), and toString() methods.

Here is an example of how to declare and use a record:
public record Employee(String name, int age) {}

This record class Employee has two components: name and age. You can create an instance of Employee and access its components like this:

Employee employee = new Employee("John Doe", 30); System.out.println(employee.name()); // prints "John Doe" System.out.println(employee.age()); // prints 30

Note that records are intended to be used as simple data carriers, so they are not suitable for every situation. If you need more complex behavior, you should use a regular class.

Records in Java, while useful for creating simple data carrier classes, do have some limitations:

No Inheritance: Records cannot extend any other class, and they are implicitly final, so no other class can extend a record.
Limited Instance Variables: All instance variables must be declared in the record header, and these variables are automatically final. You cannot declare additional fields in the body of the record.
No Argument-less Constructor: Records automatically provide a constructor with arguments for all fields, but they do not provide a no-argument constructor. This can be a limitation when working with frameworks that rely on no-argument constructors.
No Mutability: Since all fields in a record are final, records are inherently immutable. This means once a record object is created, you cannot change its state.
Limited Use of Annotations: Annotations that are applicable to classes, methods, or fields may not be applicable to record components.
Limited Scope: Records are intended to be used as simple data carriers. If you need more complex behavior, such as methods that operate on the fields, you should use a regular class.

few good reads -

Records in java

Basu — Tue, 26 Mar 2024 05:25:54 +0000

A private final field for each component of the record.
A public constructor whose signature matches the record components.
Public read accessor methods for each component of the record, with the same name and type as the component.
Implementations of equals(), hashCode(), and toString() methods.

Here is an example of how to declare and use a record:
public record Employee(String name, int age) {}

This record class Employee has two components: name and age. You can create an instance of Employee and access its components like this:

Employee employee = new Employee("John Doe", 30); System.out.println(employee.name()); // prints "John Doe" System.out.println(employee.age()); // prints 30

Note that records are intended to be used as simple data carriers, so they are not suitable for every situation. If you need more complex behavior, you should use a regular class.

Records in Java, while useful for creating simple data carrier classes, do have some limitations:

No Inheritance: Records cannot extend any other class, and they are implicitly final, so no other class can extend a record.
Limited Instance Variables: All instance variables must be declared in the record header, and these variables are automatically final. You cannot declare additional fields in the body of the record.
No Argument-less Constructor: Records automatically provide a constructor with arguments for all fields, but they do not provide a no-argument constructor. This can be a limitation when working with frameworks that rely on no-argument constructors.
No Mutability: Since all fields in a record are final, records are inherently immutable. This means once a record object is created, you cannot change its state.
Limited Use of Annotations: Annotations that are applicable to classes, methods, or fields may not be applicable to record components.
Limited Scope: Records are intended to be used as simple data carriers. If you need more complex behavior, such as methods that operate on the fields, you should use a regular class.

few good reads -

Java groupingBy, Mapping, Counting

Basu — Sun, 24 Mar 2024 09:32:22 +0000

In my earlier post, we talked about various methods provided by java.util.stream.Collectors.
This post, I am going to talk about few other methods which are very powerful and useful.

Here is my previous post -
https://dev.to/bsknath/exploring-java-collectors-utilities-10ne

Exploring Java Collectors utilities

Basu — Fri, 22 Mar 2024 12:22:56 +0000

In Java, the java.util.stream.Collectors class provides a number of useful methods for performing reductions on streams. Here are some commonly used ones:

toList(): Collects all Stream elements into a List instance.

List<String> list = stream.collect(Collectors.toList());

toSet(): Collects all Stream elements into a Set instance.

Set<String> set = stream.collect(Collectors.toSet());

toMap(): Creates a Map instance from the Stream elements, with the first function serving as the map's keys and the second function as the values.

Map<String, Integer> map = stream.collect(Collectors.toMap(Function.identity(), String::length));

joining(): Concatenates all Stream elements into a String.

String joined = stream.collect(Collectors.joining(", "));

counting(): Counts the number of elements in the Stream.

Long count = stream.collect(Collectors.counting());

summingInt(), summingLong(), summingDouble(): Sums the Stream elements.

Integer sum = stream.collect(Collectors.summingInt(Integer::intValue));

maxBy(), minBy(): Finds the maximum or minimum Stream element according to a provided Comparator.

Optional<String> max = stream.collect(Collectors.maxBy(Comparator.naturalOrder()));

collectingAndThen(): Performs an additional finishing transformation.

List<String> unmodifiableList = stream.collect(Collectors.collectingAndThen(Collectors.toList(), Collections::unmodifiableList));

partitioningBy(): Partitions the Stream elements into a Map according to a Predicate.

Map<Boolean, List<String>> partitioned = stream.collect(Collectors.partitioningBy(s -> s.length() > 5));

These are just a few examples. The Collectors class provides many more utility methods for common tasks.

Stop, think about it. Are you using the right Data Structure?

Basu — Sat, 24 Jun 2023 04:40:19 +0000

A very trivial thing but knowing which data structure or implementation maintains the order of insertion, allows duplicates, or does not accept null helps prevent unexpected outcomes. Had a similar first-hand experience a few months ago. 💀

For example, lets take Map interface. There are three classes that implements Map in java - HashMap, LinkedHashMap and TreeMap.

♨️ HashMap doesn't maintain insertion order, LinkedHashMap does, TreeMap maintains order in sorted manner.
♨️ HashMap and LinkedHashMap allows one null key object (K,V), TreeMap doesn't (it may have multiple null values, never the keys).
♨️ HashMap and LinkedHashMap both have O(1) insertion and search. TreeMap, has O(logN). Bonus - TreeMap is implemented using RB-Tree, HM - LinkedLists, LHM - Doubley-LL. These implementation are the reason for these TC.

Coding standards that everyone must follow

Basu — Thu, 15 Dec 2022 18:23:05 +0000

Introduction

First rule of coding - write clean, beautiful and elegant code
Second rule of coding - write clean, beautiful and elegant code
Take time and effort to keep our code clean and pleasing for everyone to read
Follow these basic coding principles:
KISS – Keep It Simple, Stupid
SOLID Principles
Inversion of Control
DRY – Don’t Repeat Yourself.
YAGNI – You Ain’t Gonna Need It.
Law of Demeter.
Follow Boy’s Scout rule – leave code better than you found it.
Use “Design Patterns” whenever it makes sense to do it.
Avoid Premature Optimization – Premature Optimization is the root of all evil (or at least most of it) in programming. It adds unnecessary complexity in code and makes it hard to maintain in the long run. Always measure, analyze and then optimize.
Pay attention to all IDE generated warnings/suggestions

General

Favor composition over inheritance.
Create abstraction that separate higher level concepts from lower level details.
Variables and functions should be defined close to where they are used.
When a method uses getter/setter excessively from other class, consider moving that logic to that class.
Make sure all DTOs and Value Objects are immutable.
Avoid magic numbers, Hungarian notations in code. Use named constant instead of magic numbers

Core Java

Leverage Java 8’s stream API as much as you can instead of processing collection/array explicitly. For simple operations see if makes sense to process the collection without using stream API.
Use Lombok to generate boilerplate code. But do not overuse it to generate code that you don’t need
Avoid annotations like @SneakyThrow, @data, @synchronized from Lombok
Before using @Value annotation make sure you really need all functionality provided by @Value
Create new type instead of passing around complex collection types around e.g. instead of using List> create a type that represents that collection at higher abstraction
Use enums instead of constants
Investigate if there’s a reliable third-party library available before writing your own utility methods. Apache commons, Google Guava and Spring Utils are good example of third-party libraries that provides tons of useful functionality
Use meaningful and explanatory name that reveals the intention of variable/class/method
If name requires comment, it doesn’t reveal its intention. Consider re-naming such variables
Use long and descriptive name but do not use very long variable name that it starting to look like sentence
Avoid unnecessarily acronymization of variable name. A variable called “timeSheet” is much more descriptive and meaningful than “ts”
Class Names – Classes and Objects should have noun or noun phrase names. E.g. TimeSheet, WorkEntry. It should not be a verb.
Method Names – Method should have verb or verb phrase names. E.g. approveTimeSheet, submitWorkEntry
Event Class Names - Should be verb in past tense E.g.TimeSheetSubmitted

Comments

Avoid comments. We should strive at writing self-explanatory code that requires minimum to no comments. Ideally, our code should not have any comments. If your code needs commenting, you might be doing something wrong
Do not comment change history, user story in source code – it should go as part of your commit message. All meta-data about code like author, last modified time, US number, Defect Numbers should not be part of code
Do not use Javadoc comment that says nothing more than function signature
Javadoc style comments which are mainly used for documentation should be limited for library functionality
Never ever comment out code. Just delete it.
Comment should be used only when code cannot say for itself clearly

Functions

Functions should be small. This applies to code blocks like if, while, switch blocks etc.
Function should do one thing only
If your function has more than a few arguments, see if your method has tight coupling with other classes. If you do need those arguments, try to wrap them in separate class
Avoid Boolean flag as method parameter that is used as a flag to choose behavior in method
Return java.util.Optional instead of "return null" from method

Tests

Make sure your tests cover all scenarios specified.
Write unit test for everything that could possibly break
Use IDE coverage tool to make sure you are covering all critical code path and branches
Test all boundary conditions
Write test (or tests) for each reported bug