DEV Community: Jon Handler

The Million-Tenant Problem: Why Your OpenSearch Service Architecture Breaks at Scale

Jon Handler — Fri, 24 Jul 2026 20:41:20 +0000

Almost every search application is multi-tenant, whether or not anyone called it that. A store has product categories. A SaaS product has customers. A logging platform has teams, an e-commerce site has sellers, a B2B app has accounts. Each of those is a tenant: a slice of data, and the queries that go with it, that has to stay separate from every other slice. If you run Amazon OpenSearch Service, odds are you are already in the tenancy business.

And here is the thing about tenancy: it is almost free when you have a few tenants, and it quietly turns into the whole problem when you have a lot of them. A design that is clean and obvious at a hundred tenants can fall apart at a hundred thousand, for reasons that have nothing to do with the code being wrong. Take a concrete case. You are running a fintech app, peer-to-peer payments or a neobank, and millions of users each expect to open their phone, pull up six months of history, and see it instantly. That data behaves like logs, streaming in and aging out, but it has to perform like search, because a person is staring at a spinner. It lives in the gap between the two, and the conventional wisdom for either end does not fit.

I have watched teams wrestle with this exact problem. They stand up a textbook Amazon OpenSearch Service domain, follow every log-analytics best practice, demo it, and it screams. Beautiful on the happy path, right up until a certain scale. Then the queries slow down, the costs balloon, and the architecture that purred along at 10,000 tenants starts making the noise hardware makes right before the magic smoke escapes. The question I hear next is always the same: “We thought we were doing this right. What changed?” Usually nothing changed. The design followed the playbook faithfully, right down to copying the shard sizes from a blog post. The playbook just was not written for this workload, and scale is where that bill comes due. Let me walk through where the conventional wisdom runs out, and then the handful of moves that actually hold up at a million tenants.

Where the conventional wisdom runs out

The first trap is the pure-logging approach. The standard playbook says ingest fast, roll over indices daily, keep three days hot, then tier the rest off to cheaper storage. Shard size? Go big, 50 GB is fine, because you are optimizing for write throughput. This is excellent advice for application logs that nobody reads unless something is on fire.

It falls apart the moment you serve individual tenant queries at scale. Every query fans out across every shard in the index. A terabyte in 50 shards means one user tapping their phone becomes a single request fanning out to 50 servers, all to answer somebody checking last month’s coffee purchases. The cost of coordinating that scatter-gather swamps the benefit of parallelism, especially when each tenant’s data is a thin slice smeared across all 50 shards. The cluster was tuned for throughput, and now it is being asked for the one thing that tuning works against.

The second trap is the mirror image: the pure-search approach, giving every tenant its own index. Perfect isolation, fast queries, and it works right up until a few 10s of 1000s of tenants, at which point the cluster falls over under the weight of managing so many indices. OpenSearch Service tracks metadata for every index, shard, and replica, and the cluster state balloons until it becomes the bottleneck. Now cross-tenant queries are a coordination nightmare and the master node spends its life just keeping track of everything.

The third trap is the subtlest, and the easiest to walk into: assuming a tiering strategy built for logs will hold up for user-facing queries. Teams design a gorgeous multi-tier system: hot for recent data, UltraWarm for older data, maybe cold storage for compliance. Then they discover UltraWarm adds seconds of latency, because it pulls from S3 on demand. Seconds are fine for a forensic investigation. Seconds are a catastrophe when a user is staring at a spinner waiting for last spring’s transactions.

The conventional wisdom says pick your poison, optimize for cost or optimize for latency. At a million tenants you need both, which is exactly why the standard playbooks leave you stranded by the side of the road.

The right way to think about tenant distribution

The unlock is admitting that not all tenants are created equal, and your architecture should say so out loud. The move that changes everything is routing, in the broad sense: dedicate a slice of resources to match each slice of the workload, then get every request to the right slice. Everything below is a way to do that. Custom document routing does it inside one domain. Multi-domain and cell architectures do it across many. Pick the tool that fits the shape of your tenants.

Start with the within-a-domain tool, custom document routing. It is the right lever when you are pooling many tenants into shared indices, the case where tenants vastly outnumber indices. By default, OpenSearch Service spreads documents across shards with a hash function. Great for even distribution, terrible for tenant isolation. Custom document routing lets you supply your own hash key, usually a tenant ID, so all of a tenant’s data lands on one shard. When that tenant queries, the request hits one shard instead of all 50. One query, one shard, instead of fifty servers woken up to answer it. One shard is not the only option, either. Set index.routing_partition_size and a routing value maps to a subset of shards rather than a single one, spreading a larger tenant across, say, three shards instead of one. You give up a little of the single-shard win, since queries now fan out to the subset, in exchange for spreading a big tenant’s data and load. It is the same routing idea with a dial on it: size the slice of shards to the size of the tenant.

Do the arithmetic, because the arithmetic is the whole argument. A million small tenants on random distribution means every query touches every shard: a million queries, each firing 50 network calls. With routing, each query hits one shard. You cut the networking cost by around 98%, and latency drops with it. That is not a tuning tweak. It is the difference between a system that works and one that does not, and it is one of those rare architecture changes where the math is so lopsided it feels like cheating.

The nuance is that routing only earns its keep past a certain scale. The tipping point usually sits in the tens of thousands of tenants. Below that, fussing over routing keys is effort you will not get back. Above that, it is survival. I have watched this pattern work beautifully for financial-services companies with millions of end users, where each user’s data is small but has to come back now.

Pooling handles the small tenants, but the big ones need the opposite treatment, and that is where the hybrid model comes in. The idea is to dedicate a slice of resources to each large tenant independently of the small ones, so a whale’s spikes and volume never land on the shards the minnows share. Give that tenant its own dedicated index, sized and replicated for its workload, and its load is isolated from everyone else’s. Pool the long tail of small tenants into shared indices with custom routing. Silo the whales, pool the minnows, and match the size of the resource slice to the size of the tenant. A dedicated index is the within-a-domain way to hand a tenant its own slice. When a tenant, or a group of them, needs more isolation than one index inside a shared domain can give, the slice grows to a whole domain of its own.

Hybrid is a spectrum: routing within one domain at one end, a cell per group of tenants across many managed domains in the middle, and a collection per tenant on serverless at the other. That middle option is a cell architecture: each domain is a cell that owns one group of tenants, and a routing layer in front sends each request to the cell that holds its data. The Amazon Your Orders team built exactly that for semantic order-history search across billions of records, in improving order history search with semantic search. At the serverless end, Amazon OpenSearch Serverless can run a collection per tenant behind one regional endpoint, so routing is a request header and scale-to-zero means an idle tenant costs only storage, which I cover in multi-tenant search with OpenSearch Serverless next generation. Same logic throughout: silo the whales, pool the minnows, and pick the container that fits each tenant.

Mixed workloads create real tension in shard sizing, because you are optimizing for write throughput and read latency at the same time. Write-heavy wants more, smaller shards to parallelize ingestion. Read-heavy wants fewer, larger shards to keep queries from fanning out. When both happen at once, you land in the middle on purpose. My heuristic: aim for 30 to 40 GB per shard, rather than the 50 GB you would use for pure logs or the 10 to 30 GB you would use for pure search. Benchmark against your own workload, because your mileage will absolutely vary, but that middle band tends to balance ingestion against query performance. You are not optimizing for either extreme. You are optimizing for reality, where both matter.

Tiering has a better answer now, too. Writable warm storage runs on the OpenSearch Optimized OI2 instance family and is not the old UltraWarm. The big difference is not speed. Both tiers are S3-backed with a local cache, and on the NYC Taxis benchmark writable warm matches or beats UltraWarm on most query types, so call it comparable, not faster. The difference that matters is mutability. UltraWarm is read-only, so touching a single old document means migrating the index back to hot, writing, and migrating back, a warm-to-hot-to-warm round trip that can take a couple of hours per 100 GB. Writable warm lets you write directly to warm. Late-arriving data, compliance corrections, and backfills resolve in seconds instead of hours, with no migration and no hot-node cost. For a fintech workload where a six-month-old transaction occasionally has to be corrected, that is the whole ballgame: you keep the S3 cost profile and lose the read-only handcuffs.

Where this is heading

The patterns for multi-tenant log analytics are still evolving. More teams are converging on this hybrid shape: pooled tenants with routing, dedicated resources for the power users, and tiering driven by real access patterns instead of age alone.

The next frontier is dynamic routing that reacts to tenant behavior. Picture a system that promotes a tenant from pooled to dedicated the moment its query volume spikes, then quietly demotes it when things calm down, with no operator intervention. Or routing that respects geography and time zones, noticing that your Asia-Pacific users are asleep while your North American users are hammering the cluster.

If you are building one of these systems now, instrument everything. You need per-tenant visibility into query latency, shard distribution, and resource use, because the difference between an architecture that works and one that does not is usually hiding in the metrics nobody thought to collect. Start with the hybrid model. Turn on custom routing once you cross five-digit tenant counts. And do not assume a tiering strategy designed for logs will survive contact with a user-facing query.

The million-tenant problem is not unsolvable. It just refuses to be solved by the idealized architecture in the docs, the one that works great in the diagram and has never met real traffic. Build for the workload you actually have, not the one in the tutorial.

Your Search Engine Has Been Doing the Heavy Lifting (And You Never Noticed)

Jon Handler — Fri, 24 Jul 2026 20:40:38 +0000

You type a query into a box. Three milliseconds later the thing you wanted is sitting at the top of the results, and it feels like the machine read your mind.

It did not. Your words got torn into pieces, weighed against billions of other queries, matched against an inverted index, scored by algorithms juggling dozens of relevance signals, and reassembled into a ranked list, all before you finished blinking. The magic is a very fast, very elaborate act of misdirection, and we have spent thirty years getting better at the trick. It is genuinely delightful once you see how the trick works, and by the end of this post you will.

Now large language models and semantic search have walked on stage, and everyone has decided the trick is obsolete. Most people have this backwards. Before you can talk sensibly about where search is going, you have to be honest about what search already is.

I have built search engines for two decades: large-scale comparison shopping sites, then Amazon CloudSearch fifteen years ago, and the long road from there into Amazon OpenSearch Service. I have watched several revolutions arrive. This one is real. The reaction to it is mostly confused. So let me walk you through what search actually does, why the old parts are not going anywhere, and what genuinely changes when the thing typing the query stops being human.

The story the industry tells about “traditional” search

AI-era commentary often assumes lexical search, the kind where you match words to words, is a quaint pre-AI relic we are finally evolving past. That take is the kind of wrong that gets a production system operator paged at 3 a.m.

Start with the workhorse, the inverted index. Picture the index at the back of a cookbook. Instead of reading all 400 recipes to find the ones with chocolate, you flip to the back, look up “chocolate,” and get a list of page numbers. The inverted index does the same thing for every term in every field across billions of documents, which is the difference between a cookbook and a library that answers you instantly.

Building that index is more than stuffing words into a table. The engine parses text into meaningful units, because “New York” should be matched like one term and not two. It applies language rules, so “running” and “run” match together. It weighs rare terms over common ones, because matching “ergonomic” tells you far more than matching “the.” All of that careful work happens up front, at indexing time, so your query can be cheap later. Expensive now so it is fast forever is a good bargain, and it is not primitive.

Here is the deeper point, search was never really about matching. Matching is table stakes. Search is about relevance, and not all matches are equal. Pretending they are is how you (incorrectly, usually) rank a laptop sticker above an actual laptop. Decades of unglamorous, load-bearing refinement live in that ranking: BM25, TF-IDF, field boosting, recency weighting. This is also why a search engine is not just a database with a nicer query box. Databases optimize for transactional correctness, and they were never built to apply information science to rank results by relevance. Search engines trade away some of those transactional guarantees on purpose, in exchange for thousands of queries per second at millisecond latency and a scoring layer whose entire job is relevance. “Just add search to the database” is a kluge that demos well and dies in production.

Where embeddings and LLMs actually fit

So if lexical search is not primitive, where do embeddings and large language models fit? They are not the replacement. They are the extension, and understanding exactly what they add is where it gets fun.

Lexical search is excellent at explicit terms and genuinely bad at synonyms it was never told about. Search for “laptop” against a document that only says “notebook computer” and it shrugs. Embeddings fix that, and it helps to be precise about how, because the honest explanation is more interesting than the hand-wavy one. An embedding is a projection learned from word co-occurrence. “laptop” and “notebook computer” land near each other in vector space not because the model knows what either thing is, but because the two phrases show up surrounded by the same words across the training corpus. It is correlation over usage, not comprehension. That distinction matters, and it is also kind of wonderful: you get results that feel like understanding out of pure statistics about which words keep company with which.

That correlation is genuinely useful, because it retrieves documents that share no exact terms with the query. Embeddings do not evict the inverted index, though. They move in next door, and the combination has a name: hybrid search. Lexical matching gives you precision, the exact hits on product names and SKUs where being approximately right is being wrong. Embedding-based matching gives you recall, the related documents whose surrounding words overlap even when the query terms do not appear. You want both, because a system with only one fails in a way the other would have caught. OpenSearch Service runs both over the same documents and combines the scores.

One rule survives every wave of this, and if you take one practical thing from this post, take this one: start with the document, not the model. The document is what you retrieve and what you store, so define it first. If you want to retrieve products, your document is a product. It sounds too obvious to say, yet it is easy to skip, because there is always an existing database schema right there, ready to be shoehorned into a search index. Get the document right and everything downstream falls into place: which fields are searchable, which need exact matching, whether you need vectors at all.

The searcher is changing, and search is the foundation underneath

The next wave is already here, and it reframes what search is for. For thirty years, the thing on the other end of the query was a person. That single assumption shaped every design decision. A person is impatient, so you optimize for a fast first page. A person scans a ranked list and clicks one or two results, so a strong top ten is the whole game. A person tolerates “good enough,” because human eyes do the final filtering for free. Ten blue links works because a human knows which link to pick.

An agent is a different animal. It does not scan, it consumes: it fires queries in a loop and reads the whole result set, not the first few. An agent has infinite patience, so the human problem, getting the one right answer to the top before someone gives up, mostly goes away. What it lacks is taste. It cannot glance at result seven and think “close, but not that one.” So the game shifts. Ranking, the careful ordering of a list for a glancing human, matters less. Bringing back the right matches, without polluting the set with plausible-looking wrong ones, matters more. Ranking does not vanish, since something still has to choose which candidates fit the context window, but that reranking step is a topic for the follow-up post.

Agents also change the shape of the query. Instead of one fuzzy human query hoping a good ranking sorts it out, a task planner decomposes the work into tightly scoped retrievals, each sub-agent asking for exactly the matches its step needs. The agent will act on what comes back without a human eye to catch a bad match, so the results have to be precise, structured, and self-describing.

The abstraction worth holding onto is simple: the fundamentals of retrieval do not change, but the consumer does, and the consumer sets the goal. Search is the foundation the agentic wave is built on, not a casualty of it, because any agent that looks something up before it acts is standing on a search engine.

Agent-driven search is a big topic, and it deserves more than a section at the tail of a post about fundamentals. Retrieval for agents, result formats an agent can trust, and how retrieval-augmented generation changes when the “user” is a loop and not a person, all get their own post soon. For now, the useful move is simply to notice that “who is searching” is now a real design variable, and to stop assuming the answer is always a human.

Where this is heading

The interface to search is changing far faster than the machinery underneath. Chat feels revolutionary precisely because it hides the complexity, but pull the cover off and it is still retrieval, ranking, and relevance scoring, now with embedding-based correlation stirred in and, increasingly, an agent rather than a person reading the output.

The teams that build the best search over the next few years will not be the ones chasing every new model on release day. They will be the ones who understand the fundamentals, know where a semantic layer earns its keep, and ask who is actually on the other end of the query. So define your documents clearly. Use hybrid search. Measure relevance, not just latency. The tools keep changing, and that is the fun of it. The job, closing the gap between what a searcher wants and the information that answers it, does not.