DEV Community: Jon Handler

Forget Total Recall. Give Your AI Agent Selective Memory with OpenSearch.

Jon Handler — Tue, 11 Aug 2026 22:44:22 +0000

Imagine handing an LLM the full text of Anna Karenina and asking what Levin thinks about farming. The model has the information somewhere in its window. Good luck getting a focused answer. The number of tokens you can pass to an LLM in a single call can grow to 200K and beyond, but the problem is not capacity. The problem is that LLMs diverge when there is too much information. Relevance degrades with volume. Past a threshold, adding more context makes answers worse, not better. This is why "just make the window bigger" is not a memory strategy.

AI agents need memory that is selective, persistent, and searchable. Not a transcript. Not a sliding window. A system that extracts what matters, stores it durably, and retrieves only the right pieces when the agent needs them. This is a search problem, and Amazon OpenSearch Service now provides purpose-built APIs for exactly this: agentic memory that gives your agents persistent, semantically searchable recall across conversations.

The context window is not memory

LLMs process text in a fixed-size window. GPT-4o gives you 128K tokens. Claude gives you 200K. That sounds like a lot until you try to use it as a memory system. A single week of customer interactions for one user might run 50K tokens. A month blows past any context limit. And even if the window were infinite, retrieval latency and cost scale linearly with token count. You are paying for every token you stuff in there, and most of it is irrelevant to the current question.

The naive solution—append the last N messages—fails on both ends. If N is small, you lose information that matters. If N is large, you lose relevance. The agent drowns in a sea of context where the signal-to-noise ratio degrades with every turn. The customer said they prefer conservative investments six months ago, once. That single statement buried under 200 subsequent messages is invisible to a sliding-window approach.

What you actually want is not a transcript. You want a system that extracts the important pieces, stores them persistently, embeds them for semantic retrieval, and hands back only the facts that matter for the current moment. You want the agent to know that this customer prefers conservative investments without needing to re-read six months of transcripts. That is a search problem, not a context-window problem.

And search is what OpenSearch Service does.

OpenSearch Service agentic memory

OpenSearch Service provides a purpose-built agentic memory system through its ml-commons plugin. The system organizes memory into containers—one per agent or use case—and supports four distinct memory types that work together.

Sessions track conversation metadata: who participated, when the conversation started, what state the interaction reached. Think of sessions as the filing system. Each session is a distinct interaction context, and the memory system tags all stored information with session identifiers so retrieval can scope to the right timeframe.

Working memory holds the active conversation data and agent state during an ongoing interaction. Raw messages, execution traces, current task progress, and temporary data all live here. This is the short-term scratchpad your agent writes to during a session and reads from when the LLM needs immediate context.

Long-term memory is where the real value accumulates. When inference mode is enabled (infer: true), OpenSearch Service passes the conversation through an LLM that extracts key facts, preferences, and insights. Those extracted pieces are embedded as vectors and stored persistently. Six months later, when the same customer calls back, a semantic search against long-term memory surfaces "prefers conservative investments" without loading a single old transcript.

History maintains an audit trail of every memory operation—adds, updates, and deletes—across the container. This gives you the ability to trace how an agent's knowledge evolved and debug unexpected behavior.

Memory processing strategies

Raw storage is only part of the system. OpenSearch Service offers three processing strategies that automatically organize memories as they arrive. The SEMANTIC strategy groups related memories by meaning—conversations about retirement planning cluster together even if the words used differ from session to session. USER_PREFERENCE extracts explicit preferences ("I prefer email," "don't call before 10am") into dedicated long-term entries. SUMMARY creates condensed versions of sessions so your agent can skim a six-month relationship in a few paragraphs rather than re-reading every message.

These strategies run server-side. Your agent sends a conversation specifying infer: true, and OpenSearch Service handles the extraction, embedding, and organization. Your agent code stays focused on the conversation. You can swap embedding models or change strategies without touching agent logic.

Namespaces and isolation

Memory containers support namespaces that partition information by user, session, or agent instance. A contact center serving 100K customers uses one memory container with customer_id as the namespace. Every search is scoped to the current customer—logical isolation without the overhead of 100K separate indexes. The retrieval API combines semantic search with namespace filtering in a single call, so the agent gets back only memories that belong to the right customer and match the current query.

The retrieval loop

Here is what happens during a live conversation. The agent receives a customer message. Before responding, the agent searches long-term memory with the customer's namespace and the current query ("What did we discuss about retirement?"). OpenSearch Service returns the most semantically relevant memories. The agent includes those memories in its prompt alongside the current message. The LLM generates a response informed by real history, not a sliding window of raw text. After the session ends, the full conversation is stored back into the container with inference enabled, updating long-term memory for next time.

Memory makes the difference

The difference between an agent that remembers and one that does not is the difference between a service that builds relationships and one that processes transactions. A customer who has to re-explain their situation on every interaction feels unheard. An agent that picks up where the last conversation left off feels like a partner. OpenSearch Service provides the infrastructure that makes this possible: persistent, searchable, semantically organized memory that your agent can query in milliseconds.

The APIs are available today in OpenSearch 3.3 and later. If you are building agents that talk to the same people more than once—and that is most agents worth building—give your agent a memory that outlasts the context window. The agentic memory documentation covers the full API surface, and the contact center tutorial walks through a complete implementation.

Stop Making Your Database Pretend It Can Search

Jon Handler — Thu, 06 Aug 2026 22:47:10 +0000

Databases are optimized for durable storage: transactions, constraints, consistency, recovery. Amazon OpenSearch Service is optimized for a different job: language-aware search (matching, synonyms, typo tolerance, relevance ranking), deep vector capabilities for hybrid search that combines lexical matching with semantic similarity, and sub-second analytics at scale. Both store data reliably. But each excels at what the other merely tolerates. If your application needs both great transactional writes and great search, you run both. Amazon OpenSearch Ingestion and its zero-ETL integrations now keep them in sync automatically.

The hard part has always been the sync. Your product catalog lives in PostgreSQL. Your session data lives in DynamoDB. Your search experience lives in OpenSearch Service. A customer updates their address, a product goes out of stock, a price changes. That change needs to appear in search results within seconds. Continuously. Without a fragile pipeline that breaks when someone alters a column. OpenSearch Ingestion eliminates that pipeline entirely for Amazon Aurora (Aurora), Amazon RDS (RDS), and Amazon DynamoDB (DynamoDB), with native change data capture and near-real-time synchronization out of the box.

This post walks through why the sync problem is genuinely hard, how OpenSearch Ingestion solves it, and what to expect when you wire it up.

Why Keeping Two Systems in Sync Is Genuinely Hard

The most common first attempt is dual write: the application pushes every change to both the database and OpenSearch Service directly. This works at small scale and feels clean because there is no separate sync system. It breaks once traffic grows. At scale, you are driving thousands of simultaneous connections to OpenSearch Service from your application tier, each with its own overhead. Under load, one write succeeds and the other fails, leaving the two systems out of sync. Retries help, but now your application code is handling distributed transaction semantics that belong in infrastructure, not business logic. Dual write also cannot handle schema migrations, bulk backfills, or replaying historical changes.

The next step teams try is polling. Query the database on an interval, diff against what OpenSearch Service has, push the deltas. This decouples the sync from the write path, which is an improvement. But polling intervals are a tradeoff with no stable answer: too frequent wastes resources, too infrequent means stale results. And the rate of change in a production database is bursty. A product launch triggers thousands of updates in minutes, then nothing for an hour. No fixed interval handles both.

The next level up is change data capture (CDC). MySQL has binary logs. PostgreSQL has logical replication. Read the transaction log, extract the changes, push them to OpenSearch Service. This sounds clean until you start implementing it. You need to parse log formats that differ between database versions, handle schema changes without breaking the pipeline, manage replication slot offsets, deal with connection failures and retries, and ensure exactly-once delivery. What started as "just read the logs" becomes a distributed systems problem with its own failure modes.

Then there is the initial load problem. CDC handles ongoing changes, but what about your existing million rows? You cannot run a massive SELECT against production without locking tables or degrading performance. So you end up with two completely different sync mechanisms: one for the historical backfill, one for the stream. Each has its own failure modes, its own monitoring, its own on-call rotation.

Teams build this infrastructure. It works, mostly. Then someone adds a column and the pipeline breaks. Or a schema migration changes a type and the index mapping rejects the new documents. Or the sync develops a subtle bug that silently drops updates for days before anyone notices. The real cost is not building the pipeline. The real cost is maintaining it indefinitely while the schema underneath keeps evolving.

Amazon OpenSearch Ingestion Now Connects Directly to Your Database

Amazon OpenSearch Ingestion now has native integrations with Amazon Aurora (Aurora), Amazon RDS (RDS), and Amazon DynamoDB (DynamoDB). For Aurora and RDS, it supports MySQL (version 8+) and PostgreSQL (version 16+). For DynamoDB, it uses DynamoDB Streams with point-in-time recovery for initial snapshots. All three provide automatic change data capture and near-real-time synchronization. No Lambda functions, no Kafka clusters, no Glue jobs, no custom code.

The architecture solves both the initial load and the ongoing stream in one pipeline. When you create a pipeline, OpenSearch Ingestion starts with a full snapshot export to S3. This handles the backfill: your database is not locked, the export happens in the background, and OpenSearch Service ingests from S3 at whatever rate makes sense. No production impact.

Once the snapshot is loaded, the pipeline switches to streaming mode. For MySQL, it taps into binary logs (row format, full image). For PostgreSQL, it uses logical replication. Every insert, update, and delete in your database appears in OpenSearch Service within seconds. The pipeline handles offset management, connection recovery, and delivery guarantees. You do not write retry logic. You do not manage replication slots manually.

OpenSearch Ingestion pipelines are configuration-driven. You specify schema mappings, define data mutations (rename fields, drop columns, enrich documents in flight), and control delivery behavior. The pipeline buffers requests for up to 72 hours during downstream outages, retries failed deliveries automatically, and routes undeliverable documents to a dead-letter queue. Default schema mapping is automatic (product IDs become keyword fields, timestamps are typed correctly, text columns are analyzed for full-text search), but you have full control to override any of it through the pipeline configuration.

Our team tested this with a PostgreSQL database containing a product catalog (detailed in Integrating Amazon OpenSearch Ingestion with Amazon RDS and Amazon Aurora). Setup took about twenty minutes: enable logical replication, create a Secrets Manager entry for credentials, define the pipeline in the OpenSearch console. The initial sync happened in the background. After inserting a new record into PostgreSQL, it was searchable in OpenSearch Service within seconds.

That three-second latency is the part that matters. Not the setup. Not the configuration. The fact that changes flow continuously without anyone thinking about it. The pipeline is not a batch job you schedule. It is a live connection that stays current as your database evolves.

Constraints Worth Knowing

The database and your OpenSearch Service domain must be in the same AWS account and region. You can sync one database per pipeline. Multi-AZ database clusters are not supported yet. These are planning considerations, not dealbreakers for most search and analytics use cases. If you run a multi-region architecture, you will need one pipeline per region.

Setup is prerequisite-focused rather than code-focused. For MySQL, enable binary logging with row format and full image. For PostgreSQL, enable logical replication (available in version 16+). For DynamoDB, enable DynamoDB Streams (new and old images) and point-in-time recovery (PITR) for the initial snapshot. Store your database credentials in AWS Secrets Manager. Then define your pipeline using the visual builder in the OpenSearch console or a YAML configuration. You specify which tables to sync and where in OpenSearch Service the data should land. The pipeline handles everything else.

What This Changes

This integration does not solve every data synchronization problem. It solves one specific, common, and expensive one: making your database data searchable and analyzable in real time without building custom infrastructure. If you have a product catalog in RDS and need full-text search with relevance ranking, this is now a configuration task. If you have transactional data in Aurora and need real-time analytics dashboards, you no longer need a team maintaining Kafka and custom consumers. If you have session or user-profile data in DynamoDB and need it searchable alongside your relational data, the same pipeline model applies.

The s/ETL/configuration/ substitution here is real. Teams that were spending months on sync infrastructure can now spend that time on search relevance, query tuning, and the features their users actually see. The plumbing disappears. The search quality work can begin.

If you are currently running custom sync infrastructure between RDS, Aurora, or DynamoDB and OpenSearch Service, or if you have been postponing search capabilities because the integration cost was not worth the effort, the calculation just changed. Enable replication or streams on your database, point OpenSearch Ingestion at it, and your data flows. Continuously. In seconds. Without you maintaining anything in between.

Your Search Backend Speaks MCP Now

Jon Handler — Tue, 04 Aug 2026 01:54:30 +0000

Every MCP-compatible agent in the world needs the same three things from a search backend: discover what data is available, run queries against it, and get structured results back. Claude, Amazon Q, Cursor, Kiro, Strands Agents, and a growing list of open-source frameworks all speak MCP natively now. The protocol side is settled. Your search infrastructure can now speak it back.

Amazon OpenSearch Service now does. Your domain exposes a native MCP endpoint, and agents connect directly. No custom connectors, no middleware, no per-agent integration code. This post walks through what that looks like in practice: what the endpoint exposes, how to secure it, and why the M×N integration problem disappears when your data source speaks the same protocol your agents already understand.

MCP in 60 Seconds

The Model Context Protocol (MCP) is a standard JSON-RPC interface that lets AI agents discover and call tools on external systems. An MCP server advertises what it can do (search an index, check cluster health, run an aggregation), and any MCP-compatible agent can call those tools without custom integration code. One protocol replaces all the bespoke connectors.

The problem MCP solves is combinatorial. If you have M agents connecting to N data sources, custom integrations mean M×N connectors to build and maintain. Three agents talking to five OpenSearch Service domains means fifteen connectors. Each one handles authentication, query formatting, and response parsing in its own way. Add a sixth domain or a fourth agent, and the cycle starts over. MCP collapses that to M+N: each agent speaks one protocol, each data source exposes one server, and any agent can talk to any server without additional code.

Think of it like USB. Before USB, every peripheral needed its own cable and driver. After USB, you plug in and it works. MCP is that standardization applied to the AI integration layer. Your agents are the peripherals. Your data sources are the computer. MCP is the port.

What the MCP Server Exposes

Your OpenSearch Service domain exposes the MCP endpoint at /_plugins/_ml/mcp as part of the ML Commons plugin. Agents connect directly. The endpoint advertises three types of components. Resources provide data context from your indexes. Prompts are reusable instruction templates for recurring analyses. Tools are executable functions: searching indexes, checking cluster health, analyzing performance metrics, running aggregations.

Here is what a tool call looks like. A Python agent connecting with fastmcp:

from fastmcp import Client

async with Client("https://your-domain.us-east-1.es.amazonaws.com/_plugins/_ml/mcp") as client:
    # discover what the domain exposes
    tools = await client.list_tools()
    print([t.name for t in tools])  # ['SearchIndexTool', 'ListIndexTool', 'ClusterHealthTool', ...]

    # call a tool by name
    result = await client.call_tool("SearchIndexTool", {
        "index": "products",
        "query": {"match": {"category": "electronics"}}
    })

The agent discovers available tools, calls them by name, and gets structured results back. No custom SDK, no REST client boilerplate. Any MCP-compatible agent (Amazon Q CLI, Claude, Cursor, Strands Agents) connects the same way.

The Setup (Where It Gets Fun)

For the built-in endpoint, there is nothing to configure on the server side. Your domain already exposes the MCP endpoint. What matters is getting authentication right: IAM roles and backend role mapping determine what each agent can see. Once those are in place, connecting a new agent is a configuration change. I built a domain, enabled MCP, registered tools, and used the SearchIndexTool to query my data on OpenSearch Service. The agent discovered available tools through the protocol's capability negotiation and ran queries without any custom integration code.

Access requires two layers. First, an IAM resource-based policy that lets the agent role reach the domain:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::123456789012:role/ai-agent-role"
    },
    "Action": ["es:ESHttpGet", "es:ESHttpPost"],
    "Resource": "arn:aws:es:us-east-1:123456789012:domain/my-domain/*"
  }]
}

Second, with fine-grained access control enabled, map the IAM role to an OpenSearch backend role that has permissions on the ML Commons APIs and the indexes your agent needs to search. In OpenSearch Dashboards, go to Security > Roles, create or choose a role with cluster permissions for ml_full_access (or a narrower custom permission set), add index permissions for the target indexes, then map your agent IAM role ARN to that backend role under Mapped users. Every agent that assumes the same IAM role inherits the same access. One security boundary instead of one per connector.

Scaling Without Rebuilding

The built-in MCP endpoint scales with your OpenSearch Service domain. If your domain handles the query load, the MCP endpoint handles it too. No separate scaling layer to worry about. For teams using the AgentCore-hosted path, AgentCore handles auto-scaling independently.

Adding a new AI agent takes nothing on the server side. The agent connects to your domain’s MCP endpoint and discovers available tools automatically through the protocol’s built-in capability negotiation. This is the M+N property in practice: each new agent is O(1) work, not O(N).

The M+N Payoff

Consider a team with four AI agents connecting to three OpenSearch Service domains. Under the old model, that is twelve custom integrations. With MCP, each agent points at the domain’s endpoint and discovers tools automatically. Adding agent number five is a configuration entry, not a development sprint. Adding a fourth domain means the existing agents can reach it immediately. The complexity stays linear.

The open-source OpenSearch MCP server is part of the OpenSearch project. Community-driven improvements and security updates mean you are not maintaining proprietary integration code. And because the built-in MCP endpoint on your OpenSearch Service domain uses the same protocol, agents that work with one path work with the other—no code changes required.

If your agents already speak MCP, your OpenSearch Service domain is ready to answer. Enable the ML Commons plugin (set plugins.ml_commons.mcp_server_enabled to true), register the tools you want agents to access, configure IAM and backend role mapping, and point your agent at the /_plugins/_ml/mcp endpoint. The second agent costs nothing. The tenth agent costs nothing. The protocol does what protocols are supposed to do: make the next connection free.

"Most Of Your Vectors Are Steerage. Why Are They In First Class?"

Jon Handler — Sat, 01 Aug 2026 00:39:24 +0000

I was on a call last month with a startup CTO who had just gotten their AWS bill. They had built a beautiful RAG application: semantic search, conversational AI, the works. Their vector index was humming along with about 50 million embeddings. Then they hit product-market fit.

Within six weeks, they scaled to 500 million vectors. Their monthly infrastructure costs went from $2,000 to $20,000. The real kicker? When we looked at the access patterns, over 80% of those vectors were queried less than once a week. They were paying hot-storage prices for data that was, by any honest measure, cold.

The standard advice here is "just use a cheaper vector database." The more interesting question is: why are you storing all your vectors at the same temperature in the first place?

The Cost-Recall-Latency Triangle

Vector search forces a three-way tradeoff. You can optimize for cost, recall, and latency, but you only get to pick two. Want high recall and low latency? That costs money (in-memory HNSW graphs with full-precision vectors eating RAM). Want high recall at low cost? Latency goes up. Want cheap and fast? Recall suffers.

Most teams pick a single point on this triangle and apply it uniformly to every vector in their index. That decision made sense when vector databases offered a single storage tier. It makes the same amount of sense as storing your entire filesystem on NVMe SSDs because some files need fast access.

The conventional wisdom says you pick your point on the triangle and live with it. But the conventional wisdom was written before vector storage got interesting. The better approach: tier your vectors the same way you already tier your storage. Different access patterns deserve different economics. The same embedding that costs $0.12/month in RAM might cost $0.004/month on disk and $0.0002/month in object storage. When you have 500 million of them, those decimals matter.

The Hot Tier: In-Memory HNSW and Exact k-NN

For vectors that get hit constantly (your user-facing search, your real-time recommendations, anything in the critical path of a page load), Amazon OpenSearch Service stores HNSW graphs entirely in native memory. On a single r6g.8xlarge node with 113 million vectors at 1,024 dimensions (source), this configuration delivered P90 latency of 25 ms and recall of 0.95 at 300 queries per second. Throughput scales linearly with replica shards.

There is also a case the HNSW literature tends to gloss over: exact k-NN. Exact k-NN compares every candidate by brute force rather than navigating a graph. When your query includes filters (a tenant ID, a category, a date range) that reduce the candidate set below roughly 100,000 vectors, exact k-NN outperforms approximate search. The brute-force scan finishes faster than an HNSW traversal at that scale, uses less RAM (no graph to build or maintain, no ef_construction tuning, no ef_search parameter), and returns perfect recall.

The hot tier is the right home for vectors with high query frequency, latency requirements under 50 ms, or both.

Quantization methods (FP16, binary quantization, product quantization) give you further control over the RAM-versus-recall tradeoff at each tier. That topic deserves its own article.

The Warm Tier: On-Disk Mode (Where Most of Your Vectors Should Live)

Here is the part of this story that deserves more attention than it gets.

OpenSearch Service's on-disk mode keeps a quantized navigation graph in RAM and stores the full-precision vectors on SSD. The default is binary quantization at 32× compression (each float dimension collapses to a single bit), but you can choose 2×, 4×, 8×, or 16× to trade more RAM for higher recall. When a query arrives, OpenSearch walks the compressed graph in memory, identifies candidates, then rescores against the full vectors on disk.

On the same 113M-vector benchmark (source), on-disk mode at 8× compression delivered P90 latency of 96 ms with 0.98 recall. At 32×, latency was 104 ms with 0.94 recall. Compare that to the in-memory tier's 25 ms. You trade 70-80 ms of latency for a memory reduction approaching two thirds: 1 million vectors at 256 dimensions drop from 1.31 GB (FP32) to 0.18 GB for the navigation graph, plus the derived source optimization saves up to two-thirds more by deduplicating vector storage across replicas.

Most workloads belong here. In a RAG pipeline where LLM generation takes 2-3 seconds, 100 ms for vector retrieval is invisible. The only workloads that need 25 ms retrieval are the ones in the critical path of a user-facing page load with tight SLAs.

To enable on-disk mode,

The Cold Tier: S3 Vectors for Archive Scale

For vectors that are accessed rarely, Amazon S3 Vectors provides native vector storage and search at S3 economics. You specify "engine": "s3_vector" in your index mapping on OpenSearch 2.19+ with O-series instances, and OpenSearch manages the rest. OpenSearch offloads vector data to S3, keeps metadata on the cluster for filtering, and routes the k-NN portion of each query to S3 transparently. From the application's perspective, you query the same _search API you use for in-memory and on-disk indexes.

Response times are sub-second (500-800 ms typical), and storage costs drop by up to 70% compared to in-memory indexes. S3 Vectors uses pay-per-query pricing, so you never pay for idle capacity. That model is ideal for large datasets with low to moderate query traffic (up to thousands of queries per day). For workloads with sustained high throughput (hundreds of queries per second), on-disk mode is the better complement because its costs are capacity-based. The two tiers work together. S3 Vectors handles large, infrequently accessed data, and on-disk mode handles data with steady query traffic.

S3 Vectors also works as a standalone vector store outside the OpenSearch integration. For teams using S3 Vectors independently, a one-click export moves data into OpenSearch Serverless, where queries run at sub-200ms latency. This lets teams take advantage of OpenSearch Serverless throughput when access patterns shift and cold data needs to become hot.

The Decision Framework

Where each vector lives should follow from two questions: how often does the vector get queried, and what latency does the caller need?

Vectors that get hit constantly and need sub-30ms responses belong in memory with in-memory HNSW on OpenSearch Service (FP16 recommended). Exact k-NN is worth testing when query filters narrow candidates below 100,000, which is common in multi-tenant applications. Vectors with steady query traffic where 100ms latency is acceptable belong on disk with 32× compression. Think internal search tools, batch RAG pipelines, analytics, product catalogs. On-disk mode is the default tier for most production workloads at scale. Vectors that are large in volume, accessed at low to moderate frequency, and tolerant of 500-800ms latency belong in S3 Vectors with the "engine": "s3_vector" integration: historical archives, compliance retention, seasonal catalogs outside their season. If access patterns change, a one-click export promotes cold data to OpenSearch Serverless.

The beauty of this tiering is that the calling application does not need to know about it. Every tier exposes the same OpenSearch _search API. The application sends a query. OpenSearch routes the query to the right storage layer based on which index you target. Migrations between tiers are operational decisions, not application code changes.

What Happened to the $20,000 Bill

The CTO from my opening story did the access-pattern audit. About 15% of their 500 million vectors were hot: they powered the real-time search and needed in-memory speed. Around 60% were warm: knowledge-base vectors for their RAG pipeline where 100 ms was invisible inside a 3-second LLM call. The remaining 25% were seasonal product embeddings from marketing campaigns that had ended months ago.

They moved the warm vectors to on-disk mode and the seasonal vectors to S3 Vectors. Their monthly bill dropped by roughly two-thirds without changing a single query from the application layer. The application still called the same _search API. The tiering was invisible to the calling code.

If you are running a vector workload today, the one thing worth doing this week is pulling up your access-pattern metrics. Look at which embeddings get queried at high frequency versus which ones are sitting idle. The answer tells you exactly which tier each vector belongs in. The math tends to be obvious once you look, and the savings tend to surprise people who assumed all their vectors needed to be hot.

What does your vector access-pattern distribution look like? I am curious whether others are seeing the same 80/20 split between rarely-touched and actively-queried embeddings.

Fix Your Search, Fix Your RAG Output

Jon Handler — Mon, 27 Jul 2026 20:23:49 +0000

I watched a customer demo their new RAG application last month. They had spent three months building it. The interface was beautiful. The LLM responses were eloquent and well-formatted. And almost every answer was subtly, confidently wrong.

“The model keeps hallucinating,” they told me, frustrated. They had tried three different LLMs. They had adjusted temperature settings. They had rewritten their prompts a dozen times. But here is what nobody had looked at yet: the search layer feeding context into those prompts.

This is the part of RAG that deserves more attention than it gets: most quality problems are retrieval problems. When your generative AI makes things up, it is usually because the context it received was off. Too much, too little, or just plain irrelevant. The LLM is doing exactly what you asked: generating fluent text based on the information you provided. If that information misses the mark, well, garbage in, eloquent garbage out.

The challenge is that retrieval tends to get treated as a solved problem. Chunk the documents, throw everything into a vector database, retrieve the top 10 results, done. That works for demos. In production, though, you need the system to distinguish between a precise question about part number XJ-447 and an abstract question about how plumbing works. That distinction is where things get genuinely interesting.

The Part of Chunking That Trips People Up

Here is something I find fascinating about document chunking: it looks like a technical problem, but it is actually a semantic one.

The conventional wisdom goes something like this: take your documents, split them into 512-token chunks with 50-token overlap, generate embeddings, done. Clean. Systematic. And it often destroys retrieval quality in ways that are genuinely hard to diagnose.

Think about what you are actually doing when you chunk a document. You are deciding what the atomic unit of meaning is for your system. That 512-token window might slice right through the middle of a critical explanation. It might mash together two unrelated concepts that happened to share a page. Your chunks are not just storage units. They are the answers your system will retrieve.

I have seen teams spend weeks optimizing their embedding models while using paragraph-based chunking that splits “The solution to this problem is…” from the actual solution in the next paragraph. Once you see it, the fix is obvious. But it is easy to miss because chunking happens early in the pipeline and its effects show up late, as vague or wrong answers from the LLM.

The approach that works: chunk semantically, not mechanically. If you are working with structured documents, and most enterprise content is structured, respect that structure. A section about pricing belongs together. A troubleshooting procedure should stay intact. A paragraph explaining a concept should not be split because it crossed some arbitrary token threshold.

In my own work, I use a tiered strategy: first split at structural markers (headings, section breaks), then embed sliding sentence windows and split at valleys in cosine similarity between adjacent windows. Those valleys are the points where the topic naturally shifts. This technique (well described visually here) is now built into LangChain and LlamaIndex as their default semantic chunkers. The embedding-based valley detection catches topic boundaries that structural markers miss, without the cost of sending every chunk through an LLM.

Here is the related subtlety: not every chunk needs to be retrieved independently. Sometimes you want to retrieve a full document, like when someone needs that PDF about 1957 nail prices. In those cases, use chunks as subdocuments with a parent-child relationship. The chunks help you find the right document, but you return the whole thing. Other times, especially in RAG scenarios, you want the chunk itself. You are not looking for a document about plumbing. You want the specific paragraph that explains how a P-trap works.

Most teams pick one approach and apply it everywhere, which explains why their system is great at some queries and puzzling at others.

The Precision Problem That Makes Things Interesting

Here is something about vector search that is genuinely counterintuitive once you see it: vectors never return zero results.

That is not a bug, it is math. In vector space, everything has a nearest neighbor. Ask a question about quantum physics to a database of cooking recipes, and it will cheerfully return the five “most relevant” results. They will be completely wrong, but the distance metrics will look reasonable.

This is especially tricky in RAG systems because the LLM will work with whatever you give it. Feed it those irrelevant cooking recipes as context for a quantum physics question, and it will generate a confident-sounding answer that blends both domains into nonsense. The user has no signal that the underlying search missed.

Lexical search does not have this problem. If your keywords do not match, you get nothing back. That absence is actually useful information. When someone asks about part number XJ-447, you want exact matching. You want lexical search. When they ask about “solutions for preventing pipe corrosion in coastal environments,” you want semantic search to correlate terms by the company they keep.

The right approach is not choosing between lexical and semantic search. It is knowing when to use each, and increasingly, using both together. Hybrid search strategies that combine keyword matching with semantic correlation consistently outperform either approach alone.

But you need something to orchestrate that decision. Is this a precise query or an abstract one? Does it contain specific identifiers that demand exact matching? Are there domain-specific terms that semantic search might miss? Most teams try to solve this with increasingly complex query logic. The more interesting move is letting an agent figure it out.

Building Search That Actually Works for RAG

So what does a properly architected RAG search layer look like?

Start with the retrieval target. Before you chunk anything, ask: what am I trying to retrieve? If you are building a document discovery system, your target is documents. Chunk for findability, but return the full document. If you are building a question-answering system, your target is the specific piece of information that answers the question. Chunk for semantic coherence, and return just that chunk.

For RAG specifically, smaller and more semantically coherent beats larger and more comprehensive. You are not trying to give the LLM everything that might be relevant. You are trying to give it exactly what is relevant. A tightly focused paragraph that directly addresses the question will outperform a kitchen-sink approach that includes three pages of tangentially related content.

This is where reranking enters the picture, and where it helps to be precise about what reranking actually does. Reranking is not a magic quality boost you toggle on. It is a precision tool. If your initial retrieval is pulling back mostly irrelevant results, reranking just reorders them. The conventional wisdom says to over-sample: retrieve 50 results, rerank them, take the top 10. Think about what that means. You are deliberately retrieving low-relevance results (because that is what results 11 through 50 usually are) and hoping the reranker will find hidden gems.

Sometimes that works. More often, you are adding latency and cost while marginally improving already-poor results. The better approach: write queries that retrieve high-precision results in the first place, then use reranking to fine-tune the order. If you need to retrieve 50 results to get 10 good ones, the query strategy is where the leverage is.

This is also where agentic frameworks start to make sense. Instead of building the perfect query upfront, let an agent run multiple query strategies, evaluate the results, and decide what to pass forward. Amazon OpenSearch Service has internal agentic capabilities that can orchestrate this: running lexical and semantic searches in parallel, applying different ranking strategies, and using an LLM to judge which results actually answer the question.

That last point is becoming the new standard for search quality evaluation. Traditionally, measuring precision and recall required humans to manually review results. That does not scale. But an LLM can evaluate whether retrieved chunks contain information relevant to the query, and it can do it in real time. This lets you build systems that are self-critical, that recognize when search results are poor and try a different approach rather than confidently generating wrong answers.

What This Means Going Forward

The RAG systems that work in production are not the ones with the fanciest LLMs or the most sophisticated prompts. They are the ones that treat retrieval as a first-class problem worthy of the same engineering attention as the generative layer.

We are moving away from the “embed everything, retrieve top-k, generate answer” pipeline toward systems that reason about queries, orchestrate multiple search strategies, and critically evaluate their own results before generating responses. The search layer is becoming intelligent, not just functional.

If you are building or fixing a RAG system this week, try starting not with the LLM but with a simple question: when I retrieve context for this query, am I getting back the specific information that actually answers it? Pull up your logs, look at what is being retrieved, and check whether a human could answer the question from that context alone.

Because if a human cannot, the LLM will not either. It will just be more eloquent about being wrong. And that is a problem worth the s/prompt engineering/actual engineering/.

The Million-Tenant Problem: Why Your OpenSearch Service Architecture Breaks at Scale

Jon Handler — Fri, 24 Jul 2026 20:41:20 +0000

Almost every search application is multi-tenant, whether or not anyone called it that. A store has product categories. A SaaS product has customers. A logging platform has teams, an e-commerce site has sellers, a B2B app has accounts. Each of those is a tenant: a slice of data, and the queries that go with it, that has to stay separate from every other slice. If you run Amazon OpenSearch Service, odds are you are already in the tenancy business.

And here is the thing about tenancy: it is almost free when you have a few tenants, and it quietly turns into the whole problem when you have a lot of them. A design that is clean and obvious at a hundred tenants can fall apart at a hundred thousand, for reasons that have nothing to do with the code being wrong. Take a concrete case. You are running a fintech app, peer-to-peer payments or a neobank, and millions of users each expect to open their phone, pull up six months of history, and see it instantly. That data behaves like logs, streaming in and aging out, but it has to perform like search, because a person is staring at a spinner. It lives in the gap between the two, and the conventional wisdom for either end does not fit.

I have watched teams wrestle with this exact problem. They stand up a textbook Amazon OpenSearch Service domain, follow every log-analytics best practice, demo it, and it screams. Beautiful on the happy path, right up until a certain scale. Then the queries slow down, the costs balloon, and the architecture that purred along at 10,000 tenants starts making the noise hardware makes right before the magic smoke escapes. The question I hear next is always the same: “We thought we were doing this right. What changed?” Usually nothing changed. The design followed the playbook faithfully, right down to copying the shard sizes from a blog post. The playbook just was not written for this workload, and scale is where that bill comes due. Let me walk through where the conventional wisdom runs out, and then the handful of moves that actually hold up at a million tenants.

Where the conventional wisdom runs out

The first trap is the pure-logging approach. The standard playbook says ingest fast, roll over indices daily, keep three days hot, then tier the rest off to cheaper storage. Shard size? Go big, 50 GB is fine, because you are optimizing for write throughput. This is excellent advice for application logs that nobody reads unless something is on fire.

It falls apart the moment you serve individual tenant queries at scale. Every query fans out across every shard in the index. A terabyte in 50 shards means one user tapping their phone becomes a single request fanning out to 50 servers, all to answer somebody checking last month’s coffee purchases. The cost of coordinating that scatter-gather swamps the benefit of parallelism, especially when each tenant’s data is a thin slice smeared across all 50 shards. The cluster was tuned for throughput, and now it is being asked for the one thing that tuning works against.

The second trap is the mirror image: the pure-search approach, giving every tenant its own index. Perfect isolation, fast queries, and it works right up until a few 10s of 1000s of tenants, at which point the cluster falls over under the weight of managing so many indices. OpenSearch Service tracks metadata for every index, shard, and replica, and the cluster state balloons until it becomes the bottleneck. Now cross-tenant queries are a coordination nightmare and the master node spends its life just keeping track of everything.

The third trap is the subtlest, and the easiest to walk into: assuming a tiering strategy built for logs will hold up for user-facing queries. Teams design a gorgeous multi-tier system: hot for recent data, UltraWarm for older data, maybe cold storage for compliance. Then they discover UltraWarm adds seconds of latency, because it pulls from S3 on demand. Seconds are fine for a forensic investigation. Seconds are a catastrophe when a user is staring at a spinner waiting for last spring’s transactions.

The conventional wisdom says pick your poison, optimize for cost or optimize for latency. At a million tenants you need both, which is exactly why the standard playbooks leave you stranded by the side of the road.

The right way to think about tenant distribution

The unlock is admitting that not all tenants are created equal, and your architecture should say so out loud. The move that changes everything is routing, in the broad sense: dedicate a slice of resources to match each slice of the workload, then get every request to the right slice. Everything below is a way to do that. Custom document routing does it inside one domain. Multi-domain and cell architectures do it across many. Pick the tool that fits the shape of your tenants.

Start with the within-a-domain tool, custom document routing. It is the right lever when you are pooling many tenants into shared indices, the case where tenants vastly outnumber indices. By default, OpenSearch Service spreads documents across shards with a hash function. Great for even distribution, terrible for tenant isolation. Custom document routing lets you supply your own hash key, usually a tenant ID, so all of a tenant’s data lands on one shard. When that tenant queries, the request hits one shard instead of all 50. One query, one shard, instead of fifty servers woken up to answer it. One shard is not the only option, either. Set index.routing_partition_size and a routing value maps to a subset of shards rather than a single one, spreading a larger tenant across, say, three shards instead of one. You give up a little of the single-shard win, since queries now fan out to the subset, in exchange for spreading a big tenant’s data and load. It is the same routing idea with a dial on it: size the slice of shards to the size of the tenant.

Do the arithmetic, because the arithmetic is the whole argument. A million small tenants on random distribution means every query touches every shard: a million queries, each firing 50 network calls. With routing, each query hits one shard. You cut the networking cost by around 98%, and latency drops with it. That is not a tuning tweak. It is the difference between a system that works and one that does not, and it is one of those rare architecture changes where the math is so lopsided it feels like cheating.

The nuance is that routing only earns its keep past a certain scale. The tipping point usually sits in the tens of thousands of tenants. Below that, fussing over routing keys is effort you will not get back. Above that, it is survival. I have watched this pattern work beautifully for financial-services companies with millions of end users, where each user’s data is small but has to come back now.

Pooling handles the small tenants, but the big ones need the opposite treatment, and that is where the hybrid model comes in. The idea is to dedicate a slice of resources to each large tenant independently of the small ones, so a whale’s spikes and volume never land on the shards the minnows share. Give that tenant its own dedicated index, sized and replicated for its workload, and its load is isolated from everyone else’s. Pool the long tail of small tenants into shared indices with custom routing. Silo the whales, pool the minnows, and match the size of the resource slice to the size of the tenant. A dedicated index is the within-a-domain way to hand a tenant its own slice. When a tenant, or a group of them, needs more isolation than one index inside a shared domain can give, the slice grows to a whole domain of its own.

Hybrid is a spectrum: routing within one domain at one end, a cell per group of tenants across many managed domains in the middle, and a collection per tenant on serverless at the other. That middle option is a cell architecture: each domain is a cell that owns one group of tenants, and a routing layer in front sends each request to the cell that holds its data. The Amazon Your Orders team built exactly that for semantic order-history search across billions of records, in improving order history search with semantic search. At the serverless end, Amazon OpenSearch Serverless can run a collection per tenant behind one regional endpoint, so routing is a request header and scale-to-zero means an idle tenant costs only storage, which I cover in multi-tenant search with OpenSearch Serverless next generation. Same logic throughout: silo the whales, pool the minnows, and pick the container that fits each tenant.

Mixed workloads create real tension in shard sizing, because you are optimizing for write throughput and read latency at the same time. Write-heavy wants more, smaller shards to parallelize ingestion. Read-heavy wants fewer, larger shards to keep queries from fanning out. When both happen at once, you land in the middle on purpose. My heuristic: aim for 30 to 40 GB per shard, rather than the 50 GB you would use for pure logs or the 10 to 30 GB you would use for pure search. Benchmark against your own workload, because your mileage will absolutely vary, but that middle band tends to balance ingestion against query performance. You are not optimizing for either extreme. You are optimizing for reality, where both matter.

Tiering has a better answer now, too. Writable warm storage runs on the OpenSearch Optimized OI2 instance family and is not the old UltraWarm. The big difference is not speed. Both tiers are S3-backed with a local cache, and on the NYC Taxis benchmark writable warm matches or beats UltraWarm on most query types, so call it comparable, not faster. The difference that matters is mutability. UltraWarm is read-only, so touching a single old document means migrating the index back to hot, writing, and migrating back, a warm-to-hot-to-warm round trip that can take a couple of hours per 100 GB. Writable warm lets you write directly to warm. Late-arriving data, compliance corrections, and backfills resolve in seconds instead of hours, with no migration and no hot-node cost. For a fintech workload where a six-month-old transaction occasionally has to be corrected, that is the whole ballgame: you keep the S3 cost profile and lose the read-only handcuffs.

Where this is heading

The patterns for multi-tenant log analytics are still evolving. More teams are converging on this hybrid shape: pooled tenants with routing, dedicated resources for the power users, and tiering driven by real access patterns instead of age alone.

The next frontier is dynamic routing that reacts to tenant behavior. Picture a system that promotes a tenant from pooled to dedicated the moment its query volume spikes, then quietly demotes it when things calm down, with no operator intervention. Or routing that respects geography and time zones, noticing that your Asia-Pacific users are asleep while your North American users are hammering the cluster.

If you are building one of these systems now, instrument everything. You need per-tenant visibility into query latency, shard distribution, and resource use, because the difference between an architecture that works and one that does not is usually hiding in the metrics nobody thought to collect. Start with the hybrid model. Turn on custom routing once you cross five-digit tenant counts. And do not assume a tiering strategy designed for logs will survive contact with a user-facing query.

The million-tenant problem is not unsolvable. It just refuses to be solved by the idealized architecture in the docs, the one that works great in the diagram and has never met real traffic. Build for the workload you actually have, not the one in the tutorial.

Your Search Engine Has Been Doing the Heavy Lifting (And You Never Noticed)

Jon Handler — Fri, 24 Jul 2026 20:40:38 +0000

You type a query into a box. Three milliseconds later the thing you wanted is sitting at the top of the results, and it feels like the machine read your mind.

It did not. Your words got torn into pieces, weighed against billions of other queries, matched against an inverted index, scored by algorithms juggling dozens of relevance signals, and reassembled into a ranked list, all before you finished blinking. The magic is a very fast, very elaborate act of misdirection, and we have spent thirty years getting better at the trick. It is genuinely delightful once you see how the trick works, and by the end of this post you will.

Now large language models and semantic search have walked on stage, and everyone has decided the trick is obsolete. Most people have this backwards. Before you can talk sensibly about where search is going, you have to be honest about what search already is.

I have built search engines for two decades: large-scale comparison shopping sites, then Amazon CloudSearch fifteen years ago, and the long road from there into Amazon OpenSearch Service. I have watched several revolutions arrive. This one is real. The reaction to it is mostly confused. So let me walk you through what search actually does, why the old parts are not going anywhere, and what genuinely changes when the thing typing the query stops being human.

The story the industry tells about “traditional” search

AI-era commentary often assumes lexical search, the kind where you match words to words, is a quaint pre-AI relic we are finally evolving past. That take is the kind of wrong that gets a production system operator paged at 3 a.m.

Start with the workhorse, the inverted index. Picture the index at the back of a cookbook. Instead of reading all 400 recipes to find the ones with chocolate, you flip to the back, look up “chocolate,” and get a list of page numbers. The inverted index does the same thing for every term in every field across billions of documents, which is the difference between a cookbook and a library that answers you instantly.

Building that index is more than stuffing words into a table. The engine parses text into meaningful units, because “New York” should be matched like one term and not two. It applies language rules, so “running” and “run” match together. It weighs rare terms over common ones, because matching “ergonomic” tells you far more than matching “the.” All of that careful work happens up front, at indexing time, so your query can be cheap later. Expensive now so it is fast forever is a good bargain, and it is not primitive.

Here is the deeper point, search was never really about matching. Matching is table stakes. Search is about relevance, and not all matches are equal. Pretending they are is how you (incorrectly, usually) rank a laptop sticker above an actual laptop. Decades of unglamorous, load-bearing refinement live in that ranking: BM25, TF-IDF, field boosting, recency weighting. This is also why a search engine is not just a database with a nicer query box. Databases optimize for transactional correctness, and they were never built to apply information science to rank results by relevance. Search engines trade away some of those transactional guarantees on purpose, in exchange for thousands of queries per second at millisecond latency and a scoring layer whose entire job is relevance. “Just add search to the database” is a kluge that demos well and dies in production.

Where embeddings and LLMs actually fit

So if lexical search is not primitive, where do embeddings and large language models fit? They are not the replacement. They are the extension, and understanding exactly what they add is where it gets fun.

Lexical search is excellent at explicit terms and genuinely bad at synonyms it was never told about. Search for “laptop” against a document that only says “notebook computer” and it shrugs. Embeddings fix that, and it helps to be precise about how, because the honest explanation is more interesting than the hand-wavy one. An embedding is a projection learned from word co-occurrence. “laptop” and “notebook computer” land near each other in vector space not because the model knows what either thing is, but because the two phrases show up surrounded by the same words across the training corpus. It is correlation over usage, not comprehension. That distinction matters, and it is also kind of wonderful: you get results that feel like understanding out of pure statistics about which words keep company with which.

That correlation is genuinely useful, because it retrieves documents that share no exact terms with the query. Embeddings do not evict the inverted index, though. They move in next door, and the combination has a name: hybrid search. Lexical matching gives you precision, the exact hits on product names and SKUs where being approximately right is being wrong. Embedding-based matching gives you recall, the related documents whose surrounding words overlap even when the query terms do not appear. You want both, because a system with only one fails in a way the other would have caught. OpenSearch Service runs both over the same documents and combines the scores.

One rule survives every wave of this, and if you take one practical thing from this post, take this one: start with the document, not the model. The document is what you retrieve and what you store, so define it first. If you want to retrieve products, your document is a product. It sounds too obvious to say, yet it is easy to skip, because there is always an existing database schema right there, ready to be shoehorned into a search index. Get the document right and everything downstream falls into place: which fields are searchable, which need exact matching, whether you need vectors at all.

The searcher is changing, and search is the foundation underneath

The next wave is already here, and it reframes what search is for. For thirty years, the thing on the other end of the query was a person. That single assumption shaped every design decision. A person is impatient, so you optimize for a fast first page. A person scans a ranked list and clicks one or two results, so a strong top ten is the whole game. A person tolerates “good enough,” because human eyes do the final filtering for free. Ten blue links works because a human knows which link to pick.

An agent is a different animal. It does not scan, it consumes: it fires queries in a loop and reads the whole result set, not the first few. An agent has infinite patience, so the human problem, getting the one right answer to the top before someone gives up, mostly goes away. What it lacks is taste. It cannot glance at result seven and think “close, but not that one.” So the game shifts. Ranking, the careful ordering of a list for a glancing human, matters less. Bringing back the right matches, without polluting the set with plausible-looking wrong ones, matters more. Ranking does not vanish, since something still has to choose which candidates fit the context window, but that reranking step is a topic for the follow-up post.

Agents also change the shape of the query. Instead of one fuzzy human query hoping a good ranking sorts it out, a task planner decomposes the work into tightly scoped retrievals, each sub-agent asking for exactly the matches its step needs. The agent will act on what comes back without a human eye to catch a bad match, so the results have to be precise, structured, and self-describing.

The abstraction worth holding onto is simple: the fundamentals of retrieval do not change, but the consumer does, and the consumer sets the goal. Search is the foundation the agentic wave is built on, not a casualty of it, because any agent that looks something up before it acts is standing on a search engine.

Agent-driven search is a big topic, and it deserves more than a section at the tail of a post about fundamentals. Retrieval for agents, result formats an agent can trust, and how retrieval-augmented generation changes when the “user” is a loop and not a person, all get their own post soon. For now, the useful move is simply to notice that “who is searching” is now a real design variable, and to stop assuming the answer is always a human.

Where this is heading

The interface to search is changing far faster than the machinery underneath. Chat feels revolutionary precisely because it hides the complexity, but pull the cover off and it is still retrieval, ranking, and relevance scoring, now with embedding-based correlation stirred in and, increasingly, an agent rather than a person reading the output.

The teams that build the best search over the next few years will not be the ones chasing every new model on release day. They will be the ones who understand the fundamentals, know where a semantic layer earns its keep, and ask who is actually on the other end of the query. So define your documents clearly. Use hybrid search. Measure relevance, not just latency. The tools keep changing, and that is the fun of it. The job, closing the gap between what a searcher wants and the information that answers it, does not.