DEV Community: Jon Handler

A Nightmare on Hollow Lane. Dedicated Cluster Managers Save the Day

Jon Handler — Thu, 29 Oct 2026 14:03:05 +0000

Three houses, a lane full of trick-or-treaters, and a Halloween night with no one in charge.

An Amazon OpenSearch Service domain does two kinds of work. It answers the requests your applications send, and it runs itself: choosing a leader, deciding which node holds which shard, and publishing every change to the cluster state. When a busy domain slows down, the usual fix is more or bigger data nodes, but the slowdown often comes from the second kind of work landing on the same nodes as the first. I first wrote a version of this story for my colleagues, and Halloween is the right night to bring it back, because it makes the answer easy to see: three small dedicated cluster manager nodes that give the domain its own command.

Halloween night on Hollow Lane

Every Halloween, the village of Hollowmere pools all of its candy into one shared stash and hands it out from three houses on Hollow Lane. The stash is far too big for one porch, so it sits in bowls spread across Kael's house, Mira's house, and Dorn's house. Every bowl has a twin on a different porch, so a knocked-over bowl never means lost candy. Any host can answer any knock. When a visitor wants something from a bowl on another porch, the host at the door shouts down the lane, the other hosts check their bowls, and the host hands over the whole haul.

The visitors were the real challenge of the night, and their costumes told you what each one wanted. Ghosts drifted up with sacks of candy to add to the stash, and every piece had to land in the right bowl and in that bowl's twin. Pirates came hunting treasure: one particular kind of candy, wherever on the lane it happened to sit. Vampires, who in the old stories can never resist counting, wanted only tallies, such as how many chocolate bars were left, broken down by brand. They arrived by the hundreds, and the rule of the night was old and strict. Answer fast with a treat, or get a trick.

The hosts had the candy, the bowls, and plenty of energy. What the lane did not have was anyone in charge of the night.

For the first hour the hosts kept pace. Then the village council came out with lanterns and orders, and with no one else to take them, the council went straight to the hosts. Some orders changed how the night worked. “Start a new bowl just for gummy bats!” called the mayor. “Every candy needs a label saying whether it has nuts!” said the baker. “Two pieces per pirate from now on!” cried the schoolteacher. Each order meant all three hosts had to stop, take it in, and change how they worked, in the middle of answering the door.

Other orders were bigger, and they cost more. “Who is in charge out here?” demanded the blacksmith, so the hosts had to stop and vote on a captain. “Which house keeps the caramel apples?” asked the miller, and they had to agree, then shuffle bowls when the next councilor disagreed. Then Dorn's porch light flickered out, and the other two had to work out which twin bowls now counted as the real ones and where to set out fresh twins, while the doorbells kept ringing.

The visitors did not wait. Ghosts stood on the steps holding sacks no one had time to sort. A pirate waited while Kael shouted down the lane about bowl assignments instead of caramel apples. A vampire waiting on his tally gave up and started counting the gravel. Every visitor who waited too long played a trick, and by midnight Hollow Lane was draped in toilet paper. The hosts had not failed at handing out candy. The night had failed because no one was running it.

Translating Hollow Lane

In Amazon OpenSearch Service, Hollow Lane is your domain and the three houses are its data nodes. The candy is your data, the bowls are shards, and the twin bowls are replicas, placed on different nodes so that losing one node never loses data. The visitors are the requests your applications send: ghosts are indexing requests that add documents, pirates are search queries, and vampires are aggregations. Any data node can take a request and act as its coordinator, sending it to whichever nodes hold the relevant shards and gathering their answers into one response. A treat is a fast response. A trick is a slow or failed one.

The council's orders are the domain's management work. The tactical orders are changes to the cluster state: a new bowl for gummy bats is creating an index, the nut labels are a mapping update, and two pieces per pirate is a settings change. Each change produces a new version of the cluster state that every node has to receive and apply. The strategic orders are cluster management itself: electing the cluster manager, deciding which node holds which shard, noticing when a node goes dark, promoting replicas to take its place, and computing every new version of the cluster state. Those are jobs for the domain as a whole, not for any one node. Without dedicated managers, the data nodes are the only candidates, so they do this work between requests.

On a quiet lane that overhead barely registers. The trouble grows with the domain. As shard counts climb, mappings get larger, and indexes come and go more often, the council's orders multiply, and data nodes spend more time on management and less on requests. Latency rises, indexing throughput drops, and the tricks pile up. Under heavy load it can turn into a loop: data nodes get too busy to apply cluster state updates promptly, the elected manager retries, the retries add load, and the added load slows the data nodes further.

The witches in the bell tower

The next Halloween, Hollow Lane had someone in charge. Three witches climbed the bell tower at the end of the lane. They handed out no candy and answered no doors. From the tower they could see every porch, and they kept the master list of which bowl sat where. Every councilor with an order now climbed the tower stairs instead of knocking on a host's door.

Among themselves, the witches ran the night. One led, and the other two stood ready to take over if she fell. They decided which porch kept which bowl and moved bowls only when the lane changed. They watched every porch light, and when one went out they decided which twins took over and where the new twins went. They turned the council's requests into one clear instruction at a time and called it down the lane along with a fresh copy of the master list. The hosts heard each call, adjusted, and went back to the door, and because every house had the latest list, a host always knew which porch to shout to when a pirate asked for caramel apples. Nobody on the lane stopped to vote, and the toilet paper stayed at the store.

Dedicated cluster managers in practice

Dedicated cluster manager nodes are the witches: the command role for the whole domain, held by nodes that do nothing else. They hold no data, serve no queries, and index no documents. Their job is coordination: tracking every node, index, and shard, maintaining routing information, computing a new cluster state when indexes or nodes come and go, publishing that state to every node, and sending heartbeats to confirm the data nodes are healthy. The published cluster state is the master list. Every data node gets a copy, which is how a coordinating node knows where to send each request. Data nodes still apply each update, the way the hosts adjusted to each call from the tower, but they no longer compute it or vote on who leads. You will find the setting under its older name in the console, where these are still called dedicated master nodes.

For production domains, AWS recommends three dedicated managers, and the reason is quorum. Only one manager is active at a time, while the other two wait as standbys. If the active one fails, the remaining two still form a majority and elect a replacement without pulling in the data nodes. Two managers cannot do that, because losing one leaves no majority, and without quorum the domain fails both reads and writes. Four buys nothing over three and can lose quorum when an Availability Zone goes down, which is why AWS never recommends an even number. When you deploy with Multi-AZ with Standby, the service provisions the three managers across three zones for you.

Because managers coordinate rather than store, you size them by how much they have to track, not by how much data you hold. The AWS guidance scales manager memory with the number of nodes and shards in the domain. On OpenSearch 2.17 and later, an 8 GB manager handles up to 30 nodes and 15,000 shards, and a 32 GB manager handles up to 120 nodes and 60,000 shards. Domains with many shards or large field mappings may need the next size up. Either way, the managers cost a small fraction of the data tier, which is a modest price for the stability they buy back.

Bringing it together

An Amazon OpenSearch Service domain has two kinds of work: answering the requests your applications send, and running the domain itself. Without dedicated cluster managers, nobody is assigned the second job, so the data nodes answer the door, carry out every change to the cluster state, and argue out who leads and where each shard lives, all at once. Dedicated cluster manager nodes give the domain its command. They take the management work away from the data nodes and turn every change into a published update the data nodes only have to apply. Three of them give you quorum, so a failed manager is replaced without the data nodes stopping to vote, and they run on instances sized to the domain's node and shard count rather than its data volume. If you run anything close to a production workload, put your witches in the tower before the first knock.

Stop Switching Tabs: Uplevel The SRE Chat Window with Amazon OpenSearch Service MCP Apps

Jon Handler — Tue, 27 Oct 2026 16:58:17 +0000

Your investigation agent finds the root cause in seconds. Then you open four browser tabs to check whether it is right. MCP Apps bring the evidence into the chat.

Your investigation agent can hand you a root-cause hypothesis about as fast as you can read the page. A hypothesis is not yet a fix, and before you act on one in the middle of an incident you need to see the evidence for yourself. That usually means opening the observability UI in a browser, finding the trace waterfall, pulling up the service map to scope the impact, and checking each view against what the agent said, ping-ponging between tabs while the pressure builds. The agent saved you the query time and left you the tab-switching. Amazon OpenSearch Service MCP Apps move that verification into the chat thread, so the evidence arrives next to the claim. This post builds on Implement Agent Observability: Extend Your OpenTelemetry Stack on Amazon OpenSearch Service, which covers how agent telemetry joins the OpenTelemetry stack you already run.

The proof arrives with the answer

A standard MCP tool call returns text. The agent reasons over it, and you go somewhere else to check it. MCP Apps extend the protocol with a second response channel. If a standard tool call is stdout only, an MCP App writes to two streams: a structured text summary for the agent, and an interactive visualization that your chat client renders in the conversation thread for you.

A trace investigation shows how the two halves divide the work. The agent passes a trace ID, or a service name and time range, and the tool returns a summary with the trace ID, total duration, span count, critical path, and where the failure started. The agent uses that text to choose its next step, such as pulling the log entries around the failing span. Meanwhile the same response renders a trace waterfall in the thread, and you can expand spans, inspect their attributes, and confirm the root cause yourself. OpenSearch UI builds that visualization server-side by running the query against your data, so the chart is deterministic. You are looking at the actual query result, not the model's description of it.

An MCP App renders a checkout trace waterfall in the chat thread, with the span hierarchy, span details, the AI analysis, and suggested next questions in one view. The chat client, Goose in this case, renders both the text and interactive graphic response.

When checkout errors spike, the whole investigation can start from one question about the cause. The agent queries the alerts, correlates logs with traces, and checks the service map, and its reply carries those views alongside its explanation. The waterfall above comes from exactly that kind of session. The analysis points at CartService/EmptyCart failing with a FailedPrecondition error after about 15 seconds, and the span hierarchy beneath it lets you trace the failure from the load generator through checkout to the cart service yourself. The suggested follow-ups at the bottom, such as showing the logs for this trace or checking cart service health, keep the next step in the same thread, whether that is digging deeper, drafting the incident summary, or starting the remediation.

Setting up the local server

The architecture is a local MCP server that sits between your chat client and an OpenSearch UI application. OpenSearch UI is the serverless interface for observability, and it already connects to your OpenSearch Service domains, OpenSearch Serverless collections, and Amazon Managed Service for Prometheus workspaces. Each tool call goes from your client to the local server, which signs the request with your AWS credentials and forwards it to the OpenSearch UI endpoint. The server runs on your machine, and your data stays in your AWS account. You need an OpenSearch UI application with an Observability workspace connected to at least one data source, Node.js 22 or later, and credentials that allow es:ESHttpGet and es:ESHttpPost on the application. Supported clients include Claude Desktop, VS Code with GitHub Copilot, Cursor, Goose, and ChatGPT.

Download and extract the server package, then add it to your client's MCP configuration:

{
  "mcpServers": {
    "opensearch-observability-stack-mcp": {
      "command": "node",
      "args": ["/path/to/opensearch-observability-stack-mcp/server/server.js"],
      "env": {
        "OS_UI_ENDPOINT": "application-abc123.us-west-2.opensearch.amazonaws.com",
        "AWS_REGION": "us-west-2",
        "AWS_PROFILE": "my-profile"
      }
    }
  }
}

The endpoint is the Application URL on the Applications page of the Amazon OpenSearch Service console. Restart the client and ask it to list the available observability data sources. If your domains, collections, or Prometheus workspaces come back, the connection works. The setup creates no cloud resources, so removing it means deleting the configuration entry and the local directory. The launch post walks through each step.

The chat window becomes the console

The previous post made the case for bringing application and agent telemetry together, with logs and traces in OpenSearch Service and metrics in Amazon Managed Service for Prometheus. MCP Apps are where that unification pays off for the person on call. The signals still live in several stores, but the SRE no longer has to remember which store holds which signal or which UI shows it. The agent does that routing, and the chat thread becomes one surface over every source. In that sense the agent is the new user experience for observability: you describe what you need, and the right query against the right store comes back with the right view.

Behind that surface, each tool handles a different stage of an investigation and returns the view that suits it. Triage tools surface active alerts, correlate them across data sources, and break them down by severity so the agent knows where to start. Log tools search for error patterns and cluster similar entries into a failure signature. Trace tools find the distributed trace and draw the span hierarchy and latency breakdown. To size the impact, metric tools run PromQL with threshold analysis, service performance tools report rate, errors, and duration for each service, and topology tools draw the service map with call volume and error rate on every edge. Dynamic visualization tools build line, bar, area, and metric charts from queries you describe, and a set of specialized tools covers the agents themselves, tracing LLM calls and rendering agent trace maps, alongside cluster health reports and instrumentation scoring that finds gaps in your telemetry coverage.

With those tools behind it, the console stops being a wall of dashboards you navigate and becomes a conversation you drive. You ask what is wrong, and the waterfall that shows it arrives with the answer. You ask how far it spreads, and the service map renders with the failing edges marked. You ask for the error rate since the last deploy, and the chart is built for that window when you ask, rather than waiting on a dashboard someone had to anticipate.

Trust in that conversation depends on what sits underneath it. The agent works through your permissions, and every result reports its own status. An empty result means the query finished and found nothing, which is different from a timeout, a partial scan, or a permission denial, and that status carries through each step of the investigation. The same thread can also carry the response. A runbook for a known failure can live as a skill the agent runs, so the procedure and the data it depends on show up together instead of in a wiki tab.

This approach suits teams that run observability at scale and want to own the stack: large telemetry volumes, engineers who build their own tooling, and a preference for open, extensible components. If you already run OpenSearch for logs or search, the MCP server works against the same engine and the same data you already query.

Bringing it together

Agents made root-cause analysis fast, which left verification as the slow step. Amazon OpenSearch Service MCP Apps close that gap by returning each answer twice: as text the agent reasons over, and as a deterministic visualization you can inspect in the same thread. Setup takes a local server, your existing OpenSearch UI application, and your AWS credentials, and the tools cover an investigation from the first alert to the service map. For SREs working alongside agents that investigate at machine speed, the chat window is no longer a lighter view of the observability console. It is the console.

Implement Agent Observability: Extend Your OpenTelemetry Stack on Amazon OpenSearch Service

Jon Handler — Thu, 22 Oct 2026 17:53:12 +0000

You already observe your applications. Agent observability is the same stack, reaching new signals and giving you a new way to investigate.

You get paged at 2am and work backward from a symptom: which service is erroring, what changed, which log line explains it. That loop is familiar, and your OpenTelemetry stack already feeds it. What is new is that the application now includes agents that reason, call tools, and chain steps, so the same page can turn on what an agent retrieved, which tool call stalled, or why it looped. Amazon OpenSearch Service and its observability stack cover both the agent's execution and the application's behavior in one place, so that investigation stays a single thread instead of splitting across tools. Here is how that comes together.

The signal extends

Consider what changes when the thing you are debugging is an agent. You are no longer only asking which service failed. You are asking what the agent retrieved, which tool call stalled, why it looped, and how that cascaded into the application work it triggered. That is new territory for a 2am investigation. It sits on top of the old territory rather than replacing it. The applications you already run do not go away; they gain a new kind of participant whose decisions you now have to see.

Start with what you are now collecting. Agents do not just emit logs. A single agent request can carry repeated prompts, retrieved context, tool arguments, and results across several iterations, and multi-agent or ReAct workflows push that further still, generating many times the data of a simple application trace. To see what happened inside one, you need the full execution chain: every model call, every tool invocation, every reasoning step. That is agent tracing, and the first thing to understand about it is that it extends what you already collect rather than replacing it.

The extension is mechanical, not conceptual. Agent traces arrive as OpenTelemetry spans, the same shape as the application spans you already ingest, so they land in the same store and join the same investigation. That shared landing place is the whole reason this works, because an agent's outcome usually cannot be explained from the agent trace by itself. The reason a task failed often surfaces in a downstream service, an application log, or an infrastructure metric that the agent never sees.

Picture the 2am page again. The agent returned a wrong answer, and its trace shows a tool call that took four seconds and then a retry. On its own that is a shrug. Joined to the downstream spans it triggered, the HTTP call to a search service and the database query behind it, through the identifiers propagated across all of them, the four seconds resolves into a specific slow query on a specific shard. One investigation, not two tools and a spreadsheet of timestamps. Tracing is only the entry point, too: evaluating whether the agent's answers were any good, and analyzing its behavior across sessions, read from this same execution data once it is flowing.

The economics tighten

Cost control is not new and not unique to agents; any team ingesting logs and traces at volume already decides what to keep hot, what to tier, and what to drop. What agents change is the scale, because their loops repeat system prompts, tool schemas, retrieved context, and prior results on every pass, and the volume climbs fast enough that a loose retention design turns expensive in a hurry. The architecture that answers this is the one you want for all of your observability data anyway. Incident triage needs full-text search across prompts, tool results, and logs, a grep for the one line that explains a failure. Fleet analysis needs columnar aggregation over latency, token spend, and error rates across handoffs. Both run on the same retained data, and intelligent tiering moves older data to cheaper storage on its own. It is the same unified retrieval idea that lets an agent read text, vectors, and time series through one interface, turned toward operations, so you hold one copy of the truth instead of two systems whose versions drift apart.

The actor changed

The deepest shift is who is asking. Investigation agents are becoming first-line consumers of operational data, and that is the part that upgrades the stack rather than merely extending it. An agent is good at inferring causes, but only from signals it can trust, and that puts a new demand on the interface it queries: results and their status have to be unambiguous. An empty result has to mean the query completed and found nothing, distinct from a timeout, a partial scan, a stale handle, or a permission denial. A human reads a half-empty chart and senses something is off; an agent takes the bytes at face value, so if the store conflates those cases the agent reasons confidently from a false premise. That status has to travel with the result through every step, because a wrong signal early becomes a confident wrong conclusion later.

This takes nothing away from how you work today. You keep investigating visually when that is the fastest path, and the agent works the same data through an interface built for a machine reader. Teams run many observability tools, so some data will always live outside any single store. The investigation has to cross those sources without copying everything into one system first. What you add is additive: the same OpenTelemetry data you already trust, now also legible to an agent that can act on it.

Standing it up

Putting this in place is mostly a matter of pointing instrumentation you already run at stores built for it. OpenSearch observability ingests OpenTelemetry natively. Logs and traces land in Amazon OpenSearch Service as the store for full-text search and correlation, while metrics go to Amazon Managed Service for Prometheus as the store for time-series analysis, and OpenSearch UI visualizes both together so a metric spike and the traces behind it sit on one screen. The stack correlates an agent step to the application work it triggered and tiers data by age so retention stays affordable. Because the ingestion path is OpenTelemetry, reaching from application observability to agent observability means sending the new spans, not re-platforming: you keep your query languages, your dashboards, and your pipelines, and the agent signal joins them.

Bringing it together

Put the three shifts together and the picture is simple. The signal extends, because agent traces are OpenTelemetry spans that land beside the application telemetry you already keep. The economics stay the ones you already manage, made more urgent by the volume an agent loop generates. And the investigator can now be an agent, which is what turns a bigger haystack into faster answers instead of slower ones. None of that asks you to replace your observability practice. It asks you to point the instrumentation you already run at a store built to hold all of it, and to let an agent read what it finds.

That is the foundation the next step builds on. Once application and agent telemetry live together and an agent can query them and trust the results, a chat interface can sit over the whole thing and the investigation moves into a conversation. See Implement Agent Observability: Extend Your OpenTelemetry Stack on Amazon OpenSearch Service on Medium for a fuller treatment, with a deep dive to come.

Connect your Strands agent to Amazon OpenSearch Service

Jon Handler — Thu, 15 Oct 2026 15:09:09 +0000

Your agent is only as good as what it can look up. Point it at OpenSearch Service and give it hybrid search as a tool.

An AI agent reasons well and knows little. The model is frozen at training time, so anything specific to your business, your product catalog, your support history, your internal docs, lives outside it. The agent gets that knowledge the same way an application does: it searches for it. The question is what it searches, and how well.

Amazon OpenSearch Service is the shortest path to that knowledge source. It already holds your text, and it can run lexical and semantic search over the same index, so an agent can retrieve by keyword and by word-context similarity in a single call. This post wires a Strands agent to Amazon OpenSearch Serverless with two short pieces of code: the agent itself, and a hybrid search tool it can call. Serverless means you do not size or manage a cluster; you create a collection, point the agent at it, and let it scale on its own.

The agent is a handful of lines

In Strands, the model is the agent. You hand it a set of tools, and it reasons about when to call them, reads what they return, and keeps going until it has an answer. Amazon Bedrock backs it by default, so there is no model object to construct up front. What you write is small: an import, a system prompt that tells the agent to ground its answers in what it retrieves, the tool it can call, and the question.

from strands import Agent, tool

agent = Agent(
    tools=[opensearch_hybrid_search],
    system_prompt="Answer using the knowledge base. Retrieve before you answer, and cite what you used."
)
agent("What changed in our refund policy last quarter?")

Your application receives a user's question, pulls together whatever it already knows, the session, the user's recent activity, any context it is holding, and hands that to the agent. The agent decides it needs to look something up, calls the knowledge-base tool, reads the passages that come back, and grounds its answer in them rather than in whatever the model happened to memorize at training time. Agents rarely stop at one tool call, though; they loop, calling a tool, reasoning over what it returned, and calling again until they have enough to answer. Think of it as a Unix pipe with judgment: question | retrieve | reason | answer, except the agent runs that loop itself and decides how many times to go around before it responds.

The tool is a hybrid query against OpenSearch Service

The tool runs a hybrid query with two clauses. One is lexical, a BM25 match that locks onto exact terms, product codes, and names. The other is neural, a clause that ranks documents by how closely their word-context patterns align with the query's, so it surfaces relevant passages whose wording differs from the words the user typed. A search pipeline normalizes the two score scales and combines them, so a single request returns one ranked list drawn from both signals.

@tool
def opensearch_hybrid_search(query: str) -> list[str]:
    """Search the knowledge base for passages relevant to the query.
    Use for any question about company-specific information."""
    body = {
      "query": {"hybrid": {"queries": [
        {"match": {"text": {"query": query}}},
        {"neural": {"text_embedding": {
            "query_text": query, "model_id": MODEL_ID, "k": 10}}}
      ]}}
    }
    resp = client.search(index="kb", body=body,
                         params={"search_pipeline": "hybrid-pipeline"})
    return [h["_source"]["text"] for h in resp["hits"]["hits"]]

Two fields on the index make this work: text holds the raw passage for the lexical clause, and text_embedding holds its vector for the neural clause. The search pipeline, created once on the collection, carries the normalization and combination settings so the query body stays this small. The tool hands back a list of passage strings, which is exactly what the agent needs to read and cite.

One point worth stressing, because it is where hybrid search quietly fails: both legs have to be good. Hybrid search amplifies every signal you give it, so a noisy lexical configuration produces noisy combined results. Adding a semantic clause does not paper over a weak BM25 setup. Tune the keyword side as carefully as the vector side, and the combination rewards you; neglect it, and the combination carries the noise straight through.

Getting the embeddings in: the easy path and the direct path

The neural clause needs a vector for each document, and you have two ways to produce them. The easy path is automatic semantic enrichment: you enable it on a text field, and Amazon OpenSearch Service expands that field into a sparse vector with a service-managed model at ingestion, with no embedding pipeline to wire up and no model to host. Because the model runs at ingest and queries are only tokenized, the matching clause for this path is a neural_sparse query rather than the dense neural clause shown above. For most teams this is where to start, because it removes the step that usually stalls a first vector project.

The direct path is to supply your own dense vectors. You run your embedding model, write the vectors into a knn_vector field yourself, and query them with the dense neural clause, keeping full control over the model, its dimensions, and when you re-embed. Reach for it when you have a specific model you need to use or you are already generating embeddings elsewhere in your stack. The hybrid structure is the same either way; the one clause that changes is whether you match the enriched field with neural_sparse or the dense field with neural.

Point the agent at the knowledge source

This post covers the bare-bones interaction between the application and the agent, giving you the idea on how to create and use an agent. In a real production system, the lexical clause would grow filters, boosts, and field weights tuned to your data instead of a bare match. The agent would carry more than one tool: guardrails that keep it inside policy, a step that evaluates whether the retrieved passages are good enough before it answers, and often separate lexical and vector tools with a planning step that decides which to call for a given question. You might split retrieval across several indexes, add reranking, or let the agent reformulate a query that came back thin.

An agent with no knowledge source guesses; an agent with a good one answers. An Amazon OpenSearch Serverless collection holds your knowledge, automatic semantic enrichment gets the vectors in without an embedding pipeline, and one hybrid search tool gives the agent lexical precision and semantic reach in a single call. The agent code is a dozen lines, the tool is a dozen more, and the retrieval underneath them is the part that decides whether the answers are any good. Build the knowledge source well, and the agent gets smarter for free.

Dial in your search with User Behavior Insights: from clicks to accurate results

Jon Handler — Tue, 13 Oct 2026 13:20:06 +0000

Users grade your search results by acting on them. Write down the grades yourself, and use them to make the results better.

Every action a user takes on a search result, a click, a hover, an add-to-cart, a purchase, tells you whether the ranking was right. Most search systems let that signal fall on the floor. Capture it instead and you get a continuous stream of relevance judgments, generated by the people who actually know what they were looking for, at zero labeling cost. That stream is what turns relevance tuning from guesswork into measurement.

The problem is that most teams have no structured record of what their users do. They know the query volume and maybe the zero-result rate, but they cannot answer the question a data scientist actually asks: for this query, which results did we show, and which ones did the user click? Without the causal link between a query, the results it returned, and the actions taken on those results, every relevance decision is an opinion.

User Behavior Insights (UBI) is the OpenSearch answer to that gap. Introduced in OpenSearch 2.15 and built on an open specification, UBI is a schema and a collection path for recording search behavior: the queries users submit, the results shown, and the actions taken on those results, all linked together. This post covers what UBI captures and how to collect it in Amazon OpenSearch Service. More to the point, it shows how to turn that data into results that move the numbers you care about: more of the purchases, answers, and signups your search is supposed to produce, fewer dead-end queries that send customers elsewhere, and retrieval your agents can actually trust.

What UBI actually records

UBI separates two kinds of records and ties them together. Query records capture what was searched and what the engine returned, with an identifier for the query. Event records capture what the user did next: a click, a scroll, an add-to-cart, a conversion, each tagged with the same query identifier so you can trace an action back to the exact result set that produced it. Plenty of analytics tools count clicks, but counting is not the same as explaining. UBI records which query showed which results, in which positions, and which result the user acted on, so you can judge whether a ranking did its job.

Because the schema is an open specification rather than a proprietary log format, the data you collect is portable and interoperable across tools that speak UBI. That matters when you want the behavior data to feed more than one consumer: a relevance dashboard, an offline evaluation harness, and a tuning workflow can all read the same records without a bespoke export for each.

Collecting behavior data in Amazon OpenSearch Service

The client side is where behavior originates, so UBI ships an optional JavaScript collector that manages the query identifier for a search and records events against it as the user interacts with the results. Your web page or app initializes the collector, associates it with each query, and sends click and action events as they happen. The collector handles the lifecycle of tying events back to the originating query so you do not hand-roll that plumbing.

On the server side, the open-source UBI plugin can record query data directly, avoiding a round trip to the client for the query and result payload. The plugin is part of open-source OpenSearch and is not bundled into Amazon OpenSearch Service, but you do not need the plugin to adopt UBI. Amazon OpenSearch Service documents a supported path for collecting UBI-formatted data by sending query and event records to UBI indices directly, so the managed service stores and analyzes the same schema. The data lands in OpenSearch indices, which means you query and visualize it with the same tools you already use for search.

Turning behavior into relevance

Once the events are flowing, the metrics that were previously anecdotes become queries you can run. Zero-result rate, click-through rate by position, the share of sessions that refine a query and search again, and the rank of the result users actually click all come straight out of the UBI indices, through query DSL or SQL in the Query Workbench. A dashboard over those indices turns the stream into a picture of where search is winning and where it is losing.

The behavior data also drives OpenSearch features built on top of the UBI specification. The open-source Search Relevance Workbench uses judgments derived from behavior to evaluate and compare ranking configurations, and its Hybrid Search Optimizer uses the same data to sweep normalization and weighting settings for hybrid search rather than making you guess at them. Those same behavior-derived judgments feed Learning to Rank: the Workbench turns UBI events into implicit judgment lists, and Learning to Rank, available in Amazon OpenSearch Service, trains an XGBoost or RankLib model on them to rescore results with behavioral signals that BM25 alone never sees. The click that a user made yesterday becomes an input to the ranking they see tomorrow. That is the feedback loop that separates a search system that improves from one that just runs.

Behavior data for agentic search

Your users are no longer the only ones searching. Agents now run retrievals too, and they leave behavior signals of their own. When an agent runs a retrieval, reads the results, and picks which passages to use in its answer, that selection is a relevance judgment in the same way a human click is. Record agent interactions in the UBI schema and you get a behavior stream for agent traffic alongside the human one, so you can see which retrievals actually fed answers and which the agent threw away. The stakes are higher than they look: an agent that retrieves the wrong passages produces a confidently wrong answer, and without behavior capture you hear about it from a user complaint instead of your own data.

Capturing all of this behavior is worth it on its own, but it also feeds something that pays it back. Agentic relevance tuning, delivered through the OpenSearch Relevance Agent introduced as an experimental feature in OpenSearch 3.6, takes the complexity out of relevance tuning by working directly from your UBI data. Through a natural-language chat in OpenSearch Dashboards, it reads the behavior you have captured to find where ranking falls short, proposes tuning strategies from what it finds, and validates them against offline evaluation sets before anything ships. Tuning that used to take a search expert weeks becomes a conversation, and it keeps you as the decision-maker rather than changing your ranking behind your back. The behavior stream this article is about is the fuel: the richer your UBI data, the more the agent has to reason over.

Start recording

The behavior data you need to improve search is being generated right now, on every query your users and agents run, and it disappears the moment you fail to record it. UBI is the path to keeping it: an open schema, a client collector, and a supported collection path in Amazon OpenSearch Service, all landing in indices you already know how to query. Instrument the queries that matter, watch what users and agents do with the results, and let the grades they have been giving you all along drive what you ship next.

From 1.3 TB to 66 GB: On-Disk Vector Search in OpenSearch

Jon Handler — Thu, 08 Oct 2026 16:21:14 +0000

The last turn of the dial. Move the full-precision vectors to disk, stack a few more savings, and pay for the latency you actually need.

The first two articles in this series shrank the vectors and kept them in memory. 1.3 TB of RAM for 100M Vectors? Shrink Them Before They Hit OpenSearch covered pre-quantized vectors, where you convert to FP16, byte, or binary before indexing. Native Quantization: Let OpenSearch Service Compress Your Vectors handed that conversion to the engine at index time, from 16-bit scalar quantization down to 32x with optimized scalar quantization. Both kept the working set in RAM, where search answers in milliseconds.

This article takes the last turn of the dial: it moves the full-precision vectors to disk. Amazon OpenSearch Service keeps a compressed copy in memory and searches it first, then reads the full-precision vectors from disk to rescore the best candidates. For a replicated index of 100 million embeddings at 1,536 dimensions, that drops resident memory from about 1.3 TB, the best-latency deployment with everything in RAM, to roughly 66 GB. The price is some added search latency. A few stacking techniques, covered later, trim the footprint further.

On-disk mode: compressed in memory, full precision on disk

Disk-based vector search, introduced in OpenSearch 2.17, runs a search in two phases. The compressed vectors stay resident in memory, and OpenSearch searches them first to gather more candidates than you asked for. Then it loads the full-precision vectors for those candidates from disk and rescores them, returning the top k by true similarity. The large full-precision copy lives on the secondary storage tier, not in RAM, which is where the memory savings come from. Rescoring is on by default, with an oversample factor of 2.0 at the default 32x compression, so recall stays strong despite the compressed first pass.

You enable it with a single field-level parameter, mode set to on_disk. By default that selects the Faiss engine, the HNSW method, and 32x compression, which routes through the optimized scalar quantizer OpenSearch uses at that level, bringing a replicated 100-million-vector index down to roughly 66 GB resident. The mode works with the float and half_float data types. To trade some of the savings back for recall, lower the compression with compression_level; at 4x, OpenSearch uses the Lucene engine.

PUT /my-ondisk-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "space_type": "l2",
        "data_type": "float",
        "mode": "on_disk"
      }
    }
  }
}

Latency is the thing to measure. In-memory search runs every comparison at RAM speed. On-disk search adds a second pass that reads full-precision vectors from the storage tier, so SSD throughput sets the pace. In the OpenSearch project's disk-based search benchmark on 113 million 1,024-dimension vectors, on-disk search at 32x answered in about 100 milliseconds at p90 with a single client, compared with about 24 milliseconds for the uncompressed in-memory index. Latency climbs as concurrency rises, so test at your real query load. For RAG, agent, and batch workloads that is usually a good trade. For the tightest user-facing latency budgets, keep the working set in RAM with one of the in-memory options from the earlier articles.

A useful pattern runs two indexes on one cluster: an in-memory quantized index for interactive, user-facing queries, and an on-disk index for batch or background work such as overnight RAG precomputation or periodic analytics over the full corpus. Each is sized for its own latency target, so you are not paying RAM prices for queries that can wait.

Derived source: cut disk storage, not just RAM

Derived source saves disk, not memory. Everything else in this series shrinks what sits in RAM; this setting shrinks what sits on disk. Without it, OpenSearch keeps a full copy of every vector in the stored _source field, alongside the vector data the search structures already hold, so each vector is stored twice. Derived source drops that copy from _source and rebuilds the values on demand, so the update, update_by_query, and reindex APIs keep working. Less data on disk also makes search faster: in the OpenSearch benchmarks for derived vector source, storage dropped by 3x, force merges ran about 10% faster, and search latency on the Lucene engine fell by 90%, because less unneeded vector data moves through the page cache. Derived source is on by default for vector indexes on OpenSearch 3.0 and later, and it matters most for on-disk indexes, where the full-precision vectors already occupy the storage tier. For an index created before the default, check index.knn.derived_source.enabled; the setting is static, so turning it on means reindexing.

Serving indexes larger than memory, handled for you

You do not have to load an entire Faiss index into memory to search it. Memory-optimized search, introduced in OpenSearch 3.1, memory-maps the index file and lets the operating system file cache serve reads, so a node can hold indexes larger than its physical memory and repeated queries come from cache. When you choose on-disk mode, or 32x compression with the optimized scalar quantizer, OpenSearch applies it for you behind the scenes. You can also turn it on explicitly for any Faiss HNSW index with a single index setting. On instances with large local memory and NVMe storage, memory-mapping raises effective capacity per node, with occasional p99 variability when a read misses the cache. The setting is static, so for an existing index you close the index, update the setting, and reopen it. It applies to Faiss HNSW only; IVF and product quantization indexes still load into memory.

PUT /my-vector-index
{
  "settings": {
    "index.knn": true,
    "index.knn.memory_optimized_search": true
  },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "space_type": "l2",
        "method": { "name": "hnsw", "engine": "faiss" }
      }
    }
  }
}

IVF with product quantization for billion-scale

HNSW is the default graph for most workloads, but at billion scale even a quantized graph can exceed the memory budget. Faiss offers an alternative: an inverted file (IVF) index combined with product quantization. IVF partitions the vector space into clusters with k-means and searches only the clusters near the query, while product quantization encodes each vector into a compact set of codes. There is no navigation graph to hold, so the footprint is smaller than HNSW, at the cost of a training step to learn the cluster centroids and codebooks, and typically lower recall than HNSW at the same latency. Reach for IVF with product quantization when you have billions of vectors and cost is the binding constraint. IVF is available only on the Faiss engine; the Lucene engine supports HNSW and flat indexes but not IVF. Like all product quantization, it requires training before you index.

Reminders from earlier in the series

A few techniques from the earlier articles stack with everything here. When a filter narrows the candidates to roughly 100,000 vectors or fewer, exact k-NN skips the graph entirely and returns perfect recall with no graph to hold in memory. On nodes dedicated to vector search, raising the k-NN memory circuit breaker, knn.memory.circuit_breaker.limit, above its 50% default fits more vectors per node before you add hardware. And GPU-accelerated graph construction cuts build times for full-corpus rebuilds and embedding-model upgrades; see the GPU acceleration write-up.

The full spectrum, end to end

Here is the whole dial across the three articles, measured against the same baseline: 100 million vectors at 1,536 dimensions, HNSW with m of 16, one replica. RAM figures are the resident footprint with the replica. Dollar figures are order-of-magnitude on demand before reserved-instance or savings-plan discounts, meant to show relative magnitude, not quote a price.

Method	Resident RAM (with replica)	p50 latency	Recall
FP32 baseline	~1.3 TB	single-digit ms	reference
FP16 / SQ 16-bit	~656 GB	single-digit ms	indistinguishable from FP32
INT8 / byte	~341 GB	single-digit ms	0.99 on Euclidean sets
SQ 4-bit (8x)	~184 GB	single-digit ms	0.94-0.98 with rescoring
SQ 2-bit (16x)	~105 GB	single-digit ms	moderate, benchmark it
OSQ 1-bit (32x)	~66 GB	single-digit ms	0.94-0.98 with rescoring
Product quantization	~42 GB	single-digit ms	moderate, codec-dependent
On-disk (32x default)	~66 GB resident + FP32 on disk	~100 ms p90 (benchmark)	strong, two-phase rescoring

Dimensionality reduction stacks on every row. If your embedding model emits shorter vectors, or you apply principal component analysis or Matryoshka truncation, cutting 1,536 dimensions to 768 roughly halves each RAM figure again.

Wrapping up the series: pick your point on the dial

Across all three articles, the choice comes down to a tradeoff among cost, latency, and accuracy. You can hold any two steady by giving up some of the third. Every step down the table saves memory and pays for it in recall or in latency, and oversampling with rescoring buys recall back at a latency cost. No setting is right for every workload; the right one is the point where your recall and latency requirements still hold at the lowest cost.

Two settings make good middle grounds. FP16, either pre-quantized half_float vectors or 16-bit scalar quantization in the engine, halves memory to about 656 GB with recall effectively indistinguishable from full precision and no change in latency. It is the lowest-risk first step. At the other end, 32x compression on disk takes the same index to about 66 GB resident, around a twentieth of the baseline, and two-phase rescoring against the full-precision vectors keeps recall strong, around 95% recall@100 in the OpenSearch disk-based search benchmark. The cost is latency, about 100 milliseconds at p90 in that benchmark, which suits RAG, agent, and batch workloads. Between the two, you can cover a wide range of workloads without exotic configuration.

You do not have to walk the dial by hand. Auto-optimize for Amazon OpenSearch Service takes your vectors from Amazon S3, along with the recall and latency you need, and evaluates HNSW parameters, quantization, and rescoring settings against them. It returns up to three recommended configurations, typically within 30 to 60 minutes. The jobs run on separate infrastructure that OpenSearch Service manages, so they don't consume resources on your domain, and each job is billed at a flat rate. That makes it a simple, cost-effective way to explore the dial on your own data before you commit to a configuration.

Whichever setting you choose, measure it. Start from a full-precision baseline on your own evaluation queries, record recall and p50 and p95 latency, and compare every option against it. The right point moves as the corpus grows, the embedding model changes, and the accuracy requirement shifts, so build recall measurement into the deployment pipeline and re-check when the inputs change.

Native Quantization: Let OpenSearch Service Compress Your Vectors

Jon Handler — Tue, 06 Oct 2026 21:51:48 +0000

Send FP32, get 2x to 32x compression, and leave your ingestion pipeline untouched. The engine does the work.

The previous article in this series put the quantization work on you. You convert vectors to a reduced precision before indexing, and Amazon OpenSearch Service stores exactly what you send. That gives you full control, and it gives you a standing job: pick a method, run the conversion in your pipeline, and re-check accuracy every time your embedding model changes.

There is another way. OpenSearch Service can compress at index time. You send the full-precision FP32 vectors your embedding model already produces, and the engine converts each one to a lower-bit representation during ingestion and stores the compressed vectors in the index. Both the Faiss and Lucene engines offer native quantization, each with its own methods. Your ingestion pipeline does not change. The control over the accuracy tradeoff moves out of your code and into index configuration.

The baseline is the same one from the previous article. At 100 million vectors of 1,536 dimensions, full-precision FP32 needs about 643 GB for one copy, roughly 1.3 TB with a replica. That figure is the best-latency deployment, holding every vector in memory; with memory-optimized loading you can serve the same index in less RAM and page from disk at some latency cost. Every option here reduces the footprint while leaving ingestion identical. This is the second of three articles on the quantization spectrum in OpenSearch Service. The first covered pre-quantized vectors. The third covers on-disk mode and cost stacking.

Scalar quantization: the engine compresses each dimension

Faiss scalar quantization is the native path you reach for first. You set the encoder on the knn_vector field to sq and give it a bits value, and OpenSearch Service converts each FP32 dimension to the target precision during ingestion, storing the quantized vectors in the index. Valid bit widths are 16, 4, 2, and 1, spanning the range from near-lossless to aggressive compression.

At 16 bits, scalar quantization halves memory, the same reduction as pre-quantized FP16, dropping a replicated 100-million-vector index to about 656 GB, without a conversion step in your pipeline. OpenSearch stores the 16-bit vectors and builds the HNSW graph over them, and at search time it computes distances directly on the 16-bit values, accelerated by SIMD instructions on recent processors, so there is no round trip back to full precision. The type parameter selects the 16-bit format: fp16, the default, for the highest precision within a plus or minus 65,504 range, or bf16 (introduced in OpenSearch 3.9) for the full 32-bit value range at slightly lower precision. A companion clip parameter governs fp16 only: left at its default it rejects any vector with a value outside the fp16 range, and set to true it rounds out-of-range values to the limits instead. Both type and clip apply to 16-bit quantization only.

PUT /my-sq-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "parameters": {
            "m": 16,
            "ef_construction": 256,
            "encoder": { "name": "sq", "parameters": { "bits": 16 } }
          }
        }
      }
    }
  }
}

The low-bit end: 1, 2, and 4-bit quantization

Below 16 bits, scalar quantization reaches the aggressive end of the range. Multi-bit scalar quantization added two-bit (16x) and four-bit (8x) options for Faiss in OpenSearch 3.9, and one-bit (32x) has been available since OpenSearch 3.6. These bit widths are supported for the HNSW method. You configure the multi-bit options by setting the sq encoder bits to 2 or 4, or compression_level on the field to 8x or 16x. For the 32x end, set compression_level to 32x and let OpenSearch choose the algorithm, which since 3.7 is the optimized scalar quantizer described below. At 32x, a replicated 100-million-vector index drops to about 66 GB, from the 1.3 TB FP32 baseline. Collapsing each dimension toward a single bit discards magnitude, so benchmark against your own data; at 32x the optimized scalar quantizer recovers much of that recall for you, as the next section describes.

PUT /my-sq-2bit-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "parameters": {
            "encoder": { "name": "sq", "parameters": { "bits": 2 } }
          }
        }
      }
    }
  }
}

The 32x path: optimized scalar quantization on both engines

At the 32x end of the dial, OpenSearch Service uses optimized scalar quantization (OSQ), a RaBitQ-derived technique that compresses float32 vectors down toward a bit per dimension while holding recall far better than naive binary rounding. Where naive binary quantization centers each vector on the global centroid and rounds, OSQ adds a calibration step: it computes statistics, optimizes the quantization intervals, and keeps small corrective factors with each vector that sharpen the distance estimates. The payoff is a better compression-to-quality ratio at the same 32x reduction. You do not pick the algorithm by hand. Starting in OpenSearch 3.7, asking for 32x compression selects OSQ automatically, and OpenSearch applies it across both the Faiss and Lucene engines, handling the underlying mechanism for you. Set the compression level and let the engine choose.

The reason OSQ holds recall at roughly a bit per dimension is a set of techniques OpenSearch applies for you, with no knobs to turn. It keeps small corrective factors with each quantized vector so the first-pass distances stay close to the true ones. It uses asymmetric distance computation, keeping the query vector at full precision and rescaling it to compare meaningfully against the compressed document vectors, so the query side keeps information the documents gave up. And it applies a random rotation that spreads variance across dimensions, so more signal survives the compression. Each of these was once a separate option to wire up by hand; at 32x, OpenSearch does all of it under the covers.

PUT /my-osq-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "space_type": "l2",
        "compression_level": "32x"
      }
    }
  }
}

Product quantization, briefly

Product quantization (PQ) is the one native method that works differently enough to call out. Instead of reducing the precision of each dimension, PQ splits each vector into m subvectors and encodes each one against a learned codebook, reaching compression ratios beyond what scalar quantization offers. The cost is a training step: PQ runs k-means over a representative sample to build the codebooks before you can index, which scalar quantization does not require. That training requirement, plus a larger accuracy gap at high compression, makes PQ a fit for very large corpora, hundreds of millions to billions of vectors, where the savings justify the added operational step. For the parameters and the training workflow, see the OpenSearch product quantization documentation.OpenSearch product quantization documentation

Oversampling to recover accuracy

The low-bit methods search over reduced-precision vectors, so the first-pass scores are approximate. Across these methods, you raise recall at query time with oversampling: set an oversample_factor and OpenSearch retrieves that multiple of k candidates before ranking, trading latency for accuracy, with 2x to 4x a reasonable starting point. The same rescore.oversample_factor parameter applies whether you are running scalar quantization or the optimized scalar quantizer at 32x. Measure recall and latency together on your own evaluation set, because the right oversampling depends on your data and your bit width.

GET /my-sq-index/_search
{
  "size": 10,
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.1, 0.2],
        "k": 10,
        "method_parameters": { "ef_search": 100 },
        "rescore": { "oversample_factor": 3.0 }
      }
    }
  }
}

When exact k-NN is the better call

In-engine quantization still builds and searches an HNSW graph, and that graph is what holds memory. When your queries pre-filter to a small candidate set, you can skip the graph entirely. Exact k-NN scores the query against every vector that passes the filter, with no graph to build or hold in RAM, and when a structured filter narrows the field to roughly 100,000 vectors or fewer it returns perfect recall at latency competitive with the approximate path. It stacks with quantization too: a quantized exact scan over a filtered subset is both small and precise. Reach for exact k-NN when the access pattern is search within this tenant, this category, this date range rather than search the whole corpus. The graph earns its memory only when the candidate set stays large.

Pick the method, keep the pipeline

The native methods trade a little configuration for a large cut in memory while your ingestion stays exactly as it was. Scalar quantization at 16 bits halves memory at recall close to full precision, and the 2- and 4-bit settings take you to 16x and 8x. At the 32x end, you set the compression level and the optimized scalar quantizer takes over on either engine, recovering recall with corrective factors, asymmetric distance computation, and random rotation that you never configure. Product quantization serves the largest corpora, and oversampling tunes recall across all of them. When your queries filter down to a small set, exact k-NN sidesteps the graph cost altogether.

The next article moves from compressing the vectors to moving them off memory. On-disk mode keeps a compressed graph in RAM and the full-precision vectors on disk, with a two-phase search that rescores against those full-precision vectors, and cost-stacking techniques layer on top of any method here for more savings.

1.3 TB of RAM for 100M Vectors? Shrink Them Before They Hit OpenSearch

Jon Handler — Fri, 02 Oct 2026 13:51:05 +0000

FP16, INT8, and binary. Pick your spot on the accuracy-cost dial before Amazon OpenSearch Service ever sees the vector.

If you use vectors for search, you are always trading off cost, latency, and accuracy. OpenSearch Service gives you a set of knobs to adjust that balance, depending on where you want the emphasis. The one that dominates the bill is memory, because the approximate-nearest-neighbor graph that makes vector search fast lives in RAM, and RAM is the resource you provision the most of. How much you need comes down to one choice you make before indexing a single document: how much precision each vector carries.

At 100 million vectors of 1,536 dimensions, you need about 643 GB for a single copy. Run it with one replica, which most production deployments do, and the resident footprint is about 1.3 TB. That full footprint is what you deploy to hold every vector in memory for the best latency; with memory-optimized loading you can run in less RAM and page from disk on demand, trading some latency for a smaller bill. That number is not fixed either way. You decide how much of it to pay for, by choosing how much precision each vector carries before OpenSearch Service ever stores it, and you make that choice knowing what accuracy it costs.

Quantization is a dial with a continuous range of settings, each trading a measured amount of accuracy and latency for a lower memory bill. This is the first of three articles walking the full spectrum available in Amazon OpenSearch Service. Here you shrink the vectors yourself before they reach the engine. The next two cover OpenSearch-native quantization, where the engine compresses for you, and on-disk mode, where a compressed copy stays in memory and the full-precision vectors move to disk for a second pass that restores accuracy.

The baseline: full-precision FP32

Every compression option is measured against the full-precision baseline, so start there. A single FP32 vector at 1,536 dimensions holds 1,536 values of 4 bytes each, which is 6,144 bytes. The OpenSearch estimate for Faiss HNSW adds the graph that connects those vectors and multiplies the whole by a storage factor: 1.1 times the quantity 4 times dimension plus 8 times m. The 4 times dimension term is the raw vector, 4 bytes per FP32 dimension. The 8 times m term is the graph: m is how many neighbor links HNSW keeps per vector (m defaults to 16), and each link is stored as an 8-byte node identifier, so the graph adds 8 times m bytes per vector. The 1.1 is the source-to-index ratio, the stored index running about ten percent larger than the raw input.

memory_per_vector = 1.1 x (4 x dimension + 8 x m)
                  = 1.1 x (4 x 1,536 + 8 x 16)
                  = 1.1 x (6,144 + 128)
                  = ~6,899 bytes/vector

Per copy:        100,000,000 x 6,899 bytes = ~643 GB
With 1 replica:  643 GB x 2               = ~1,285 GB (~1.3 TB)

The raw vectors dominate that number. A replica holds a full second copy of the graph in memory for availability and query throughput, so every footprint figure in this article is per copy, and you double it for one replica. 1.3 TB is the resident anchor for a replicated 100-million-vector index. Every option below reduces it.

Turn 1.3 TB into instances to see the bill. Take the r8gd.2xlarge, with 64 GB of RAM. By default OpenSearch Service gives half of that to the JVM heap, leaving 32 GB off-heap, and the k-NN memory circuit breaker (knn.memory.circuit_breaker.limit) caps native vector memory at 50% of what remains, about 16 GB per instance. At 16 GB usable, 1.3 TB needs roughly 80 instances. If the node is dedicated to vector search and not running heavy text queries that lean on the heap, raise the circuit breaker to 75% of off-heap memory, which lifts usable vector memory to about 24 GB per instance and brings the count down to roughly 54 instances. On demand, 54 to 80 r8gd.2xlarge instances runs on the order of $31,000 to $46,000 per month, before reserved-instance or savings-plan discounts. Treat the dollars as order-of-magnitude. The point is that full precision at this scale is a five-figure monthly line item, and the dial below moves it.

Pre-quantized vectors: you shrink, OpenSearch stores

The first category of compression happens before your vectors reach OpenSearch Service. You, your embedding model, or a step in your ingestion pipeline produce vectors in a reduced-precision format, and OpenSearch Service stores and searches exactly what you send. The engine does no conversion here. The appeal is control: you pick the method, you measure recall against your own evaluation set, and nothing about the accuracy tradeoff is hidden. Three formats are available, and they sit at different points on the dial: FP16, INT8, and binary. The footprints below are per copy; double each for a replica, the same as the baseline.

FP16: half the bytes, nearly identical recall

Half-precision floating point stores each dimension in 16 bits instead of 32. Per-vector size drops by exactly half, from 6,144 to 3,072 bytes, and the 100-million-vector footprint falls from about 643 GB to roughly 328 GB per copy, about 655 GB with a replica. Many embedding models carry little useful signal in the low-order bits of each FP32 dimension, so truncating to FP16 costs almost nothing in quality. OpenSearch benchmarks put FP16 recall at effectively indistinguishable from full precision, because most datasets do not use the full 32-bit range. Some models emit FP16 natively, in which case there is no conversion step to add.

In OpenSearch Service, the data type for native FP16 is half_float, introduced in OpenSearch 3.9 and supported for the HNSW method on both the Faiss and Lucene engines. You ingest and query half_float vectors the same way as float vectors, and each dimension must fall within the representable range of roughly plus or minus 65,504. The mapping:

PUT /my-vector-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "data_type": "half_float",
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "faiss"
        }
      }
    }
  }
}

INT8 byte vectors: a quarter of the footprint

Byte vectors store each dimension as a signed 8-bit integer in the range -128 to 127. At one byte per dimension, a 1,536-dimension vector occupies 1,536 bytes, a quarter of the FP32 baseline, and the footprint drops to roughly 170 GB per copy, about 341 GB with a replica. The cost is a conversion step: you map your FP32 embeddings into the byte range before indexing. A linear scaling from the observed min and max across a representative sample works for many models, and calibrated or per-channel approaches narrow the accuracy gap further. OpenSearch benchmarks on byte vectors report recall@100 of 0.99 on Euclidean datasets, matching full precision, with angular datasets measured by cosine similarity giving up more and benefiting most from careful calibration. Measure it against your own queries before you commit.

PUT /my-byte-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 1536,
        "data_type": "byte",
        "space_type": "l2",
        "method": {
          "name": "hnsw",
          "engine": "lucene"
        }
      }
    }
  }
}

Binary vectors: 32x smaller, a different bargain

Binary is the aggressive end of the pre-quantization dial. Each dimension collapses to a single bit, one if the value is non-negative and zero otherwise. A 1,536-dimension vector becomes 1,536 bits, which is 192 bytes, 32 times smaller than FP32, and the 100-million-vector footprint lands near 33 GB per copy, about 66 GB with a replica, small enough to fit on a single large instance either way. The tradeoff is that binary discards magnitude entirely. For high-dimensional embeddings whose vectors are well separated by angle, which is common for modern text embeddings, binary retains a surprising share of the original recall. OpenSearch large-scale tests with rescoring hold recall@100 in the 0.94 to 0.98 range across 8x, 16x, and 32x compression, close to the in-memory baseline. For embeddings that lean on subtle magnitude differences, the drop is steeper. Always benchmark against your own data.

When you convert to binary yourself and store the result, you pack the bits into bytes: eight bits become one int8 value, so the dimension you declare must be a multiple of 8, and the engine compares vectors with Hamming distance. This pre-converted path is Faiss-only. If you would rather hand FP32 vectors to the engine and let it compress them, OpenSearch does that for you at the aggressive end of the dial through the optimized scalar quantizer, which is the subject of the next article.

A production pattern pairs a fast binary first pass with a full-precision second pass that restores ordering. OpenSearch Service implements that pattern for you in on-disk mode rather than leaving you to wire up a separate full-precision field, so the mechanics are covered in the third article of this series.

When exact k-NN beats the graph entirely

Quantization assumes you are running approximate search over an HNSW graph, and that graph is the thing eating your RAM. If your queries pre-filter to a small candidate set, you may not need the graph at all. Exact k-NN scores the query against every vector that passes the filter, with no graph to build, hold in memory, or rebuild on merge. When a structured filter narrows the field to roughly 100,000 vectors or fewer, exact search returns perfect recall at latency that is competitive with the approximate path, and it stacks cleanly with quantization: a byte or binary exact scan over a filtered subset is both inexpensive and precise. Reach for exact k-NN when your access pattern is 'search within this tenant, this category, this date range' rather than 'search the whole corpus.' The graph earns its memory only when the candidate set stays large.

Dimensionality reduction stacks with all of it

Precision is one axis. Dimension count is another, and the two multiply. Reducing 1,536-dimension embeddings to 768, whether through a model that emits shorter vectors or a reduction step such as principal component analysis, halves memory on its own and composes with any format above. FP16 at 768 dimensions is half of FP16 at 1,536. Binary at 768 is half of binary at 1,536. The dial is really a surface, precision on one axis and dimensionality on the other, and you can move along both.

Pick your spot on the dial

Pre-quantized vectors are the easiest segment of the spectrum to reason about, because you own the conversion and you measure the result. For many workloads FP16 alone halves the memory bill at recall indistinguishable from full precision, and that is often the end of the story. Byte vectors take you to a quarter of the baseline when a few points of recall are acceptable. Binary takes you to a thirty-second when your embeddings tolerate it and a rescoring pass covers the rest. And when your queries filter down to a small set, exact k-NN sidesteps the graph cost altogether.

When you want more compression without managing the conversion yourself, OpenSearch Service will do the work. The next article covers OpenSearch-native scalar and binary quantization, where you send full-precision vectors and the engine compresses them at index time, with oversampling to recover accuracy. After that, on-disk mode keeps a compressed copy in memory and moves the full-precision vectors to disk for a second-pass rescore, the lowest-cost end of the dial.

Migrating From Keyword Search to Vector Search on OpenSearch: A Guide for Teams Still on Lexical-Only

Jon Handler — Tue, 29 Sep 2026 22:43:14 +0000

Your users learned to ask questions somewhere else, and now your agents are asking too. Your search bar is the last place that still wants keywords.

A citizen types "can I renew my passport if it expired two years ago" into a government services portal, and the search bar hands back every page that contains the words "passport," "renew," and "expired," ranked by how often those words appear. The one page that answers the question sits on the third results screen, if it surfaces at all. The citizen gives up and calls the contact center, which costs the agency real money on a query the website should have answered for free.

If you run search for an organization, you have watched some version of this play out. Your users arrive from ChatGPT, Perplexity, and Claude, where they ask full questions and get direct answers. Then they reach your search box and have to translate the question back into the two or three keywords they hope your index contains. The friction is not subtle, and it shows up in your analytics as abandoned searches, repeated queries, and support tickets for answers that already exist in your content.

The reason to move to vector search is not that the technology is new or interesting. The reason is that the cost of staying on keyword-only search now shows up on a budget line, and it is about to show up again as you put agents in front of your content. An agent answering a user's question runs its own searches, and it hits the same wall your users do when the index only matches exact terms. This guide covers when keyword search is actually failing you, what vector and hybrid search on Amazon OpenSearch Service change for both people and agents, and how to stage the migration without ripping out the system you already run.

When keyword search is actually failing you

Keyword search still does real work, and moving to Amazon OpenSearch Service does not mean giving it up. Exact product codes, known document titles, specific error strings, faceted navigation, and structured filters all run on BM25 ranking that returns the right result quickly and transparently, and OpenSearch Service runs that lexical path natively. The point is not to replace keyword search. It is to stop asking keyword search to answer the queries it was never built for.

The failure mode appears when queries stop looking like keywords and start looking like questions. A search for "vehicle hire" against a catalog that only says "car rental" returns nothing, because the two strings share no terms even though a person would treat them as the same request. Keyword search matches the characters the user typed against the characters in your index. When those two vocabularies diverge, the query fails silently, and the user rarely tells you. They just leave.

Pull your search logs and sort by zero-result and low-engagement queries, and the failures line up in a clear pattern. They cluster on natural-language phrasings, multi-part questions, and searches that use different words than your content authors used. Those are the queries bleeding engagement, and they are exactly the queries that keyword matching cannot rescue no matter how carefully you tune the ranking.

What vector and hybrid search change

Vector search matches on the company words keep rather than the exact characters a user typed. An embedding model maps text to coordinates so that phrases appearing in similar contexts across the training corpus land near each other. "Vehicle hire" and "car rental" end up close because they show up in overlapping surrounding text, not because a model comprehends either phrase. The practical effect is that a query retrieves relevant documents even when the author and the user chose different words for the same thing. Correlation over word usage, not understanding, is what closes the vocabulary gap.

Vector search alone overcorrects. Ask it for an exact product code and it may return a cluster of conceptually adjacent items while burying the precise match the user actually wanted. This is why production search combines both approaches. Hybrid search runs the keyword path and the vector path together and merges the results, so exact matches keep ranking high while related-but-differently-worded documents fill in the results a keyword-only system would have missed. OpenSearch Service runs hybrid search natively, and the balance between the two paths is a setting you tune to your traffic rather than a rewrite you commit to.

Increasingly this is not one-shot retrieval-augmented generation but agent-driven question answering. An agent decides how to answer a question: it can run several retrievals, refine them based on what came back, pull from more than one index, and compose an answer grounded in what it found. OpenSearch Service supports this directly with agentic search, where the agent chooses the lexical or vector path per query based on the query itself, using exact matching for a date filter or a product code and vector retrieval for a conceptual question. Grounding the agent in content it retrieves from your own indices is what keeps the answer built from your documents rather than from the model's training data alone. Retrieval quality sets the ceiling on answer quality, so the move to better search pays off twice: once in direct results and again in every agent you build on top of them.

How to stage the migration

Treat the move as an augmentation, not a rip-and-replace. Your keyword index keeps running and keeps serving the exact-match and filter traffic it already handles well. You add a vector path alongside it and route more query types through the hybrid pipeline as your confidence grows. Nothing about this requires a flag-day cutover, and the staged approach means you can measure each step against the system it replaces.

Start where keyword search fails hardest, which your zero-result logs already identified. Stand up an embedding pipeline in OpenSearch Service so documents get vectorized as they are indexed, point a hybrid query at the failing query set, and compare relevance against your current system on the same queries. The Search Relevance Workbench in OpenSearch Dashboards runs keyword and neural queries side by side against the same index, so you can see the difference on your own queries rather than take it on faith, and it is enough of a topic on its own that it deserves its own article. Amazon OpenSearch Service can host the embedding model inside the engine and vectorize both documents at index time and queries at search time, so you are not standing up a separate model-serving tier to get started. Once the hybrid pipeline wins on the queries that were failing, widen its coverage.

Measure the transition on user behavior, not just offline relevance scores. The catch is that most teams do not capture user behavior in any structured way, so the signal that would prove the migration worked never gets recorded. This is what User Behavior Insights (UBI) addresses: a standard schema that ties each query to the results shown and the actions the user took on them, so zero-result rate, search refinement, and click-through depth become data you can query rather than anecdotes. You can collect UBI-formatted data with Amazon OpenSearch Service and analyze it in the same engine that serves the search. Those metrics also make the business case for the next phase, because they translate search quality into the operational costs your organization already tracks, like the support contacts that duplicate content the site should have surfaced.

The cost of waiting

The gap between what your users expect and what your search bar delivers widens every day they spend in a conversational interface somewhere else. When one organization in your sector ships natural-language search, the expectation resets for everyone your users interact with next, including you. The migration stops being a project you might schedule and becomes a gap your users feel on every visit.

Amazon OpenSearch Service gives you a path that does not start with tearing down what you have. Keep the keyword search that works, add vector and hybrid search where questions have replaced keywords, and let agents run retrieval across your indices when you are ready to return answers instead of links. The technology is production-ready today. The queries your users and your agents are already typing are the ones telling you where to start.

OpenSearch Unifies the Retrieval Layer for Structured, Time Series, and Full Text Data

Jon Handler — Thu, 24 Sep 2026 20:07:35 +0000

Text, time series, and structured data in one engine. One query interface. One place for agents to look.

For a decade, search quality meant one thing: getting the right link into the top slot. Ten blue links. Click-through rate. Precision at rank one. The entire search industry organized itself around a single user staring at a results page, deciding which link to click. We got very good at that problem.

Agents changed the question. An agent does not click links. An agent synthesizes. It pulls a fact from a research document, a number from a time-series index, a constraint from a structured database, and weaves them into a response. The agent does not care whether the data came from a full-text index, a vector embedding, or an aggregation pipeline. It cares that the data is accurate, that the retrieval was fast, and that it did not have to call four different backends with four different query languages to assemble one answer.

That shift, from person-as-consumer to agent-as-consumer, changes what a search engine needs to be. It is no longer enough to do one type of retrieval well. The engine needs to unify text retrieval (structured and unstructured), time-series analytics, and hybrid search behind a single query interface. Amazon OpenSearch Service does all three.

The financial analyst's agent

Consider a financial services company building an agent to help its research analysts. The agent needs to answer questions like "What is the market sentiment on Acme Corp following their Q3 earnings, and how has the stock performed since the product launch in June?" That question touches four data types: news articles (unstructured text), internal research reports (semi-structured documents), transaction records (structured data), and stock price history (time series). Each data type lives in a different system today. The agent has to call each one, reconcile the results, and hope the latency budget survives.

OpenSearch Service consolidates those retrievals. The text data, the structured records, the time-series metrics, and the vector embeddings all live in the same engine. The agent can choose the right retrieval strategy per question: a pure lexical query when the analyst asks for a specific company by name, a semantic query when the question is conceptual, a hybrid query that combines both when the question has precise and fuzzy elements, or a multi-clause query that pulls text and time-series data together in a single call. The engine supports all of these through one query interface. The integration complexity that used to live in the agent's orchestration layer moves into the search engine, where it belongs.

Text retrieval: lexical, semantic, and hybrid together

Search engines have been refining text retrieval for more than thirty years. Inverted indices, BM25 scoring, boolean query algebra, stemming, synonyms, phrase matching, reranking: the toolbox is deep and battle-tested. OpenSearch inherits all of that. Lexical search through BM25 gives you precise term matching, boolean queries, function_score for custom relevance logic, and integration with reranking models. When the analyst searches for "Acme Corp revenue guidance," the lexical path finds documents that contain those exact terms. Lexical search remains the foundation: it handles exact matching, structured field filters, and queries where the user's words map directly to the vocabulary in the index.

Semantic search through vector embeddings finds results where the surrounding text contexts are similar, even when the exact words differ. If a user searches for "laptop" and your index contains "notebook computer," lexical search returns nothing because the terms do not overlap. Semantic search surfaces the match because "laptop" and "notebook computer" appear in overlapping lexical contexts across the training corpus. The match is based on distributional patterns in how words co-occur, not on shared terms. When the analyst searches for "market outlook," semantic search surfaces documents about "forward guidance" and "earnings projections" for the same reason. OpenSearch supports both exact and approximate nearest-neighbor search, with quantization methods and tiered storage that let you optimize vector costs as the collection grows. Auto-optimization tunes the model and indexing parameters so you do not have to.

Hybrid search combines both paths in a single query with per-clause weighting. This is the default for production retrieval today, and for good reason: the lexical leg catches exact terminology while the semantic leg catches contextual relevance. For agents, the per-clause weighting is especially useful. The agent can emphasize the lexical clause when the user asks for a specific company name and emphasize the semantic clause when the user asks a conceptual question. One query, adjustable emphasis.

Semantic search does not paper over weak lexical search. Builders today often hope that adding a vector path will compensate for a poorly tuned BM25 configuration, but hybrid search amplifies both signals. If the lexical side is noisy, the combined results are noisy. In our testing, tuning the lexical side of hybrid search produced significantly better results overall than tuning the semantic side alone. Both legs have to be good for hybrid to deliver.

Back to the financial analyst: the agent searches for information about Acme Corp's innovation strategy. The lexical clause matches documents mentioning "Acme Corp" by name. The semantic clause surfaces research reports about the company's R&D investments and patent filings, even when those reports never use the word "innovation." The combined result set gives the agent the full picture without requiring two separate queries.

Time series retrieval: aggregation and analysis

Time-series data gives agents the ability to aggregate, trend, and correlate. Stock prices, server metrics, transaction volumes, sensor readings: these are all sequences of values over time, and agents need to query them the same way they query text. OpenSearch Service provides an optimized engine for time-series data with fast ingestion, sub-second aggregations, and tiered storage for data that regulatory requirements demand you keep for years.

Agent observability is a natural fit here. The financial company wants to control its AI costs and understand how its agents perform. OpenSearch's observability stack ingests OpenTelemetry traces and metrics from the agent infrastructure. The company builds KPI dashboards around agent interactions: latency per tool call, token consumption per query, success and failure rates across agent-to-agent handoffs. The same engine that stores the analyst's research documents also stores the operational data about how the agent system itself is performing.

Hybrid queries can bring time-series data into the same result set as text retrieval. The analyst wants to see Acme Corp's stock performance over the last six months and correlate price movements with product launch dates. The time-series clause returns the aggregated price data. The text clause returns the news articles and press releases around each launch. The agent receives both in a single response and synthesizes the correlation without orchestrating separate backend calls.

Why OpenSearch fits

The information agents need spans every data type in the stack: unstructured text, structured records, vector embeddings, and time-series metrics. Most retrieval architectures treat these as separate concerns, each with a dedicated system. OpenSearch fits naturally because it handles all of them natively. Full-text search, vector search, hybrid search, sub-second aggregations over time-series data, and structured field queries all run inside the same engine, against the same indices, through the same query API.

That native breadth is what makes OpenSearch a natural retrieval layer for agents. The agent does not need to learn four query languages or stitch together results from four response formats. The agent issues one query with clauses that target different data types, and the engine handles the internal fan-out, score normalization, and result fusion. The agent gets back one ranked list and spends its compute on reasoning, not plumbing.

The value compounds as the data grows. Adding a new data type to the retrieval layer means adding an index and a query clause, not integrating a new backend. The financial company that started with research documents and stock prices can add regulatory filings, earnings call transcripts, and agent telemetry data to the same engine without changing the agent's retrieval architecture.

The engine backs that breadth with production-grade performance. Amazon OpenSearch Service delivers single-digit-millisecond query latencies, sustains thousands to tens of thousands of queries per second, and scales vector storage into the hundreds of billions of embeddings. Time-series ingestion runs at hundreds of thousands of events per second with sub-second aggregation response times. These are not theoretical limits. They are the operating parameters of production workloads running on the service today. When an agent needs to retrieve across text, vectors, and time series in one call, the latency budget has to hold for all three data types. OpenSearch Service holds it.

Where this is heading

The shift from people to agents as search consumers is still early, but the retrieval patterns are taking shape. Agents need unified retrieval across data types, not a patchwork of specialized backends stitched together with orchestration code. They need a single query interface that handles text, vectors, time series, and structured data with per-clause weighting so the agent can tune the emphasis on each data type per query.

OpenSearch Service delivers that unified interface at production scale. Lexical search, semantic search, hybrid search with reciprocal rank fusion, time-series analytics, observability, and tiered storage all run inside the same engine with single-digit-millisecond latencies and thousands of queries per second. The retrieval layer does not need to be assembled from parts. It is one service, one query language, one place for your agents to look.

Hybrid Search Explained: Combining Lexical and Semantic Search in OpenSearch

Jon Handler — Tue, 22 Sep 2026 16:23:51 +0000

Your keyword search is precise. Your vector search is contextual. Use both.

Hybrid search in Amazon OpenSearch Service combines BM25 keyword matching with vector similarity in a single query. You get the precision of lexical search (exact terms, filters, structured fields) and the contextual reach of semantic search (vocabulary mismatch handled, related terms surfaced) without choosing between them. As of OpenSearch 2.11, score normalization and combination run natively inside the search pipeline, so you do not need to build the merge logic yourself.

I ran an experiment recently that made the case for hybrid search more clearly than any architecture diagram could.

One Query, Three Approaches

I indexed a product catalog and searched for "women's shoes." With pure lexical search, the results included two women's shoes (matched on both words) and two men's shoes (matched on "shoes" alone). Technically correct: the query terms appeared in all four documents. But the user's intent was clearly gendered, and lexical search had no way to capture that.

With pure semantic search, all four results were boots. The vector space placed boots close to "shoes" based on co-occurrence patterns in the training corpus, which is accurate as far as distributional similarity goes. But the results lost variety entirely. If the user wanted sandals, flats, or sneakers, semantic search had narrowed the options to a single style.

With hybrid search, the top result was a boat shoe (strong lexical match on the query terms) and the remaining results included boots and other styles (semantic similarity expanding the candidate set). The user got what they searched for at the top, plus contextually related options below. Precision where it matters, discovery where it helps.

Score Normalization Is the Hard Part (and OpenSearch Handles It)

Combining lexical and semantic scores is not as easy as adding them together. BM25 returns scores from 1 to infinity. FAISS vector similarity returns scores between 0 and 1. Without normalization, the BM25 scores dominate every time, and the semantic signal disappears.

OpenSearch 2.11 introduced a score normalization processor in search pipelines that handles this at the coordinator level, globally across all shards. You configure the normalization technique (min-max or L2), the combination method (arithmetic mean, geometric mean, or harmonic mean), and the weight distribution between lexical and semantic scores. The OpenSearch team's benchmarks on public datasets show min-max normalization with arithmetic mean delivers strong results across a range of datasets, but the right settings depend on your data and your users.

I found equal weighting (0.5 lexical, 0.5 semantic) provided the best balance of relevance and discovery in my tests. Higher semantic weights pushed results toward the pure semantic outcome, which is useful when user intent is ambiguous but loses precision when the user knows exactly what they want. Lower semantic weights approach pure lexical behavior. The weight is a dial, and the right setting is a product decision, not a technical one.

Neither Lexical Nor Semantic Is Enough Alone

Lexical search is fast, interpretable, and works without any ML model. It handles exact matches, filters on structured fields, and generalizes across domains without fine-tuning. For queries where the user types the exact term that exists in your data, lexical search is already optimal. The failure mode is vocabulary mismatch: the user says "beach shoes" and your catalog says "water-resistant footwear."

Semantic search via vector embeddings handles vocabulary mismatch by correlating terms that co-occur in similar contexts across the training corpus. But vector search comes with costs: you need an embedding model (either hosted or via a service like Amazon Bedrock), the vectors consume memory, and the retrieval can over-cluster results around a narrow region of the vector space, as the boots example showed. Pure semantic search also loses the ability to do exact filtering and structured field matching that lexical search handles natively.

Hybrid search gives you both. Exact matches rank high because BM25 rewards them. Semantically related results fill in below because vector similarity surfaces them. Filters on structured fields (price range, category, availability) work through the lexical path. The combined result set is richer than either approach alone.

Implementing Hybrid Search in OpenSearch Service

The implementation runs through search pipelines. You create a pipeline with a normalization processor that specifies the normalization technique, combination method, and weights. At query time, you send a hybrid query that includes both a BM25 text match and a k-NN vector query. OpenSearch Service runs both queries in parallel, normalizes the scores using your configured method, combines them with your specified weights, and returns a single ranked result set.

The key configuration choices are the normalization technique and the weight distribution. For normalization, min-max rescales both score sets to a 0-1 range. L2 normalizes by the Euclidean magnitude of the score vector. In practice, min-max with arithmetic mean is a strong default. For weights, start at 0.5/0.5 and adjust based on how your users search: if most queries are specific product names, lean lexical; if most queries are natural language descriptions, lean semantic.

On the semantic side, you need an embedding model. OpenSearch Service integrates with Amazon Bedrock for embedding generation, or you can host your own model on SageMaker. The ingest pipeline with a text_embedding processor converts documents to vectors at index time. At query time, the same model converts the user's query to a vector for the k-NN search leg of the hybrid query.

Where Hybrid Search Fits

E-commerce search is the obvious case: users switch between exact product names and natural language descriptions within the same session. Internal knowledge bases are another strong fit, where users search for concepts they cannot name precisely. RAG retrieval pipelines benefit from hybrid search because the lexical path catches exact terminology (error codes, API names, product IDs) while the semantic path catches conceptual relevance.

If you are running pure lexical search today, adding the semantic path is where the lift comes from. If you have already invested in semantic search, adding the lexical path back in restores the precision you may have lost. Either way, hybrid search is the baseline for production search experiences now, not an advanced feature.

Start by running your current query set through a hybrid pipeline alongside your existing implementation. Measure not just relevance metrics but user behavior: click-through rates, time to conversion, search refinement patterns. The difference is usually visible within days.

Semantic Search Setup in Amazon OpenSearch Service: From Zero to Vector Search in 15 Minutes

Jon Handler — Thu, 10 Sep 2026 19:10:44 +0000

Tight integration brings the embedding model to the search engine. Vectors happen automagically.

If you have been considering semantic search, you already know the list of unknowns. Which embedding model do you choose? How do you host it? How do you get your documents converted to vectors and keep those vectors in sync as your data changes? How do you handle vectorization at query time without adding latency? And once you have vectors, how do you wire everything together so that your search index, your embedding model, and your ingest pipeline all talk to each other without a custom integration layer in between?

Those questions can take weeks of work to resolve, and for a lot of organizations they represent a huge hill to climb. Selecting and deploying an embedding model on SageMaker. Writing batch processing jobs to convert content into vectors. Building orchestration code to keep the ML pipeline and the search index in sync. Managing two separate systems that both need to know when your content or your model changes. The cognitive overhead alone can be a barrier, before you write a single search query.

The neural plugin in Amazon OpenSearch Service makes this dramatically easier. Through the OpenSearch Service console, you set up an ML connector that links your domain to an embedding model hosted on SageMaker or Amazon Bedrock. An ingest pipeline with a text_embedding processor then calls that model automatically every time you index a document, and queries convert to vectors at search time through the same connector. The embedding model still runs on SageMaker or Bedrock, but the integration, the vectorization pipeline, and the query-time conversion all live inside OpenSearch Service, so you are managing one workflow instead of stitching together three. If you want an even lighter path, Automatic Semantic Enrichment (which I wrote about in an earlier post) adds sparse vector enrichment at index time without deploying any model at all.

I set this up recently for a product catalog, and the whole process from zero to working semantic search took about fifteen minutes.

What the Old Playbook Looked Like

Without the neural plugin, implementing semantic search on OpenSearch Service means treating the search engine as a passive vector store. You deploy embedding models separately, build SageMaker processing jobs to convert your content into vectors, write orchestration code to call those endpoints, and manage the batch ingestion pipeline that feeds vectors into your index. When your content updates, both systems need to know. When your embedding model changes, you reprocess everything.

The architecture worked, but the operational tax was real. I spent more time debugging the plumbing between the embedding service and the search index than I spent on search relevance. The integration layer became its own project, and keeping the ML pipeline and the search index in sync consumed engineering cycles that had nothing to do with the actual search experience.

The Neural Plugin Changes the Contract

The neural plugin integrates embedding models directly into the search workflow. A pre-built CloudFormation template (available in the Integrations tab of the Amazon OpenSearch Service console) deploys the embedding model (Amazon Titan Text Embeddings through Amazon Bedrock, or a model on Amazon SageMaker), creates the IAM roles, and configures the ML connector in a single stack. The model runs on SageMaker or Bedrock, and the CloudFormation stack outputs an internal OpenSearch model ID that you use in subsequent steps. OpenSearch Service calls the model automatically at ingest and query time through the connector. With the model deployed, OpenSearch Dashboards provides a visual workflow experience for building the end-to-end AI search flow through the AI Search Flows interface (under OpenSearch Plugins). The same setup is available via the ML Commons API for automation.

When you create an OpenSearch ingest pipeline with a text_embedding processor, the neural plugin automatically calls your embedding model as documents enter the index. You specify which text fields should generate embeddings (product descriptions, document content, whatever drives your search experience), and the pipeline handles vectorization at ingest time. The entire workflow runs inside OpenSearch Service, so you are not maintaining batch jobs, custom code, or a sync layer between two systems.

At query time, the same model converts the user's search text into a vector, and OpenSearch Service runs a k-NN search against the stored embeddings. The entire semantic search workflow (text to vector to search to results) happens inside the search engine. You interact with plain text on both sides: index text documents, search with text queries. The vectors are internal.

What Semantic Search Actually Does to Your Results

I tested this against a product catalog. A keyword search for "what to wear in office" returned random clothing items: t-shirts, boots, undergarments. The query matched on individual words without any sense of the relationship between them. A semantic search for the same query returned dress suits, professional shoes, ties, and business attire. The embedding model had learned from training data that these terms co-occur in professional-clothing contexts, and the vector similarity surfaced results that lexical matching missed entirely.

For "accessory for hike," keyword search returned nothing useful. Semantic search returned backpacks, water bottles, and outdoor gear. The vocabulary mismatch between the query and the catalog vocabulary was total, and semantic search bridged the gap without a single synonym mapping.

The Search Relevance Workbench in OpenSearch Dashboards lets you run keyword and neural queries side-by-side against the same index. When stakeholders ask whether semantic search justifies the investment, the results comparison makes the case directly. The difference between matched-on-keywords and matched-on-co-occurrence-patterns is visible in the first three results. OpenSearch Dashboards also includes a visual builder for these AI-powered search flows, so you can configure and test ingest pipelines, ML connectors, and search pipelines through a graphical interface rather than writing JSON by hand.

Getting Started in 15 Minutes

Before you start, two prerequisites need to be in place. You need model access enabled in Amazon Bedrock for your chosen embedding model, and you need to add the Lambda invoke role as a backend role in your OpenSearch ML Commons configuration. Budget 10 minutes for this setup before you touch CloudFormation. With the prerequisites in place, deploy the CloudFormation template that provisions the embedding model, the ML connector, and the IAM roles. The stack outputs an internal OpenSearch model ID that you use for the rest of the setup. Create an ingest pipeline with the text_embedding processor, create an index that uses the pipeline, and start indexing documents.

The operational burden of adding ML-powered capabilities to search has dropped by an order of magnitude. The neural plugin, the CloudFormation templates, and the managed ML connectors are not doing anything fundamentally new. They are collapsing the integration layer that used to sit between your search engine and your embedding model. The search engine now handles both sides of the conversation.

If you have been putting off semantic search because the ML pipeline seemed too heavy, that calculation has changed. Deploy the CloudFormation stack, create an ingest pipeline, index a few hundred documents, and run the comparison tool. The difference in search quality is measurable in minutes, and the setup takes a fraction of the time the old approach required.