Users grade your search results by acting on them. Write down the grades yourself, and use them to make the results better.
Every action a user takes on a search result, a click, a hover, an add-to-cart, a purchase, tells you whether the ranking was right. Most search systems let that signal fall on the floor. Capture it instead and you get a continuous stream of relevance judgments, generated by the people who actually know what they were looking for, at zero labeling cost. That stream is what turns relevance tuning from guesswork into measurement.
The problem is that most teams have no structured record of what their users do. They know the query volume and maybe the zero-result rate, but they cannot answer the question a data scientist actually asks: for this query, which results did we show, and which ones did the user click? Without the causal link between a query, the results it returned, and the actions taken on those results, every relevance decision is an opinion.
User Behavior Insights (UBI) is the OpenSearch answer to that gap. Introduced in OpenSearch 2.15 and built on an open specification, UBI is a schema and a collection path for recording search behavior: the queries users submit, the results shown, and the actions taken on those results, all linked together. This post covers what UBI captures and how to collect it in Amazon OpenSearch Service. More to the point, it shows how to turn that data into results that move the numbers you care about: more of the purchases, answers, and signups your search is supposed to produce, fewer dead-end queries that send customers elsewhere, and retrieval your agents can actually trust.
What UBI actually records
UBI separates two kinds of records and ties them together. Query records capture what was searched and what the engine returned, with an identifier for the query. Event records capture what the user did next: a click, a scroll, an add-to-cart, a conversion, each tagged with the same query identifier so you can trace an action back to the exact result set that produced it. Plenty of analytics tools count clicks, but counting is not the same as explaining. UBI records which query showed which results, in which positions, and which result the user acted on, so you can judge whether a ranking did its job.
Because the schema is an open specification rather than a proprietary log format, the data you collect is portable and interoperable across tools that speak UBI. That matters when you want the behavior data to feed more than one consumer: a relevance dashboard, an offline evaluation harness, and a tuning workflow can all read the same records without a bespoke export for each.
Collecting behavior data in Amazon OpenSearch Service
The client side is where behavior originates, so UBI ships an optional JavaScript collector that manages the query identifier for a search and records events against it as the user interacts with the results. Your web page or app initializes the collector, associates it with each query, and sends click and action events as they happen. The collector handles the lifecycle of tying events back to the originating query so you do not hand-roll that plumbing.
On the server side, the open-source UBI plugin can record query data directly, avoiding a round trip to the client for the query and result payload. The plugin is part of open-source OpenSearch and is not bundled into Amazon OpenSearch Service, but you do not need the plugin to adopt UBI. Amazon OpenSearch Service documents a supported path for collecting UBI-formatted data by sending query and event records to UBI indices directly, so the managed service stores and analyzes the same schema. The data lands in OpenSearch indices, which means you query and visualize it with the same tools you already use for search.
Turning behavior into relevance
Once the events are flowing, the metrics that were previously anecdotes become queries you can run. Zero-result rate, click-through rate by position, the share of sessions that refine a query and search again, and the rank of the result users actually click all come straight out of the UBI indices, through query DSL or SQL in the Query Workbench. A dashboard over those indices turns the stream into a picture of where search is winning and where it is losing.
The behavior data also drives OpenSearch features built on top of the UBI specification. The open-source Search Relevance Workbench uses judgments derived from behavior to evaluate and compare ranking configurations, and its Hybrid Search Optimizer uses the same data to sweep normalization and weighting settings for hybrid search rather than making you guess at them. Those same behavior-derived judgments feed Learning to Rank: the Workbench turns UBI events into implicit judgment lists, and Learning to Rank, available in Amazon OpenSearch Service, trains an XGBoost or RankLib model on them to rescore results with behavioral signals that BM25 alone never sees. The click that a user made yesterday becomes an input to the ranking they see tomorrow. That is the feedback loop that separates a search system that improves from one that just runs.
Behavior data for agentic search
Your users are no longer the only ones searching. Agents now run retrievals too, and they leave behavior signals of their own. When an agent runs a retrieval, reads the results, and picks which passages to use in its answer, that selection is a relevance judgment in the same way a human click is. Record agent interactions in the UBI schema and you get a behavior stream for agent traffic alongside the human one, so you can see which retrievals actually fed answers and which the agent threw away. The stakes are higher than they look: an agent that retrieves the wrong passages produces a confidently wrong answer, and without behavior capture you hear about it from a user complaint instead of your own data.
Capturing all of this behavior is worth it on its own, but it also feeds something that pays it back. Agentic relevance tuning, delivered through the OpenSearch Relevance Agent introduced as an experimental feature in OpenSearch 3.6, takes the complexity out of relevance tuning by working directly from your UBI data. Through a natural-language chat in OpenSearch Dashboards, it reads the behavior you have captured to find where ranking falls short, proposes tuning strategies from what it finds, and validates them against offline evaluation sets before anything ships. Tuning that used to take a search expert weeks becomes a conversation, and it keeps you as the decision-maker rather than changing your ranking behind your back. The behavior stream this article is about is the fuel: the richer your UBI data, the more the agent has to reason over.
Start recording
The behavior data you need to improve search is being generated right now, on every query your users and agents run, and it disappears the moment you fail to record it. UBI is the path to keeping it: an open schema, a client collector, and a supported collection path in Amazon OpenSearch Service, all landing in indices you already know how to query. Instrument the queries that matter, watch what users and agents do with the results, and let the grades they have been giving you all along drive what you ship next.
Top comments (0)