If you have built a RAG pipeline or a search feature, you have probably shipped a ranking function and never touched it again. This post is about changing that with a few ideas borrowed from reinforcement learning.
Most retrieval systems are frozen on the day they ship. The embedding model was trained once, the ranking formula was written once, and from then on every query is scored the same way, whether the last thousand users found the results useful or ignored them completely. That is a strange design for a system that sees feedback on every single request.
Reinforcement learning offers a way out. The ideas come from game AI and robotics, but they map cleanly onto search, RAG pipelines and agent memory.
Why Static Ranking Stops Improving
A retrieval system takes an action every time it answers a query: it returns a ranked set of results. It then gets to observe what happened. Did the user click the first result or scroll past it? Did the model actually use the retrieved passage in its answer? Did the user rephrase the question a moment later?
Those behavioral signals are implicit rewards. A static system throws them away. An adaptive system treats them as training data, running a simple loop: serve results with the current ranking policy, observe the outcome, turn the outcome into a reward, and adjust the ranking so the expected reward goes up next time.
Designing a Reward Function From Noisy Signals
The loop is easy to describe and hard to get right, because feedback in retrieval is noisy, delayed and ambiguous. A result nobody clicked might be irrelevant, or the snippet might have answered the question already. A reformulated query might mean the results were bad, or that the user moved on to a related topic.
Single signals are easy to game. Click through rate rewards clickbait, dwell time rewards long content regardless of quality, and reformulation rate punishes exploration. Composite rewards that weight several signals together hold up much better, and the strongest signal of all is task completion: the bug got fixed, the ticket got resolved, the user found what they needed. There is a practical walkthrough of designing reward functions for memory systems if you want to go deeper on the tradeoffs.
Bandits, Replay and When to Update
You do not need a full deep RL stack to start. Multi armed bandits capture most of the value for ranking. Each ranking strategy is an arm, and the system balances exploiting the best known ranking against exploring alternatives that might be better. Epsilon greedy is the simplest version, serving a variation a small percentage of the time. Thompson sampling is smarter, exploring strategies in proportion to how uncertain their quality still is. Contextual bandits go one step further and learn different strategies for different query types or users.
Experience replay, borrowed from game playing agents, stores tuples of query, results served and user behavior, then reprocesses them in batches. That decouples learning from serving, so updates never add latency to live queries, and it averages out the noise in individual interactions. Most production systems end up with a hybrid: a fast online layer for small immediate adjustments and a slower batch layer for changes to the ranking formula itself.
Evidence Gating and Production Guardrails
The biggest risk in production is learning the wrong lesson from a handful of coincidences. Evidence gating addresses this by refusing to change behavior until a pattern shows up across multiple independent interactions. A memory that helped once might be luck. A memory that helped across ten queries from five different users is evidence. The idea is covered in more detail in this piece on evidence gated learning.
A few other guardrails matter just as much. Start new users and new content from a strong baseline, like similarity plus recency, so cold start is not a coin flip. Make ranking changes gradual and reversible. Log every policy change with the evidence behind it, and track retrieval metrics like mean reciprocal rank and recall at k so you can roll back when a change hurts.
The Takeaway
A retrieval layer that never learns is leaving its best training data on the floor. Start with a clear reward, add a bandit or a simple feedback loop, gate every update behind real evidence, and keep a baseline to fall back on. The full guide to reinforcement learning for AI systems walks through each of these pieces, from feedback loops to production monitoring.
Top comments (0)