Originally published at Trida AI.
Most retrieval failures we see are not retrieval failures. The search worked. The problem is that the system spent the same effort on every question: a planning loop with several model calls to answer a lookup that sits in one checklist, and a single hopeful search for a comparison spread across four documents. Effort was allocated by habit, not by the question in front of it.
That is why a small change in Amazon Bedrock Managed Knowledge Base caught our attention. The AgenticRetrieveStream API already offered agentic retrieval, where a model plans searches instead of running one. AWS has now widened one parameter, maxAgentIteration, to a range of 0–5, and in doing so turned retrieval depth into a per-request choice. One dial, three very different machines behind it.
What one parameter now covers
At 0, the service runs one retrieval and returns the passages. No model is called at all, and asking it to also generate an answer at this setting is rejected as invalid. At 1, it retrieves once and generates an answer from what it found: classic single-shot RAG, with no planning and no query rewriting in between. From 2 to 5, the agentic mode takes over. A model breaks the question into sub-queries, searches for each part, judges whether the evidence covers the question, and searches again where it finds gaps.
The counting matters, because it is where the money goes. One iteration equals one foundation model call, and the last call is always the one that writes the answer. So a setting of 4 does not buy four searches. It buys up to three planning rounds, each a query rewrite plus a retrieval, and then the answer. A setting of 5 on a question that needed one lookup can cost up to four extra model calls confirming what the first retrieval already held, on the rounds where the planner does not stop early.
Two details keep teams honest. First, all three modes can search multiple knowledge bases, so single-shot does not mean single-source; what the higher settings add is decomposition and the sufficiency check, not reach. Second, an initial speculative retrieval runs ahead of the first planning step and never counts against the limit, which means the dial prices the thinking, not the first look.
How the setting is chosen for each request. The class is decided in advance; the parameter is set per request.
The evidence cuts both ways
The case for the high settings is real, and AWS's own numbers show where it lives. On MuSiQue, a benchmark of multi-hop questions, AWS reports agentic retrieval improving recall by 20 points over single-shot retrieval, with the largest gains on the hardest questions. Read the next sentence of that result too: on single-hop questions the gain drops under five points. Depth pays where the question genuinely has parts. On a question with one part, it mostly pays the vendor.
That asymmetry is the whole argument. Agentic retrieval is priced per call, on top of the underlying retrieval calls, and each planning round adds its latency in sequence, because the planner has to read what it found before deciding what to look for next. None of that is an argument against the mode. It is an argument against leaving it on for everything, which is what happens when a team sets 5 once, sees the hard questions improve, and never looks at what the easy majority is costing.
Set it by query class, not by request
The parameter is per request, but we would not let it be decided per request, and certainly not by the model answering the question. Decide it in advance, per class of question, the way you would set any other budget:
- Passages only (0) when something downstream does its own reasoning: another agent, a reviewer who wants the source text in front of them, a UI that shows evidence rather than prose. If a person asked for the document, do not summarise it at them.
- Single-shot (1) as the default for direct, well-scoped questions, which in most internal assistants is most of the traffic. One retrieval, one answer; the only faster setting returns passages, not an answer.
- Agentic (2–5) for questions with visible structure: comparisons, multi-part asks, anything whose evidence lives in more than one knowledge base. Start at 3 for a single knowledge base and reserve 4–5 for the comparative, cross-source classes, then adjust from traces rather than taste.
What makes this stick is the trace. Agentic retrieval streams trace events for its planning and retrieval steps, so you can see, per class, how many rounds actually ran and whether the extra rounds found anything the first search missed. Review that monthly. A class whose planner almost always stops after one round is a class you are overpaying for. A class at setting 1 whose answers keep missing a second document is a class to promote. The dial is only useful if somebody reads the meter.
One comparison question, run at two settings. The extra rounds at setting 4 are what found the third part. Example content.
One question, two settings
Take a question of the kind that lands in a support assistant: "How do the refund rules differ between annual and monthly plans, and what exceptions apply to education accounts?" At setting 1, the service retrieves once against the whole sentence. The passages that come back cluster around whichever plan type dominates the wording, and the answer is fluent, cited, and quietly missing a leg. Nothing in the response tells you a part was never searched for.
At setting 4, the same question is taken apart first: one sub-query per plan type, one for the education exceptions. The sufficiency check is the step that earns the money here. It can see that the exceptions evidence is thin and spend another round on it before answering. The answer costs several model calls instead of one and arrives later. For this question, that trade is obviously right. For "what is the refund window on a monthly plan?", the identical machinery would be theatre.
Where the dial does not help
A higher setting cannot fix a corpus problem. If the passage that answers the question was never indexed, or was chunked away from the sentence that qualifies it, more planning rounds just search the same gap more thoroughly. Nor does the parameter make the routing decision for you: AWS is explicit that choosing the mode is application logic, so the classifier, the class definitions and the defaults are yours to build and yours to get wrong. And the managed service sets the terms of the trade: agentic retrieval runs only against fully managed Bedrock knowledge bases, so teams running their own vector store are building this dial themselves or going without.
Our read is that the interesting part of this release is not the agentic mode, which existed, or the single-shot mode, which is the oldest pattern in RAG. It is that a major platform has put the choice in one named, per-request parameter, with the cost of each notch legible in model calls. Retrieval depth is becoming a budgeted resource, like tokens or time. Budget it the same way: default low, raise it where the evidence says the question earned it, and check the meter often enough that the setting stays a decision instead of becoming a habit.
Work with us
If your retrieval bill is set by one global configuration nobody has revisited since launch, the fix is rarely a new search stack. It is classifying the questions, pricing each class honestly, and wiring the depth to the class. Bring us the assistant where every lookup is paying for a planning loop, or where comparisons keep coming back half-answered, through Trida AI's intake and we will show you where the dial should sit for each class of question you actually get.


Top comments (0)