AWS Bedrock's built-in eval graded my agent green. It only looks at a quarter of it.

I built an agent on AWS Bedrock the low-code way. Pick a foundation model, wire an action group to a Lambda, attach a Knowledge Base, add a Guardrail. Then I ran Bedrock's built-in Model Evaluation job, got good scores, and shipped.

Production disagreed almost immediately. The agent was dropping a chunk of its tool calls on a task the built-in eval had passed. The model was fine. Everything wrapped around the model was not, and the built-in check could not see any of it.

Here is what I learned about actually evaluating a Bedrock agent, in plain terms.

The built-in eval scores the model, and your agent is four things

This is the core mistake, and it is baked into the tool. Bedrock's Model Evaluation job runs a dataset against one foundation model and scores it for accuracy and safety. That is a fine sanity check on the raw model.

But a Bedrock agent is not just a model. It is a model, plus action groups (the tool-call layer that runs your Lambdas), plus a Knowledge Base (the retrieval layer), plus Guardrails (the safety layer). The built-in eval sees the first piece and is blind to the other three. So three-quarters of what your user actually experiences never gets graded.

It also runs once, as a one-shot job. There is no path from "I re-indexed the Knowledge Base" or "I tweaked the Guardrail" back into a check that gates your next deploy. Production agents need the eval to run on every change, not once at the start.

So the fix is to stop grading the model and start grading the three layers around it. Three axes.

Axis one: did it call the right tool, correctly

Action groups are where my agent broke first. Getting this right means three separate things, and all three matter:

• Did it pick the right tool, including picking none? This is the one people forget. You need test cases where the correct answer is to not call any tool at all. Without those, the failure where a prompt tweak makes the agent call tools too eagerly stays invisible until users complain.
• Were the arguments actually right? Not just valid against the schema, but semantically correct. A date like "2026-01-01" can be perfectly valid and still wrong if the user said "next Friday." Schema-valid and correct are two different checks.
• Did it use what the tool returned? I hit exactly this. A tool returned a real account balance, and the model answered without reading it, just making up a plausible number. If the agent ignores its own tool output and falls back on invented knowledge, every earlier check can pass and the answer is still wrong.

Axis two: did it retrieve the right documents

If you use a Bedrock Knowledge Base, the retrieval happens inside the agent run, which means a bad-retrieval bug looks exactly like a bad-model bug unless you score retrieval on its own.

So I score the retrieval step separately from the final answer. Build a small set of questions with the specific documents that should come back for each, and check how often the right ones actually land in the results. Then check the answer separately. Now you can tell the two apart: if retrieval is bad, fix the chunking or the embeddings; if retrieval is good but the answer is ungrounded, fix the prompt or the model. Without the split you spend a week tuning the wrong layer.

Two Bedrock gotchas worth knowing before they bite you:

• Changing the embedding model quietly drops your recall. Swapping the embeddings behind a Knowledge Base reshuffles which chunks come back, and I have seen it drop retrieval quality by roughly ten points with nothing else changed. Re-run the retrieval check after any embedding change.
• Non-English content degrades silently. If your docs are multilingual, tag each test case by language and watch the non-English subsets on their own. The regression where German or Hindi retrieval quietly falls behind English is one of the most common and least noticed.

Axis three: are the guardrails catching attacks without blocking real users

Bedrock Guardrails fire on word lists and rules. They are fast and predictable, and they fail in two opposite directions at once. They miss attacks that are phrased around the filter, and they block legitimate queries you forgot to test.

So I score them on two labelled sets:

• A benign set of normal queries that should pass. What fraction did the Guardrail wrongly block? That is your false-block rate, and it is quietly making your support queue longer.
• An adversarial set of jailbreaks, injections, and real policy violations that should be blocked. What fraction did it actually catch?

Report both, per Guardrail version. A policy that catches almost every attack but blocks a tenth of real users is not obviously better than one that lets a few more through but rarely annoys anyone. The eval just shows you the trade-off honestly so you can pick the operating point on purpose instead of by accident.

Gate on each axis, not on one number

The tempting move is to average everything into one agent score and gate on that. Do not. An aggregate of 0.85 can easily hide a 0.62 on parameter correctness behind a 0.97 on tool selection, and the production failure rides on the weak one.

So I set a separate threshold per axis: tool selection, parameter correctness, retrieval quality, groundedness, guardrail precision, guardrail recall. When the gate fails, the name of the failing check is the root cause. One bisect instead of three days.

The trap that is unique to Bedrock: the model swap

Bedrock makes changing the model behind your agent a one-line change. That is convenient and it is a trap. A pass on one model is not a pass on another. I have watched one model handle an action group perfectly while another dropped it on the same input. So run the same suite against every model your agent could actually resolve to, not just the one you developed against.

The mistakes I would warn a friend about, all learned the hard way: trusting the built-in eval as the agent eval, testing on one model when you swap models in production, treating the Knowledge Base as a black box instead of scoring retrieval on its own, and scoring a Guardrail on attacks only while never checking how much real traffic it blocks.

The lesson I keep coming back to is that Bedrock's low-code assembly hides how many moving parts your agent actually has, and the built-in eval only looks at one of them. Once I scored the three layers around the model, the failures that had been slipping to production were obvious before the deploy.

If you want the deeper version, with the exact retrieval and guardrail rubrics and how to wire the per-axis gate into CI, this piece goes through all three axes in detail.

If you run Bedrock agents, I am curious which layer broke on you first. For me it was action groups, an agent that called the right tool and then ignored what it returned.