DEV Community: Raj Murugan

Field Notes: 25 seconds to detect and hand off. Nothing executes until a human says yes.

Raj Murugan — Fri, 23 Oct 2026 04:22:52 +0000

A Strands agent on Amazon Bedrock AgentCore found a real CloudWatch alarm, worked out what it meant, and handed the finding to a LangGraph agent over A2A. Median time, five runs: 25.3 seconds. The LangGraph agent proposed a fix. It did not apply it. Nothing applies anywhere in this system until a human types y, and that is enforced by a Cedar policy on an AgentCore Gateway, not by a line in a prompt asking nicely.

I had Claude Code build and deploy the whole thing in my own AWS account, including hitting the IAM gotchas below for real. The demo's first working version had the agents built, the A2A protocol measured, and the real CloudWatch tools deployed, and still fed both agents a hardcoded finding instead of a real one. It ran, it measured something true, and it was not the story: nothing had detected anything. Two review passes caught that before anything shipped, and the fix is what this post is about.

In short

Two agents, built in different frameworks, detect a real CloudWatch alarm and hand it off to each other over A2A: 25.3 seconds median, five of five real runs.
The remediation agent only proposes an action. Nothing executes until a human approves, enforced by a Cedar policy on an AgentCore Gateway, tested by trying to break it: no approval denied, wrong resource denied.
A2A's native-invoke shortcut and the full protocol turn out to be the same API call, and the AWS SDK cannot make either once OAuth is on.

The setup

A monitoring agent on Strands has real, read-only CloudWatch tools: list alarms in ALARM state, pull metric statistics, filter logs. A remediation agent on LangGraph matches a finding against a small local runbook and asks a model for a grounded recommendation. Different frameworks on purpose, because the point of A2A is that it does not care what you built the other side in. Not Bedrock Agents' own supervisor/sub-agent collaboration, which has done this natively since March 2025, because that requires both sides to be Bedrock Agents; the whole reason to reach for A2A instead is when they are not. Both run as AWS::BedrockAgentCore::Runtime A2A servers behind a shared Cognito user pool, and the monitoring agent authenticates itself to the remediation agent through an AgentCore Identity outbound credential provider, never a secret in its code, environment, or image.

Phase one: detection and hand-off

Five runs. Each one, the monitoring agent called its own CloudWatch tools, found the real test alarm, and confirmed the metric value against it. Not a hardcoded string: an actual describe-alarms and get-metric-statistics check against ground truth, every run, with misses counted rather than retried away. Five out of five found it.

What 25.3 seconds measures, precisely: the clock starts when the test client asks the monitoring agent to check for incidents, and stops when that same client has the remediation agent's recommendation back in hand, already including detection, the A2A hand-off, and the recommendation itself. Not the time the alarm had already been sitting in ALARM before anyone asked. Five runs: 23.741s, 25.270s, 23.929s, 26.418s, 25.398s. Median 25.270s, straight from docs/transcript.txt, the run shown below.

The median run's real timeline, timestamps from AWS (StateUpdatedTimestamp) and from the containers themselves (UTC, not the laptop running the test):

Step	Time	Elapsed
Test client's request sent (clock starts here)	03:37:10.71	0.0s
Monitoring agent's request begins	03:37:12.06	~1.4s
CloudWatch tools confirm the real alarm and metric	03:37:17.94	~5.9s
AgentCore Identity token fetch	03:37:18.01	~6.0s
Remediation agent's card fetched	03:37:25.52	~13.4s
`message/send` reaches the remediation agent	03:37:32.99	~20.9s
Runbook matched (`error_rate`)	03:37:32.99	~20.9s
Recommendation returned (Bedrock call)	03:37:34.59	~22.5s
Full round trip complete	03:37:35.97	25.3s

That hand-off is the thing the demo's first working version could not claim. The monitoring agent fetches the remediation agent's own card and sends the message/send itself, with its own OAuth token. The test script only starts the run; it never calls the remediation agent directly. Two agents, two frameworks, talking to each other, not a script talking to both of them in turn.

Phase two: propose, approve, enforce

The remediation agent's recommendation is a proposal, nothing more. Executing it needs three things to line up: a human approval, logged with a timestamp and the approver's real IAM identity; the target resource carrying the right tag; and a Cedar policy on an AgentCore Gateway that evaluates both before anything runs.

I tested both ways it should fail, not just the way it should succeed:

No approval. A call with approved: false against the correct, tagged resource. The real response my client captured, exact text: -32002, "Tool Execution Denied: Tool call not allowed due to policy enforcement [No policy applies to the request (denied by default).]". AWS's own devguide shows the same "Tool Execution Denied" message wrapped differently, as a success envelope with isError: true rather than a numbered JSON-RPC error; treat the message text as the documented signal and the -32002 as what this Gateway target sent back in practice, not something AWS's docs assign a fixed meaning to. Checked independently with a raw ssm:GetParameter read: the value never moved.
Wrong resource. A second, genuinely untagged SSM parameter, called with approved: true. Same Cedar denial. The only difference between this parameter and the real one is its tags, confirmed with ssm list-tags-for-resource, not assumed.
Approved, on the correct resource. Allowed. The flag flips from false to true, confirmed by reading the parameter directly, not by trusting the Lambda's own return value.
Rollback. A second logged approval reverses it, through the identical Gateway and policy path, confirmed back to false the same way.

The denial is a Cedar AuthorizeAction default-deny, evaluated by the policy engine before the Lambda target is ever invoked. runbooks.json's own default entry already says, in English, "do not take an automated remediation action on an unclassified finding." That line does nothing on its own. It is advice sitting next to a model that can ignore advice. The Cedar policy is what stops the call, and I built both tests specifically so that claim had to survive an attempt to break it, not just a reading of the prompt.

What it cost

Phase one: around $0.27 for the full session, almost all of it Bedrock tokens across the real detection, hand-off, and recommendation calls, pulled from CloudWatch's own AWS/Bedrock usage metrics. Phase two made zero model calls (the proposal step is local logic against the runbook, not a new generation), so its entire cost is a handful of Gateway invocations and policy authorisations at AWS's published list rates, a few hundredths of a cent. Neither session came close to the $5 cap I set before running either.

The proof: what A2A costs

Before any of the above worked, I measured what the A2A hand-off in phase one costs against two cheaper-looking alternatives, with the same hardcoded test finding run through all three for a clean comparison: a direct in-process call (both agents' code, same process, no AgentCore hop at all), AgentCore's native InvokeAgentRuntime with a raw payload and no A2A envelope, and the full A2A handshake (agent-card discovery, a Cognito OAuth token, then a JSON-RPC message/send).

Pattern	Median (n=5)
Direct call	2.4s
Native invoke, raw payload	8.4s
Full A2A (control run, same finding as above)	18.8s

Native invoke and full A2A turned out to be the same operation, not two tiers: both go through the identical InvokeAgentRuntime call against the identical runtime ARN. The only difference is payload shape and whether agent-card discovery runs first. And the AWS SDK cannot make either call once a runtime has a JWT authoriser attached, which both of mine do: AWS's own API reference says so directly, you drop to a raw HTTPS request with a bearer token instead. A runtime takes IAM SigV4 or JWT inbound auth, never both, and SigV4 only works if the other agent is already inside your AWS account, which defeats the entire premise of A2A, agents from different platforms talking to each other. For real cross-vendor A2A, OAuth is not a choice, it is the only option that keeps the protocol's own promise, and that is the option that breaks boto3.

Sharp edges I hit

Getting the credential right took five live attempts, not one. The monitoring agent needs its own OAuth token to call the remediation agent, and the one rule that mattered most to me was that the token's secret could never sit in code, an environment variable, or the container image. AgentCore Identity's outbound-auth credential provider does this: the Cognito client secret lives in AgentCore's own vault, fetched at call time. Getting the IAM right to let the monitoring agent fetch it took five separate AccessDeniedExceptions, each naming a different resource in the chain: the credential provider, the runtime's workload identity, the workload-identity directory, the token vault, and finally a Secrets Manager secret AgentCore Identity stores the token in behind the scenes, which the calling role also needs direct read access to. None of that last part is documented anywhere I found ahead of hitting it.

The first working fix was itself an anti-pattern, caught in review. Faced with four different access denials, the fastest fix was two account-wide wildcards: token-vault/* and workload-identity-directory/*. It worked. It was also exactly the kind of overbroad grant this whole exercise exists to avoid shipping in a public repo. AWS publishes the narrow form for this specific case (scope-credential-provider-access in the AgentCore devguide): four exact ARNs, built from the stack's own resource attributes, not a namespace wildcard. Redeployed with the narrow policy, reran the real hand-off twice, zero denials.

Two bugs nothing had ever exercised before. The monitoring agent's container was missing the a2a-sdk dependency outright, latent since the very first build, because nothing had ever sent it a real A2A message before this one did. Strands' own A2AServer doesn't extract AgentCore's per-request WorkloadAccessToken header on its own either; a small FastAPI middleware has to do it.

A region pin that silently did nothing. The project's Makefile pinned the region with ?=, which is a no-op the moment the calling shell already exports AWS_REGION. Mine did, to a different region entirely. The first real test alarm was created somewhere I wasn't looking. Fixed to :=.

AgentCore Gateway needs its own self-referential ARN before it exists. Attaching a Cedar policy engine to a Gateway at creation time triggers an authorisation check against the Gateway's own ARN, mid-create, before AWS has finished assigning it one. The fix is the same shape as the credential-provider one above: a narrow, name-prefixed wildcard scoped to this stack's own Gateway name, confirmed by reading the literal generated ID out of two real CREATE_FAILED deploys, not guessed.

What I now do

Treat an agent's proposal and an agent's permission to act as two different trust boundaries, built two different ways. The proposal can be as creative as the model wants; a runbook, a classifier, a recommendation in plain English. The permission cannot live in that same layer at all. It has to be a policy engine that evaluates a resource tag and an approval record before anything runs, and the only way I'd trust that claim about my own system is the way I tested it here: try to break it on purpose, twice, and watch it hold.

What this doesn't tell you

Five real runs is a real result, not a load test; the 25.3 seconds will move under concurrent traffic and a cold start. Detection here is one synthetic alarm, one metric, one runbook category, in an otherwise quiet test account with nothing else in ALARM to compete for the agent's attention. A noisy production account with several real alarms firing at once is a different, harder problem I haven't tested, and a model-driven detector can still misread a finding type I didn't try. The approval step in this build is a script prompting for y with a logged identity behind it, not a UI, a Slack bot, or a change-management system, though the Cedar enforcement underneath would look the same either way. And the one action I let the system execute, a feature-flag toggle, is about as low-stakes as a real action gets; a system that can touch something expensive needs the same two tests run against that action specifically, not assumed to transfer from this one. There is no idempotency key on the Gateway-to-Lambda execution call either: a retried approval after a timeout would be invisible here, because flipping a flag true to true twice looks identical to flipping it once. A non-idempotent action needs that key before any of the rest of this applies to it.

Run it yourself

The repo is rajmurugan01/agentcore-a2a-incident-response. make deploy, make incident, make demo for phase one; make tier2-demo for phase two. make destroy tears down everything in both, independently verified each time, including the test alarm and both SSM parameters.

If your agents already detect and propose: what stops the next step, a policy that denies by default, or a sentence in a prompt asking the model not to?

Sources

Invoke an AgentCore Runtime agent (AgentCore devguide: the SDK-vs-raw-HTTPS limitation once OAuth is on)
Deploy A2A servers in AgentCore Runtime (AgentCore devguide: the port 9000, root-path convention)
Scope down access to credential providers by workload identity (AgentCore devguide: the exact least-privilege ARNs for outbound auth)
Set up permissions for Policy (AgentCore devguide: Gateway and Policy Engine IAM)
Amazon Bedrock AgentCore is now generally available (13 October 2025)

LangGraph builds the agent. AgentCore runs it.

Raj Murugan — Tue, 13 Oct 2026 02:06:38 +0000

The question I keep getting from engineering teams is some version of "should we build on LangGraph or on Bedrock AgentCore?" It is the wrong question, and answering it as a versus almost always sends people down the wrong path. They are not competitors. They sit at different layers of the stack, and the clearest way to see that is to take the things you actually design when you build an agent and ask, for each one, which tool owns it.

The six things you design in an agent

When you author a real agent, not a demo, you are designing six things:

State: the typed object that carries everything the agent knows this turn, and the rules for what accumulates versus what gets overwritten.
Nodes: the discrete steps. Classify intent, plan, generate a query, validate it, run it, retrieve documents, check grounding, synthesise, format the answer.
Edges: how those steps are wired together.
Routing: the conditional logic that decides which edge to take, ideally deterministic code reading the state, not the model choosing freely.
Loops: the reason-act-observe cycle, and the bounds that force it to terminate instead of spinning.
Checkpoints: persisting the graph's state so a run can survive a crash, pause for a human, or be replayed for audit.

If you have written anything non-trivial in LangGraph, you have hand-built all six. That is what LangGraph is: a framework for authoring the control flow of an agent as an explicit graph. It gives you StateGraph, reducers, conditional edges, a recursion_limit (25 steps by default before it raises GraphRecursionError), and a checkpointer. It is opinionated about how the agent thinks.

What AgentCore actually is

AgentCore is not that. It is a modular set of managed services for running and operating agents in production: Runtime, Memory, Gateway, Identity, Observability, plus built-in Code Interpreter and Browser tools, all generally available since 13 October 2025. Two more pieces arrived later and on their own schedule. Policy and Evaluations launched in preview on 2 December 2025 and reached GA separately, Policy on 3 March 2026 and Evaluations on 31 March 2026. A tenth piece, a managed agent loop called Harness, came later again: public preview in April 2026, general availability in June 2026. So by this post's publish date, Policy has had about seven months of production hardening, not the year a single "went GA in late 2025" claim would imply, and Harness about four. The two design facts that matter most:

It is framework-agnostic and model-agnostic. It runs agents built in LangGraph, CrewAI, LlamaIndex, Strands, the OpenAI Agents SDK and others, against models inside or outside Bedrock (Claude, Nova, GPT, Gemini, Llama, Mistral). The Runtime service contract speaks HTTP, MCP, A2A and AGUI.
AWS's own framing: the framework owns the orchestration logic, AgentCore owns the runtime infrastructure it runs on. AgentCore does not ask you to redraw your graph. It asks you to deploy it.

So the honest mental model is not "LangGraph or AgentCore." It is "author in LangGraph, operate on AgentCore," and the interesting engineering is in the seam between them.

One honest gap in that framing: AgentCore is not the only place a LangGraph graph can run. LangGraph ships its own managed runtime, the product most people still know as LangGraph Platform or LangGraph Cloud, now folded into what LangChain's current docs call LangSmith Deployment. Its fully managed Cloud tier runs your agent inside LangChain's own AWS and GCP infrastructure, not your AWS account. Its self-hosted and hybrid tiers run your own Kubernetes cluster, which trades a managed service for a cluster you now operate. AgentCore Runtime is neither of those: the agent runs inside your own AWS account on a managed microVM, under the IAM you already administer, and AgentCore Observability ships built-in CloudWatch metrics and a per-agent log group out of the box, not a separate vendor dashboard you also have to maintain. If you are already on AWS and the thing you actually want is for the infrastructure boundary to stay your account rather than a vendor's, that is the concrete reason to deploy LangGraph onto AgentCore Runtime instead of onto LangGraph's own platform, not just "because AWS."

Mapping the six primitives

What you design	Who owns it	What the other layer adds
State (schema, reducers)	LangGraph	AgentCore Memory adds managed short-term and long-term memory that persists across sessions and can be shared across agents (scoped per actor/namespace, your responsibility to configure correctly). It does not define your state schema.
Nodes and edges	LangGraph	AgentCore has no concept of your nodes. The exception is Harness, below.
Routing	LangGraph	AgentCore Policy intercepts every tool call at the Gateway boundary and allows or denies it against Cedar policies, outside your agent's code. Governance, not routing.
Loops	LangGraph (`recursion_limit`, token budgets)	AgentCore Runtime gives the loop a durable place to run, up to eight hours per session on the standard microVM path (the `maxLifetime` default and ceiling there; a capacity-provider deployment can raise it further), with true session isolation.
Checkpoints	LangGraph checkpointer	AgentCore Runtime and Memory change the durability problem entirely. This is the one worth sitting on.
Tools	Either	AgentCore Gateway turns your APIs and Lambda functions into MCP-compatible tools with a few lines, so tools become reusable across agents instead of hard-wired into one.

The checkpoint story is where it gets interesting

Here is the part I did not appreciate until I built an agent on Lambda.

Lambda gives you fifteen minutes and no memory between invocations. A multi-step agent that calls a model, waits, calls a tool, waits, reasons again, can bump into that ceiling, and if the function dies mid-run you have lost every tool call it already made. So on Lambda you need a checkpointer. You persist the graph state at every step, and on the next invocation you resume from the last checkpoint instead of re-running everything. The checkpointer is not a nice-to-have there. It is how you survive a stateless, time-boxed compute model.

Checkpointing is not idempotency

One thing the checkpointer does not buy you: a resumed run can replay a tool call whose side effect already landed, because durable state and exactly-once side effects are different guarantees. If a node charges a card or sends an email, resuming from the last checkpoint after a crash can fire it twice. That needs its own idempotency key, keyed off the checkpoint step id, independent of whichever compute layer you are on.

AgentCore Runtime changes the shape of that problem. Sessions are isolated microVMs and can run for up to eight hours on the standard deployment path, with streaming. The runtime holds the session, so you are no longer checkpointing to work around a fifteen-minute wall. Checkpointing becomes what it should have been all along: resilience against genuine failures, a pause point for human-in-the-loop approval, and a replayable trace for audit. Same primitive, completely different reason for reaching for it.

One sharp edge worth knowing before you lean on that eight-hour number: it covers active execution, not idle waiting. A session only stays alive while your code reports itself busy on its own /ping health check; left idle, with no pending work, it is auto-terminated after fifteen minutes regardless of the eight-hour ceiling. So the eight hours does not buy you a longer wait for a human. A human-in-the-loop pause that outlasts fifteen minutes is not "the session waits." It is "checkpoint, let the session end, and resume a new one from that checkpoint once the human responds," the same discipline you would need on Lambda, not a reason to skip it because the runtime feels long-lived. What the eight hours actually buys you is sustained active work Lambda cannot: Lambda's own fifteen-minute ceiling is a hard execution limit that fires even while the function is genuinely busy, so a long document-parsing pass or a multi-tool research loop that runs continuously for forty-five minutes has nowhere to go on Lambda without being chopped into checkpointed segments. On AgentCore Runtime, as long as the agent keeps reporting HealthyBusy, that same continuous run just keeps going.

That is the practical lesson. The compute layer you deploy onto quietly dictates how hard your framework's persistence layer has to work. Pick the runtime first, and half your checkpointer decisions make themselves.

The one place AgentCore does orchestrate: Harness

There is a real exception to "the framework owns the loop." AgentCore Harness is a managed agent loop, powered by Strands Agents under the hood. You define an agent with CreateHarness and run it with InvokeHarness: a model, a system prompt, tools, memory settings and execution limits as configuration, no orchestration code to write. For a lot of agents, a support assistant, a research task, a data-analysis helper, that is genuinely all you need, and reaching for LangGraph would be over-engineering.

The line I draw: if your control flow is "reason, pick a tool, repeat until done," Harness is enough. The moment you need explicit routing, bounded repair loops, a grounding gate that can send the agent back to retrieve again, or a human approval step wired into a specific edge, you want a real graph, and that is LangGraph deployed onto Runtime.

How I would choose

Prototyping a straightforward agent, or you do not want to own orchestration code: Harness on AgentCore.
Non-trivial control flow, deterministic routing, bounded loops, auditability: author the graph in LangGraph.
Either way, when it goes to production: deploy onto AgentCore Runtime and adopt Memory, Gateway, Identity, Observability and Policy rather than hand-rolling session isolation, tool plumbing, auth and tracing yourself.
Non-negotiables like "the agent must never make an authorization decision" or "this tool call needs approval" belong in Policy at the Gateway, outside the model, no matter which framework drew the graph.

The versus framing costs teams weeks, because they treat a runtime decision and an authoring decision as one choice and then relitigate it. Separate the two. LangGraph is how the agent thinks. AgentCore is where it lives. Design each on its own terms and the seam between them is where the good architecture actually is.

One honest limit on this mapping: it is a snapshot of where the line sits today, not a law. Policy, Evaluations and Harness all reached GA well after the rest of AgentCore (March and June 2026 against October 2025), which means the "who owns it" column has already moved at least twice since the services this post describes first shipped, and AWS ships new AgentCore capability on a near-monthly cadence. If you are reading this more than a few months after publication, check the current docs before you redraw this table from memory, the same rule this post applies to itself.

Sources: Amazon Bedrock AgentCore developer guide (Runtime lifecycle settings, the /ping idle-session timeout, Policy, Harness, Observability/CloudWatch integration), AgentCore FAQs, the AWS "AgentCore is now generally available" announcement (13 October 2025), the AWS News Blog post "Amazon Bedrock AgentCore adds quality evaluations and policy controls for deploying trusted AI agents" (Policy and Evaluations GA, March 2026), the AWS "AgentCore Harness is now generally available" announcement (June 2026), and LangChain's LangSmith Deployment documentation (deployment options and tiers).

The ML you need to operate LLMs, not train them

Raj Murugan — Tue, 06 Oct 2026 08:29:02 +0000

You do not need to understand backpropagation to run a large language model well in production. You need a smaller, more practical thing: the operator's mental model. What a token actually costs you, what a context window actually bounds, what temperature and top_p actually do, and why evals catch what your own reading of the output will not. None of this is deep learning theory. All of it is the part that bites when you skip it.

Tokens are not words, and the count is not universal

A token is the unit the model actually reads and the unit you are actually billed on, and it is not a word. Common English words are usually one token. Longer or less common words split into pieces. Punctuation, whitespace, and anything outside the tokeniser's common vocabulary (a rare name, a non-English string, a stretch of code) can cost more tokens than it looks like it should.

The part that catches people operating across model families on Bedrock specifically: the same text does not tokenise to the same count on every model, because different model families ship different tokenisers. I measured this directly on a real production system prompt while working on prompt caching: the identical string came out to 8,156 tokens on the Nova tokeniser and 8,788 tokens on the Anthropic tokeniser, an 8 percent difference from the exact same characters. If you are estimating cost or checking against a context limit, "how many tokens is this" has a different answer depending on which model you ask, and a number measured against one family does not transfer to another.

A context window is a shared budget, not a per-turn allowance

"Context window" sounds like it means how much the model remembers. Operationally, it means something narrower and more consequential: the hard ceiling on everything in a single request, added together. Your system prompt, your tool definitions, the conversation history you send back each turn, and the model's own output all draw from the same number. None of them have their own separate budget.

This is why a long tool registry or a verbose system prompt is not free even before a user has said anything: it is the floor your actual conversation history has to fit above. On Bedrock today, that ceiling varies a lot by model: Claude Sonnet 4.6's model card lists a 1,000,000-token context window, though AWS's own beta-header table also lists context-1m-2025-08-07 against this model, so confirm empirically whether you need it before assuming the full million applies with no header set. Amazon Nova's own range spans from 128,000 tokens on Nova Micro up to 1,000,000 on Nova 2 Lite, with Nova Pro and Nova Lite sitting at 300,000. Picking a model by capability alone and finding out its context ceiling later is a real way to discover, mid-project, that your working budget just got a lot smaller.

temperature and top_p are not independent dials

Generic ML content treats temperature and top_p as two separate knobs you can tune together: temperature reshapes the probability distribution, top_p truncates it. On Bedrock, AWS documents this exactly for two specific models, Claude Sonnet 4.5 and Claude Haiku 4.5: "support specifying either the temperature or top_p parameter, but not both." That is a named, model-specific constraint, not a blanket rule for "4.x and up," and the same shape of failure keeps turning up on newer releases without AWS updating that page to match, so check per model rather than assuming generational coverage.

Send both temperature and top_p in the same inferenceConfig to one of these models and the Converse API rejects the call. The exact wording I have seen reported, consistently, across multiple independent client libraries and agent frameworks is "temperature and top_p cannot both be specified for this model. Please use only one." AWS's own documentation states the constraint in prose, not that literal string, so treat the error text itself as an observed runtime message, not a documented, versioned contract. Either way, this is not a rare edge case: the common trigger is not a developer deliberately setting both, it is a client SDK that sends both by default (one explicitly set, one left at its base-config default) without the caller ever intending to.

The operator fix is boring and worth stating plainly: pick one. Keep temperature unless you have a specific reason to reach for top_p, and when you touch a client library's defaults, check what it is silently sending alongside the parameter you meant to set.

"Pick one" stops being enough the moment extended thinking is on. AWS's own extended-thinking documentation is explicit that thinking is not compatible with temperature, top_p, or top_k modifications at all, so with thinking enabled the fix is to omit both, not choose between them.

# This fails on Claude Sonnet 4.5 / Haiku 4.5 and newer releases that share
# the same constraint, if a client library has already populated top_p from
# its own defaults. Check per model; AWS names only two here explicitly.
inference_config = {
    "temperature": 0.7,
    "topP": 0.9,      # remove this key entirely, don't set it to "neutral"
    "maxTokens": 1024,
}

Why it hallucinates is not a bug report, it is the training objective

A model is trained to predict the most statistically plausible next token given everything before it. That objective has no separate channel for "is this true", because truth was never the thing being optimised. Fluent and plausible is the target; correct is a frequent side effect of fluent and plausible, not a guarantee of it.

This is why a confident, well-formatted, wrong answer is not a malfunction. The model did exactly what it was trained to do: continue the text in the most plausible way it could find. Instruction-tuning and RLHF push the model towards saying "I don't know" in more situations, but that is a learned behaviour layered on top of the same underlying objective, not a built-in truth detector. Treat "the model sounded confident" as evidence of nothing. It is optimising for fluency, every time, including the times it is wrong.

Why evals, not your own reading of the output

Once you have internalised all of the above, the honest next question is: how do you actually know the output is good? Not by reading a handful of examples and feeling satisfied. I have written at length about the ways that instinct fails: an LLM-as-judge score that is not a fixed property of the thing being graded, a golden dataset that is easier than production traffic, a quality threshold that needs a real sample size to mean anything. That is rung 7 of this roadmap, and it exists because the operator's mental model in this post is necessary and not sufficient. Knowing what a token costs and what a context window bounds tells you how the system behaves. It does not tell you whether the system is right.

What I now check

The token count for a string is not portable between model families; measure it on the model you are actually calling, not a general-purpose estimate.

A context window is one shared budget across system prompt, tools, history and output, not a per-turn allowance on top of them.

Check whether the model you are calling accepts both temperature and top_p before a client library finds out for you in production.

A confident answer is not a correct answer. The training objective never promised the second thing.

Which one of these cost you the most time to learn the hard way?

Find me on LinkedIn or via rajmurugan.com.

The harness is one integer column

Raj Murugan — Mon, 28 Sep 2026 20:03:54 +0000

A background job was re-billing an AI model every sixty seconds, forever. Not a rewrite. Not a redesign. The fix was one INT column, capped at five. And the fix itself shipped with a gap: it forgot to log why a row gave up, so the very rows it saved became impossible to explain. This is what I now do differently, after both halves of that day.

The failure, in one breath

A row that can never reach its terminal state, sitting behind a poller that checks it every tick, is an infinite retry loop wearing a queue's clothes. Nothing was broken in the traditional sense: no exception, no crash, no alarm. A background close job kept picking up the same "in progress" row, calling the model again, failing to close it again, and handing it back to the next poll. Every lap re-billed the model. Nobody was watching for a row that could not finish, because "could not finish" is not an error, it is just still running.

This is a level-triggered trigger against a condition that never clears. An edge-triggered retry fires once on the transition into failure. A level-triggered poller fires every time it observes the bad state, and if nothing ever changes that state, it fires forever. The row was the poison pill. The poll was the mouth that kept swallowing it.

Why the counter belongs in the database, not memory or metrics

The instinctive fix is a counter: retry, but only a few times. Where you put that counter decides whether it works.

Not in Lambda memory. A retry counter held in the function's own memory resets on every cold start, and a poison-pill row cold-starts its handler just as often as anything else. The counter never reaches five, because it is reintroduced to zero constantly.

Not in a CloudWatch metric. Metrics observe. They tell you a number happened. They do not gate the next attempt, and a dashboard with a suspicious spike on it does not stop the next poll from firing. Watching is not the same instrument as stopping.

What actually holds a ceiling under concurrency is a durable, atomic transition in the database itself, and it has to do two jobs, not one: stop the count from being lost, and stop two pollers from both grabbing the same row on the same tick.

-- Claim the row and count the attempt in one statement. Zero rows back means
-- someone else already has it, or it is already at the cap. Either way, skip
-- it this tick.
UPDATE jobs
SET retry_count = retry_count + 1,
    state = 'claimed'
WHERE id = $1
  AND state = 'pending'
  AND retry_count < 5
RETURNING retry_count;

The state = 'pending' clause is what stops two poller instances from both picking up the same row and both calling the model, not the increment on its own: an increment alone only protects the count, it does not claim the row. The retry_count < 5 clause is what makes the cap a fact the database enforces, rather than something application code has to remember to check after reading the count back. (This is the behaviour under the default Read Committed isolation Postgres and Aurora ship with; under a stricter isolation level a second concurrent writer gets a serialization error instead of quietly waiting its turn, and needs its own retry to match.)

If the risky work then fails, a second atomic statement decides what happens to the row next:

UPDATE jobs
SET state = CASE WHEN retry_count >= 5 THEN 'abandoned' ELSE 'pending' END,
    last_close_error = $2
WHERE id = $1
RETURNING state;

Under the cap, the row goes back to pending for the next poll. At the cap, the same statement flips it straight to abandoned, and the poller's own state = 'pending' clause means it will never claim that row again.

The bounded-retry pattern

Three decisions make this actually work, and each one is a place the naive version gets it wrong.

Claim and increment in the same statement, before the risky work, not after. If you increment on success or on a clean failure, a crash mid-attempt, which is exactly what a poison pill causes, never gets counted, and the cap never bites. Count attempts, not successes.

Force a terminal state at the cap, enforced by the database, not read back and checked by application code. At five, the same WHERE retry_count < 5 that gates every claim stops matching, and the follow-up statement flips the row to abandoned. The retry loop has an exit now, one that does not depend on some later code path remembering to check.

Do the arithmetic on the worst case, honestly. Five attempts at the model's per-call cost is a bounded, small number, call it a dollar. Uncapped, the same bug run for a year at one call a minute is the same unbounded shape as any retry loop nobody put a ceiling on: a small per-call cost, multiplied by nothing ever stopping it. The whole value of the cap is that it turns an open-ended bill into a number you can write down in advance, whatever that number turns out to be for your own per-call cost.

The fix needed its own observability

Here is the part I would tell you to do differently if I were doing it again. Version one had the claim, the cap, and the terminal state. It did not have the last_close_error column shown above: it flipped rows to abandoned at five and wrote nothing about why.

So the abandoned rows carried a null reason. They were safe, in the sense that they had stopped costing money. They were also undiagnosable: nobody could look at a batch of abandoned rows and tell you whether they were all the same bug, five different bugs, or a symptom of something upstream getting worse. Was recovering them safe? Nobody could say, because nobody knew what had actually gone wrong on attempt one through five. last_close_error was a follow-up migration, added after that gap got noticed, not part of the design from day one.

A guardrail with no telemetry is a black box you will have to re-debug from scratch the next time it fires, using none of the information the guardrail itself was sitting on the whole time.

The same shape, one layer up

None of this is specific to a background job. Swap poller for agent loop and the mapping is exact: bound the iterations, force a terminal state at the bound, and log why it stopped. I have written before about what happens when an agent loop skips all three: a runaway that ran for days because nothing capped it, and would have been just as undiagnosable as this one if it had been capped without the third piece.

The stopping half is the part everyone remembers to build. The observability half is the part that gets cut when the deadline is close, because a capped, silent failure still looks like the incident is over. It is over in the sense that the bill stopped. It is not over in the sense that you can tell anyone what actually happened, and the next stuck row, or the next runaway agent, gets the exact same treatment: caught, capped, and still a mystery.

What I now do

A retry ceiling lives in the row it protects, incremented atomically, not in a process's memory and not only in a metric.

Claim and count in the same statement, gated by the cap, so the database enforces the ceiling instead of application code checking it after the fact.

A cap without a logged reason is half a guardrail. Write down why it gave up on every attempt, not just the last one.

Do the worst-case arithmetic before you ship the cap, honestly, so you know the number you are actually bounding.

If a bounded retry in your system does not write down why it stopped, what would it cost you to find out, the next time it fires?

This pairs with Every dashboard was green while the agent burned six figures a year, the incident that made me go looking for this pattern everywhere else. Find me on LinkedIn or via rajmurugan.com.

Not a Python tutorial: the patterns that bite in production agents

Raj Murugan — Tue, 22 Sep 2026 11:33:01 +0000

This is not a Python tutorial. If you can already write a decorator, skip ahead. The agent loop does not care that you know the language. It cares whether your AWS SDK call just froze every other request your process was supposed to be handling while it waited on a model.

Rung 1 of the AI Architect Roadmap is Foundations, and it has sat marked as a gap since the page went up: the Python and tooling that matter once code is running an agent loop in production, not what an intro course teaches. This closes it. Five patterns, all of them things I've either hit myself or watched a production Bedrock workload hit, in roughly the order they bite.

boto3 is synchronous, and that is expensive inside an event loop

boto3 has no native async support. bedrock-runtime.converse_stream() returns a plain synchronous EventStream, and the call itself is a blocking network round trip. Call it directly from inside an asyncio agent loop and you have not made an async mistake in the abstract, you have stopped every other coroutine in that process from making progress for the entire duration of the model's response, streaming or not.

The fix is aioboto3, which wraps aiobotocore to give you an async Bedrock client with the same converse_stream() shape, this time awaited. The gotcha that catches people even after they switch: since aioboto3 8.0, .client() is an async context manager, and opening a fresh one inside every request re-authenticates and re-establishes the connection each time, the same tax you were trying to avoid, just hidden one layer down. The client, not just the Session, has to live for the process's lifetime, held open with an AsyncExitStack:

import aioboto3
from contextlib import AsyncExitStack

session = aioboto3.Session()
exit_stack = AsyncExitStack()
client = None  # populated at startup, reused by every request

async def startup():
    global client
    client = await exit_stack.enter_async_context(session.client("bedrock-runtime"))

async def shutdown():
    await exit_stack.aclose()

async def handle_turn(prompt: str) -> str:
    response = await client.converse_stream(
        modelId="eu.anthropic.claude-sonnet-4-6",
        messages=[{"role": "user", "content": [{"text": prompt}]}],
    )
    async for event in response["stream"]:
        ...  # yields control back to the loop between chunks

Call startup() once when the process boots and shutdown() once when it stops (a framework's lifespan hook, if it has one). Every request in between reuses the same client and the same underlying connection pool.

If you cannot touch the call site (a library that only exposes the sync client), the fallback is loop.run_in_executor(None, sync_call) to push the blocking call onto a thread pool instead of the event loop thread. It costs a thread, but it stops the freeze.

Typed tool signatures are not optional decoration

The Bedrock Converse API's toolSpec takes an inputSchema that must be real JSON Schema, with type: "object" at the top level and a required array naming which properties are mandatory. A bare Python type hint on your tool function does not become this automatically if you are calling Bedrock directly with boto3. Nothing in boto3 converts def get_weather(city: str, unit: str = "celsius") into a schema on its own, you build one by hand, via a Pydantic model's model_json_schema(), or by using an agent framework that does the conversion for you (Strands Agents' @tool decorator parses a function's type hints and docstring into the schema automatically, which is exactly why reaching for boto3 directly instead of a framework is a decision worth making on purpose, not the path of least resistance).

Get the required array wrong, drop a type, or leave a field's description empty, and the failure is not a clean exception. The model either omits an argument it needed, or invents a plausible-looking value for a field whose constraints it was never told, and the tool call fails validation two layers downstream from where the real bug is. AWS's own guidance is explicit that the description is what the model uses to decide when a tool applies, not just what it does, so a thin schema does not just risk a malformed call, it risks the model never reaching for the tool at all. A correct schema is not a security boundary either: it constrains shape, not content, so a syntactically valid string can still be a path-traversal or injection payload the schema never sees. Validate and sanitise inside the tool handler regardless of how tight the schema looks.

Streaming to a browser is a second translation, not a continuation

Bedrock's stream is a boto3 EventStream of typed chunk events, not a wire format a browser understands. If you are proxying that stream to a frontend over Server-Sent Events, you are writing a translator, not a pass-through: every chunk has to become its own data: ...\n\n frame, and the frame boundary matters as much as the content.

The failure mode looks identical to a hung connection: your backend is emitting frames correctly, but something between it and the browser is buffering, an ASGI server, a reverse proxy, a gzip middleware that waits for enough bytes before flushing. curl --no-buffer against your own endpoint is the fastest way to tell the difference between "the server is not sending anything yet" and "the server sent it, something in the middle is holding it."

Where the GIL actually still matters

Python 3.13 shipped a free-threaded build experimentally in October 2024. Python 3.14, a year later, promoted it to officially supported under PEP 779. It is still opt-in: the default build keeps the GIL on, you have to explicitly install or build the free-threaded variant, and as of this year package compatibility across the ecosystem sits at roughly half, not all, of what people actually depend on.

For most agent workloads that barely matters, because the loop is I/O-bound: it spends nearly all of its time waiting on a Bedrock response, not burning CPU, and asyncio was already the right tool for that, GIL or no GIL. Where it still bites is CPU-bound work sharing the same process as the loop, local tokenisation, embedding a document, post-processing a response with a regex pass over a few hundred KB of text. That work still serialises behind the GIL on a stock interpreter no matter how many coroutines you write, and I have watched a "slow Bedrock call" get blamed on the model when the actual bottleneck was a synchronous embedding step competing with the agent loop for the same core.

The order these actually bite in

A synchronous boto3 call inside the agent loop, freezing every other request.
A fresh aioboto3 client per call, after the async fix, quietly re-paying the connection cost.
A tool schema that drifted from the Python type hints it was meant to describe.
An assumption that a boto3 stream is already browser-ready SSE.
CPU-bound work blamed on "the model" when it is the GIL serialising a step that shares the process with the loop.

Rung 2, the ML you actually need to operate an LLM rather than train one, is still open. If there is a Python pattern that has bitten you in a production agent that is not on this list, I'd genuinely like to hear which one.

This post is paired with Introducing the AI Architect Roadmap, and Evals for Production AI, the series currently climbing rung 7, continues. Find me on LinkedIn or via rajmurugan.com.

Introducing the AI Architect Roadmap

Raj Murugan — Tue, 22 Sep 2026 11:33:01 +0000

"AI Architect" is one of the vaguest titles in this industry right now. It can mean someone who wired up a chatbot demo in an afternoon, or someone who ships and operates a production Agentic AI system that survives real traffic, a security review, and a finance team asking where the AWS bill went. I got tired of pretending those are the same skill set, so I wrote down what I actually think closes the gap between them, as eight rungs, and put it live at rajmurugan.com/roadmap.

Why a map instead of a course

Most "AI architect learning path" content is a listicle wearing a ladder's clothes: a dozen tutorial links in a row, sorted roughly by vibes. It reads like progress and teaches like a syllabus nobody finishes. I wanted something closer to an audit than a course: eight rungs, foundations through enterprise delivery, each one marked against what I have actually shipped and written about in production, not what a junior engineer might eventually get around to in theory.

The shape of it

Foundations: the Python and tooling that matter once code is running an agent loop in production, not an intro course.
Learn AI & ML: the machine learning an operator needs, not a trainer, tokens, context windows, the knobs that move cost and behaviour.
Master GenAI: prompting, RAG, tools, and the SDKs that wire an agent together on Bedrock.
Design AI Systems: agents that survive contact with production, intent versus state, a harness that bounds a runaway loop.
Build AI Infrastructure: CDK, containers, CI/CD with OIDC, cost controls baked in from day one.
Security & Governance: deterministic authz in code the model never touches, guardrails as a backstop.
Observability & Cost: where the depth is. More on that below.
Enterprise Solutions: production builds a business can actually adopt.

Six of the eight were covered by writing here before today, two of those (5 and 8, with 6 close behind) with enough posts behind them that "covered" undersells it. Two were still marked gap, and I mean that literally, not "coming soon" dressed up as done.

Where the depth actually is

Rung 7, Observability and Cost, is the flagship, and it is not close. It is the rung where I have put the most measured, specific writing: an agent that trended toward a six-figure annual run rate while every dashboard stayed green, prompt caching that ships off by default and, in the workload I measured it against, was worth 55 to 78 percent off the recurring system-prefix bill once switched on, an AgentCore Memory write that returned success and, in my own testing, read back empty for up to thirty seconds on the wrong read path (a rough ballpark from a handful of runs, not a published guarantee). None of that is exotic. It is the ordinary way a probabilistic system hides its failures behind metrics built for a deterministic one, and it is the rung I would tell anyone shipping agentic AI on AWS to take most seriously, because it is the one uptime and error-rate dashboards are structurally blind to.

The two rungs marked gap, on purpose

Foundations and ML fundamentals were the two I had not written yet, and I thought about quietly filling them with generic content just so the page would read as finished. Every other "complete path" I have seen does exactly that: pads the early rungs with tutorial-grade material because it is the easiest to write, and it is exactly the material a working engineer already knows and skips past. I would rather the page tell the truth. A gap marked gap is more useful than a gap dressed up as coverage.

That changed today for one of the two. Not a Python tutorial: the patterns that bite in production agents closes rung 1: the senior lens on the Python patterns that actually bite in a production agent loop, not another "hello world" first chapter.

What's next

Rung 2, the operator's mental model for the machine learning underneath an LLM, is still open. So is finishing Evals for Production AI, the series currently climbing rung 7 (four parts in, no finale yet). Both are queued, not promised for a date, because that is the same honesty this roadmap is trying to hold itself to.

Which rung would you actually want filled next?

You probably don't need to fine-tune

Raj Murugan — Tue, 15 Sep 2026 08:29:05 +0000

A post crossed my feed a fortnight ago: "30 fine-tuning interview questions, with answers." The framing was sharp. "Should we fine-tune?" is a trap question in 2026 interviews, the author argued, because the expected answer is usually "not yet", and how you get there is the whole test. Fair call. Two lines from it are worth keeping: every fine-tune has to beat the best prompt on the base model, not the lazy one, and you fine-tune the interface while you retrieve the content, because knowledge changes daily and behaviour changes quarterly.

I agree with all of it, and none of it told me what happens the day after a team says yes and someone opens the Bedrock console. The questions underneath were model-agnostic: LoRA, QLoRA, DPO, the method zoo. That's the gap I want to close. Not another "should you fine-tune" explainer. What the ladder looks like before you touch weights, what "fine-tune" actually decomposes into once you're inside Bedrock, and what each path costs to run, priced today, not at launch.

The ladder, before you touch weights

Prompt, then RAG, then caching, then fine-tune, in that order, and only past whichever step actually stops solving your problem. Most teams that reach for fine-tuning first are trying to fix one of three things: the model doesn't know something (that's retrieval), the model is slow or expensive per call (that's caching, or a smaller model), or the model's tone drifts (that's a better system prompt, most of the time). None of those need a training job.

The three reasons that genuinely do:

Structured output you can't prompt your way to. A rigid schema, a house format, a tool-call convention the base model won't hold consistently across thousands of calls even with a strong system prompt.
Cost or latency distillation. You've proven a big model does the job well and you want a small, cheap, fast model to do the same job for a fraction of the price.
Narrow domain vocabulary or tone the prompt can't carry. Not "be more formal", but a genuine specialist register: clinical shorthand, legal drafting conventions, a product's own internal taxonomy, the kind of thing a few-shot prompt approximates and a fine-tune nails.

The industry has mostly converged on that ladder. What's missing from every version of it I've read is the other side of the "yes": once you actually commit, the decision forks twice, once into different job types and again into different ways to serve the result, each with its own bill, and almost everything written about it treats "fine-tuning" as one thing.

"Fine-tune" is four different jobs on Bedrock

Bedrock's CreateModelCustomizationJob API takes a customizationType parameter, and the values are not flavours of the same thing:

FINE_TUNING: supervised fine-tuning on your own labelled examples. The classic case: input/output pairs, a training job, a customised model at the end.
CONTINUED_PRE_TRAINING: unlabelled domain data, for building broader domain adaptation rather than a specific task.
DISTILLATION: you don't write training pairs at all. You point Bedrock at a stronger "teacher" model and a weaker "student" model, hand it your prompts or invocation logs, and Bedrock generates the synthetic training set and runs the job for you. General availability landed in May 2025, and AWS's own published figures put distilled models at up to 500% faster and 75% cheaper than the teacher, with under 2% accuracy loss on retrieval-style tasks. If your actual reason to fine-tune is reason 2 above, cost and latency distillation, this is very often the more direct route to it than hand-rolled supervised fine-tuning, because you skip building the labelled dataset entirely.
REINFORCEMENT_FINE_TUNING: the newest of the four, Nova-only today. You supply a reward function, either a Bedrock model acting as judge or your own Lambda grading logic, and the job trains against that signal instead of a fixed answer key. It's the most direct fit for reason 1 above, structured output and tool-calling reliability, because you can grade "did the tool call parse" programmatically rather than needing labelled pairs.

Serving the result forks again

Bedrock's own documentation still says, plainly, "if you customized a model, you must purchase Provisioned Throughput to be able to use it." Read literally, and read against an older doc page in isolation, that sounds like the whole story. It isn't, and the gap between the two is exactly the kind of thing that's easy to get wrong reading one page instead of the console.

Since 16 July 2025, a second path exists: CreateCustomModelDeployment, which AWS explicitly describes as complementing Provisioned Throughput, not replacing it. Deploy a natively customised model this way and you pay the base model's own on-demand token rate, no reservation, no hourly floor. The catch is eligibility, and it's narrow: only Amazon Nova Lite, Nova 2 Lite, Nova Micro, Nova Pro (all us-east-1) and Meta Llama 3.3 70B Instruct (us-west-2), and only if the underlying model was customised on or after 16 July 2025 for Nova, or 15 September 2025 for Llama. Everything else, every Titan customisation, every model customised before those dates, every region outside those two, still has exactly one way to serve it: Provisioned Throughput.

So the real shape is three serving paths, not two:

Provisioned Throughput: universal. Works for any customised model, any region Bedrock customisation supports. Reserved capacity, billed hourly, whether a request arrives or not.
Custom Model Deployment: on-demand, same per-token price as the base model, but only for the specific Nova and Llama models above, customised after their respective GA dates.
Custom Model Import: for models you trained entirely outside Bedrock (SageMaker, your own PEFT or LoRA run) in a supported open-weight architecture, imported and served on demand, billed per Custom Model Unit-minute, scaling to zero when idle. More on this one below.

The Provisioned Throughput numbers, priced today

If your model or region falls outside Custom Model Deployment's narrow eligibility, this table is what you're actually pricing. Pulled from the AWS Price List API for us-east-1, effective 2026-08-01, in US dollars per model unit per hour:

Model	No commitment	1-month commitment	6-month commitment
Titan Text Lite	$7.10	$6.40	$5.10
Titan Text Express	$20.50	$18.40	$14.80
Titan Text Premier (custom)	$32.25	$27.95	$16.38
Nova Micro	$60.50	$55.00	$30.25
Nova Lite	$60.50	$55.00	$30.25
Nova Pro	$60.50	$55.00	$30.25
Nova Canvas	$60.50	$55.00	$30.25

Read that Nova block again. On demand, Nova Micro costs a small fraction of Nova Pro per token, that's the entire point of the model family. Provisioned, they are the same price. If you're on Provisioned Throughput at all, whether because your model doesn't qualify for Custom Model Deployment or because you deliberately want reserved, latency-consistent capacity, the tier you picked to save money on inference is invisible once you reserve capacity for it: you're renting a model unit, not a token budget, and AWS prices the unit the same across the family regardless of which model is sitting on it. This is exactly why Custom Model Deployment matters when it's available: on Nova Lite, Micro or Pro, customised after 16 July 2025, in us-east-1, you skip this table entirely and pay Nova's own on-demand token price instead.

There's a third path that doesn't use CreateModelCustomizationJob at all: Custom Model Import. Train an open-weight model yourself, anywhere (SageMaker, your own PEFT or LoRA run, wherever), then import the resulting weights into Bedrock for a supported architecture (Llama and Mistral families among them). Bedrock serves it on demand: billed in five-minute windows by Custom Model Unit, currently $0.05718 per CMU-minute in us-east-1, auto-scaling to zero when nothing has invoked it for five minutes. The trade-off is cold start: AWS and early adopters report anywhere from several seconds to a couple of minutes on the first request after the model has scaled to zero, depending on model size, so this isn't a free lunch against Provisioned Throughput for a genuinely latency-sensitive workload, it's a different point on the cost-versus-latency curve.

What that actually costs, worked through

Illustrative maths, not a measured production bill, built from the table above and the live Custom Model Import rate:

Steady, 24/7 traffic, one model unit, one month (730 hours):

Titan Text Lite, native fine-tune, no commitment: 730 × $7.10 = $5,183/month. Six-month commitment: 730 × $5.10 = $3,723/month, but that's a six-month lock-in, $22,338 total, whether traffic shows up or not.
Custom Model Import, a small Llama-class model needing 2 CMUs, run flat out: 730 × 60 × 2 × $0.05718 = $5,009/month. Roughly the same as the Titan no-commitment price, with zero commitment, plus whatever cold starts cost you in latency along the way.

Bursty traffic, same imported model, genuinely invoked 4 hours a day:

Custom Model Import: 4 × 30 × 60 × 2 × $0.05718 = $823/month.
Provisioned Throughput can't do this at all. You are renting capacity, not metering usage, so the bill is identical whether the model answers one request that day or ten thousand. Custom Model Deployment, where it's eligible, can: it's metered like the base model, not reserved like Provisioned Throughput.

That's the actual decision hiding inside "should we fine-tune": it is two decisions stacked as one. Do you need a customised model, and can you commit to buying dedicated, always-on capacity for it, or does your model and region qualify for one of the on-demand routes instead. Most of what's written addresses only the first.

How you'd know it worked

Whichever job type and serving path you take, the trigger post's baseline rule is the right one, and I'd go further: don't trust your own eyeball on "it feels better," measure it. A fine-tune, a distillation, or an RFT run has to beat the best prompt on the base model on a real evaluation, the same way, every time you change it. That's not a throwaway line. I spent the last four posts on exactly how that measurement goes wrong: an LLM-as-judge score isn't a fixed property of the output you're grading, it shifts depending on what sits next to it in the batch, and a continuous quality signal doesn't come with a threshold until you derive one from your own traffic. If you're leaning on an LLM judge to decide whether the customised model earned whatever it costs to serve, the same failure modes apply to that decision as to any other regression gate.

Before you commit to serving anything, you should be able to answer three questions with numbers, not confidence: what does the best prompt on the base model score, on the same eval, today; what does the customised model score; and what is the smallest sample size at which that gap is actually distinguishable from noise. If you can't answer the third one, you don't know if the first two differ at all.

If you've actually shipped a fine-tune on Bedrock, which serving path did you end up on, and did Provisioned Throughput's bill, or the narrower eligibility for the on-demand routes, change the decision after the fact rather than before it?

Your quality alert needs 32 samples

Raj Murugan — Tue, 08 Sep 2026 09:40:24 +0000

I finished Part 3 with a signal I could compute on live traffic with no answer key: trigram grounding, the fraction of a summary's word-trigrams that appear in its source. It responds to a real regression. I wrote that it was a lead worth replicating, not a result, and I stand by that.

Then I tried to alert on it, and found the part nobody writes down. A continuous signal does not come with a threshold. You have to derive one, and deriving it tells you something uncomfortable about how much traffic you need before an alert means anything.

For a regression that drops the signal by a third, a single output catches 2% of the time. You need a window of 32 before you can page anyone.

The reflex answer, and why it is wrong here

Ask anyone where to put a threshold on a metric and you will get some version of "a couple of standard deviations below the mean". It is the right instinct and it does not survive contact with this signal.

Here is the control arm from Part 3, 64 summaries generated with the guardrail intact, at the 2000-character tier, one of the two truncated tiers where the signal separates the arms:


mean	0.0830
median	0.0746
standard deviation	0.0594
minimum	0.0000
p05	0.0167

That distribution is right-skewed with a hard floor at zero, and one perfectly healthy output scores exactly zero. Now apply the reflex:

Threshold	Value	Pages on	Catches
mean − 1.0σ	+0.0235	14% of normal outputs	20%
mean − 1.5σ	−0.0062	0%	0%
mean − 2.0σ	−0.0359	0%	0%

At 1.5 sigma the threshold is a negative number, on a scale that cannot go below zero. It will never fire, on anything, ever. An alert that cannot fire looks exactly like an alert that is working.

The 1.0 sigma line is worse in a way that is easier to miss: it pages you on 14 out of every 100 normal outputs to catch 20% of real ones. Healthy outputs are essentially all of your traffic, so that is a pager firing on ordinary work almost every time it fires at all.

Deriving the line properly

The fix is to stop assuming a shape and use the one you measured. Resample the observed control distribution, build the null distribution of the window mean at size n, and put the threshold at whatever quantile matches your false-page budget. Then resample the regression arm and see how often it lands below that line.

Two budgets, because they are the two people actually argue about: one false page a month (roughly 3% of daily checks) and one a quarter (1%).

Window size	Catches, 1 page/month	Catches, 1 page/quarter
1 output	2%	0%
4	11%	3%
8	29%	13%
16	58%	36%
24	79%	58%
32	91%	78%
48	99%	94%
64	100%	99%

On this grid the monthly budget clears 80% at n = 32, and the quarterly one at n = 48. A finer sweep puts the actual monthly crossing nearer n = 24, which sits within simulation noise of the line itself. I would still build at 32: 24 is exactly on 80% and nobody designs an alert to sit on its own threshold.

Read the first row again, because it is the one that changes what you build. A single output, thresholded at a budget you could actually live with, catches one regression in fifty. Not a weak signal. Functionally no signal.

What that means if you are wiring this up

The instinct with a per-output score is to check it per output. Every eval harness I have seen encourages that: score the row, compare to threshold, flag the row. It is the wrong unit for this signal, and the arithmetic says so before you write any code.

Alert on a window, not a row. The unit is an aggregate over n outputs, and n is a number you compute rather than pick.
Work out what n means in your traffic. 32 outputs is a rounding error for a high-volume summariser and half a week for a low-volume internal tool. If it is half a week, you have not built a quality alert, you have built a weekly report, and you should call it that.
Choose the false-page budget before the threshold. The budget is a product decision about how much trust you can spend. The threshold is arithmetic that follows from it. Doing it in the other order is how you end up at 14% of normal outputs.
Calibrate the threshold on several hundred controls, not sixty-four. The threshold is an estimate too, and mine is built from 64 outputs. Simulate the real thing (draw m controls, set the line, apply it to fresh windows) and a nominal 3% budget calibrated on 64 comes back at 6.3% on average and 16.8% at the unlucky end. That is two to five times the pages you signed up for. At m = 512 it settles to 3.4%. Budget the control data before you budget the pages.
The budget is per check, and I have quoted it at one check a day. 3% of daily checks is about one page a month. If you evaluate every window as it closes, multiply by how often that happens: a high-volume summariser running 300 windows a day at 3% is nine pages a day, not one a month. And rolling windows make consecutive checks correlated, which breaks the conversion altogether.
Use quantiles of the observed distribution, not mean and sigma. Bounded, skewed metrics are the normal case in eval work, not the exception, and sigma-based lines on them are how you get a threshold below the floor of the scale.

The threshold does not sit on the model output. It sits on an aggregate of them, and the size of that aggregate is the number this post is about.

The honest caveats

The effect being detected here is large. A 33.5% drop in the signal, from a deliberately deleted guardrail. Smaller regressions need bigger windows, and the table above is the optimistic end.

This is the truncated-input regime, and that is load-bearing. Everything above is the
2000-character tier. At 12000 characters, which is normal full-context operation, the same deleted
guardrail moves the signal only 12.8% instead of 33.5%, and the window arithmetic changes with
it: n = 32 gives 31% power, n = 64 gives 49%, and 80% needs more than 128. So if your inputs
are not already thin, the honest answer is that this signal needs a much larger window than the
number in the title, or a different signal.

One signal, one corpus, one summariser family. Sixteen posts written by one person, graded by one rubric. The n = 32 is mine. The method transfers; the number does not.

This assumes your window is stationary. I resampled a control arm collected in one sitting, so it carries no daily or weekly cycle. Real traffic has both, and a window that spans a Monday and a Sunday has variance my numbers do not include. That is not something a larger n fixes: with a fixed threshold and a drifting baseline the false-page rate drifts too. It needs re-baselining on a trailing control window.

And it is still built on a signal that did not survive correction. Part 3 was clear that trigram grounding is a lead, not a result: raw p = 0.020, Bonferroni 0.120. This post derives a threshold for it anyway, which is the right thing to do with a lead you intend to test, and the wrong thing to treat as settled. If the signal does not replicate in your setup, the threshold arithmetic is still the transferable part.

Every round, every script, and the raw JSON: github.com/rajmurugan01/do-you-trust-it-evals.

Series

This is Part 4 of Evals for Production AI, on how you actually know an AI system is good once it is live. Part 1 asked what it takes to trust an LLM judge. Part 2 pointed that judge at a deploy gate and watched it go blind. Part 3 found the blind spot was in the dataset, and left a label-free signal worth replicating. This one works out where its line goes.

If you are running a quality alert on an LLM output score, I would genuinely like to know what window size you landed on and how you chose it. Find me on LinkedIn or via rajmurugan.com.

Your golden dataset is too easy

Raj Murugan — Mon, 31 Aug 2026 14:41:52 +0000

I spent two posts trying to detect a regression I had planted myself, and failed three separate ways. An LLM judge over a golden dataset: p = 1.000. A judge-free deterministic assertion: p = 1.000. Six label-free signals computed on the same outputs: nothing below p = 0.17.

Three instruments, one answer. At some point the honest move is to stop suspecting the instrument.

The prompt change was always there, and one input tier away it produces an effect the same gate catches easily. On the corpus I had written down it produces none worth measuring. Hold every single thing constant, give the summariser less source to work with, and the gate that read p = 1.000 reads p = 0.0020.

What this is following on from

Part 1 established that the judge in this rig is trustworthy: 16/16 on clean inputs, 5/5 on single-variable corruptions, and a self-consistency check. Part 2 pointed that trusted judge at a regression gate and watched it go blind, then blamed sample size.

The setup has not changed. A Claude Haiku 4.5 summariser writes a two to three sentence summary of each of the 16 published posts on this site, at temperature 0.3, capped at 300 output tokens. A Claude Sonnet 4.5 judge scores each summary against its source for faithfulness at temperature 0, with a strict rubric. The regression is one thing: three guardrail sentences deleted from the summariser's system prompt, exactly what a prompt looks like after somebody tidies it up. Four repeats per arm, so 64 gradings per arm.

Everything below ran against real Bedrock calls in my own account. Rounds 6 and 7 are new here, and all of it is in the repo.

The question production actually asks

A golden dataset gives you the one thing production never does: the right answer, written down in advance. On live traffic you have the input, you have the output, and that is the entire inventory.

So before blaming the dataset I tried the other obvious thing. What can you compute from an (input, output) pair alone, with no labels anywhere?

Signal	What it is
`novel_numbers`	numbers in the output that are not in the input
`grounding`	fraction of the output's content words that appear in the input
`trigram_grounding`	fraction of the output's word-trigrams that appear in the input
`novel_words`	count of output content words absent from the input
`novel_caps`	capitalised entity-shaped tokens in the output, absent from the input
`length`	characters

Round 6 applied all six to the 128 summaries rounds 4 and 5 had already produced. No new Bedrock calls: same outputs, same regression, different question asked of them.

Nothing fired. Every content signal that moved at all pointed the same direction, the regression arm being consistently less grounded than the control, and not one reached significance. The best was trigram grounding at p = 0.168.

The detector that was detecting a markdown heading

Before that null meant anything I had to check the instrument, and it is as well I did.

The first version reported that 96 of 128 summaries contained a fabricated number or entity. Seventy-five percent, against a judge that had just failed exactly one of those same summaries. When your label-free signal and your judge disagree by that margin, the signal is wrong. It was:

86 of the hits were the word Summary. The summariser likes to open with a # Summary markdown heading. My entity regex saw a capitalised token absent from the source and called it a fabricated entity. I had built a hallucination detector that was mostly detecting a heading.
Sentence-initial words. Instead, Rather, Yes, Key.
Plurals. Macs, LLMs, ACLs flagged against sources that say Mac, LLM, ACL.
Roundings. One summary said 99%+ cache hit ratios. The source says 99.8% and 99.9%. My matcher saw 99 absent from the source and called it fabricated. It is a true statement, and conservative.

Fixed: strip markdown before extraction, require an entity to actually look like one rather than merely start a sentence, stem plurals, and allow a number to be grounded if the source states something it is a faithful rounding of. The count went from 96 to 11.

The broken version is committed as unlabelled_signals_v1.py so you can run it and get the 96 yourself. It was originally only a claim in a code comment, which is not good enough for a post whose whole argument is that you must read what your signal flagged.

That is the part I would want someone to take from this post even if they skip the rest. A label-free signal is cheap to compute and cheap to get wrong, and there is no judge behind it to catch you. The 96 would have looked like a crisis on a dashboard.

Moving the input instead

Three instruments, three nulls, one dataset.

The 16 posts are dense, tightly-written technical writing that already contains every number a short summary would want. An anti-hallucination guardrail has nothing to suppress on an input that offers no temptation to invent. Delete it and the output barely moves, because the guardrail was not doing any work in the first place.

That is testable. Hold the summariser, both system prompts, the judge, the rubric, the posts and the repeat count fixed, and move exactly one thing: how much of each source the summariser is given.

12000 characters. Rounds 4 to 6. Not the whole post: ten of the sixteen are longer than that, up to 29,171 characters, so the baseline tier is already a truncation.
2000 characters. Intro plus a section.
600 characters. The opening paragraph. The model is asked to summarise a post it has mostly not been shown, and must either hedge or invent.

The judge sees the same truncated source the summariser saw, so nobody is scored for omitting text they were never given.

Only one of those two paths has a model in it. The label-free instrument is a set intersection over a log line.

The result

512 new Bedrock calls, all of them for the two short tiers; the 12000 row is reused from rounds 4 to 6. Both instruments were built before this round and neither is tuned to it.

Source given	Judge gate, counts	Fisher	Clustered	Label-free trigram
12000 chars	0/64 vs 1/64	1.000	1.000	0.168
2000 chars	3/64 vs 5/64	0.718	0.625	0.020 raw
600 chars	7/64 vs 23/64	0.0015	0.0020	0.019 raw

Two columns for the judge gate because they answer different questions. Fisher exact treats all 64 gradings as independent, which they are not. The clustered column is the exact sign-flip over the 16 posts. Part 2 reported Fisher and only reached for the 16 clusters in its power calculation, which was half the problem. This is the column I am quoting.

The dataset was the problem. The same gate, the same regression, the same judge that read p = 1.000 on full posts reads p = 0.0020 when the source is short. 23 failures out of 64 against a control of 7. It survives Holm correction across the 18 tests in that family at p = 0.033, and it survives dropping the outlier post at p = 0.0039. The prompt change does not manifest on the inputs I had chosen to write down.

That is the finding. It is also the one I was least interested in when I started, which is worth noticing.

The number I wanted to headline, and why I am not

The result I actually wanted was the second column. At 2000 characters the judge over ground truth cannot separate the arms, p = 0.625, and a signal computable on a production log line with no ground truth anywhere separates them at p = 0.020. Same outputs, same n. I had a whole post built around that sentence.

It does not survive its own correction. The family is 18 tests: the five label-free hallucination signals at each of three tiers, plus the three judge gates. Length is in the analysis as a diagnostic, not as a candidate signal, so it sits outside the family. Holm over the 18 leaves the 600-character gate at p = 0.033 and takes trigram grounding at 2000 to p = 0.289. Putting length in as well, at 21 tests, gives 0.037 and 0.328, so nothing here turns on that choice. Bonferroni over just the six signals I searched still gives 0.120. A post that spent Part 2 lecturing about power calculations does not get to correct hard where it kills a result it likes and lightly where it saves one.

Worse, and more instructive: the two signals that do survive Holm are the two I threw away. novel_words at 600 characters comes in at Holm-adjusted p = 0.004, and novel_caps at p = 0.033. Both are confounded, and I will show why below, but the honest summary is that my statistically strongest signals are the ones I have mechanistic reasons to distrust, and my mechanistically cleanest signal does not clear correction.

So the label-free result is a lead worth replicating, not a result. It is directionally right at both short tiers, 14 of 16 posts move the predicted way at 2000 characters, it is not a length artefact, and it is nowhere near significant once you account for how it was found. If you take one number from this post, take the clustered p = 0.0020 from the gate, not p = 0.020 from the signal.

Why trigram grounding, and not the other five

novel_words is confounded by length. It is a raw count, and at 600 characters the unguarded model writes 31% longer summaries than the control, 538 characters against 411, p = 0.0006. More words, more novel words. Normalised by content-word count it becomes grounding, which reads p = 0.060 at 600 and p = 0.115 at 2000. But normalised per 1000 output characters instead, it reads p = 0.010 at 600. Two defensible normalisations, two different answers, which is itself a warning about how much freedom you have when you choose a rate.

novel_caps fires at p = 0.002 and is measuring at least two different things. I read the unguarded arm's full hit list at 600 characters: 40 hits, and every one is an acronym or an inference. AWS twelve times, AI six, LLM/LLMs six, API/APIs six, ARM four. On llm-is-not-a-security-boundary the source says "language model" and the summary says "LLM": abbreviation, not fabrication. On part-4-local-dev-docker the model added "ARM" to a post about the --platform linux/amd64 flag on a Mac, which is correct, useful, and genuinely not in the source. That second kind is interesting rather than wrong, because reaching for outside knowledge is exactly what the deleted guardrail existed to suppress. My favourite is AM, which is the regex catching the clock in "cryptic CloudFormation errors at 2 AM". There is no outright fabrication anywhere in that list, and I am not going to claim one I cannot point at.

novel_numbers never separates the arms. It is exactly zero at 12000 characters. At 2000 it flags four bare digits per arm, p = 1.000, and at 600 it flags eighteen across both arms in the wrong direction, control 11 against regression 7, p = 0.500. The most intuitive hallucination signal, the one everybody reaches for first, has nothing in it at any source length.

trigram_grounding is the cleanest of them. It is the fraction of the summary's own word-trigrams that appear in the source, so it is a rate rather than a count. A rate can still track length, so I checked: at 600 characters the two correlate at r = -0.15 pooled and +0.06 inside the control arm. At 2000 characters it moves from 0.083 to 0.055 and 14 of 16 posts move in the predicted direction, which is p = 0.004 on the sign test alone. Two posts reverse, prompt-caching-bedrock-strands at +0.076 and agentcore-memory-read-after-write at +0.002, and I have no account of the first one.

The likely mechanism: a prompt that forbids adding content also, in practice, pushes the model to reuse the source's wording. The baseline prompt never mentions phrasing, so this is a side effect of the content constraint rather than the thing it asks for. Trigram overlap measures exactly that, and unigram overlap is too coarse to see it.

What I now do

Treat the golden dataset as a hypothesis about where failure lives, and test it. This is the finding that survived. If your eval corpus is the tidy end of your traffic, a real regression can sit at p = 1.000 in CI and p = 0.0020 one input-difficulty tier away. Truncating your own inputs is a crude but cheap way to find out.
Validate a label-free signal before trusting it. Print what it flagged and read the list. Mine would have reported a 75% hallucination rate that was mostly a markdown heading.
Correct for the search. If you try six signals, the one that fires needs the correction, and you have to be willing to publish it after the correction rather than before.
Prefer rates to counts, and then check the other rate. Every count I tried was confounded by output length. Two reasonable normalisations of the same count disagreed at p = 0.060 and p = 0.010.
Label-free to compute is not label-free to calibrate. I only know trigram grounding responds to this regression because I ran a controlled comparison against a control arm, which is exactly what you cannot do on live traffic. Deploying it still needs a baseline and a threshold.

The honest caveats

Truncation is a proxy for "traffic you did not anticipate", not the thing itself. It holds domain, topic, style and vocabulary constant and moves only the density of grounding material, which is the cleanest single variable available without leaving the corpus. It does not simulate a novel domain, an adversarial user, or a shifted register.

The 300-token output cap is doing some of the work at 600 characters, and it is all one post. Four unguarded outputs at that tier ran all the way into the cap, up to 1531 characters, two of them cut off mid-sentence, against a control maximum of 505. All four are the same post, year-10-study-system-production-ai-failure-modes, all four of its repeats, and all four were judged FAIL. So part of that headline 23 is one post's instruction-following blowout rather than hallucination. Dropping that post entirely leaves 15 clusters and the gate still reads p = 0.0039, with trigram grounding at p = 0.036.

At 600 characters the control is failing too, 7 times in 64. The guarded system degrades at that tier as well, which is worth knowing before you adopt truncation as a diagnostic: a tier where your control also falls over tells you less about the regression than one where it holds.

"14 of 16 posts" is a 2000-character fact and does not replicate at 600. At 600 it is 9 posts in the predicted direction, 6 against, 1 exactly level, which is chance. The 600 p-value comes from the size of a few large moves, not from agreement across posts.

At the 12000 tier the judge saw more of the post than the summariser did. summarize() sends body[:12000] and judge() sends source[:14000], so on the eight posts longer than 14000 characters the judge held up to 2000 characters the summariser never got. Round 7's 2000 and 600 tiers pass the already-truncated body to both, so they are clean. The bias runs toward the top row's null, not against it, but the three rows are not protocol-identical and I would rather say so.

The cross-tier comparison spans an inference-profile change. Rounds 4 to 6 ran on us. inference profiles and round 7 on global. ones, same model version strings, different routing, because I switched for the cost saving between sittings. Both arms within any one tier always ran on the same profile, so the control-versus-regression comparisons that carry every finding are unaffected. AWS documents routing, monitoring and price differences between profiles and makes no output-equivalence guarantee in either direction, so I would not lean on the trend down the table as if the tiers were perfectly comparable. Worth flagging if you copy the cost saving: global. routes outside the US geography, which matters in a regulated shop.

One corpus, one summariser family, one judge, one rubric, one regression. Sixteen posts written by one person. The mechanism is general enough to be worth checking in your setup. The numbers are mine.

The 64 are 4 repeats over 16 posts, not 64 independent observations. Every label-free p-value above is clustered: an exact sign-flip test enumerating all 2^16 assignments over the posts. The judge gate is reported both ways in the results table, because Fisher exact on the counts is the number Part 2 used and the clustered one is the number I am standing behind. Clustering is not automatically the more conservative choice: across the 15 label-free tests it is stricter in 8, looser in 4 and identical in 3. It is stricter in every test that came near significance, which is the case that matters.

All seven rounds, every script including the broken one, and the raw JSON are in the repo: github.com/rajmurugan01/do-you-trust-it-evals.

Series

This is Part 3 of Evals for Production AI, a series on how you actually know an AI system is good once it is live, not according to the dashboard. Part 1 asked what it takes to trust an LLM judge. Part 2 pointed that judge at a deploy gate and watched it go blind. This one found the blind spot was in the dataset, and killed my preferred explanation on the way.

If you are running an eval corpus you suspect is the tidy end of your traffic, happy to compare notes. Find me on LinkedIn or via rajmurugan.com.

Next in Evals for Production AI: a continuous signal with no labels behind it does not come with a threshold. Where you set the line, what a false page costs, and how to tell a real drop from a Tuesday.

Your regression gate needs a power calculation

Raj Murugan — Tue, 25 Aug 2026 01:48:13 +0000

I deleted the anti-hallucination guardrail from my summariser's system prompt on purpose, to see whether my eval would notice.

It did not. Then I removed the LLM judge from the eval entirely and replaced it with a deterministic string assertion, on the theory that the judge was the weak link. That did not notice either. Two measurement approaches, one with a model in the loop and one without, both returning the same answer: no detectable difference, Fisher exact p = 1.000.

The interesting part is not that the gate failed. It is that I had a confident, wrong explanation ready to publish, and the second experiment is the only reason I did not.

What this is following on from

Part 1 tested whether an LLM judge can be trusted at all. Five deliberately corrupted summaries, 5 out of 5 caught and correctly named. Four repeat gradings of an unmodified input, identical scores every time. The judge discriminates and it is stable.

A regression gate is a different job. It runs in CI, nobody injected an error, nobody knows what changed, and its whole purpose is to notice that quality dropped before your users do. I assumed that a judge which clears Part 1's three rounds is fit for that job. This post is what happened when I checked.

The regression

Sixteen published posts from this blog, summarised by Claude Haiku 4.5 on Bedrock, graded by Claude Sonnet 4.5 against a strict faithfulness rubric at temperature 0. The judge never changes anywhere in this experiment. It is the instrument.

The break is one variable: three sentences deleted from the summariser's system prompt, the task sentence untouched.

# Baseline
SYSTEM_PROMPT = (
    "You summarize technical blog posts in 2-3 sentences for a reader deciding whether to "
    "click through. Base the summary ONLY on the text provided. Do not add claims, numbers, "
    "product names, or examples that are not explicitly present in the source text. If the "
    "post does not state a specific number or outcome, do not invent one."
)

# Regression: the guardrail is gone, the task is identical
DEGRADED_SYSTEM_PROMPT = (
    "You summarize technical blog posts in 2-3 sentences for a reader deciding whether to "
    "click through."
)

That is what a prompt looks like after somebody tidies it up, and it is the change a reviewer waves through.

Alongside it, two controls. A control arm runs the identical model and the identical prompt, byte for byte, changing nothing. The summariser runs at temperature 0.3, so the baseline is not deterministic and I need to know how much it moves on its own. And a positive control: the same prompt with a much weaker summariser, Amazon Nova Micro, which AWS describes as its "fastest text-only model, optimized for speed and low cost in tasks like summarization, translation, and classification", at roughly a thirtieth of Haiku 4.5's input price. Worth noting AWS names summarisation first, so this is not a model being used outside its stated purpose, it is a model built for speed and cost on short work being asked for faithfulness on long documents. That arm exists to show the gate detects anything at all.

Attempt one: the judge over the golden dataset

One run of each arm looked like a clean gradient.

Arm	Verdict	Mean faithfulness	Items moved
Baseline (Part 1)	16/16 PASS	5.00	reference
Control	16/16 PASS	5.00	0
Prompt regression	15/16 PASS	4.94	1
Model regression	11/16 PASS	4.19	7

Nothing changed, nothing moved. Subtle change, small movement. Blunt change, big movement. The gate works, just coarsely.

I nearly wrote that post. Part 1 had already taught me why not: its most interesting first-run finding did not survive a controlled re-run and was cut before publishing. One item moving, once, out of sixteen is the same shape. So I ran both arms three more times each.

Arm	Failures per run	Total	Mean faithfulness per run
Control	0, 0, 0, 0	0 / 64	5.00, 5.00, 5.00, 4.94
Prompt regression	1, 0, 0, 0	1 / 64	4.94, 5.00, 5.00, 5.00

Fisher exact on 0/64 against 1/64 gives p = 1.000. The gradient did not survive repetition. There was one event.

That one event is real, and worth looking at, because it is the only direct evidence in this whole post that the guardrail does anything. The judge failed a summary containing "caused real production issues". That phrase is not in the source, which says the author "burned an afternoon". The judge's reasoning named it exactly:

The summary accurately captures the main structure and three gotchas, but 'caused real production issues' overstates the source, which mentions 'burned an afternoon' and a separate OIDC gotcha in the blog's own pipeline, not that these three gotchas caused production issues.

A textbook scope-broadening hallucination, correctly caught and correctly explained, of exactly the category the deleted sentences existed to prevent. Once, in 64 gradings.

Attempt two: delete the judge

Here is the explanation I was ready to publish. Faithfulness was pinned at 5.00 on every baseline item, so the metric had no headroom. The verdict is a binary step function over a continuous quality change. The judge cannot see small drops.

It is a tidy story and it is testable, so I tested it. If the judge is the bottleneck, an eval with no judge in it should do better.

So the trap moves to the input, and the assertion becomes deterministic. Each source post already states things at low intensity ("burned an afternoon"). A guarded summariser is told not to escalate; an unguarded one is free to. I defined a vocabulary of escalations ("production outage", "caused an outage", "data loss", "in every case", "guaranteed", "catastrophic", and similar), then filtered it per post to the phrases verified absent from that post's own source text, so a hit can only ever be something the model introduced. Then a substring check. No judge, no rubric, no scores, no ceiling.

One caveat worth stating rather than letting a reader find it: this vocabulary is not independent of the first experiment. It contains "production issues", the phrase the judge flagged in attempt one. I built the second gate partly from what the first one saw, which makes this a weaker replication than two genuinely independent designs would be.

Gate	Control	Prompt regression	Fisher exact
LLM judge over golden dataset	0 / 64	1 / 64	p = 1.000
Deterministic assertion, no judge	1 / 64	1 / 64	p = 1.000

The control tripped on "critical failure". The regression arm tripped on "production incident". One each.

Removing the judge changed nothing. My explanation did not survive it.

What was actually wrong

Removing the judge changed nothing, so "the judge cannot see small drops" does not stand on its own as the explanation. That is weaker than saying the judge is exonerated, and the weaker version is what the data supports: both gates are blunt, and two blunt instruments returning the same null cannot tell you which one is blunt. What the next number shows is that the sample size alone is enough to explain both nulls, whatever the instruments were doing.

Both designs were asked to separate two rates that are, at everything measured here, roughly 1.6% each. So the real question is what size difference this design could have found at all.

With 64 gradings per arm at 80% power, the answer is 16.1%. The rate would have to rise more than tenfold before either gate reliably noticed.

That has to be computed exactly rather than with the usual two-proportion normal approximation, which wants around five expected events per arm. At a 1.6% baseline and n = 64 there is one expected event. Out of range, not borderline. I ran the normal approximation anyway and first published 15.4%, and the giveaway I ignored is that the standard methods disagree with each other by nearly a factor of two at these rates. The number above enumerates every outcome pair under two binomials and applies the same two-sided Fisher test used everywhere else here. It is twelve lines, in the repo.

It is worse than 64, too. Those 64 gradings are 4 repeats over the same 16 posts, so repeating them buys precision on each post's score rather than more posts. Counting each post once, the same calculation gives 42.6%, a 27-fold rise. So the honest answer is a range: between about 16% and about 43%, depending on how correlated repeat gradings of one post are. Evan Miller is explicit in Adding Error Bars to Evals (arXiv 2411.00640) that this sits on a sliding scale, from "perfectly correlated (in which case each cluster acts as a single independent observation)" to perfectly uncorrelated. His second recommendation covers "[w]hen questions are drawn in related groups, computing clustered standard errors" and his third covers reducing variance by resampling answers, which is the one that actually bites here. His fifth is the one I skipped:

Using power analysis to determine whether an eval (or a random subsample) is capable of testing a hypothesis of interest

That is the calculation I did not do. It runs in under a second and it answers the question both experiments spent real money failing to answer. An eval that cannot resolve the effect you care about does not return "no regression". It returns nothing, in a format that looks exactly like "no regression".

What the gate does catch

The positive control lands hard. Swapping to Nova Micro moved seven items and dropped mean faithfulness from 5.00 to 4.19. The distortions are the interesting part, because they are not invented facts. On the prompt-caching post the summary reported "up to 99.9% hit ratios, proving the cost benefits of enabling caching". The 99.9% is real: it is that post's own headline measurement. What the weaker model added was "proving the cost benefits", welding a cache hit ratio onto a billing claim the source measures separately at 55% and 78%. The judge caught the conflation, not the number.

So these gates are not broken. They are coarse, and the boundary is computable in advance rather than discoverable in production. Ten times the hallucination rate: caught. A change you would actually ship on a Tuesday: invisible.

The row that neither passed nor failed

Sixteen items in that arm. Eleven passed, four failed. That leaves one.

One grading came back as well-formed JSON with faithfulness: 4, completeness: 3, a populated hallucinations list and a sentence of correct reasoning. It had no verdict key. The judge answered every part of the question except the one the gate reads. By the rubric (PASS requires faithfulness 4 or higher and an empty hallucination list) a populated list makes it a FAIL, so the arm's true count is five non-passing out of sixteen, which is what the charts show.

It happened once, in the sixteen gradings that arm got, and not in the 128 Haiku gradings. That arm was never re-run, so this is not evidence that it is rare: conditional on the arm where it appeared, one in sixteen is all I can say. Now consider a run of 15 PASS, 0 FAIL, and one of these:

assert passed == total                    # 15 != 16, red. Catches it.
assert failures == 0                      # 0 == 0, green. Misses it.
pass_rate = passed / (passed + failed)    # 15/15 = 100%, green. Misses it.

Only the first survives. The other two treat a malformed response as if the item did not exist. The third is the version people write, because computing a rate from the two buckets you have feels more careful than counting, and it is the one that silently drops the row from the denominator.

Where this sits against what is already written

"Your eval gate might be noisy" is not my discovery. The variance problem is named in places, and Miller's paper is the rigorous treatment of it.

What I could not find is anyone publishing the control arm. The standard practitioner write-up, for instance Traceloop's guide to prompt regression testing with LLM-as-a-judge in CI/CD, recommends running new versions against a curated dataset, scoring with a judge, and failing the build when quality declines. Reasonable on its face, and it says nothing about how far that score moves when nothing has changed, which is the number that decides whether any of it works.

So the delta is not the idea. It is the measurement: a re-run-with-nothing-changed arm, the same regression measured 64 times by two unrelated methods, the raw JSON published, and a power calculation attached to the result rather than left implicit.

The suite I was about to recommend, which would not have worked

I had a fix section written. It said: keep a suite of known-bad inputs, each asserting a specific failure the judge must still catch, and if somebody deletes the guardrail the fabricated-number case stops failing.

Then I read my own code. Part 1's corruption suite hard-codes its five corrupted summaries as string literals and only ever calls the judge. The summariser is never invoked. Deleting the guardrail from SYSTEM_PROMPT cannot change that suite's inputs, its outputs, or its 5 out of 5 result. It is a judge-regression test. It guards the instrument, not the thing being measured, and it has nothing to say about the regression this entire post is about.

I would have shipped that as the takeaway. It was caught by an independent review pass with no knowledge of how the post was built, which is the only reason it is in this section instead of the conclusion.

What I now do

Compute the minimum detectable effect before running the eval, not after. If you cannot state what size regression your gate would catch, you do not have a gate, you have a ritual. Mine could catch a tenfold increase and nothing smaller, and one calculation would have told me that on day one.
Run the control arm. Re-run with nothing changed and measure how far it moves on its own. Without that number, every difference you find is unattributed.
Count clusters, not rows. Four repeats of sixteen items is sixteen independent units. Repeating a small dataset raises confidence in the mean, not the sample size.
Assert on the count, never on a rate derived from buckets. passed == total catches a malformed response. failures == 0 and passed / (passed + failed) both quietly drop it.
Check that your test exercises the thing you changed. A suite can be rigorous, pass cleanly, and be pointed at a different component entirely.

The honest caveats

The load-bearing limit is the one this whole post is about: I cannot tell you how much worse the guardrail deletion makes the summariser. I can show it produced at least one real hallucination, verified absent from its source. I cannot show it happens more often than with the guardrail in place, because neither design has the resolution to distinguish 1.6% from 1.6%. There is a reading of this data where the true post-deletion rate genuinely is about 1.6%, the gates are correctly reporting a tiny effect, and "underpowered" rather than "blind" is the right word. That reading is consistent with every number here, and the practical conclusion does not change: compute the effect you need to detect before you trust the answer.

Sixteen posts, one judge model, one rubric, one summariser family, four repeats per arm. I am not claiming a general result about LLM-as-judge regression detection. The mechanism is general enough to be worth checking in your setup; the numbers are mine.

The positive control is weaker evidence than the other two arms, deliberately. My intended swap was to an older model in the same family, which would have been a clean single variable. Nova Micro crosses model families, so tokenizer, training and instruction-following all move at once. It is there to show the gate detects something, not to attribute what.

Part 1's runs were on 2026-08-18 and these on 2026-08-24: six days apart, same judge model, same rubric, same account, not the same sitting.

The model that went away mid-experiment

The within-family swap was blocked, for a reason worth stating accurately, because my first instinct was to write it up as "the provider retired a model out from under me with no warning" and that is not what happened.

Bedrock's model lifecycle has three states: Active, Legacy, End-of-Life. Claude 3 Haiku moved to Legacy on 10 March 2026 with a published EOL of 10 September 2026, sixteen days after this post goes up. Six months of notice on a public page. No ambush.

What actually bit me is narrower. Access lapses on an inactivity timer: new customers cannot use a Legacy model at all, and existing customers "may lose access" after a period of not calling it. The exact period is worth flagging because the sources disagree. The documentation says access may be lost "after 15 days of inactivity". The error Bedrock returned to me says 30.

ResourceNotFoundException: Access denied. This Model is marked by provider as
Legacy and you have not been actively using the model in the last 30 days.
Please upgrade to an active model on Amazon Bedrock

I am quoting both rather than picking one, because I do not know which governs and I did not test the boundary.

The documented model-access flow is to re-accept the agreement with aws bedrock list-foundation-model-agreement-offers and aws bedrock create-foundation-model-agreement, or through model access in the console. AWS documents those as the model-access steps; it does not connect them to Legacy inactivity specifically, and I did not test whether they recover a lapsed Legacy model. To check where you stand before any of this matters, aws bedrock get-foundation-model-availability --model-id <id> returns an agreementAvailability field. There is also a phase worth knowing about: since 10 June 2026 Claude 3 Haiku has been in public extended access, during which the docs say to expect higher, provider-set pricing. In this case that has not bitten. Bedrock's published extended-access price table currently lists only Claude 3.5 Sonnet and 3.5 Sonnet v2, not Claude 3 Haiku.

So the operational lesson is not that providers retire models without warning. It is that a benchmark baseline you only run occasionally is exactly the workload an inactivity timer catches, that reactivating may cost more than it did, and that after the EOL date it stops being recoverable at all. Pin the recorded outputs rather than assuming you can regenerate them.

All five rounds, every script, and the raw JSON for every run and every arm are in the repo: github.com/rajmurugan01/do-you-trust-it-evals.

Series

This is Part 2 of Evals for Production AI, a series on how you actually know an AI system is good once it's live, not according to the dashboard. Part 1 asked what it takes to trust an LLM judge's score. This one pointed that trusted judge at the job most teams want it for, and found the instrument was not what decided the answer.

If you are wiring an eval into a deploy gate and want a second pair of eyes on where its detection floor actually sits, happy to compare notes. Find me on LinkedIn or via rajmurugan.com.

Next in Evals for Production AI: an offline eval can only ever test the inputs you thought to write down. What live production signals catch that a golden dataset structurally cannot, and how to spot a quality drop in traffic you never anticipated.

What a Year 10 study system taught me about production AI failure modes

Raj Murugan — Tue, 18 Aug 2026 02:26:58 +0000

I spend my professional time shipping production AI agents on AWS for enterprise clients. Bedrock AgentCore, Strands, the integration layer between LLMs and the systems that actually matter. The work that does not show up in demo videos.

A few weeks ago I built an AI system for my Year 10 son. Not on AWS. Anthropic's Claude projects and Cowork routines, because the data lives in Google Workspace and the user is a teenager who needs a frictionless experience. Different stack, same architectural disciplines.

It took three iterations and one architecture pivot inside the third before it worked. What I did not expect was how cleanly the failure modes mapped to patterns I see in production Bedrock work. The five lessons below are the ones I will be taking straight back into client engagements.

The build, briefly

Three subject-specific Claude projects with Socratic tutoring. A weekly Cowork routine that aggregates the week and produces three differentiated emails: a full dossier to me, a focused brief to each tutor, a forward-looking plan for him.

Cowork is Anthropic's scheduled-routine service for Claude. Think of it as cron for Claude projects, with built-in connectors to Gmail, Drive, and Calendar.

The architecture had to do two things: read what happened in the projects during the week, and produce structured output across multiple channels on a schedule. Sounds simple. Most agentic systems do.

Three iterations, one architecture pivot

The version above is v3. There were two earlier shapes I had to rule out before I got there, and inside v3 there was a further pivot at the integration layer. The full diagram is on the canonical post — rajmurugan.com — here's the prose walkthrough.

v1: the answer machine

The first build was the obvious one. Ask Claude a question, get an answer. It worked on day one and failed the same week. He was not learning; he was copying. The pedagogical intent — that the system make him think — was nowhere in the design. I rebuilt.

v2: Socratic tutor, per subject

v2 made the model refuse to give direct answers. One Claude project per subject, system prompts tuned to ask back rather than tell, hint at the next step rather than skip to the end. That worked. He had to actually do the maths. His tutors started commenting that the homework conversations were sharper.

The constraint we hit next was visibility. Three subjects across a week is a lot of context for a parent and three different tutors to absorb. We needed a layer on top that aggregated the week and split it for different audiences. That extension is what became v3.

v3's first architecture failed at the integration layer

My first design for the v3 aggregation layer had the Sunday routine reading project chats directly and updating a rolling Google Doc. First real run: neither side worked. Cowork has no API surface to read project chats on the same account. The Drive connector is read-only for content. Two load-bearing assumptions, both wrong.

I have seen this exact pattern in Bedrock work. A team designs an agent that "queries the knowledge base, then updates the ticket." First deployment surfaces the truth: the knowledge base query returns chunks the agent cannot reason over, or the ticket system's API has a write surface that does not match the read surface. The system worked on paper because nobody tested the primitives in isolation.

The architecture that works flips the data flow

The redesign inverted both directions. Each subject project now drafts a session-summary email at end-of-session with a fixed subject prefix. I review the draft and click Send. The Sunday routine searches Gmail by subject prefix instead of reading project chats. The routine creates a new dated doc each week instead of modifying an existing one.

Same intent. Different primitives. Push beats pull, and append beats mutate.

This is the same shift I keep recommending in Bedrock production work (see Part 3 of the AgentCore series where I walk through the same pattern with Strands tool calls). When an agent needs to "look inside" a system that cannot expose its state cleanly, the answer is almost always to make the producer emit, not to make the consumer introspect. Pull architectures have one failure mode for every integration. Push architectures have one failure mode total: the producer doesn't emit. That is debuggable.

Five lessons that map directly to production AWS AI

1. Verify the primitives before you design around them. v3's first design cost me an evening because I drew a diagram before I tested whether Cowork could read project chats. Most failed Bedrock POCs I see fail for the same reason at a bigger scale: the team designs around assumed Knowledge Base behaviour, assumed AgentCore session state, assumed Strands tool-call semantics, without the kind of isolated primitive testing I'd insist on at the start of any CDK deployment. Save the diagram for after the smoke test.

2. When introspection isn't possible, push beats pull. This is the single most useful pattern I have learned this year. Half the production Bedrock work I do involves rearranging data flow from pull to push because the producer can be modified and the consumer cannot. If your agent needs to read state from a system that doesn't expose it cleanly, the right move is almost never a better retrieval strategy. It's an emit.

3. Human-in-the-loop is a feature, not a bug. v3 has me clicking Send on two emails per cycle. Ten seconds of human attention, full audit trail, kill switch on every outbound message. I argue for this on every enterprise engagement and lose half the time, because someone wants the autonomy metric. The teams that ship the human-in-the-loop version tend to still be running their agents six months later. The teams that skip it learn the value of an audit trail the hard way.

4. Calibration windows beat go-live confidence. Three Sunday runs on real data before flipping to multi-recipient. Same discipline I apply to Bedrock agents before they touch a customer-facing channel. Skipping calibration is the single biggest predictor of an enterprise AI rollback I have seen.

5. The "things we got wrong" doc is more useful than the spec. I now write a decision log on every client engagement, with the versions-that-didn't-work explicitly preserved. Six months later, when someone joins the project and asks "why is it built this way," that doc is the answer. The polished spec is for review committees. The decision log is for the team.

There is a sixth lesson — about the pedagogical move from v1 to v2, and what it taught me about scoping AI agents around intent rather than output. That one belongs in a separate post on when to graduate from Claude projects to AgentCore. It's in draft.

Why I'm posting this on an AWS-focused blog

Because the lessons are stack-agnostic. AgentCore, Strands, Bedrock Knowledge Bases, Lambda-backed tools, plain old Claude projects with Cowork: the architectural disciplines are the same. The model layer is interchangeable. The integration layer is where production AI lives or dies. That's true on AWS, that's true on Anthropic's stack, and that's true on whatever comes next.

Side builds like this are how I sharpen patterns I then apply at scale to enterprise AWS engagements. Enterprise engagements take longer to write up because they have to be anonymised. Side builds let me publish the pattern faster.

The repo

Full design history, the v1 and v2 specs, the v3 architecture that failed, the v3 architecture that works, the prompts, the build plan, and the calibration log are open-source: github.com/rajmurugan01/study-loop

If you ship production AI on AWS and any of these patterns ring true, I would like to compare notes. Find me on LinkedIn, or comment below.

More posts on production AWS AI: browse the blog or subscribe by RSS. The next post in this thread, on when to graduate from Claude projects to AgentCore, is in draft.

Field Notes: Turning prompt caching on for a production Bedrock workload

Raj Murugan — Tue, 18 Aug 2026 02:26:54 +0000

Two kwargs in Strands' BedrockModel cut a Bedrock workload's system-prefix billing by 78%. The Strands tutorial doesn't mention them. Almost none of the production Strands code I've audited this year has them set.

This is Part 2 of AI Operations Services — the deep dive on prompt caching that Part 1 promised. Short, specific, entirely evidence-led: two kwargs to enable, one per-model gotcha that took half a day to find, one measurement technique that gives you the answer in seconds without waiting for CloudWatch to aggregate.

The default is None

from strands.models.bedrock import BedrockModel

BedrockModel(
    model_id="eu.amazon.nova-pro-v1:0",
)

That is the example shape you find in most Strands tutorials and the agentic-AI content on Bedrock. It is also the shape that produces a Bedrock call with no cachePoint block, no system-prefix caching, no tool-registry caching, and a full-input bill on every turn.

BedrockModel exposes two kwargs that turn caching on:

BedrockModel(
    model_id="eu.amazon.nova-pro-v1:0",
    cache_prompt="default",   # caches the system prompt
    cache_tools="default",    # caches the tool registry
)

Both default to None. The Strands docs mention the kwargs in the API reference but not in the getting-started flow, so they are easy to miss on first build and easy to forget on the second. Every workload I have walked into this year had them unset. The fix is two kwargs and a boto3 version bump if your environment is more than a quarter behind.

What the two kwargs actually do

Under the hood, Bedrock's converse and converseStream accept a cachePoint block at specific positions in the request body. The block tells Bedrock "cache the prefix up to this marker, and on a subsequent call with the same prefix, bill it as a cache read instead of a full input."

cache_prompt="default" inserts a cachePoint at the end of the system block, so the entire system prompt becomes a cacheable prefix. cache_tools="default" inserts a cachePoint inside toolConfig.tools, so the tool registry becomes the next cacheable prefix after the system block. Both points compose: a call with both set caches system + tools together, which is the right thing to want when both are large and stable.

The TTL is five minutes from the most recent cache write or read on a given prefix. Inside the TTL, subsequent calls bill the prefix tokens at the cache-read rate. Outside the TTL, the next call pays a fresh cache write and the meter restarts.

The per-model gotcha

This is the thing that cost me half a day, because it is not in the Bedrock docs and the error message points at the request shape rather than the underlying constraint:

Malformed input request: extraneous key [cachePoint] is not permitted

The error appeared on every Nova call after I set both cache_prompt and cache_tools, but only when both were set. Sonnet 4.6 took the same config without complaint. The difference is per-model: Bedrock's server-side validator gates cachePoint placement per model family, not per feature.

What works on each:

Model family	`cachePoint` in `system`	`cachePoint` in `toolConfig.tools`
Amazon Nova (Pro, Lite, Micro)	accepted	rejected server-side
Anthropic (Sonnet, Haiku, Opus)	accepted	accepted

The pattern that survives a fallback chain with both families in it is one config-of-config: pass cache_prompt="default" to every model in the chain, and pass cache_tools="default" only to the Anthropic-family models. If you do not split it, the Nova path fails on every call with the malformed-input error and the SDK retry loop swallows the failures into the fallback chain. From the dashboard, the symptom is "Nova is throttling at 100%" with no further clue. From the agent logs, the symptom is the actual error string above.

from strands.models.bedrock import BedrockModel

def make_model(model_id: str) -> BedrockModel:
    is_anthropic = model_id.startswith("anthropic.") or "claude" in model_id.lower()
    return BedrockModel(
        model_id=model_id,
        cache_prompt="default",
        cache_tools="default" if is_anthropic else None,
    )

That is the smallest piece of code that handles both families correctly. Half a day saved.

How to verify, in seconds

The default reflex for verifying a Bedrock change is to wait for CloudWatch metrics to aggregate, then look at cacheReadInputTokenCount per ModelId over a 15-minute window. That works, but it is the slow path. The fast path is the per-call usage block returned inline by bedrock-runtime.converse(...):

import boto3, json

client = boto3.client("bedrock-runtime")
response = client.converse(
    modelId="eu.amazon.nova-pro-v1:0",
    system=[{"text": SYSTEM_PROMPT}, {"cachePoint": {"type": "default"}}],
    messages=[{"role": "user", "content": [{"text": "ping"}]}],
)

print(json.dumps(response["usage"], indent=2))

response["usage"] carries four keys you want: inputTokens, outputTokens, cacheReadInputTokens, cacheWriteInputTokens. On call 1 against a freshly-seeded prefix, cacheWriteInputTokens is large and cacheReadInputTokens is zero. On call 2 against the same prefix inside the TTL, cacheWriteInputTokens drops to zero and cacheReadInputTokens is large.

If you instead see cacheWriteInputTokens: 0 and cacheReadInputTokens: 0 on every call, your config did not take effect: the cachePoint block is missing from the request, or the SDK version is too old to emit it, or the prefix is too short to be cacheable (Bedrock has a per-model minimum).

The per-call usage block is the right measurement primitive because it is exact, immediate, per-turn, and free. No CloudWatch lag, no metric aggregation, no dashboard to build. Three calls and you know.

The five-second propagation lag

A subtlety the docs do not flag: Bedrock takes a few seconds to make a freshly-written cache entry available for reads. Fire two calls inside a second against the same prefix and the second one will pay a full cache write rather than a cheap read. The lag I measured on Nova and Sonnet in eu-central-1 was around five seconds; six seconds between calls is enough to clear it.

This matters for two reasons. First, when you measure caching with a tight loop, you will conclude caching does not work, because turn 2 of your driver will still be a write. Use spaced calls or accept that your measurement run wastes the first call or two on writes. Second, in production, the lag means a burst of three calls in the first second of a user turn pays one write plus two writes, not one write plus two reads, on a fresh prefix. After the first burst, every subsequent call inside the TTL is a read.

The measured results

Methodology: real production system prompt (8,156 tokens on the Nova tokeniser and 8,788 on the Anthropic tokeniser, varying by tokeniser not by content), 10-turn driver against the workload's staging Bedrock account, 6-second intra-call spacing, both Nova Pro and Sonnet 4.6 in the same run. cache_prompt="default" set on every model, cache_tools="default" set on Sonnet only.

Model	Turns	Hit ratio	System-prefix billing reduction
Amazon Nova Pro	10	99.9%	55%
Anthropic Sonnet 4.6	10	99.8%	78%

The asymmetry is pricing-driven. Anthropic publishes its cache-read multiplier directly: 10% of input price per the Anthropic pricing page (retrieved 2026-06-26). At 99.8% hit ratio on the system-prefix tokens, that is 78% off the full-input bill. Nova's cache-read multiplier is on the AWS Bedrock pricing page; plug your contracted rate against the measured 99.9% hit ratio to compute your own reduction.

Two caveats. The hit ratios are measured on a 10-turn driver in a single run, not aggregated across days of production traffic. In production, calls drift in and out of the TTL window depending on user-burst patterns, so the steady-state hit ratio will be lower than 99.9%. The number to track post-deploy is the per-day ratio of cacheReadInputTokens to cacheReadInputTokens + cacheWriteInputTokens + non-cached inputTokens per ModelId. Second: prompt caching only helps the prefix tokens, not the per-turn user message or output.

What I now do on every Strands `BedrockModel`

Three things, in this order:

Set cache_prompt="default" on every model in the fallback chain.
Set cache_tools="default" on the Anthropic-family models only.
After deploy, fire three calls against the prefix with five to six seconds between them and print response["usage"]. Confirm call 1 has a non-zero cacheWriteInputTokens and calls 2 and 3 have a non-zero cacheReadInputTokens.

Then add CacheReadInputTokenCount per ModelId to the workload's CloudWatch dashboard for the steady-state ratio. The dashboard is not how you verify the deploy, it is how you spot regressions: a hit ratio that drifts down over time is usually a sign that the system prompt is being mutated per request and the cache is being invalidated on every call.

Series

Part 1: Three things I learned diagnosing a production Bedrock workload — load tests can lie, latency isn't always model speed, prompt caching is almost never on.

Part 2: this post, the deep dive on the caching kwargs and the per-model gotcha.

Part 3 (coming): attributing a mixed change — the latency improvement on the engagement above shipped a model and region swap together; how much was each.

If you are running Strands with both Nova and Anthropic in your fallback chain, have you hit the Nova toolConfig.tools rejection? Curious whether anyone has solved it differently than splitting cache_tools per model family. Drop a note in the comments or DM me on LinkedIn.