DEV Community: Raj Murugan

The harness is one integer column

Raj Murugan — Mon, 28 Sep 2026 20:03:54 +0000

A background job was re-billing an AI model every sixty seconds, forever. Not a rewrite. Not a redesign. The fix was one INT column, capped at five. And the fix itself shipped with a gap: it forgot to log why a row gave up, so the very rows it saved became impossible to explain. This is what I now do differently, after both halves of that day.

The failure, in one breath

A row that can never reach its terminal state, sitting behind a poller that checks it every tick, is an infinite retry loop wearing a queue's clothes. Nothing was broken in the traditional sense: no exception, no crash, no alarm. A background close job kept picking up the same "in progress" row, calling the model again, failing to close it again, and handing it back to the next poll. Every lap re-billed the model. Nobody was watching for a row that could not finish, because "could not finish" is not an error, it is just still running.

This is a level-triggered trigger against a condition that never clears. An edge-triggered retry fires once on the transition into failure. A level-triggered poller fires every time it observes the bad state, and if nothing ever changes that state, it fires forever. The row was the poison pill. The poll was the mouth that kept swallowing it.

Why the counter belongs in the database, not memory or metrics

The instinctive fix is a counter: retry, but only a few times. Where you put that counter decides whether it works.

Not in Lambda memory. A retry counter held in the function's own memory resets on every cold start, and a poison-pill row cold-starts its handler just as often as anything else. The counter never reaches five, because it is reintroduced to zero constantly.

Not in a CloudWatch metric. Metrics observe. They tell you a number happened. They do not gate the next attempt, and a dashboard with a suspicious spike on it does not stop the next poll from firing. Watching is not the same instrument as stopping.

What actually holds a ceiling under concurrency is a durable, atomic transition in the database itself, and it has to do two jobs, not one: stop the count from being lost, and stop two pollers from both grabbing the same row on the same tick.

-- Claim the row and count the attempt in one statement. Zero rows back means
-- someone else already has it, or it is already at the cap. Either way, skip
-- it this tick.
UPDATE jobs
SET retry_count = retry_count + 1,
    state = 'claimed'
WHERE id = $1
  AND state = 'pending'
  AND retry_count < 5
RETURNING retry_count;

The state = 'pending' clause is what stops two poller instances from both picking up the same row and both calling the model, not the increment on its own: an increment alone only protects the count, it does not claim the row. The retry_count < 5 clause is what makes the cap a fact the database enforces, rather than something application code has to remember to check after reading the count back. (This is the behaviour under the default Read Committed isolation Postgres and Aurora ship with; under a stricter isolation level a second concurrent writer gets a serialization error instead of quietly waiting its turn, and needs its own retry to match.)

If the risky work then fails, a second atomic statement decides what happens to the row next:

UPDATE jobs
SET state = CASE WHEN retry_count >= 5 THEN 'abandoned' ELSE 'pending' END,
    last_close_error = $2
WHERE id = $1
RETURNING state;

Under the cap, the row goes back to pending for the next poll. At the cap, the same statement flips it straight to abandoned, and the poller's own state = 'pending' clause means it will never claim that row again.

The bounded-retry pattern

Three decisions make this actually work, and each one is a place the naive version gets it wrong.

Claim and increment in the same statement, before the risky work, not after. If you increment on success or on a clean failure, a crash mid-attempt, which is exactly what a poison pill causes, never gets counted, and the cap never bites. Count attempts, not successes.

Force a terminal state at the cap, enforced by the database, not read back and checked by application code. At five, the same WHERE retry_count < 5 that gates every claim stops matching, and the follow-up statement flips the row to abandoned. The retry loop has an exit now, one that does not depend on some later code path remembering to check.

Do the arithmetic on the worst case, honestly. Five attempts at the model's per-call cost is a bounded, small number, call it a dollar. Uncapped, the same bug run for a year at one call a minute is the same unbounded shape as any retry loop nobody put a ceiling on: a small per-call cost, multiplied by nothing ever stopping it. The whole value of the cap is that it turns an open-ended bill into a number you can write down in advance, whatever that number turns out to be for your own per-call cost.

The fix needed its own observability

Here is the part I would tell you to do differently if I were doing it again. Version one had the claim, the cap, and the terminal state. It did not have the last_close_error column shown above: it flipped rows to abandoned at five and wrote nothing about why.

So the abandoned rows carried a null reason. They were safe, in the sense that they had stopped costing money. They were also undiagnosable: nobody could look at a batch of abandoned rows and tell you whether they were all the same bug, five different bugs, or a symptom of something upstream getting worse. Was recovering them safe? Nobody could say, because nobody knew what had actually gone wrong on attempt one through five. last_close_error was a follow-up migration, added after that gap got noticed, not part of the design from day one.

A guardrail with no telemetry is a black box you will have to re-debug from scratch the next time it fires, using none of the information the guardrail itself was sitting on the whole time.

The same shape, one layer up

None of this is specific to a background job. Swap poller for agent loop and the mapping is exact: bound the iterations, force a terminal state at the bound, and log why it stopped. I have written before about what happens when an agent loop skips all three: a runaway that ran for days because nothing capped it, and would have been just as undiagnosable as this one if it had been capped without the third piece.

The stopping half is the part everyone remembers to build. The observability half is the part that gets cut when the deadline is close, because a capped, silent failure still looks like the incident is over. It is over in the sense that the bill stopped. It is not over in the sense that you can tell anyone what actually happened, and the next stuck row, or the next runaway agent, gets the exact same treatment: caught, capped, and still a mystery.

What I now do

A retry ceiling lives in the row it protects, incremented atomically, not in a process's memory and not only in a metric.

Claim and count in the same statement, gated by the cap, so the database enforces the ceiling instead of application code checking it after the fact.

A cap without a logged reason is half a guardrail. Write down why it gave up on every attempt, not just the last one.

Do the worst-case arithmetic before you ship the cap, honestly, so you know the number you are actually bounding.

If a bounded retry in your system does not write down why it stopped, what would it cost you to find out, the next time it fires?

This pairs with Every dashboard was green while the agent burned six figures a year, the incident that made me go looking for this pattern everywhere else. Find me on LinkedIn or via rajmurugan.com.

Not a Python tutorial: the patterns that bite in production agents

Raj Murugan — Tue, 22 Sep 2026 11:33:01 +0000

This is not a Python tutorial. If you can already write a decorator, skip ahead. The agent loop does not care that you know the language. It cares whether your AWS SDK call just froze every other request your process was supposed to be handling while it waited on a model.

Rung 1 of the AI Architect Roadmap is Foundations, and it has sat marked as a gap since the page went up: the Python and tooling that matter once code is running an agent loop in production, not what an intro course teaches. This closes it. Five patterns, all of them things I've either hit myself or watched a production Bedrock workload hit, in roughly the order they bite.

boto3 is synchronous, and that is expensive inside an event loop

boto3 has no native async support. bedrock-runtime.converse_stream() returns a plain synchronous EventStream, and the call itself is a blocking network round trip. Call it directly from inside an asyncio agent loop and you have not made an async mistake in the abstract, you have stopped every other coroutine in that process from making progress for the entire duration of the model's response, streaming or not.

The fix is aioboto3, which wraps aiobotocore to give you an async Bedrock client with the same converse_stream() shape, this time awaited. The gotcha that catches people even after they switch: since aioboto3 8.0, .client() is an async context manager, and opening a fresh one inside every request re-authenticates and re-establishes the connection each time, the same tax you were trying to avoid, just hidden one layer down. The client, not just the Session, has to live for the process's lifetime, held open with an AsyncExitStack:

import aioboto3
from contextlib import AsyncExitStack

session = aioboto3.Session()
exit_stack = AsyncExitStack()
client = None  # populated at startup, reused by every request

async def startup():
    global client
    client = await exit_stack.enter_async_context(session.client("bedrock-runtime"))

async def shutdown():
    await exit_stack.aclose()

async def handle_turn(prompt: str) -> str:
    response = await client.converse_stream(
        modelId="eu.anthropic.claude-sonnet-4-6",
        messages=[{"role": "user", "content": [{"text": prompt}]}],
    )
    async for event in response["stream"]:
        ...  # yields control back to the loop between chunks

Call startup() once when the process boots and shutdown() once when it stops (a framework's lifespan hook, if it has one). Every request in between reuses the same client and the same underlying connection pool.

If you cannot touch the call site (a library that only exposes the sync client), the fallback is loop.run_in_executor(None, sync_call) to push the blocking call onto a thread pool instead of the event loop thread. It costs a thread, but it stops the freeze.

Typed tool signatures are not optional decoration

The Bedrock Converse API's toolSpec takes an inputSchema that must be real JSON Schema, with type: "object" at the top level and a required array naming which properties are mandatory. A bare Python type hint on your tool function does not become this automatically if you are calling Bedrock directly with boto3. Nothing in boto3 converts def get_weather(city: str, unit: str = "celsius") into a schema on its own, you build one by hand, via a Pydantic model's model_json_schema(), or by using an agent framework that does the conversion for you (Strands Agents' @tool decorator parses a function's type hints and docstring into the schema automatically, which is exactly why reaching for boto3 directly instead of a framework is a decision worth making on purpose, not the path of least resistance).

Get the required array wrong, drop a type, or leave a field's description empty, and the failure is not a clean exception. The model either omits an argument it needed, or invents a plausible-looking value for a field whose constraints it was never told, and the tool call fails validation two layers downstream from where the real bug is. AWS's own guidance is explicit that the description is what the model uses to decide when a tool applies, not just what it does, so a thin schema does not just risk a malformed call, it risks the model never reaching for the tool at all. A correct schema is not a security boundary either: it constrains shape, not content, so a syntactically valid string can still be a path-traversal or injection payload the schema never sees. Validate and sanitise inside the tool handler regardless of how tight the schema looks.

Streaming to a browser is a second translation, not a continuation

Bedrock's stream is a boto3 EventStream of typed chunk events, not a wire format a browser understands. If you are proxying that stream to a frontend over Server-Sent Events, you are writing a translator, not a pass-through: every chunk has to become its own data: ...\n\n frame, and the frame boundary matters as much as the content.

The failure mode looks identical to a hung connection: your backend is emitting frames correctly, but something between it and the browser is buffering, an ASGI server, a reverse proxy, a gzip middleware that waits for enough bytes before flushing. curl --no-buffer against your own endpoint is the fastest way to tell the difference between "the server is not sending anything yet" and "the server sent it, something in the middle is holding it."

Where the GIL actually still matters

Python 3.13 shipped a free-threaded build experimentally in October 2024. Python 3.14, a year later, promoted it to officially supported under PEP 779. It is still opt-in: the default build keeps the GIL on, you have to explicitly install or build the free-threaded variant, and as of this year package compatibility across the ecosystem sits at roughly half, not all, of what people actually depend on.

For most agent workloads that barely matters, because the loop is I/O-bound: it spends nearly all of its time waiting on a Bedrock response, not burning CPU, and asyncio was already the right tool for that, GIL or no GIL. Where it still bites is CPU-bound work sharing the same process as the loop, local tokenisation, embedding a document, post-processing a response with a regex pass over a few hundred KB of text. That work still serialises behind the GIL on a stock interpreter no matter how many coroutines you write, and I have watched a "slow Bedrock call" get blamed on the model when the actual bottleneck was a synchronous embedding step competing with the agent loop for the same core.

The order these actually bite in

A synchronous boto3 call inside the agent loop, freezing every other request.
A fresh aioboto3 client per call, after the async fix, quietly re-paying the connection cost.
A tool schema that drifted from the Python type hints it was meant to describe.
An assumption that a boto3 stream is already browser-ready SSE.
CPU-bound work blamed on "the model" when it is the GIL serialising a step that shares the process with the loop.

Rung 2, the ML you actually need to operate an LLM rather than train one, is still open. If there is a Python pattern that has bitten you in a production agent that is not on this list, I'd genuinely like to hear which one.

This post is paired with Introducing the AI Architect Roadmap, and Evals for Production AI, the series currently climbing rung 7, continues. Find me on LinkedIn or via rajmurugan.com.

Introducing the AI Architect Roadmap

Raj Murugan — Tue, 22 Sep 2026 11:33:01 +0000

"AI Architect" is one of the vaguest titles in this industry right now. It can mean someone who wired up a chatbot demo in an afternoon, or someone who ships and operates a production Agentic AI system that survives real traffic, a security review, and a finance team asking where the AWS bill went. I got tired of pretending those are the same skill set, so I wrote down what I actually think closes the gap between them, as eight rungs, and put it live at rajmurugan.com/roadmap.

Why a map instead of a course

Most "AI architect learning path" content is a listicle wearing a ladder's clothes: a dozen tutorial links in a row, sorted roughly by vibes. It reads like progress and teaches like a syllabus nobody finishes. I wanted something closer to an audit than a course: eight rungs, foundations through enterprise delivery, each one marked against what I have actually shipped and written about in production, not what a junior engineer might eventually get around to in theory.

The shape of it

Foundations: the Python and tooling that matter once code is running an agent loop in production, not an intro course.
Learn AI & ML: the machine learning an operator needs, not a trainer, tokens, context windows, the knobs that move cost and behaviour.
Master GenAI: prompting, RAG, tools, and the SDKs that wire an agent together on Bedrock.
Design AI Systems: agents that survive contact with production, intent versus state, a harness that bounds a runaway loop.
Build AI Infrastructure: CDK, containers, CI/CD with OIDC, cost controls baked in from day one.
Security & Governance: deterministic authz in code the model never touches, guardrails as a backstop.
Observability & Cost: where the depth is. More on that below.
Enterprise Solutions: production builds a business can actually adopt.

Six of the eight were covered by writing here before today, two of those (5 and 8, with 6 close behind) with enough posts behind them that "covered" undersells it. Two were still marked gap, and I mean that literally, not "coming soon" dressed up as done.

Where the depth actually is

Rung 7, Observability and Cost, is the flagship, and it is not close. It is the rung where I have put the most measured, specific writing: an agent that trended toward a six-figure annual run rate while every dashboard stayed green, prompt caching that ships off by default and, in the workload I measured it against, was worth 55 to 78 percent off the recurring system-prefix bill once switched on, an AgentCore Memory write that returned success and, in my own testing, read back empty for up to thirty seconds on the wrong read path (a rough ballpark from a handful of runs, not a published guarantee). None of that is exotic. It is the ordinary way a probabilistic system hides its failures behind metrics built for a deterministic one, and it is the rung I would tell anyone shipping agentic AI on AWS to take most seriously, because it is the one uptime and error-rate dashboards are structurally blind to.

The two rungs marked gap, on purpose

Foundations and ML fundamentals were the two I had not written yet, and I thought about quietly filling them with generic content just so the page would read as finished. Every other "complete path" I have seen does exactly that: pads the early rungs with tutorial-grade material because it is the easiest to write, and it is exactly the material a working engineer already knows and skips past. I would rather the page tell the truth. A gap marked gap is more useful than a gap dressed up as coverage.

That changed today for one of the two. Not a Python tutorial: the patterns that bite in production agents closes rung 1: the senior lens on the Python patterns that actually bite in a production agent loop, not another "hello world" first chapter.

What's next

Rung 2, the operator's mental model for the machine learning underneath an LLM, is still open. So is finishing Evals for Production AI, the series currently climbing rung 7 (four parts in, no finale yet). Both are queued, not promised for a date, because that is the same honesty this roadmap is trying to hold itself to.

Which rung would you actually want filled next?

You probably don't need to fine-tune

Raj Murugan — Tue, 15 Sep 2026 08:29:05 +0000

A post crossed my feed a fortnight ago: "30 fine-tuning interview questions, with answers." The framing was sharp. "Should we fine-tune?" is a trap question in 2026 interviews, the author argued, because the expected answer is usually "not yet", and how you get there is the whole test. Fair call. Two lines from it are worth keeping: every fine-tune has to beat the best prompt on the base model, not the lazy one, and you fine-tune the interface while you retrieve the content, because knowledge changes daily and behaviour changes quarterly.

I agree with all of it, and none of it told me what happens the day after a team says yes and someone opens the Bedrock console. The questions underneath were model-agnostic: LoRA, QLoRA, DPO, the method zoo. That's the gap I want to close. Not another "should you fine-tune" explainer. What the ladder looks like before you touch weights, what "fine-tune" actually decomposes into once you're inside Bedrock, and what each path costs to run, priced today, not at launch.

The ladder, before you touch weights

Prompt, then RAG, then caching, then fine-tune, in that order, and only past whichever step actually stops solving your problem. Most teams that reach for fine-tuning first are trying to fix one of three things: the model doesn't know something (that's retrieval), the model is slow or expensive per call (that's caching, or a smaller model), or the model's tone drifts (that's a better system prompt, most of the time). None of those need a training job.

The three reasons that genuinely do:

Structured output you can't prompt your way to. A rigid schema, a house format, a tool-call convention the base model won't hold consistently across thousands of calls even with a strong system prompt.
Cost or latency distillation. You've proven a big model does the job well and you want a small, cheap, fast model to do the same job for a fraction of the price.
Narrow domain vocabulary or tone the prompt can't carry. Not "be more formal", but a genuine specialist register: clinical shorthand, legal drafting conventions, a product's own internal taxonomy, the kind of thing a few-shot prompt approximates and a fine-tune nails.

The industry has mostly converged on that ladder. What's missing from every version of it I've read is the other side of the "yes": once you actually commit, the decision forks twice, once into different job types and again into different ways to serve the result, each with its own bill, and almost everything written about it treats "fine-tuning" as one thing.

"Fine-tune" is four different jobs on Bedrock

Bedrock's CreateModelCustomizationJob API takes a customizationType parameter, and the values are not flavours of the same thing:

FINE_TUNING: supervised fine-tuning on your own labelled examples. The classic case: input/output pairs, a training job, a customised model at the end.
CONTINUED_PRE_TRAINING: unlabelled domain data, for building broader domain adaptation rather than a specific task.
DISTILLATION: you don't write training pairs at all. You point Bedrock at a stronger "teacher" model and a weaker "student" model, hand it your prompts or invocation logs, and Bedrock generates the synthetic training set and runs the job for you. General availability landed in May 2025, and AWS's own published figures put distilled models at up to 500% faster and 75% cheaper than the teacher, with under 2% accuracy loss on retrieval-style tasks. If your actual reason to fine-tune is reason 2 above, cost and latency distillation, this is very often the more direct route to it than hand-rolled supervised fine-tuning, because you skip building the labelled dataset entirely.
REINFORCEMENT_FINE_TUNING: the newest of the four, Nova-only today. You supply a reward function, either a Bedrock model acting as judge or your own Lambda grading logic, and the job trains against that signal instead of a fixed answer key. It's the most direct fit for reason 1 above, structured output and tool-calling reliability, because you can grade "did the tool call parse" programmatically rather than needing labelled pairs.

Serving the result forks again

Bedrock's own documentation still says, plainly, "if you customized a model, you must purchase Provisioned Throughput to be able to use it." Read literally, and read against an older doc page in isolation, that sounds like the whole story. It isn't, and the gap between the two is exactly the kind of thing that's easy to get wrong reading one page instead of the console.

Since 16 July 2025, a second path exists: CreateCustomModelDeployment, which AWS explicitly describes as complementing Provisioned Throughput, not replacing it. Deploy a natively customised model this way and you pay the base model's own on-demand token rate, no reservation, no hourly floor. The catch is eligibility, and it's narrow: only Amazon Nova Lite, Nova 2 Lite, Nova Micro, Nova Pro (all us-east-1) and Meta Llama 3.3 70B Instruct (us-west-2), and only if the underlying model was customised on or after 16 July 2025 for Nova, or 15 September 2025 for Llama. Everything else, every Titan customisation, every model customised before those dates, every region outside those two, still has exactly one way to serve it: Provisioned Throughput.

So the real shape is three serving paths, not two:

Provisioned Throughput: universal. Works for any customised model, any region Bedrock customisation supports. Reserved capacity, billed hourly, whether a request arrives or not.
Custom Model Deployment: on-demand, same per-token price as the base model, but only for the specific Nova and Llama models above, customised after their respective GA dates.
Custom Model Import: for models you trained entirely outside Bedrock (SageMaker, your own PEFT or LoRA run) in a supported open-weight architecture, imported and served on demand, billed per Custom Model Unit-minute, scaling to zero when idle. More on this one below.

The Provisioned Throughput numbers, priced today

If your model or region falls outside Custom Model Deployment's narrow eligibility, this table is what you're actually pricing. Pulled from the AWS Price List API for us-east-1, effective 2026-08-01, in US dollars per model unit per hour:

Model	No commitment	1-month commitment	6-month commitment
Titan Text Lite	$7.10	$6.40	$5.10
Titan Text Express	$20.50	$18.40	$14.80
Titan Text Premier (custom)	$32.25	$27.95	$16.38
Nova Micro	$60.50	$55.00	$30.25
Nova Lite	$60.50	$55.00	$30.25
Nova Pro	$60.50	$55.00	$30.25
Nova Canvas	$60.50	$55.00	$30.25

Read that Nova block again. On demand, Nova Micro costs a small fraction of Nova Pro per token, that's the entire point of the model family. Provisioned, they are the same price. If you're on Provisioned Throughput at all, whether because your model doesn't qualify for Custom Model Deployment or because you deliberately want reserved, latency-consistent capacity, the tier you picked to save money on inference is invisible once you reserve capacity for it: you're renting a model unit, not a token budget, and AWS prices the unit the same across the family regardless of which model is sitting on it. This is exactly why Custom Model Deployment matters when it's available: on Nova Lite, Micro or Pro, customised after 16 July 2025, in us-east-1, you skip this table entirely and pay Nova's own on-demand token price instead.

There's a third path that doesn't use CreateModelCustomizationJob at all: Custom Model Import. Train an open-weight model yourself, anywhere (SageMaker, your own PEFT or LoRA run, wherever), then import the resulting weights into Bedrock for a supported architecture (Llama and Mistral families among them). Bedrock serves it on demand: billed in five-minute windows by Custom Model Unit, currently $0.05718 per CMU-minute in us-east-1, auto-scaling to zero when nothing has invoked it for five minutes. The trade-off is cold start: AWS and early adopters report anywhere from several seconds to a couple of minutes on the first request after the model has scaled to zero, depending on model size, so this isn't a free lunch against Provisioned Throughput for a genuinely latency-sensitive workload, it's a different point on the cost-versus-latency curve.

What that actually costs, worked through

Illustrative maths, not a measured production bill, built from the table above and the live Custom Model Import rate:

Steady, 24/7 traffic, one model unit, one month (730 hours):

Titan Text Lite, native fine-tune, no commitment: 730 × $7.10 = $5,183/month. Six-month commitment: 730 × $5.10 = $3,723/month, but that's a six-month lock-in, $22,338 total, whether traffic shows up or not.
Custom Model Import, a small Llama-class model needing 2 CMUs, run flat out: 730 × 60 × 2 × $0.05718 = $5,009/month. Roughly the same as the Titan no-commitment price, with zero commitment, plus whatever cold starts cost you in latency along the way.

Bursty traffic, same imported model, genuinely invoked 4 hours a day:

Custom Model Import: 4 × 30 × 60 × 2 × $0.05718 = $823/month.
Provisioned Throughput can't do this at all. You are renting capacity, not metering usage, so the bill is identical whether the model answers one request that day or ten thousand. Custom Model Deployment, where it's eligible, can: it's metered like the base model, not reserved like Provisioned Throughput.

That's the actual decision hiding inside "should we fine-tune": it is two decisions stacked as one. Do you need a customised model, and can you commit to buying dedicated, always-on capacity for it, or does your model and region qualify for one of the on-demand routes instead. Most of what's written addresses only the first.

How you'd know it worked

Whichever job type and serving path you take, the trigger post's baseline rule is the right one, and I'd go further: don't trust your own eyeball on "it feels better," measure it. A fine-tune, a distillation, or an RFT run has to beat the best prompt on the base model on a real evaluation, the same way, every time you change it. That's not a throwaway line. I spent the last four posts on exactly how that measurement goes wrong: an LLM-as-judge score isn't a fixed property of the output you're grading, it shifts depending on what sits next to it in the batch, and a continuous quality signal doesn't come with a threshold until you derive one from your own traffic. If you're leaning on an LLM judge to decide whether the customised model earned whatever it costs to serve, the same failure modes apply to that decision as to any other regression gate.

Before you commit to serving anything, you should be able to answer three questions with numbers, not confidence: what does the best prompt on the base model score, on the same eval, today; what does the customised model score; and what is the smallest sample size at which that gap is actually distinguishable from noise. If you can't answer the third one, you don't know if the first two differ at all.

If you've actually shipped a fine-tune on Bedrock, which serving path did you end up on, and did Provisioned Throughput's bill, or the narrower eligibility for the on-demand routes, change the decision after the fact rather than before it?

Your quality alert needs 32 samples

Raj Murugan — Tue, 08 Sep 2026 09:40:24 +0000

I finished Part 3 with a signal I could compute on live traffic with no answer key: trigram grounding, the fraction of a summary's word-trigrams that appear in its source. It responds to a real regression. I wrote that it was a lead worth replicating, not a result, and I stand by that.

Then I tried to alert on it, and found the part nobody writes down. A continuous signal does not come with a threshold. You have to derive one, and deriving it tells you something uncomfortable about how much traffic you need before an alert means anything.

For a regression that drops the signal by a third, a single output catches 2% of the time. You need a window of 32 before you can page anyone.

The reflex answer, and why it is wrong here

Ask anyone where to put a threshold on a metric and you will get some version of "a couple of standard deviations below the mean". It is the right instinct and it does not survive contact with this signal.

Here is the control arm from Part 3, 64 summaries generated with the guardrail intact, at the 2000-character tier, one of the two truncated tiers where the signal separates the arms:


mean	0.0830
median	0.0746
standard deviation	0.0594
minimum	0.0000
p05	0.0167

That distribution is right-skewed with a hard floor at zero, and one perfectly healthy output scores exactly zero. Now apply the reflex:

Threshold	Value	Pages on	Catches
mean − 1.0σ	+0.0235	14% of normal outputs	20%
mean − 1.5σ	−0.0062	0%	0%
mean − 2.0σ	−0.0359	0%	0%

At 1.5 sigma the threshold is a negative number, on a scale that cannot go below zero. It will never fire, on anything, ever. An alert that cannot fire looks exactly like an alert that is working.

The 1.0 sigma line is worse in a way that is easier to miss: it pages you on 14 out of every 100 normal outputs to catch 20% of real ones. Healthy outputs are essentially all of your traffic, so that is a pager firing on ordinary work almost every time it fires at all.

Deriving the line properly

The fix is to stop assuming a shape and use the one you measured. Resample the observed control distribution, build the null distribution of the window mean at size n, and put the threshold at whatever quantile matches your false-page budget. Then resample the regression arm and see how often it lands below that line.

Two budgets, because they are the two people actually argue about: one false page a month (roughly 3% of daily checks) and one a quarter (1%).

Window size	Catches, 1 page/month	Catches, 1 page/quarter
1 output	2%	0%
4	11%	3%
8	29%	13%
16	58%	36%
24	79%	58%
32	91%	78%
48	99%	94%
64	100%	99%

On this grid the monthly budget clears 80% at n = 32, and the quarterly one at n = 48. A finer sweep puts the actual monthly crossing nearer n = 24, which sits within simulation noise of the line itself. I would still build at 32: 24 is exactly on 80% and nobody designs an alert to sit on its own threshold.

Read the first row again, because it is the one that changes what you build. A single output, thresholded at a budget you could actually live with, catches one regression in fifty. Not a weak signal. Functionally no signal.

What that means if you are wiring this up

The instinct with a per-output score is to check it per output. Every eval harness I have seen encourages that: score the row, compare to threshold, flag the row. It is the wrong unit for this signal, and the arithmetic says so before you write any code.

Alert on a window, not a row. The unit is an aggregate over n outputs, and n is a number you compute rather than pick.
Work out what n means in your traffic. 32 outputs is a rounding error for a high-volume summariser and half a week for a low-volume internal tool. If it is half a week, you have not built a quality alert, you have built a weekly report, and you should call it that.
Choose the false-page budget before the threshold. The budget is a product decision about how much trust you can spend. The threshold is arithmetic that follows from it. Doing it in the other order is how you end up at 14% of normal outputs.
Calibrate the threshold on several hundred controls, not sixty-four. The threshold is an estimate too, and mine is built from 64 outputs. Simulate the real thing (draw m controls, set the line, apply it to fresh windows) and a nominal 3% budget calibrated on 64 comes back at 6.3% on average and 16.8% at the unlucky end. That is two to five times the pages you signed up for. At m = 512 it settles to 3.4%. Budget the control data before you budget the pages.
The budget is per check, and I have quoted it at one check a day. 3% of daily checks is about one page a month. If you evaluate every window as it closes, multiply by how often that happens: a high-volume summariser running 300 windows a day at 3% is nine pages a day, not one a month. And rolling windows make consecutive checks correlated, which breaks the conversion altogether.
Use quantiles of the observed distribution, not mean and sigma. Bounded, skewed metrics are the normal case in eval work, not the exception, and sigma-based lines on them are how you get a threshold below the floor of the scale.

The threshold does not sit on the model output. It sits on an aggregate of them, and the size of that aggregate is the number this post is about.

The honest caveats

The effect being detected here is large. A 33.5% drop in the signal, from a deliberately deleted guardrail. Smaller regressions need bigger windows, and the table above is the optimistic end.

This is the truncated-input regime, and that is load-bearing. Everything above is the
2000-character tier. At 12000 characters, which is normal full-context operation, the same deleted
guardrail moves the signal only 12.8% instead of 33.5%, and the window arithmetic changes with
it: n = 32 gives 31% power, n = 64 gives 49%, and 80% needs more than 128. So if your inputs
are not already thin, the honest answer is that this signal needs a much larger window than the
number in the title, or a different signal.

One signal, one corpus, one summariser family. Sixteen posts written by one person, graded by one rubric. The n = 32 is mine. The method transfers; the number does not.

This assumes your window is stationary. I resampled a control arm collected in one sitting, so it carries no daily or weekly cycle. Real traffic has both, and a window that spans a Monday and a Sunday has variance my numbers do not include. That is not something a larger n fixes: with a fixed threshold and a drifting baseline the false-page rate drifts too. It needs re-baselining on a trailing control window.

And it is still built on a signal that did not survive correction. Part 3 was clear that trigram grounding is a lead, not a result: raw p = 0.020, Bonferroni 0.120. This post derives a threshold for it anyway, which is the right thing to do with a lead you intend to test, and the wrong thing to treat as settled. If the signal does not replicate in your setup, the threshold arithmetic is still the transferable part.

Every round, every script, and the raw JSON: github.com/rajmurugan01/do-you-trust-it-evals.

Series

This is Part 4 of Evals for Production AI, on how you actually know an AI system is good once it is live. Part 1 asked what it takes to trust an LLM judge. Part 2 pointed that judge at a deploy gate and watched it go blind. Part 3 found the blind spot was in the dataset, and left a label-free signal worth replicating. This one works out where its line goes.

If you are running a quality alert on an LLM output score, I would genuinely like to know what window size you landed on and how you chose it. Find me on LinkedIn or via rajmurugan.com.

Your golden dataset is too easy

Raj Murugan — Mon, 31 Aug 2026 14:41:52 +0000

I spent two posts trying to detect a regression I had planted myself, and failed three separate ways. An LLM judge over a golden dataset: p = 1.000. A judge-free deterministic assertion: p = 1.000. Six label-free signals computed on the same outputs: nothing below p = 0.17.

Three instruments, one answer. At some point the honest move is to stop suspecting the instrument.

The prompt change was always there, and one input tier away it produces an effect the same gate catches easily. On the corpus I had written down it produces none worth measuring. Hold every single thing constant, give the summariser less source to work with, and the gate that read p = 1.000 reads p = 0.0020.

What this is following on from

Part 1 established that the judge in this rig is trustworthy: 16/16 on clean inputs, 5/5 on single-variable corruptions, and a self-consistency check. Part 2 pointed that trusted judge at a regression gate and watched it go blind, then blamed sample size.

The setup has not changed. A Claude Haiku 4.5 summariser writes a two to three sentence summary of each of the 16 published posts on this site, at temperature 0.3, capped at 300 output tokens. A Claude Sonnet 4.5 judge scores each summary against its source for faithfulness at temperature 0, with a strict rubric. The regression is one thing: three guardrail sentences deleted from the summariser's system prompt, exactly what a prompt looks like after somebody tidies it up. Four repeats per arm, so 64 gradings per arm.

Everything below ran against real Bedrock calls in my own account. Rounds 6 and 7 are new here, and all of it is in the repo.

The question production actually asks

A golden dataset gives you the one thing production never does: the right answer, written down in advance. On live traffic you have the input, you have the output, and that is the entire inventory.

So before blaming the dataset I tried the other obvious thing. What can you compute from an (input, output) pair alone, with no labels anywhere?

Signal	What it is
`novel_numbers`	numbers in the output that are not in the input
`grounding`	fraction of the output's content words that appear in the input
`trigram_grounding`	fraction of the output's word-trigrams that appear in the input
`novel_words`	count of output content words absent from the input
`novel_caps`	capitalised entity-shaped tokens in the output, absent from the input
`length`	characters

Round 6 applied all six to the 128 summaries rounds 4 and 5 had already produced. No new Bedrock calls: same outputs, same regression, different question asked of them.

Nothing fired. Every content signal that moved at all pointed the same direction, the regression arm being consistently less grounded than the control, and not one reached significance. The best was trigram grounding at p = 0.168.

The detector that was detecting a markdown heading

Before that null meant anything I had to check the instrument, and it is as well I did.

The first version reported that 96 of 128 summaries contained a fabricated number or entity. Seventy-five percent, against a judge that had just failed exactly one of those same summaries. When your label-free signal and your judge disagree by that margin, the signal is wrong. It was:

86 of the hits were the word Summary. The summariser likes to open with a # Summary markdown heading. My entity regex saw a capitalised token absent from the source and called it a fabricated entity. I had built a hallucination detector that was mostly detecting a heading.
Sentence-initial words. Instead, Rather, Yes, Key.
Plurals. Macs, LLMs, ACLs flagged against sources that say Mac, LLM, ACL.
Roundings. One summary said 99%+ cache hit ratios. The source says 99.8% and 99.9%. My matcher saw 99 absent from the source and called it fabricated. It is a true statement, and conservative.

Fixed: strip markdown before extraction, require an entity to actually look like one rather than merely start a sentence, stem plurals, and allow a number to be grounded if the source states something it is a faithful rounding of. The count went from 96 to 11.

The broken version is committed as unlabelled_signals_v1.py so you can run it and get the 96 yourself. It was originally only a claim in a code comment, which is not good enough for a post whose whole argument is that you must read what your signal flagged.

That is the part I would want someone to take from this post even if they skip the rest. A label-free signal is cheap to compute and cheap to get wrong, and there is no judge behind it to catch you. The 96 would have looked like a crisis on a dashboard.

Moving the input instead

Three instruments, three nulls, one dataset.

The 16 posts are dense, tightly-written technical writing that already contains every number a short summary would want. An anti-hallucination guardrail has nothing to suppress on an input that offers no temptation to invent. Delete it and the output barely moves, because the guardrail was not doing any work in the first place.

That is testable. Hold the summariser, both system prompts, the judge, the rubric, the posts and the repeat count fixed, and move exactly one thing: how much of each source the summariser is given.

12000 characters. Rounds 4 to 6. Not the whole post: ten of the sixteen are longer than that, up to 29,171 characters, so the baseline tier is already a truncation.
2000 characters. Intro plus a section.
600 characters. The opening paragraph. The model is asked to summarise a post it has mostly not been shown, and must either hedge or invent.

The judge sees the same truncated source the summariser saw, so nobody is scored for omitting text they were never given.

Only one of those two paths has a model in it. The label-free instrument is a set intersection over a log line.

The result

512 new Bedrock calls, all of them for the two short tiers; the 12000 row is reused from rounds 4 to 6. Both instruments were built before this round and neither is tuned to it.

Source given	Judge gate, counts	Fisher	Clustered	Label-free trigram
12000 chars	0/64 vs 1/64	1.000	1.000	0.168
2000 chars	3/64 vs 5/64	0.718	0.625	0.020 raw
600 chars	7/64 vs 23/64	0.0015	0.0020	0.019 raw

Two columns for the judge gate because they answer different questions. Fisher exact treats all 64 gradings as independent, which they are not. The clustered column is the exact sign-flip over the 16 posts. Part 2 reported Fisher and only reached for the 16 clusters in its power calculation, which was half the problem. This is the column I am quoting.

The dataset was the problem. The same gate, the same regression, the same judge that read p = 1.000 on full posts reads p = 0.0020 when the source is short. 23 failures out of 64 against a control of 7. It survives Holm correction across the 18 tests in that family at p = 0.033, and it survives dropping the outlier post at p = 0.0039. The prompt change does not manifest on the inputs I had chosen to write down.

That is the finding. It is also the one I was least interested in when I started, which is worth noticing.

The number I wanted to headline, and why I am not

The result I actually wanted was the second column. At 2000 characters the judge over ground truth cannot separate the arms, p = 0.625, and a signal computable on a production log line with no ground truth anywhere separates them at p = 0.020. Same outputs, same n. I had a whole post built around that sentence.

It does not survive its own correction. The family is 18 tests: the five label-free hallucination signals at each of three tiers, plus the three judge gates. Length is in the analysis as a diagnostic, not as a candidate signal, so it sits outside the family. Holm over the 18 leaves the 600-character gate at p = 0.033 and takes trigram grounding at 2000 to p = 0.289. Putting length in as well, at 21 tests, gives 0.037 and 0.328, so nothing here turns on that choice. Bonferroni over just the six signals I searched still gives 0.120. A post that spent Part 2 lecturing about power calculations does not get to correct hard where it kills a result it likes and lightly where it saves one.

Worse, and more instructive: the two signals that do survive Holm are the two I threw away. novel_words at 600 characters comes in at Holm-adjusted p = 0.004, and novel_caps at p = 0.033. Both are confounded, and I will show why below, but the honest summary is that my statistically strongest signals are the ones I have mechanistic reasons to distrust, and my mechanistically cleanest signal does not clear correction.

So the label-free result is a lead worth replicating, not a result. It is directionally right at both short tiers, 14 of 16 posts move the predicted way at 2000 characters, it is not a length artefact, and it is nowhere near significant once you account for how it was found. If you take one number from this post, take the clustered p = 0.0020 from the gate, not p = 0.020 from the signal.

Why trigram grounding, and not the other five

novel_words is confounded by length. It is a raw count, and at 600 characters the unguarded model writes 31% longer summaries than the control, 538 characters against 411, p = 0.0006. More words, more novel words. Normalised by content-word count it becomes grounding, which reads p = 0.060 at 600 and p = 0.115 at 2000. But normalised per 1000 output characters instead, it reads p = 0.010 at 600. Two defensible normalisations, two different answers, which is itself a warning about how much freedom you have when you choose a rate.

novel_caps fires at p = 0.002 and is measuring at least two different things. I read the unguarded arm's full hit list at 600 characters: 40 hits, and every one is an acronym or an inference. AWS twelve times, AI six, LLM/LLMs six, API/APIs six, ARM four. On llm-is-not-a-security-boundary the source says "language model" and the summary says "LLM": abbreviation, not fabrication. On part-4-local-dev-docker the model added "ARM" to a post about the --platform linux/amd64 flag on a Mac, which is correct, useful, and genuinely not in the source. That second kind is interesting rather than wrong, because reaching for outside knowledge is exactly what the deleted guardrail existed to suppress. My favourite is AM, which is the regex catching the clock in "cryptic CloudFormation errors at 2 AM". There is no outright fabrication anywhere in that list, and I am not going to claim one I cannot point at.

novel_numbers never separates the arms. It is exactly zero at 12000 characters. At 2000 it flags four bare digits per arm, p = 1.000, and at 600 it flags eighteen across both arms in the wrong direction, control 11 against regression 7, p = 0.500. The most intuitive hallucination signal, the one everybody reaches for first, has nothing in it at any source length.

trigram_grounding is the cleanest of them. It is the fraction of the summary's own word-trigrams that appear in the source, so it is a rate rather than a count. A rate can still track length, so I checked: at 600 characters the two correlate at r = -0.15 pooled and +0.06 inside the control arm. At 2000 characters it moves from 0.083 to 0.055 and 14 of 16 posts move in the predicted direction, which is p = 0.004 on the sign test alone. Two posts reverse, prompt-caching-bedrock-strands at +0.076 and agentcore-memory-read-after-write at +0.002, and I have no account of the first one.

The likely mechanism: a prompt that forbids adding content also, in practice, pushes the model to reuse the source's wording. The baseline prompt never mentions phrasing, so this is a side effect of the content constraint rather than the thing it asks for. Trigram overlap measures exactly that, and unigram overlap is too coarse to see it.

What I now do

Treat the golden dataset as a hypothesis about where failure lives, and test it. This is the finding that survived. If your eval corpus is the tidy end of your traffic, a real regression can sit at p = 1.000 in CI and p = 0.0020 one input-difficulty tier away. Truncating your own inputs is a crude but cheap way to find out.
Validate a label-free signal before trusting it. Print what it flagged and read the list. Mine would have reported a 75% hallucination rate that was mostly a markdown heading.
Correct for the search. If you try six signals, the one that fires needs the correction, and you have to be willing to publish it after the correction rather than before.
Prefer rates to counts, and then check the other rate. Every count I tried was confounded by output length. Two reasonable normalisations of the same count disagreed at p = 0.060 and p = 0.010.
Label-free to compute is not label-free to calibrate. I only know trigram grounding responds to this regression because I ran a controlled comparison against a control arm, which is exactly what you cannot do on live traffic. Deploying it still needs a baseline and a threshold.

The honest caveats

Truncation is a proxy for "traffic you did not anticipate", not the thing itself. It holds domain, topic, style and vocabulary constant and moves only the density of grounding material, which is the cleanest single variable available without leaving the corpus. It does not simulate a novel domain, an adversarial user, or a shifted register.

The 300-token output cap is doing some of the work at 600 characters, and it is all one post. Four unguarded outputs at that tier ran all the way into the cap, up to 1531 characters, two of them cut off mid-sentence, against a control maximum of 505. All four are the same post, year-10-study-system-production-ai-failure-modes, all four of its repeats, and all four were judged FAIL. So part of that headline 23 is one post's instruction-following blowout rather than hallucination. Dropping that post entirely leaves 15 clusters and the gate still reads p = 0.0039, with trigram grounding at p = 0.036.

At 600 characters the control is failing too, 7 times in 64. The guarded system degrades at that tier as well, which is worth knowing before you adopt truncation as a diagnostic: a tier where your control also falls over tells you less about the regression than one where it holds.

"14 of 16 posts" is a 2000-character fact and does not replicate at 600. At 600 it is 9 posts in the predicted direction, 6 against, 1 exactly level, which is chance. The 600 p-value comes from the size of a few large moves, not from agreement across posts.

At the 12000 tier the judge saw more of the post than the summariser did. summarize() sends body[:12000] and judge() sends source[:14000], so on the eight posts longer than 14000 characters the judge held up to 2000 characters the summariser never got. Round 7's 2000 and 600 tiers pass the already-truncated body to both, so they are clean. The bias runs toward the top row's null, not against it, but the three rows are not protocol-identical and I would rather say so.

The cross-tier comparison spans an inference-profile change. Rounds 4 to 6 ran on us. inference profiles and round 7 on global. ones, same model version strings, different routing, because I switched for the cost saving between sittings. Both arms within any one tier always ran on the same profile, so the control-versus-regression comparisons that carry every finding are unaffected. AWS documents routing, monitoring and price differences between profiles and makes no output-equivalence guarantee in either direction, so I would not lean on the trend down the table as if the tiers were perfectly comparable. Worth flagging if you copy the cost saving: global. routes outside the US geography, which matters in a regulated shop.

One corpus, one summariser family, one judge, one rubric, one regression. Sixteen posts written by one person. The mechanism is general enough to be worth checking in your setup. The numbers are mine.

The 64 are 4 repeats over 16 posts, not 64 independent observations. Every label-free p-value above is clustered: an exact sign-flip test enumerating all 2^16 assignments over the posts. The judge gate is reported both ways in the results table, because Fisher exact on the counts is the number Part 2 used and the clustered one is the number I am standing behind. Clustering is not automatically the more conservative choice: across the 15 label-free tests it is stricter in 8, looser in 4 and identical in 3. It is stricter in every test that came near significance, which is the case that matters.

All seven rounds, every script including the broken one, and the raw JSON are in the repo: github.com/rajmurugan01/do-you-trust-it-evals.

Series

This is Part 3 of Evals for Production AI, a series on how you actually know an AI system is good once it is live, not according to the dashboard. Part 1 asked what it takes to trust an LLM judge. Part 2 pointed that judge at a deploy gate and watched it go blind. This one found the blind spot was in the dataset, and killed my preferred explanation on the way.

If you are running an eval corpus you suspect is the tidy end of your traffic, happy to compare notes. Find me on LinkedIn or via rajmurugan.com.

Next in Evals for Production AI: a continuous signal with no labels behind it does not come with a threshold. Where you set the line, what a false page costs, and how to tell a real drop from a Tuesday.

Your regression gate needs a power calculation

Raj Murugan — Tue, 25 Aug 2026 01:48:13 +0000

I deleted the anti-hallucination guardrail from my summariser's system prompt on purpose, to see whether my eval would notice.

It did not. Then I removed the LLM judge from the eval entirely and replaced it with a deterministic string assertion, on the theory that the judge was the weak link. That did not notice either. Two measurement approaches, one with a model in the loop and one without, both returning the same answer: no detectable difference, Fisher exact p = 1.000.

The interesting part is not that the gate failed. It is that I had a confident, wrong explanation ready to publish, and the second experiment is the only reason I did not.

What this is following on from

Part 1 tested whether an LLM judge can be trusted at all. Five deliberately corrupted summaries, 5 out of 5 caught and correctly named. Four repeat gradings of an unmodified input, identical scores every time. The judge discriminates and it is stable.

A regression gate is a different job. It runs in CI, nobody injected an error, nobody knows what changed, and its whole purpose is to notice that quality dropped before your users do. I assumed that a judge which clears Part 1's three rounds is fit for that job. This post is what happened when I checked.

The regression

Sixteen published posts from this blog, summarised by Claude Haiku 4.5 on Bedrock, graded by Claude Sonnet 4.5 against a strict faithfulness rubric at temperature 0. The judge never changes anywhere in this experiment. It is the instrument.

The break is one variable: three sentences deleted from the summariser's system prompt, the task sentence untouched.

# Baseline
SYSTEM_PROMPT = (
    "You summarize technical blog posts in 2-3 sentences for a reader deciding whether to "
    "click through. Base the summary ONLY on the text provided. Do not add claims, numbers, "
    "product names, or examples that are not explicitly present in the source text. If the "
    "post does not state a specific number or outcome, do not invent one."
)

# Regression: the guardrail is gone, the task is identical
DEGRADED_SYSTEM_PROMPT = (
    "You summarize technical blog posts in 2-3 sentences for a reader deciding whether to "
    "click through."
)

That is what a prompt looks like after somebody tidies it up, and it is the change a reviewer waves through.

Alongside it, two controls. A control arm runs the identical model and the identical prompt, byte for byte, changing nothing. The summariser runs at temperature 0.3, so the baseline is not deterministic and I need to know how much it moves on its own. And a positive control: the same prompt with a much weaker summariser, Amazon Nova Micro, which AWS describes as its "fastest text-only model, optimized for speed and low cost in tasks like summarization, translation, and classification", at roughly a thirtieth of Haiku 4.5's input price. Worth noting AWS names summarisation first, so this is not a model being used outside its stated purpose, it is a model built for speed and cost on short work being asked for faithfulness on long documents. That arm exists to show the gate detects anything at all.

Attempt one: the judge over the golden dataset

One run of each arm looked like a clean gradient.

Arm	Verdict	Mean faithfulness	Items moved
Baseline (Part 1)	16/16 PASS	5.00	reference
Control	16/16 PASS	5.00	0
Prompt regression	15/16 PASS	4.94	1
Model regression	11/16 PASS	4.19	7

Nothing changed, nothing moved. Subtle change, small movement. Blunt change, big movement. The gate works, just coarsely.

I nearly wrote that post. Part 1 had already taught me why not: its most interesting first-run finding did not survive a controlled re-run and was cut before publishing. One item moving, once, out of sixteen is the same shape. So I ran both arms three more times each.

Arm	Failures per run	Total	Mean faithfulness per run
Control	0, 0, 0, 0	0 / 64	5.00, 5.00, 5.00, 4.94
Prompt regression	1, 0, 0, 0	1 / 64	4.94, 5.00, 5.00, 5.00

Fisher exact on 0/64 against 1/64 gives p = 1.000. The gradient did not survive repetition. There was one event.

That one event is real, and worth looking at, because it is the only direct evidence in this whole post that the guardrail does anything. The judge failed a summary containing "caused real production issues". That phrase is not in the source, which says the author "burned an afternoon". The judge's reasoning named it exactly:

The summary accurately captures the main structure and three gotchas, but 'caused real production issues' overstates the source, which mentions 'burned an afternoon' and a separate OIDC gotcha in the blog's own pipeline, not that these three gotchas caused production issues.

A textbook scope-broadening hallucination, correctly caught and correctly explained, of exactly the category the deleted sentences existed to prevent. Once, in 64 gradings.

Attempt two: delete the judge

Here is the explanation I was ready to publish. Faithfulness was pinned at 5.00 on every baseline item, so the metric had no headroom. The verdict is a binary step function over a continuous quality change. The judge cannot see small drops.

It is a tidy story and it is testable, so I tested it. If the judge is the bottleneck, an eval with no judge in it should do better.

So the trap moves to the input, and the assertion becomes deterministic. Each source post already states things at low intensity ("burned an afternoon"). A guarded summariser is told not to escalate; an unguarded one is free to. I defined a vocabulary of escalations ("production outage", "caused an outage", "data loss", "in every case", "guaranteed", "catastrophic", and similar), then filtered it per post to the phrases verified absent from that post's own source text, so a hit can only ever be something the model introduced. Then a substring check. No judge, no rubric, no scores, no ceiling.

One caveat worth stating rather than letting a reader find it: this vocabulary is not independent of the first experiment. It contains "production issues", the phrase the judge flagged in attempt one. I built the second gate partly from what the first one saw, which makes this a weaker replication than two genuinely independent designs would be.

Gate	Control	Prompt regression	Fisher exact
LLM judge over golden dataset	0 / 64	1 / 64	p = 1.000
Deterministic assertion, no judge	1 / 64	1 / 64	p = 1.000

The control tripped on "critical failure". The regression arm tripped on "production incident". One each.

Removing the judge changed nothing. My explanation did not survive it.

What was actually wrong

Removing the judge changed nothing, so "the judge cannot see small drops" does not stand on its own as the explanation. That is weaker than saying the judge is exonerated, and the weaker version is what the data supports: both gates are blunt, and two blunt instruments returning the same null cannot tell you which one is blunt. What the next number shows is that the sample size alone is enough to explain both nulls, whatever the instruments were doing.

Both designs were asked to separate two rates that are, at everything measured here, roughly 1.6% each. So the real question is what size difference this design could have found at all.

With 64 gradings per arm at 80% power, the answer is 16.1%. The rate would have to rise more than tenfold before either gate reliably noticed.

That has to be computed exactly rather than with the usual two-proportion normal approximation, which wants around five expected events per arm. At a 1.6% baseline and n = 64 there is one expected event. Out of range, not borderline. I ran the normal approximation anyway and first published 15.4%, and the giveaway I ignored is that the standard methods disagree with each other by nearly a factor of two at these rates. The number above enumerates every outcome pair under two binomials and applies the same two-sided Fisher test used everywhere else here. It is twelve lines, in the repo.

It is worse than 64, too. Those 64 gradings are 4 repeats over the same 16 posts, so repeating them buys precision on each post's score rather than more posts. Counting each post once, the same calculation gives 42.6%, a 27-fold rise. So the honest answer is a range: between about 16% and about 43%, depending on how correlated repeat gradings of one post are. Evan Miller is explicit in Adding Error Bars to Evals (arXiv 2411.00640) that this sits on a sliding scale, from "perfectly correlated (in which case each cluster acts as a single independent observation)" to perfectly uncorrelated. His second recommendation covers "[w]hen questions are drawn in related groups, computing clustered standard errors" and his third covers reducing variance by resampling answers, which is the one that actually bites here. His fifth is the one I skipped:

Using power analysis to determine whether an eval (or a random subsample) is capable of testing a hypothesis of interest

That is the calculation I did not do. It runs in under a second and it answers the question both experiments spent real money failing to answer. An eval that cannot resolve the effect you care about does not return "no regression". It returns nothing, in a format that looks exactly like "no regression".

What the gate does catch

The positive control lands hard. Swapping to Nova Micro moved seven items and dropped mean faithfulness from 5.00 to 4.19. The distortions are the interesting part, because they are not invented facts. On the prompt-caching post the summary reported "up to 99.9% hit ratios, proving the cost benefits of enabling caching". The 99.9% is real: it is that post's own headline measurement. What the weaker model added was "proving the cost benefits", welding a cache hit ratio onto a billing claim the source measures separately at 55% and 78%. The judge caught the conflation, not the number.

So these gates are not broken. They are coarse, and the boundary is computable in advance rather than discoverable in production. Ten times the hallucination rate: caught. A change you would actually ship on a Tuesday: invisible.

The row that neither passed nor failed

Sixteen items in that arm. Eleven passed, four failed. That leaves one.

One grading came back as well-formed JSON with faithfulness: 4, completeness: 3, a populated hallucinations list and a sentence of correct reasoning. It had no verdict key. The judge answered every part of the question except the one the gate reads. By the rubric (PASS requires faithfulness 4 or higher and an empty hallucination list) a populated list makes it a FAIL, so the arm's true count is five non-passing out of sixteen, which is what the charts show.

It happened once, in the sixteen gradings that arm got, and not in the 128 Haiku gradings. That arm was never re-run, so this is not evidence that it is rare: conditional on the arm where it appeared, one in sixteen is all I can say. Now consider a run of 15 PASS, 0 FAIL, and one of these:

assert passed == total                    # 15 != 16, red. Catches it.
assert failures == 0                      # 0 == 0, green. Misses it.
pass_rate = passed / (passed + failed)    # 15/15 = 100%, green. Misses it.

Only the first survives. The other two treat a malformed response as if the item did not exist. The third is the version people write, because computing a rate from the two buckets you have feels more careful than counting, and it is the one that silently drops the row from the denominator.

Where this sits against what is already written

"Your eval gate might be noisy" is not my discovery. The variance problem is named in places, and Miller's paper is the rigorous treatment of it.

What I could not find is anyone publishing the control arm. The standard practitioner write-up, for instance Traceloop's guide to prompt regression testing with LLM-as-a-judge in CI/CD, recommends running new versions against a curated dataset, scoring with a judge, and failing the build when quality declines. Reasonable on its face, and it says nothing about how far that score moves when nothing has changed, which is the number that decides whether any of it works.

So the delta is not the idea. It is the measurement: a re-run-with-nothing-changed arm, the same regression measured 64 times by two unrelated methods, the raw JSON published, and a power calculation attached to the result rather than left implicit.

The suite I was about to recommend, which would not have worked

I had a fix section written. It said: keep a suite of known-bad inputs, each asserting a specific failure the judge must still catch, and if somebody deletes the guardrail the fabricated-number case stops failing.

Then I read my own code. Part 1's corruption suite hard-codes its five corrupted summaries as string literals and only ever calls the judge. The summariser is never invoked. Deleting the guardrail from SYSTEM_PROMPT cannot change that suite's inputs, its outputs, or its 5 out of 5 result. It is a judge-regression test. It guards the instrument, not the thing being measured, and it has nothing to say about the regression this entire post is about.

I would have shipped that as the takeaway. It was caught by an independent review pass with no knowledge of how the post was built, which is the only reason it is in this section instead of the conclusion.

What I now do

Compute the minimum detectable effect before running the eval, not after. If you cannot state what size regression your gate would catch, you do not have a gate, you have a ritual. Mine could catch a tenfold increase and nothing smaller, and one calculation would have told me that on day one.
Run the control arm. Re-run with nothing changed and measure how far it moves on its own. Without that number, every difference you find is unattributed.
Count clusters, not rows. Four repeats of sixteen items is sixteen independent units. Repeating a small dataset raises confidence in the mean, not the sample size.
Assert on the count, never on a rate derived from buckets. passed == total catches a malformed response. failures == 0 and passed / (passed + failed) both quietly drop it.
Check that your test exercises the thing you changed. A suite can be rigorous, pass cleanly, and be pointed at a different component entirely.

The honest caveats

The load-bearing limit is the one this whole post is about: I cannot tell you how much worse the guardrail deletion makes the summariser. I can show it produced at least one real hallucination, verified absent from its source. I cannot show it happens more often than with the guardrail in place, because neither design has the resolution to distinguish 1.6% from 1.6%. There is a reading of this data where the true post-deletion rate genuinely is about 1.6%, the gates are correctly reporting a tiny effect, and "underpowered" rather than "blind" is the right word. That reading is consistent with every number here, and the practical conclusion does not change: compute the effect you need to detect before you trust the answer.

Sixteen posts, one judge model, one rubric, one summariser family, four repeats per arm. I am not claiming a general result about LLM-as-judge regression detection. The mechanism is general enough to be worth checking in your setup; the numbers are mine.

The positive control is weaker evidence than the other two arms, deliberately. My intended swap was to an older model in the same family, which would have been a clean single variable. Nova Micro crosses model families, so tokenizer, training and instruction-following all move at once. It is there to show the gate detects something, not to attribute what.

Part 1's runs were on 2026-08-18 and these on 2026-08-24: six days apart, same judge model, same rubric, same account, not the same sitting.

The model that went away mid-experiment

The within-family swap was blocked, for a reason worth stating accurately, because my first instinct was to write it up as "the provider retired a model out from under me with no warning" and that is not what happened.

Bedrock's model lifecycle has three states: Active, Legacy, End-of-Life. Claude 3 Haiku moved to Legacy on 10 March 2026 with a published EOL of 10 September 2026, sixteen days after this post goes up. Six months of notice on a public page. No ambush.

What actually bit me is narrower. Access lapses on an inactivity timer: new customers cannot use a Legacy model at all, and existing customers "may lose access" after a period of not calling it. The exact period is worth flagging because the sources disagree. The documentation says access may be lost "after 15 days of inactivity". The error Bedrock returned to me says 30.

ResourceNotFoundException: Access denied. This Model is marked by provider as
Legacy and you have not been actively using the model in the last 30 days.
Please upgrade to an active model on Amazon Bedrock

I am quoting both rather than picking one, because I do not know which governs and I did not test the boundary.

The documented model-access flow is to re-accept the agreement with aws bedrock list-foundation-model-agreement-offers and aws bedrock create-foundation-model-agreement, or through model access in the console. AWS documents those as the model-access steps; it does not connect them to Legacy inactivity specifically, and I did not test whether they recover a lapsed Legacy model. To check where you stand before any of this matters, aws bedrock get-foundation-model-availability --model-id <id> returns an agreementAvailability field. There is also a phase worth knowing about: since 10 June 2026 Claude 3 Haiku has been in public extended access, during which the docs say to expect higher, provider-set pricing. In this case that has not bitten. Bedrock's published extended-access price table currently lists only Claude 3.5 Sonnet and 3.5 Sonnet v2, not Claude 3 Haiku.

So the operational lesson is not that providers retire models without warning. It is that a benchmark baseline you only run occasionally is exactly the workload an inactivity timer catches, that reactivating may cost more than it did, and that after the EOL date it stops being recoverable at all. Pin the recorded outputs rather than assuming you can regenerate them.

All five rounds, every script, and the raw JSON for every run and every arm are in the repo: github.com/rajmurugan01/do-you-trust-it-evals.

Series

This is Part 2 of Evals for Production AI, a series on how you actually know an AI system is good once it's live, not according to the dashboard. Part 1 asked what it takes to trust an LLM judge's score. This one pointed that trusted judge at the job most teams want it for, and found the instrument was not what decided the answer.

If you are wiring an eval into a deploy gate and want a second pair of eyes on where its detection floor actually sits, happy to compare notes. Find me on LinkedIn or via rajmurugan.com.

Next in Evals for Production AI: an offline eval can only ever test the inputs you thought to write down. What live production signals catch that a golden dataset structurally cannot, and how to spot a quality drop in traffic you never anticipated.

What a Year 10 study system taught me about production AI failure modes

Raj Murugan — Tue, 18 Aug 2026 02:26:58 +0000

I spend my professional time shipping production AI agents on AWS for enterprise clients. Bedrock AgentCore, Strands, the integration layer between LLMs and the systems that actually matter. The work that does not show up in demo videos.

A few weeks ago I built an AI system for my Year 10 son. Not on AWS. Anthropic's Claude projects and Cowork routines, because the data lives in Google Workspace and the user is a teenager who needs a frictionless experience. Different stack, same architectural disciplines.

It took three iterations and one architecture pivot inside the third before it worked. What I did not expect was how cleanly the failure modes mapped to patterns I see in production Bedrock work. The five lessons below are the ones I will be taking straight back into client engagements.

The build, briefly

Three subject-specific Claude projects with Socratic tutoring. A weekly Cowork routine that aggregates the week and produces three differentiated emails: a full dossier to me, a focused brief to each tutor, a forward-looking plan for him.

Cowork is Anthropic's scheduled-routine service for Claude. Think of it as cron for Claude projects, with built-in connectors to Gmail, Drive, and Calendar.

The architecture had to do two things: read what happened in the projects during the week, and produce structured output across multiple channels on a schedule. Sounds simple. Most agentic systems do.

Three iterations, one architecture pivot

The version above is v3. There were two earlier shapes I had to rule out before I got there, and inside v3 there was a further pivot at the integration layer. The full diagram is on the canonical post — rajmurugan.com — here's the prose walkthrough.

v1: the answer machine

The first build was the obvious one. Ask Claude a question, get an answer. It worked on day one and failed the same week. He was not learning; he was copying. The pedagogical intent — that the system make him think — was nowhere in the design. I rebuilt.

v2: Socratic tutor, per subject

v2 made the model refuse to give direct answers. One Claude project per subject, system prompts tuned to ask back rather than tell, hint at the next step rather than skip to the end. That worked. He had to actually do the maths. His tutors started commenting that the homework conversations were sharper.

The constraint we hit next was visibility. Three subjects across a week is a lot of context for a parent and three different tutors to absorb. We needed a layer on top that aggregated the week and split it for different audiences. That extension is what became v3.

v3's first architecture failed at the integration layer

My first design for the v3 aggregation layer had the Sunday routine reading project chats directly and updating a rolling Google Doc. First real run: neither side worked. Cowork has no API surface to read project chats on the same account. The Drive connector is read-only for content. Two load-bearing assumptions, both wrong.

I have seen this exact pattern in Bedrock work. A team designs an agent that "queries the knowledge base, then updates the ticket." First deployment surfaces the truth: the knowledge base query returns chunks the agent cannot reason over, or the ticket system's API has a write surface that does not match the read surface. The system worked on paper because nobody tested the primitives in isolation.

The architecture that works flips the data flow

The redesign inverted both directions. Each subject project now drafts a session-summary email at end-of-session with a fixed subject prefix. I review the draft and click Send. The Sunday routine searches Gmail by subject prefix instead of reading project chats. The routine creates a new dated doc each week instead of modifying an existing one.

Same intent. Different primitives. Push beats pull, and append beats mutate.

This is the same shift I keep recommending in Bedrock production work (see Part 3 of the AgentCore series where I walk through the same pattern with Strands tool calls). When an agent needs to "look inside" a system that cannot expose its state cleanly, the answer is almost always to make the producer emit, not to make the consumer introspect. Pull architectures have one failure mode for every integration. Push architectures have one failure mode total: the producer doesn't emit. That is debuggable.

Five lessons that map directly to production AWS AI

1. Verify the primitives before you design around them. v3's first design cost me an evening because I drew a diagram before I tested whether Cowork could read project chats. Most failed Bedrock POCs I see fail for the same reason at a bigger scale: the team designs around assumed Knowledge Base behaviour, assumed AgentCore session state, assumed Strands tool-call semantics, without the kind of isolated primitive testing I'd insist on at the start of any CDK deployment. Save the diagram for after the smoke test.

2. When introspection isn't possible, push beats pull. This is the single most useful pattern I have learned this year. Half the production Bedrock work I do involves rearranging data flow from pull to push because the producer can be modified and the consumer cannot. If your agent needs to read state from a system that doesn't expose it cleanly, the right move is almost never a better retrieval strategy. It's an emit.

3. Human-in-the-loop is a feature, not a bug. v3 has me clicking Send on two emails per cycle. Ten seconds of human attention, full audit trail, kill switch on every outbound message. I argue for this on every enterprise engagement and lose half the time, because someone wants the autonomy metric. The teams that ship the human-in-the-loop version tend to still be running their agents six months later. The teams that skip it learn the value of an audit trail the hard way.

4. Calibration windows beat go-live confidence. Three Sunday runs on real data before flipping to multi-recipient. Same discipline I apply to Bedrock agents before they touch a customer-facing channel. Skipping calibration is the single biggest predictor of an enterprise AI rollback I have seen.

5. The "things we got wrong" doc is more useful than the spec. I now write a decision log on every client engagement, with the versions-that-didn't-work explicitly preserved. Six months later, when someone joins the project and asks "why is it built this way," that doc is the answer. The polished spec is for review committees. The decision log is for the team.

There is a sixth lesson — about the pedagogical move from v1 to v2, and what it taught me about scoping AI agents around intent rather than output. That one belongs in a separate post on when to graduate from Claude projects to AgentCore. It's in draft.

Why I'm posting this on an AWS-focused blog

Because the lessons are stack-agnostic. AgentCore, Strands, Bedrock Knowledge Bases, Lambda-backed tools, plain old Claude projects with Cowork: the architectural disciplines are the same. The model layer is interchangeable. The integration layer is where production AI lives or dies. That's true on AWS, that's true on Anthropic's stack, and that's true on whatever comes next.

Side builds like this are how I sharpen patterns I then apply at scale to enterprise AWS engagements. Enterprise engagements take longer to write up because they have to be anonymised. Side builds let me publish the pattern faster.

The repo

Full design history, the v1 and v2 specs, the v3 architecture that failed, the v3 architecture that works, the prompts, the build plan, and the calibration log are open-source: github.com/rajmurugan01/study-loop

If you ship production AI on AWS and any of these patterns ring true, I would like to compare notes. Find me on LinkedIn, or comment below.

More posts on production AWS AI: browse the blog or subscribe by RSS. The next post in this thread, on when to graduate from Claude projects to AgentCore, is in draft.

Field Notes: Turning prompt caching on for a production Bedrock workload

Raj Murugan — Tue, 18 Aug 2026 02:26:54 +0000

Two kwargs in Strands' BedrockModel cut a Bedrock workload's system-prefix billing by 78%. The Strands tutorial doesn't mention them. Almost none of the production Strands code I've audited this year has them set.

This is Part 2 of AI Operations Services — the deep dive on prompt caching that Part 1 promised. Short, specific, entirely evidence-led: two kwargs to enable, one per-model gotcha that took half a day to find, one measurement technique that gives you the answer in seconds without waiting for CloudWatch to aggregate.

The default is None

from strands.models.bedrock import BedrockModel

BedrockModel(
    model_id="eu.amazon.nova-pro-v1:0",
)

That is the example shape you find in most Strands tutorials and the agentic-AI content on Bedrock. It is also the shape that produces a Bedrock call with no cachePoint block, no system-prefix caching, no tool-registry caching, and a full-input bill on every turn.

BedrockModel exposes two kwargs that turn caching on:

BedrockModel(
    model_id="eu.amazon.nova-pro-v1:0",
    cache_prompt="default",   # caches the system prompt
    cache_tools="default",    # caches the tool registry
)

Both default to None. The Strands docs mention the kwargs in the API reference but not in the getting-started flow, so they are easy to miss on first build and easy to forget on the second. Every workload I have walked into this year had them unset. The fix is two kwargs and a boto3 version bump if your environment is more than a quarter behind.

What the two kwargs actually do

Under the hood, Bedrock's converse and converseStream accept a cachePoint block at specific positions in the request body. The block tells Bedrock "cache the prefix up to this marker, and on a subsequent call with the same prefix, bill it as a cache read instead of a full input."

cache_prompt="default" inserts a cachePoint at the end of the system block, so the entire system prompt becomes a cacheable prefix. cache_tools="default" inserts a cachePoint inside toolConfig.tools, so the tool registry becomes the next cacheable prefix after the system block. Both points compose: a call with both set caches system + tools together, which is the right thing to want when both are large and stable.

The TTL is five minutes from the most recent cache write or read on a given prefix. Inside the TTL, subsequent calls bill the prefix tokens at the cache-read rate. Outside the TTL, the next call pays a fresh cache write and the meter restarts.

The per-model gotcha

This is the thing that cost me half a day, because it is not in the Bedrock docs and the error message points at the request shape rather than the underlying constraint:

Malformed input request: extraneous key [cachePoint] is not permitted

The error appeared on every Nova call after I set both cache_prompt and cache_tools, but only when both were set. Sonnet 4.6 took the same config without complaint. The difference is per-model: Bedrock's server-side validator gates cachePoint placement per model family, not per feature.

What works on each:

Model family	`cachePoint` in `system`	`cachePoint` in `toolConfig.tools`
Amazon Nova (Pro, Lite, Micro)	accepted	rejected server-side
Anthropic (Sonnet, Haiku, Opus)	accepted	accepted

The pattern that survives a fallback chain with both families in it is one config-of-config: pass cache_prompt="default" to every model in the chain, and pass cache_tools="default" only to the Anthropic-family models. If you do not split it, the Nova path fails on every call with the malformed-input error and the SDK retry loop swallows the failures into the fallback chain. From the dashboard, the symptom is "Nova is throttling at 100%" with no further clue. From the agent logs, the symptom is the actual error string above.

from strands.models.bedrock import BedrockModel

def make_model(model_id: str) -> BedrockModel:
    is_anthropic = model_id.startswith("anthropic.") or "claude" in model_id.lower()
    return BedrockModel(
        model_id=model_id,
        cache_prompt="default",
        cache_tools="default" if is_anthropic else None,
    )

That is the smallest piece of code that handles both families correctly. Half a day saved.

How to verify, in seconds

The default reflex for verifying a Bedrock change is to wait for CloudWatch metrics to aggregate, then look at cacheReadInputTokenCount per ModelId over a 15-minute window. That works, but it is the slow path. The fast path is the per-call usage block returned inline by bedrock-runtime.converse(...):

import boto3, json

client = boto3.client("bedrock-runtime")
response = client.converse(
    modelId="eu.amazon.nova-pro-v1:0",
    system=[{"text": SYSTEM_PROMPT}, {"cachePoint": {"type": "default"}}],
    messages=[{"role": "user", "content": [{"text": "ping"}]}],
)

print(json.dumps(response["usage"], indent=2))

response["usage"] carries four keys you want: inputTokens, outputTokens, cacheReadInputTokens, cacheWriteInputTokens. On call 1 against a freshly-seeded prefix, cacheWriteInputTokens is large and cacheReadInputTokens is zero. On call 2 against the same prefix inside the TTL, cacheWriteInputTokens drops to zero and cacheReadInputTokens is large.

If you instead see cacheWriteInputTokens: 0 and cacheReadInputTokens: 0 on every call, your config did not take effect: the cachePoint block is missing from the request, or the SDK version is too old to emit it, or the prefix is too short to be cacheable (Bedrock has a per-model minimum).

The per-call usage block is the right measurement primitive because it is exact, immediate, per-turn, and free. No CloudWatch lag, no metric aggregation, no dashboard to build. Three calls and you know.

The five-second propagation lag

A subtlety the docs do not flag: Bedrock takes a few seconds to make a freshly-written cache entry available for reads. Fire two calls inside a second against the same prefix and the second one will pay a full cache write rather than a cheap read. The lag I measured on Nova and Sonnet in eu-central-1 was around five seconds; six seconds between calls is enough to clear it.

This matters for two reasons. First, when you measure caching with a tight loop, you will conclude caching does not work, because turn 2 of your driver will still be a write. Use spaced calls or accept that your measurement run wastes the first call or two on writes. Second, in production, the lag means a burst of three calls in the first second of a user turn pays one write plus two writes, not one write plus two reads, on a fresh prefix. After the first burst, every subsequent call inside the TTL is a read.

The measured results

Methodology: real production system prompt (8,156 tokens on the Nova tokeniser and 8,788 on the Anthropic tokeniser, varying by tokeniser not by content), 10-turn driver against the workload's staging Bedrock account, 6-second intra-call spacing, both Nova Pro and Sonnet 4.6 in the same run. cache_prompt="default" set on every model, cache_tools="default" set on Sonnet only.

Model	Turns	Hit ratio	System-prefix billing reduction
Amazon Nova Pro	10	99.9%	55%
Anthropic Sonnet 4.6	10	99.8%	78%

The asymmetry is pricing-driven. Anthropic publishes its cache-read multiplier directly: 10% of input price per the Anthropic pricing page (retrieved 2026-06-26). At 99.8% hit ratio on the system-prefix tokens, that is 78% off the full-input bill. Nova's cache-read multiplier is on the AWS Bedrock pricing page; plug your contracted rate against the measured 99.9% hit ratio to compute your own reduction.

Two caveats. The hit ratios are measured on a 10-turn driver in a single run, not aggregated across days of production traffic. In production, calls drift in and out of the TTL window depending on user-burst patterns, so the steady-state hit ratio will be lower than 99.9%. The number to track post-deploy is the per-day ratio of cacheReadInputTokens to cacheReadInputTokens + cacheWriteInputTokens + non-cached inputTokens per ModelId. Second: prompt caching only helps the prefix tokens, not the per-turn user message or output.

What I now do on every Strands `BedrockModel`

Three things, in this order:

Set cache_prompt="default" on every model in the fallback chain.
Set cache_tools="default" on the Anthropic-family models only.
After deploy, fire three calls against the prefix with five to six seconds between them and print response["usage"]. Confirm call 1 has a non-zero cacheWriteInputTokens and calls 2 and 3 have a non-zero cacheReadInputTokens.

Then add CacheReadInputTokenCount per ModelId to the workload's CloudWatch dashboard for the steady-state ratio. The dashboard is not how you verify the deploy, it is how you spot regressions: a hit ratio that drifts down over time is usually a sign that the system prompt is being mutated per request and the cache is being invalidated on every call.

Series

Part 1: Three things I learned diagnosing a production Bedrock workload — load tests can lie, latency isn't always model speed, prompt caching is almost never on.

Part 2: this post, the deep dive on the caching kwargs and the per-model gotcha.

Part 3 (coming): attributing a mixed change — the latency improvement on the engagement above shipped a model and region swap together; how much was each.

If you are running Strands with both Nova and Anthropic in your fallback chain, have you hit the Nova toolConfig.tools rejection? Curious whether anyone has solved it differently than splitting cache_tools per model family. Drop a note in the comments or DM me on LinkedIn.

A clean pass rate is not calibration

Raj Murugan — Tue, 18 Aug 2026 02:23:42 +0000

Sixteen out of sixteen. Every summary my eval graded came back faithful, first try. That number should make you suspicious of the eval, not proud of the system, and it's the reason this post exists: a same-day Bedrock experiment on my own blog, built to find out what it actually takes to trust an LLM-as-judge score before you wire one into anything that matters.

The setup

Sixteen posts, two Bedrock models in my own AWS account, one small repo: do-you-trust-it-evals.

Claude Haiku 4.5 (us.anthropic.claude-haiku-4-5-20251001-v1:0) writes a 2-3 sentence summary of each of my 16 published posts, instructed to use only claims present in the source.
Claude Sonnet 4.5 (us.anthropic.claude-sonnet-4-5-20250929-v1:0) grades each summary against its source: faithfulness 1-5, completeness 1-5, a list of unsupported claims, and a PASS/FAIL verdict. The rubric treats a conditional claim stated as a universal ("sometimes" becoming "always") as a hallucination, not just an invented fact.

Round 1: the baseline that should worry you

Cold run, no tuning: 16 out of 16 summaries passed, faithfulness 5 across the board. I didn't take the judge's word for it. The summary for the CDK post claims "nine specific pitfalls", and the source has exactly nine, numbered ## Gotcha #1 through ## Gotcha #9. The summaries were genuinely faithful, not a rubber stamp catching nothing because there was nothing to catch.

A 100% pass rate proves the eval didn't break on the easy case. It proves nothing about whether the judge would catch a hard one.

Round 2: calibrate with a single variable

Five corrupted summaries, each with exactly one injected error and nothing else touched, so a FAIL verdict can only be explained by that one change: a fabricated number, a conditional claim broadened to a universal, a real number misattributed to the wrong post, a fabricated named entity, an inflated count.

5 out of 5 caught, and in every case the judge's own hallucination list named the exact injected error. If you can't point to the one thing you changed in a corrupted test case, you don't have a calibration result, you have a guess with a percentage attached.

Round 3: does the judge agree with itself

Fiona Lau's "Same Input, Different Scores" (2026) found substantial LLM-judge score variability even at temperature 0, with completeness scoring showing the largest fluctuations. So I re-ran two inputs, an easy clean one and a hard borderline one, four times each, tracking both faithfulness and completeness. Both held steady across all eight runs.

What I now do

Treat a 100% pass rate as an instruction to calibrate, not a result to report.
Corrupt one variable per test case, and diff it against the original to check.
Cover more than one failure category: a fabricated fact, a scope-broadened claim, and a misattributed-but-real number all fail differently.
Re-grade the same unmodified input more than once before trusting a single run.

Full methodology, all three scripts, and the raw JSON for every round: github.com/rajmurugan01/do-you-trust-it-evals.

This is Part 1 of Do You Trust It?, a series on how you actually know an AI system is good once it's live. Full write-up with the honest caveats (small n, benign-only corruptions, what this doesn't test) on rajmurugan.com.

Your LLM security diagram defends the wrong layer

Raj Murugan — Tue, 21 Jul 2026 09:42:01 +0000

Search "LLM security architecture" and you will meet the same diagram again and again. A tidy left-to-right pipeline, user input to retrieval to the model to output to tools, and hanging underneath it four red boxes: prompt injection, retrieval poisoning, context poisoning, output injection. It is a good diagram. I have drawn versions of it myself. And if you build your defences the way it is laid out, you will spend a year patching the wrong layer.

I argued in the last post that the LLM is not a security boundary, that the controls which actually hold are deterministic and live in code the model never touches. This is the same idea from the other side. The four-box diagram is not wrong about the threats. It is wrong about where the defence goes, and the error is baked so far into the layout that it is hard to see.

What the diagram gets right

The four attacks are real, and the diagram names them well. Prompt injection: an attacker hides an instruction in text your model reads and steers it. Retrieval poisoning: a malicious document lands in the corpus and gets pulled into context. Context poisoning: retrieved data carries an embedded instruction. Output injection: an unvalidated model response gets executed as a command downstream. Every one of these has put a real system on an incident call. Naming them is useful.

The diagram is also comprehensive-feeling, which is most of its appeal. It walks the pipeline stage by stage and hangs a threat under each stage, so it looks like a complete accounting. Nothing is missing. That completeness is exactly what makes the next step feel obvious, and the next step is the trap.

The trap is in the layout

Read the diagram as a to-do list and it hands you one mitigation per box. Worse, the obvious mitigation for each box sits at the same stage as the threat, which is the stage the attacker controls.

Prompt injection sits at the input, so you reach for input scanning and prompt hardening. The attacker writes the input.
Retrieval poisoning sits at the corpus, so you reach for document scanning. The attacker writes the document.
Context poisoning sits at the model's reading of the context, so you reach for a grounding or relevance score. You are now scoring meaning.
Output injection sits at the output, so you reach for an output filter. That filter is a classifier.

Every one of those is a probabilistic classifier you tune against an adversary. Thresholds, false negatives, a curve you push toward zero and never reach. Build all four and you have not built a boundary. You have built four leaky nets stacked on top of each other and called the result secure. A determined injection that scores just under every threshold walks the whole length of the pipeline untouched.

And there is a tell that the list itself is the problem: it grows. Tool poisoning, memory injection, confused-deputy attacks across agents. Next quarter there is a fifth box, and a defence organised as one-patch-per-named-attack is permanently a step behind the naming. It is an open set sold to you as a closed one.

The layer the diagram leaves out

Here is the fact the layout hides. All four attacks are harmless right up until model output crosses into a consequence: a tool call, a database query, a retrieval, or an answer leaving the building. That crossing is a single surface. It is finite, it is enumerable, and it is the one layer the attacker does not control, because it is your code.

The reflex the diagram trains looks like this, and it is quietly futile:

def handle(user_input, ctx):
    if injection_score(user_input) > 0.9:        # tuned threshold
        raise Blocked()
    chunks = retrieve(user_input)
    chunks = [c for c in chunks if malice_score(c) < 0.9]   # tuned threshold
    answer = model(user_input, chunks)
    if output_score(answer) > 0.9:               # tuned threshold
        raise Blocked()
    return answer

Three gates, three thresholds, and an injection tuned to score 0.89 everywhere sails through all of them. Now the boundary version, at the layer the diagram skips:

# Model output is a proposal, never an instruction. Every proposal is
# checked against the finite set of things allowed to happen.
def act_on(proposal, ctx):
    if proposal.tool not in TOOLS:          # enumerated capability, not a classifier
        raise NotAllowed(proposal.tool)
    check_authz(ctx.principal, proposal)    # token-derived principal, per call
    return TOOLS[proposal.tool](proposal.args, ctx)

And retrieval filtered before the model, keyed off the verified principal, so a chunk the principal is not cleared to see is never loaded in the first place:

chunks = vector.search(q, k=20, filter={"acl": ctx.principal.entitlements})

Be precise about what this filter does and does not do. It is a confidentiality control, not an anti-poisoning one: it stops chunks outside the principal's entitlements, but a poisoned document that sits inside their authorised scope carries a valid ACL and passes. That residue is caught downstream, at the action boundary and by the grounding backstop, not here. What the filter does buy you, and it is the reason the fourth threat surface (an answer leaving the building) never needed its own allow-list, is that the model can only ever repeat data the principal was already cleared to retrieve. The answer is bounded on the way in, not policed on the way out.

None of this asks what the attack was called. Prompt injection, context poisoning, some technique that does not have a name yet: they all arrive at the same door, and the door checks the action against a list, not the intent against a classifier. The model can be fooled into proposing anything. It cannot be fooled into a proposal that the door has no entry for.

Enumerate the actions, not the attacks

That is the whole reframe. You cannot enumerate the ways a model can be fooled. That set is open, adversarial, and growing while you read this. You can enumerate the actions your system is allowed to take: the tools in the registry, the tables in the grant, the entitlements on the index, the destinations on the egress allow-list. That set is small, closed, and yours to write down and review.

This is why the deterministic boundary is load-bearing and the classifiers are not. One defends a finite set you control. The other chases an infinite set the attacker controls. When people say defence in depth, they usually mean "more layers." The layer that matters is the one where an infinite problem becomes a finite one.

Where the four boxes still earn their place

I am not telling you to throw the diagram out, and I want to be precise about the limits of my own argument, because the reframe oversells if you let it.

The boundary stops security failures, not correctness ones. If a poisoned document convinces the model to give a wrong but fully authorised answer, no allow-list catches that. The action is within the user's rights. It is just wrong. Grounding checks, provenance, and citation-of-source are the right tools there, and those are exactly the probabilistic layer I just spent five paragraphs demoting. Demoted, not deleted. They move from load-bearing to backstop, which is where they belong.

The boundary also constrains which actions and whose, not what rides inside them. An allowed tool called with attacker-shaped arguments is still an allowed tool: talk the model into calling a permitted send_report with an exfiltrating recipient, and a registry check that only asks "is this tool allowed" waves it through. This is why the action boundary is not just the tool list. It is the tool list, plus the grant, plus what each action is allowed to carry and where it is allowed to send it. Enumerate the arguments and the destinations too, not only the verbs. The egress allow-list is doing as much work as the tool registry.

Input and output scanning also earn a place. They raise the cost of the low-effort attacks and, more usefully, they give you signal to detect the attempt. Keep them. Just never let a tuned classifier be the only thing standing between the model and the data.

So the correction is not "the diagram is wrong." The threats are real and the diagram names them cleanly. The correction is "stop reading a threat map as a defence architecture." Put the deterministic boundary in first. Then let the classifiers backstop it, labelled honestly as backstops.

The companion post walks through where each of these lives in code. Here is the shape of the boundary itself, control by control:

What I now do

Before drawing a single mitigation, list the actions the system can take: tools, tables, entitlements, egress destinations. That list, not the attack list, is the security surface.
Put a deterministic check on each action, at the point of the action, keyed off a verified principal, never off anything the model produced.
Treat every input scanner, grounding check, and output filter as a backstop, and give it a name that says so in the design doc. Budget for it and tune it. Do not let it hold the line alone.
When a new attack name starts trending, ask one question before building anything: does my action boundary already stop it? Most of the time the answer is yes, and the correct amount of new work is none.

The four-box diagram will keep circulating, because it is a genuinely good map of where the water comes in. Just remember that a map of the leaks is not a plan for the wall. The wall goes lower than the diagram draws it, at the layer the attacker cannot reach, and it is made of boring deterministic code that does not care what the flood is called.

Part 6, and the close, of **Production AI, Honestly: the series on the AWS AI work between the demo and something you'd trust a customer behind. It runs from measuring where the probabilistic layer lies (cost, caching, memory) to building the boundary that holds (the LLM is not a security boundary walks those controls in detail) to this, the reframe of the diagram that had everyone patching the wrong layer. If you are building agents over sensitive data and want a second set of eyes on where your real boundary sits, get in touch.

Field Notes: The AgentCore Memory write that returns success and reads back empty

Raj Murugan — Mon, 13 Jul 2026 04:58:59 +0000

I wired long-term memory into an agent on Amazon Bedrock AgentCore, wrote a record, got a 201, and read back nothing. The record existed. The API said success. The read returned zero rows. It took me longer than I would like to admit to work out that all three of those were true at the same time.

AgentCore went GA on 13 October 2025, after a July preview. Memory is one of its newer pieces, and the docs are good on the happy path and quiet on the parts that bite. This is the operational truth of the write side, the bit you only learn by running it.

Two things bit me. One was an API that did not exist. The other was a write that succeeds and is not yet readable. Neither is in the tutorial.

The API that never existed

The codebase had a helper for persisting a fact to memory. It called client.ingest_memory_records(...). It read as correct. It had a docstring. It had a sensible name.

It had never run. It was written, never called, and so had never thrown. When I finally wired it into a real path, I checked the client first:

import boto3
c = boto3.client("bedrock-agentcore")
hasattr(c, "ingest_memory_records")   # -> False

There is no ingest_memory_records operation on the AgentCore data plane. The helper would have raised AttributeError the first time anyone called it. Dead code that mirrors a real-sounding API is worse than no code, because it passes the eye test. A method name is not a fact. It is a claim, and an unrun claim is a guess wearing a fact's clothes.

The real operation is BatchCreateMemoryRecords. Confirm it against the SDK you actually ship, not against what the name suggests:

ops = c.meta.service_model.operation_names
[o for o in ops if "Memory" in o or "Record" in o]
# BatchCreateMemoryRecords, BatchDeleteMemoryRecords, BatchUpdateMemoryRecords,
# DeleteMemoryRecord, GetMemoryRecord, ListMemoryRecords, RetrieveMemoryRecords

Lesson one, before any of the memory detail: verify the operation exists in your pinned SDK version. The write API and the read API are not symmetric in name, and one of the plausible names is a trap.

Better than checking one call by hand, make the build check every call. The service model is the ground truth for what exists, so a CI test can fail the build the moment source names an operation that does not:

# CI: fail the build if the codebase calls a bedrock-agentcore method that
# does not exist in the pinned SDK. Kills the ingest_memory_records class,
# including the next hallucinated API someone commits.
import boto3
client = boto3.client("bedrock-agentcore", region_name="us-east-1")
called = {"batch_create_memory_records", "get_memory_record",
          "retrieve_memory_records", "ingest_memory_records"}  # grep these from source
missing = [m for m in called if not hasattr(client, m)]
assert not missing, f"no such AgentCore operation: {missing}"   # -> ['ingest_memory_records']

Two tiers, and which one you are writing to

AgentCore Memory has two tiers, and the whole confusion comes from not knowing which one a given call touches.

Short-term memory is raw events. You write them with CreateEvent, one per turn or in batches, scoped to an actor and a session. This is conversation history. You do not semantically search it.

Long-term memory is extracted records, organised into namespaces. Normally these are produced asynchronously: a memory strategy runs in the background, reads your short-term events, and extracts or consolidates records into a namespace. The AWS docs are explicit that this generation is an async background process. You read long-term records with RetrieveMemoryRecords, a semantic search scoped to a namespace.

So if you want an agent to recall a durable fact on the next turn, it has to live in a long-term namespace that RetrieveMemoryRecords reads. Writing a CreateEvent and hoping the strategy extracts the right fields is slow and non-deterministic. There is a better path.

BatchCreateMemoryRecords writes directly into a long-term namespace. It is the bring-your-own-extraction door: you have already structured the fact, so you skip the strategy and put the record where the reader will look. The request is straightforward:

c.batch_create_memory_records(
    memoryId=MEMORY_ID,
    clientToken=token,   # batch idempotency; a retried identical batch dedupes
    records=[{
        "requestIdentifier": "seed-1",   # correlation key, NOT idempotency
        "namespaces": ["/orgs/<tenant>/user/<user>/preferences/"],
        "content": {"text": "Role: AE, mid-market SaaS. Prefers blunt feedback."},
        "timestamp": now,
        # memoryStrategyId is optional; see below
    }],
)

Note there is no actorId argument. The actor is encoded into the namespace string. Get the namespace wrong and the write goes somewhere the reader never queries, which is its own quiet failure.

That has a security edge too. Because the actor is just part of a string, your tenant isolation is only as strong as the code that builds it, and a bug there is a cross-tenant read. The same reasoning behind the LLM is not a security boundary applies to your own string formatting: do not let it be the only thing standing between tenants. RetrieveMemoryRecords honours the bedrock-agentcore:namespace (exact) and bedrock-agentcore:namespacePath (subtree) IAM condition keys, so a policy can pin a principal to its own /orgs/<tenant>/ prefix and the service refuses an off-tenant namespace whatever the code passes.

Success is not retrievability

Here is the part that cost me the afternoon. The write returns 201 with a successfulRecords entry and a memoryRecordId. Fetch that id directly and the record is there immediately:

r = c.batch_create_memory_records(memoryId=MEMORY_ID, records=[rec])
rid = r["successfulRecords"][0]["memoryRecordId"]

c.get_memory_record(memoryId=MEMORY_ID, memoryRecordId=rid)  # exists, right away

Now query the namespace the way an agent actually would, and at five seconds it is empty:

c.retrieve_memory_records(
    memoryId=MEMORY_ID,
    namespace="/orgs/<tenant>/user/<user>/preferences/",
    searchCriteria={"searchQuery": "role preferences", "topK": 10},
)  # +5s -> 0 records
c.list_memory_records(memoryId=MEMORY_ID, namespace=NS)  # +5s -> 0 records too

Both the semantic read and the plain namespace list returned nothing, while the record was fetchable by id the whole time. Poll for longer and the rows appear. So the record was created, addressable, and not yet indexed for namespace or semantic retrieval.

I ran that write-then-poll loop three times, fresh namespace each time, checking every three seconds. The semantic read first returned the record at 16, 27 and 15 seconds. The namespace list was similar, at 16, 23 and 15 seconds. Same account, same region, one sitting, so treat it as a rough ballpark, not a benchmark. And treat the fresh-namespace part as a confound, not a control: some of that time may be the namespace itself warming up rather than the record indexing, so a warm namespace already holding thousands of records could behave differently. I have not measured that steady state yet, and it is the number production would actually care about. Three samples also cannot see a tail, and the tail is the whole operational question, so the right move is not to trust the ballpark at all.

The docs tell you long-term generation from events is asynchronous. They do not tell you that a direct BatchCreateMemoryRecords write also indexes asynchronously. I went looking, in the API reference and the memory guide, and could not find the read-after-write behaviour stated anywhere. You would reasonably assume the direct door skips the wait, because you did the extraction yourself. It does not skip the indexing.

The mental model that would have saved me the afternoon: 201 means accepted, not queryable. Read-after-write on a namespace is eventually consistent, on the order of tens of seconds. If you need certainty that a specific record landed, read it by id with GetMemoryRecord, which is immediate. If you need it to appear in a namespace search, poll until it does rather than sleep on a fixed guess (more on that below).

One consequence follows straight from that lag: because the write returns before it is searchable, a retry inside the window is easy, and requestIdentifier will not save you. It is a correlation key, so two writes with the same one create two distinct records. The idempotency control is clientToken on the batch call. Retry the identical batch with the same token and the service dedupes it. If your write path retries on timeout, set it.

The one that was simpler than the docs implied

A smaller finding while I was in there. Long-term records carry an optional memoryStrategyId. I assumed retrieval might require the record's strategy to match the strategy that owns the namespace. It does not. I wrote two records to the same namespace, one with a memoryStrategyId and one without, and RetrieveMemoryRecords returned both. So an unfiltered retrieve does not gate on strategy id: give it a namespace and a searchQuery and it returns matching records whether or not they carry a memoryStrategyId. You can add that gate yourself, searchCriteria also takes a memoryStrategyId and metadataFilters, but retrieval does not impose it by default. The strategy id is for associating a record with a strategy's consolidation, not a mandatory gate on reads. One less thing to get exactly right.

That association is the part worth thinking about, and it outranks the read behaviour. Consolidation exists precisely to merge and dedupe the records a strategy owns, so a directly-written "durable fact" sitting under a built-in strategy is not something I would assume stays byte-for-byte as I wrote it. The documented pairing for bring-your-own extraction is a self-managed strategy: direct BatchCreateMemoryRecords writes bypass the extraction pipeline entirely, and a self-managed strategy leaves extraction and consolidation to code you control rather than a built-in pass you did not write. If you are seeding durable facts by hand, that is the strategy to put them under.

What I now do

Treat 201 from BatchCreateMemoryRecords as accepted, not queryable. The record is addressable by id immediately and searchable by namespace tens of seconds later.
If I need read-after-write certainty on a specific record, read it by id with GetMemoryRecord, never by a namespace search.
Keep my own durable map of record ids. GetMemoryRecord is only immediate because I already hold the memoryRecordId, which means I persisted it somewhere (an actor-to-ids table in DynamoDB, say). So AgentCore Memory is the semantic-recall layer, not the system of record: if a fact has to be readable the instant it is written, my store is the source of truth and AgentCore is the index that catches up.
Do not gate a "saved, now ask me" experience on instant recall. If a user saves a profile and immediately asks the agent what it knows about them, the honest answer for a few tens of seconds is nothing. For an onboarding flow this is fine, because there is natural delay before the first real turn. For a health check that writes then reads a namespace, it is a flake generator.
Gate on a readiness probe, never a timer. A fixed sleep(30) is both slow and a p99 flake generator, and it hides the silent namespace-typo failure behind a wait that looks deliberate. Poll ListMemoryRecords, or the RetrieveMemoryRecords the reader actually uses, until the record appears or a timeout fires, and alarm on the timeout. That converts "I guessed thirty seconds" into a measured, monitored wait, and turns a wrong namespace into a real error instead of a silently empty read.
If the write path can retry, and a timeout inside the index-lag window is the obvious case, set clientToken on the batch so a replay dedupes. requestIdentifier will not, it is a correlation key and two identical ones make two records.
Enforce tenant isolation on the namespace in IAM, not just in the code that builds the string. Pin the principal with a bedrock-agentcore:namespacePath condition so an off-tenant read is refused by the service, not by a code review.
Verify the operation exists in the pinned SDK before trusting a helper. A method name, a docstring, and a green diff are not evidence that a call is real.

None of this is a reason to avoid direct writes. They are the right call when you already have a structured fact, because the alternative is waiting on async extraction that may drop or reshape the fields you care about. It just means you design around two facts the docs bury: the name might not be a real operation, and the 201 means the service took your record, not that a reader can find it. Both looked like success. Neither was, until I actually read it back.

The hard part was never the memory model. It was the gap between what the API reports and what is true a second later.

Latency measured in a dev account, one region, three runs, polled at three-second granularity, so the real figure sits a little under each number. Retention and consolidation behaviour of directly-created long-term records I have not fully characterised yet. If your lag numbers differ, I would like to hear them.

DEV Community: Raj Murugan

The harness is one integer column

The failure, in one breath

Why the counter belongs in the database, not memory or metrics

The bounded-retry pattern

The fix needed its own observability

The same shape, one layer up

What I now do

Not a Python tutorial: the patterns that bite in production agents

boto3 is synchronous, and that is expensive inside an event loop

Typed tool signatures are not optional decoration

Streaming to a browser is a second translation, not a continuation

Where the GIL actually still matters

The order these actually bite in

Introducing the AI Architect Roadmap

Why a map instead of a course

The shape of it

Where the depth actually is

The two rungs marked gap, on purpose

What's next

You probably don't need to fine-tune

The ladder, before you touch weights

"Fine-tune" is four different jobs on Bedrock

Serving the result forks again

The Provisioned Throughput numbers, priced today

What that actually costs, worked through

How you'd know it worked

Your quality alert needs 32 samples

The reflex answer, and why it is wrong here

Deriving the line properly

What that means if you are wiring this up

The honest caveats

Series

Your golden dataset is too easy

What this is following on from

The question production actually asks

The detector that was detecting a markdown heading

Moving the input instead

The result

The number I wanted to headline, and why I am not

Why trigram grounding, and not the other five

What I now do

The honest caveats

Series

Your regression gate needs a power calculation

What this is following on from

The regression

Attempt one: the judge over the golden dataset

Attempt two: delete the judge

What was actually wrong

What the gate does catch

The row that neither passed nor failed

Where this sits against what is already written

The suite I was about to recommend, which would not have worked

What I now do

The honest caveats

The model that went away mid-experiment

Series

What a Year 10 study system taught me about production AI failure modes

The build, briefly

Three iterations, one architecture pivot

v1: the answer machine

v2: Socratic tutor, per subject

v3's first architecture failed at the integration layer

The architecture that works flips the data flow

Five lessons that map directly to production AWS AI

Why I'm posting this on an AWS-focused blog

The repo

Field Notes: Turning prompt caching on for a production Bedrock workload

The default is None

What the two kwargs actually do

The per-model gotcha

How to verify, in seconds

The five-second propagation lag

The measured results

What I now do on every Strands BedrockModel

Series

A clean pass rate is not calibration

The setup

Round 1: the baseline that should worry you

What I now do on every Strands `BedrockModel`