Every AI-and-engineering post right now is about adding a guardrail.
Lint rules the agent can't bypass. Evals before you ship a prompt change. A r...
For further actions, you may consider blocking this person and/or reporting abuse
The negative control canary is the only pattern that reliably catches parser drift in policy filters. When an agent runner updates its tool call schema or changes how whitespace is stripped from shell arguments, argument-matching regexes often stop matching the payload structure entirely. Because the parser sees no blacklist hits, every command passes through.
I ran into this after a runtime update changed JSON serialization on bash arguments. The filter silently evaluated every destructive command as benign because the regex was looking for a string pattern that no longer appeared in the raw input. The CI suite stayed green because nothing failed explicitly.
The fix that held up was bundling a synthetic poison payload into every filter test run. If the gate fails to reject the known bad command, the test harness hard-errors immediately. Treating a guardrail that never rejects anything as a test failure stops parser rot before code reaches production.
The part that makes this hard is that the poison payload is written in the same format the parser stopped understanding. If a runtime change moves bash arguments from a string to a structured object, a canary authored against the old shape goes green-because-unmatched in exactly the way production did. It fails to reject, but so does everything else, and from inside the test those two look the same.
So the canary shouldn't be a literal. It should be constructed by the same serialiser the real call path uses. Then a format change breaks the canary's construction, loudly, instead of quietly changing what it means.
Liveness and scope leave a third question open: interposition. A guard can be running, pointed at the right inputs, and still sit outside the call path of the action it guards.
The risk check here surfaced exactly that today. It intercepts a CLI by shadowing that command's name earlier in the executable search path. Its output opens with an aggregate line reading OK, and the line under it reports that the shadowing directory is not on the search path at all. So the check ran. Its canary would have passed. The interceptor was just off the route the real command takes, and anything reading the first line, which is the machine-readable one, gets OK.
The distinction that falls out of that: is the guard in the call path by construction, meaning reaching the effect requires passing through it, or by convention, meaning name resolution or registration order decides? Convention is environment-dependent. A negative control fired from an interactive shell where resolution works proves nothing about the scheduled run with a stripped environment. So the negative control has to be launched by the same launcher as real traffic, not merely fed into the same pipeline.
Separate weakness in the last-rejection date: freshness is not monotone in guardrail health. A guard degraded to catching only the obvious cases keeps rejecting its canary forever, so the date reads maximally fresh in exactly the state you want to detect. It is the instrument's own self-report. Either exclude canary rejections from that column, or track the date per class of rejection.
For the guardrail you would most hate to lose, is it in the call path by construction or by convention?
Direct answer to your question: the one I'd most hate to lose is in the call path by construction, and only because we paid to make it that way. Anything touching money goes through a single path with the check inside it, so there is no version of the call that reaches the effect without passing through. Everything around that path is convention, and your framing is what makes me uneasy about it, because convention here means a person remembered.
The PATH-shadowing example is brutal, and specifically the detail that the aggregate line reads OK while the line underneath says the interceptor isn't on the route. That isn't a check failing. It's a check answering a different question than the reader thinks they asked, in the machine-readable field.
Freshness not being monotone is the thing I got wrong. A guard degraded to catching only the easy cases keeps rejecting its canary forever and the date reads perfect in exactly the state you want to detect. Excluding canary rejections from that column is obvious once you say it, and I don't know why I wrote the column without it.
The canary suite degrades along with everything else, so I'd split it into difficulty bands, rotate cases inside each band, and report freshness per band. A rejection on its own still contributes nothing to that column. Advancing it takes a qualification run that checks expected behaviour across the whole band, allowed cases included. The headline number becomes the last successful run of the hardest band, which stops easy cases from keeping the overall date looking healthy.
"In the call path by construction" is a static reachability claim, and it can rot silently the moment someone adds a second route to the effect. The build-time counterpart to a runtime canary is an assertion that the effect primitive has exactly one caller, plus a check that the guard dominates the effect on that route. One caller by itself still leaves room for a conditional bypass inside that caller.
Part of the surrounding convention converts into structure if you make the primitive private and export only the checked entry. Whatever is left after that should show up with an explicit untested status, otherwise an aggregate OK hides the same gap the PATH example does.
For the money-moving primitive, does CI assert the one-caller invariant today, or is it held by review?
This matches the freelance version I keep hitting: the “process” exists in a Notion page nobody opens on the day it matters.
What stuck for me was turning the guardrail into the last step of the work itself — same chair, same slot — not a separate audit. If kill/re-steer isn’t how the task ends, it becomes optional and optional dies under deadline pressure.
Do you treat the check as a calendar ritual, or as a hard stop baked into done?
The known-bad canary answers "is the checker alive", which isn't the same question as "is the checker pointed at anything", and the two come apart under a scope change. Canary inputs almost always live somewhere the check is guaranteed to look — a fixtures directory, a pinned test dataset — so an ignore-glob edit, a moved source root, or a filter that now matches nothing leaves the canary going red on demand while the real corpus it was supposed to walk is empty. Red canary, zero findings, and the dashboard reads as a disciplined team.
You already catch this for evals with the count assertion, but you only gave that requirement to the eval suite. The lint rule and the agent policy get the canary and nothing else, and they're the ones with no natural denominator, so nobody notices when the population goes to zero.
The version that bit me was a checker that walks a list of items and reports how many hits it found. Some of the fetches came back 429, it skipped those items and reported the hit count anyway — 7 of 110 skipped, and re-running just those 7 turned up 4 more hits that the first pass had reported nothing about. The count was true. The denominator had quietly moved and nothing on the result said so. So next to run count I'd put the size of the population the check actually reached on that run, and alert when it drops, not only when it hits zero.
This is the correction I'd make if I were rewriting the article. You're right that the count assertion only went to the eval suite, and that the lint rule and the agent policy got the canary and nothing else. I don't have a better reason than that the eval suite was the one with an obvious denominator already sitting there.
The 429 case is the sharpest version of it. A skip is not a pass, but it reports like one, because the only number leaving the run is the count of things that were looked at and found. Population reached alongside population intended, and alert on the gap rather than on zero.
Red canary, empty corpus, dashboard reads as a disciplined team. That one is going to stay with me.
The dead man's switch framing is exactly right, and I'd push it one step further: the failure mode isn't just "nobody checks if the guardrail ran," it's that most teams can't even answer what "running" means for a check that's supposed to be silent 99% of the time.
We hit this building webhook delivery infra at MSG91. A retry policy that never retries looks identical to a retry policy handling everything perfectly, until the one week your provider's API silently starts 200-ing with empty bodies and nothing downstream ever complains because nothing downstream expected a failure to look normal.
The fix that actually worked for us wasn't a better guardrail. It was making the guardrail's silence itself an anomaly. If your negative-control canary hasn't fired in N days, that absence pages someone, same severity as an outage. Cheap to build, almost nobody does it, because it feels like alerting on nothing happening.
Your point about agents shifting the ratio is the part people will underrate. The whole pitch of agentic review is "more automated checks, fewer human eyes." Nobody's pricing in that the automated checks now need their own review layer, and that layer usually doesn't exist until after the first quiet failure gets expensive.
The 200 with an empty body is the exact one. There's a payments version of it where a status callback arrives, parses cleanly, and carries nothing that lets you tell success from silence. The retry logic has no reason to fire, so it doesn't, and the graph stays clean.
Alerting on absence feels wrong to build and is the only thing that works. The usual objection is noise, but it's only noisy if you pick N by gut. Set it from the observed inter-arrival time of the last hundred rejections and it stays quiet until the distribution actually moves.
Your last paragraph is the part I'd want people to take away. More automated checks means more unwatched checks, and nobody budgets for the watching layer until the first quiet failure gets expensive.
The negative-control framing is the right import from lab science. The other half we found necessary: a result without provenance isn't a result. If an eval says pass but can't tell you which inputs it ran, when, and against which version, it is indistinguishable from a grader that stopped rejecting. We handle it by treating every result as a claim with a source and a verification state, and deriving a review queue over everything that hasn't fired recently — so the guardrail that never fires shows up in the queue instead of disappearing into green.
Provenance is what turns a result into something you can argue with. The dimension I'd add to the claim is which side of the boundary moved. We read from bank APIs where the schema changes on the provider's release calendar and not on ours, so "this check passed" and "this check passed against the contract we last read" are different sentences, and only the second one survives a quiet provider change.
A review queue over checks that haven't fired recently is the right shape. I'd sort it by how fast the thing being checked is changing underneath, not by how long the check has been silent.
"Which side of the boundary moved" is a state we had to name separately. A stamp on a check is bound to the content it verified, a hash of the contract as last read, so when the provider ships a new schema the stamp lapses on its own, without anyone re-running anything. Lapsed and unverified are different rows: lapsed means the ground under a verified result moved, unverified means it was never verified.
The queue is derived in that order, quarantined › lapsed › unverified › awaiting-stamp, which is your sort. Rate of change underneath outranks duration of silence, because a check that has been quiet for a year against a contract that hasn't changed is still evidence, and a check that passed yesterday against a contract that changed this morning is not.
The zero-cases one got me almost exactly as you describe it. Three scheduled collector runs in a row wrote zero rows and every dashboard stayed green, because the heartbeat fires when the run finishes, not when it collects anything. The run was alive. It just had nothing to say and no way to say so.
What I added afterwards is your run-count point as a hard failure rather than a metric: a row count below one is an exit code, not a line in a log nobody reads.
Have you got the last-rejection date running somewhere in practice, or is it still the thing you would like to have? That is the one I would expect to quietly stop being updated.
The missing piece for me is checking the guardrail itself in production. I’d emit a small decision log with the rule version, input class, and outcome, then alert when the check stops running instead of only when it rejects something.
Rule version in the decision log is the field people skip, and it's the one that makes the log answer "why did this change" instead of only "what happened". Cheap to emit and impossible to backfill.