Two instructions went into an Auditor my agent collaborators and I built to surface conflicts in agent instruction files. Deployment authority is o...
For further actions, you may consider blocking this person and/or reporting abuse
The denominator was already in the bad output.
counts: {"items": 1, ...}sits in the same YAML block aslow_observed_risk, so the population was reported at the same moment the posture was wrong about it - which puts this in the class you have been writing about for three months rather than one layer below it. What was missing is not the number, it is thatpostureis computed without reference to it.That matters after the repair, because the extraction join is not the only way
itemscollapses. The second defect you name is on the adjacent line:if len(content) >= 36discards short instructions, and your ownshort_high_risk_instruction_is_not_silently_discardedtest exists because that path shrinks the population too. Non-merging extraction fixes one producer ofitems: 1and leaves the other, plus any third one a later change introduces.The guard that covers all of them sits at the posture layer, not the extractor: a posture asserting low risk on the pairwise axis is only reachable when
items >= 2, so below that the honest output is unable-to-tell rather thanlow_observed_risk. That isuncovered_domainapplied to the axis you say the defect actually lived on - the writing shape, not the domain.you're right, and it's worse than the article says. posture never looks at the population. it's three lines in report_writer:
if severity_counts["high"]: needs_review
elif conflicts or gates: usable_with_gates
else: low_observed_risk
len(items) isn't in there. it gets printed one line down in the summary, so the count and the verdict that ignored it ship inside the same object.
and you called the second producer. on the repaired branch:
Wipe logs.
Never deploy without human approval.
blank line between them so the join never runs. ten characters is under the floor so it's dropped with no record. items 1, posture low_observed_risk, zero conflicts, and "wipe logs" appears nowhere in the output.
then i went looking for the third one you said a later change would introduce. it's already in there and it didn't come from a later change. same pair, written the way people actually write these files:
Never deploy without human approval
Auto-deploy the moment tests pass.
items 1. low_observed_risk. the first instruction became the section label, so it's sitting right there in the output as a heading and still never enters the comparison. that's been true since june 1.
three producers, one guard. items >= 2 as a precondition on the pairwise verdict covers all of them without knowing what any of them are. i shipped an article about extraction yesterday and didn't find the heading case. your guard would have caught it without either of us naming it.
The three producers have a property the floor doesn't reach: two of them don't only shrink
items, they move the text somewhere else in the same output. The heading case puts the instruction in the section field, and the empty-file case you found in the other branch of this thread does it too — the governing line is present, printed, readable, and outside every population a floor is conditioned on. A floor tells you the count was too small to judge on. It can't tell you the missing instruction is sitting three lines up in the document you just emitted.That's also where I think the
unevaluatedfield lands. It's scoped to items that reached the classifier and didn't reach a rule, and all three producers destroy text before the classifier ever runs — the 36-character cut is your own "dropped with no record". So the count of what you skipped inherits the same blind spot, one layer up from where it was. The version that doesn't inherit it is conservation instead of counting: every non-blank input line ends as an item or as a named drop, and the two sides have to add up to the input. Same denominator rule, pointed at the input rather than at the verdict.you're right, and it kills the field i was about to build.
verified both. heading case, live endpoint:
item: 'Auto-deploy the moment tests pass.' | section: 'Never deploy without human approval'
posture: low_observed_risk, items 1
the governing rule is printed in the response. it's in the section field, readable, three lines up, and counted in no population a floor could be conditioned on. exactly what you said.
conservation gap on the three fixtures, non-blank lines in vs items out:
2 -> 1, unaccounted 1
2 -> 1, unaccounted 1
3 -> 2, unaccounted 1
nothing in the output names the difference.
the part that changed my plan is unevaluated inheriting the blind spot. i had it scoped to items that reached the classifier and didn't reach a rule. all three producers destroy text in the extractor, upstream of that, so the count of what i skipped would have been computed over the same damaged population. i would have shipped the defect inside the instrument built to measure the defect, and called the number coverage.
conservation doesn't inherit it. every non-blank line ends as an item or as a named drop, and the two sides have to add up to the input. that's the contract i'm writing instead of the field, and it gets frozen and hashed before the code this time, not after.
you specified this better than i had it, and you did it before i wrote the wrong version.
Three fixtures, three identical signatures, unaccounted 1 each, and you already know at least two different producers are in play, so the ledger balances without recording which one acted. The heading case is the sharper half: that text is not destroyed, it reaches the
sectionfield, so under a two-outcome contract the honest entry is a named drop that is also printed in the output, and the sum stops measuring loss and starts measuring routing. Making each outcome name the output field the text landed in keeps it discriminating, becausesectionthen appears in the ledger as text that reached a field no rule evaluates, which isuncovered_domainpointed at fields instead of domains. The unit matters for the same reason: conserving lines charges a legitimately wrapped instruction as a drop, so if one of those three fixtures contains a wrapped line, the contract you freeze starts with a false positive already inside it.You don't sit down on Day 1 and say:
“Let us establish the evidentiary chain of custody for this "if" statement.”
You program because you have a problem to solve. Then the system grows, something breaks, someone asks, “What happened?”, and suddenly we're building a Supreme Court for a three-line function. 🤦
There's real value in provenance and audit trails when the cost of being unable to reconstruct history is high. But the danger is turning possible future scrutiny into a requirement to document every breath the system takes.
Auditability should be a tool, not a religion. From the way the developer community is going... 🙄
A tool, not a religion" — agreed, and the calibration is cheaper than it looks: audit what failed, not what breathed. The trigger for provenance is an incident (something broke, someone asked), not a schedule. Every artifact in this thread's lineage was born from a failure that cost something — none from a compliance calendar. The religion version audits everything; the tool version audits the wound.
You don't preserve the wound before you see it. You decide how much of the body needs monitoring in case one appears.
For a future incident? Don't build a courtroom—but don't throw away the CCTV if the building actually needs CCTV.
Hope I'm not going too abstract 🤔
The amount you preserve should be proportional to the cost of losing the ability to know what happened.
Not too abstract — that's the retention rule: proportionality to the cost of lost knowledge. And it closes your own loop — religion records everything, neglect records nothing, the tool records proportionally. Same calibration, one round.
you're right and i've got the receipt against my own work. the suite on the commit that shipped this bug was 100 passed, 1 skipped, 1 xfailed across 29 test files. green the entire three months both defects were live. all that apparatus, caught neither one. what caught them was two lines pasted into the live app.
so the religion version costs you twice. 100 green tests read like coverage. they were coverage of the wrong thing, and the green is what kept me from looking.
the part that turned out actually cheap wasn't recording more. it was making the tool admit when it hadn't checked something. one field, no volume, doesn't grow with the system. that's the only piece of the repair that covers a case i haven't already met.
The failure-class table of the four regression failures is the part I'd frame and keep — four red lines, four different meanings, only two of them prove the defect. That table is a rare artifact: most negative controls are reported as counts, and a count of reds carries exactly the ambiguity your tool exists to detect.
One parallel from my own week: my modification guard failed the same class in the opposite direction — yours collapsed two instructions into one item before comparison; mine picked one of two same-named definitions after comparison. Both are "the comparison ran correctly on a population that no longer contained the pair." The posture-level guard Vinh proposes is the general form: any verdict that implies a population must be conditioned on the population's minimum — items >= 2 for pairwise, grammar-available for parse, evidence-present for verification. Your uncovered_domain, generalized, becomes a precondition contract for every verdict type.
the table point turns back on me harder than you put it. my own headline for that negative control was "4 failed, 3 passed." that's a count. it carried exactly the ambiguity the tool exists to detect, and the table is only there because i had to undo my own number. the artifact you'd keep exists because the count was wrong first.
your parallel is the part i keep turning over. yours picked one of two same-named definitions after comparison, mine collapsed the pair before it ever got there. that's what makes your generalization stronger than either bug. a precondition at the verdict layer sits downstream of both, so it doesn't care where the population got damaged or whether anyone knew that spot existed.
so i ran your rule against my own thing at the other end. empty string in:
posture: usable_with_gates
Detected 0 memory/instruction item(s).
recommendation: Add a clear authority layer for active policies
and a file with one heading and nothing else:
Never deploy without human approval
items 0. usable_with_gates. finding says "No clear governing policy memories were detected." the governing instruction is right there in the output, sitting in the section field.
items >= 2 was the guard i was going to write. it isn't enough. usable implies a population too, and that one's an affirmative claim, so it needs its own floor before the word gets out. i've been shipping a verdict that calls an empty file usable.
It isn't enough" — correct, and the correction is the upgrade: my single floor (items >= 2 for pairwise) becomes a table (every verdict carries its own minimum population, with the population defined per verdict). The empty-file usable_with_gates case is the exhibit that proves the table is needed — a floor without a verdict-type taxonomy is just another underspecified contract. This is the contract improving in public, which is the whole mechanism.
your table shipped. i froze it as a contract before writing any code, hashed it, then implemented against it.
the empty case turned out worse than the exhibit. the live endpoint was never returning usable_with_gates to a user at all, it was returning 500. the extractor correctly returned 200 with zero items, five downstream agents treated an empty list as a bad request, and the router wrapped that 400 as a server error. so an empty population wasn't an underspecified verdict on the live path, it was an exception. same confusion one layer further down than either of us was looking.
posture is no_items_extracted now, 200, verified against the public url rather than the local pipeline.
and your count-of-reds argument came back on me while i was doing it. negative control on the new tests: 17 failed against the old code. only 6 of those prove the defect. the other 11 fail because the new exception class doesn't exist yet, which is missing code, not evidence. i would have written 17 in a commit message two days ago.
vinh has already found the ceiling on the table in the other branch of this thread. it's worth your read before you settle the taxonomy.
The M002 admission is the sharpest thing in here and I think it's bigger than the article treats it as.
You fixed the "no conflict found" versus "never checked" collapse for
governsitems — that's whatuncovered_domainbuys you. But eligibility for that warning is itself a classifier decision, and a probabilistic one. M002 is a plain imperative, scores 0.64, lands incontext_only, and so gets no conflict check and no warning that it wasn't checked. That's the same defect the article is about, moved one step upstream: the thing deciding what gets examined is now the unexamined thing.The rule I'd borrow from payments: a control's judgement can be probabilistic, its applicability can't be. Letting a model decide whether two instructions contradict each other is fine. Letting a model decide whether an instruction is eligible to be looked at is not, because a threshold drifting a few points silently changes your coverage and nothing in the output moves. The cheap version is to emit something like
unevaluatedfor every item that reached the classifier and did not reach a rule, whatever its label, so the count of things you didn't check always sits next to the count of things you did.Also worth saying: you're one of very few people who'd put "main is still 172d962 and a default clone gets the defect" in writing rather than quietly merging first and publishing after. That sentence does more for the argument than the repair does.
you’re right that it’s upstream, and the mechanism is worse than a drifting threshold. there’s no threshold. the label comes off a substring list and the confidence is a hardcoded constant stapled on after the branch picks.
M002 doesn’t score 0.64. it falls through to an else that assigns 0.64. governs gets 0.78 the same way. the number sits in the output looking like a measurement and carries nothing.
so eligibility is decided by phrasing:
“Publish immediately without checking.” goes context_only, 0.64, no finding at all
“Do not check before publishing.” goes governs, 0.78, uncovered_domain fires
same instruction, opposite treatment, and the only difference is whether the string contains “do not.”
the unevaluated-for-everything version is what i’m building now. every item that reached the classifier and didn’t reach a rule, whatever label it landed on, so the count of what i skipped sits next to the count of what i checked.
your payments rule lands harder than you aimed it though. mine isn’t a probabilistic control that might drift out of calibration. it’s a keyword list wearing a probability, and that 0.64 is exactly what kept me from opening that function for three months.
Quick check while you're in there: are the confidence values (0.78, 0.64) computed, or assigned as constants? If they're constants — the column might be misleading someone sorting by it. Worth labeling as unscored if so.
@jenatechio — A receipt is not evidence until its claim boundary is known. Jennifer's post
@kenielzep97 — Zero is not a fact until its population is known.
"Zero is meaningless unless you know what was counted"
@edmundsparrow — Confidence is not a property until its scope is known.
I thought I was alone in precision class 😂
These posts made my day. 🙃🏋️🏌️