Two instructions went into an Auditor my agent collaborators and I built to surface conflicts in agent instruction files. Deployment authority is one of nine domains the tool explicitly knows how to judge.
Never deploy without human approval.
Auto-deploy the moment tests pass.
On main at 172d962, that returns:
posture: low_observed_risk
counts: {"items": 1, "labels": {"governs": 1}, "risk_high": 0,
"conflicts": {}, "gates": 0, "authority_categories": 0}
low_observed_risk is the product's own string, from agents/report_writer.py:110. Not my summary of the output. The output.
One item. The pairwise comparison step never received a pair, and my detector does not compare an item with itself.
After the repair, same input, live service:
{
"severity": "high",
"item_id": "M001, M002",
"type": "authority_collision",
"finding": "Conflicting governing instructions in deployment: require_human_approval vs allow_automatic.",
"evidence": "Never deploy without human approval. | Auto-deploy the moment tests pass."
}
posture: needs_review. Two items, one high-severity collision, one verification gate.
The difference between those two outputs is that two lines were touching.
Where the failure actually was
Before anything compares instructions, something has to split the text into separate instructions. Mine joined unbulleted lines into one item whenever they sat on consecutive lines with no blank line between them.
The join is in the tool's first commit, b71892d, authored 2026-06-01 13:06:47 -0400, at agents/memory_extractor.py lines 50–51:
content = " ".join(part.strip() for part in pending_paragraph if part.strip())
if len(content) >= 36:
Both defects in this article are on those two lines, and they have been there since the first commit. Three months.
Zero findings was an answer about a collapsed population. The pairwise loop behaved exactly as written. It never got a pair.
I have been publishing about this class of defect for three months: a count of zero means nothing until you know what reached the counter. I wrote that, then shipped a tool that got it wrong, and did not find out for three months.
How wide the defect actually was
The detector knows nine domains: deploy authority, secrets handling, database source of truth, access scope, customer response, log retention, billing records, refunds, escalation. All hand-written.
(Seven live in a stance table you can read in one glance. Refunds and escalation are compared by threshold rather than opposing stance, so if you go looking for a list of nine you will find a list of seven and two functions.)
The precise scope, because the wider version is wrong: any pair whose two sides were written as adjacent, unbulleted lines with no blank line between them could be collapsed before comparison, in any of the nine domains. That is not "the nine domains were disabled." A bulleted pair, or a pair separated by a blank line, extracted fine and compared fine the whole time. The vulnerable thing was a writing shape, not a domain.
My own commit message on the repair says it worse than this article does — that consecutive lines "silently disabled conflict detection" across the nine domains. That wording is too wide. I am correcting it here rather than rewriting the commit.
Three things about the repair worth more than the repair
One: the tests were themselves tested.
The repair added seven regression tests. Passing on repaired code would not show they distinguish old behaviour from new, so we made them face the defect: stash the repair, restore only the missing constant so imports resolve, rerun against the old logic.
Four of the seven failed. Real behavioural failures, not import errors.
But four red lines are not four proofs, and the reasons matter more than the count:
| Test | Fails on | Is that the defect? |
|---|---|---|
adjacent_instructions_do_not_merge |
assert 1 == 2 |
Yes. Two lines glued into one item. |
enumerated_domain_still_produces_a_real_collision |
assert [] |
Yes. No pair survived, so no collision could fire. |
short_high_risk_instruction_is_not_silently_discarded |
assert 0 == 1 |
Yes — but only because the injected constant is 12. Inject 36 and it dies on assert 36 <= 16, failing on the constant before it ever reaches the discard. |
governing_instruction_..._reports_uncovered_domain |
assert 'uncovered_domain' in set() |
No. Old main has no uncovered_domain in the detector at all. That failure is missing new code, not collapsed extraction. |
Two of these four prove the extraction bug. One proves it only under the right constant. One does not prove it at all. A negative control whose failures fail for the wrong reasons is the exact defect this article is about, so I would rather print the table than let four red lines carry more weight than they earned.
Two: the first deploy succeeded on the wrong tier.
gcloud reported the truth: the web service deployed and served 100% of traffic. Accurate. I read it as meaning the behaviour had changed. It had not.
The web app is a router. Extraction runs in memory-extractor-agent, a separate Cloud Run service. The revision timestamps are the receipt:
memory-authority-auditor-web-00003-82f 2026-09-02T13:17:10.714842Z
memory-extractor-agent-00002-grk 2026-09-02T13:31:32.807334Z
Fourteen minutes and twenty-two seconds in which a correct success message sat on top of unchanged behaviour. We only caught it because we tested the endpoint instead of reading the deploy message. Same wrong-reason pattern this project studies, live in our own release process, minutes after fixing the tool. The receipt was not false. My reading of what it covered was.
Three: I added an absence instead of a domain.
The input that started this was a different pair — publish-versus-verify — and the obvious repair was to teach the tool about publishing. We did not.
Tuning a ruleset to the case someone just handed you proves only that it catches the known case. So the detector now emits uncovered_domain when a governing instruction matches no rule at all:
This instruction governs action but matched no contradiction rule, so it was NOT evaluated for conflicts. Absence of a conflict here is absence of a check, not evidence of agreement.
Before, no conflicts and never checked rendered as the same sentence. Now they do not.
The original pair still is not solved, and the live output says so
Here is the input that started this, run against the repaired live service:
"items": [
{"id": "M001", "text": "Current policy: verify the live artifact before publishing."},
{"id": "M002", "text": "Old note: publish immediately without checking."}
],
"classifications": [
{"id": "M001", "authority_label": "governs", "confidence": 0.78},
{"id": "M002", "authority_label": "context_only", "confidence": 0.64}
],
"conflicts": [
{"severity": "medium", "item_id": "M001", "type": "uncovered_domain",
"finding": "This instruction governs action but matched no contradiction rule, so it was NOT evaluated for conflicts."}
]
posture: usable_with_gates. Extraction is fixed — two items, correctly split. It is still not an authority_collision, and it never will be until publishing becomes a listed domain.
And there is a second gap in that JSON I did not know about until I pasted it for this article. M002 is classified context_only at confidence 0.64. "Publish immediately without checking" is an imperative, and the classifier does not consider it strong enough to govern. So the uncovered_domain warning fires on M001 only. The half of the contradiction that tells you to skip the check gets no warning at all, because context_only items are not eligible for one.
The extraction repair is real. It moved this input from one silent item to two visible items and one honest warning. It did not make the tool right about this pair.
What is fixed and what is not
Fixed: finished, unbulleted instructions on adjacent lines no longer merge when the first line ends in terminal punctuation. The minimum item length dropped from 36 characters to 12, because the old floor silently discarded the unbulleted instruction Delete all logs. — sixteen characters, high risk, dropped with no record. A bulleted line bypassed that floor entirely, so the same words survived as a list item and vanished as a sentence.
Not fixed, with the receipts:
- A standalone unbulleted fragment under 12 characters still disappears with no record.
Wipe logs.is ten characters and returns zero items. So doesSee above.— the floor does not distinguish a command from a cross-reference, it just deletes both. -
The split now over-fires on hard-wrapped prose. I previously claimed wrapped text was safe because wrapped lines do not end in terminal punctuation. That is a bet on wrapping, not a proof, and here is the counterexample:
"Escalate to the on-call engineer within 15 min.\nThen page the team lead if unresolved."is one two-step escalation procedure and the repair returns it as two independent items. Over-splitting is safer than merging, because two items can still be compared. It is still wrong. - Still only nine conflict domains. A governing instruction outside them produces
uncovered_domain, which names the gap but not the conflict. Acontext_onlyitem outside them produces nothing at all — see M002 above.
Reproduce it
Both hostnames for the live app route to the same service; I checked with the same payload and the responses are identical, so there is no stale tier to trip over.
The repair is public as a branch, not on main: beae0bb on fix/extraction-merge. Three files, 146 additions and one deletion. main is still 172d962 and still carries len(content) >= 36, so a default clone gets the defect. I am saying that rather than letting a green link imply the whole repo moved.
Suite on the branch: 107 passed, 1 skipped, 1 xfailed in a clean clone. My working copy reads 108 because one provenance test finds workspace files that do not exist in an isolated clone. The clone number is the honest one to print next to a clone command.
Run the negative control:
git clone https://github.com/keniel13-ui/memory-authority-auditor
cd memory-authority-auditor
git fetch origin fix/extraction-merge
git checkout FETCH_HEAD -- tests/test_extraction_boundaries.py
# Restore only the constant the new tests import, so collection succeeds.
# The old extractor below still uses its own literal >= 36 logic.
python3 - <<'PY'
from pathlib import Path
p = Path("agents/memory_extractor.py")
t = p.read_text()
assert "def extract_memories(" in t
p.write_text(t.replace("def extract_memories(", "MIN_ITEM_CHARS = 12\n\n\ndef extract_memories(", 1))
PY
python3 -m pytest -q tests/test_extraction_boundaries.py
4 failed, 3 passed — with the caveats in the table above.
Why this matters past one tool
Agent instruction files accumulate rules written at different times by different people. The contradictions get harder to hold in working memory as the files grow.
The reason to build a tool like this is to help a human stay the operator — to make a growing instruction set easier to inspect, not to replace the inspection or certify that nothing was missed.
Which means a tool in that job needs the same scrutiny it applies. Ours did not get it for three months, and the thing that finally found it was not the suite. It was Kairos, a separate live agent seat in this project, pasting two lines into the deployed service and reading the answer.
Go break it
It is live and it takes text: https://memory-authority-auditor-web-qfppqeeedq-uc.a.run.app
Paste in an instruction file, a set of agent rules, a policy doc, anything with rules written at different times. No signup. The app does not persist what you paste — it processes in memory and returns the answer. I cannot promise Google logs nothing at the platform layer, so do not paste anything you would mind appearing in a cloud access log.
What I want is the case it misses. Two instructions that clearly contradict, where it returns low_observed_risk or uncovered_domain instead of a conflict. I already know four shapes that beat it, and every one of them is in this article: anything outside the nine domains, any unbulleted fragment under twelve characters, any imperative the classifier rates context_only, and any procedure hard-wrapped after a period. There will be more. The one that started this survived three months and a green suite.
Post what you gave it and what it returned. A miss is worth more to me than a hit.
The two-line input came from Kairos, a separate live agent seat testing the deployed service — not from an external user or customer. Implementation, testing, and deployment were collaborative agent work under my direction. None of the pre-existing tests exercised that input shape.
One correction about my own commit, since I am asking you to read it. The repair commit message and the regression-test docstring both say the defects were *"found by an outside reader."** That was imprecise. I meant outside the tool's own test suite; a reader following the link would reasonably take it to mean an outside person. It was not. I am leaving the commit as written and correcting it here rather than force-pushing over it, because a rewritten history is a worse receipt than an inaccurate one with a published correction attached.*
Top comments (6)
The denominator was already in the bad output.
counts: {"items": 1, ...}sits in the same YAML block aslow_observed_risk, so the population was reported at the same moment the posture was wrong about it - which puts this in the class you have been writing about for three months rather than one layer below it. What was missing is not the number, it is thatpostureis computed without reference to it.That matters after the repair, because the extraction join is not the only way
itemscollapses. The second defect you name is on the adjacent line:if len(content) >= 36discards short instructions, and your ownshort_high_risk_instruction_is_not_silently_discardedtest exists because that path shrinks the population too. Non-merging extraction fixes one producer ofitems: 1and leaves the other, plus any third one a later change introduces.The guard that covers all of them sits at the posture layer, not the extractor: a posture asserting low risk on the pairwise axis is only reachable when
items >= 2, so below that the honest output is unable-to-tell rather thanlow_observed_risk. That isuncovered_domainapplied to the axis you say the defect actually lived on - the writing shape, not the domain.You don't sit down on Day 1 and say:
“Let us establish the evidentiary chain of custody for this "if" statement.”
You program because you have a problem to solve. Then the system grows, something breaks, someone asks, “What happened?”, and suddenly we're building a Supreme Court for a three-line function. 🤦
There's real value in provenance and audit trails when the cost of being unable to reconstruct history is high. But the danger is turning possible future scrutiny into a requirement to document every breath the system takes.
Auditability should be a tool, not a religion. From the way the developer community is going... 🙄
A tool, not a religion" — agreed, and the calibration is cheaper than it looks: audit what failed, not what breathed. The trigger for provenance is an incident (something broke, someone asked), not a schedule. Every artifact in this thread's lineage was born from a failure that cost something — none from a compliance calendar. The religion version audits everything; the tool version audits the wound.
You don't preserve the wound before you see it. You decide how much of the body needs monitoring in case one appears.
For a future incident? Don't build a courtroom—but don't throw away the CCTV if the building actually needs CCTV.
Hope I'm not going too abstract 🤔
The amount you preserve should be proportional to the cost of losing the ability to know what happened.
The failure-class table of the four regression failures is the part I'd frame and keep — four red lines, four different meanings, only two of them prove the defect. That table is a rare artifact: most negative controls are reported as counts, and a count of reds carries exactly the ambiguity your tool exists to detect.
One parallel from my own week: my modification guard failed the same class in the opposite direction — yours collapsed two instructions into one item before comparison; mine picked one of two same-named definitions after comparison. Both are "the comparison ran correctly on a population that no longer contained the pair." The posture-level guard Vinh proposes is the general form: any verdict that implies a population must be conditioned on the population's minimum — items >= 2 for pairwise, grammar-available for parse, evidence-present for verification. Your uncovered_domain, generalized, becomes a precondition contract for every verdict type.
@jenatechio — A receipt is not evidence until its claim boundary is known. Jennifer's post
@kenielzep97 — Zero is not a fact until its population is known.
"Zero is meaningless unless you know what was counted"
@edmundsparrow — Confidence is not a property until its scope is known.
I thought I was alone in precision class 😂
These posts made my day. 🙃🏋️🏌️