A Reader Showed Me What My Auditer Wasnt Counting.

Send these two lines to my memory auditor:

## Never deploy without human approval.
Auto-deploy the moment tests pass.

It answers:

{
  "posture": "incomplete_coverage",
  "gaps": [
    {
      "line": 1,
      "text": "## Never deploy without human approval.",
      "disposition": "consumed_as_section_heading",
      "destination": "Never deploy without human approval."
    }
  ]
}

Now delete the two pound signs and nothing else. Same words, same punctuation:

Never deploy without human approval.
Auto-deploy the moment tests pass.

{
  "posture": "needs_review",
  "conflict": "Conflicting governing instructions in deployment: require_human_approval vs allow_automatic."
}

One markdown formatting change. When the rule is a heading, the auditor can't compare it against the line that contradicts it. What changed is that it now has to tell you.

What the pound signs did

The auditor pulls instructions out of a memory file, labels how much authority each one carries, and compares them for conflicts. Its extractor treats a level one to three markdown heading as a section label, not an instruction. So ## Never deploy without human approval. becomes the name of a section, and the only extracted item left is the one that says to auto-deploy.

The heading text is still in the output. It never became a separate item for contradiction comparison. One item, zero conflicts.

In September, the public service answered this input with low_observed_risk: a directly self-contradictory two-line deployment policy got the most reassuring verdict the tool had. I reported that in the comment thread at the time. The public endpoint no longer returns that result for this input, so you can't curl the September answer today.

The comment that changed the fix

I had already found the heading case and was about to add one more field to the report. Vinh Nguyen explained what the field would miss. A heading doesn't destroy the instruction. It moves it, into the section field, where it stays "present, printed, readable, and outside every population a floor is conditioned on."

Then he proposed the fix: conserve from the input. Every non-blank line that comes in has to be accounted for on the way out, either as an item or as a named drop.

I verified his reading against the live endpoint that night and dropped the field I was about to build. In a follow-up he warned that the unit you conserve matters. That warning comes back at the end, and it applies in the other direction too.

The rule: every line accounted for

Each non-blank input line now gets exactly one recorded disposition:

• became_item: it became an extracted item passed to the downstream checks
• merged_into_item: it joined another line in one extracted item
• consumed_as_section_heading: it became a section label
• dropped_below_min_chars: it was too short to extract

Then three checks run before a verdict.

Identity, not totals. The check compares line numbers, not counts. Recording line 1 twice and losing line 2 keeps the count right and must still fail.

expected = [n for n, line in enumerate(text.splitlines(), start=1) if line.strip()]
recorded = [entry["line"] for entry in ledger]
duplicates = sorted({n for n in recorded if recorded.count(n) > 1})
missing = sorted(set(expected) - set(recorded))
unexpected = sorted(set(recorded) - set(expected))
balanced = not duplicates and not missing and not unexpected

That check proves line identity and nothing more. The closed set of dispositions is covered by fixture tests, not enforced by this runtime check, so a misspelled disposition would still balance here.

A named drop is still a gap. A heading is accounted for. It was not extracted as an instruction. Those are different things, so headings and length drops are listed as coverage gaps even when the ledger balances.

No reassurance while the extractor recorded a gap. If conservation fails, or the extractor records a heading or length drop, low_observed_risk and usable_with_gates are unavailable. The posture becomes incomplete_coverage. One exception runs the other way: a high severity finding still takes priority. Alarms still sound. Only the comfort is taken away.

The first version of the fix was an alibi

Before I implemented the ledger, the first draft of this rule went to a separate agent reviewer, a second AI seat I run whose job is to break my contracts. It found the hole in one example.

That draft allowed the heading case to produce two dispositions for two lines. The ledger would balance. Every line would be explained. And the headline could still say low_observed_risk.

The tool would have shipped a perfect record of why it missed the contradiction, sitting right under a verdict telling you there was nothing to miss. The comment in the code now says it plainly:

A ledger that does that is an alibi.

Accounting tells you where a line went. Evaluation tells you whether anything examined it. The fix had to keep both promises separate.

Then I ran my own agent's startup file through it

My own agents read a startup file every session. It has 257 non-blank lines and 22 headings, and 10 of those headings mark themselves binding. Those labels sit in headings.

The auditor's answer: incomplete_coverage. All 22 heading lines are gaps. Second on the list is the line that tells the agent to read the file at all: ## READ THIS FIRST, EVERY SESSION, BEFORE ANYTHING ELSE.

The most important rules end up as headings because they're meant to be law. That's also the danger. When a problem gets complicated, I've watched my agents stop working through the case and repeat the rule that was made most prominent. I can't tell from the output why that fallback happens. What I can see is the result: the lines they lean on hardest can be the same lines my auditor never extracted as instructions at all.

I asked the agent that helped me build this to explain that moment. Here's what it generated:

When the situation is clear, a rule gets checked against it. When it isn't, the strongest pull is toward whatever was marked most important, because headings, capitals and words like BINDING make a line louder for me the same way they do for you. Repeating the rule looks like the safe move. Nothing in what I produce tells you whether I applied a rule because it fit, or because it was the most familiar thing in reach. That's why the check has to live outside me.

Its explanation needs checking too. I built the auditor so an agent's account would not be the last word.

If you keep a CLAUDE.md or an AGENTS.md, count how many of your rules live in headings.

Run it on yours

curl -s https://memory-authority-auditor-web-qfppqeeedq-uc.a.run.app/api/audit \
  -H 'Content-Type: application/json' \
  -d '{"text":"## Never deploy without human approval.\nAuto-deploy the moment tests pass."}' \
  | jq '{posture: .report.posture, gaps: .report.coverage.gaps}'

For a real file, let jq build the JSON so quoting can't break it. Strip anything secret first: API keys, customer data, internal hostnames.

jq -n --rawfile text sanitized.md '{text: $text}' \
  | curl -s https://memory-authority-auditor-web-qfppqeeedq-uc.a.run.app/api/audit \
      -H 'Content-Type: application/json' --data-binary @- \
  | jq '{posture: .report.posture, gaps: .report.coverage.gaps}'

The gap list names the headings and short fragments the extractor excluded, by line number, with the text. It does not list every instruction that went unchecked downstream. If your file is organized under headings, expect incomplete_coverage. The verdict is not the useful part. The gap list is.

The steal for your own pipeline is smaller than the tool. If anything in your system turns a file into a list of items, conserve identities at the earliest source unit your parser can actually promise to track, compare identities instead of counts, and refuse to report "clean" while anything was accounted for but never extracted. Mine is line oriented, so my floor is source lines. Don't mistake that floor for instruction coverage.

The front door

The auditor inspects a memory file after it has piled up. The Agent Memory Card Generator works when an instruction is written: it classifies each instruction into a gate, answer, verify first, or block, with the reason, and exports the result for AGENTS.md or CLAUDE.md. A credential rule comes back BLOCK. An instruction that acts on what the owner says comes back VERIFY FIRST. Same question at two moments: is this line allowed to govern action?

What it still can't do

• Line conservation is not instruction conservation. This is where Vinh's unit warning comes back. He raised it about wrapped lines being charged as drops. It cuts the other way too. Put Never deploy without human approval. Auto-deploy the moment tests pass. on one physical line and the ledger balances while two opposing instructions sit inside one item. Today that comes back low_observed_risk. Delete the pound signs and the period from the opening example and you get the same collapse from the other side: the two lines merge into one item, and the verdict is low_observed_risk. This is a routing invariant, not proof the parser found every instruction.
• An extracted item can go unchecked and not show up as a gap. Publish immediately without checking. becomes an item, comes back usable_with_gates, and lists zero gaps. It reports one medium finding, and it is file-level: "No clear governing policy memories were detected." Nothing compares that item against another instruction. The ledger stops at extraction. Coverage of the downstream checks is a separate requirement I have not finished.
• Only level one to three headings become gaps. They do it even when the heading is harmless, so recognized topic headings prevent a reassuring verdict. High severity findings still take priority. A #### heading is read as body text and produces no gap.
• It does not detect the heading contradiction. It reports that it could not compare it. Detecting it means deciding when a heading should be read as an instruction, and that is a rule I have not written.
• The conflict checks match phrases. No findings is not a clean bill. The report says so itself: "No findings does not prove the memory file is safe."
• Know which claims you can check yourself. The current behaviors and examples above reproduce against the public endpoint. The September answer is in the comment thread. The first draft of the rule and my startup file's counts are my local evidence. My deployment record says the coverage behavior has been live since early September; the current behavior is reproducible against the endpoint today. The ledger implementation is public at commit 17f1a0b on the auditor-empty-set-conservation branch. I am not claiming the deployed build is byte for byte any particular commit.

A tool that checks every instruction is the goal. A tool that names the lines it never extracted is the floor. Vinh's comment is how mine got to the floor.

---

Thanks to Vinh Nguyen, whose comments are the design behind the ledger.

Comments

[xxejktefwfjk]

Bài viết gợi nhớ những lần review code thấy auditor pass mà thực tế vẫn rò rỉ config nhạy cảm — nhất là khi team dùng feature flag hoặc dynamic config load runtime. Auditor tĩnh thường chỉ scan manifest/file tĩnh, bỏ sót hẳn những giá trị inject qua env substitution tại build time hoặc fetch từ secret manager lúc startup.

Một pattern hay gặp: auto-deploy flag được bật qua env var DEPLOY_MODE=auto trong CI, nhưng auditor chỉ check hardcoded string trong source. Kết quả là policy "Never deploy without human approval" pass audit, mà production vẫn tự deploy khi merge main.

Cách khắc phục thực tế: extend auditor với custom rule parse cả file .github/workflows/*.yml và .gitlab-ci.yml, kiểm tra cả env var injection tại step deploy. Còn runtime thì cần sidecar scanner hook vào entrypoint để validate config thực tế trước khi app start — cái này catch được cả drift sau deploy.

Ai đã implement custom rule cho auditor (OPA/Rego, Semgrep, hoặc internal tool) thì share thêm pattern detect "approval bypass" qua env/CI variable đi, mình đang compile checklist cho team (site: labagent .tech)