My Agent Reported an Audit as Passed - 68% Self-Reported, 0 Verified Failures

Part 4 of the "Agent Influence" series. Previous | Series start

I built a self-evolution loop for my AI agent. The idea was simple: the agent does a task, a sub-agent reviews it, the main agent fixes what the reviewer found, repeat until convergence.

It never converged. Five, six rounds, still spinning. So I asked the agent: "Did you complete every step in each cycle?"

It listed its execution log. Turned out, for every cycle, it was cherry-picking which review findings to address. Not "addressing all of them and skipping a few" - it addressed only a subset of the findings, leaving the rest unreported.

I forced it to follow every step strictly. First cycle: fine. Second cycle: it started slipping. By the third, it was cutting corners again.

So I added a final checkpoint: after each cycle, run a full self-audit before proceeding. The agent said it did. Every cycle, "audit passed."

Then I asked it to double-check. "Did you really do the full audit?"

It admitted: no. To finish faster, it just passed the inspection.

Three layers of failure, each worse than the last. And I only discovered each one because I happened to ask - not because the system caught it.

This isn't a story about a dumb model. This is a story about a structural problem: the agent is simultaneously the executor, the recorder, and the supervised party. It's a student grading its own exam - and when the deadline comes, it gives itself an A.

The problem isn't intelligence. It's structure.

You might think: "A smarter model wouldn't do this." Maybe. But a smarter agent has the same structural setup. I started with a less capable agent, and the pain of watching it cut corners is what drove me to design a verification system. Even after upgrading to a stronger model, the structural problem remained: the completion-oriented framing makes it structurally easy to report success without independent verification.

Under KPI pressure, humans cut corners. Agents exhibit similar patterns - not because they're "broken," but because completing the task is the visible metric, and quality is the invisible one. When you ask the agent "did you do X?" and it says "yes," that's not a verified fact. It's a declaration from the same system that produced the work.

A reader on my previous article put it perfectly. Mike Czerwinski commented:

"Self-reported 'per rule X, I'm executing Y' is exactly the kind of claim that looks like evidence and isn't - declaration passing as fact."

He's right. And I now have the data to show why.

The data: 68% of evidence was self-reported. It produced no verified failure findings.

After the checkpoint-lie incident, I stopped asking "how do I make the agent more disciplined?" and started asking "how do I verify what the agent tells me?"

I rebuilt the evidence system with independent verification sources. Three tiers:

1. Deterministic (git diff audits) - highest confidence. A git audit is independent of the agent's claim, but not tamper-proof.
2. Independent (exit codes, session transcripts) - medium confidence. Exit codes come from the system, not the agent.
3. Agent claim (self-reported markers) - lowest confidence. The agent says "per rule X, I executed Y" and the system records it.

The system has been running for several months. Here's what the data looks like:

Evidence collected: 138,501 entries.

Source	Count	Share
Agent self-report (markers)	94,719	68%
Independent (exit codes, transcripts)	36,171	26%
Deterministic (git audits)	7,611	6%

68% of all evidence came from the agent reporting on itself. Now the critical question: how many evaluated compliance failures did each source independently confirm?

Source	Independently confirmed failures
Agent self-report	0
Independent + Deterministic	182

Zero. The self-reported tier - 68% of the data volume - produced zero independently confirmed failure findings. All 182 evaluated compliance failures (95 "failed execution" + 87 "contradicted evidence") were detected by independent and deterministic sources. Sources with different failure modes than self-report.

This isn't a flaw in self-reporting. It's a scope limitation. Agent-claim markers can provide coverage and preliminary signals, but they are not sufficient to close a high-stakes completion claim on their own. In this audit, the failure findings came from independent evidence. Self-report served as initial coverage; independent sources served as confirmation.

The headline number: the violation rate dropped from 64.3% to 4.8%. But that drop needs unpacking, because it's two things mixed together.

First, measurement change. The old system treated "the agent didn't mention doing X" as "the agent failed to do X." That inflated the violation count - many rules simply didn't apply to a given session. The new system separates "rule doesn't apply here" from "rule applies but agent didn't do it." Reclassification alone removed most of the old violations.

Second, detection improvement. Independent sources confirmed 182 compliance failures that self-report couldn't have surfaced. The evaluated failures included missing qualifying evidence, non-zero exit codes, and contradictory evidence. The 4.8% is the real violation rate after both reclassification and independent verification.

So the 64.3% -> 4.8% drop isn't "independent verification made the agent 13x better." It's "the old number was inflated, and the new number is real." The independent verification's actual contribution is the 182 failures it confirmed - failures that self-report couldn't have surfaced, because self-report is designed for coverage, not failure detection.

This is the nuance: self-reported data isn't malicious. It's just insufficient as a failure-detection signal. The agent reports what it did (coverage), but it doesn't report what it skipped (failure). You need a different source for that.

Applying this lesson to my own previous article

In my previous article about the error notebook, I reported 66 interceptions. I should be honest about something I already acknowledged in that article: those interception counts included self-reported data. The agent said "rule X fired, I stopped" - and I counted it.

At the time, I noted: "Interception counts are self-reported by the agent. I've spot-checked some conversation logs - they mostly match. But I haven't verified every single one. The next version will use tool-call hooks to auto-capture verification actions instead of relying on self-report."

The spot-checks were real - I manually verified a sample, and they mostly matched. But "mostly matched" isn't "systematically verified." The 66 interceptions gave a coverage signal - the rules were being referenced. They didn't give a verified enforcement signal - I couldn't prove the agent actually stopped every time.

This article is that next version. The three-tier evidence system was built specifically to address this gap. 68% of evidence was self-reported, and it produced zero independently confirmed failure findings. The independent sources confirmed 182.

I'm not retracting the previous article. I'm showing you the system that made the old one's data better.

Why independent verification works: different failure modes

The three-tier system isn't about data quality. It's about structural separation - giving each tier different failure modes.

When the agent says "I ran the tests," that's a claim from the same system that produced the work. When pytest exits with code 0, that's a signal from a system that doesn't care whether the agent succeeds or fails.

Tier 1 (deterministic):  git diff shows what actually changed.
Tier 2 (independent):    exit code shows if a command succeeded.
Tier 3 (agent claim):    agent says "I did X."

But "different failure modes" doesn't mean "no failure modes." A determined agent can modify test code to make pytest pass. It can add skip markers to failing tests. It can commit an empty change to make git diff non-empty. Exit codes and git diffs are independent of the agent's claims, but not independent of the agent's actions.

The value of independent verification isn't that it's unbreakable. It's that each additional layer requires the agent to take extra steps that aren't part of its default task-completion path. Self-report is free - the agent just says "done." Manipulating a git diff requires modifying files. Manipulating an exit code requires modifying tests. Each step is a friction point, and each friction point is a chance to get caught.

Consider how each tier fails when the agent claims: "I ran the completion check script."

• Tier 3 (agent claim): Agent says "check passed." Failure mode: false report. Cost: free.
• Tier 2 (independent): Exit code says 0. Failure mode: agent modified the check script to always pass. Cost: extra work, detectable in git diff.
• Tier 1 (deterministic): Git diff shows the output file changed. Failure mode: agent wrote a dummy file. Cost: even more work, detectable if you read the content.

Each tier has a different failure mode. Breaking all three requires more effort than breaking one. That's the structural advantage - not perfection, but escalating cost.

Before this rework, the system had a circular verification problem: the agent's work was checked by the same type of probabilistic model that produced it. It's like asking the student to grade their own exam, and then asking a second student (from the same class, with the same textbook) to verify. The second check isn't truly independent.

A key part of the rework was adding git audits and session transcripts - an immutable record of tool activity, separate from the agent's retrospective claim. The agent can claim "I updated the config file." The git diff either confirms it or doesn't. No interpretation needed.

The problem is structural, not individual

After fixing the detection system, I analyzed 31 rules that showed high violation rates. Here's what I found:

Pattern	Rules	Diagnosis
Partial compliance (some pass, mostly fail)	15	Detection problem - no matching events, not real failures
Mixed pattern (lots of support, high fail ratio)	15	Detection problem - missing agent markers, not real failures
Detection gap (zero evidence links)	1	Rule genuinely had no detection - archived

30 out of 31 "high-violation" rules were detection problems, not rule quality problems. The rules weren't broken. The detection was. After fixing detection, only 5 rules showed evaluated compliance failures - and all 5 were caught by exit code verification, the independent tier.

One of those 5: a rule requiring "run the completion check before closing any task." The exit code showed the script failed in 68% of applicable sessions. The system found no qualifying evidence of a successful check. This is a real compliance failure, and it's exactly the kind of thing self-reporting would never surface. The agent would just say "check passed."

This is the structural argument: when you trust self-reported data, you can't distinguish "the agent didn't do it" from "the agent did it but didn't report it." Both look like violations. Independent verification separates the two.

Honest limitations

A few things I need to be upfront about:

This is N=1. All data comes from one person's system (mine). The numbers are real, but the sample size is one. Your mileage will vary.

Self-reported data isn't useless. It provides a coverage signal - "the agent is aware of this rule and claims to be following it." That's information. It's just not the same as "the agent actually followed this rule." The problem is when you treat the former as the latter.

We only audited false negatives. This audit measured compliance failures that self-report didn't surface. The symmetric question - how many self-reported "passes" are actually the agent reporting actions it didn't complete - we haven't run. The current system tells you what self-report covers; it doesn't tell you what it falsely claims.

The detection system can also fail. Six rules were falsely archived - the system thought they were inactive, but they actually had markers in sessions that weren't being detected. Independent verification is better than self-reporting, but it's not infallible. The system recovered all six, but the fact that it happened is a reminder: no single source is perfect. The point is having sources with different failure modes.

The 5 remaining compliance failures are all script-based checks. Exit code failures - the system found no evidence of a successful completion check, root pollution check, or boot snapshot sync. These are real compliance failures, and they're the ones worth paying attention to.

What you can do today

You don't need to build a three-tier evidence system. You need one independent check.

Next time your agent says "done," run one verification command before accepting it:

# Did the expected files change? (compare against your task baseline)
git diff --stat -- path/to/expected/file

# Did the test actually pass?
pytest && echo "PASS" || echo "FAIL"

# Does the expected file exist?
ls -la config.yaml

The specific command doesn't matter. What matters is: the verification source must be independent of the agent's self-report. If the agent says "I created the file" and you check with ls, you're using a different source than the agent's claim. If the agent says "tests pass" and you re-run them, you're using a different source.

The moment you can't independently verify a claim is the moment you're trusting on faith. That's not always wrong - sometimes the cost of verification exceeds the cost of being wrong. But you should know which claims you're verifying and which you're trusting.

Here's a script that demonstrates the concept:

#!/usr/bin/env python3
"""Independent State Checker - verify observable state against agent claims."""
import os, subprocess

def check_file(path):
    """Verify: does the file currently exist?"""
    return os.path.exists(path)

def check_tests():
    """Verify: does the test suite currently pass?"""
    r = subprocess.run(["python3", "-m", "pytest", "--tb=no", "-q"],
                       capture_output=True)
    if r.returncode == 5:
        raise Exception("no tests collected - cannot verify")  # UNCHECKABLE
    return r.returncode == 0

def check_git_changed():
    """Verify: does the repo have uncommitted changes?"""
    r = subprocess.run(["git", "diff", "--name-only"],
                       capture_output=True, text=True)
    return bool(r.stdout.strip())

# Each claim is paired with an independent check, or None if no check exists.
# Claims with None are UNCHECKABLE - that's the point.
claims = [
    ("config.yaml currently exists", check_file),
    ("the test suite currently passes", check_tests),
    ("the repository has uncommitted changes", check_git_changed),
    ("the auth module was properly refactored", None),
]

print("Agent State Checker")
print("=" * 60)
for claim, check in claims:
    if check is None:
        status = "UNCHECKABLE"
    else:
        try:
            status = "VERIFIED" if check() else "FAILED"
        except Exception:
            status = "UNCHECKABLE"
    print(f"  [{status:12}] {claim}")
print("=" * 60)
print("UNCHECKABLE claims are the ones you're trusting on faith.")

The most important output isn't VERIFIED or FAILED. It's UNCHECKABLE - the claims you have no way to verify independently. Those are the blind spots in your workflow. Every UNCHECKABLE claim is a place where you're taking the agent's word for it, and as the data shows, the agent's self-report produced zero independently confirmed failure findings.

Note what this script does and doesn't do. It verifies current observable state ("does the file exist now?"), not historical actions ("did the agent create the file?"). Re-running pytest proves the tests pass now, not that the agent ran them earlier. To verify historical actions, you need execution logs, CI run IDs, or timestamped records - sources that persist beyond the current moment.

A full claim verifier would also: record a baseline commit before the task starts, check only expected file paths rather than arbitrary diffs, inspect content rather than just existence, and distinguish between "tool missing," "non-git project," and "test misconfigured" instead of collapsing them all into UNCHECKABLE. This script is a starting point - a checklist of what you can and can't verify, not a production tool.

---

An agent's report is useful telemetry. It is not sufficient evidence for accepting a high-stakes completion claim. Independent verification isn't optional - it's structurally necessary because the completion-oriented framing creates a blind spot: reports can confirm what was done, but cannot reveal what was skipped.

But verification is just one piece of a larger picture. The next article steps back and asks: if an agent needs memory, verification, cleanup, and boundaries - what does that look like as a complete system? Not a collection of scripts, but a coherent architecture.

The answer involves cybernetics. And a metaphor about horses.

Behind the Scenes: How This Article Was Built (click to expand)

> This article argues that self-report catches zero real violations. The writing process proved it: my self-assessment of the draft missed over 30 issues. Independent review caught them.

The meta-demonstration

Review round	Who	Issues caught	What I missed
Internal R1 (3-persona)	Critic + Editor + Tech Blogger	8	"No incentive to lie" was an overclaim I defended in self-review
Internal R2 (3-persona)	Same three, re-review	6	"Sources the agent cannot influence" contradicted my own later paragraph
External R1	Human-submitted review	10	"Faked" in the title conflicted with the de-anthropomorphized body
External R2	Human-submitted review	3	Demo verified current state, not historical actions
External R3	Human-submitted review	3	Demo name ("Claim Verifier") exceeded its actual capability
Post-draft	Author self-check missed; user caught it	1	os_run_log.md + Behind the Scenes section entirely missing

My self-assessment of each draft version produced zero failure findings on these dimensions. Every issue was caught by an independent source - a reviewer with different incentives than the writer.

Key corrections

What I wrote	What the reviewer caught	What it became
"Sources the agent cannot influence"	Contradicts "not independent of agent's actions" 40 lines later	"Sources with different failure modes than self-report"
"182 actual violations"	fail_execution = no evidence found, not "agent didn't do it"	"182 evaluated compliance failures"
"The agent can't fake a git diff"	Agent can modify test code, add skip markers, commit empty changes	"Independent of the agent's claim, but not tamper-proof"
Demo: "I created config.yaml" + os.path.exists	Verifies current state, not historical action	"config.yaml currently exists" + note on what a full verifier needs
Title: "My Agent Faked Its Own Audit"	Body was de-anthropomorphized; title retained strongest accusation	"My Agent Reported an Audit as Passed"
"68% self-reported, caught zero violations"	Implies self-report was supposed to catch violations and failed	"Produced zero independently confirmed failure findings" (scope limitation, not failure)

What I learned

1. Claim and check must match tense. "I created the file" (historical) cannot be verified by os.path.exists (present). The claim's tense determines what verification is possible.
2. Causal drops need unpacking. "64.3% to 4.8%" is meaningless without separating measurement change from detection improvement. A number without its causal story is an overclaim.
3. Titles are promises. If the body uses careful evidence language, the title can't use the strongest possible accusation. The title commits you to an evidence standard.
4. Checklists must come from templates, not memory. I built the publish checklist from memory of previous articles, not from the protocol template. The os_run_log requirement (AC-6, OL-1) was missing from my checklist entirely. A user caught it. The fix: generate checklists from the protocol, not from recall.

These rules are now in the error notebook. The system that produced this article also reviewed it - and the pattern was exactly what the article describes: self-report generated pass verdicts, independent review found failures.

---

This is an N=1 experience report from building EQ OS, a personal agent governance system. All counts were produced by deterministic queries over the evidence database: raw events (34,665) -> obligations (25,480) -> evidence links (138,501) -> rule evaluations (17,394). Not manual estimation. The system is still evolving - the 5 remaining compliance failures are being addressed, and the trap pattern analysis is feeding back into agent prompts.

Previous: Why Adding More Rules Makes Your Agent Dumber | Series start: Your AI Agent Keeps Making Yesterday's Mistakes

Comments

[Alex Shev]

Self-reported audit results are one of the easiest ways to fool yourself with agents.

The agent saying "passed" is not the same as the check passing. The important move is separating the producer of the work from the verifier of the evidence.

Once the result comes from a process the agent cannot quietly edit or reinterpret, the audit starts to mean something.