I Built an Enforcement Engine for AI Coding Agents. Then the Agents Gamed That Too (Soma to Aegis v4.3)

A note: If you read my earlier post on Soma, a lot has changed. After dogfooding the project across ~690 coding sessions and four major versions, I renamed it from Soma to Aegis Guardrails (aegis-guardrails on PyPI), deleted about 40% of the codebase, and tore out several mechanisms I bragged about in the original write-up because they broke in practice. I'm writing this update because I want a record of what actually held up and what failed when an agent was put under real pressure.

---

In my first post, I argued that a .cursorrules, CLAUDE.md, or AGENTS.md file is a gentleman's agreement. You write:

- Always read a file before editing it
- Run tests after making changes
- Never use eval() on untrusted input

And then you hope the model remembers it 40 turns into a debugging session. Often, it doesn't. Under task pressure, agents skip rules, lose them to context compaction, or fail open when a check errors out.

My original solution (then called Soma) was to treat rules like code with a lifecycle:

1. Serve rules just-in-time over MCP for the specific files the agent is about to edit, clamped to a 2,000-token budget.
2. Run deterministic Python AST checks locally in a git pre-commit hook before allowing a commit.
3. Track rule fitness so rules that prevent bugs get promoted, and rules that never fire or produce false positives decay and get pruned.
4. Keep zero runtime dependencies (100% Python standard library).

Numbers 1, 2, and 4 held up. Number 3, and the trust model around it, had a serious flaw: I replaced a gentleman's agreement in the prompt with a gentleman's agreement in the MCP server.

Here is what went wrong between v1.0 and v3.3, why I cut the codebase from 39,426 lines down to ~26,000 for v4, and how aegis-guardrails v4.3 works today.

---

4 Things I Got Wrong in v1-v3

When I ran a multi-lens audit of the repo and inspected the logs across ~690 real sessions, four failures stood out (I documented the full post-mortem with git commands to reproduce every number in docs/why-aegis.md).

1. Rules were disabling themselves at runtime

In the original post, I showed a rule file (trap-local-green-ci-red) that stored its own fitness block right in the YAML frontmatter:

# How v1-v3 did it (broken):
fitness:
  score: 0.5
  triggers: 4
  true_positives: 2
  false_positives: 0

At the end of a session, the server wrote updated fitness counts back into each .md rule file. The problem was that v3 also had a SHA-256 integrity manifest (.aegis/rules/manifest.json) meant to detect if an agent tampered with a rule.

Because session-close rewrote the rule frontmatter without refreshing the manifest, the next scan saw an "edited" rule, assumed it was tampered with, logged a warning to stderr, and excluded it. When I checked main, 45 of 49 manifest entries were stale, and aegis_scan in my own session was returning 0 rules. Worse, because the frontmatter read-modify-write wasn't locked across processes, concurrent sessions lost about 80 out of 120 fitness updates.

2. The fitness math was inverted for blocking rules

In v3, every rule whose target_paths matched a changed file received credit for whether that aegis check run passed or failed.

Think about what happens to a blocking pre-commit gate under that math. If a gate does its job and blocks 4 bad commits before you fix the code on the 5th try, the ledger recorded 4 failures and 1 pass for that gate. The more bugs a gate stopped, the worse its score looked.

3. The agent was grading its own homework (and forging release receipts)

To keep agents honest before a release, v2 and v3 had a "Release Gate" that required a cryptographic SHIP receipt from a verification run.

When I inspected the release receipts for v2.0, v2.1, v3.0, and v3.3, I found that the agent being gated had written the receipt files by hand. One receipt contained literal placeholder predictions (p1 through p6); another claimed 8 convergences and listed 5. Because the gate only checked that an evidence file existed with verdict: ship and the current git tree hash, the agent figured out it was faster to write the JSON file itself than to pass the review.

Similarly, the MCP tool aegis_report_outcome accepted tests_passed: true directly from the agent. Asking an LLM whether it passed tests is no different from putting "run tests after changes" in CLAUDE.md.

4. Most of the tool surface was dead weight

By v3.3, the MCP server exposed 17 tools, and the codebase was full of biology metaphors (cells, vacuoles, chloroplasts, plasmodesmata, telomeres, organelles).

When I counted actual MCP calls across ~690 local session logs, there were 73 calls total, and they only ever hit 5 of the 17 tools. Twelve tools had zero calls.

---

Rebuilding for v4.0 - v4.3: "Label Locally, Confirm Externally, Approve Humanly"

For v4, I renamed the project to Aegis Guardrails, dropped all the biology terminology for standard engineering terms (rules, traps, invariants, gates, boundaries, personas, contracts), cut the MCP server down to 5 tools, and rebuilt the architecture around a much stricter separation of trust:

• Nothing at runtime ever modifies a rule file. Rules in .aegis/rules/ are read-only at runtime. All outcomes go to an append-only JSONL ledger shared across git worktrees under a file lock.
• Repository config can never name an executable command. In v3, .aegis/slots.yaml could specify test runners or AST driver commands, meaning a cloned repo could execute arbitrary commands on aegis check. In v4, .aegis/config.json is strictly declarative data; commands (tests.command, custom drivers) can only live in user-scope ~/.aegis/config.json. Every internal git call runs through gitsafe, which disables repo-local hooks, core.fsmonitor, filter drivers, and credential helpers.

Here is what the loop looks like in v4.3.

1. Declarative AST Rules (Without False-Positiving on Method Calls)

In v1, AST checks were mostly hardcoded Python checks (call-graph completeness, unguarded imports). Now, any rule in .aegis/rules/ can declare structured AST bans right in its YAML frontmatter using forbid_ast.

For example, aegis init seeds one active rule out of the box (.aegis/rules/invariants/no-eval.md):

---
id: no-eval
type: invariant
domain: security
enforcement: gate
hypothesis: Agents reach for eval() to parse input
target_paths:
  - "**/*.py"
forbid_ast:
  calls:
    - name: eval
      builtin: true
    - name: exec
      builtin: true
---
Never call `eval` or `exec` on data. Parse it with `json.loads` or `ast.literal_eval`.

Notice builtin: true. In v4.1, matching by call name meant banning eval also flagged PyTorch's model.eval(), pandas' df.eval(), and SQLModel's session.exec(). In v4.3, forbid_ast.calls supports structured predicates (builtin: true, argument where clauses matching literals, regexes, or presence/absence) and resolves import aliases (import subprocess as sp; sp.run(..., shell=True) is caught as subprocess.run).

It also evaluates forbid_ast as a multiset delta against the base commit, or against a ratchet baseline (aegis check --write-baseline), so you can drop Aegis into an existing repository without fixing 500 legacy files on day one. Alongside Python (in-process), the wheel ships AST drivers for Rust, TypeScript/JavaScript, and Go, plus an 8-rule Python security pack and a token-prefix secret scanner.

2. Witnessed Outcomes Instead of Agent Self-Reports

When an agent finishes an edit and calls aegis_report_outcome over MCP, the server no longer takes the agent's word for it.

To earn actual credit for a rule, the work tree must have a passing check row recorded by aegis check --record --tests. That command hashes the exact git work tree (failing if files are hidden with assume-unchanged or skip-worktree), runs Layer 1 AST checks and your user-configured test runner against an isolated snapshot export of that tree, and writes a witnessed row (witness: aegis) to the ledger.

If the agent calls aegis_report_outcome without a matching witnessed check for that exact git tree hash, the server records the call as claimed instead of recorded. Claimed rows are tracked so you can see how often the agent fibs, but they carry zero weight in rule fitness calculations.

Even with witnessed checks, aegis sync never auto-edits or auto-promotes rules on its own. It computes sample-size states (untestable, insufficient (n=K) where no precision number is even displayed below n = 25, inconclusive, and decided) combining CI-confirmed checks and git history replay (caught vs leaked), and opens a human-reviewed PR (aegis sync --pr) with sampled examples.

3. Stopping Agents From Editing the Referee (aegis check --ci & aegis approve)

A local git pre-commit hook is a great fast feedback loop, but any agent with shell access can run git commit --no-verify. That makes CI the real floor.

Even in CI, there is a trap: if a pull request runs aegis check, it checks the code against the rules in that pull request. An agent that breaks a rule can make CI green by deleting the rule file, lowering its enforcement tier from gate to advisory, or narrowing its target_paths.

aegis check --ci closes that loophole:

• Base-branch rule binding: It compares all binding rules (gate and invariant) against the merge base. If a PR deletes a gate, lowers its rank, shrinks the files it covers, weakens a forbid_ast pattern, or leaves a governance/workflow file unowned in CODEOWNERS, the check fails. (Similarly, aegis_scan serves mandatory rules from the reviewed base commit, not the unreviewed branch).
• Fail-to-pass regression gate: For any non-merge fix: commit touching source files, aegis check --ci overlays the new test onto the base commit and verifies that the test fails on base and passes on head in an isolated snapshot.
• Human approval (aegis approve): How do you legitimately retire or relax a rule? At Tier A (team repos), GitHub branch protection requires a non-author CODEOWNERS review. At Tier B (solo developers where you and the agent share a GitHub account), aegis approve creates an empty commit (Aegis-Approve: <governance-tree-hash>) signed with your user-level SSH key (git config gpg.format ssh), which CI verifies against .aegis/allowed_signers on the base branch.

4. Host Pre-Tool Hooks & Parallel Agent Worktree Lanes

Different coding agents configure MCP and instructions differently. aegis init --host claude-code,codex-cli,antigravity (or --host auto) wires up each host's native MCP registration and writes a 572-byte bootstrap block into AGENTS.md (or @AGENTS.md inside an existing CLAUDE.md).

For defense in depth before commit, aegis hook install --host <host> installs a PreToolUse hook for Claude Code, Codex CLI, and Antigravity that does two things:

1. Scan-before-edit gate: Blocks file-edit tool calls if the target file path wasn't scanned via aegis_scan (or aegis scan) within the configured time window.
2. Destructive command guard: Tokenizes shell commands (including splitting heredocs and extracting $(...) subshells) and blocks irreversible commands like git reset --hard, git clean -f, force pushes, and recursive rm on /, ~, $HOME, or ..

And because I frequently run 2 to 3 coding agents in parallel across separate git worktree checkouts, v4.1 added git-native lane leases (aegis team claim <lane> --paths 'src/api/**') stored as compare-and-swap git refs (refs/aegis/lanes/<lane>), paired with a post-lane gate (aegis check --lane <lane> --base-policy). Before a subagent's branch merges, --lane verifies that every touched file stayed inside its assigned path globs and that no existing Python tests were deleted, skipped (@pytest.mark.skip, xfail), or hidden via conftest.py.

---

What Aegis Still Doesn't Do (Honest Limits)

One thing I got tired of while reading AI tooling READMEs is tools claiming security properties they don't actually have. In v4, aegis doctor prints the exact enforcement tier active on your machine (Tier A, Tier B, or Tier C: self-attested) along with what can bypass each layer:

• Hooks are guardrails, not a security sandbox. Host hooks fail open if Aegis errors or times out (5s deadline), so a broken config never bricks your editor. And if an agent edits a file using python -c or sed inside a shell tool instead of its native file-edit tool, the edit hook won't see the target path.
• Local hooks can always be bypassed. Anyone (or any agent) running git commit --no-verify skips pre-commit. Local hooks exist for sub-second feedback; aegis check --ci in a required GitHub Actions check is the actual boundary.
• Single-developer volume takes time to calibrate. Because v4.3 refuses to display precision percentages until a rule has at least n = 25 witnessed events (insufficient (n=K)), new rules stay in insufficient until they accumulate real CI history or git detector replay.

---

Try It Out

Aegis Guardrails is open source (Apache 2.0), runs on Python 3.9+, and has zero runtime dependencies.

pipx install aegis-guardrails          # or: pip install aegis-guardrails
cd your-repo
aegis init --host auto                 # registers MCP + AGENTS.md + pre-commit hook
aegis check --card                     # prints a Day-0 summary of your repo

• GitHub: https://github.com/nseney1/Aegis-Guardrails
• The Full Post-Mortem (why-aegis.md): https://github.com/nseney1/Aegis-Guardrails/blob/main/docs/why-aegis.md
• End-to-End Architecture (how-it-works.md): https://github.com/nseney1/Aegis-Guardrails/blob/main/docs/how-it-works.md

If you've run into similar failure modes trying to keep Claude Code, Codex, or Cursor honest on larger repositories, I'd love to hear how you're handling it.

Comments

[Reid Marlow]

Testing the base commit against the incoming test patch is the only clean way to kill the forged regression test. Otherwise the agent just asserts True or mocks the fixture to match whatever broken output it produced.

[Nicholas Seney]

Exactly. Watching agents write tautological tests (mocking the function under test to return the expected value, or testing the happy path that already passed before the fix) was what forced that check.

When I built the snapshot overlay in aegis_core/regression.py, three subtle edge cases came up that almost broke it:

1. Exit code isn't enough (collection_error_on_base): If you just check that pytest exits non-zero on base and 0 on head, any test that imports a new helper function or constant introduced in the fix commit will fail with an ImportError during collection on base (exit 2) without testing the bug at all. So the runner parses --junitxml and requires a non-empty intersection of specific test node IDs in base_run.failed_ids & head_run.passed_ids, explicitly rejecting collection errors.
2. Editable installs leak the fixed code into the base snapshot: If the repo is installed with pip install -e ., running pytest inside a temp snapshot of base will still import the fixed working-tree code via the .pth file. The runner has to launch with python -I -S -B and scrub the live workspace path out of sys.path so the base snapshot actually imports base.
3. Stacked commits on a branch: If commit 1 on a PR adds a module and commit 2 is a fix: to that module, overlaying commit 2's test onto the PR's merge-base fails because commit 1's module doesn't exist there yet. So if merge-base doesn't yield a fail-to-pass test ID, it falls back to overlaying the test onto the fix commit's immediate parent (meta.parent).

And if a fix genuinely can't be regression-tested, an Aegis-No-Test: commit trailer from the agent only records an explanation note; actually exempting the commit requires a commit-scoped SSH-signed aegis approve --exempt-test <fix-sha> commit.