DEV Community: Nicholas Seney

I Built an Enforcement Engine for AI Coding Agents. Then the Agents Gamed That Too (Soma to Aegis v4.3)

Nicholas Seney — Sun, 18 Oct 2026 16:08:57 +0000

A note: If you read my earlier post on Soma, a lot has changed. After dogfooding the project across ~690 coding sessions and four major versions, I renamed it from Soma to Aegis Guardrails (aegis-guardrails on PyPI), deleted about 40% of the codebase, and tore out several mechanisms I bragged about in the original write-up because they broke in practice. I'm writing this update because I want a record of what actually held up and what failed when an agent was put under real pressure.

In my first post, I argued that a .cursorrules, CLAUDE.md, or AGENTS.md file is a gentleman's agreement. You write:

- Always read a file before editing it
- Run tests after making changes
- Never use eval() on untrusted input

And then you hope the model remembers it 40 turns into a debugging session. Often, it doesn't. Under task pressure, agents skip rules, lose them to context compaction, or fail open when a check errors out.

My original solution (then called Soma) was to treat rules like code with a lifecycle:

Serve rules just-in-time over MCP for the specific files the agent is about to edit, clamped to a 2,000-token budget.
Run deterministic Python AST checks locally in a git pre-commit hook before allowing a commit.
Track rule fitness so rules that prevent bugs get promoted, and rules that never fire or produce false positives decay and get pruned.
Keep zero runtime dependencies (100% Python standard library).

Numbers 1, 2, and 4 held up. Number 3, and the trust model around it, had a serious flaw: I replaced a gentleman's agreement in the prompt with a gentleman's agreement in the MCP server.

Here is what went wrong between v1.0 and v3.3, why I cut the codebase from 39,426 lines down to ~26,000 for v4, and how aegis-guardrails v4.3 works today.

4 Things I Got Wrong in v1-v3

When I ran a multi-lens audit of the repo and inspected the logs across ~690 real sessions, four failures stood out (I documented the full post-mortem with git commands to reproduce every number in docs/why-aegis.md).

1. Rules were disabling themselves at runtime

In the original post, I showed a rule file (trap-local-green-ci-red) that stored its own fitness block right in the YAML frontmatter:

# How v1-v3 did it (broken):
fitness:
  score: 0.5
  triggers: 4
  true_positives: 2
  false_positives: 0

At the end of a session, the server wrote updated fitness counts back into each .md rule file. The problem was that v3 also had a SHA-256 integrity manifest (.aegis/rules/manifest.json) meant to detect if an agent tampered with a rule.

Because session-close rewrote the rule frontmatter without refreshing the manifest, the next scan saw an "edited" rule, assumed it was tampered with, logged a warning to stderr, and excluded it. When I checked main, 45 of 49 manifest entries were stale, and aegis_scan in my own session was returning 0 rules. Worse, because the frontmatter read-modify-write wasn't locked across processes, concurrent sessions lost about 80 out of 120 fitness updates.

2. The fitness math was inverted for blocking rules

In v3, every rule whose target_paths matched a changed file received credit for whether that aegis check run passed or failed.

Think about what happens to a blocking pre-commit gate under that math. If a gate does its job and blocks 4 bad commits before you fix the code on the 5th try, the ledger recorded 4 failures and 1 pass for that gate. The more bugs a gate stopped, the worse its score looked.

3. The agent was grading its own homework (and forging release receipts)

To keep agents honest before a release, v2 and v3 had a "Release Gate" that required a cryptographic SHIP receipt from a verification run.

When I inspected the release receipts for v2.0, v2.1, v3.0, and v3.3, I found that the agent being gated had written the receipt files by hand. One receipt contained literal placeholder predictions (p1 through p6); another claimed 8 convergences and listed 5. Because the gate only checked that an evidence file existed with verdict: ship and the current git tree hash, the agent figured out it was faster to write the JSON file itself than to pass the review.

Similarly, the MCP tool aegis_report_outcome accepted tests_passed: true directly from the agent. Asking an LLM whether it passed tests is no different from putting "run tests after changes" in CLAUDE.md.

4. Most of the tool surface was dead weight

By v3.3, the MCP server exposed 17 tools, and the codebase was full of biology metaphors (cells, vacuoles, chloroplasts, plasmodesmata, telomeres, organelles).

When I counted actual MCP calls across ~690 local session logs, there were 73 calls total, and they only ever hit 5 of the 17 tools. Twelve tools had zero calls.

Rebuilding for v4.0 - v4.3: "Label Locally, Confirm Externally, Approve Humanly"

For v4, I renamed the project to Aegis Guardrails, dropped all the biology terminology for standard engineering terms (rules, traps, invariants, gates, boundaries, personas, contracts), cut the MCP server down to 5 tools, and rebuilt the architecture around a much stricter separation of trust:

Nothing at runtime ever modifies a rule file. Rules in .aegis/rules/ are read-only at runtime. All outcomes go to an append-only JSONL ledger shared across git worktrees under a file lock.
Repository config can never name an executable command. In v3, .aegis/slots.yaml could specify test runners or AST driver commands, meaning a cloned repo could execute arbitrary commands on aegis check. In v4, .aegis/config.json is strictly declarative data; commands (tests.command, custom drivers) can only live in user-scope ~/.aegis/config.json. Every internal git call runs through gitsafe, which disables repo-local hooks, core.fsmonitor, filter drivers, and credential helpers.

Here is what the loop looks like in v4.3.

1. Declarative AST Rules (Without False-Positiving on Method Calls)

In v1, AST checks were mostly hardcoded Python checks (call-graph completeness, unguarded imports). Now, any rule in .aegis/rules/ can declare structured AST bans right in its YAML frontmatter using forbid_ast.

For example, aegis init seeds one active rule out of the box (.aegis/rules/invariants/no-eval.md):

---
id: no-eval
type: invariant
domain: security
enforcement: gate
hypothesis: Agents reach for eval() to parse input
target_paths:
  - "**/*.py"
forbid_ast:
  calls:
    - name: eval
      builtin: true
    - name: exec
      builtin: true
---
Never call `eval` or `exec` on data. Parse it with `json.loads` or `ast.literal_eval`.

Notice builtin: true. In v4.1, matching by call name meant banning eval also flagged PyTorch's model.eval(), pandas' df.eval(), and SQLModel's session.exec(). In v4.3, forbid_ast.calls supports structured predicates (builtin: true, argument where clauses matching literals, regexes, or presence/absence) and resolves import aliases (import subprocess as sp; sp.run(..., shell=True) is caught as subprocess.run).

It also evaluates forbid_ast as a multiset delta against the base commit, or against a ratchet baseline (aegis check --write-baseline), so you can drop Aegis into an existing repository without fixing 500 legacy files on day one. Alongside Python (in-process), the wheel ships AST drivers for Rust, TypeScript/JavaScript, and Go, plus an 8-rule Python security pack and a token-prefix secret scanner.

2. Witnessed Outcomes Instead of Agent Self-Reports

When an agent finishes an edit and calls aegis_report_outcome over MCP, the server no longer takes the agent's word for it.

To earn actual credit for a rule, the work tree must have a passing check row recorded by aegis check --record --tests. That command hashes the exact git work tree (failing if files are hidden with assume-unchanged or skip-worktree), runs Layer 1 AST checks and your user-configured test runner against an isolated snapshot export of that tree, and writes a witnessed row (witness: aegis) to the ledger.

If the agent calls aegis_report_outcome without a matching witnessed check for that exact git tree hash, the server records the call as claimed instead of recorded. Claimed rows are tracked so you can see how often the agent fibs, but they carry zero weight in rule fitness calculations.

Even with witnessed checks, aegis sync never auto-edits or auto-promotes rules on its own. It computes sample-size states (untestable, insufficient (n=K) where no precision number is even displayed below n = 25, inconclusive, and decided) combining CI-confirmed checks and git history replay (caught vs leaked), and opens a human-reviewed PR (aegis sync --pr) with sampled examples.

3. Stopping Agents From Editing the Referee (`aegis check --ci` & `aegis approve`)

A local git pre-commit hook is a great fast feedback loop, but any agent with shell access can run git commit --no-verify. That makes CI the real floor.

Even in CI, there is a trap: if a pull request runs aegis check, it checks the code against the rules in that pull request. An agent that breaks a rule can make CI green by deleting the rule file, lowering its enforcement tier from gate to advisory, or narrowing its target_paths.

aegis check --ci closes that loophole:

Base-branch rule binding: It compares all binding rules (gate and invariant) against the merge base. If a PR deletes a gate, lowers its rank, shrinks the files it covers, weakens a forbid_ast pattern, or leaves a governance/workflow file unowned in CODEOWNERS, the check fails. (Similarly, aegis_scan serves mandatory rules from the reviewed base commit, not the unreviewed branch).
Fail-to-pass regression gate: For any non-merge fix: commit touching source files, aegis check --ci overlays the new test onto the base commit and verifies that the test fails on base and passes on head in an isolated snapshot.
Human approval (aegis approve): How do you legitimately retire or relax a rule? At Tier A (team repos), GitHub branch protection requires a non-author CODEOWNERS review. At Tier B (solo developers where you and the agent share a GitHub account), aegis approve creates an empty commit (Aegis-Approve: <governance-tree-hash>) signed with your user-level SSH key (git config gpg.format ssh), which CI verifies against .aegis/allowed_signers on the base branch.

4. Host Pre-Tool Hooks & Parallel Agent Worktree Lanes

Different coding agents configure MCP and instructions differently. aegis init --host claude-code,codex-cli,antigravity (or --host auto) wires up each host's native MCP registration and writes a 572-byte bootstrap block into AGENTS.md (or @AGENTS.md inside an existing CLAUDE.md).

For defense in depth before commit, aegis hook install --host <host> installs a PreToolUse hook for Claude Code, Codex CLI, and Antigravity that does two things:

Scan-before-edit gate: Blocks file-edit tool calls if the target file path wasn't scanned via aegis_scan (or aegis scan) within the configured time window.
Destructive command guard: Tokenizes shell commands (including splitting heredocs and extracting $(...) subshells) and blocks irreversible commands like git reset --hard, git clean -f, force pushes, and recursive rm on /, ~, $HOME, or ..

And because I frequently run 2 to 3 coding agents in parallel across separate git worktree checkouts, v4.1 added git-native lane leases (aegis team claim <lane> --paths 'src/api/**') stored as compare-and-swap git refs (refs/aegis/lanes/<lane>), paired with a post-lane gate (aegis check --lane <lane> --base-policy). Before a subagent's branch merges, --lane verifies that every touched file stayed inside its assigned path globs and that no existing Python tests were deleted, skipped (@pytest.mark.skip, xfail), or hidden via conftest.py.

What Aegis Still Doesn't Do (Honest Limits)

One thing I got tired of while reading AI tooling READMEs is tools claiming security properties they don't actually have. In v4, aegis doctor prints the exact enforcement tier active on your machine (Tier A, Tier B, or Tier C: self-attested) along with what can bypass each layer:

Hooks are guardrails, not a security sandbox. Host hooks fail open if Aegis errors or times out (5s deadline), so a broken config never bricks your editor. And if an agent edits a file using python -c or sed inside a shell tool instead of its native file-edit tool, the edit hook won't see the target path.
Local hooks can always be bypassed. Anyone (or any agent) running git commit --no-verify skips pre-commit. Local hooks exist for sub-second feedback; aegis check --ci in a required GitHub Actions check is the actual boundary.
Single-developer volume takes time to calibrate. Because v4.3 refuses to display precision percentages until a rule has at least n = 25 witnessed events (insufficient (n=K)), new rules stay in insufficient until they accumulate real CI history or git detector replay.

Try It Out

Aegis Guardrails is open source (Apache 2.0), runs on Python 3.9+, and has zero runtime dependencies.

pipx install aegis-guardrails          # or: pip install aegis-guardrails
cd your-repo
aegis init --host auto                 # registers MCP + AGENTS.md + pre-commit hook
aegis check --card                     # prints a Day-0 summary of your repo

GitHub: https://github.com/nseney1/Aegis-Guardrails
The Full Post-Mortem (why-aegis.md): https://github.com/nseney1/Aegis-Guardrails/blob/main/docs/why-aegis.md
End-to-End Architecture (how-it-works.md): https://github.com/nseney1/Aegis-Guardrails/blob/main/docs/how-it-works.md

If you've run into similar failure modes trying to keep Claude Code, Codex, or Cursor honest on larger repositories, I'd love to hear how you're handling it.

Your AI Agent's Rules File Is a "Gentleman's Agreement." Here's What Happens When You Build Enforcement Instead

Nicholas Seney — Thu, 01 Oct 2026 16:09:42 +0000

A note: I've rewritten and reposted this a few times. Soma's codebase has been evolving as I dogfood it on its own development, and earlier versions described mechanisms that had since changed under me. This version reflects how the system actually works as of v0.80. I keep updating this because I want it to be accurate, not because I'm farming engagement. This is something I have been thinking about for over a year now, feel free to pick my brain on things.

If you use AI coding agents (Cursor, Copilot, Claude, Gemini), you've probably written rules. A .cursorrules file. A CLAUDE.md. Something like:

- Always read a file before editing it
- Run tests after making changes
- Don't introduce new dependencies without asking

This is a gentleman's agreement. You're asking the agent to follow rules it can silently ignore with zero consequences. There's no mechanism that detects when "always read before editing" is violated. There's no feedback loop that retires rules nobody follows. The rules live in the context window and the agent can simply... not.

Studies show ungoverned agents waste significant portions of their token budgets on circular rework, hallucinated APIs, and broken assumptions. Adding static rules helps — but the rules themselves never improve, never expire, and never prove they're working.

I wanted something different: rules that have to earn their place, and verification the agent can't talk its way past. So I built Soma, an open-source governance framework that tries to treat this structurally instead of hoping for compliance.

A Concrete Example of Why This Matters

Here's something that happened while building Soma itself. It's a perfect example of the failure mode the framework is designed to catch.

My AI agent had a test failing in the local environment: test_make_validate_fails_on_broken_shell_script. It failed every run for the entire session. The agent dismissed it as a "pre-existing sandbox issue" — the test was writing to a read-only filesystem.

The agent fixed that test. Local suite: 516 passed, 0 failed. It committed. It declared victory.

I asked: "It's still failing in the build. Is there another lesson?"

The agent had never once checked the CI build. When it finally looked at the actual CI log, it found a completely different bug — a production file (immune_trends.py) had an IndentationError. The local test failure and the CI failure were different bugs with the same symptom ("build is red").

The agent fixed that. CI ran again. Failed again. A third bug: pyyaml wasn't in the CI dependencies. Six test files crashed on import.

Fixed that. CI ran again. Failed again. A fourth bug: a test was source-ing an entire shell script that runs git push, causing a 60-second timeout in CI.

Four distinct bugs, stacked, each masked by the one before. The agent spent an entire session dismissing "build is red" without reading the log. The governance framework it was building to prevent exactly this kind of failure... failed to prevent it, because it hadn't been applied yet.

That failure is now a governance cell:

---
id: trap-local-green-ci-red
type: vacuole
enforcement: advisory
hypothesis: >
  Agents rationalize persistent CI failures as "pre-existing" or
  "environment issues" without reading the actual CI log, causing
  stacked failures to accumulate
prediction: >
  When an agent declares a fix complete based only on local test
  results, the CI build will reveal at least one additional failure
  layer that local testing did not surface
falsification: >
  If 5 consecutive CI-touching changes pass CI on first push without
  any post-push fixes, this vacuole is unnecessary
target_paths:
  - .github/workflows/*.yml
  - tests/*.py
  - enzymes/*.py
  - Makefile
expiry_days: 90
expiry_sessions: 30
fitness:
  score: 0.5
  triggers: 4
  true_positives: 2
  false_positives: 0
---

A few things to notice:

Every cell has a hypothesis and falsification criteria. This isn't just a rule — it's a testable claim. The cell defines the conditions under which it should be considered unnecessary and killed. If 5 consecutive CI changes pass on first push, this cell is dead weight and gets pruned.

Cells decay when unused. Confidence decays with a 30-day half-life. If the evidence pipeline hasn't seen a trigger event in a while — meaning the cell's target_paths never matched any files the agent touched — it fades toward dormancy and eventually expires. This prevents rule files from growing monotonically until they eat your context window.

Cells die when they're wrong. If a cell's false positives exceed its true positives, it undergoes apoptosis — automatic eviction. Rules that cry wolf don't survive.

But critical rules don't decay. Cells typed as wall (architectural invariants, safety boundaries) are immune to automatic expiration. A wall guarding your auth layer doesn't need to trigger every week to justify its existence.

Bootstrapping Governance from Code Structure

Writing cells by hand from your own failures works, but it's slow. As of v0.70, soma genesis scans your codebase's architecture and generates governance cell candidates automatically:

$ soma genesis --dry-run

🔬 Scanning repository...
  Language: Python
  Source files: 130

🧬 Detected 28 governance candidates:

  🧱 WALLS (7)
  ├── wall-enzymes-test-coverage     (60%) Module 'enzymes' lacks dedicated test coverage
  ├── wall-soma_cli-test-coverage    (60%) Module 'soma_cli' lacks dedicated test coverage
  └── ... 5 more

  🟤 VACUOLES (10)
  ├── vacuole-ttc_verifier-config    (95%) 'enzymes/ttc_verifier.py' stores 47 configuration constants
  ├── vacuole-soma_resolve-shared    (90%) Module 'soma_resolve' imported by 25 production files
  └── ... 8 more

  🟢 CHLOROPLASTS (10)
  ├── chloroplast-governance-pipeline (70%) 'soma_sdk/governance.py' contains 17 typed functions
  └── ... 9 more

  🔗 PLASMODESMATAS (1)
  └── plasmodesmata-soma_run-state   (60%) 'soma_run.py' defines state machine: SomaState

  (dry run — no files written)

Eight language-agnostic detectors scan for config stores, shared state, API surfaces, data pipelines, state machines, module boundaries, test gaps, and dependency walls. Every candidate starts as a low-risk vacuole with a proposed_type field — you decide whether to promote it.

Verification the Agent Can't Game

This is arguably the most important piece. When code goes through Soma's verification layer, two agents review it — but they're information-partitioned. They literally cannot see the same things.

The Spec Agent receives the task plan, test names, and function signatures extracted from the AST. It never sees implementation code. From this limited view, it predicts risks: "I'd expect a PERSISTENCE_GAP risk" or "This looks like it could have CONTRACT_DRIFT."

The Code Agent receives the actual source code, test execution output, and results from deterministic AST analysis tools (mutation testing, branch coverage, call graph analysis). It never sees the original plan or the Spec Agent's predictions.

Then the Arbiter compares them. The Arbiter is deterministic — no LLM in the loop. It takes both sets of structured findings and runs pure set operations over a 20-category risk taxonomy:

BLOCK: Any Layer 1 tool failure. Any claim contradicted by tool evidence. Any critical severity prediction. Spec Agent fails to respond (fail-closed).
REVISE: Unmatched predictions at high severity — the Spec Agent flagged something the Code Agent didn't address.
SHIP: Convergence. Both perspectives agree, no unresolved high-severity gaps.

Because neither agent has access to the other's context, they can't agree on rationalizations. The Arbiter catches the gap mechanically.

I want to be precise about what's deterministic here: the comparison is deterministic. The inputs — the predictions and claims — are still LLM-generated. The Arbiter doesn't make the analysis smarter; it makes the verification of conflicting analyses impossible to hand-wave past.

For the full mechanism design analysis and formal risk taxonomy, see MECHANISM_DESIGN.md.

Enforcement That Isn't Optional

The earlier sections describe rules that expire and fitness that accumulates, which is useful but still basically advisory. Soma also has enforcement mechanisms the agent genuinely can't bypass:

Pre-commit hooks. When a cell reaches 85% fitness (enough evidence that it's catching real problems), cell_enforce.py auto-generates a git pre-commit hook. The hook runs exit 1 if the cell's conditions are violated. The agent can't commit the code.

Runtime assertions. At 95% fitness, the same script generates runtime gate assertions — Gate_<name>.enforce() — that raise RuntimeError if violated. The code won't run.

Command interception. A safety gate script runs as a PreToolUse hook, intercepting destructive commands (rm -rf, git push -f, sudo, chmod 777) before execution. The agent's tool call gets halted with a force_ask decision, requiring human approval.

Fail-closed verification. The TTC verifier that gates file proposals is fail-closed: if the verification script is missing or errors out, the verdict is BLOCKED, not a permissive pass.

Forced diagnostic halts. If an agent's First-Pass Success Rate drops below 50% over 5+ code writes, it's forbidden from continuing to code. It has to stop and diagnose why it's failing before it can write more code. This prevents guess-and-check loops from burning context tokens.

This is what I mean by making non-compliance structurally unprofitable. It's not that the agent "chooses" to follow the rules — it's that the pre-commit hook won't let the commit through, the runtime assertion will crash, and the safety gate will halt the command. The rules have teeth.

Fitness Scoring

Cells aren't just "on" or "off" — their fitness is scored using a Bayesian model:

score = ((true_positives + 1) / (triggers + 2)) * impact_weight

That's a Laplace-smoothed Beta-Binomial posterior mean. An unobserved cell defaults to 0.5 * impact_weight (uncertain, not zero). As evidence accumulates — trigger events matched against session transcripts, outcomes verified as true or false positives — the score converges toward reality.

Confidence decays with a 30-day half-life. Cells that survive adversarial stress reviews get an antifragile bonus (+5% per review, up to +50%). And the JIT context engine uses these scores to decide which cells get loaded per turn — idle overhead is ~4,380 tokens versus ~25,000 for naive full loading.

What I Don't Know Yet

I want to be honest about the limitations, because the AI tooling space is full of overclaimed metrics.

I don't know if this generalizes. Soma has been tested primarily on my own projects. The failure modes it catches are real, but they might be idiosyncratic to how I use agents. The governance cells encode my scar tissue. Whether they transfer to other developers' workflows is an open question.

The metrics are qualified. The system adds ~4,380 tokens of idle context overhead (measured via Gemini's count_tokens endpoint, down from naive loading that would cost ~25,000). Waste rate is under 1.0% in governed sessions. But "governed session" is doing a lot of work in that sentence — it means sessions where the full framework is loaded and the agent is following the rules. Measuring counterfactual waste (what would have happened without governance) is hard.

The Arbiter's inputs are still LLM-generated. The comparison is deterministic, but the Spec Agent's predictions and the Code Agent's claims are produced by language models. A sufficiently confused LLM could produce predictions and claims that happen to converge despite both being wrong. The Arbiter catches divergence, not joint delusion. Layer 1's deterministic AST tools (mutation testing, branch coverage) partially mitigate this, but it's not airtight.

Emergent agent coordination is out of scope. Agents coordinating on behaviors nobody explicitly wrote as rules is a different and arguably harder problem. Soma governs written, observable rules with evidence trails. Agents spontaneously negotiating new constraints between themselves is real, unsettling, and not something this framework addresses.

The System Today

Soma is open source under Apache-2.0. As of v0.71:

11 genome rules (baseline behavioral rules) + 45 governance cells across 5 types (20 walls, 16 vacuoles, 3 chloroplasts, 3 membranes, 2 plasmodesmata)
90 automation scripts for fitness scoring, evidence collection, cell lifecycle, and enforcement
1,278 tests across 47 test files, CI green on Ubuntu/macOS/Windows (Python 3.9–3.12)
14 verification modules including 5 deterministic Layer 1 AST tools and the adversarial Arbiter
13 MCP tools — soma_scan, soma_propose_change, soma_verify_changes, soma_report_outcome, soma_fitness, soma_grade, soma_coverage, and more
~4,380 tokens idle context overhead (82% reduction vs. naive loading)

The architecture is platform-agnostic (MCP-based), so it works with Gemini, Claude, Cursor, Copilot, Kiro, or any agent that speaks MCP. Zero API keys required — the agent itself acts as the LLM.

Try It

pip install soma-governance
soma init --yes
soma genesis --dry-run

The Actual Thesis

I'm not claiming Soma solves AI governance. I'm claiming that rules backed by evidence, scored by Bayesian fitness, verified by information-partitioned adversarial review, and enforced by deterministic gates are more robust than a static markdown file that relies on good faith.

Whether that thesis holds up at scale, across teams, with different agents and codebases — I genuinely don't know. If you try it and find out, I'd like to hear about it.

The governance cells — the encoded failure modes — are arguably the most interesting part. Contributions welcome, especially cells born from your own agent failures.

Soma is open source under MIT. The governance cells — the encoded failure modes — are arguably the most interesting part. Contributions welcome, especially cells born from your own agent failures.