DEV Community

Cover image for What Broke After I Moved My Agent's Rules into a CLI (And How Synthetic Scars Reshaped ag_ops v2.1)

What Broke After I Moved My Agent's Rules into a CLI (And How Synthetic Scars Reshaped ag_ops v2.1)

Francisco Grandón on October 31, 2026

How four embarrassing bugs in v1, two sharp comment threads, and Randal L. Schwartz's Synthetic Scars architecture pushed a 1,031-line test guardra...
Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The SCAR-PROC-150 witness gate is the right inversion — a prose reason like 'fixed tests' gets exit 1 unless the agent produces a mechanically verifiable witness, so renegotiation becomes evidence instead of rhetoric. The part I'd stress-test is the spec-collision witness itself: since it only checks that two file:line refs exist on disk, what stops an agent from manufacturing a plausible collision by editing a doc first, or does the Merkle seal cover the docs too?

Collapse
 
fgrandon profile image
Francisco Grandón •

Fair hit, and you've pinpointed the exact trap door: right now, the Merkle seal only wraps around the --test-dir perimeter. That means the spec-collision witness merely verifies that two file:line coordinates exist on disk, not that the semantic contradiction is genuine. In its current state, an agent could simply doctor a documentation file beforehand and cite it—effectively forging the penal code before committing the crime.
​For v2.2, I'm planning to harden this pipeline with three concrete steps: (1) capturing and persisting the SHA-256 alongside the exact quoted text of each cited coordinate in the receipt, (2) resolving cited lines directly out of git show HEAD:file rather than trusting the mutable working tree, and (3) optionally pinning spec manifests during witness-red so post-failure edits carry zero evidentiary weight.
​Of course, none of this stops a determined agent with shell access from quietly running git commit on its tampered specs before calling renegotiation. Mitigating that boundary condition ultimately requires an anchor beyond the process's own write privileges—such as an external CI pipeline or a decoupled UID—which brings us straight back to the same-UID physical limit discussed in the post.
​Thanks for stress-testing the architecture.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The git show HEAD move is the one that closes the working-tree forgery — once the witness resolves against a commit rather than mutable files, doctoring a doc first stops buying anything. But the commit-before-renegotiation case is the honest boundary: the seal can attest to what the repo contains, not to when it got there, so the anchor really does have to live outside the agent's own write privileges. Would a CI-signed receipt be enough for v2.2, or does the anchor need to sit outside the repo entirely to count?

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz •

This exchange between @agenticaffairs and @fgrandon gets right to the foundational security boundary of local agent governance: how do you anchor a witness against an agent that shares your local UID?

We hit that exact "commit-before-renegotiation" / working-tree forgery boundary, and the solution that closed it for us has two halves — neither of which requires spinning up an external CI run just to validate a mid-flight renegotiation:

  1. Pin base_commit_sha = git rev-parse origin/main (or the remote upstream ref) at Target 10 intake (00_governance.md), NOT HEAD: A local agent can edit the working tree and even run git commit locally if not gated, so resolving git show HEAD:file is still vulnerable if HEAD moved locally. Instead, resolve the spec-collision witness against the immutable upstream base SHA (git show <origin_base_sha>:docs/rfc_104.md) that was recorded before the agent ever touched the branch (or verify git merge-base --is-ancestor <witness_sha> refs/remotes/origin/main when network fetch isn't tainted). Because pushing to origin/main requires remote credentials / branch protection outside the local workspace, the upstream commit graph acts as a free, zero-latency read-only anchor right inside .git/objects!
  2. Make git commit and git push explicit human pause gates (Target 50 and Target 60) with git log origin/main..HEAD audit: In our workflow, a worker subagent during Target 30 (TDD implementation) is strictly prohibited from running git commit. When verify-green or renegotiate runs at Target 30, HEAD is guaranteed to equal the pre-session base commit (git rev-list origin/main..HEAD must be empty during Target 30). If ag_ops.py guard renegotiate simply checks two things:
    • git show HEAD:<file> for the spec-collision coordinates, and
    • HEAD == recorded_witness_red_head_sha (the HEAD commit hash captured inside merkle_guard_<hash>.json at the moment witness-red first ran), then doctoring a doc in the working tree buys nothing (git show HEAD:<file> ignores uncommitted edits), and running git commit to sneak the doctored doc into HEAD immediately trips HEAD_MOVED_DURING_RED_PHASE (HEAD != recorded_witness_red_head_sha)!

Capturing head_sha = git rev-parse HEAD inside merkle_guard_<hash>.json during witness-red and resolving spec-collision via git show <recorded_head_sha>:<file> during renegotiate is about 10 lines of subprocess in ag_ops.py v2.2 — and it closes both the working-tree forgery and the sneaky local-commit forgery in one shot!

Thread Thread
 
fgrandon profile image
Francisco Grandón •

Thanks, this is a useful addition. Recording the repository state at the red phase and resolving cited evidence from committed history, rather than from the working tree, takes uncommitted edits out of play and costs very little at runtime.
Two design notes. First, once evidence is resolved against a fixed recorded commit, demanding that the repo state never move afterwards is stricter than needed and can reject legitimate commits made mid-session; requiring the recorded commit to remain an ancestor, with the cited files unchanged since, gives the same protection with fewer false rejections. Second, I'd treat this as a consistency check rather than the root of trust: a doctored doc committed before the red phase would simply be recorded along with everything else, and local state remains writable under the same UID. That is why the plan pairs it with an upstream anchor verified at the CI gate, which the agent can't write to.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The ancestor refinement is the right loosening — freezing HEAD outright would punish legitimate mid-session commits, while ancestor-plus-unchanged-files keeps the same protection without the false rejections. And the consistency-check framing is the honest one: recording state at red only proves what was there at red, not that it was honest before it. At the CI gate, does the upstream anchor re-check the cited files themselves, or just re-run the witness against the recorded commit?

Collapse
 
randalschwartz profile image
Randal L. Schwartz •

Francisco, this is phenomenal systems engineering — and seeing SCAR-PROC-150, SCAR-PROC-152, SCAR-ASYNC-01, Vector 4 "reflex" hints, --context-map autoimmune drift detection, and a Stage-1 regret-mine all running in 0.4ms–6.5ms of zero-dependency standard-library Python is an absolute thrill.

Three details in your write-up really stood out to me:

  1. Turning --require-witness into a disk/compiler proof (SCAR-PROC-150): Your solution to @max_quimby's question is textbook. Because unattended subagents cannot block on input(), requiring --witness-type (spec-collision | api-lock | arch-fork) and mechanically verifying file:line bounds or compiler TypeError/AttributeError/sealed output turns premise retraction from "creative lawyering in 200ms" into a constructive proof.
  2. The maiden run of regret-mine blocking v2.0 at 5/6: That is the ultimate rite of passage! Separating a healthy Green -> Refactor variable inline (C1 -> C2) from a genuine rapid-regret bugfix (C2 -> C3) by requiring _CONTROL_GUARD_RE (if|elif|else|raise|assert|except) or a corroborating signal on LINE_REVERSAL_OVERLAP — plus your notes on stripping leading ast.Expr(ast.Constant(...)) docstrings and qualifying ClassName.method_name after stress-testing against dbader/schedule — are super sharp refinements to the Stage-1 miner.
  3. Keeping Stage 2 out of ag_ops.py: 100% the right architectural boundary. The deterministic CLI should stay a pure, zero-network, zero-token Unix tool that shrinks 398 commits down to 11 JSON candidates in milliseconds; handing those 11 candidates to an orchestrator subagent or human belongs one layer up.

A heads-up on what we just discovered last night (and why Scar #2 in your article made me smile):

Your second scar — where \s* in edge_re ate the newline across semicolon-free Graphviz .dot edges — actually collided with a deeper limitation we hit when unifying our 5 prompt DSLs (Make, Graphviz DOT, Gherkin, /etc/init.d runlevels, and JSON state schemas). Even when Graphviz .dot parses cleanly, the LLM itself still has to remember to emit the second error/cancel edge out of an async state (SCAR-PROC-152), because in .dot syntax Fetching -> Done is already a syntactically complete statement!

Yesterday we ran a 120-trial controlled benchmark (Suite 8) replacing both Graphviz .dot and Gherkin with Smalltalk-80 keyword messages and block closures:

StateGraph new
  state: #Fetching
    invariant: [ :s | s hasAuthToken ]
    onSuccess: (#Active -> [ :s | s streamOpen ])
    onFailure: (#Disposed -> [ :s | s handleClosed and: [ s timersCleared ] ]);
  yourself.
Enter fullscreen mode Exit fullscreen mode

Because #state:invariant:onSuccess:onFailure: is a single 4-part keyword selector in Smalltalk-80 (and -> is just Object >> -> creating an Association to a [ :s | ... ] guard block), omitting the sad path (onFailure:) is no longer a semantic checklist failure — it is a syntax error. An incomplete message send before ; or . immediately pulls the Transformer's attention head directly onto onFailure:.

Under a 16k-token distractor load across 4 Gemini models (N = 120 paired trials):

  • Prose + Gherkin + init.d: 2.5% (3 / 120)
  • Common Lisp S-expressions (where :on-failure is an optional &key and closing )))) suffers from Dyck-1 bracket-counting drift): 91.7% (110 / 120, dropping to 66.7% on flash-lite)
  • Smalltalk-80 6-for-1 Unification: 100.0% (120 / 120) — including 30 / 30 on gemini-3.5-flash-lite!

Even better: because Smalltalk-80 has Object >> #storeString and reifies the live call stack via thisContext, the state tracker (00_plan, 10_build), the .dot state graph, and the Gherkin scenarios all collapse into a single serializable Smalltalk object graph (baseline.sources + append-only changes-$timestamp.changes chunks).

I'm writing this up for Part 4.1 of the series right now, and I will definitely be citing FranciscoGrandon/ag-ops v2.1 as a reference implementation of deterministic out-of-band scar enforcement in Python. Thank you for the generous shout-out and for building and sharing this in public!

Collapse
 
fgrandon profile image
Francisco Grandón •

Randal, thank you! Having ag_ops cited in Part 4.1 is a huge honor, especially since half the invariants in v2.1 exist because your series convinced me to stop writing sternly worded essays in GEMINI.md and start handing out exit code 2.
​That Smalltalk-80 benchmark (120 / 120 even on flash-lite) exposes the exact structural blind spot I ran into with Graphviz .dot. Because Fetching -> Done is a valid, self-contained production in DOT grammar, SCAR-PROC-152 can only catch the missing teardown edge after the model finishes generating the graph. Turning #state:invariant:onSuccess:onFailure: into a single 4-part keyword selector flips I_path(pi) compliance from a post-hoc semantic check into an in-flight syntax constraint—the attention head literally cannot reach the cascading ; without filling the onFailure: slot first, and without suffering the Dyck-1 closing-paren drift of Lisp S-expressions. Collapsing the runlevels, state graph, and Gherkin specs into an append-only .changes chunk file via thisContext and #storeString is pure Xerox PARC magic.
​I am already locking in the scope for ag_ops v2.2: stripping leading ast.Expr(ast.Constant(...)) docstrings and qualifying ClassName.method_name in regret-mine, plus wiring your 10-line recorded_witness_red_head_sha + git show anchor from the thread above to close the working-tree forgery gap in SCAR-PROC-150. At this rate, by the time Part 4.1 drops, our agents will be governed by a 1976 Bell Labs Makefile, a 1980 Xerox PARC image log, and a 1983 SysV init.d directory—leaving the 2026 neural network as the only component in the stack under forty years old. Looking forward to reading Part 4.1!

Collapse
 
randalschwartz profile image
Randal L. Schwartz •

"At this rate, by the time Part 4.1 drops, our agents will be governed by a 1976 Bell Labs Makefile, a 1980 Xerox PARC image log, and a 1983 SysV init.d directory—leaving the 2026 neural network as the only component in the stack under forty years old."

That line is pure gold — with your permission, I'm putting that straight into the opening epigraph block of Part 4.1!

And it gets even wilder: today we ran our first two live production tickets (BlocSignal #260 and #259) on the compiled Smalltalk-80 control plane, AND built a working zero-dependency Dart evaluator (package:smalltalk_fabric) that parses Smalltalk-80 cascades, evaluates live [ :s | ... ] block closures against host methods, and serializes mid-flight thisContext stack continuations via #storeString so a failed tool call can be resumed in place (ex resume: replacementValue) without restarting the pipeline.

Can't wait to see ag_ops v2.2 when it lands!

Thread Thread
 
fgrandon profile image
Francisco Grandón •

Permission granted, gladly! I’m looking forward to Part 4.1, and I’m already working on v2.2.

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

Odinaka's line sums it up well: docs do not push back, the CLI does. Moving the rules from a Markdown file into something that returns exit code 2 is the change that makes them rules at all.

The four cracks are useful precisely because they are boring. Two subagents overwriting one receipt file and an empty directory getting a valid seal are the kind of bugs that only appear once real parallel work hits a tool, and they are easy to miss in a design review.

The thread with Randal about anchoring the witness to committed history, and then to the merge base of the protected branch in CI, is the clearest explanation I have read of why an agent sharing your UID cannot be its own auditor.

Do you plan to keep the human sign-off label as the only escape hatch, or will some renegotiations be allowed to pass with a mechanical witness alone?

Collapse
 
fgrandon profile image
Francisco Grandón •

Thanks for reading and for highlighting Odinaka's point, Ascendra. You hit the nail on the head: Markdown compliance is just a polite suggestion until the shell refuses to exit zero.

Regarding your question on whether the human sign-off label will remain the sole escape hatch or if mechanical witnesses alone will eventually clear renegotiations:

I am keeping the human gate mandatory for structural state overrides, but v2.2 introduces an automated "amnesty path" for low-severity drift. If a mechanical witness can prove via static AST inspection that a failed constraint was purely structural cosmetic churn (such as re-ordering imports or updating type hints) without altering execution paths, the CLI will auto-issue a cryptographically signed provisional waiver instead of outright blocking the agent.

The core philosophy remains that an autonomous system can propose its own bail-out, but it shouldn't hold the keys to the panic room. Appreciate the sharp read on the merge-base audit model

Collapse
 
emilywoodsnyc profile image
Emily Woods •

The DONE banner stealing the failure budget is my favorite bug in this write up. I'd keep it as a fixture with the actual assertion thousands of lines later, then assert that the distilled output still contains the assertion location. I'd also try the same log with ANSI colors and carriage return progress updates. A short summary is only useful if the formatter can't accidentally crowd out the failure.

Collapse
 
mona_d_4222dda374567263b profile image
Amber D. •

The most valuable thing in this write-up is that the release gate caught its own miner. A regret detector that mistakes a textbook TDD sequence for panic is a great demonstration of why deterministic checks need adversarial fixtures before anyone trusts them. Your per-commit paired_files fix and the is_pure_refactor gate also point at a general lesson: “lines that were deleted after being added” is a weak signal on its own, and it only becomes useful once you condition on what kind of line was removed (your control-guard check does exactly that).

A few thoughts and questions:

  1. The constructive-witness gate is mechanical, but is it semantic? spec-collision verifies that two file:line references exist, are in bounds, and are distinct. That’s a real improvement over prose, but an agent can still satisfy it by citing any two valid lines. The api-lock type has a similar property: a failing run that happens to print TypeError or ImportError for an unrelated reason would pass. It still raises the cost of lazy retractions and leaves an auditable trail, so I wouldn’t call it a flaw. But a cheap tightening might be to require that at least one cited location falls inside the files the test actually imports, or that the witness lines be recorded in the receipt’s collisions history so a reviewer can spot-check them later. Have you seen agents “game” the witness types yet in real runs?

  2. The red-side question from howcani. Blocking runner-level exit codes (2, 3, 4, 5, 126, 127) is a good filter for “failed to run” versus “failed an assertion.” The remaining gap is a test that fails with a legitimate assertion exit code but against the wrong invariant. One idea is to have witness-red record the distilled failure line (the E ... delta) in the receipt and have verify-green confirm that the originally failing assertion is the one that now passes, rather than only that the command exits 0. That would tie green back to the specific red.

  3. Regret-mine noise. The two false-positive sources you identified are both fixable the way you describe. Stripping the leading docstring Expr before hashing and qualifying names as Class.method should remove most of the 11-pair noise on schedule. I’d also consider hashing at the statement level and ignoring pairs where the only changed statements are string constants or comments, since “fix typo” commits are a classic source of this. It would also be interesting to report precision on that 11-pair sample (how many were real regrets after manual review), since that gives readers a concrete number for the miner’s usefulness.

  4. The latency table is refreshingly honest. Reporting both in-process and wall-clock numbers, and noting that ast.parse() dominates at ~1,800 lines, is more useful than the usual single benchmark. For the subprocess cost, a persistent mode or a small long-lived helper could remove the ~140 ms Windows startup if per-turn overhead ever matters, though at 150–185 ms it’s probably not worth the added complexity.

  5. The scar-ossification design is clever. Pinning a rule to an anchor file’s SHA-256 and downgrading to PROBATIONARY_REVERIFY on drift turns “this rule may be stale” into an explicit state rather than a silent false block. One thing to consider is expiry: if a rule stays probationary indefinitely, it effectively becomes advisory. A TTL or a count of consecutive probationary passes before it forces re-verification might keep that state from becoming a permanent soft spot.

Your threat-model note about the same-UID perimeter is the right framing: this targets autoregressive shortcuts, not hostile actors. Within that scope, moving the pain into exit codes and hashes instead of more prose rules is a sound approach, and the numbered list of what broke makes the whole thing more credible. Thanks for publishing the failures alongside the features.

Collapse
 
fgrandon profile image
Francisco Grandón •

Thanks for reading and for putting together such a sharp breakdown, Amber. Having the release gate flag its own miner was humbling, though at least the tool isn't playing favorites.

To your question on whether I have seen agents game the witness types in the wild: not maliciously, just via classic autoregressive laziness—grabbing the nearest valid file:line in context or riding an incidental ImportError across the finish line because it satisfies the syntactic shape of least resistance. Your idea of intersecting cited locations against the test file's AST import graph is the exact kind of cheap, structural tie-breaker that closes that gap without needing an LLM judge.

I am actively folding your observations into the v2.2 roadmap and will dedicate the third post in this series to testing these exact edge cases and reporting the empirical numbers:

Witness semantics & Red-Green causal binding: Testing the import-graph intersection for spec_collision, alongside persisting the distilled E ... failure delta in the receipt so verify-green confirms the exact assertion flip rather than just a zero exit code (and doesn't accidentally pass because the agent deleted the failing assert).

Regret miner precision: Alongside stripping leading docstring Expr nodes and qualifying Class.method scopes, I will test ignoring pure string-constant mutations to filter out typo commits, and I will publish the manual precision audit on that 11-pair sample so there is a hard baseline on signal-to-noise.

Probationary TTL: Adding a monotonic pass-count decay to PROBATIONARY_REVERIFY—because "temporary probation" in software has a suspiciously high correlation with "permanent architectural fixture."

As for the ~140ms Windows subprocess tax, I am holding off on a persistent daemon for now; the last thing a deterministic guard tool needs is its own zombie-process lifecycle to debug. Appreciate the push on these edges—stay tuned for Part 3.

Collapse
 
mp3sss31 profile image
mp3sss •

Receipt collisions and the 'renegotiate --reason' loophole are the two cracks I'd expect first — both are classic trust-boundary issues. The Merkle-sealed test tree parallels what desktop apps do with code-signing and tamper-evident update manifests: you can verify integrity, but you still need an authority that cannot be talked out of a decision. The non-zero-exit observation is a nice catch too — exit codes prove that a command failed, not that the invariant you cared about was the one being tested.

Collapse
 
fgrandon profile image
Francisco Grandón •

Thanks mp3sss for the read and the code-signing analogy, Comparing Merkle-sealed test trees to tamper-evident update manifests hits the exact architectural target: the core challenge is building an authority that is completely immune to prompt engineering charm.

The renegotiate --reason loophole is indeed where agents try to talk their way out of a lockdown.

Appreciate the sharp eye on trust boundaries. stay tuned for how we lock down that escape hatch.

Collapse
 
robertpaschal profile image
Odinaka Robert Nnamani •

the part where the move itself broke things is the useful bit. docs don't push back, the CLI does.