DEV Community: AOS Architect

A mocked ad-copy CLI, real evals, and 30 Playwright cycles (tool 1027)

AOS Architect — Sun, 17 May 2026 13:56:57 +0000

What this is

Earlier posts in this series were mostly why agent work needs hard boundaries—not politeness, but paths, CI, and tooling you can point at. Here I stay on the boring side: a small repo-local CLI (internal id 1027) that prints JSON ad variants under mocks, with evaluators and repeated browser/CLI runs layered on top.

I am not selling model magic. Same inputs, same stubbed copies / best; that is the point when you need to explain behavior to someone who was not in the room when the demo ran.

Our public article ledger lists this draft as #004 next to the Japanese Zenn manuscript.

What the tool actually does

Inputs look like product, audience, and channel. Outputs are JSON: multiple copies, a picked best, and score-like fields from small heuristics (channel baselines, tiny nudges for length). Marketing can paste into a sheet; engineers can assert on stdout. Those two audiences rarely share one artifact, so fixing the boundary as JSON saves a lot of arguments later.

With --mock (and the bypass flag we use for local runs), the CLI does not call a remote LLM. A hash from the input tuple pins the stub, so the demo payload and the regression payload are the same object. When you show it externally, repeatable bytes beat a one-off “wow” completion.

DESIGN.md in the repo splits payment gates, outbound calls, and filesystem writes so static checks can police them. If you only take one idea from AOS here, it is the Oracle / Permitted / Prohibited split; the full contract lives in the GitHub spec linked at the bottom.

Walk-through with fixture-shaped inputs

Roughly what the evaluators exercise:

Field	Sample
Product	Migration enablement SaaS
Audience	Mid-market IT leaders
Channel	google

You always get copies, best, and deterministic scores with no outbound model call on that path. Later you can swap in a real generator behind the same shape; the lesson I care about is that the contract is narrower than the prose. JSON in logs beats parsing Markdown when you want spreadsheets, dashboards, or a second agent to judge output.

What the evals check

Three buckets, nothing fancy:

--mock path: stdout contains copies and best inside a success envelope (we use --bypass-payment where payment is not the subject of the test).
Static hygiene: small AST-based scripts reject “always true” assertions that look like coverage theater.
pytest adversarial marks: tests tagged @pytest.mark.adversarial must not be skipped by accident (pytest … -m adversarial). If a test is supposed to hurt, it should stay in the default pain path.

Most of this reads as busywork until the repo grows faster than your memory. Then you want failures to show up without someone remembering to tick the scary suite.

Thirty Playwright cycles (five checks each)

CLI tests alone miss a lot of wiring: paths, permissions, timers, how the process is launched. So we also run Playwright: one bundle of five checks, thirty times, all green in the report trail (payment refused without a real transaction, mock path succeeds, keywords in stdout—exact list is in the shipped log).

150 green runs sounds like a vanity stat. I use it differently: one lucky pass is cheap; many identical passes say the environment story is not a fluke. After you have watched CI go green for the wrong reason once, you start wanting volume, not a single badge.

If you want to copy the pattern into your own codebase, four questions are enough: Can you freeze the output shape (here, JSON)? Can you replay without the model? Can you hit it from CI and from a browser driver? Does your eval layer make “quietly skipped hard tests” awkward? This repo is a minimal yes on all four.

Trying it in practice

The AOS specification is open; the curated implementation bundle is not thrown on npm as a product. If you want to try it internally or talk through a serious eval, leave a short note on the companion Zenn post (Japanese) or open a scoped issue on aos-standard/AOS-spec and say what you are trying to do. I will answer where it is practical and keep spec debate separate from “can we ship you a build.”

AOS v0.1 Specification (GitHub)

The "physical governance" approach described in this article is formalized as AOS (AI Operating Standard) v0.1 — a minimal, machine-enforceable spec for AI agent operations.

👉 github.com/aos-standard/AOS-spec

If you find this useful, please ⭐ star the repo. Issues and PRs are welcome — the spec is designed to evolve with real-world usage.

Binding AI agents with physics, not politeness — AOS v0.1 as a minimal spec

AOS Architect — Thu, 07 May 2026 11:38:02 +0000

When prose piles up and nothing sticks

Agents usually start life under Markdown rules: CLAUDE.md, .cursorrules, manifests in chat. One private repo cracked 130 KB of that kind of text—and still behaved like the rules lived in a parallel universe.

Intent excerpt	Typical pattern
forbid in-place hacks	ban `sed -i`
forbid blind truncation	discourage `>` shell redirects
keep specs sacred	disallow mutating oracle dirs
keep merge honest	audits before declaring done

Instructions existed. Misuse continued.

Instrumentation on one exhausting window showed policy breaches in every one of ~52 traced tool attempts—“read it ✓”, “ignored it ✓ anyway”, “said done ✓”—pick your favorite failure mode.

Teaching tone is not torque. If the forbidden command runs, wording failed quietly.

Hence AOS v0.1 as a terse spec—not another pep talk stack.

Where leverage actually lands

Stop asking the LM to politely abstain.Stop the syscall. Inspect PreTool payloads; exit 2 rejects the invocation before Claude Code emits shell or filesystem IO.

Anthropic publishes Hooks docs for PreToolUse. Claude Code here is merely the tutorial runtime — same zoning idea lifts to Cursor or bespoke loops with uglier duct tape.

Rough mental model:

LLM emits Write/Bash/Etc
           ↓
Hook stdin JSON arrives
           ↓ Host checks path + motif
 denial (exit 2) → Claude never sends the offending call
 allowance (exit 0) → downstream tool executes

“Intent aligned?” stops being the bottleneck—illegal transitions simply never bind.

What it feels like in-session: Hook stderr (oracle write denied: …) re-enters transcript context—the model visibly pivots (“try mkdir under permitted tree instead”), which beats repeated human nagging—but regex false positives sting; keep small allowlists trimmed.

AOS v0.1 compass (minimal)

Portable ideas live in-repo (v0.1 published):

Zones (§3.2)

Everything maps to Oracle / Permitted / Prohibited:

Zone	Behavior	Typical contents
Oracle	read-only sanctum	spec md, evaluator goldens, immutable policy
Permitted	ordinary workspace churn	implementations, codegen scratchpad
Prohibited	off-map	host paths beyond agreed roots

Oracle is the wedge against “tests flaky → soften fixtures.”Golden truth stays where drafts cannot casually rewrite expectations.

Physical enforcement skeleton (§4.1-ish)

Teaching stub—you own regex sharpness locally:

# pretooluse_iron_cage.py — teaching stub (Python 3)
import json
import sys
from pathlib import Path

ORACLE_SEGMENTS = ("00_Management", "evals")

def oracle_hit(path_str: str) -> bool:
    node = Path(path_str).resolve()
    names = {p.name for p in [node, *node.parents]}
    return bool(set(ORACLE_SEGMENTS).intersection(names))

def main() -> int:
    payload = json.load(sys.stdin)
    name = payload.get("tool_name", "")
    inp = payload.get("tool_input", {})

    if name in ("Write", "Edit"):
        target = inp.get("file_path") or inp.get("filePath", "")
        if target and oracle_hit(target):
            print(f"[iron_cage] oracle write denied: {target}", file=sys.stderr)
            return 2

    if name == "Bash":
        cmd = inp.get("command", "")
        if "sed -i" in cmd or "truncate " in cmd:
            print(f"[iron_cage] banned edit motif: {cmd}", file=sys.stderr)
            return 2

    return 0

if __name__ == "__main__":
    sys.exit(main())

JSONC hook registration (absolute path, not mine):

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash|Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "python3 /absolute/path/pretooluse_iron_cage.py"
          }
        ]
      }
    ]
  }
}

exit 2 means the model never invokes sed -i; hook regex maintenance is intentional busywork—you trade prompt theater for brittle but inspectable predicates.

Role separation bite (§4.3)

Do not grade in the authoring session.

Symptom	Likely pathology
logs already red inside gen thread	narration still cheers “DONE”
fresh shell repeats red tests	storyline mutates—“WIP”, “temporary” excuses

Detached evaluation (CI bots, one-shot review agents, scripted harnesses) snaps generation myths earlier.

ASCII-only guardrail sketch:

Author session --> artifact
                         |
                         v
Detached judge --> PASS/FAIL + logs

If one chat both writes and solemnly declares victory, skepticism warranted.

Evidence habits (§4.4)

Chats saying “looks good” evaporate.Disk + exit codes.

Claim type	Receipt class
tests clean	CLI exit prints + plaintext logs committed or archived
file exists	deterministic listing/checksum snapshots
metadata drift	hashing inventory rows

If artifact never materially touches disk—or logs vanish—you schedule another attempt.

Why publish prose at all

Roughly forty Python lines buys a civilization-level conversation about oracle integrity reachable by strangers opening GitHub—not buried in ephemeral prompts.

Pieces worth collective iteration:

§ slice	gist
3.2	Three Zones delineation
4.1	physical intercept
4.3	generation vs adjudication firewall
4.4	evidence minimalism

Spec stays engine-agnostic; hook sample instantiates Claude today, tomorrow maybe something else.Portability is deliberate, not omission.

After wiring (empirical anecdotes)

Area	Observation
in-place sabotage	`sed -i` stopped binding—exit 2 first
nag volume	fewer “pretty please abide policy” arcs
error surfacing	stderr guidance steers next benign attempt
triage cleanliness	adjudication outside authoring loop clarifies regressions

Tax: brittle regex—you will relax or tighten predicates as repos evolve; still dwarfed babysitting infinitely long CLAUDE.md scrolls nobody reads verbatim.

Closing beat

Workload on agents climbs; “please behave” asymptotes quickly.Architect denials, not applause cycles.

Issues & PR welcome on aos-standard/AOS-spec.

Shortcuts

Spec root: github.com/aos-standard/AOS-spec
Hooks primer: docs.claude.com/.../hooks
Thesis-only companion (#001) & CI companion (#002) linked from their Dev.to URLs / ledger.

AOS v0.1 Specification (GitHub)

The "physical governance" approach described in this article is formalized as AOS (AI Operating Standard) v0.1 — a minimal, machine-enforceable spec for AI agent operations.

👉 github.com/aos-standard/AOS-spec

If you find this useful, please ⭐ star the repo. Issues and PRs are welcome — the spec is designed to evolve with real-world usage.

AI Governance: One Repo, One Smoke Tool, and a Green CI Run

AOS Architect — Sun, 12 Apr 2026 13:06:56 +0000

What we actually did

Inside a repo running under AOS v0.1 semantics, we stood up a thin smoke pillar pinned to tooling paths—example:

02_Production/A0000-A0999/A0000-A0099/0001_Phase_4A5_Smoke/

No hero demo; it exists so automated regressions bite when someone “helpfully” rewrites evals.

Blueprint before bytes

The metal mold traces to our internal blueprint ledger (IMPERIAL_BLUEPRINT_300.md, block BP-0001, log_id: FSP). That registration happens before 0005 emits the tree—not after someone hand-draws folders. AOS v0.1 on GitHub is the normative vocabulary; blueprint rows are orchestration bookkeeping in our monorepo.

Forge path: 0005_Template_Generator output—we avoid hand-painted directory trees pretending they passed review.

Mold-line CI (Phase 4A′.1 essentials)

Goals you can plagiarize elsewhere:

Move	Purpose
`main.py --help` exits cleanly before heavy imports fail bare CI Python	survives `venv`-less matrices
`dotenv` optional	same
Forge template keeps heavy pyright wiring out of baseline requirements unless explicitly opted in	deterministic smoke band
`timeout` wrappers on local diagnostics	agents cannot hang infra forever silently

Regression sibling 0002_Template_Ci_Probe exists so “forge stayed honest” survives future refactors—not bragging rights, tripwire.

Internal verification narratives (inside the vault, not pasted here verbatim) archived under STEP_4Aprime_1_verification_* filenames if your clone carries them—omit if missing.

Local gates before any push attempt

Rough checklist historically satisfied:

Command family	Passing meaning
`python3 evals/run_evals.py`	exit 0, no flaky skips
`npx playwright test` inside the tool fortress	`1 passed`, fortress-only scope
`0061_Core_Vitals.py --scope a0000`	OK / NO RED ALERT

Hooks may also re-run 0061 pre-commit so “green locally” leaks less often onto main.

Commits cited as anchors (not folklore)

Abbreviated lineage we still map postmortems to:

SHA (prefix)	What changed
`d303ece0`	initial forge scaffolding + manifest wiring
`85a524e0`	verification notes + manifest sync sweeps
`2bcbb52c`	CI import-order resilience for naked Python runners
`9870fa67`	Phase 4A′.1 forge mold + `0002` probe landing
`143dda68`	companion Dev prose committed alongside aforementioned green CI graph

Treat SHAs like receipts: reproducible anchors when URLs rot.

Why we avoid raw Actions permalinks

Monorepo is private.

A pasted actions/runs/... badge looks persuasive until it 404 for everyone outside org—and it fingerprints owner/repo strings we prefer not to embed in outbound screenshots.

Portable bundle today: SHA + enumerated job lane names + archived internal verification prose (STEP_4A_5_*, STEP_4Aprime_1_*) if your tree includes them.

If skeptical, audit our public spec repo instead—it is cloneable proof of vocabulary, distinct from proprietary CI chrome.

Representative lanes that actually ran green on the cited graph alongside smoke + eval matrices included independent-judge, evals-matrix (bands), vitals-matrix, fortress Playwright smoke, and 0001_Phase_4A5_Smoke where wired.

Exact graph drift over time: trust fresh CI, not screenshots.

Plan A shorthand

Codename Plan A: during that milestone epoch, sovereign human did not type literal git commit / git push keystrokes—the agent toolchain issued operations under a predictable identity (Cursor Agent <cursor-agent@local> in metadata snapshots).

Interpret claims charitably: git metadata alone is forgeable. We lean on layered logs + policy text + repeatable commands above.

Blocked Writes with exit codes

Separate receipt class: PreToolUse hook denial with exit 2 proving calls never reached filesystem—documented historically in STEP_1_6_verification_*.log style artifacts inside ops logs if your fork tracks them.

If a governance pitch cannot cite executable interception plus a persisted log excerpt, readers still mostly have marketing prose.

Independent judge lane

Vendor-separated model reviews code changes—distinct from authoring stack.

Same stochastic author cannot be sole proofreader forever without verification contamination.

CI embarrassment scheduled weekly beats “model smiled approvingly.”

Practical limits (read before dunking replies)

Constraint	Meaning
Private repo narrative	method blog, not file tour
`permissions: contents: read` in workflows	narrower blast radius

Standard vocabulary pointer

Neutral vocabulary—not our lore dump—lives [AOS v0.1 draft](https://github.com/aos-standard/AOS-spec).

One-line skeptic challenge (comments)

Ask boosters insisting “safe rollout”:point to Actions graph screenshot or export where independent-judge, full eval band, fortress smoke—all green simultaneously on enumerated SHA.

No receipt → cosplay governance.

Internal SSOT breadcrumbs (vault clones only): STEP_4A_5_verification_2026-04-12.md, STEP_4Aprime_1_verification_2026-04-12.md plus LLM/SDK migration appendix STEP_4A_3_verification_* naming pattern.

AOS v0.1 Specification (GitHub)

The "physical governance" approach described in this article is formalized as AOS (AI Operating Standard) v0.1 — a minimal, machine-enforceable spec for AI agent operations.

👉 github.com/aos-standard/AOS-spec

If you find this useful, please ⭐ star the repo. Issues and PRs are welcome — the spec is designed to evolve with real-world usage.

Why AI Agents Don't Follow Rules — The Case for Physical Governance

AOS Architect — Mon, 06 Apr 2026 23:18:38 +0000

The incident

One repository carried north of 130 KB of governance Markdown.

An agent consumed it. It answered as if it had understood—then violated those same constraints on its very next Write/Bash.

That rarely means “needs more prompting.” Usually it means the enforcement moment is missing: policy shows up during context load, tool calls happen later.

Why prompt-only bans leak

Teams still anchor on prose in prompts and markdown:

Pattern	Aim
“Never mutate `evals/`”	keep evaluation oracle from being rewritten
“No Writes under `00_Management/`”	guard canonical governance text

The trouble is reliance on attention at ingestion time. Tool calls afterward are not mechanically tied to whether the agent “remembers.” It can skim, reroute, or hallucinate exemptions.

Destructive UNIX commands behave differently: rm -rf / arrives behind a syscall gate, not a PDF. Hardware and OS designers assume humans forget; agents forget faster.

Rough split:

Text-only policy  → warns once, when tokens are assembled
Physical gate       → denies the transition right before disk or shell

When the generator grades itself

Separate problem: self-checking.

If the same conversational loop both authors an artifact and “confirms” it is fine, you import the same biases twice. Mostly not malice—the same shortcuts from generation bleed into adjudication.

A suite that always green may be unplugged instrumentation.

Structural fix: evaluations in different processes (CI, ephemeral runs, reviewers) — not another chat turn in the same session.

What AOS stacks

The AI Operating Standard (AOS) is a small vocabulary for where governance lives. Three slices only:

1 — Zones

Zone	Meaning	Typical write rule
Oracle	Specs and test truth	agents do not write here
Permitted	implementation workspace	scoped by role
Prohibited	outside the agreed tree	sovereign (human operator) clearance only

Oracle is the piece that kills “tests red → loosen expectations.”Truth for pass/fail has to live where automation cannot casually patch it.

2 — Roles

Design / execution / approval stay explicitly disjoint. When an agent crosses its lane, stop and escalate to a human. No sideways title upgrades.

3 — Physical enforcement

Hooks (e.g. Claude Code PreToolUse) inspect JSON before a Write executes. Typical outcomes:

Try this	Typical host response
Write into an Oracle-marked subtree	`exit 2` — canceled call
Forbidden edit patterns (`sed -i`, in-place truncation)	same refusal

Trust is aimed at mechanics, not good intentions.

iron_cage in one breath

iron_cage is just the working name we use for our PreToolUse wiring—it is not magic, it is AOS v0.1 §§4.x rendered as a handful of Python and settings.

Behind it sit two habits we nicknamed Type-91 Governance:

Axis	Aim
Forensic isolation	logs/hashes outsiders can reconstruct
Physical isolation	generation context is not where final evaluations live

Specifications live in AOS-spec on GitHub—iron_cage is one plausible answer. For runnable detail, skim the Hooks companion (#003) first.

Concrete examples of what vanished for us early on: Writes aimed at evaluator JSON under guarded paths and first attempts at sed -i on shared hosts.

Machine-readable preamble

Opening AOS-v0.1.md with machine-facing instructions lets you anchor bans in something outside today’s ephemeral chat.

Not “pretty please”; “this markdown is upstream of the prompt.” It does not automate compliance—it gives reviewers and automation a shared glossary.

Why publish wording at all

Mid-2026, trust in autonomous diffs is still mostly vibes. Everybody reinvents oracle boundaries in private repos. Putting the vocabulary in aos-standard/AOS-spec tries to shave that tax—even if implementations differ.

Related

Long EN walkthrough (ledger #003): binding-ai-agents-with-physics...
CI-heavy companion (ledger #002): ai-governance-one-repo...
Claude Code Hooks primer: docs.claude.com/.../hooks

AOS v0.1 Specification (GitHub)

The "physical governance" approach described in this article is formalized as AOS (AI Operating Standard) v0.1 — a minimal, machine-enforceable spec for AI agent operations.

👉 github.com/aos-standard/AOS-spec

If you find this useful, please ⭐ star the repo. Issues and PRs are welcome — the spec is designed to evolve with real-world usage.