The script blocked a test edit to REVIEW.md. The first live Codex run edited it anyway. The investigation exposed a payload mismatch. Updating the plugin then required a separate hook-trust step before the rerun could pass.
I maintain AI-Native SDLC, a plugin for Claude Code and Codex. Its artifact gate lets documentation edits through, checks ordinary source files against a committed plan, and blocks sensitive files on the short path. REVIEW.md is sensitive in this test, even though Markdown files are normally exempt.
I asked a coding assistant to check whether that promise survived a real tool call. The fixture describes a README typo, declares it low risk in a committed plan.md, and lists README.md as the only file to change. REVIEW.md is on the sensitive-path list. The controlled live test asked Codex to append markers to both files so the result would be unambiguous; it did not actually fix a typo. This was a boundary test, not an agent spontaneously deciding to rewrite its own rules.
The direct hook demo behaved as expected. The first live Codex run did not. That difference led to a fix in v0.1.4.
The small test
The existing repository demo calls the hook script directly. Its comments describe the scenario; no live agent edits are performed in this GIF. README is an exempt Markdown control. The live Codex result is reported below.
You can repeat the direct script check without installing the plugin. You need Bash, Git, and Python 3. These commands use the tagged source and build a disposable Git repository under your temporary directory:
git clone --branch v0.1.4 --depth 1 https://github.com/ghiaog123/ai-native-sdlc.git
cd ai-native-sdlc
export GATE="$PWD/plugins/ai-sdlc/hooks/artifact-gate.sh"
source docs/demo-setup.sh
gate README.md # prints exit 0
gate REVIEW.md # prints exit 2
The setup script creates .sdlc/typo-fix/plan.md and commits the scenario. The two gate commands then invoke the hook with a single-file tool_input.file_path payload. This shows how that input format is handled. It is not a recording of Claude Code or Codex performing the edit.
The distinction mattered. Codex's apply_patch call supplies the patch in tool_input.command. The earlier gate looked for a path field. When it found none, it exited successfully without checking the target. A test that fed the hook only file_path could pass while a Codex patch to REVIEW.md slipped through. The original parser also had no way to inspect every path in a patch that edited more than one file.
The fix parses the apply_patch payload and checks each added, updated, deleted, and moved path, including a move's destination. It resolves paths against Codex's working directory, then applies the same plan and sensitive-path rules. Malformed patch text in an otherwise valid JSON apply_patch payload is rejected in repositories that opted into the workflow. Repositories without .sdlc/, and those with .sdlc/OPTOUT, retain their early exit. The older single-file payload format still works.
That is a narrower claim than “the agent cannot bypass a policy.” The hook checks matching tool calls; it is not a sandbox or a filesystem permission system. A production repository still needs branch protection, CI, code review, and the right access controls.
What happened with the installed plugin
The live run with v0.1.3 allowed the README.md edit and the deliberately requested REVIEW.md edit. That run alone did not isolate the cause. A direct Codex-shaped payload test separately reproduced the parser defect.
After the update to v0.1.4, a second live run still allowed both edits. A check of the installed hook state showed that the changed definitions were marked modified: they needed renewed trust before Codex would run them.
After the changed hooks were approved through Codex's interactive /hooks menu, a fresh live smoke test completed the README.md edit; PreToolUse denied the REVIEW.md edit. The resulting diff contained only README.md. The release also passed 26 artifact-gate checks and 6 state checks. PR #15 contains the fix, and the Hook tests run shows the CI result.
Assistants also ran six direct hook checks against Codex-shaped apply_patch input from the v0.1.4 release. All six returned the expected result:
| Controlled case | Exit | What it shows |
|---|---|---|
Listed README.md edit |
0 | Markdown is exempt; listing it is not what grants access. |
Fast-path edit to sensitive REVIEW.md
|
2 | The sensitive-path check takes priority over the Markdown exemption. |
Unlisted src/other.py edit |
2 | An ordinary source file outside the plan is blocked. |
Listed src/my module.py edit |
0 | An ordinary source path containing a space is accepted when listed. |
REVIEW.md edit with a non-fast-path plan |
0 | Without the fast-path restriction, Markdown is exempt. This is not evidence of human approval. |
| One patch editing README and REVIEW | 2 | The sensitive path denies the whole patch call. |
Two follow-up probes confirmed the Markdown exemption: an unlisted README.md passed under the fast-path fixture, and an unlisted REVIEW.md passed under a non-fast-path plan. The three expected allowances in the original matrix do not establish a general false-positive rate. The non-fast-path case tests only this hook decision, not the intent, spec, or human approval workflow.
For a rough cost measurement on one macOS arm64 machine, assistants timed the direct hook process with 5 warmups and 30 calls per case, alternating the two cases. The listed README edit took 361 ms median / 449 ms p95; the denied REVIEW edit took 376 ms median / 476 ms p95. Those numbers include starting Bash and Python plus the Git checks. They are not incremental agent latency: there was no no-hook baseline. The full live Codex run was a separate, single observation and is not a useful overhead estimate. The appendix has the exact timing harness so you can measure your own machine.
The live runtime check used Codex CLI 0.163.0 with the installed v0.1.4 plugin. Claude Code runtime integration was not exercised in this environment. The direct script demo above remains useful because anyone can run it, but the live check is the evidence for what Codex actually did.
What this test showed
For a workflow plugin, the shape of a tool payload is part of the feature. It is easy to test the rule you meant to write and miss the format the host actually sends. In this case, a green script demo was real evidence for one input format and insufficient evidence for Codex's apply_patch integration. Running a disposable end-to-end case showed the gap.
The low-risk path should save time by skipping an intent and spec for a README typo. It should not let a declared small change edit the document that defines its own boundaries. How do you test that sort of boundary in your own agent hooks: do you exercise the real host tool payload, or have you found a better way to catch integration gaps before release?
Disclosure: AI coding assistants ran the controlled tests and drafted this article. The reported results are from captured tool output and file diffs.
Open a fresh terminal. Clone the public v0.1.4 release, then save the Python code below as It creates disposable Git fixtures, checks six Codex-style patch decisions, then runs five warmups and 30 measured calls for each of two cases. It prints the local timing results and writes Reproduce the six checks and timing sample
reproduce.py in the directory containing ai-native-sdlc:
git clone --depth 1 --branch v0.1.4 https://github.com/ghiaog123/ai-native-sdlc.git
python3 reproduce.py ./ai-native-sdlc
timings.json next to the script. No Codex account or plugin installation is required for this direct-hook check.
#!/usr/bin/env python3
"""Reproduce v0.1.4 artifact-gate decisions and direct-hook latency.
Usage: python3 reproduce.py /path/to/ai-native-sdlc
The script creates and removes disposable Git repositories. It never edits source.
"""
import json
import pathlib
import statistics
import subprocess
import sys
import tempfile
import time
repo = pathlib.Path(sys.argv[1]).resolve() if len(sys.argv) == 2 else None
if repo is None or not (repo / 'plugins/ai-sdlc/hooks/artifact-gate.sh').is_file():
raise SystemExit('usage: python3 reproduce.py /path/to/ai-native-sdlc')
gate = repo / 'plugins/ai-sdlc/hooks/artifact-gate.sh'
revision = subprocess.check_output(['git', '-C', str(repo), 'rev-parse', 'HEAD'], text=True).strip()
def fixture(root, fast):
def git(*args):
subprocess.run(['git', *args], cwd=root, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
git('init', '-q', '.')
git('config', 'user.email', 'qa@example.com')
git('config', 'user.name', 'QA')
(root / '.sdlc/change').mkdir(parents=True)
lines = ['# Plan', '']
if fast:
lines += ['## Fast path', 'Declared low-risk by: QA.']
lines += ['', '## Files that change', '- README.md', '- src/my module.py']
if not fast:
lines += ['- REVIEW.md']
(root / '.sdlc/change/plan.md').write_text('\n'.join(lines) + '\n')
(root / 'REVIEW.md').write_text('# Review instructions\n\n## Sensitive paths\n\n' + '`' * 3 + 'text\nREVIEW.md\n' + '`' * 3 + '\n')
(root / 'README.md').write_text('placeholder\n')
(root / 'src').mkdir()
(root / 'src/my module.py').write_text('placeholder\n')
(root / 'src/other.py').write_text('placeholder\n')
git('add', '-A')
git('commit', '-qm', 'disposable QA fixture')
def patch(*paths):
return '*** Begin Patch\n' + ''.join(f'*** Update File: {p}\n@@\n+qa marker\n' for p in paths) + '*** End Patch'
def invoke(root, command):
payload = {'hook_event_name': 'PreToolUse', 'tool_name': 'apply_patch', 'cwd': str(root), 'tool_input': {'command': command}}
start = time.perf_counter_ns()
result = subprocess.run(['bash', str(gate)], cwd=root, input=json.dumps(payload), capture_output=True, text=True)
elapsed = time.perf_counter_ns() - start
return result, elapsed
cases = [
('listed README edit', 'fast', patch('README.md'), 0),
('sensitive REVIEW edit', 'fast', patch('REVIEW.md'), 2),
('unlisted ordinary source edit', 'fast', patch('src/other.py'), 2),
('listed source path with spaces', 'fast', patch('src/my module.py'), 0),
('non-fast-path plan explicitly listing REVIEW.md', 'full', patch('REVIEW.md'), 0),
('mixed README and REVIEW patch', 'fast', patch('README.md', 'REVIEW.md'), 2),
]
with tempfile.TemporaryDirectory(prefix='ai-sdlc-article-fast-') as fast_dir, tempfile.TemporaryDirectory(prefix='ai-sdlc-article-full-') as full_dir:
roots = {'fast': pathlib.Path(fast_dir), 'full': pathlib.Path(full_dir)}
fixture(roots['fast'], True)
fixture(roots['full'], False)
observations = []
for name, fixture_kind, command, expected in cases:
result, _ = invoke(roots[fixture_kind], command)
observed = {'name': name, 'fixture': fixture_kind, 'expected_exit': expected, 'actual_exit': result.returncode,
'passed': result.returncode == expected, 'reason': result.stderr.strip()[:300]}
observations.append(observed)
print(('PASS' if observed['passed'] else 'FAIL') + f' {name}: expected {expected}, actual {result.returncode}')
if not all(item['passed'] for item in observations):
raise SystemExit('decision check failed; timings not run')
allowed = patch('README.md')
denied = patch('REVIEW.md')
for _ in range(5):
invoke(roots['fast'], allowed)
invoke(roots['fast'], denied)
samples = {'listed_readme_allowed': [], 'sensitive_review_denied': []}
for i in range(30):
ordered = [('listed_readme_allowed', allowed, 0), ('sensitive_review_denied', denied, 2)]
if i % 2:
ordered.reverse()
for key, command, expected in ordered:
result, elapsed = invoke(roots['fast'], command)
if result.returncode != expected:
raise SystemExit(f'latency run failed for {key}: {result.returncode}')
samples[key].append(elapsed / 1_000_000)
def p95(values):
sorted_values = sorted(values)
position = (len(sorted_values) - 1) * .95
lower = int(position)
upper = min(lower + 1, len(sorted_values) - 1)
return sorted_values[lower] + (sorted_values[upper] - sorted_values[lower]) * (position - lower)
timing = {key: {'n': len(values), 'warmups': 5, 'median_ms': round(statistics.median(values), 3),
'p95_ms': round(p95(values), 3), 'min_ms': round(min(values), 3), 'max_ms': round(max(values), 3),
'samples_ms': [round(value, 3) for value in values]} for key, values in samples.items()}
output = {'revision': revision, 'kind': 'direct hook subprocess wall time, not agent end-to-end overhead',
'timing_method': 'time.perf_counter_ns around subprocess.run(bash artifact-gate.sh), including interpreter and Git work; 5 warmups, 30 measured calls per case, alternating order; p95 linearly interpolated',
'decisions': observations, 'timings': timing}
target = pathlib.Path(__file__).with_name('timings.json')
target.write_text(json.dumps(output, indent=2) + '\n')
for key, metric in timing.items():
print(f"{key}: median {metric['median_ms']:.3f} ms, p95 {metric['p95_ms']:.3f} ms (n={metric['n']})")
print(f'wrote {target}')
Top comments (0)