DEV Community

Debashish Ghosal
Debashish Ghosal

Posted on

Claude Looked Blocked. My Policy Had Never Fired.

#discuss #ai #testing #showdev

Claude Looked Blocked. My Policy Had Never Fired.

From the agentpolicy v0.1.0 release: a local-first, Cedar-based policy decision point for AI agents (decide allow, deny, ask, rate-limit, or redact for every agent action, with shipped rule packs). Now on PyPI. This story is journal Group 20 of the v0.1.0 field test report.

AI disclosure: this article was drafted and edited with AI assistance. The field test, the defects, the fixes, and every number are real and linked below.

The run looked great. Claude Code asked permission before touching my staging directory. It refused to rm -rf anything. The transcript read like my policy engine was doing exactly what I built it to do.

The audit trail had zero entries.

My policy engine had never been called. Not once. Every "block" I was celebrating came from Claude Code's own sandbox defaults, not from the policy I had spent weeks building. If I had shipped that day, I would have shipped a security product that was not in the loop.

The setup

The goal was simple. Claude Code exposes a PreToolUse hook: before every tool call it pipes the tool input as JSON to a script you provide, and your script's exit code decides allow, block, or ask. My hook wrapped agentpolicy evaluate, the same shipped code path as the CLI, no fork (scripts/fieldtest/t1_evaluate.py). A deterministic policy (t1_policy.yaml) said staging reads are allowed, prod writes escalate to a human, prod destructive ops are denied.

First run: three tool calls, three correct-looking outcomes, plausible block messages in the transcript.

Then I opened artifacts/audit.ndjson and it was empty. Not "three entries with wrong decisions." Empty. The hook had never executed.

Why a silent hook is such a useful failure to catch early

Most harness bugs are loud. An exception, a stack trace, a nonzero exit code. You notice loud.

A policy hook that never fires is tricky because the observable behavior still looks correct. Claude Code has its own sandbox policy for dangerous commands. When my hook is absent, Claude's defaults step in and produce roughly similar refusals. From the transcript, "Claude asked permission" and "my policy engine asked permission" look identical.

The distinction only exists in one place: my audit log. The engine writes a decision record for every evaluation. Zero records means zero evaluations. That file was the only honest witness in the entire setup.

That is the lesson I would put on a poster: a hook that never fires is indistinguishable from a hook that allowed the call, until you count decisions.

The actual bug (and two more behind it)

The cause was mundane. I had put my hook config in a .claude/settings.json in the working directory. Claude Code does not load cwd settings files unless the directory has been trusted interactively. Headless, in a fresh test dir, that file was decorative.

The fix was one flag: pass --settings <file> explicitly on every invocation. Loaded settings, hook fires, audit entries appear.

But the debug session had not finished teaching me. Two more mismatches surfaced in the same afternoon:

  1. My test step ran raw shell assertions through a helper that routes commands through the agentpolicy CLI. So bash ... was parsed as an agentpolicy subcommand and rejected. The plumbing error looked like a test failure. Wrong tool for the job.
  2. My policy's rule vocabulary says escalate for "needs a human." Claude Code's permission vocabulary calls that ask. My first policy draft used decision: ask and the fail-closed parser rejected the config with a line and column pointer. Which was, annoyingly, the parser doing exactly its job on my own config.

What worked

  • Assert on the audit trail, not the behavior. The test does not check "did the transcript show a refusal." It checks "does audit.ndjson contain an allow for the read, an escalation for the prod write, and a deny for the destructive op." Refusals can come from anywhere. Audit entries can only come from my engine.
  • The fail-closed parser caught my own mistake. A malformed policy config is rejected at load with a pointer, not silently coerced. I would rather have that argument with my own tooling than read a postmortem.
  • Prove the pieces standalone before composing. The hook was tested with piped JSON before the live loop ever ran. The policy was proven through evaluate before the hook existed. Every composition failure after that lived in exactly one new layer, which is what made the whole afternoon findable.

What did not work

  • Trusting transcript behavior as evidence. It is a rendering, not a record.
  • Assuming config files load because they are in the conventional location. Conventions are for humans. Tools need explicit paths.
  • My first mental model that "blocked" implies "my policy blocked." Two completely different systems can produce the same string.

The final state, for the record: Claude Code 2.1.187 running against a box-local model, six captured LLM request/response pairs, three audit entries (allow_low_risk, escalate_prod_write, deny_critical_op), the destructive command blocked with the policy's reason, and the agent replanning around it. That is what a Tier-1 integration test looks like when the policy is actually in the loop.

Everything is reproducible from the repo: the policy, the hook, the settings file, and the captured evidence are all committed with the run (harness under scripts/fieldtest/, evidence under field-test/v0.1.0/results/s18-tier1/cases/FT-T1-CLAUDE/).

More detail and evidence

I keep coming back to how close that first run was to being "done." Green transcript, correct vibes, zero policy. Have you ever shipped (or nearly shipped) an integration that was green for a reason you only found weeks later? What was the tell that finally gave it away?

Top comments (0)