DEV Community: Aoba Works

Claude Code Hooks Not Working: 8 Fixes and a Debug Script

Aoba Works — Sat, 07 Nov 2026 17:34:28 +0000

Originally published at Unattended.

Claude Code hooks not working? Run /hooks first: if the hook isn't listed, Claude Code never loaded it, so check the file location and JSON. If it is listed, the official docs are blunt: "If the hook appears but doesn't fire, the matcher is the usual cause." If the matcher is right, the problem is usually the exit code, the script's environment, or the way you start claude -p.

Which option is right for you?

For most Claude Code users, the best start is /hooks plus the debug script below. On a tight budget, stick to command hooks and that script, because only prompt and agent hooks call a Claude model. If nothing makes sense, run claude --safe-mode, then try a clean config, to isolate your customizations.

Why are my Claude Code hooks not running or firing?

Claude Code hooks usually fail to fire because the hook never loaded, the matcher misses, or you are watching the wrong event.

Work through the eight fixes in this order. The earlier ones are quicker to rule out.

#	Symptom	Likely cause	Fix
1	Not listed in `/hooks`	Wrong file, bad JSON or untrusted folder	Move it to a settings file and validate it
2	Listed, never fires	Matcher typo or wrong case	Use the exact tool name
3	Fires, doesn't block	Exit 1 or misplaced JSON	Use exit 2 or valid `hookSpecificOutput`
4	Stop hook keeps Claude going	No `stop_hook_active` check	Exit 0 when it is `true`
5	Notification stays silent	Expecting an alert on every response	Match a real notification type
6	`hook error` in the transcript	Permissions, path or missing `jq`	Run the script by hand
7	Works in the terminal, not in cron	Working directory	`cd` into the repo first
8	Works interactively, not in a script	`--bare`	Remove it for runs that need hooks

Also confirm the event itself. PreToolUse fires before a tool runs and PostToolUse after it, and a successful hook shows nothing in the transcript. A working hook and a dead one can therefore look the same, which is why the logging script later in this guide helps. If you're unsure which event you need, see our Claude Code hooks lifecycle guide.

An Edit|Write hook can also look broken when Claude changes a file through a shell command. The guide notes that Claude can modify files with shell commands, so audit hooks should also match Bash|PowerShell or scan the working tree from a Stop hook.

Where should hooks go in settings.json (and why is my JSON ignored)?

Put hooks under the "hooks" key in a Claude Code settings file; broken JSON makes Claude Code skip that file.

Location	Applies to	Common trap
`~/.claude/settings.json`	All your projects	`~/.claude.json` is a different file, and hooks don't belong there
`.claude/settings.json`	One project	Loads from the current directory with no parent-directory fallback
`.claude/settings.local.json`	One project, you only	Loads from the git root even when you start in a subfolder
Plugin `hooks/hooks.json`	While the plugin is enabled	Only plugins get a standalone hooks file

The second and third rows produce a confusing result. If you start Claude Code in repo/src, your local file can still load while the committed one doesn't.

The nesting must be event, then matcher group, then handler:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/check.sh",
            "args": []
          }
        ]
      }
    ]
  }
}

Settings files are strict JSON: a // comment or trailing comma is a syntax error. Interactive sessions show a Settings Error dialog for that. An individually invalid entry, such as an unknown event name, gets a Settings Warning instead, and the rest of the file keeps working. To check files without opening a session, run claude doctor.

Three other things can hide a valid hook:

Workspace trust. An interactive session holds back settings-file hooks until you accept the trust dialog for the folder, including hooks in your user file.
disableAllHooks. Precedence applies, so a project value can override your user value.
Managed policy. If /hooks shows Only hooks from managed settings run here, allowManagedHooksOnly is on and your own hooks won't run or appear.

Is the hook matcher case-sensitive? Why doesn't my matcher match?

Yes, Claude Code hook matchers are case-sensitive, so bash never matches the Bash tool.

Tool names are capitalized: Bash, Edit, Write, Read. The rest depends on how the matcher string is evaluated. *, an empty string or no matcher means match all. Strings containing only letters, digits, _, -, spaces, , and | are exact names or lists. Anything else is an unanchored JavaScript regular expression.

Matcher	Result	Use instead
`"bash"`	Matches nothing	`"Bash"`
`["Edit","Write"]`	Invalid entry; under `PreToolUse` or `PermissionRequest`, the file's other hooks don't load either	`"Edit\
{% raw %}`"Edit,Write"`	Matches both on v2.1.191 or later, nothing on older versions	`"Edit\
{% raw %}`"Edit.*"`	Also matches `NotebookEdit`	`"^Edit$"`
`"mcp__memory"`	An exact string that matches no tool	`"mcp__memory__.*"`

Not every event matches on a tool name. SessionStart matches how the session started, Notification matches its type, and Stop and UserPromptSubmit have no matcher support, so they fire every time.

The if field has its own trap. It only works on tool events, and adding it to any other event prevents the hook from running.

Exit code vs. stdout JSON: why is my hook output ignored or not blocking?

Claude Code treats exit 2 as a block and exit 0 as success; exit 1 is normally non-blocking, so the action still proceeds.

This table summarizes the hooks guide and hooks reference:

Exit code	Stdout	Outcome
0	One JSON object and nothing else	Fields are applied
0	Anything not starting with `{`	Treated as plain text, with no error shown
2	Anything	Blocks on events that can block; stderr is the reason unless JSON supplies one
Other	Valid JSON that passes validation	The JSON alone decides; the exit code is ignored
Other	Empty, plain text or invalid JSON	Non-blocking error; the action goes ahead

The current docs say a hook that prints JSON and exits 2 still blocks, and Claude Code still reads the JSON fields. They also advise choosing one approach per hook, and that keeps scripts easier to reason about. Use exit 2 with a stderr message for a simple block, or exit 0 with JSON for structured control:

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "deny",
    "permissionDecisionReason": "Use rg instead of grep"
  }
}

The silent failures usually come from one of these:

Shell profile output. An unconditional echo in ~/.zshrc can land ahead of your JSON. Search the debug log for Hook output does not start with {.
Wrong level. permissionDecision at the top level parses but is ignored. Search for Hook JSON output had unrecognized keys.
Too late to block. PostToolUse can't undo an action because the tool already ran.
Treating exit 0 as approval. On PreToolUse, exit 0 leaves the normal permission flow in place.
Timeouts. A timed-out PreToolUse hook doesn't block the call unless onFailure: "block" is set.

For copy-paste patterns, see our PreToolUse block scripts.

Why is my Claude Code Stop hook not working (and what is stop_hook_active)?

A Claude Code Stop hook fires when Claude finishes responding, not at task completion, and should exit 0 when stop_hook_active is true.

stop_hook_active is true when Claude is already continuing because of a stop hook. Without that check, your hook can keep sending Claude back to work until the safety cap applies: "Claude Code overrides a Stop hook after it blocks eight times in a row with no tool call from Claude in between."

Add this check to the top of the script. It's the guard from the official guide:

#!/bin/bash
INPUT=$(cat)
if [ "$(echo "$INPUT" | jq -r '.stop_hook_active')" = "true" ]; then
  exit 0  # Allow Claude to stop
fi
# ... rest of your hook logic

Then check these in order:

Remove any matcher. Stop ignores it, so it can't narrow the hook.
Block with exit 2 and a stderr message, or with decision: "block" plus a reason. The reason is required with that decision.
Read last_assistant_message from the input rather than parsing the transcript, because the transcript is written asynchronously and may lag.
Raise the cap with CLAUDE_CODE_STOP_HOOK_BLOCK_CAP only if your hook truly needs more than eight rounds.

Scheduled jobs have one more gap. Stop hooks don't fire on user interrupts, and API errors fire StopFailure instead. A cleanup script attached only to Stop will therefore miss a rate-limited run. Our Stop hook guide covers loop-safe patterns.

Why is my Claude Code Notification hook not working?

A Claude Code Notification hook fires only for specific notification types, such as permission_prompt and idle_prompt, not after every response.

Symptom	Cause	Fix
No alert when Claude finishes a reply	Notification isn't a reply-finished event	Use a `Stop` hook
Quick approvals never alert	`permission_prompt` needs the prompt to wait about six seconds	Leave the prompt open while testing
Exit 2 doesn't stop anything	Notification can't block; its exit code and stderr are ignored	Use it for side effects only
Matcher `"permission"` never hits	Values are exact types	Use `permission_prompt`, or omit the matcher
OS alert never appears	The command itself fails	Run it outside Claude Code first

Turning off desktop alerts isn't the cause. preferredNotifChannel changes how you're alerted, not whether your hook runs.

Test the notifier on its own before blaming the hook. On macOS, for example, run osascript -e 'display notification "test"'. On Linux, try notify-send test. If neither shows anything in your terminal, the hook won't fix that.

Unattended runs need a different plan. The docs don't list Notification behavior for each mode, but claude -p denies calls that need approval rather than prompting. Our inference is that you shouldn't rely on permission_prompt for cron alerts. Use Stop, StopFailure or a PermissionRequest hook instead.

How do I fix hook script permissions, paths and environment problems?

Claude Code can only run a hook script that is executable, findable and fast enough; check chmod +x, paths, jq and timeout.

Make the script executable with chmod +x ./my-hook.sh, and start it with a shebang such as #!/bin/bash.
Run it by hand with the official test, then read the exit code:

echo '{"tool_name":"Bash","tool_input":{"command":"ls"}}' | ./my-hook.sh
echo $?

Reference the script with an absolute path or ${CLAUDE_PROJECT_DIR}. Add "args": [] to use exec form, which starts it without a shell and needs no quoting. In shell form, wrap the path in double quotes:

{ "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/check.sh" }

If you see jq: command not found, install jq or parse with Python or Node.js. Under cron, use absolute tool paths inside the script, because cron's PATH is usually far smaller than your login shell's.
Compare your runtime with the default timeouts (as of November 2026). Set a timeout in seconds to override them.

Hook type or event	Default timeout
`command`, `http`, `mcp_tool`	10 minutes
`UserPromptSubmit`, `PreModelSwitch`, `PostModelSwitch`	30 seconds
`MessageDisplay`	10 seconds
`prompt`	30 seconds
`agent`	60 seconds
`SessionEnd`, any type	Shared 1.5-second budget, raised up to 60 seconds by a longer per-hook `timeout`

The SessionEnd budget catches many cleanup scripts. On Windows, a shell-form hook with no shell set runs in PowerShell when Git Bash is absent, so a Bash script may fail there.

Do Claude Code hooks work in headless mode (claude -p)?

Yes, Claude Code hooks run in claude -p, unless you add --bare, which skips hook discovery and ignores hooks passed through --settings.

The official wording is direct: "Hooks still don't run in bare mode when you pass them with --settings." The same page calls --bare the recommended mode for scripted calls and says it "will become the default for -p in a future release." So a job that depends on hooks could break after an upgrade. Check the release notes before updating that machine, and keep the test run below handy.

Headless behavior	What it breaks	Fix
`.claude/settings.json` loads from the current directory	Cron starts elsewhere and project hooks never load	`cd /path/to/repo && claude -p ...`
No dialog for broken settings	The file or value is skipped and the run continues	Run `claude doctor` after an odd run
No trust dialog; the folder counts as trusted	A cloned repo's hooks run on your machine	Review `.claude/`, or pass `--settings '{"disableAllHooks": true}'`
`PermissionRequest` still runs outside `dontAsk`	A call no hook decides is denied	Return a decision or add allow rules
Async hooks are killed at teardown	Background work ends with `cancelled`	Start a fully detached process

To see what a headless run actually does, use this example. Replace the path with your repo:

cd /path/to/repo && claude -p "List this folder using Bash" \
  --allowedTools "Bash(ls *)" \
  --output-format stream-json --verbose --include-hook-events \
  --debug-file /tmp/claude-hooks.log

Don't treat a missing event as proof of failure. Notification, SessionEnd, PreCompact and PostCompact never produce a hook_started event, even with that flag. To turn this into a scheduled job, follow our cron setup guide. For the flag and auth details, see the headless mode guide.

How do I reload hooks: do I need /hooks or a restart?

Usually neither: Claude Code normally reloads hook edits automatically, and /hooks is a read-only view rather than a reload button.

The settings docs say Claude Code watches your settings files and applies hooks edits to the running session. Advice that hooks only load at startup doesn't match the current documentation.

Save the file and wait a few seconds.
Run /hooks again. Rerunning it refreshes the view.
Select the hook to see its full command and where it's defined. Choose All events to see events with no hooks.
If nothing changed, restart the session, because the watcher may have missed the edit.

Two exceptions are worth knowing. Skill frontmatter hooks don't appear until the skill is invoked. And before v2.1.257, edits in a .claude/ folder created mid-session weren't detected, so run claude -v if you're on an old install.

How do I debug Claude Code hooks with a payload-logging script?

Log the JSON each command hook receives on stdin to see exactly what fired and what your script gets.

1. Save this as .claude/hooks/debug-log.sh. It writes nothing to stdout and always exits 0, so it won't make a decision for Claude Code:

#!/bin/bash
# Logs every hook payload. For temporary debugging only.
umask 077
LOG="$HOME/.claude/hook-debug.log"
payload="$(cat)"
{
  printf -- '--- %s pwd=%s project=%s\n' \
    "$(date -u +%FT%TZ)" "$PWD" "$CLAUDE_PROJECT_DIR"
  printf 'PATH=%s\n' "$PATH"
  printf '%s\n' "$payload"
} >> "$LOG"
exit 0

2. Make it executable with chmod +x .claude/hooks/debug-log.sh.

3. Register it in .claude/settings.local.json. With no matcher, it fires on every occurrence. Merge this into any existing "hooks" key, then add the same entry under SessionStart, PostToolUse, Notification, Stop and the event you're debugging if it's missing:

{
  "hooks": {
    "PreToolUse": [{ "hooks": [{ "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/debug-log.sh", "args": [] }] }]
  }
}

4. Trigger the behavior, then run tail -f ~/.claude/hook-debug.log in another terminal. Matching hooks run in parallel, so this logger sits alongside your real hook. Each payload includes common fields such as session_id, transcript_path, cwd, permission_mode and hook_event_name, and tool events add tool_name and tool_input.

What the log shows	What it means	Next check
No file, no entries	Hooks aren't loading	`/hooks`, the file location, `--bare`
`SessionStart` only	Your event didn't occur	Wrong event, or no tool call happened
`tool_name` differs from your matcher	Matcher miss	Copy the exact name
Event logged, your script has no effect	Script failure or wrong output	Run it by hand, then check the exit code
Different `pwd` or `PATH` under cron	Environment mismatch	`cd` first, use absolute paths

5. Read Claude Code's own log. Start with claude --debug-file /tmp/claude.log, or run /debug mid-session to enable logging and find its path. Plain claude --debug can look broken, since "The --debug flag doesn't print to the terminal." Its log goes to ~/.claude/debug/<session-id>.txt.

On v2.1.296 or later, search for Hooks: lines, such as Hooks: PostToolUse:Write [.claude/hooks/log-write.sh] finished with status 0 (31ms). Other endings include timed out after <N>ms and failed to start. For matcher counts, set CLAUDE_CODE_DEBUG_LOG_LEVEL=verbose.

In a live session, Ctrl+O opens the transcript view, where a non-blocking failure appears as a <hook name> hook error notice.

6. Clean up. Remove the entries and delete the log when you're done, since payloads can include prompts, commands and file contents. Command hooks run with your full user permissions, so read any hook you copy before installing it.

FAQ

Where is the Claude Code hooks documentation?

The hooks reference covers event schemas, output fields and debug logs. The hooks guide covers setup and troubleshooting. For settings problems, use Debug your configuration.

Do Claude Code hooks run in parallel and what is the default timeout?

Yes, matching hooks run in parallel, and an identical handler defined in several settings files runs once. Command hooks default to 10 minutes as of November 2026, with shorter defaults on some events. Override them per hook with timeout, in seconds.

Can a hook block a tool call before it runs?

Yes. Use PreToolUse with exit 2 or permissionDecision: "deny". A deny holds even in bypassPermissions mode, with one documented exception: on machines without managed settings or a Team or Enterprise /login, an installed mod can approve a call your hook blocked. An "allow" from a hook doesn't override deny rules.

Why does my hook run but Claude ignores its output?

Claude never sees stderr from a hook that exits 0, and most events don't add plain stdout to Claude's context. Return hookSpecificOutput.additionalContext, nested correctly, because a top-level additionalContext is silently ignored. Output over 10,000 characters is saved to a file, and Claude gets the path plus a 2,000-character preview.

Next steps

If you'd rather not dig through logs yourself: run /debug followed by your issue in a session. It turns on debug logging and prompts Claude to diagnose the problem, while /doctor proposes fixes and asks before applying them.

Claude Code is included in paid plans: Pro is $20 billed monthly or $17 a month billed annually, Max is $100 a month, and a Team standard seat is $25 billed monthly or $20 billed annually (as of November 2026; check the official page). Pro suits a solo builder with one or two automations, while Max makes more sense only if you already hit Pro's usage limits. Neither suits someone who wants a person to take responsibility for the fix, and Claude's diagnosis can be wrong, so test the result with the manual command above.

Related guides: Hooks lifecycle, Stop hook without loops, Cron job setup.

Sources

The linked claims above come from these official pages, read on November 8, 2026.

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

AI Agent vs Automation: A 5-Question Test for Claude Code

Aoba Works — Fri, 06 Nov 2026 17:15:53 +0000

Originally published at Unattended.

The difference between an AI agent and automation comes down to who picks the next step. With automation, you write every step ahead of time and the same input always takes the same path. With an AI agent such as Claude Code, the model looks at the situation and chooses its own next action. Anthropic's engineering guide puts it this way: workflows are systems "orchestrated through predefined code paths," and agents are systems "where LLMs dynamically direct their own processes and tool usage" (Anthropic, Building Effective Agents). Here is the practical test. If a task needs judgment at points where the right path depends on messy input, use an agent at those points. If the task is just steps on predictable input, write a script.

Which option is right for you?

Most solo builders should start with a script. Add a Claude Code agent step only where the script keeps failing on inputs it can't predict.

Your situation	What to pick first	First page to open
Beginner, never automated anything	A plain shell script or cron job. You can read every line, and debugging is easy	Claude Code Cron Job Setup
Daily dev work (PR reviews, failing tests, triage)	A Claude Code agent in headless mode (`claude -p`) with a short tool allowlist	Run Claude Code programmatically
Work that lives on GitHub (issues, PRs)	The Claude Code GitHub Action, triggered by `@claude` or a schedule	Claude Code GitHub Actions
Tight budget, fixed and structured inputs	A script, with no model calls. Each run costs nothing beyond compute	Make: when to use AI agents
Mostly fixed steps plus one fuzzy decision	A hybrid: script around a single `claude -p --json-schema` call	The "How do you combine both?" section below
Unattended runs overnight	A headless agent with strict permissions and retry handling	Permissions headless template

One reason for each pick. Scripts fail loudly and in the same place every time, so they suit beginners. Agents are worth their cost when inputs vary, as with diffs, logs and tickets. The GitHub Action saves you from writing your own server. A hybrid keeps the model's freedom limited to one step.

What Is the Difference Between AI Agent and Automation?

Automation runs steps you defined in advance. An AI agent like Claude Code decides which steps to run, based on what it observes as it works.

Under the hood, Anthropic describes the building block of agentic systems as an LLM enhanced with retrieval, tools and memory (Building Effective Agents). Claude Code is that building block packaged as a tool. It reads files, runs shell commands, edits code, and loops until it decides the job is finished. A cron job that runs rsync never decides anything. It does exactly what the line says.

	Scripted automation	AI agent (e.g. Claude Code)
Who chooses the next step	You, in code	The model, at runtime
Same input → same output?	Yes	Not guaranteed
Handles inputs you didn't anticipate	No, it errors or does the wrong thing silently	Often, by reasoning about them
Cost per run	Compute only	Model tokens plus compute (`--output-format json` reports `total_cost_usd`, per the headless docs)
Speed	Milliseconds to seconds	Seconds to minutes
Failure mode	Crash at a known line	Plausible but wrong action, or errors that compound
Debugging	Read the code	Read the transcript

For example, a script that moves invoices from an inbox to a folder works perfectly until a vendor sends a PDF named scan001.pdf with no invoice number in the subject line. An agent can open the PDF, find the number, and file it. It can also misread it. Both outcomes are possible, and that is the trade.

Anthropic is direct about the cost: "agentic systems often trade latency and cost for better task performance" (Building Effective Agents). The same article warns about the "potential for compounding errors" in autonomous runs, which is why agents need guardrails that scripts don't.

AI Agent vs Automation Workflow: Which One Fits Your Task?

A task needs a Claude Code agent when it requires branching judgment on varied or unstructured input. If it only needs repeatable steps on structured input, a script fits better.

Here are the five questions. Answer them for one specific recurring task, not for "my work" in general.

Can you write the steps down completely today? If yes, and they cover at least nine out of ten real cases, write a script. If you keep writing "it depends", that is where judgment lives.
Is the input structured? JSON from an API, a CSV with fixed columns, and a database row are structured. A bug report, a stack trace, a git diff or a customer email is not. Unstructured input points toward an agent.
Does the number of steps vary? Anthropic recommends agents for "open-ended problems where it's difficult or impossible to predict the required number of steps" (Building Effective Agents). Fixing a failing test might take one edit or twelve. Syncing two APIs always takes the same calls.
What does a wrong answer cost? If a mistake is cheap and reversible, such as a draft comment or a label, an agent is fine. If it is expensive or permanent, such as a payment, a deleted record or a production deploy, keep that step scripted, or let the agent only propose and have a human approve.
Does it run often enough that the cost per run matters? A job that runs every 5 minutes pays model costs 288 times a day. If a script can do it, the script wins on budget.

How to score it: two or more answers that point to an agent (no to Q1, unstructured for Q2, variable for Q3) means an agent step is justified. A high-cost answer on Q4 means that step stays scripted or gated, whatever the other answers say.

Example, labeled as an illustration. "Every morning, check the CI failures from last night and open a fix PR." Q1: no, because every failure is different. Q2: unstructured logs. Q3: variable. Q4: medium, since a PR is reviewable before merge. Q5: once a day. The result is an agent, with the PR as the human checkpoint. Compare "Every morning, back up the Postgres database to S3." Q1: yes. Q2: structured. Q3: fixed. Q4: high. The result is a script.

Common mistake: running the test on the whole pipeline instead of on each step. "Triage support tickets" fails the test as one unit. Split it into fetch, classify, route and reply, and usually only "classify" needs an agent.

AI Agent vs Agentic AI vs Automation: How Do the Terms Relate?

Automation is the umbrella. An AI agent such as Claude Code is one kind of automation that makes decisions. "Agentic AI" describes systems built from agent-like parts.

Anthropic's own vocabulary is the clearest anchor. Its guide says "we categorize all these variations as agentic systems," then splits them into workflows and agents (Building Effective Agents). So in Anthropic's usage, "agentic" covers both LLM workflows (fixed code paths that call a model) and true agents (the model directs the path).

Term	What it means in practice	Example
Automation	Any process that replaces manual work, AI or not	Cron + bash backup, a Zapier zap, an RPA bot
LLM workflow	Fixed code path with model calls at defined steps	Script that sends each ticket to Claude for a label, then routes by label
AI agent	Model decides its own steps and tool use in a loop	`claude -p "Run the test suite and fix any failures"`
Agentic AI / agentic system	Umbrella for the two above, and for multi-agent setups such as an orchestrator with workers	Claude Code spawning subagents for parallel research

Vendors stretch these words. Some product pages call any chatbot with a webhook "agentic AI". When you see the term, ask one question: who chooses the next step, the code or the model? That answer tells you more than the label does.

Is AI Part of Automation, or Separate From It?

AI is one way to automate, and Claude Code is one AI tool among several. Automation is the goal, and AI, scripting and RPA are methods for reaching it.

The methods sit side by side:

Rule-based scripting: bash, Python, cron, CI pipelines. You write every rule yourself.
RPA (robotic process automation): UiPath defines it as software robots that "handle repetitive, rule-based tasks like entering data, moving files, or processing transactions" (UiPath). It is still rules, applied through user interfaces.
No-code workflow tools: Zapier, Make and n8n. These are rules you build visually, often with an optional AI step.
AI agents: Claude Code, the Agent SDK. The model makes the decisions.

UiPath now describes RPA as "the execution layer" under AI agents (UiPath). That reflects the shape most teams end up with: deterministic tools do the actions, and AI decides which action to take.

For example, think of a solo SaaS founder's weekly churn report. A scheduled script pulls cancellations from Stripe's API, which is plain automation. A Claude call reads the free-text cancellation reasons and groups them into themes, which is AI used inside automation. The script then emails the summary, which is automation again. Calling this either "AI" or "automation" would be only half right.

AI Agent Automation Examples: Which Tasks Need an Agent?

Tasks with varied input need an agent. Claude Code agents beat scripts when the fix depends on what the agent finds, as in debugging, code review, triage and research.

These come straight from the patterns in Anthropic's docs:

Fixing failing tests. The headless docs show claude -p "Run the test suite and fix any failures" --allowedTools "Bash,Read,Edit" (headless docs). No script can know ahead of time which file to edit.
Explaining build errors. cat build-error.txt | claude -p 'concisely explain the root cause of this build error' > output.txt (headless docs). Error logs are unstructured by nature.
Pull request review. The GitHub Action's review workflow posts inline comments on each issue it finds, and it skips drafts and PRs it judges not to need review (GitHub Actions docs). "Judges" is the key word there.
Issue-to-PR. Comment @claude fix the TypeError in the user dashboard component on an issue and the Action works on it (GitHub Actions docs).
Commit messages from staged changes. The docs scope this with --allowedTools "Bash(git diff *),Bash(git log *),Bash(git status *),Bash(git commit *)" (headless docs).

Pitfalls and fixes:

The agent stops partway through an unattended run. In -p mode with Claude Code v2.1.302 or later, runs start in Manual mode when nothing sets a permission mode, and a call that needs approval is denied (headless docs). Fix: allowlist exactly the tools the job needs. Why Claude Code stops mid-automation walks through the causes.
The agent waits for an approval nobody will give. Add --permission-prompts none (requires v2.1.259 or later). Prompts are then denied and Claude is told not to retry them (headless docs).
The agent does too much. Put limits on it: --max-turns, a tool allowlist, and written rules. See the guardrails example.
The agent behaves differently on another machine. Use --bare. It skips the hooks, plugins, MCP servers and CLAUDE.md files found on the host. Note that bare mode needs ANTHROPIC_API_KEY and doesn't use your subscription login (headless docs).

Who agents are not for: anyone who needs byte-identical output on every run, such as compliance exports or invoice numbering. Agents also cost money on every run, and that cost varies with codebase size and task complexity (GitHub Actions docs).

AI Automation Examples: Which Tasks Should Stay Scripted?

Backups, syncs between two APIs, templated emails and scheduled reports from fixed queries should stay scripted. They need no judgment, and a script is cheaper, faster and repeatable.

Make's own guidance says it plainly: for jobs like notifications and record updates, "classic automation is faster, cheaper, and easier to maintain" (Make). Make lists the conditions where agents don't fit: rules that are fixed and stable, inputs that are clean and structured, and speed and consistency that matter more than judgment.

Examples where a script is the better tool:

Task	Why a script wins
Nightly `pg_dump` to S3	Same command every night, and a wrong action is costly
Copy new Stripe customers into your CRM	Structured JSON on both ends, with a fixed field mapping
Send a welcome email from a template	No decision at all
Rotate logs older than 14 days	`find -mtime +14 -delete` is one line and has no edge cases
Bump a version and tag a release	Deterministic, and reproducibility matters

Pitfalls and fixes:

Putting a model call into a task that has no decision in it. This adds cost and latency and makes the result less consistent. Fix: delete the model call and see whether anything gets worse.
Using a session-scoped /loop for something durable. Recurring /loop tasks expire 7 days after creation and only fire while the session runs (scheduled tasks docs). Fix: use system cron, a Desktop scheduled task, a cloud Routine, or GitHub Actions. Cloud Routines have a 1-hour minimum interval, while Desktop tasks and /loop go down to 1 minute (as of October 2026, per the same page).
Assuming GitHub scheduled workflows run forever. In public repositories, GitHub disables a schedule after 60 days without repository activity (GitHub Actions docs). Fix: add a check that alerts you when a run is missing, or schedule the job on a machine you control.

Who scripts are not for: tasks whose inputs change faster than you can update the rules. If you keep patching regexes for new edge cases every week, that is the sign to move that one step to an agent.

Using AI for Automation: How Do You Combine Both?

Use a hybrid. Scripted automation does the deterministic steps, and a single Claude Code call sits at the one decision point that needs judgment.

Anthropic's guide describes this shape. "Prompt chaining" splits a task into a sequence of steps, and "routing classifies an input and directs it to a specialized followup task" (Building Effective Agents). It lists five workflow patterns in total: prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer. For solo builders, routing is the most useful one: the model labels the input, and your code acts on the label.

Steps to build one (the example below is an illustration of support-ticket triage):

Write the deterministic parts first. Fetch the input and save it to a file. Piped stdin to claude -p is capped at 10MB, and for anything bigger the docs say to write it to a file and reference the path (headless docs).
Make the model return a fixed shape. Use --output-format json with --json-schema. The result arrives in the structured_output field (headless docs). An enum in the schema keeps labels to a closed set your script can branch on.
Give the agent step the fewest tools possible. A classifier needs Read and nothing else.
Branch in code, not in the prompt. jq reads the label, and case decides what happens next.
Check the exit code. Claude Code "exits with code 0 on success and a non-zero code when the run fails" (headless docs).
Schedule it with cron. See Claude Code Cron Job Setup.

#!/usr/bin/env bash
# Example: hybrid triage. Script steps around one agent step.
set -euo pipefail

./fetch_new_tickets.sh > /tmp/tickets.json   # deterministic (your own script)

schema='{"type":"object","properties":{"items":{"type":"array","items":{"type":"object","properties":{"id":{"type":"string"},"label":{"type":"string","enum":["refund","bug","other"]}},"required":["id","label"]}}},"required":["items"]}'

out=$(claude --bare -p "Read /tmp/tickets.json. Label each ticket refund, bug, or other." \
  --allowedTools "Read" \
  --output-format json \
  --json-schema "$schema") || { echo "agent step failed" >&2; exit 1; }

echo "$out" | jq -r '.total_cost_usd'          # log spend per run
echo "$out" | jq -c '.structured_output.items[]' | while read -r t; do
  case "$(echo "$t" | jq -r .label)" in
    refund) ./route_to_billing.sh "$t" ;;      # deterministic
    bug)    ./open_issue.sh "$t" ;;            # deterministic
    *)      ./send_template_reply.sh "$t" ;;   # deterministic
  esac
done

Common mistakes with hybrids:

Letting the agent take the actions too. If Claude both classifies and issues the refund, you have handed a high-cost Q4 step to the model. Keep actions in your own scripts.
Ignoring rate limits. A cron job that hits a limit at 3 a.m. fails silently unless you handle it. The stream emits system/api_retry events with fields like attempt and retry_delay_ms (headless docs). See the rate limit retry setup.
Not tracking token spend. total_cost_usd is a client-side estimate and can differ from your bill (headless docs). Log it anyway, and read how to see token usage and reduce it.

AI Agent and Automation Course: Is It Worth Taking One?

You usually don't need a paid course. Anthropic's Claude Code documentation, the Building Effective Agents article and the cookbook cover the core material at no cost.

What the free material covers:

Concepts and patterns: Building Effective Agents, published December 19, 2024. It defines workflows and agents and the five patterns.
Running agents headless: the headless docs and the GitHub Actions docs, with copy-paste commands and YAML.
Code recipes: the anthropics/claude-cookbooks repo, "a collection of notebooks/recipes" that includes tool use and agent examples.
Structured lessons: the Anthropic Academy course catalog lists more than 20 courses, including Claude Code 101, Claude Code in Action, Introduction to subagents and Introduction to agent skills. The catalog page doesn't show prices. Third-party roundups describe the courses as free with a certificate (Analytics Vidhya), so confirm on the course page before you enroll.

When a course helps: you learn better with a sequenced curriculum and exercises than by reading reference docs. When it doesn't: you already know the problem you want to solve. Building that one hybrid script teaches you more than a general course will.

Pitfalls when choosing a third-party course: tools change quickly. The headless docs alone mention behavior changes at several 2.1.x versions. A course recorded a year ago may teach flags that have since been renamed. Check the recording date. Be wary of any course that promises income from "AI automation agencies". That promise is about marketing, not about the skill you are trying to learn.

Next steps

Do it yourself today:

Pick one recurring task and run the five-question test on each of its steps. Write the answers down.
If every step comes out "script", write the script and schedule it with cron. You're done, and no model is involved.
If one step comes out "agent", try that step alone first: claude -p "<your instruction>" --allowedTools "Read" --output-format json. Then wrap it with the hybrid script above.

If you'd rather not build it yourself:

Claude Pro: $20/month billed monthly, or $17/month with annual billing ($200 up front). Claude Code is included (Claude pricing, as of October 2026; check the official page). It suits a solo builder running a few agent jobs a day. It doesn't suit --bare scripts, which need an API key instead of your subscription login (headless docs).
Claude Max: from $100/month, billed monthly (Claude pricing, as of October 2026; check the official page). It suits heavy daily agent use. It is overkill if your agent step runs once a day.
Claude Code GitHub Action: no server to run, and setup is /install-github-app (GitHub Actions docs). It suits repo-centric work. It doesn't suit local-only files, and it uses GitHub Actions minutes as well as API tokens or your subscription.
Anthropic Academy courses: structured lessons at anthropic.skilljar.com. They suit people who like guided exercises. They won't build your specific pipeline for you.

Related guides on this site:

FAQ

What is AI agent and automation?

Automation is any setup that does manual work for you by following rules you wrote. An AI agent is a kind of automation where a model such as Claude decides its own next steps and which tools to use, in a loop. You can combine them: scripts for the fixed steps, an agent for the judgment calls.

What is an AI agent vs automation?

A scripted automation takes the same path every time for the same input. An AI agent may take a different path each run, depending on what it finds. Anthropic describes agents as systems where the model maintains "control over how they accomplish tasks" (Building Effective Agents). Choose based on whether the task needs judgment.

AI agents vs automation: what are people saying on Reddit?

This guide doesn't summarize Reddit threads, because forum opinions can't be sourced the way documentation can. When you read them, judge each claim with the five-question test. Ask whether the commenter is describing structured or unstructured input, and whether a wrong action in their case could be undone. Claims about limits or pricing should be checked against the vendor's official page.

Automation vs AI: which term should I use?

Use "automation" for the goal and "AI" for the method. A cron job is automation without AI. A script that calls Claude to classify text is AI-assisted automation. Say "agent" only when the model chooses the steps, which avoids the vague way "agentic" often gets used.

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code PreToolUse Hook: Copy-Paste Block Scripts

Aoba Works — Thu, 05 Nov 2026 23:29:51 +0000

Originally published at Unattended.

A Claude Code PreToolUse hook is a script that inspects each tool call and can allow, deny, or escalate it. The docs say it "Runs after Claude creates tool parameters and before processing the tool call." To block a risky command safely, register a Bash matcher, read tool_input.command from stdin, and either print a JSON deny with exit 0 or exit 2.

Which option is right for you?

Claude Code PreToolUse setups differ by how much failure you can tolerate, so start from your situation.

Situation	What to pick first	First page to open	Why
Beginner, first hook	Short `exit 2` script	Hooks guide	Fewest moving parts, and the block message is plain stderr
Daily work, recurring headless runs	JSON `deny`, `onFailure: "block"`, and permission deny rules	Headless permissions template	A crashed hook blocks the call instead of letting it through
Budget first	Local `command` hook, not a `prompt` hook	Token usage guide	Prompt hooks send input to a Claude model; a shell script doesn't
You need a hard filesystem or network boundary	Sandbox plus hook	Sandbox guide	Pattern matching can miss a command; OS-level limits still apply
Running on a repo you didn't write	`--bare`, or hooks off for that run	Trust table	`-p` runs that repo's hooks without a trust dialog

What is the Claude Code PreToolUse hook and when does it run?

The Claude Code PreToolUse hook runs after Claude builds a tool call and before that call executes, whatever the permission mode.

Docs say it fires before any permission-mode check, in every mode. The one exception is EndConversation, which skips PreToolUse and PostToolUse. The nearby tool events do different jobs:

Event	When it fires	Can it stop the call?
`PreToolUse`	Before a tool call executes	Yes, on every matched call
`PermissionRequest`	When a call needs a permission decision	Yes, but through its `decision` object; exit 2 isn't honored
`PostToolUse`	After a call succeeds	No, the tool already ran

Use PreToolUse for a block list, because it sees calls that an allow rule would otherwise wave through. A hook that exits 2 stops the call before permission rules are evaluated.

What JSON does a PreToolUse hook receive on stdin?

A Claude Code PreToolUse hook receives one JSON object on stdin, containing the common fields plus tool_name, tool_input, and tool_use_id.

This sample Bash call is assembled from the hooks reference; the values are examples:

{
  "session_id": "abc123",
  "transcript_path": "/home/you/.claude/projects/demo/session.jsonl",
  "cwd": "/home/you/project",
  "permission_mode": "default",
  "hook_event_name": "PreToolUse",
  "tool_name": "Bash",
  "tool_input": {
    "command": "npm test",
    "description": "Run test suite",
    "timeout": 120000,
    "run_in_background": false
  },
  "tool_use_id": "toolu_01ABC123..."
}

tool_input changes with the tool, so read the field that belongs to it:

Tool	Fields in `tool_input`	Field to check for blocking
`Bash`	`command`, `description`, `timeout`, `run_in_background`	`command`
`Read`	`file_path`, `offset`, `limit`	`file_path`
`Edit`	`file_path`, `old_string`, `new_string`, `replace_all`	`file_path`
`Write`	`file_path`, `content`	`file_path`

Two details trip up scripts:

permission_mode can be default, plan, acceptEdits, auto, dontAsk, or bypassPermissions. Manual mode arrives as default, so never compare against "manual".
description and content are free text written by the model. Match on command or file_path, and always quote shell variables.

How do you allow, deny, or ask from a PreToolUse hook?

A Claude Code PreToolUse hook decides by exiting 0 with permissionDecision JSON on stdout, or by exiting 2 to block with stderr as the message.

The deny form looks like this:

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "deny",
    "permissionDecisionReason": "Force pushes are blocked. Do not retry; note it in your report."
  }
}

Here is the full contract, drawn from the hooks reference unless linked otherwise:

Exit code	Stdout	What happens on PreToolUse
0	Empty	The hook makes no decision, so the normal permission flow applies
0	JSON `allow`	Skips the prompt, but deny and ask rules are still evaluated
0	JSON `deny`	Blocks the call and shows the reason to Claude
0	JSON `ask`	Prompts the user, or denies in a `-p` run where nobody can answer
0	JSON `defer`	Ends the turn for later resume, in `-p` only, and only with a single tool call in the turn
2	Anything	Blocks; the message is stderr, and even a JSON `allow` can't override it
1 or other	Anything	Non-blocking error; the call goes ahead with a `hook error` notice
0	Invalid JSON	Non-blocking parse or validation failure; the call goes ahead
None	Timed out	Doesn't block, unless the hook sets `onFailure: "block"`

The docs are blunt about exit 1: "If your hook is meant to enforce a policy, use exit 2." A script that calls exit 1 on a dangerous command logs a complaint and lets the command run.

Other rules worth keeping nearby:

Pick one style per hook. The docs say to use exit codes alone, or exit 0 and print JSON.
Conflicting hooks. Precedence is deny > defer > ask > allow.
Old field names. Top-level decision and reason are deprecated for this event, with approve and block mapped to allow and deny.
Stopping the whole run. The universal field continue: false stops Claude entirely and takes precedence over event-specific decisions. Reserve it for commands where carrying on would make things worse.

For a first experiment, this exit-code version is enough:

#!/usr/bin/env bash
cmd=$(jq -r '.tool_input.command // empty')
case "$cmd" in
  *"rm -rf"*) echo "Blocked: rm -rf is not allowed" >&2; exit 2 ;;
esac
exit 0

It has a weakness: if jq is missing, cmd is empty and every command passes. The next script closes that gap.

How do you block risky Bash commands with a copy-paste script?

Register a Claude Code PreToolUse hook on Bash that matches tool_input.command against risky patterns and returns a JSON deny.

1. Create the folder.

mkdir -p .claude/hooks

2. Save this as .claude/hooks/block-risky-bash.sh.

#!/usr/bin/env bash
# Example block list. Edit the rules to fit your project.
set -u

input=$(cat)
cmd=$(printf '%s' "$input" | jq -r '.tool_input.command // empty') || {
  echo "Blocked: hook could not parse its input (is jq installed?)" >&2
  exit 2
}

deny() {
  printf '%s\t%s\n' "$(date -u +%FT%TZ)" "$cmd" \
    >> "${CLAUDE_PROJECT_DIR:-.}/.claude/hooks/denied.log" 2>/dev/null || true
  jq -nc --arg reason "$1" '{hookSpecificOutput: {
    hookEventName: "PreToolUse",
    permissionDecision: "deny",
    permissionDecisionReason: $reason}}'
  exit 0
}

check() {
  if printf '%s' "$cmd" | grep -Eq -- "$1"; then deny "$2"; fi
}

start='(^|[;&|(`[:space:]/])'

check "${start}"'rm[[:space:]]+([^;&|]*[[:space:]])?(-[[:alnum:]]*[rR][[:alnum:]]*|--recursive)([[:space:]]|$)' \
  "Recursive delete is blocked. Remove specific files, or skip this step and note it in your report."
check 'git[[:space:]]+([^;&|]*[[:space:]])?push([[:space:]]+[^;&|]*)?[[:space:]](--force[^[:space:]]*|-[[:alnum:]]*f[[:alnum:]]*)([[:space:]]|$)' \
  "Force pushes are blocked. Do not retry; note it in your report."
check 'git[[:space:]]+([^;&|]*[[:space:]])?reset[[:space:]]+([^;&|]*[[:space:]])?--hard' \
  "git reset --hard is blocked. Commit or stash instead."
check '(curl|wget)[[:space:]].*\|[[:space:]]*(sudo[[:space:]]+)?(ba|z|da)?sh([[:space:]]|$)' \
  "Piping a download into a shell is blocked. Download and review the file instead."
check "${start}"'sudo[[:space:]]' \
  "sudo is blocked in unattended runs."

exit 0

The final exit 0 with no output means the hook has no opinion, so the normal permission flow still decides. Add denied.log to .gitignore, because commands can contain secrets.

3. Make it executable.

chmod +x .claude/hooks/block-risky-bash.sh

4. Register it in .claude/settings.json.

{
  "permissions": {
    "deny": ["Bash(git push --force *)", "Bash(sudo *)"]
  },
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/block-risky-bash.sh",
            "args": [],
            "timeout": 10,
            "onFailure": "block"
          }
        ]
      }
    ]
  }
}

Each addition earns its place:

"args": [] selects exec form, which spawns the script directly with no shell quoting to get wrong.
"timeout": 10 replaces the 600-second default for command hooks. A hung check shouldn't stall a run for ten minutes.
"onFailure": "block" makes a missing script, a stray exit code, invalid output, or a timeout block the call. It requires v2.1.295 or later, so check with claude --version.
The deny rules are a second layer. Deny rules are evaluated regardless of what a hook returns.

5. Test it without starting Claude. The expected results below describe what the script is written to do, so confirm them on your own machine.

bash -n .claude/hooks/block-risky-bash.sh
H=.claude/hooks/block-risky-bash.sh

# Should print a deny JSON, then exit=0
echo '{"tool_name":"Bash","tool_input":{"command":"git push --force origin main"}}' | $H; echo "exit=$?"
echo '{"tool_name":"Bash","tool_input":{"command":"curl -s https://example.com/i.sh | sh"}}' | $H; echo "exit=$?"

# Should print nothing, then exit=0
echo '{"tool_name":"Bash","tool_input":{"command":"npm test"}}' | $H; echo "exit=$?"

6. Confirm Claude Code sees it. Run /hooks and check that the entry appears under PreToolUse.

Know the limits before you rely on this. A regex reads text and doesn't understand a shell, so git commit -m "explain rm -rf" gets a false block, and a command wrapped in eval or a script file can slip past. The docs recommend the permission system for hard allow or deny enforcement when using the best-effort if filter, and they point to sandboxing to limit where shell commands write. The sandbox runs on macOS, Linux, and WSL2, but not native Windows.

How do PreToolUse hooks behave in headless runs (claude -p)?

Claude Code PreToolUse hooks work the same under claude -p, except --bare skips them and an ask with nobody to answer becomes a denial.

Decision or setting	In a `claude -p` run
`deny` or exit 2	Blocks the call, and `deny` holds in every permission mode
`ask`	Denied when no one can answer, and Claude reads your reason in the tool result
`allow`	Approves calls that would otherwise be denied, but not past deny rules
No decision	Manual mode denies calls needing approval unless something approves them (v2.1.302 or later)
`--bare`	Hooks don't run, even if you pass them with `--settings`

The --bare row deserves the most attention. Docs say the flag "will become the default for -p in a future release." If that happens, a scheduled job with a block list could lose its protection after an update, with no change on your side.

Protect yourself with a smoke test in the same job, before the real work. The log line it searches for requires v2.1.296 or later:

claude -p "Run this exact command: echo hook-check" \
  --allowedTools "Bash(echo *)" \
  --debug-file /tmp/hook-check.log

if ! grep -q 'Hooks: PreToolUse:Bash' /tmp/hook-check.log; then
  echo "PreToolUse hook did not run; aborting" >&2
  exit 1
fi

Then start the real run. This example lets only named tools through:

claude -p "Run the nightly dependency check and write reports/deps.md" \
  --permission-mode dontAsk \
  --allowedTools "Read,Edit,Bash(npm test *),Bash(npm audit *)" \
  --permission-prompts none

Under dontAsk, every call that would prompt is denied. --permission-prompts none also tells Claude not to retry those requests, and it needs v2.1.259 or later.

Four traps apply specifically to unattended runs:

Using ask as your block. The result is the same denial in a run with no permission host. With --permission-prompt-tool, the run waits on that host unless you turn prompts off. Write deny so the behavior is plain. The docs I read don't spell out a hook's ask under bypassPermissions, which is one more reason to avoid it there.
Vague deny reasons. Claude reads the reason and may try a different route. Say what to do next, such as "skip this step and note it in your report."
Allow rules in an untrusted folder. In a never-trusted folder, claude -p ignores permissions.allow from project settings and prints a warning to stderr. Pass them with --allowedTools, or trust the folder.
Installed mods. With no managed settings and no Team or Enterprise /login, a mod can approve a call your hook blocked. A run using an API key or CLAUDE_CODE_OAUTH_TOKEN counts as not using that login. Review what's installed with /plugin before relying on a block.

With stream-json, permission denials appear as permission_denied messages and in the result's permission_denials. The page doesn't say whether hook denials are included, so the script keeps its own denied.log. Check it after each run, and see catching silent failures for how to alert on it.

Why does Claude Code show a PreToolUse hook error (Bash, Read, Edit)?

Claude Code shows a PreToolUse hook error when your hook fails without a deliberate block. Common causes are a wrong exit code, invalid JSON, a missing script, or a missing dependency.

The important part is that on a non-blocking error, the action goes ahead. PreToolUse:Bash hook error usually means your guard didn't run properly and the command ran anyway. The same applies to Read and Edit, which differ only in the matched tool.

Follow these steps in order:

Run the script by hand, as the troubleshooting guide suggests:

   echo '{"tool_name":"Bash","tool_input":{"command":"ls"}}' | ./my-hook.sh
   echo $?

Press Ctrl+O to open the transcript view and read the notice.
Start with claude --debug-file /tmp/claude.log, or run /debug mid-session to enable logging. Then search for your hook's Hooks: line, which reports status and duration.

What you see	Likely cause	Fix
`command not found`	Relative or unresolved script path	Use `${CLAUDE_PROJECT_DIR}` and `"args": []`
`jq: command not found`	`jq` isn't installed on this host	Install it, which is easy to forget on a new VPS
Script never runs	Not executable	`chmod +x` the script
`Failed with non-blocking status code:`	Exit code other than 0 or 2	Exit 0 with JSON, or exit 2 to block
JSON parse message	Stdout looked like JSON but wasn't valid	Build output with `jq` rather than string concatenation
JSON validation message	Valid JSON that fails the schema	Check field names and nesting
No error, but the decision is ignored	Profile `echo` before the JSON, or `permissionDecision` at the top level	Search the debug log for `Hook output does not start with {` or `Hook JSON output had unrecognized keys`
`blocking because Git Bash wasn't found`	Windows without Git Bash	Follow the linked Windows fix
Every tool blocked	Syntax error in the script	See below

That last row is the reverse failure. Bash can exit 2 on a syntax error, which Claude Code reads as "block." One developer reported being locked out of Read, Edit, and Bash for about two hours when a broken hook had an empty matcher. Run bash -n before every deploy. To recover, edit the file outside Claude or start with --settings '{"disableAllHooks": true}'.

Two public bug reports may match what you see, and neither ended with a confirmed fix:

#34859 reports the hook error label on every call despite exit 0, on v2.1.76. A bot closed it as not planned.
#3514 reports exit 2 with continue: false failing to stop a Bash call, on v1.0.51. Commenters said a permissionDecision: "deny" response did block the call.

Both reports are old compared with current docs. Test the block on your installed version rather than assuming either way.

How do you write matchers for Bash, Read, and Edit PreToolUse hooks?

A Claude Code PreToolUse matcher filters by tool name and is case-sensitive, so Bash matches and bash never fires.

Matcher	How it's evaluated	What it matches
`Bash`	Exact string	Only the Bash tool
`Read\	Edit\	Write`
`*`, empty, or omitted	Match all	Every tool, so a broken script can block everything
`Edit.*`	Unanchored regex	`Edit` and also `NotebookEdit`
`^Edit$`	Anchored regex	Only `Edit`
`mcp__memory__.*`	Regex	All tools on the `memory` MCP server
`mcp__memory`	Exact string	Nothing, because the `.*` is required

To protect files, use a separate script, since these tools carry file_path and no command. Save this example as .claude/hooks/protect-files.sh:

#!/usr/bin/env bash
set -u
path=$(jq -r '.tool_input.file_path // empty') || {
  echo "Blocked: hook could not parse its input" >&2
  exit 2
}
case "$path" in
  */../*|*.env|*.env.*|*/.git/*|*/.ssh/*|*.pem)
    jq -nc --arg p "$path" '{hookSpecificOutput: {
      hookEventName: "PreToolUse",
      permissionDecision: "deny",
      permissionDecisionReason: ("Protected path: " + $p + ". Do not retry.")}}'
    ;;
esac
exit 0

Register it as a second entry in the same PreToolUse array, with "matcher": "Read|Edit|Write". Note that *.env.* also catches .env.example, so narrow it if your agent needs that file.

This hook can't see cat .env, because that's a Bash call. Add a Bash rule for such reads, or rely on the sandbox for shell commands.

The optional if field can avoid spawning a process, for example "if": "Bash(git *)". It checks subcommands too, so npm test && git push matches. Treat it as a performance filter and keep your block logic in the script, since the guide calls the filter best-effort.

What is stop_hook_active and when do you need it?

stop_hook_active is a Claude Code Stop hook input field that is true when Claude is already continuing because of a stop hook.

PreToolUse doesn't use it. You need it only when a Stop hook returns decision: "block" to keep Claude working, for example until tests pass. Check it first and let Claude stop if it's set, using the pattern from the guide:

#!/bin/bash
INPUT=$(cat)
if [ "$(echo "$INPUT" | jq -r '.stop_hook_active')" = "true" ]; then
  exit 0  # Allow Claude to stop
fi
# ... rest of your hook logic

Without that check, the loop isn't literally endless. Claude Code overrides the block after eight consecutive continuations, and the count resets each time Claude calls a tool. You can change the cap with CLAUDE_CODE_STOP_HOOK_BLOCK_CAP. In a paid run, those eight rounds still consume tokens. The Stop hook guide covers the design in detail.

How do you check Claude Code usage when agents run unattended?

Claude Code usage appears through /usage in a session and through total_cost_usd in claude -p --output-format json output for scripted runs.

Where you run	What to use	What it tells you	Limit
Interactive session	`/usage` (`/cost` and `/stats` are aliases)	Session cost, plan limits, activity stats	Plan breakdown covers this machine only
`claude -p` script	`--output-format json`	`total_cost_usd` and per-model cost	A client-side estimate that can differ from your bill
Claude Console (API)	Cost page and workspace spend limits	Spend reporting and a cap	API organizations only
Your own stack	OpenTelemetry export	Per-user token and cost metrics in near real time	You have to run the observability stack

To record each run's estimate:

claude -p "Run the nightly dependency check" --output-format json \
  | jq -r '[now, .total_cost_usd] | @tsv' >> usage.tsv

On a subscription, ignore the dollar figure in the Session block, because it isn't relevant for billing. Read the plan usage bars instead, and note that the /usage breakdown reports the share of recent usage from loops, skills, and subagents. Hard caps are also available: --max-budget-usd stops a run's remaining background work when it reaches the cap.

For scale, Anthropic reports an average around $13 per developer per active day across enterprise deployments (as of November 2026; check the official page). That's an enterprise figure, so a small cron job may look very different. Measure your own runs.

Next steps

Claude Code PreToolUse protection can be running today with three small actions.

Do it yourself:

Save block-risky-bash.sh, add the settings entry with onFailure: "block", and run the bash -n and piped tests.
Add the smoke test to the top of your scheduled job, so a run aborts if the hook stops loading.
Check denied.log after your next few runs, then adjust false blocks and add any missing patterns.

If you'd rather not maintain scripts: try auto mode, where a classifier reviews most actions, using --permission-mode auto --permission-prompts none. It suits people who want fewer prompts without keeping a block list. It doesn't suit anyone who needs a fixed, auditable rule that gives the same answer every time, and you'll need to confirm it's available for your account. Claude Code comes with Pro at $17/month billed annually or $20 monthly, and Max from $100/month (as of November 2026; check the official page). You can also keep your hooks alongside it, since a hook's deny holds in every permission mode.

Related guides on this site:

FAQ

Exit 1 doesn't block a Claude Code hook, updatedInput can rewrite a tool call, and deny survives bypass mode.

Does exit code 1 block a tool call in a Claude Code hook?

No. Exit 1 without valid JSON is a non-blocking error, so the tool call goes ahead and you see a hook error notice. The exception is a hook with onFailure: "block", where any exit code other than 0 or 2 counts as a failure and blocks. Use exit 2 or a JSON deny for an intentional block.

Where is the Claude Code hooks documentation?

Anthropic keeps it in two places. The Hooks reference holds event schemas and JSON formats, and the Hooks guide holds setup steps and troubleshooting. Inside a session, /hooks shows what's configured.

Can a PreToolUse hook change the tool input before it runs?

Yes, through updatedInput in hookSpecificOutput. It replaces the entire input object, so include the fields you aren't changing. Permission rules are evaluated against the input your hook returns. Avoid having two hooks rewrite the same tool's input, because Claude Code doesn't merge them or guarantee which one is used.

Do PreToolUse hooks still run with --dangerously-skip-permissions?

Yes. A hook returning deny "blocks the tool even in bypassPermissions mode or with --dangerously-skip-permissions." That doesn't make bypass mode safe, since docs advise using it only in isolated environments such as containers or VMs. A block list only stops the patterns you thought to write.

Sources

These official docs and public reports were read on 2026-11-06 and support the claims above.

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code GitHub Actions Permissions: Least-Privilege YAML

Aoba Works — Wed, 04 Nov 2026 16:15:45 +0000

Originally published at Unattended.

Claude Code GitHub Actions permissions come in two layers. The first is the Claude GitHub App, which needs Contents, Issues and Pull requests set to read and write. The second is the permissions: block in your workflow, which scopes the GITHUB_TOKEN and must include id-token: write for the action's default authentication. On top of those, claude_args: --allowedTools decides which tools Claude can actually run, and your Anthropic credential sits in a GitHub secret. Below you'll find the exact YAML for review, scheduled and code-writing jobs, plus a least-privilege table built from the official Claude Code GitHub Actions docs.

Which option is right for you?

Pick the job you actually want automated and start from the matching official example. Don't start from a broad template and trim it down.

Your situation	What to pick first	Limitation / not for	First page to open
Beginner, one repo, already use Claude Code locally	Run `/install-github-app` (quick setup)	Installs the official app with its full permission set, including Actions, Workflows and Repository hooks write, and you can't accept a subset. Not for you if you want only the three scopes the action uses (build a custom app instead)	Claude Code GitHub Actions: Quick setup
You want automatic PR review and nothing else	Code-review skill workflow with read-only token scopes	Comments only and never pushes fixes. On public repos it doesn't run on fork PRs because secrets are withheld. Not for you if you want Claude to fix what it finds	Run a skill example
Daily work: `@claude` fixes issues and pushes commits	Interactive mode workflow (`examples/claude.yml`)	Needs write scopes on the workflow token, only users with write access can trigger it, and it can't edit `.github/workflows/`. Not for repos where untrusted users comment	examples/claude.yml
Recurring report or triage on a cron	Automation mode with a `prompt` and named MCP tools	Runs only from the default branch. In public repos it is disabled after 60 days of inactivity. Output stays in the run log unless you grant a posting tool. Not for one-off requests	Run on a schedule
Tight budget, already pay for Pro/Max	`CLAUDE_CODE_OAUTH_TOKEN` from `claude setup-token`	Tied to the subscription of whoever generated it and subject to plan usage limits. Not for org-wide shared secrets	Authentication docs
Company with an AWS/GCP/Azure account and a no-static-keys rule	Bedrock, Vertex or Foundry via OIDC	Needs a cloud role plus a custom GitHub App. Not for quick experiments or anyone without cloud admin access	Cloud providers page

What permissions does Claude Code GitHub Actions need?

It needs two things: the Claude GitHub App and a workflow permissions: block that includes id-token: write. The app needs Contents, Issues and Pull requests set to read and write. The app and the workflow token are two separate systems, and most permission confusion comes from mixing them up.

The first layer is the Claude GitHub App. The docs state the action "relies on three of the app's permissions": Contents, Issues and Pull requests, all read and write (Claude Code docs). When you install the app you actually grant a bigger set, because the app is shared with Code Review and web auto-fix. It includes Actions (read and write), Checks (read and write), Workflows (read and write), Repository hooks (read and write) and several read-only scopes. You can't trim that set: "GitHub doesn't let you accept a subset" (Claude Code docs). If that's a problem for you, the docs point to building a custom GitHub App with only Contents, Issues and Pull requests. That app works for the action only, though. Code Review and web auto-fix still need the official app.

The second layer is the workflow's GITHUB_TOKEN scopes. These come from the permissions: key at workflow or job level. GitHub's rule is strict: "If you specify the access for any of these permissions, all of those that are not specified are set to none" (GitHub workflow syntax). So once you write a block, list every scope you need.

Layer	Where you set it	What it controls	Minimum for this action
Claude GitHub App	github.com/apps/claude install screen	What the app token can do in the repo	Contents, Issues, Pull requests: read and write
Workflow `permissions:`	`.github/workflows/*.yml`	`GITHUB_TOKEN` scopes and the OIDC token	`id-token: write` plus the read scopes your job needs
`claude_args --allowedTools`	`with:` input of the action step	Which tools Claude may call	Only the tools the task uses

The id-token: write scope is easy to miss. The official example labels it "required for the Claude Code GitHub Action's default GitHub App authentication" (Claude Code docs). Without it you get OIDC errors, as the action FAQ explains.

Two limits apply whatever you grant. Per the FAQ, the app has no workflow write access through the action, so Claude won't edit .github/workflows/. It also won't rebase, merge or force-push, even if you allow Bash(git rebase:*), because that rule lives in its system prompt (action FAQ).

How do you set a least-privilege permissions block for Claude Code?

Start with read-only scopes plus id-token: write. Add contents: write, pull-requests: write or issues: write only for jobs that change code or post with the GITHUB_TOKEN. The Claude Code docs and the security doc give 4 example blocks, and they show how far apart the needs are.

Job type	`permissions:` block (from official examples)	Why
PR review via code-review skill	`contents: read`, `pull-requests: read`, `issues: read`, `id-token: write`	The app token posts the inline comments, so the workflow token only reads
Scheduled report	`contents: read`, `issues: read`, `id-token: write`	Reads commits and issues through GitHub MCP tools; output goes to the run log
`@claude` interactive (writes code)	`contents: write`, `pull-requests: write`, `issues: write`, `id-token: write`, `actions: read`	Pushes commits and updates comments; `actions: read` lets Claude see CI results
Issue labeling for non-write users	`issues: write` only, with `github_token` passed	The security doc's example of an "extremely limited" workflow

Sources: Claude Code GitHub Actions docs for the first three rows and action security doc for the last.

Here's the interactive block as the docs publish it, set at job level:

jobs:
  claude:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
      issues: write
      id-token: write
      actions: read

A few things trip people up:

Job-level vs workflow-level. GitHub accepts permissions "for an entire workflow or for individual jobs" (GitHub Docs). Put it on the Claude job so that other jobs in the same file don't inherit write access.
Missing actions: read. If you want Claude to read CI failures, you need the actions: read scope and the action input additional_permissions: "actions: read", which is "for viewing workflow results" (usage.md).
Using write-all for convenience. That gives the token every scope. The docs' own advice is to "grant the GITHUB_TOKEN the least required access" (GitHub Docs).

If you also run Claude headless on your own machine, the same mindset applies to local settings. See our headless permissions template.

How do you use allowed_tools and claude_args in Claude Code GitHub Actions?

In v1 of Claude Code GitHub Actions, tool limits and every other CLI flag go through claude_args (usage.md). The old allowed_tools input is deprecated. Pass --allowedTools with the specific tools and Bash patterns the job needs.

The old input is marked "DEPRECATED: Use claude_args with --allowedTools instead" (usage.md). The claude_args input "accepts any Claude Code CLI argument" (Claude Code docs), including --max-turns, --model, --mcp-config, --allowedTools (alias --allowed-tools) and --debug.

Defaults matter here. "The Bash tool is disabled by default for security" (action FAQ). For a plain-text automation prompt, the docs say Claude "has no shell or GitHub API access until you grant the tools the prompt needs" (Claude Code docs). So you're starting from a closed position and opening what each job needs.

Task	Example `--allowedTools` value	Notes
Post inline review comments	`mcp__github_inline_comment__create_inline_comment`	Required even if the skill already lists it; the MCP server only starts when this flag names it
Read commits and issues for a report	`mcp__github__list_commits,mcp__github__list_issues`	From the official daily-report example
Run tests and lint	`Bash(npm test:),Bash(npm run lint:)`	Example; narrower than the FAQ's `Bash(npm:*)`
Read an issue for triage	`Bash(gh issue view:*)`	The security doc's recommended scope for non-write users
Edit files	`Edit,Read,Write`	From the migration guide example in usage.md

Here is a multi-line claude_args example. Replace <model-id> with a model ID your account can use. This article doesn't recommend a specific model ID.

claude_args: |
  --model <model-id>
  --max-turns 8
  --allowedTools "Read,Edit,Bash(npm test:*),Bash(npm run lint:*)"

Common mistakes and fixes:

Bash(npm:*) is wider than it looks. It covers npm install and npm publish as well as npm test. Name subcommands when you can. Our allowed-tools syntax guide covers how the pattern matching works.
Review posts nothing. Either the --comment argument is missing from the review prompt, or claude_args doesn't name the inline-comment MCP tool. Without --comment, "Claude posts nothing" and findings stay in the run log (Claude Code docs).
Still using @beta keys. direct_prompt, mode, max_turns and model are beta-era inputs. Move them to prompt and claude_args. The custom_instructions input becomes --append-system-prompt (upgrade section).

You can also pass tool rules as JSON through the settings input with a permissions.allow rule, if you'd rather keep them in a settings file (Claude Code docs).

How do you store the Anthropic API key safely as a GitHub secret?

Save the credential as a GitHub Actions secret and reference it only as ${{ secrets.NAME }}. Name it ANTHROPIC_API_KEY (or CLAUDE_CODE_OAUTH_TOKEN for a subscription token). The action's security doc puts it bluntly: "Never hardcode your Anthropic API key or OAuth token in workflow files!" (security.md).

These are the steps for a repository secret, using GitHub's UI labels (GitHub Docs):

Open the repo and click Settings (or the More dropdown if Settings isn't shown).
In the sidebar's Security section, click Secrets and variables, then Actions.
On the Secrets tab, click New repository secret.
Enter ANTHROPIC_API_KEY as Name, paste the key from the Claude Console as Secret, and click Add secret.
In the workflow, add anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} under with:.

For a subscription token, run claude setup-token locally, save the output as CLAUDE_CODE_OAUTH_TOKEN, and use the input claude_code_oauth_token instead. The docs list it for Pro, Max, Ultra, Team and Enterprise plans (Claude Code docs).

If you want a human gate on the riskier workflows, use an environment secret. Go to Settings → Environments, pick the environment, and under Environment secrets click Add secret (GitHub Docs). Then add environment: <name> to the job and set protection rules for that environment, described in GitHub's "Managing environments for deployment" page.

Pitfalls:

Sharing an OAuth token across an org. The docs recommend a Console API key for shared secrets, since an OAuth token is tied to the subscription of whoever ran claude setup-token.
Thinking deleted means revoked. Deleting the GitHub secret leaves the credential valid. To retire it, also delete the key in the Console (Claude Code docs).
Leaking via logs. show_full_output is off by default, and the security doc warns that logs "are publicly visible in GitHub Actions for public repositories!" (security.md). Leave it off unless you're debugging a private repo.

How do you set up an unattended PR review workflow with Claude Code?

Run the official code-review plugin on a pull_request trigger with read-only scopes. Add id-token: write to the workflow scopes and a claude_args line that allows only the inline-comment tool. This is the official example, with two optional additions marked in comments:

name: Code Review
on:
  pull_request:
    types: [opened, synchronize, ready_for_review, reopened]
    paths:            # example addition: only review source changes
      - "src/**"
jobs:
  review:
    runs-on: ubuntu-latest
    timeout-minutes: 15   # example addition: cap runaway runs
    permissions:
      contents: read
      pull-requests: read
      issues: read
      id-token: write
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 1
      - uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          plugin_marketplaces: "https://github.com/anthropics/claude-plugins-official.git"
          plugins: "code-review@claude-plugins-official"
          prompt: "/code-review:code-review --comment ${{ github.repository }}/pull/${{ github.event.pull_request.number }}"
          claude_args: '--allowedTools "mcp__github_inline_comment__create_inline_comment"'

The base file, including actions/checkout@v6, fetch-depth: 1 and claude-code-action@v1, comes from the Claude Code docs, "Run a skill".

Steps:

Install the Claude GitHub App on the repo.
Add the ANTHROPIC_API_KEY secret (previous section).
Save the file as .github/workflows/claude-code-review.yml on a branch and open a PR. That PR should trigger the first review.
Check the run under the Actions tab, then look for inline comments on the PR.

Some behavior is built in. Claude skips draft and closed PRs, PRs it judges trivial or automated, and PRs that already have a Claude comment. On public repos, the docs note GitHub withholds secrets from fork PRs, "so the review runs only on pull requests from branches in the same repository" (Claude Code docs).

To limit by branch, add branches: [main] under pull_request. That matches the PR's base branch. Put your review criteria in CLAUDE.md and keep the file short, because Claude reads it on every run. Our CLAUDE.md template for unattended agents has a structure you can copy.

If you'd rather not maintain a workflow at all, Anthropic's separate Code Review product runs reviews without a workflow file.

How do you run Claude Code on a schedule or on issues without babysitting?

Give Claude Code a prompt (automation mode) on schedule or issues triggers. Pair it with narrow scopes, --max-turns and timeout-minutes so each run is bounded. Use @claude mentions (interactive mode) only when a human starts each request.

	Interactive mode	Automation mode
How it starts	No `prompt` input; waits for `@claude`	`prompt` input set; runs on the event
Where results go	Comment on the issue or PR	Workflow run log, unless the prompt and a tool let it post
Good for	Ad-hoc fixes you request	Cron reports, triage, recurring checks
Who can trigger	Users with write access	Same check on issue/PR events; `schedule` skips it

Source: Claude Code docs, "Interactive and automation modes".

Here's a bounded nightly example, adapted from the official daily report. The concurrency, timeout and turn cap are example additions:

name: Daily Report
on:
  schedule:
    - cron: "0 9 * * *"
concurrency:
  group: claude-daily-report
  cancel-in-progress: true
jobs:
  report:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
      issues: read
      id-token: write
    steps:
      - uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
          prompt: "Generate a summary of yesterday's commits and open issues"
          claude_args: |
            --max-turns 6
            --allowedTools "mcp__github__list_commits,mcp__github__list_issues"

Scheduling has some catches. GitHub runs scheduled workflows "only from the default branch," and in public repos it disables the schedule after 60 days without repository activity (Claude Code docs). Scheduled runs are attributed to whoever last changed the cron line. If that user is a bot, the human-actor check rejects the run unless the bot is in allowed_bots.

Loops are the other trap. If your workflow posts a comment containing @claude using GITHUB_TOKEN, nothing triggers, because "The github-actions user cannot trigger subsequent GitHub Actions workflows" (action FAQ). That's GitHub preventing infinite loops on purpose, so keep it. If you need chaining, use your own app token rather than widening allowed_bots to '*'. For machine-side scheduling outside GitHub, see our cron job setup guide.

Is it safe to run Claude Code on pull requests from forks or untrusted users?

Only with tight limits, because PR bodies, issues and comments can carry prompt injection. The defaults help: fork runs get no secrets, and non-write users are blocked.

GitHub's rule: "With the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository" (GitHub Docs). That's why a plain pull_request review simply doesn't run on forks. Switching to pull_request_target to get fork coverage changes the risk. The security doc warns that pull_request_target and workflow_run "execute with the base repository's secrets" (security.md).

Setting	Risk	Safer choice
`pull_request` (default)	Low; no secrets on forks	Keep for review
`pull_request_target` + checkout PR head into root	High; untrusted code runs with your secrets	Check out the head into a subfolder (`path: pr-head`) and pass `--add-dir pr-head`
`allowed_non_write_users: "*"`	High; bypasses the main access check	Avoid, or pair with `issues: write` only and `Bash(gh issue view:*)`
`allowed_bots: "*"` on a public repo	External apps may trigger runs	List bot names explicitly
Personal access token as `github_token`	Static token can leak via injection	Use `${{ secrets.GITHUB_TOKEN }}`, which rotates per run

Source: security.md.

The core rule is short: "Do not check out an untrusted ref into the workspace root before this action" (security.md). On PRs, the action restores Claude config paths like .claude/, .mcp.json and CLAUDE.md from the base branch. The security doc notes that files such as package.json, lockfiles and Makefile stay at the PR head. Our inference from that is that an allowed command like npm test, which runs scripts defined in package.json, would execute whatever the PR author wrote there.

The action strips HTML comments, invisible characters and hidden attributes before Claude sees them, though the doc admits "new bypass techniques may emerge." On public repos, include_comments_by_actor lets you allowlist whose comments reach Claude. For more rule ideas, see our guardrails examples.

Should you use an Anthropic API key or OIDC (Bedrock/Vertex) for Claude Code GitHub Actions?

An Anthropic API key secret is the simplest choice for solo builders. OIDC through Bedrock, Vertex, Foundry or Anthropic workload identity federation avoids long-lived keys but needs cloud-side setup.

Method	Secret stored	Extra setup	Suits	Doesn't suit
`ANTHROPIC_API_KEY`	Long-lived key	None beyond the secret	Solo builders, small teams	Orgs that ban static keys
`CLAUDE_CODE_OAUTH_TOKEN`	Subscription token	`claude setup-token`	Pro/Max users who want plan billing	Org-wide shared secrets
Anthropic workload identity federation	None (IDs only)	Console federation rule and service account; `id-token: write`	Teams on the Claude API wanting no keys	Quick experiments
Bedrock / Agent Platform / Foundry	None static; cloud OIDC role	Cloud role, custom GitHub App with `APP_ID`, `APP_PRIVATE_KEY`	Companies billing through a cloud account	Anyone without cloud admin access

Sources: Claude Code docs and cloud providers page.

For federation, set anthropic_federation_rule_id (fdrl_...) and anthropic_organization_id. You can optionally add anthropic_service_account_id (svac_...) and anthropic_workspace_id. The docs say the action needs id-token: write for that exchange "even when you pass your own github_token" (Claude Code docs). For cloud providers, set use_bedrock: "true", use_vertex: "true" or use_foundry: "true" and follow the provider page. Bedrock setups typically store an AWS_ROLE_TO_ASSUME secret, which is a role ARN, not a key.

For a wider comparison of auth and billing when Claude runs headless, see our headless mode guide.

How do you cap cost and runaway runs in Claude Code GitHub Actions?

Cap Claude Code GitHub Actions cost with --max-turns, job timeout-minutes, concurrency groups and narrow triggers, plus a cheaper model for routine work. Each run uses both GitHub Actions minutes and Claude tokens (Claude Code docs).

The default ceiling is generous. GitHub's job timeout is "The maximum number of minutes to let a job run before GitHub automatically cancels it. Default: 360" (GitHub workflow syntax). That's six hours of runner time if a run wanders. Setting timeout-minutes: 10 to 20 on review and report jobs is a reasonable starting point (example values; tune to your repo).

Control	Where	What it stops
`--max-turns N`	`claude_args`	Long agent loops within one run
`timeout-minutes`	Job	Runner minutes burning after a hang
`concurrency` + `cancel-in-progress: true`	Workflow or job	Stacked runs on rapid pushes
`paths:` / `types:` filters	`on:`	Runs on docs-only or irrelevant changes
`if: contains(github.event.comment.body, '@claude')`	Job	Runner start on every comment
`--model`	`claude_args`	Paying top-tier rates for simple tasks

The concurrency option means "cancel any currently running job or workflow in the same concurrency group" (GitHub workflow syntax). For PR review, group by PR number: group: review-${{ github.event.pull_request.number }}.

On pricing, API runs bill per token at the rates on Claude's pricing page. With an OAuth token, runs draw on your subscription instead, and current plan prices are listed on the same pricing page. Subscription use is still subject to plan usage limits, so a busy repo may hit them. To see where tokens go, read our token usage guide.

Why is Claude Code failing with permission errors in GitHub Actions?

Most Claude Code GitHub Actions permission errors come from one of five causes. Either a scope is missing from permissions:, the Claude GitHub App isn't installed, the trigger user lacks write access, a tool isn't in --allowedTools, or a github_token was passed by mistake.

Symptom	Likely cause	Fix
OIDC authentication error	No `id-token: write`	Add it to the job's `permissions:` (FAQ)
"You don't have permission" style rejection	Commenter lacks write access	Grant write, or use `allowed_non_write_users` with care
Nothing happens on `@claude`	App not installed, workflows disabled, or `/claude` instead of `@claude`	Check each in the docs' troubleshooting list
`403 Resource not accessible by integration`	App token can't call the `/user` endpoint	See the FAQ entry; often caused by custom steps calling `/user`
Claude says it can't run a command	Bash not allowed	Add `Bash(cmd:*)` to `--allowedTools`
Review finishes but no comments	Missing `--comment` or inline-comment tool	Add both, per the review example
CI doesn't run on Claude's commits	`github_token: ${{ secrets.GITHUB_TOKEN }}` passed	Remove it so the action uses the app, or use a custom app token
Comments not from `claude[bot]`	`github_token` provided	Remove it unless using a custom app
Scheduled run rejected	Last cron editor is a bot	Add that bot to `allowed_bots`
Claude won't edit workflow files	App lacks workflow write via the action	Edit `.github/workflows/` yourself

Sources: Claude Code troubleshooting and action FAQ.

A quick diagnostic order:

Open the failed run under Actions and find the first red step.
If the error mentions OIDC or token exchange, check permissions: first.
If Claude ran but did nothing useful, rerun with --debug in claude_args (private repos only, given public logs).
Test the API key locally with claude before debugging YAML further, as the docs suggest.

Next steps

If maintaining workflow files isn't the route you want, there are two alternatives.

Anthropic's Code Review runs automatic PR reviews without YAML. It suits teams that only want review. It doesn't suit you if you need cron jobs, issue-to-PR automation or custom tool lists. It also requires the official Claude GitHub App with its full permission set (Claude Code docs). If cost predictability matters more than per-token billing, a Claude Pro or Max subscription with CLAUDE_CODE_OAUTH_TOKEN is another route. Plans are listed on Claude's pricing page. It's a poor fit for org-wide shared secrets.

Related guides on this site:

FAQ

Does Claude Code GitHub Actions need id-token: write?

Yes, with the default Claude GitHub App authentication. The action uses the workflow's OIDC token to get its app token, and the action FAQ says you "must add the id-token: write permission." Anthropic workload identity federation needs it too, even if you pass your own github_token (Claude Code docs).

Can Claude Code push commits or open PRs from a workflow?

It can push commits to a branch (prefix claude/ by default) when the job has write scopes. But "Claude doesn't create PRs by default." It posts a link to a pre-filled PR page so a human opens the PR and branch protection still applies (action FAQ).

Should I pin the claude-code-action to a commit SHA?

The official examples use the moving @v1 tag, which picks up fixes automatically. You can see it in examples/claude.yml and the workflows in the Claude Code docs. GitHub's security hardening guide says pinning an action to a full-length commit SHA is the only way to use it as an immutable release. That makes a SHA the safer choice for workflows that hold write scopes or secrets. The trade-off is that you have to bump it yourself. By the same guidance, a branch ref like @main changes with every commit, so avoid it in production workflows.

Can I use Claude Code in GitHub Actions without an API key?

Yes. You can use a CLAUDE_CODE_OAUTH_TOKEN from claude setup-token on a Claude subscription. You can also use Anthropic workload identity federation, or OIDC with Amazon Bedrock, Google Cloud's Agent Platform or Microsoft Foundry (Claude Code docs). Each option still needs the Claude GitHub App or a custom app for GitHub access.

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code Stuck on Compacting Conversation: Cron Watchdog

Aoba Works — Tue, 03 Nov 2026 16:14:29 +0000

Originally published at Unattended.

If Claude Code is stuck on "Compacting conversation," stop the process. Then start a fresh session with a short written handoff instead of waiting or resuming. In an unattended claude -p run, wrap the command in a wall-clock timeout and use the PreCompact/PostCompact hooks to timestamp each compaction. If one runs too long, kill the run and retry clean. The rest of this guide covers why compaction stalls, what the real thresholds are, and how to make a cron job recover without you.

The official docs warn that /compact "reads the conversation it summarizes, so compacting a large context is itself a large request" (Claude Code docs: Manage costs). A large request can fail, get retried or run slowly, and in a headless run nobody sees the spinner.

Which option is right for you?

Pick based on how you run Claude Code today. Each row gives the first page to open.

Your situation	What to pick first	First page to open
Beginner, interactive terminal, spinner frozen right now	Press Esc, then `/compact` with a focus, or `/clear`	Errors: Prompt is too long
Daily work: cron or CI job that sometimes never finishes	Wrapper script with a hard timeout and a compaction watchdog (below)	Run Claude Code programmatically
Long multi-hour agent tasks	Compact earlier with `CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=50`, and split work into sessions	Set the auto-compact window
Budget-sensitive (API key billing)	Cap each run with `--max-budget-usd` and `--max-turns`, and turn auto-compact off for short jobs	CLI reference
Not sure if it's compaction at all	Check the stream for `api_retry` events before blaming compaction	/why-claude-code-stops-mid-automation-and-how-to-fix-it/

Here's why each one fits. Interactive users can see and answer the error. Cron users can't, so they need a script that decides for them. Long tasks gain the most from compacting at a smaller, cheaper context. Budget users mainly need a ceiling so a stuck loop can't run up a bill.

Why does Claude Code get stuck on "Compacting conversation"?

Claude Code gets stuck when the summarization request fails, retries or loops. Most often the context is too big to summarize, or the API is slow or throttled.

The documented and reported causes fall into five groups:

The summary doesn't fit. The errors page lists Prompt is too long · automatic compaction failed: Compaction would make the conversation longer. That message means the summary would be bigger than the original, and the fix is /clear (Errors reference). The same page says a single-exchange conversation "cannot be compacted" at all.
Retries that look like a hang. Claude Code "retries transient failures up to 10 times with exponential backoff before showing you an error." Each attempt gets up to API_TIMEOUT_MS (600000 ms, 10 minutes by default) to receive response headers (Errors: Automatic retries; Environment variables). A big compaction request that hits an overloaded API can therefore sit for a long time before it fails.
The retry watchdog you turned on. Setting CLAUDE_CODE_RETRY_WATCHDOG=1 makes Claude Code retry 429 and 529 errors indefinitely. On v2.1.199 and later it also raises the retry count for other transient errors to 300, "roughly three hours of backoff" (Errors: Tune retry behavior). That's useful for cron. It also means a compaction can wait hours unless you set CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS.
Thrashing loops. GitHub issue #6004 describes an endless cycle: read files, compact, lose the state, then read the same files again. It was reported on v1.0.83, still seen on v2.0.6x, and closed as "not planned" in February 2026. The usual trigger is one large file or tool output that fills the context again right after each summary.
Summarizer overflow (older versions). Issue #23751 reported Conversation too long failures at about 48% context usage on v2.1.32–v2.1.34, with image-heavy sessions hit hardest. It is marked fixed as of v2.1.85. If your cron box runs an old pinned version, update it first.

Here's why headless makes this worse. An interactive user sees the error and presses Esc. A claude -p process has no human, so a failed compaction ends in one of two ways. It either exits non-zero, which is easy to handle, or it keeps retrying or looping, which looks like a frozen job.

Example: a nightly job that reads a 10,000-line log file can fill most of the context in one tool call. Compaction summarizes it, the agent reads the log again, and the cycle repeats until the job hits a limit or you notice in the morning.

What triggers auto-compact, and can you change the threshold?

By default, Claude Code auto-compacts when the conversation reaches the model's context limit. You can lower that point with a setting, a CLI flag or an environment variable.

The "92–95%" figure you see quoted online doesn't match current docs. According to Default auto-compact thresholds (as of October 2026):

Most sessions compact when the conversation reaches the model's context limit.
Models with a native 1M window (Fable models, Sonnet 5 and later, Haiku 5.5, Opus 4.7 and later) compact "at about 967K tokens by default."
Sonnet 4.6 and Opus 4.6 without extended context compact at the 200K boundary.
With CLAUDE_CODE_DISABLE_1M_CONTEXT=1, native-1M models compact at 200K.
Unrecognized model IDs, such as an LLM gateway alias, compact at whatever window Claude Code assumes for that ID.

The four ways to change it

Where	How	Scope
In a session	`/autocompact 500k` (or `/autocompact auto` to reset)	Saved for the current model
Settings file	`"autoCompactWindow": 200000` in `~/.claude/settings.json`	Every model
One launch	`claude --autocompact 500k` (v2.1.221+)	That run only
Scripts/CI	`CLAUDE_CODE_AUTO_COMPACT_WINDOW=200000`	Wins over the others

Accepted values run from 100K to 1M tokens, and Claude Code caps the window at the model's context size (model config; settings reference).

Use CLAUDE_AUTOCOMPACT_PCT_OVERRIDE to compact earlier as a percentage. The docs say "the variable can't raise the threshold, so values above the default percentage are ignored" (Environment variables). It only applies in sessions that compact before the model's context limit. So pair it with CLAUDE_CODE_AUTO_COMPACT_WINDOW if you want it to take effect.

Turning it off

Set "autoCompactEnabled": false in a settings file (settings reference). With auto-compact off, a full context produces a Context limit reached error instead of a compaction (Errors). For a short cron job that's often what you want: a fast, loud failure your script can catch. For long jobs, it means the run just stops at the limit.

There's also a hook-level off switch. A PreCompact hook that exits with code 2 blocks compaction (Hooks reference). See /claude-code-hooks-lifecycle-every-event-explained/ for where it fires in the lifecycle.

How do you detect a stuck compaction in a headless or cron run?

To spot a stuck compaction in headless Claude Code, mark when each compaction starts and ends. If a start has no matching end within your limit, the run is stuck.

Exit codes alone aren't enough. The docs say Claude Code "exits with code 0 on success and a non-zero code when the run fails" (Headless docs). A hung process never exits, though, so you also need signals that work while it's still running. There are three:

Compaction markers from hooks. PreCompact fires "before context compaction begins" and PostCompact fires "after context compaction completes successfully." Both accept an auto or manual matcher (Hooks reference). Have the first write a timestamp file and the second delete it. A timestamp file older than your limit means compaction is stuck.
Stream inactivity. Run with --output-format stream-json --verbose and write stdout to a file. If the file hasn't changed in N minutes, something is wrong. Note that PreCompact and PostCompact "never produce a hook_started event" in the stream (CLI reference), so the hook marker files are the more reliable signal.
Retry events. Before each retry, the stream includes a system/api_retry event with attempt, max_retries, retry_delay_ms and an error category such as rate_limit or overloaded (Headless docs). Several of these right after the marker file appears means compaction is fighting the API rather than looping.

Steps to add compaction markers

Create a job folder, for example mkdir -p ~/jobs/nightly/state.
Save this as ~/jobs/nightly/compact-watch.json. It's an example; change the paths to match yours:

{
  "hooks": {
    "PreCompact": [
      { "matcher": "auto",
        "hooks": [{ "type": "command",
          "command": "date +%s > \"$HOME/jobs/nightly/state/compact.started\"" }] }
    ],
    "PostCompact": [
      { "matcher": "auto",
        "hooks": [{ "type": "command",
          "command": "rm -f \"$HOME/jobs/nightly/state/compact.started\"; date >> \"$HOME/jobs/nightly/state/compact.log\"" }] }
    ]
  }
}

Pass it with --settings ./compact-watch.json. In --bare mode Claude Code skips hooks it would normally discover on disk. Settings you pass with --settings are still loaded (Headless: bare mode).
Watch the state/ folder from your wrapper script (next section).

A common mistake is putting the hooks only in .claude/settings.json and then running with --bare. The hooks silently never fire. The fix is to pass them through --settings.

How do you set a timeout and kill a hung Claude Code process?

Claude Code has no built-in wall-clock limit for claude -p. Wrap it in timeout or a watchdog loop, send SIGTERM first, then SIGKILL after a grace period.

What the docs give you for runaway work: --max-turns "exits with an error when the limit is reached" and has no limit by default. --max-budget-usd stops the run once estimated spend reaches the cap (CLI reference). Neither one measures time. A single compaction that keeps retrying uses no extra turns, so you need a clock as well.

Option A: GNU timeout (Linux)

timeout -k 30s 90m claude -p "$(cat task.md)" --output-format json --max-turns 40
echo "exit=$?"

GNU timeout sends TERM when the limit is reached and KILL 30 seconds later because of -k. It exits with status 124 when the command timed out (GNU coreutils: timeout invocation).

Here's what happens inside Claude Code on SIGTERM: "Claude Code exits with code 143," leaves the current turn unfinished, kills any running Bash process tree and runs SessionEnd hooks (Headless: Stop a run with SIGTERM). If you'd rather end the turn cleanly, the docs say to send SIGINT first.

Option B: watchdog loop (works anywhere, catches compaction specifically)

This example script combines a total time limit, a compaction limit and a stream-inactivity check. The limits are example values; tune them to your job.

#!/usr/bin/env bash
# run-agent.sh — example watchdog for one unattended Claude Code run
set -u
JOB="$HOME/jobs/nightly"; STATE="$JOB/state"
COMPACT_LIMIT=900     # example: 15 min max for one compaction
HARD_LIMIT=5400       # example: 90 min for the whole run
IDLE_MIN=20           # example: no stream output for 20 min = stalled
mkdir -p "$STATE"; rm -f "$STATE/compact.started"

cd "$JOB/repo" || exit 1
claude -p "$(cat "$JOB/task.md")" \
  --settings "$JOB/compact-watch.json" \
  --output-format stream-json --verbose \
  --max-turns 40 --permission-prompts none \
  > "$STATE/stream.jsonl" 2> "$STATE/stderr.log" &
PID=$!; START=$(date +%s)

stop() { echo "$(date) $1" >> "$STATE/watchdog.log"
         kill -INT "$PID" 2>/dev/null; sleep 10
         kill -TERM "$PID" 2>/dev/null; sleep 20
         kill -KILL "$PID" 2>/dev/null; exit "$2"; }

while kill -0 "$PID" 2>/dev/null; do
  sleep 30; NOW=$(date +%s)
  if [ -f "$STATE/compact.started" ] && \
     [ $((NOW - $(cat "$STATE/compact.started"))) -gt "$COMPACT_LIMIT" ]; then
    stop "compaction stalled" 75
  fi
  [ $((NOW - START)) -gt "$HARD_LIMIT" ] && stop "hard limit" 124
  [ -n "$(find "$STATE/stream.jsonl" -mmin +"$IDLE_MIN")" ] && stop "stream idle" 76
done
wait "$PID"; exit $?

The exit codes 75 and 76 are this script's own convention, so your retry layer can tell a stalled compaction apart from a normal failure. --permission-prompts none (v2.1.259+) stops the run from waiting on a permission prompt nobody will answer (Headless docs). That's a separate kind of hang that looks the same from outside.

Pitfalls

Killing too early. A normal compaction of a large context takes real time, and each API attempt can wait up to 10 minutes for headers by default. A 2-minute limit will kill healthy runs. Start around 15 minutes and look at compact.log to see how long yours actually take.
Mistaking background wait for a hang. After the final turn, claude -p can wait up to 10 minutes for background subagents or commands (CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS, default 600000) (Environment variables). If stderr shows Waiting for background work to finish, it isn't compaction.
Killing only the parent. If you launch through bash -c or npx, signal the process group. Otherwise a child claude can keep running.

How do you avoid hitting compaction at all?

Keep each Claude Code session well below the auto-compact window. Filter big tool outputs, send verbose work to subagents, and give each task its own session.

These are the levers that matter most for unattended jobs, roughly in order of payoff:

One job, one session. Start every cron run fresh (no --continue). The docs recommend /clear "when switching to unrelated work," and a new -p process does the same thing (Context window docs).
Filter output before Claude sees it. The docs' own example is a PreToolUse hook that rewrites test commands to keep only failures. They point out that grepping a 10,000-line log for ERROR can cut context "from tens of thousands of tokens to hundreds" (Manage costs). Copy-paste versions are in /claude-code-hooks-examples-7-copy-paste-scripts/.
Delegate large reads to subagents. Their file contents stay in their own context window, and only a summary comes back (Context window docs).
Use --bare for scripts. The docs say "--bare is the recommended mode for scripted and SDK calls" (Headless docs). It skips CLAUDE.md, MCP servers, skills and auto memory, so the run starts with less context. Bare mode needs ANTHROPIC_API_KEY, because it doesn't use your subscription login.
Trim always-loaded files. Keep CLAUDE.md under 200 lines and move workflow-specific instructions into skills, which load only when invoked (Manage costs).
Compact earlier if you must compact. A summary of a 150K context is a smaller request than a summary of 950K. On a native-1M model, example settings like CLAUDE_CODE_AUTO_COMPACT_WINDOW=300000 keep each compaction request moderate.

To measure what's filling the context, run /context in an interactive session on the same repo. It "shows your conversation size in tokens" broken down by category (Errors). /claude-code-how-to-see-token-usage-and-reduce-it/ walks through reading that output.

A common mistake is asking an agent to "read all files in src/ and summarize". That's the quickest way to fill the window. Name the files instead, or have a subagent do the reading.

How do you resume a session safely after a compaction failure?

After a failed compaction, don't resume the stuck Claude Code session. Start a new session seeded with a short handoff file, and keep the old transcript for debugging.

Here's why. --resume and --continue reload the stored conversation. That's the same oversized history that just failed to compact, so the next turn is likely to trigger the same compaction. Two details make this worse after a kill:

After SIGTERM, "Claude Code leaves the interrupted turn as it is, and your next prompt drives the conversation." Setting CLAUDE_CODE_RESUME_INTERRUPTED_TURN=1 makes it continue the interrupted turn instead (Headless docs). That's the opposite of what you want here.
If you resume with different system-prompt flags, they only take effect "once the conversation is compacted or when you start a new conversation" (CLI reference).

Safer recovery steps

Record the failed session ID. With --output-format stream-json, the system/init event and the final result carry session_id. Save it to state/last_session from your wrapper.
Keep a progress file the agent writes. Add this line to task.md (example wording): "After each completed step, append one line to PROGRESS.md: step, files changed, what's next."
Start fresh with a handoff. Example: claude -p "Continue the task in task.md. Completed steps are in PROGRESS.md; verify the last one, then continue." --settings compact-watch.json
Only resume if you need the history. Fork it so the original stays intact: claude --resume "$ID" --fork-session (CLI reference). Then run /compact with a narrow focus, for example /compact keep only the list of changed files and the failing test name.
Check the code state. A killed run may have left half-written files. git status and git stash handle most cases. /claude-code-checkpoints-how-rollback-works-in-automations/ explains what Claude Code's own checkpoints do and don't cover.

In an interactive terminal, the errors page and community reports give a shorter path. Press Esc twice to step back a few messages, then run /compact again. If that still has no room, use /clear; the old conversation stays reachable through /resume (Errors).

How do you build auto-recovery for unattended runs (cron, CI)?

A watchdog kills the stalled Claude Code run, and a wrapper retries it once in a fresh session. The wrapper also sorts exit codes and logs anything it can't fix.

Building on the watchdog script above, a minimal recovery layer looks like this (example):

#!/usr/bin/env bash
# nightly.sh — called from cron
JOB="$HOME/jobs/nightly"
for attempt in 1 2; do
  "$JOB/run-agent.sh"; code=$?
  case $code in
    0)  echo "$(date) ok attempt=$attempt" >> "$JOB/state/history.log"; exit 0 ;;
    75|76|124|143)  # stalled compaction, idle stream, hard limit, SIGTERM
        echo "$(date) stalled code=$code attempt=$attempt" >> "$JOB/state/history.log"
        mv "$JOB/state/stream.jsonl" "$JOB/state/stream.$(date +%s).jsonl" ;;
    *)  echo "$(date) failed code=$code" >> "$JOB/state/history.log"; break ;;
  esac
done
exit 1

And the crontab line, for example: 15 2 * * * /home/you/jobs/nightly/nightly.sh >> /home/you/jobs/nightly/state/cron.log 2>&1. Environment details, such as PATH and auth tokens under cron, are covered in /claude-code-cron-job-setup-run-automations-overnight/.

Design decisions that matter

Retry once, not forever. If the second fresh session also stalls at compaction, the task itself is too big for one session. Retrying again just burns tokens. Issue #6004 includes one user's report of $800 in API charges from a compaction loop (GitHub #6004).
Bound the API retry wait. If you use CLAUDE_CODE_RETRY_WATCHDOG=1 to ride out rate limits, also set CLAUDE_CODE_RETRY_WATCHDOG_MAX_WAIT_MS (v2.1.295+). Otherwise a single compaction request can wait out 429/529 errors without limit (Errors). Rate-limit handling for cron is covered in /claude-code-rate-limit-reached-retry-setup-for-cron-jobs/.
Cap money as well as time. Add --max-budget-usd (print mode only). The docs warn spend "can pass the cap, so leave headroom" (CLI reference).
Alerting is up to you. Claude Code doesn't page anyone. Have the wrapper's failure branch call whatever you already use: a Slack incoming webhook via curl, mail, or your CI's failure notification.
Lower retries if you want fast failure. CLAUDE_CODE_MAX_RETRIES defaults to 10. The docs suggest you "lower it to surface failures faster in scripts" (Errors).

A common mistake is letting the retry layer use --continue. The retry then inherits the same bloated context and stalls the same way. Always retry with a new session plus the progress file.

Claude Code auto-compact vs manual context management: which is more reliable for long runs?

For unattended long runs, manual context management in Claude Code is more predictable. Auto-compact is convenient, but it adds a failure point your script can't control.

Approach	How you set it	Strength	Limitation	Best for
Default auto-compact	Nothing	Zero setup, the run continues past the limit	Compaction happens at the largest, most expensive context, so it's more exposed to slow or failed requests	Interactive work
Earlier auto-compact	`CLAUDE_CODE_AUTO_COMPACT_WINDOW`, `CLAUDE_AUTOCOMPACT_PCT_OVERRIDE`	Smaller summary requests	Compacts more often; each summary can drop detail	Multi-hour single tasks
Auto-compact off + chunked sessions	`"autoCompactEnabled": false`, one task per `-p` run, PROGRESS.md handoff	Deterministic: a full context becomes a clean error	You design the task split up front; a run can stop at the limit	Cron jobs with clear steps
Subagent isolation	Delegate reads/tests to subagents	Main context stays small	Subagents have their own windows and can compact too (sub-agents docs)	Research-heavy jobs
Block via hook	`PreCompact` exits 2	Fine-grained control, e.g. block only during a critical step	Blocking leaves you at the limit with no relief	Rare, advanced cases

My pick for the reader of this guide: for recurring jobs, use chunked sessions with auto-compact either off or set to a lower window, plus the watchdog. Here's why. A job that fails loudly at step 3 of 5 is easy to resume from PROGRESS.md. A job that freezes in a summary at 2 a.m. costs you a night.

One pitfall with turning compaction off: if a single step needs more context than the window holds, it will always fail. Split that step further, or move its large reads into a subagent. /ai-agent-vs-automation-a-5-question-test-for-claude-code/ helps decide whether a step needs an agent at all.

Next steps

Do today (DIY):

Run claude --version on the machine that runs your cron. If it's older than v2.1.85, update with claude update. The summarizer overflow bug in issue #23751 was fixed in that version.
Add compact-watch.json and run-agent.sh from this guide to one job, with generous limits (15 min per compaction, 90 min total). Run it for a week and read compact.log to see real compaction durations.
Add a "write one line to PROGRESS.md per step" instruction to your task prompt, so any kill can be retried in a fresh session.

If you'd rather not do it yourself: Anthropic's official Claude Code GitHub Actions integration runs Claude Code inside a CI job. The CI runner gives you job timeouts and failure notifications, so you don't maintain a cron box. It suits people already on GitHub. It doesn't suit jobs that need your local machine or private network, and you still pay for Actions minutes plus Claude usage.

If your real problem is hitting usage limits rather than hangs, Claude subscriptions include Claude Code. Pro is "$20 if billed monthly" and Max starts "from $100 per month" (as of October 2026; check the official pricing page). A bigger plan doesn't fix a stuck compaction, though. It only gives you more headroom before rate limits slow the retries.

Related guides on this site:

FAQ

Why is Claude Code frozen on compacting conversation?

Usually the summary request is very large and is either retrying against a slow or throttled API, or caught in a re-read-and-compact loop. Each attempt can wait up to 10 minutes for response headers by default, and Claude Code retries up to 10 times (Errors). In a terminal, press Esc and run /compact with a focus, or /clear.

How do I force Claude Code to stop compacting?

Interactively, press Esc to interrupt, then /clear for a fresh context. In scripts, send SIGINT or SIGTERM to the process; SIGTERM exits with code 143 (Headless docs). To prevent it, a PreCompact hook that exits with code 2 blocks compaction.

Does Claude Code auto-compact affect cron jobs?

Yes. claude -p runs use the same auto-compact thresholds as interactive sessions, but nobody sees the spinner. A stalled compaction holds the job until something kills it. Add a wall-clock timeout and hook-based compaction markers so the job fails and retries on its own.

Can I disable auto-compact in Claude Code?

Yes. Set "autoCompactEnabled": false in a settings file (settings reference). After that, a full context ends with a Context limit reached error instead of compacting. That suits short, chunked cron jobs better than long single tasks.

What exit code does Claude Code return on timeout?

Claude Code has no built-in time limit for -p, so the code depends on what stopped it. GNU timeout returns 124, SIGTERM gives 143, and --max-turns exits with an error status when the turn limit is hit (GNU timeout; Headless docs; CLI reference).

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code Stop Hook: Block Early Exit Without Loops

Aoba Works — Mon, 02 Nov 2026 16:07:45 +0000

Originally published at Unattended.

A Claude Code Stop hook is a command that Claude Code runs when the main agent finishes responding. If the command prints {"decision": "block", "reason": "..."}, Claude keeps working and treats the reason as its next instruction. To make an unattended run continue until a real check passes, have the hook run your tests or look for the output file, block with a specific reason while the check fails, and stop blocking after a fixed number of retries so the run can't loop forever.

Which option is right for you?

The best choice is the lightest Claude Code tool that can verify your "done" condition. Start from the row that matches your situation:

Your situation	What to pick first	First page to open	Why
Beginner, interactive session, one task	`/goal` with a condition such as "npm test exits 0"	Keep Claude working toward a goal	Needs no config file. It lasts for the current session only
Daily unattended runs (cron, CI, VPS) with a test command	A command Stop hook plus your own retry counter	Hooks reference: Stop	The check is deterministic and the same in every session
Tight budget	Command hook, a retry cap of 2–3, and `--max-budget-usd` on the run	Manage costs effectively	A shell check uses no model tokens. Only the retries do
The "done" condition needs judgment (prose quality, completeness)	A prompt-based Stop hook (`"type": "prompt"`)	Prompt-based hooks	A model judges it, so it costs a model call each time it runs
Subagents should check their own work	A SubagentStop hook	Hooks reference: SubagentStop	It fires when a subagent finishes. Stop doesn't fire then

If you're unsure, pick the command hook. You can test it from a terminal, and it gives the same answer every time.

What is a Claude Code Stop hook and when does it run?

A Claude Code Stop hook is a handler registered under the Stop event. Claude Code runs it every time the main agent ends a turn. The hooks reference says Stop "Runs when the main Claude Code agent has finished responding". It doesn't run when you interrupt, and API errors fire a separate StopFailure event instead (hooks reference).

This matters for unattended work because the model decides when it is finished, and it can be wrong. A Stop hook lets a script that you control decide whether the work is actually done.

The hook receives JSON on stdin. These are the fields that matter for a "keep working" hook (hooks reference):

Field	What it tells you	Use it for
`session_id`	A unique ID for the session	Naming a retry-counter file
`stop_hook_active`	`true` when Claude is already continuing because a Stop hook blocked	Spotting a repeat pass
`last_assistant_message`	The text of Claude's final reply	Logging and notifications. The docs recommend it over reading `transcript_path`
`background_tasks`, `session_crons`	Work still in flight or scheduled	Telling "done" apart from "waiting on a background job"
`cwd`, `transcript_path`	Working directory and transcript file	Finding project files

Some points to know before you write one:

Stop has no matcher support. It fires on every occurrence (hooks guide). In an interactive session, a hook that runs tests also runs after a one-line answer to a quick question. For example, you could have your cron wrapper export a variable such as UNATTENDED=1 and make the script exit 0 when that variable isn't set.
Stop hooks fire "whenever Claude finishes responding, not only at task completion" (hooks guide). Your check has to work at any point in the task.
A command hook has a 600-second default timeout (hooks reference). A slow test suite needs a larger timeout value.

For where Stop sits among the other events, see Claude Code Hooks Lifecycle: Every Event, Explained.

Claude Code Stop hook example: how do you keep working until tests pass?

Run your real check and block with a specific reason until it passes. When it passes, exit 0 without a decision. The decision fields for Stop are decision ("block" prevents stopping; leave it out to allow the stop) and reason, which the docs call "Required when decision is "block"".

Step 1: Register the hook

Add this to .claude/settings.json in your project. Use .claude/settings.local.json if you don't want to commit it.

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/verify-before-stop.sh",
            "timeout": 900
          }
        ]
      }
    ]
  }
}

Notes:

There is no "matcher", because Stop doesn't support one.
$CLAUDE_PROJECT_DIR avoids "command not found" errors when the working directory changes (hooks guide).
timeout is in seconds. 900 is an example value for a slow suite.

Step 2: Write the script

Save the following as .claude/hooks/verify-before-stop.sh and run chmod +x .claude/hooks/verify-before-stop.sh. In this example, "done" means npm test exits 0 and output/report.md exists and isn't empty. Swap in your own checks.

#!/bin/bash
# Example Stop hook: block until tests pass and the report exists, max N retries.
INPUT=$(cat)
SESSION_ID=$(printf '%s' "$INPUT" | jq -r '.session_id')
ACTIVE=$(printf '%s' "$INPUT" | jq -r '.stop_hook_active')
MAX_RETRIES="${STOP_HOOK_MAX_RETRIES:-3}"

STATE_DIR="${CLAUDE_PROJECT_DIR:-.}/.claude/state"
mkdir -p "$STATE_DIR"
COUNT_FILE="$STATE_DIR/stop-$SESSION_ID.count"
LOG="$STATE_DIR/stop-hook.log"

# A fresh stop (not a continuation) starts a new retry budget.
[ "$ACTIVE" != "true" ] && echo 0 > "$COUNT_FILE"
COUNT=$(cat "$COUNT_FILE" 2>/dev/null || echo 0)

# The real check. Its output goes to a file, never to stdout.
if npm test --silent >"$STATE_DIR/last-test.txt" 2>&1 && [ -s output/report.md ]; then
  echo "$(date -u +%FT%TZ) $SESSION_ID PASS retries=$COUNT" >>"$LOG"
  rm -f "$COUNT_FILE"
  exit 0                      # no decision = Claude may stop
fi

# Retry budget spent: allow the stop, record the failure, warn the user.
if [ "$COUNT" -ge "$MAX_RETRIES" ]; then
  echo "$(date -u +%FT%TZ) $SESSION_ID GAVE_UP retries=$COUNT" >>"$LOG"
  rm -f "$COUNT_FILE"
  jq -n --arg n "$COUNT" \
    '{systemMessage: ("Stop hook gave up after " + $n + " retries; checks still failing.")}'
  exit 0
fi

# Block and say exactly what is still broken.
echo $((COUNT + 1)) > "$COUNT_FILE"
FAILS=$(tail -n 30 "$STATE_DIR/last-test.txt")
jq -n --arg fails "$FAILS" '{
  decision: "block",
  reason: ("Not done. npm test must exit 0 and output/report.md must exist and be non-empty. Last test output:\n" + $fails)
}'
exit 0

systemMessage is a general JSON output field that shows a message to the user. The guide lists it as something a successful hook's JSON can surface (hooks guide).

Step 3: Test it before you run it unattended

Pipe sample input into the script yourself:

echo '{"session_id":"test1","stop_hook_active":false}' | .claude/hooks/verify-before-stop.sh
echo $?

You should see one JSON object with "decision": "block" while the tests fail, nothing once they pass, and an exit code of 0 in both cases.

The exit-code-2 alternative

There's a shorter version. Write the reason to stderr and exit 2:

echo "npm test is failing; fix it before finishing" >&2
exit 2

For Stop, exit 2 "Prevents Claude from stopping, continues the conversation", and stderr reaches Claude the same way reason does. The docs say to use one approach per hook (hooks guide). The JSON form is the better fit for unattended runs, because the same script can also return systemMessage when it gives up.

How do you use stop_hook_active so a Stop hook never loops forever?

In Claude Code, stop_hook_active is true when Claude is already continuing because a Stop hook blocked. Read it to tell a first stop from a repeat, and pair it with a counter to set a real cap.

The guide's minimal pattern exits early when the flag is true (hooks guide):

if [ "$(echo "$INPUT" | jq -r '.stop_hook_active')" = "true" ]; then
  exit 0  # Allow Claude to stop
fi

That pattern prevents loops, but it allows exactly one retry. The first stop blocks. Every later stop in that continuation arrives with stop_hook_active: true, so the hook lets it through. If tests are still red after one round of fixes, the run ends red. The counter in the script above allows up to MAX_RETRIES blocks per task turn instead.

Pitfalls:

Don't key the counter on the project directory only. Parallel sessions would use up each other's retries. Keying it by session_id keeps two cron jobs in the same repo from sharing a budget.
Put state under the project and outside git. Add .claude/state/ to .gitignore, or Claude may commit your counter files.
Make "give up" visible. Exiting 0 silently on the last retry looks the same as success in a cron log. Write a marker line your wrapper can grep for. AI Agent Monitoring in Claude Code: Catch Silent Failures shows how to alert on it.
Choose a small cap. Each retry is a full extra turn with your whole context. A cap of 2–3 (an example value, not an official recommendation) is enough for tests to go green after a real fix. If the suite still fails after that, the cause usually needs a human.

What is CLAUDE_CODE_STOP_HOOK_BLOCK_CAP and how do you raise the limit?

CLAUDE_CODE_STOP_HOOK_BLOCK_CAP is the Claude Code environment variable that raises the built-in limit on consecutive Stop hook continuations. The default is 8. The hooks reference says that after stop hooks have continued the turn eight times in a row, "Claude Code overrides the next block and ends the turn" (hooks reference, as of November 2026).

When the cap is reached, the guide describes the symptom: Claude "ends the turn with a warning that the Stop hook blocked too many consecutive times" (hooks guide). That is where the "set claude_code_stop_hook_block_cap to raise this limit" text comes from.

One detail matters here: "The count of consecutive continuations resets each time Claude calls a tool." The built-in cap catches a hook that blocks while Claude only replies with text. It does not catch the common unattended case, where Claude edits files and reruns tests on each pass. Each tool call resets the count, so going by the docs' wording, a test-gated hook can keep blocking well past 8. That's why your own counter is the cap that protects your token budget.

How to raise it (only if you need to)

First confirm the hook isn't blocking on a condition that can never pass, such as a missing file path or a test that needs network access.
Set the variable for the run. In a shell or cron wrapper:

   CLAUDE_CODE_STOP_HOOK_BLOCK_CAP=12 claude -p "..."

Or in a settings file, using the env key, which Claude Code reads "no matter how claude was launched" (environment variables):

   { "env": { "CLAUDE_CODE_STOP_HOOK_BLOCK_CAP": "12" } }

Keep your script's MAX_RETRIES at or below what you're willing to pay for.

The guide's condition for raising it is when your hook "legitimately needs more than eight iterations to converge". Keep the built-in cap in place on unattended runs.

The additionalContext output, which sends non-error feedback, goes through "the same loop protections" as decision: "block" (hooks reference). Switching to it doesn't get around the cap.

How do you fix a Claude Code Stop hook "JSON validation failed" error?

In Claude Code, this error means the hook's stdout parsed as JSON but didn't match the schema for the event. The guide explains that a JSON validation message means stdout "parsed as JSON but failed schema validation", and a parse message means it "looked like a JSON object but wasn't valid JSON". Both can happen "even on exit 0" (hooks guide).

Common causes in Stop hooks:

What your hook printed	Problem	Fix
`{"decision": "allow"}` or `"approve"`	Stop only documents `"block"`	Leave `decision` out, or print nothing, to allow the stop
`{"decision": "block"}` with no `reason`	`reason` is required with block	Add a specific `reason` string
`{"hookSpecificOutput": {"permissionDecision": "deny"}}`	That shape belongs to PreToolUse	Use the top-level `decision`/`reason`
`{"additionalContext": "..."}` at the top level	It belongs inside `hookSpecificOutput` with `"hookEventName": "Stop"`	Nest it. Fields at the top level are silently ignored (hooks guide)
`{"reason": "Line 1\nfailed "quote""}` built with echo	Unescaped quotes or newlines cause a parse error	Build it with `jq -n --arg`. The guide recommends a JSON encoder "instead of string concatenation" (hooks guide)

A related failure produces no error at all: the decision is simply ignored. The rule is "Print nothing but the JSON object to stdout." Output that doesn't start with { is read as plain text, and the usual cause is an echo in ~/.zshrc or ~/.bashrc. The guide's fix is to wrap profile echoes in if [[ $- == *i* ]]; then ... fi so they run only in interactive shells (hooks guide).

Steps to fix:

Start Claude Code with claude --debug-file /tmp/claude.log, then trigger a stop.
In the log, search for Hook output does not start with { (extra output) or Hook JSON output had unrecognized keys (fields at the wrong level). Both strings come from the hooks guide.
Send all debug prints to stderr (>&2) or a log file, and redirect test output to a file as the example script does.
Run the script by hand and pipe its output through jq .. If jq can't parse it, Claude Code can't either.

Claude Code Stop hook not working: what should you check?

When a Claude Code Stop hook doesn't fire or doesn't block, the usual causes are registration, the path, permissions, or stdout. Work through this table from top to bottom:

Check	How	Typical fix
1. Is it loaded?	Type `/hooks`. It opens "a read-only browser for your configured hooks"	Fix the JSON (no trailing commas or comments) and the file location
2. Correct key?	It must be under `"Stop"` (case-sensitive), inside a `"hooks"` array	Copy the registration block above
3. Executable?	`ls -l .claude/hooks/`	`chmod +x` the script
4. Right path?	Look for "command not found" in the transcript	Use `"$CLAUDE_PROJECT_DIR"/...` or an absolute path
5. Clean stdout?	Pipe sample JSON in by hand	Move logs to stderr and silence profile echoes
6. Dependencies?	Look for `jq: command not found`	Install jq or rewrite the script in Python
7. Managed policy?	`/hooks` shows `Only hooks from managed settings run here`	Your organization set `allowManagedHooksOnly`. Ask an admin (hooks guide)

Two Stop-specific traps:

Exit 1 doesn't block. Exit codes other than 0 and 2 without valid JSON are treated as non-blocking errors on most events (hooks guide), so exit 1 on failing tests lets Claude stop.
A broken exit-2 hook can trap Claude. On Stop, exit 2 "sends Claude back to keep working, and Claude can't repair a hook that won't run" (hooks reference). The reference treats one case as non-blocking: an exit 2 with empty stdout and a missing-file message such as No such file or directory on stderr. Other crashes still block. This is another reason to prefer the JSON form, where a crashed script doesn't print "decision": "block".

To see what the hook did, press Ctrl+O to open the transcript view. A block shows the reason, and a non-blocking error shows a <hook name> hook error notice (hooks guide). If Claude Code stops mid-run for reasons unrelated to hooks, see Why Claude Code Stops Mid-Automation (and How to Fix It).

Stop vs SubagentStop: which one covers subagents?

In Claude Code, Stop covers the main agent and SubagentStop covers subagents. The hooks guide lists SubagentStop as firing "When a subagent finishes" (hooks guide). A Stop hook alone won't check subagent output.

	Stop	SubagentStop
Fires when	The main agent finishes a turn	A subagent finishes
Matcher	None, always fires	Agent type, e.g. `Explore` or a custom agent name
Extra input	`stop_hook_active`, `last_assistant_message`, `background_tasks`, `session_crons`	`stop_hook_active`, `agent_id`, `agent_type`, `agent_transcript_path`, `last_assistant_message`
Block effect	Claude keeps working	"keeps the subagent running and delivers the reason to it as its next instruction" (hooks reference)

SubagentStop uses the same decision format as Stop. The counter script works with one change: key the counter on agent_id instead of session_id, because one session can run many subagents.

{
  "hooks": {
    "SubagentStop": [
      {
        "matcher": "test-writer",
        "hooks": [
          { "type": "command", "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/verify-subagent.sh" }
        ]
      }
    ]
  }
}

Pitfall: internal agents, such as prompt suggestions and /btw side questions, also trigger SubagentStop, sometimes with an empty agent_type (hooks reference). Always set a matcher with your agent's name. To choose the agent's tools, see Claude Code Subagent allowed-tools: Syntax, Examples, Fixes.

How do you disable a Claude Code hook quickly?

To turn off all Claude Code hooks without deleting them, set "disableAllHooks": true in a settings file. To turn off a single hook, remove its entry from the file that defines it.

Find which file defines it. /hooks lists configured hooks by event. Hooks can live in ~/.claude/settings.json (all projects), .claude/settings.json (project, committed), .claude/settings.local.json (project, not shared), managed policy, plugins, skills, or subagent frontmatter (hooks guide).
Turn off one hook. Delete its object from the "Stop" array. JSON settings don't allow comments, so you can't comment it out.
Turn everything off. Add "disableAllHooks": true. Precedence rules apply, so "a project's settings file can override yours", and managed hooks keep running unless the setting is also set in managed settings (hooks guide). This also turns off /goal, which runs on the hooks system (goal docs).
Stop a run that's already looping. Interrupt the session. Stop hooks don't run when you interrupt (hooks reference). Fix the script, then restart.

Settings edits are normally picked up while Claude Code is running. If a change doesn't appear after a few seconds, restart the session (hooks guide).

How do you watch token usage from Stop hook retries?

Use Claude Code's /usage command in a session, cap spend on headless runs, and count blocks in your hook's log. Every block starts another turn, and that turn re-sends your whole context.

Inside a session, run /usage. Its Session block shows total cost and usage by model. The docs call the dollar figure "an estimate" and point API users to the Console Usage page for real billing (costs). On Pro, Max, Ultra, Team, and Enterprise plans, the same screen shows plan usage bars and a usage breakdown.
For headless runs, add --max-budget-usd as a hard ceiling. The costs page says session cost counts toward it (costs; flags in the CLI reference).
From your hook log, count retries per run:

   grep -c GAVE_UP .claude/state/stop-hook.log
   grep PASS .claude/state/stop-hook.log | tail -n 20

If many runs pass only after 2–3 retries, your prompt or CLAUDE.md probably isn't telling Claude to run the tests before it finishes.

For a sense of scale, Anthropic reports an enterprise average of "around $13 per developer per active day" (costs, as of November 2026). A hook that loops all night can push a single run well past that.

To get alerted, have the hook notify you when it gives up. For example, on macOS add osascript -e 'display notification "Stop hook gave up" with title "Claude Code"' to the give-up branch, or use notify-send on a Linux desktop. On a headless server, append to a file that your cron wrapper checks. The Notification event also exists for cases where Claude Code is waiting on you, such as permission_prompt or idle_prompt (hooks guide). More on cutting usage: Claude Code: How to See Token Usage (and Reduce It).

Next steps

Things you can do today:

Copy the registration block and script above. Change the check to your real "done" condition, set STOP_HOOK_MAX_RETRIES=3, and pipe sample JSON into it until the output is a single clean JSON object.
Add .claude/state/ to .gitignore, and add a grep GAVE_UP line to the end of your cron wrapper so a failed run doesn't look like a success.
Add --max-budget-usd to your claude -p command as a second limit on spend.

If you'd rather not write a script: /goal is built in and costs nothing extra to set up. Its evaluator runs on the small fast model, and the docs say evaluation tokens are "typically negligible compared to main-turn spend" (goal docs). It suits interactive sessions and one-off claude -p "/goal ..." runs. It is a poor fit if you need a deterministic check, because the evaluator "doesn't run commands or read files independently" and only judges what Claude has shown in the conversation. It also lasts for one session only.

For running Claude Code itself, Anthropic's subscription plans (Pro, Max, Ultra, Team, Enterprise) are listed at claude.com/pricing (as of November 2026; check the official page). Pay-per-token API billing through the Claude Console lets you set workspace spend limits (costs). Neither one replaces a retry cap in your hook.

Related guides on this site:

FAQ

Where can I find Claude Code Stop hook examples on GitHub?

Anthropic's own repository has a complete hook reference implementation, the bash command validator example. It's a PreToolUse hook, but its input parsing and exit-code handling carry over to Stop. Community discussions also exist, such as this thread in ianymu/claude-verify-before-stop, but they aren't official sources. Check any community example for a retry cap and for clean stdout before you rely on it.

Why does Claude Code show "running stop hook" and how long should it take?

That status means your Stop hook command is running. It lasts as long as your check takes, such as the full test suite. Command hooks time out after 600 seconds by default (hooks reference). If it seems stuck, run the script by hand to time it, and raise timeout or run a faster test subset.

Can a Stop hook send me a notification when Claude finishes or gives up?

Yes. A Stop hook can run any command, so add a desktop notification or a webhook call to the branch that allows the stop, and use last_assistant_message for the summary text. The docs recommend that field over reading the transcript at Stop time (hooks reference). On servers, a log line that your cron wrapper checks is more reliable than a desktop notification.

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code Subagent allowed-tools: Syntax, Examples, Fixes

Aoba Works — Sun, 01 Nov 2026 16:15:10 +0000

Originally published at Unattended.

Claude Code subagents don't have an allowed-tools field. The key you want in a subagent's .md frontmatter is tools. It's an allowlist, written as a comma-separated string like tools: Read, Grep, Glob or as a YAML list. disallowedTools is the matching denylist. allowed-tools belongs to Skills, and it does something else: it pre-approves tools for one turn and restricts nothing. If you leave tools out, the subagent "Inherits every tool available to subagents if omitted."

Which option is right for you?

The best place to start depends on whether you want a safe default fast, a dependable daily setup, or the lowest token cost.

Your situation	What to pick first	First page to open
Beginner, first custom subagent	`tools: Read, Grep, Glob` (read-only). Nothing can break	Create custom subagents
Daily work, subagent edits code and runs tests	`tools: Read, Edit, Bash, Grep, Glob` plus a `permissions.deny` rule for risky Bash commands	Configure permissions
Subagent needs one MCP server (GitHub, Slack)	`tools: Read, mcp__github__*`, or scope the server with `mcpServers`	Sub-agents: MCP section
You want everything except writes	`disallowedTools: Write, Edit` and keep inheriting	Sub-agents: restrict tools
Budget: keep context and tokens small	Inline MCP servers in the subagent so the parent never loads their tool descriptions	Sub-agents: scope MCP servers
Unattended or cron runs	Explicit `tools` list plus a `PreToolUse` hook as a second check	Hooks reference

The reason for each row: an explicit read-only list can't fail in a dangerous way. A Bash deny rule blocks particular commands without taking Bash away. A server wildcard keeps the list short. A denylist is the least work when you only fear one or two tools. Inline MCP keeps tool definitions out of the main context. Hooks still run when a frontmatter list turns out wrong.

What is the allowed-tools field in Claude Code subagents?

In Claude Code subagents, the "allowed-tools" job is done by the tools frontmatter key, an allowlist that limits which tools that subagent can call. Many people search for allowed-tools because they've seen it in Skills or the Agent SDK, so the three names are worth separating before you write any YAML.

Where you write it	Key name	What it does
Subagent file (`.claude/agents/*.md`)	`tools`	Allowlist: the subagent can use only these tools
Subagent file	`disallowedTools`	Denylist: removes tools from the inherited or listed pool
Skill file (`SKILL.md`)	`allowed-tools`	Pre-approves listed tools for the turn that invokes the skill. Other tools stay callable
Agent SDK options	`allowedTools` / `allowed_tools`	Auto-approves tools before your permission callback runs

The Skills documentation says the Skills allowed-tools grant clears when you send your next message, and that it does not restrict which tools are available. The Agent SDK permissions page treats allowedTools as approval too. Put allowed-tools: in a subagent file and you get neither behavior: it isn't a supported subagent field, so the subagent inherits everything.

Here is a minimal, valid subagent. Save it as .claude/agents/code-reviewer.md (example):

---
name: code-reviewer
description: Reviews diffs for bugs and style issues. Use after code changes.
tools: Read, Grep, Glob
model: sonnet
---
You are a code reviewer. Read the changed files, report problems with file and line, and never modify files.

Some official limits are worth knowing (as of October 2026, sub-agents docs). name can be up to 256 characters and can't contain :. Claude Code warns at startup when the combined descriptions of your custom subagents go over 15,000 tokens. By default, 20 subagents running at once is the ceiling, and the next spawn fails with Concurrent subagent limit reached.

How do you write the allowed-tools syntax (wildcards, lists, MCP tool names)?

Claude Code's subagent tools field takes exact tool names, either as a comma-separated string or a YAML list. The documented wildcards are for MCP servers, such as mcp__github__*.

Steps to add or change a tool list

Open (or create) the file: .claude/agents/<name>.md for one project, or ~/.claude/agents/<name>.md for all your projects. A subagent with the same name in the project folder wins over the one in your home folder. The priority table puts managed settings first, then the --agents CLI flag, then project, then user, then plugins.
Add a tools: line between the --- markers. Spell tool names with their exact capitalization (Read, not read).
Save the file, then start a new session or ask Claude to use the subagent by name.
On current versions, typing /agents only reminds you to ask Claude or edit .claude/agents/ and ~/.claude/agents/ directly. The interactive wizard with Running and Library tabs exists only on v2.1.197 and earlier (sub-agents docs).

Syntax patterns from the official docs

String form:

tools: Read, Grep, Glob, Bash

YAML list form (same meaning):

tools:
  - Read
  - Grep
  - Glob
  - Bash

Every tool from one MCP server, plus Read:

tools: Read, mcp__github__*

Everything inherited except one MCP server:

disallowedTools: mcp__github

No MCP tools at all, built-ins kept:

disallowedTools: mcp__*

Those three MCP patterns come straight from the sub-agents table. One MCP tool is named mcp__<server>__<tool>, the same format hook matchers use, for example mcp__memory__create_entities.

What the docs don't promise

Command-level patterns in tools. Skills accept Bash(git add *). For subagents, the docs only describe what a specifier does in disallowedTools: "An entry with a specifier, such as Bash(git push *), still removes the whole tool." To keep Bash and block only git push, add Bash(git push *) to permissions.deny in settings. That rule covers the main conversation and subagents.
Agent(worker, researcher). This allowlist of spawnable subagent types works only for an agent running as the main thread through claude --agent. Inside an ordinary subagent definition, the docs say the type list in parentheses is ignored.
Task vs Agent. In v2.1.63 the subagent-spawning tool was renamed from Task to Agent, and older Task(...) references still work as aliases (GitHub issue #29677). Hook scripts that check tool_name == "Task" stopped matching, so use Agent in anything new.

Defining the list from the CLI

For one-off or scripted runs, the --agents flag takes JSON, with tools as an array (sub-agents docs):

claude --agents '{
  "code-reviewer": {
    "description": "Expert code reviewer. Use proactively after code changes.",
    "prompt": "You are a senior code reviewer.",
    "tools": ["Read", "Grep", "Glob", "Bash"],
    "model": "sonnet"
  }
}'

In non-interactive mode you can pass a file path instead, such as claude -p --agents ./agents.json "Review my changes". Interactive sessions refuse a file path (sub-agents docs).

Inherit vs restrict: should a subagent inherit all tools or list them explicitly?

For a Claude Code subagent that runs without you watching, list its tools explicitly. Inheriting is fine for interactive helpers you supervise. If you leave tools out, the subagent picks up every built-in and MCP tool the main conversation has, after a couple of filters. Listing tools gives you predictable behavior, and in exchange you have to update the list when the job changes.

	Omit `tools` (inherit)	`tools` allowlist	`disallowedTools` denylist
New MCP servers you add later	Picked up automatically	Not available until you add them	Picked up automatically
Risk of surprise side effects	Highest	Lowest	Medium
Maintenance	None	Update when the job changes	Update when a new risky tool appears
Good for	Ad-hoc helpers in an interactive session	Cron jobs, reviewers, anything unattended	"Everything but writes" style agents
Failure mode	Does too much	Missing tool, or launch refused if nothing resolves	Forgetting a new dangerous tool

When both fields are set: "If both are set, disallowedTools is applied first, then tools is resolved against the remaining pool." A tool that appears in both lists ends up removed.

Two inheritance details tend to catch people:

Permission mode can be overridden. When the main conversation runs in bypassPermissions, acceptEdits, or auto mode, the subagent runs in that same mode and the docs say your permissionMode setting is ignored. The tool list is your real limit in those modes, so a tight list matters more than the permission mode you picked. To see how modes and rules combine for headless runs, read /claude-code-permissions-best-practices-headless-template/.
Built-in agents already restrict. The built-in Explore and Plan agents use "read-only tools; Write and Edit are denied," and general-purpose gets every tool available to subagents (sub-agents docs). If one of those already fits the job, you don't need a custom file.

A practical rule: if the subagent can run when you're not at the keyboard, such as from cron, CI, or a background session, write out its tools. If you start it yourself and watch the output, inheriting is acceptable.

Which tools can be assigned to a Claude Code subagent?

A Claude Code subagent can use any built-in or connected MCP tool, except a few that are always withheld. Built-in names come from the tools reference, which says they "are the exact strings you use in permission rules, subagent tool lists, and hook matchers."

Tools a background subagent keeps

Subagents run in the background by default. A background subagent keeps every MCP tool but only these built-in tools (as of October 2026, sub-agents docs):

Group	Tool names
Read and search	`Read`, `Grep`, `Glob`, `LSP`
Change files	`Edit`, `Write`, `NotebookEdit`
Run commands	`Bash`, `PowerShell`, `Monitor`, `TaskStop`
Web	`WebFetch`, `WebSearch`
Workflow	`TodoWrite`, `Skill`, `ToolSearch`, `SendMessage`, `Artifact`
Worktrees	`EnterWorktree`, `ExitWorktree`

The list also includes SubagentHandback for a subagent that reports through it, and Agent, which follows the depth limit. Before v2.1.280, background subagents couldn't use LSP (sub-agents docs).

Tools no subagent gets, even if listed

AskUserQuestion, EndConversation, EnterPlanMode, ScheduleWakeup, WaitForMcpServers, Workflow, ExitPlanMode (unless permissionMode is plan), and Agent once the subagent hits the depth limit. Subagents can spawn their own subagents "up to three layers below the main conversation" by default. You change that limit with CLAUDE_CODE_MAX_SUBAGENT_SPAWN_DEPTH.

That affects how you design the job. A subagent can't stop and ask you a question, so its prompt has to contain everything it needs. And because the background filter drops some tools silently, "the same definition can resolve to different tools in the foreground and the background."

Starter lists by job (examples)

Read-only auditor: Read, Grep, Glob
Bug fixer that runs tests: Read, Edit, Bash, Grep, Glob
Docs writer with citations: Read, Write, WebFetch, WebSearch
Issue triager: Read, mcp__github__*

What happens when a subagent's tools don't match its job (failure modes)?

When a Claude Code subagent lacks a tool it needs, it either can't launch or does a weaker version of the task. When it has more tools than it needs, it can change things you never asked it to touch. Here are the failures you're most likely to hit, with fixes.

1. Every entry is wrong, so the subagent won't start

If nothing in tools resolves to a real tool, because of typos, lowercase names, or tools subagents can't have, Claude Code "usually refuses to launch the subagent," and the Agent tool returns an error naming the entries it couldn't resolve (sub-agents docs; errors page). The error text contains "spawned with zero tools."
Fix: check capitalization against the tools reference, and remove anything from the withheld list above.

2. Some entries are wrong, and nothing tells you

The error only appears when the list resolves to nothing. Suppose you write tools: Read, edit, Bash (example). edit doesn't resolve, so the subagent starts with Read and Bash. It may then try to make changes through shell commands, or report that it couldn't edit.
Fix: after any edit to the list, ask the subagent to name the tools it has before the first real run.

3. Background runs lose a tool you listed

A tool that isn't on the background keep-list gets removed with no error. A definition that works when you watch it in the foreground can then break overnight.
Fix: build your lists only from the background table above, or set background: false for interactive use. When a cron run stops halfway, also read /why-claude-code-stops-mid-automation-and-how-to-fix-it/.

4. `allowed-tools` in a subagent file

That key isn't a subagent field, so the subagent inherits every tool, and the safety limit you thought you'd set isn't there.
Fix: rename the key to tools.

5. A Bash specifier that removes all of Bash

disallowedTools: Bash(rm *) takes away the whole Bash tool, so test runs fail too.
Fix: move the pattern to permissions.deny in .claude/settings.json.

6. Too many tools on a read-only job

For example, a "reviewer" with inherited Bash and Write might run a formatter or rewrite files while it reviews. In acceptEdits or auto mode, nothing asks you first.
Fix: use an explicit read-only list, and add a hook (next sections) that denies writes from that agent type. For broader rule sets you can paste in, see /ai-agent-guardrails-example-copy-paste-claude-code-rules/.

7. A new MCP server reaches every inheriting agent

Connect a Slack server, and every subagent without a tools list can now post messages.
Fix: add disallowedTools: mcp__* to agents that should stay local, or define the server inline with mcpServers so only one agent connects to it.

What are Claude Code hooks and how do they relate to subagents?

Claude Code hooks are commands (or HTTP calls, MCP tools, prompts, or agents) that run automatically at fixed lifecycle points. They give you a second layer of enforcement that keeps working even when a subagent's tool list is wrong. The tools field decides what a subagent can try. A hook can inspect each attempt as it happens and block it.

Hooks reach subagents in two ways (sub-agents docs):

Session-wide, from settings. Hooks in settings files, managed policy, and plugins all apply inside subagents, so "a PreToolUse hook in settings.json also runs before every tool a subagent uses."
Per subagent, from frontmatter. A hooks: block in the agent file runs only while that subagent is active and is removed when it finishes. A Stop hook in frontmatter becomes SubagentStop (hooks reference).

Each hook gets JSON input that includes agent_id and agent_type (the subagent's name), so one settings-level script can apply different rules to different agents. SubagentStart and SubagentStop also accept a matcher on the agent's name. Plugin-scoped names like my-plugin:db-agent contain a colon, which makes them regex, so anchor them as ^my-plugin:db-agent$.

Example frontmatter hook, adapted from the docs:

---
name: db-migrator
description: Runs database migrations
tools: Read, Bash
hooks:
  PreToolUse:
    - matcher: "Bash"
      hooks:
        - type: command
          command: "./scripts/security-check.sh"
---

For a full map of every event and when it fires, see /claude-code-hooks-lifecycle-every-event-explained/.

What do Claude Code hooks PreToolUse, PostToolUse, UserPromptSubmit, and Notification do?

In Claude Code, PreToolUse runs before a tool call and can block it, and PostToolUse runs after the tool succeeds. UserPromptSubmit runs when a prompt is submitted. Notification runs when Claude Code sends a notification and is for side effects only.

Event	When it fires	Matcher filters on	Can it block?	Typical subagent use
`PreToolUse`	Before a tool executes	Tool name (`Bash`, `Edit\	Write`,` mcp_memory_.*`)	Yes: exit code 2, or `permissionDecision: "deny"`
`PostToolUse`	After a tool succeeds	Tool name	Not the tool, which already ran. `decision: "block"` stops Claude from continuing	Run a linter after a subagent edits
`UserPromptSubmit`	When a prompt is submitted, before Claude processes it	No matcher; fires every time	Yes: exit code 2 or `decision: "block"`	Add project context to every turn
`Notification`	When Claude Code sends a notification	`permission_prompt`, `idle_prompt`, `auth_success`, `agent_needs_input`, `agent_completed`, and others	No, exit code is ignored	Alert you when an unattended run needs input

All four come from the hooks reference (as of October 2026). A few more details:

PreToolUse decisions: permissionDecision accepts "allow", "deny", "ask", or "defer". updatedInput can rewrite the tool's arguments before they run, and additionalContext passes a note to Claude.
PostToolUse: updatedToolOutput can replace what the tool returned.
UserPromptSubmit also fires on turns Claude Code starts by itself, which matters for automations.
Default timeouts: 600 seconds for command, HTTP, and MCP tool hooks, 30 for prompt hooks, and 60 for agent hooks.

Exit codes work like this. 0 means success, and any JSON output is applied. 2 is a blocking error on events that can block. Any other code is a non-blocking error, and the action goes ahead unless you set onFailure: "block". For ready-made scripts on these events, see /claude-code-hooks-examples-7-copy-paste-scripts/.

What is a Claude Code hook and how do you configure one?

A Claude Code hook is a JSON settings entry that runs a handler when an event matches. The handler is usually a shell command, and the entry links it to an event and a matcher. Here is how to set up a hook that stops a read-only subagent from writing files, even if its tool list was set up wrong.

Steps

Choose the scope. .claude/settings.json is for one project and is shared through git. .claude/settings.local.json is for one project and is gitignored. ~/.claude/settings.json covers all your projects (hooks reference).
Create the script at .claude/hooks/deny-reviewer-writes.sh (example):

#!/bin/bash
INPUT=$(cat)
AGENT=$(echo "$INPUT" | jq -r '.agent_type // empty')

if [ "$AGENT" = "code-reviewer" ]; then
  jq -n '{
    hookSpecificOutput: {
      hookEventName: "PreToolUse",
      permissionDecision: "deny",
      permissionDecisionReason: "code-reviewer is read-only"
    }
  }'
else
  exit 0
fi

Make it executable: chmod +x .claude/hooks/deny-reviewer-writes.sh. You also need jq installed.
Register it in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Edit|Write|NotebookEdit",
        "hooks": [
          {
            "type": "command",
            "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/deny-reviewer-writes.sh"
          }
        ]
      }
    ]
  }
}

Open /hooks in a session to confirm the hook is listed and to see which file it came from.
Run the reviewer on a small diff and ask it to fix something. The write attempt should come back denied, with your reason.

Matcher rules to get right

A matcher that is "*", "", or left out matches everything. A value made only of letters, digits, _, -, spaces, , and | is treated as an exact name or a list. Anything else is treated as an unanchored JavaScript regex (hooks reference). So mcp__github__.* is a regex that covers every GitHub MCP tool. You can narrow further with the if field, which takes permission-rule syntax such as "Bash(rm *)".

Common mistakes and fixes

Script paths relative to cwd break in worktrees. Use ${CLAUDE_PROJECT_DIR}. In worktrees it stays at the main checkout, while cwd points to the worktree.
Checking tool_name == "Task". Since v2.1.63 the name is Agent (issue #29677).
Expecting Notification to block. It can't. Use PreToolUse or a permission rule.
A crashing script lets the action through. A non-zero exit other than 2 doesn't block. For security hooks, set "onFailure": "block".
Hooks seem to do nothing. Look for "disableAllHooks": true in your settings, or a --settings override in your cron command. Cron wrappers are covered in /claude-code-cron-job-setup-run-automations-overnight/.

Next steps

Do today (DIY):

Search your agents for the wrong key: grep -rn "allowed-tools" .claude/agents ~/.claude/agents. Rename every match to tools.
For each subagent that runs unattended, write out a tools list built only from the background keep-list, then ask the subagent to list its tools before its next real run.
Add one PreToolUse hook like the one above for your most sensitive agent, and put Bash(git push *) in permissions.deny if no agent should push.

If you'd rather not set this up by hand: the main official option is a Claude subscription that includes Claude Code, so you can ask Claude to write and check agent files for you. Claude Pro is $20/month ($17/month billed annually), and Max starts at $100/month with 5x or 20x Pro's usage (as of October 2026; check the official pricing page). Pro suits solo builders running a few scheduled subagents. Max suits people who run many parallel or long jobs and keep hitting limits. Neither plan reviews your configuration for you, and the Free plan doesn't include Claude Code. If your runs are small and occasional, pay-as-you-go API use may cost less. Token tracking is covered in /claude-code-how-to-see-token-usage-and-reduce-it/.

Related guides on this site:

/claude-code-permissions-best-practices-headless-template/
/claude-code-hooks-lifecycle-every-event-explained/
/claude-code-hooks-examples-7-copy-paste-scripts/
/ai-agent-guardrails-example-copy-paste-claude-code-rules/
/claude-code-checkpoints-how-rollback-works-in-automations/

FAQ

What tools can a Claude Code subagent use?

If you don't set a tools field, it inherits the main conversation's built-in and MCP tools, minus a few that are always withheld, such as AskUserQuestion and EnterPlanMode. Background subagents, the default, keep only a fixed set of built-ins like Read, Edit, Write, Bash, Grep, Glob, WebFetch, and WebSearch, plus all MCP tools. Use tools or disallowedTools to narrow that set.

What are Claude Code hooks?

Hooks are handlers you configure in settings files or agent frontmatter, usually shell commands, that run automatically at lifecycle events such as PreToolUse, PostToolUse, SessionStart, and SubagentStop. They get JSON describing the event and can allow, deny, or change some actions. Hooks set in settings also run inside subagents.

Where are the docs for PreToolUse, PostToolUse, UserPromptSubmit, and Notification?

All four are on the official hooks reference, with input fields, matcher values, and decision options for each. The sub-agents page covers how those events behave inside subagents and the SubagentStart/SubagentStop matchers.

How do I make a Claude Code hook block a command?

Register a PreToolUse hook with a matcher such as Bash, and either exit with code 2 or print JSON with permissionDecision: "deny" and a reason. Add "onFailure": "block" if a crashing script should also block. For simple command patterns, a permissions.deny rule is often enough and needs no script.

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code Output Format JSON: Fields, jq & Python Parsing

Aoba Works — Sat, 31 Oct 2026 16:27:17 +0000

Originally published at Unattended.

To get JSON from Claude Code, run claude -p "your prompt" --output-format json. When the run finishes, it prints one JSON object with the answer in result and run metadata next to it: type, subtype, is_error, duration_ms, num_turns, session_id, total_cost_usd and usage. The official docs call this format "structured JSON with result, session ID, and metadata". If you want events while the run is still going, use --output-format stream-json. It prints one JSON object per line, and the last line is the same result object.

One thing to know before you write a parser: many older posts mention a cost_usd field. The current Agent SDK reference calls the cost field total_cost_usd. If your script reads .cost_usd, it will quietly get null back.

Which option is right for you?

Use json if you only need the final outcome. Use stream-json if you need progress, tool calls or retry events while the run is in progress.

Your situation	What to pick first	First page to open
Beginner, first headless script	`--output-format json` plus `jq -r '.result'`	Run Claude Code programmatically
Nightly cron job that must alert on failure	`json`, then check `subtype` and `is_error`	/claude-code-cron-job-setup-run-automations-overnight/
Long runs you want to watch live, or retry visibility	`stream-json --verbose`	Stream responses section
Budget-conscious, tracking spend per run	`json`, log `total_cost_usd` and `usage`	/claude-code-how-to-see-token-usage-and-reduce-it/
You need typed fields (lists, flags) and not prose	`json` + `--json-schema`, read `structured_output`	Get structured output
You only want to read a saved output file nicely	VS Code's built-in JSON formatter	VS Code JSON docs

Why each pick works:

Beginner: json gives you one object, so one jq call answers "did it work, and what did it say?"
Cron job: Claude Code returns a non-zero exit code when a run fails, but the JSON tells you how it failed (error_max_turns and error_max_budget_usd call for different fixes).
Live monitoring: stream-json shows system/api_retry events, so you can tell a stalled run apart from one that is waiting on a retry.
Budget: total_cost_usd gives you a spend figure for every run without logging into a dashboard.

What does --output-format json return in Claude Code?

--output-format json returns one result object when the run ends. That object holds the final text in result, plus status, timing, cost and token metadata.

The headless docs describe the payload and say it includes "total_cost_usd and a per-model cost breakdown" (headless docs). The full field list comes from the ResultMessage definition in the Python Agent SDK reference, which describes the same result message the CLI prints (as of October 2026):

Field	Type	What it tells your script
`type`	string	Always `"result"` for this object. In `stream-json`, use it to find the final line
`subtype`	string	`"success"`, `"error_during_execution"`, `"error_max_turns"`, `"error_max_budget_usd"` or `"error_max_structured_output_retries"`
`is_error`	boolean	`true` when the run ended in an error state
`result`	string or absent	The final text answer
`structured_output`	any, optional	Present only if you passed `--json-schema` and the run produced valid output
`duration_ms`	integer	Wall-clock time for the whole run
`duration_api_ms`	integer	Time spent waiting on model API calls
`num_turns`	integer	How many agent turns ran (compare it with your `--max-turns`)
`session_id`	string	Pass to `--resume` to continue this conversation
`stop_reason`	string or null	Why the model stopped
`total_cost_usd`	number	Estimated spend for the conversation
`usage`	object	Token counts (input, output, cache creation, cache read)
per-model usage	object	Per-model breakdown. Named `modelUsage` in the TypeScript reference, `model_usage` in the Python SDK
`permission_denials`	array	Tool calls that were denied, which is useful in unattended runs
`errors`	array, optional	Error details when the run failed

The Python reference says it plainly: "The subtype field determines which other fields are populated." So read subtype first and the other fields after it.

An illustrative json result

Here is the shape of the output, with example values made up for illustration and some fields left out:

{
  "type": "result",
  "subtype": "success",
  "is_error": false,
  "duration_ms": 18432,
  "duration_api_ms": 15110,
  "num_turns": 4,
  "result": "Found 2 failing tests in auth.spec.ts and fixed both.",
  "session_id": "example-session-id",
  "total_cost_usd": 0.0412,
  "usage": { "input_tokens": 1200, "output_tokens": 640 }
}

The exact set of keys can change between Claude Code versions. Before you hard-code anything, run this once on your machine:

claude -p "Say hi" --output-format json | jq 'keys'

Three facts about cost and exit codes

Cost is an estimate. The docs say: "Both figures are client-side estimates and can differ from your actual bill." Use it for trends and alerts. Don't use it for invoicing.
Cost adds up across a session. If you use --continue or --resume, the run reports the conversation's whole total, including what earlier runs spent (headless docs). Summing total_cost_usd across resumed runs therefore double-counts.
Exit code plus stdout. Claude Code "exits with code 0 on success and a non-zero code when the run fails." Invalid flags go to stderr. But "When a failure happens inside the run, such as missing authentication, Claude Code prints the failure as the result on stdout." So stdout can contain an error message sitting in the result field.

Getting typed data with --json-schema

If your monitor needs fields and not prose, add --json-schema. The validated data shows up in structured_output. The docs warn that when a run ends without valid structured output, "the response has no structured_output field" (headless docs). Check that the field exists before you read it. If the schema itself is invalid, claude exits with Error: --json-schema is not a valid JSON Schema.

How do you use Claude Code output format stream-json?

Claude Code's stream-json format prints newline-delimited JSON events while the agent works, and finishes with a result line. Read it one line at a time.

The documented command for live token streaming is:

claude -p "Explain recursion" --output-format stream-json --verbose --include-partial-messages

"The last line of the stream is a result message with the final response text, cost, and session metadata." Before that line you'll see these event types (from the headless docs):

Event (`type` / `subtype`)	When it appears	Useful for
`system` / `init`	First, unless hook or plugin-install events come before it	Model, tools, MCP servers, plugins, `plugin_errors`, `mcp_server_errors`
`assistant`, `user`	Every turn, including subagent messages (`parent_tool_use_id` set)	Logging tool calls and results
`stream_event`	Only with `--include-partial-messages`	Live text deltas
`system` / `api_retry`	Before each retry of a retryable API error	`attempt`, `max_retries`, `retry_delay_ms`, `error`
`system` / `permission_denied`	When a tool call is denied	Finding out which permissions are missing
`result`	Last line	Same fields as the `json` format

Steps: consume stream-json in a pipeline

Run with --output-format stream-json --verbose. Add --include-partial-messages only if you need token-level text.
Read stdout line by line. Each line is a complete JSON object, and the stream as a whole is not one JSON document.
When you see type == "system" with subtype == "init", check mcp_server_errors and plugin_errors. Claude Code skips invalid --mcp-config entries and "the run continues and exits cleanly," so a non-empty array is your only signal (headless docs).
On system/api_retry, log attempt and error. The error values include rate_limit, overloaded, billing_error and authentication_failed.
On type == "result", apply the same success checks you'd apply to the json format.

stream-json pitfalls

Running jq '.result' on the whole stream. jq applies the filter to every line, so you get a column of nulls with the answer at the end. Filter with select(.type == "result").
No text deltas. Without --include-partial-messages, you won't get stream_event lines. You get whole messages only.
A slow reader holds the process open. If your consumer reads slowly, Claude Code waits for queued output to drain before it exits, "capped at 30 seconds" (headless docs).
Subagent noise. Messages from subagents appear in the same stream. If you only want the main conversation, filter on parent_tool_use_id == null.

How do you parse Claude Code JSON output with jq?

Pipe claude -p ... --output-format json into jq -r '.result' for the text, or into jq field paths such as .total_cost_usd and .is_error for monitoring values.

The jq flags that matter here, quoted from the jq manual:

-r: a string result "will be written directly to standard output rather than being formatted as a JSON string with quotes."
-j: "Like -r but jq won't print a newline after each output."
-e: sets the exit status to 1 "if the last output value was either false or null," which makes jq usable as a test.
-c: puts "each JSON object on a single line," which is handy for appending to a log.

Copy-paste jq snippets

Save the output once, then read it as many times as you need. Each of these is an example; adjust paths to your setup:

# 1. Run once, keep the raw JSON
out=$(claude -p "Run the test suite and summarize failures" \
  --output-format json --allowedTools "Bash,Read")

# 2. Just the answer
echo "$out" | jq -r '.result'

# 3. Monitoring values in one line (tab-separated)
echo "$out" | jq -r '[.subtype, .is_error, .duration_ms, .num_turns, .total_cost_usd] | @tsv'

# 4. Fail the script unless the run succeeded
echo "$out" | jq -e '.subtype == "success" and .is_error == false' > /dev/null \
  || { echo "Claude run failed: $(echo "$out" | jq -r '.subtype')"; exit 1; }

# 5. Append a compact record to a JSONL log for later analysis
echo "$out" | jq -c '{ts: now, subtype, is_error, duration_ms, num_turns, total_cost_usd, session_id}' \
  >> ~/claude-runs.jsonl

# 6. Default a missing field instead of printing null
echo "$out" | jq -r '.structured_output // "NO_STRUCTURED_OUTPUT"'

Snippet 6 relies on jq's alternative operator //, which returns the right side when the left side is false or null (jq manual).

For stream-json, the docs filter text deltas like this:

claude -p "Write a poem" --output-format stream-json --verbose --include-partial-messages | \
  jq -rj 'select(.type == "stream_event" and .event.delta.type? == "text_delta") | .event.delta.text'

To keep only the final record from a stream, as an example:

claude -p "Nightly dependency audit" --output-format stream-json --verbose \
  | tee run.jsonl \
  | jq -c 'select(.type == "result") | {subtype, is_error, total_cost_usd}'

tee saves the full transcript, so you can debug later without re-running and paying again.

jq pitfalls

Quoting. Put jq programs in single quotes so the shell doesn't expand $ or !.
jq -e and false. jq -e '.is_error' exits 1 when is_error is false, which is the healthy case. Test the condition you want to be true, as snippet 4 does.
Parsing twice. Each claude -p call is a new paid run. Capture the output in a variable or file first.

How do you parse Claude Code JSON output with Python?

For the json format, pass the subprocess stdout to json.loads(). For stream-json, call json.loads() on each line. In both cases, branch on subtype and is_error.

Python's json.loads handles one document. "If the data being deserialized is not a valid JSON document, a JSONDecodeError will be raised." That's why stream output needs a per-line loop.

Example: a minimal monitor for json output

This is an illustrative script. Replace alert() with whatever notifier you already use:

import json
import subprocess
import sys

def alert(msg: str) -> None:
    print(f"ALERT: {msg}", file=sys.stderr)  # swap for your webhook, mail, etc.

proc = subprocess.run(
    ["claude", "-p", "Summarize yesterday's error log in logs/app.log",
     "--output-format", "json", "--allowedTools", "Read"],
    capture_output=True, text=True, timeout=1800,
)

try:
    data = json.loads(proc.stdout)
except json.JSONDecodeError:
    alert(f"No valid JSON (exit {proc.returncode}). stderr: {proc.stderr[:500]}")
    sys.exit(1)

subtype = data.get("subtype")
if proc.returncode != 0 or data.get("is_error") or subtype != "success":
    alert(f"Run failed: subtype={subtype} turns={data.get('num_turns')}")
    sys.exit(1)

cost = data.get("total_cost_usd") or 0
if cost > 1.00:  # example threshold
    alert(f"Run cost ${cost:.2f}, above threshold")

print(data.get("result", ""))

Keep stdout and stderr separate (capture_output=True does that). The stderr text then shows up in your alert without breaking the JSON parse.

Example: reading stream-json line by line

import json, subprocess

cmd = ["claude", "-p", "Refactor utils.py and run tests",
       "--output-format", "stream-json", "--verbose"]

with subprocess.Popen(cmd, stdout=subprocess.PIPE, text=True) as proc:
    final = None
    for line in proc.stdout:
        line = line.strip()
        if not line:
            continue
        event = json.loads(line)
        t, st = event.get("type"), event.get("subtype")
        if t == "system" and st == "init" and event.get("mcp_server_errors"):
            print("MCP config problem:", event["mcp_server_errors"])
        elif t == "system" and st == "api_retry":
            print(f"retry {event['attempt']}/{event['max_retries']}: {event['error']}")
        elif t == "result":
            final = event

if final is None:
    print("Stream ended without a result line (killed or crashed?)")
elif final.get("is_error"):
    print("Failed:", final.get("subtype"))

Python pitfalls

Treating final is None as success. If something sends the process SIGTERM, Claude Code "exits with code 143" and "records no result" for the turn in progress (headless docs). A missing result line means failure.
Using subprocess.run for long streams. It buffers everything until the process exits, so you lose the live view. Use Popen and iterate.
Python SDK names versus raw JSON names. If you later switch to the Python Agent SDK, the per-model field becomes model_usage. In raw CLI JSON, check the key name with jq 'keys'.

If you want the bigger picture on catching runs that "succeed" but did nothing useful, see /ai-agent-monitoring-in-claude-code-catch-silent-failures/.

How do you format a JSON file in VS Code?

Open the .json file and run Format Document. The shortcut is ⇧⌥F on Mac, Shift+Alt+F on Windows, or Ctrl+Shift+I on Linux, or you can right-click and choose Format Document.

That shortcut comes straight from the VS Code JSON docs: "You can format your JSON document using ⇧⌥F (Windows Shift+Alt+F, Linux Ctrl+Shift+I) or Format Document from the context menu."

Steps for a saved Claude Code output:

Save the output: claude -p "..." --output-format json > run.json.
In VS Code, choose File > Open File... and select run.json.
Check that the status bar in the bottom-right says JSON. If it says Plain Text, click it and pick JSON.
Press the Format Document shortcut, or right-click > Format Document.
To format only part of the file, select it and use Format Selection (⌘K ⌘F on Mac, Ctrl+K Ctrl+F on Windows/Linux), per the VS Code basic editing docs.

Formatting pitfalls

stream-json files aren't JSON documents. A .jsonl file with many objects won't format as one document. Convert it to an array first, for example jq -s '.' run.jsonl > run.json, or pull out just the result line with jq 'select(.type=="result")' run.jsonl > result.json.
JSON with Comments warnings. In JSONC mode, VS Code "accepts trailing commas, but they are discouraged and the editor will display a warning" (VS Code JSON docs). Claude Code output contains neither comments nor trailing commas, so a warning there usually means the file was edited by hand.
"There is no formatter for 'plaintext' files." The language mode is wrong. Fix it as in step 3.

VS Code JSON formatter: built-in or Prettier?

For reading Claude Code output, VS Code's built-in JSON formatter is enough. Prettier is worth adding only when a team needs the same formatting rules everywhere.

	Built-in JSON formatter	Prettier extension (`esbenp.prettier-vscode`)
Install	Ships with VS Code	Marketplace extension. Free, listed with over 72 million installs (Marketplace, as of October 2026; check the official page)
Config	VS Code settings only	`.prettierrc` in the repo, shared with CI
Defaults	Follows your editor indent settings	`printWidth` 80, `tabWidth` 2 (Prettier options)
Same rules in CLI/CI	No	Yes (`npx prettier --check`)
Good for	Reading `run.json`, quick fixes	Teams committing JSON fixtures or config

Who Prettier is not for: if you just want to read the occasional output file, Prettier adds an extension, a config file and a second formatter that can fight with the built-in one. One limitation: once two formatters are installed, VS Code may ask you to pick a default, and formatting does nothing until you do.

Who the built-in formatter is not for: teams that want a pre-commit or CI check. It only runs inside the editor.

My pick for this audience is the built-in formatter for viewing output, plus jq . on the command line, which pretty-prints by default. Add Prettier only when JSON files go into a shared repo.

How do you auto-format JSON on save in Visual Studio Code?

To auto-format JSON in Visual Studio Code, turn on editor.formatOnSave and set a default formatter for the [json] language in settings.json.

VS Code documents three automatic triggers: editor.formatOnType, editor.formatOnSave ("Format a file on save") and editor.formatOnPaste (basic editing docs).

Steps:

Open the Command Palette (⇧⌘P on Mac, Ctrl+Shift+P on Windows/Linux).
Run Preferences: Open User Settings (JSON), or Preferences: Open Workspace Settings (JSON) to apply it to one project only.
Add a per-language block. Example with the built-in formatter:

{
  "[json]": {
    "editor.defaultFormatter": "vscode.json-language-features",
    "editor.formatOnSave": true
  },
  "[jsonc]": {
    "editor.defaultFormatter": "vscode.json-language-features",
    "editor.formatOnSave": true
  }
}

To use Prettier instead, swap the formatter ID for esbenp.prettier-vscode. The Marketplace page shows the same per-language pattern.
Save the file, open a run.json, change something and save it. It should re-indent.

Two catches. Format-on-save doesn't run when a script writes the file. It only runs when you save in the editor. And workspace settings override user settings, so if nothing happens, check .vscode/settings.json in the repo.

What breaks when you pipe Claude Code JSON into monitoring scripts?

Four things break most pipelines: mixed stderr, exit-code-only checks, a missing result line, and misread cost.

Symptom	Likely cause	Fix
`jq: parse error` or `JSONDecodeError`	You used `2>&1`, so stderr text landed in the JSON stream	Send stderr to its own file: `2>>claude.err`
Alert never fires but the work wasn't done	Script checks exit code only, or reads `.cost_usd` (null) (Python SDK reference)	Check `subtype == "success"` and `is_error == false`, and use `total_cost_usd`
Cost charts jump upward	Summing `total_cost_usd` across `--resume` runs that report cumulative totals (headless docs)	Log `session_id` and keep only the latest value per session
`structured_output` is null	Run declined or failed schema retries (`error_max_structured_output_retries`) (Python SDK reference)	Check the field exists, alert on that subtype
MCP tools silently missing	Invalid `--mcp-config` entry skipped, run "exits cleanly" (headless docs)	Fail on non-empty `mcp_server_errors` in `system/init`
No result line at all	Process killed with SIGTERM (exit 143) (headless docs)	Treat a missing result as failure, and send SIGINT for graceful stops
Run hangs after the answer	Waiting on background work, 10-minute idle cap by default (headless docs)	Set `CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS` or avoid background tasks in `-p`
Piped input rejected	Piped stdin is capped at 10MB (headless docs)	Write the data to a file and reference its path in the prompt

Sources for the table rows: headless docs and the Python SDK reference.

The stderr row is the one people hit most often. The docs note that the "Waiting for background work to finish" message goes to stderr, and that with JSON output it prints only when stdout isn't a terminal, "so the JSON your script reads never carries it" (headless docs). If you merge the two streams yourself, that guarantee is gone.

For CI and cron, the docs recommend --bare, which skips local hooks, plugins, MCP servers and CLAUDE.md, so every machine gets the same system/init (headless docs). It needs ANTHROPIC_API_KEY because bare mode doesn't use your subscription login. The headless flags and auth trade-offs are covered in /claude-code-headless-mode-flags-auth-and-cost-2026/. If your alerts keep showing rate_limit retries, see /claude-code-rate-limit-reached-retry-setup-for-cron-jobs/.

Next steps

Three things you can do today:

Run claude -p "Say hi" --output-format json | jq 'keys' and save the key list. That's the schema for your installed version.
Wrap your existing cron command with jq snippet 4 (fail unless subtype == "success") and snippet 5 (append a JSONL record). That gives you an alert and a cost log in about ten lines.
If any job uses --mcp-config, switch it to stream-json --verbose and fail on a non-empty mcp_server_errors.

If you'd rather not build and maintain this yourself: Claude Code's official GitHub Actions integration runs headless jobs inside GitHub, so run status and logs live in the Actions UI and you don't need your own parser. It suits people whose recurring work already lives in a GitHub repo. It's a poor fit for jobs that need files on your own server, or for anyone who wants per-run cost alerts (you'd still parse the JSON for those). Costs are your GitHub Actions minutes plus your Claude usage, through an API key or a plan. Claude Pro is listed at "$20 if billed monthly" and Max "From $100 Per month" on Claude's pricing page (as of October 2026; check the official page). For scripted --bare runs, remember they need an API key and won't use your subscription login.

Related guides on this site:

/claude-code-cron-job-setup-run-automations-overnight/: scheduling the runs you're now parsing
/ai-agent-monitoring-in-claude-code-catch-silent-failures/: what to check beyond is_error
/claude-code-how-to-see-token-usage-and-reduce-it/: turning usage numbers into savings

FAQ

How do I format a JSON file in VS Code?

Open the file, check that the status bar language mode is JSON, then press ⇧⌥F on Mac, Shift+Alt+F on Windows or Ctrl+Shift+I on Linux. You can also right-click and choose Format Document. For Claude Code stream-json files, convert to an array first with jq -s '.', because a multi-object file isn't one JSON document.

Which VS Code JSON formatter should I use?

For reading and occasionally fixing JSON, use the built-in formatter that ships with VS Code. Pick Prettier if your team commits JSON files and wants the same rules checked in CI. Running both without setting editor.defaultFormatter leads to "which formatter?" prompts.

What is Claude Code output format stream-json?

It's a newline-delimited JSON format. Each line is an event (system/init, assistant, user, system/api_retry, optional stream_event deltas), and the final line is a result object with the answer, cost and session metadata. The docs use it with --verbose, and with --include-partial-messages if you want token-level text.

How do I format JSON in Visual Studio Code automatically?

Add "editor.formatOnSave": true and an "editor.defaultFormatter" inside a "[json]" block in your settings JSON. Formatting happens when you save in the editor. Files written by scripts stay as they are until you open and save them.

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code Regression Testing After Prompt Changes: Steps

Aoba Works — Fri, 30 Oct 2026 16:16:57 +0000

Originally published at Unattended.

To regression-test Claude Code after a prompt change, run a fixed set of test cases several times with your current CLAUDE.md and save the results. Then run the same cases with the edited file, grade both sets the same way, and compare pass counts per case. Use a threshold so one random failure doesn't block a good edit and a real drop doesn't slip through. Anthropic's own eval guide says regression evals ask "Does the agent still handle all the tasks it used to?" and "should have a nearly 100% pass rate." This guide covers that workflow end to end, with scripts you can adapt.

Which option is right for you?

Choose your setup by how often the agent runs unattended and how much you want to spend on test runs.

Your situation	What to pick first	First page to open
Beginner with one recurring Claude Code job	5 cases × 3 runs, run by hand with a shell script before each CLAUDE.md edit	Headless mode docs
Daily work use (agent runs every night)	10–15 cases × 5 runs, triggered by GitHub Actions on any PR touching `CLAUDE.md`	GitHub Actions docs
Tight budget	Code-only graders (no LLM judge) and a per-run `--max-budget-usd` cap	Headless mode: cost fields
Switching models (e.g. Sonnet to Opus)	Same suite, with the full model ID and effort level pinned on both sides	Model configuration
You'd rather use a ready-made eval tool	Promptfoo's Claude Agent SDK provider	Promptfoo provider page

If you haven't set up an eval loop yet, start with the pillar guide, AI Agent Evaluation Framework: Claude Code Eval Loop Steps. This article covers one narrower job: comparing before and after a change.

Why do prompt changes to CLAUDE.md silently break agent behavior?

CLAUDE.md edits change the context Claude reasons over. A rule that used to work can get less weight or be contradicted by a new line, and nothing throws an error.

The Claude Code memory docs say this directly: "Claude treats them as context, not enforced configuration." The same page adds that "the more specific and concise your instructions, the more consistently Claude follows them." So there's no parser that rejects a bad CLAUDE.md. A vague new rule, or one that conflicts with an old one, just changes the odds of what the agent does.

Anthropic shipped a regression like this in Claude Code itself. Its April 2026 postmortem traces one of three quality problems to a system prompt line added on April 16 that capped text between tool calls and final responses at short word counts. A broader eval later "showed a 3% drop for both Opus 4.6 and 4.7," and the prompt was reverted in the April 20 release. In Anthropic's words, "one addition to the system prompt caused an outsized effect on intelligence in Claude Code." Their first eval set didn't catch it. The fix they committed to was "a broad suite of per-model evals for every system prompt change" and ablations "to understand the impact of each line."

For example, suppose your nightly dependency-update agent has a CLAUDE.md rule saying "never edit files under migrations/." You add a line for brevity: "Keep final reports under 50 words." Nothing obvious breaks. But the shorter report may drop the "files changed" list your morning check depends on. The agent may also skip the step where it re-reads the rules before editing. You'd only see this by comparing runs.

Pitfalls specific to CLAUDE.md:

Edits outside the file you're looking at. User-level ~/.claude/CLAUDE.md, .claude/rules/, and auto memory all load too (memory docs), so the "before" and "after" may differ in more than one file.
Hard rules written as prose. If an action must never happen, the docs point to a PreToolUse hook rather than a CLAUDE.md line. A regression suite measures how well guidance is followed. A hook enforces it. Our guardrails guide shows which rules belong where.

How do you set up a baseline run before editing a prompt?

Run your current CLAUDE.md against a fixed set of representative inputs and save every output before you change anything. That snapshot is your baseline.

There's one trap to clear first. Headless mode has a --bare flag that skips "auto-discovery of hooks, skills, … auto memory, and CLAUDE.md" (headless docs). The same page says "--bare is the recommended mode for scripted and SDK calls, and will become the default for -p in a future release." That's the right choice for most scripts. For a test of CLAUDE.md, though, it would quietly test nothing. So the baseline runs a normal claude -p inside a fresh copy of a fixture project, with only the CLAUDE.md you're testing in it.

Steps:

Create an evals/ folder in your repo with cases/ (one JSON file per case), fixtures/ (small sample projects), and schema.json (the output shape you'll grade).
Pin everything that isn't the variable you're testing: the full model ID (claude-opus-5-5, per the model config page), the effort level, and the Claude Code version. The setup page shows how to install an exact version, for example curl -fsSL https://claude.ai/install.sh | bash -s 2.1.89, and how to stop background updates with DISABLE_AUTOUPDATER (setup docs).
Ask for structured output. With --output-format json --json-schema '<schema>', the response puts schema-shaped data in a structured_output field. If the run ends without valid output, that field is missing (headless docs).
Copy the current CLAUDE.md to evals/baseline/CLAUDE.md and commit it, so the baseline is reproducible.
Run every case N times (3 to start) and save each run's JSON, the list of changed files, and the test exit code.

An example evals/schema.json for a "fix failing tests" job:

{
  "type": "object",
  "properties": {
    "status": { "enum": ["done", "blocked"] },
    "files_changed": { "type": "array", "items": { "type": "string" } },
    "tests_passed": { "type": "boolean" },
    "summary": { "type": "string" }
  },
  "required": ["status", "files_changed", "tests_passed", "summary"]
}

An example case file, evals/cases/fix-date-parse.json:

{
  "id": "fix-date-parse",
  "fixture": "evals/fixtures/date-utils",
  "prompt": "Run the tests and fix the failing date parsing test. Do not touch the migrations folder.",
  "allowed_paths": "^src/",
  "assert": ".structured_output.status == \"done\""
}

The runner script, evals/run.sh (an example to adapt; it assumes jq and git are installed):

#!/usr/bin/env bash
# usage: evals/run.sh <label> <path/to/CLAUDE.md> [runs]
set -euo pipefail
ROOT=$(pwd); LABEL=$1; MEMORY=$2; RUNS=${3:-3}
MODEL=${EVAL_MODEL:-claude-opus-5-5}; EFFORT=${EVAL_EFFORT:-high}
SCHEMA=$(cat evals/schema.json)

for case in evals/cases/*.json; do
  id=$(jq -r .id "$case"); prompt=$(jq -r .prompt "$case")
  fixture=$(jq -r .fixture "$case")
  for i in $(seq 1 "$RUNS"); do
    work=$(mktemp -d); out="$ROOT/results/$LABEL/$id/run$i"; mkdir -p "$out"
    cp -R "$fixture"/. "$work"/ && cp "$MEMORY" "$work/CLAUDE.md"
    (cd "$work" && git init -q && git add -A && git commit -qm base)
    (cd "$work" && claude -p "$prompt" \
       --model "$MODEL" --effort "$EFFORT" \
       --output-format json --json-schema "$SCHEMA" \
       --allowedTools "Read,Edit,Bash(npm test *)" \
       --permission-mode dontAsk --max-turns 30 --max-budget-usd 1 \
       > "$out/result.json") || echo $? > "$out/claude_exit"
    (cd "$work" && git diff --name-only > "$out/changed.txt" \
       && { npm test --silent >/dev/null 2>&1; echo $? > "$out/tests_exit"; })
  done
done

Each run starts from a clean copy because "each trial should be 'isolated' by starting from a clean environment" (Anthropic). Leftover files from run 2 shouldn't help or hurt run 3. Run the baseline with bash evals/run.sh baseline evals/baseline/CLAUDE.md 3.

The flags above are explained in more depth in Claude Code Headless Mode: Flags, Auth and Cost (2026). In short, dontAsk denies anything that would prompt, so the run can't hang waiting for approval.

How do you build a before/after diff workflow for prompt changes?

Run the same cases through the old and new CLAUDE.md, then compare graded structured fields per case. Don't compare raw text, which differs on every run even when behavior hasn't changed.

The workflow:

Save your edit as evals/candidate/CLAUDE.md. Leave the real CLAUDE.md alone until the comparison passes.
Run bash evals/run.sh candidate evals/candidate/CLAUDE.md 3.
Grade both folders with the same grader (next section).
Compare per-case pass counts with evals/compare.sh, and apply the threshold from the flaky-run section.
For every case that got worse, open the transcripts of a failing candidate run next to a passing baseline run. Anthropic's guide warns: "You won't know if your graders are working well unless you read the transcripts" (source).
Only then copy the candidate into the real CLAUDE.md and commit it, with the comparison summary in the commit message.

Example evals/compare.sh:

#!/usr/bin/env bash
# usage: evals/compare.sh baseline candidate
set -euo pipefail
join -t $'\t' <(sort "results/$1/summary.tsv") <(sort "results/$2/summary.tsv") |
while IFS=$'\t' read -r id base_pass base_n cand_pass cand_n; do
  verdict="ok"
  if   [ "$cand_pass" -lt "$base_pass" ] && [ $((base_pass - cand_pass)) -ge 2 ]; then verdict="REGRESSION"
  elif [ "$cand_pass" -lt "$base_pass" ]; then verdict="flaky-rerun"
  fi
  printf "%-24s base %s/%s  cand %s/%s  %s\n" "$id" "$base_pass" "$base_n" "$cand_pass" "$cand_n" "$verdict"
done | tee "results/compare-$2.txt"
! grep -q REGRESSION "results/compare-$2.txt"

The last line makes the script exit non-zero when any case regressed. A CI job can then fail the pull request.

Pitfalls:

Changing two things at once. If you edit CLAUDE.md and bump the model in the same run, you can't tell which one caused a drop. Anthropic's postmortem calls this kind of one-change-at-a-time testing "ablations."
Diffing summary. Free-text fields should be graded by a rubric, not a string diff.
Forgetting cost. The JSON output includes total_cost_usd per run (headless docs), described there as a client-side estimate. Sum it with jq -s 'map(.total_cost_usd // 0) | add' results/candidate/*/*/result.json so you know what each comparison costs.

What makes a good grader for automated regression checks?

A good grader checks what "done" means for your task, using code first. It's a rule-based check, or a separate Claude call with an explicit rubric. "Looks right" isn't a grader.

Start with code. Anthropic's eval guide describes code-based graders as "fast, cheap, objective, reproducible, easy to debug," and advises that "it's often better to grade what the agent produced, not the path it took" (source). For a coding agent, the outcome usually lives in the workspace: did the tests pass, which files changed, does the structured output say done?

Example evals/grade.sh, which writes one summary.tsv line per case (id, passes, runs):

#!/usr/bin/env bash
# usage: evals/grade.sh <label>
set -euo pipefail
LABEL=$1; : > "results/$LABEL/summary.tsv"
for case in evals/cases/*.json; do
  id=$(jq -r .id "$case"); allow=$(jq -r .allowed_paths "$case"); assert=$(jq -r .assert "$case")
  pass=0; n=0
  for run in results/$LABEL/$id/run*; do
    n=$((n+1)); ok=1
    jq -e '.structured_output' "$run/result.json" >/dev/null 2>&1 || ok=0
    [ "$(cat "$run/tests_exit" 2>/dev/null)" = "0" ] || ok=0
    grep -vE "$allow" "$run/changed.txt" | grep -q . && ok=0      # touched a forbidden path
    jq -e "$assert" "$run/result.json" >/dev/null 2>&1 || ok=0
    [ $ok -eq 1 ] && pass=$((pass+1))
  done
  printf "%s\t%s\t%s\n" "$id" "$pass" "$n" >> "results/$LABEL/summary.tsv"
done

Add a model-based grader only for parts code can't check, like whether a report is accurate or follows a tone rule. Make it a separate claude -p --bare call. Bare mode is correct here, because the judge shouldn't load your agent's CLAUDE.md, and in bare mode Claude Code needs ANTHROPIC_API_KEY set (headless docs). Rules that keep it honest:

Grade one dimension per judge call. Anthropic recommends grading "each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions."
Give it a way out. Allow "unknown" as a verdict, as the same guide suggests, and count unknowns as "needs a human look," not as passes.
Use a detailed rubric and ask for reasons before the score. Anthropic's test-development docs list detailed rubrics and reasoning-first grading as LLM-grading tips.
Calibrate. Hand-grade a sample of runs and check the judge agrees with you. Anthropic says judges "should be closely calibrated with human experts."

An example judge call:

jq -r '.structured_output.summary' "$run/result.json" | claude --bare -p \
  "Rubric: PASS only if the summary lists every changed file by path. FAIL if any changed file is missing. UNKNOWN if you cannot tell. Changed files: $(tr '\n' ' ' < "$run/changed.txt"). Explain briefly, then give the verdict." \
  --model claude-opus-5-5 --output-format json \
  --json-schema '{"type":"object","properties":{"reason":{"type":"string"},"verdict":{"enum":["pass","fail","unknown"]}},"required":["reason","verdict"]}' \
  | jq -r '.structured_output.verdict'

Limitation: an LLM judge is also nondeterministic and costs tokens on every graded run. If you're on a tight budget, keep judges for the two or three cases where prose quality actually matters.

How do you set a flaky-run threshold to avoid false alarms?

Run each Claude Code test case several times (3–5) and flag a regression only when the candidate's pass count drops by a margin set in advance. One random failure shouldn't block a good edit.

The need for this comes straight from Anthropic's guide: "Agent behavior varies between runs, which makes evaluation results harder to interpret." The guide's math shows why one run isn't enough. With a 75% per-trial success rate and three trials, "the probability of passing all three is (0.75)³ ≈ 42%" (source). A single pass or fail tells you very little about a case like that.

Two rules make the threshold workable:

Only cases that pass reliably in the baseline belong in the regression suite. Anthropic says high-pass-rate capability evals can "graduate" to become a regression suite. A case that passes 3/5 on the baseline is a capability problem, so fix it separately.
Decide the threshold before you look at results, and write it into compare.sh.

Example thresholds to start from. These are starting points, not Anthropic numbers, so tune them to your budget:

Runs per case	Keep case in suite if baseline is	Candidate result: ok	Candidate result: rerun (flaky)	Candidate result: regression
3	3/3	3/3	2/3 → run 3 more, judge on 6	1/3 or 0/3
5	5/5	5/5	4/5 → run 5 more, judge on 10	3/5 or lower
10	9/10 or 10/10	within 1 of baseline	2 below baseline	3+ below baseline

When a case lands in the "rerun" band, rerun only that case. After the extra runs, call it a regression if the combined candidate rate is clearly below the baseline rate, and read the failed transcripts either way. Our metrics article, AI Agent Evaluation Metrics: Pass Rate Across Repeat Runs, explains when to report pass@k and when to report pass^k.

Pitfalls:

Retrying until green. If you rerun a failing case until it passes, you've measured pass@k, which is the "at least one success" number. An unattended agent needs the "every run succeeds" view.
Ignoring infrastructure failures. A run that died from a rate limit isn't a prompt regression. Record the Claude Code exit code (claude_exit) and exclude API errors from the pass count. For handling them, see Claude Code Rate Limit Reached: Retry Setup for Cron Jobs.
Expecting a temperature knob. The headless docs don't document a sampling-temperature flag for claude -p, so repeat runs are how you deal with variance, not settings.

How do you regression-test after swapping models (e.g. Sonnet to Opus)?

Treat a Claude Code model swap exactly like a prompt change. Rerun the same baseline suite with the new model pinned, then compare. Different models read the same CLAUDE.md differently.

Steps:

Pin both sides by full ID, not alias. The model config page warns that "aliases point to the recommended version for your provider and update over time," and says to use the full name (for example claude-opus-5-5) or ANTHROPIC_DEFAULT_OPUS_MODEL to pin (model config). If your cron job says --model opus, the model can change under you with no edit on your side.
Pin effort too. Default effort differs by model. As of October 2026, the docs list medium as the default for Opus 5.5, Sonnet 5.5 and Haiku 5.5, xhigh for Opus 4.7, and high for the others (model config). Anthropic's postmortem traces another of its quality issues to a default effort change from high to medium. Pass --effort explicitly on both runs.
Run EVAL_MODEL=<old-id> bash evals/run.sh model-old evals/baseline/CLAUDE.md 5, then the same with the new ID and label model-new.
Grade both, compare, and read failing transcripts.
If the new model regresses only on cases tied to one CLAUDE.md rule, rewrite that rule and test again. That rewrite is now a prompt change, so it goes through the same loop.

For example, a subagent that was moved to a cheaper model may stop writing its files_changed list. Your schema check catches that, which a string diff wouldn't. If you're choosing models per role, Claude Code Subagents Model Selection: Role-by-Role Table covers that choice. This suite is how you confirm it.

Pitfalls:

Testing the new model on a different Claude Code version. The CLI updates itself by default. The stable channel serves "a version that is typically about one week old, skipping releases with major regressions" (setup docs), but pinning a version during the comparison is more reliable.
Retiring the old model too early. Keep the old run's results. They're your evidence if the new model's problems show up a week later.

What should go into a minimal test case suite for an agent?

A minimal suite needs 5–15 cases drawn from your real recurring job. Include everyday tasks, edge inputs, one case per critical CLAUDE.md rule, and every past failure.

Anthropic's guide says "20-50 simple tasks drawn from real failures is a great start" (source). That's the target for a mature agent. For one solo-run job, 5–15 cases at 3–5 runs each already means 15–75 runs per side, and that's what keeps a comparison affordable enough to run every time. Grow toward 20–50 as failures give you new cases. Anthropic's test docs lean the same way: "Prioritize volume over quality" (develop tests). More cases with automated grading beat a few hand-graded ones.

An example suite plan for one nightly job:

Slot	How many	Example
Everyday task	3–5	Fix one failing unit test; update one dependency pin
One per critical CLAUDE.md rule	2–4	A task where the easy fix is in `migrations/` (rule says never touch it)
Past real failures	All you have	The run that rewrote a lockfile; the run that reported "done" with failing tests
Edge inputs	1–3	Empty issue text; an overly long log; a task that should end in `blocked`
Stop case	1	A request it should refuse or escalate, graded on `status == "blocked"`

The rule-specific cases do a second job. If CLAUDE.md quietly stops loading, for example because a future release makes --bare the default for -p, those cases fail in both runs. That tells you the harness broke, not the prompt.

Write each case so the result is clear-cut. Anthropic's test is that "a good task is one where two domain experts would independently reach the same pass/fail verdict." If you can't write the assert line, the case isn't ready yet.

How do you automate regression tests to run on every prompt change?

Wrap the run, grade and compare steps in one script, and trigger it from CI. Run it whenever CLAUDE.md, .claude/, model settings or the eval files change.

Add a single entry point, evals/check.sh:

#!/usr/bin/env bash
set -euo pipefail
RUNS=${RUNS:-3}
bash evals/run.sh baseline evals/baseline/CLAUDE.md "$RUNS"
bash evals/run.sh candidate CLAUDE.md "$RUNS"
bash evals/grade.sh baseline && bash evals/grade.sh candidate
bash evals/compare.sh baseline candidate

Then add a GitHub Actions workflow, .github/workflows/prompt-regression.yml:

name: Prompt regression
on:
  pull_request:
    paths: ["CLAUDE.md", ".claude/**", "evals/**"]
jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v6
      - name: Install pinned Claude Code
        run: curl -fsSL https://claude.ai/install.sh | bash -s 2.1.89
      - name: Run regression suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          DISABLE_AUTOUPDATER: "1"
          EVAL_MODEL: claude-opus-5-5
          RUNS: "3"
        run: bash evals/check.sh
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-results, path: results/ }

Notes on this setup:

Version 2.1.89 is the example version from the setup docs. Replace it with the version you actually run in production.
Store the key as a GitHub secret. The GitHub Actions docs say "Never commit API keys or OAuth tokens directly to your repository," and accept CLAUDE_CODE_OAUTH_TOKEN (generated with claude setup-token) if you'd rather bill a Pro or Max subscription.
Each run uses GitHub Actions minutes plus API tokens, per the same page. Set timeout-minutes and keep RUNS low on pull requests.
The claude-code-action (anthropics/claude-code-action@v1) is built for having Claude do work in a repo. For an eval harness, calling the CLI directly as above is simpler.

To catch drift you didn't cause, like a CLI update or a model alias moving, also run the suite on a schedule against the current production CLAUDE.md. Note the docs' warning that GitHub "disables the schedule after 60 days without repository activity" in public repos. If your agent runs on a VPS, the same check.sh fits into the crontab described in Claude Code Cron Job Setup: Run Automations Overnight.

What are common pitfalls when catching prompt regressions?

The usual Claude Code regression-testing mistakes are diffing raw text, using too few runs, and leaving the grader itself unversioned. Each one makes the comparison measure the wrong thing.

Pitfall	What goes wrong	Fix
Raw text diff	Every run "differs," so real changes get lost in the noise	Grade `structured_output` fields and workspace outcomes
One run per side	A 75%-reliable case looks broken or fine by chance	3–5 runs plus a rerun band (see threshold table)
Unversioned grader	You edit the judge rubric and the "regression" is really a grader change	Keep grader scripts and judge prompts in git; change them in a separate PR and re-grade the old results
`--bare` on the agent run	CLAUDE.md never loads, so both sides test nothing	Bare mode for the judge only; rule-specific cases as a load check
Alias model names	`opus` moves to a new version between baseline and candidate	Full model IDs plus explicit `--effort`
Host config leaking in	Your local `~/.claude` hooks or auto memory shape results on your laptop but not in CI	Run comparisons on a clean CI runner, or one machine for both sides
Saturated suite	Everything passes 5/5 forever and catches nothing	Add every new production failure as a case; Anthropic notes "eval saturation occurs when an agent passes all of the solvable tasks"

For example, if you tighten the judge rubric and grade only the candidate with it, scores drop even though the agent didn't change. Re-grade both sides with the same grader version every time.

Once a change ships, regression tests stop and production monitoring takes over. AI Agent Monitoring in Claude Code: Catch Silent Failures covers that half.

Next steps

Do these today:

Copy your current CLAUDE.md to evals/baseline/CLAUDE.md, write 5 case files from last month's real jobs (include at least one failure), and run evals/run.sh 3 times per case.
Change every --model opus/sonnet in your cron jobs and scripts to a full model ID, and add an explicit --effort.
Make your next CLAUDE.md edit go through evals/check.sh before it reaches the file your agent actually reads. The structure in CLAUDE.md Best Practices: A Template for Unattended Agents makes rule-specific cases easier to write.

If you'd rather not build the harness yourself:

Promptfoo is an open-source eval tool with a Claude Agent SDK provider (anthropic:claude-agent-sdk, alias anthropic:claude-code). It supports options like working_dir, max_turns and copy_working_dir for isolated runs. It suits developers who want config files and a results UI instead of shell scripts. It's not a good fit if you can't run Node/npm in your environment. One limitation: you still have to write the cases and assertions, which is where most of the effort goes.
For billing, CI runs that use an API key are charged per token (pricing). A subscription token instead bills your plan: Claude Pro is listed at $20 if billed monthly, and Max plans start at $100 per month (as of October 2026; check the official pricing page). A subscription suits a solo builder running small suites. It doesn't suit large nightly suites shared across a team, where an API key with a budget cap is easier to control.

Related guides on this site:

FAQ

How often should you re-run regression tests on a Claude Code agent?

Run them on every change to CLAUDE.md, .claude/ settings, the model ID or the effort level, plus on a schedule (weekly is a reasonable example) to catch drift from CLI or model updates. The scheduled run uses the production CLAUDE.md as both baseline and candidate inputs over time, so compare it against the last saved results. If nothing changed and nothing failed, the run only cost tokens.

Can you use Claude itself as the grader for regression testing?

Yes, for parts code can't check, such as whether a summary is accurate. Run it as a separate claude --bare -p call with a written rubric, one dimension per call, and an unknown option, as Anthropic's eval guide recommends. Calibrate it against your own hand grading, and keep the rubric in git so grader changes don't look like agent regressions.

What's a good sample size to avoid flaky test results?

Three runs per case is the cheapest useful setting, and five is a solid default for agents that run unattended every day. Add a rerun band so borderline results (2/3 or 4/5) get more runs before you decide. Anthropic's pass^k example shows that a 75%-reliable task passes three straight runs only about 42% of the time, so one or two runs per side can't separate a flaky case from a broken one.

Does editing CLAUDE.md require a full regression suite every time?

For an agent that runs unattended, yes: run the full suite, because a line about one task can change behavior on another, as Anthropic's own postmortem shows. Keep the suite small (5–15 cases) so this stays cheap. Pure typo fixes are the only reasonable exception, and even then the scheduled run will catch anything you missed.

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code MCP Servers: Setup, Config & Cron-Safe Template

Aoba Works — Thu, 29 Oct 2026 16:20:46 +0000

Originally published at Unattended.

Claude Code MCP servers are external programs or hosted endpoints that give Claude Code extra tools, such as a browser, GitHub, Sentry or a database. You add one with claude mcp add, check it with claude mcp list and remove it with claude mcp remove <name>. In the words of the official MCP docs, "MCP servers give Claude Code access to your tools, databases, and APIs." Most guides stop at the interactive setup. This one also covers what changes when nobody is at the keyboard: which servers a cron run loads, how to scope their permissions, which timeouts apply, and what happens when a server goes down partway through a job.

Which option is right for you?

Choose the scope and transport that fit how you work, then open the matching page. Each row gives a one-line reason.

Your situation	What to pick first	First page to open
Beginner, never added a server	One hosted HTTP server at the default `local` scope. It needs no install or sign-in	Connect to MCP servers (quickstart)
Daily interactive work across many repos	`--scope user` for the 1–3 servers you use everywhere, because they load in every project	MCP reference: scopes
Small team sharing one repo	`--scope project` (`.mcp.json` committed), so teammates get the same config	Quickstart: Edit .mcp.json directly
Cron / headless jobs	A separate JSON file passed with `--mcp-config` plus `--strict-mcp-config`, so the job loads only what you list	Run Claude Code programmatically
Tight token budget	Fewer servers, because every connected server's tool names load into each session	Quickstart note on context

For a cron job, keep the automation's server list out of your personal config. The rest of this guide explains why.

What is a Claude Code MCP server?

A Claude Code MCP server is a process or hosted URL that exposes extra tools to Claude Code over the Model Context Protocol. Claude Code calls those tools like its built-in ones. The quickstart describes it as letting Claude Code use tools "beyond its built-in set, such as searching an issue tracker, querying a database, or controlling a web browser."

Servers come in two shapes:

Local (stdio): Claude Code starts the program as a subprocess on your machine. The Playwright server is one example. Use this shape for anything that needs local resources such as a browser, the filesystem or a database socket (quickstart).
Remote (HTTP): the vendor hosts the server and you register its URL. Anthropic's remote MCP announcement pitches this as the lower-maintenance option because you have no local server to manage.

Each tool appears to Claude under a prefixed name, mcp__<server>__<tool>. That name matters later, because permission rules match against it (permissions docs).

So, can Claude Code use MCP servers? Yes, in every surface. The CLI, the desktop app, VS Code and cloud sessions can all connect. Connectors you add at claude.ai also load in the CLI when you sign in with that account (quickstart: Connect from other surfaces).

How do you install Claude Code MCP servers?

For most servers, installing is the same step as registering. You run claude mcp add, and either Claude Code fetches the package through npx the first time it starts the server, or you point it at a hosted URL.

Prerequisites, per the quickstart:

Install and authenticate Claude Code.
Open a terminal in a project directory. An empty directory works.
For npx-based local servers, install Node.js. The Playwright server, for example, requires Node.js 18 or later.

Then run the add command in your shell, not inside a claude session:

# Remote example from the docs: no auth needed
claude mcp add --transport http claude-code-docs https://code.claude.com/docs/mcp

# Local example from the docs: Playwright browser server
claude mcp add playwright -- npx -y @playwright/mcp@latest

Common install mistakes and fixes:

"Added" but ✘ Failed to connect. The Added message only means the entry was saved. On the first check, npx may still be downloading the package, so wait and run claude mcp list again (quickstart).
Startup times out. The default startup timeout is 30 seconds. Raise it with MCP_TIMEOUT=60000 claude. The value is in milliseconds (quickstart troubleshooting).
Need to diagnose a local server. Run the server command yourself, for example npx -y @playwright/mcp@latest. If it starts and waits for input, the server works and the problem is in your Claude Code entry.

If you already set up servers in the Claude Desktop chat app, claude mcp add-from-claude-desktop copies them into the CLI on macOS or WSL (quickstart).

How do you configure Claude Code MCP servers (config file)?

Claude Code stores MCP server configuration in ~/.claude.json or a project's .mcp.json, depending on scope. ~/.claude.json holds the local and user scopes, and .mcp.json in the project root holds the project scope. The file depends on the --scope you pick (quickstart).

Scope	File	Who gets it
`local` (default)	`~/.claude.json`, under this project's entry	Only you, only this project
`project`	`.mcp.json` in project root	Everyone who clones the repo
`user`	`~/.claude.json`, top-level `mcpServers`	Only you, all projects

On Windows, ~/.claude.json resolves to %USERPROFILE%\.claude.json. If CLAUDE_CONFIG_DIR is set, Claude Code reads the file from that directory instead (quickstart). When one name exists in several scopes, precedence runs local, then project, then user, then plugins, then claude.ai connectors (MCP reference).

Here is a hand-written .mcp.json example. The first two entries come from the docs. The ${...} variables and the timeout field are documented options:

{
  "mcpServers": {
    "claude-code-docs": {
      "type": "http",
      "url": "https://code.claude.com/docs/mcp"
    },
    "playwright": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@playwright/mcp@latest"]
    },
    "internal-api": {
      "type": "http",
      "url": "${API_BASE_URL:-https://api.example.com}/mcp",
      "headers": { "Authorization": "Bearer ${API_KEY}" },
      "timeout": 600000
    }
  }
}

The details that matter, all from the MCP reference:

${VAR} and ${VAR:-default} expand in command, args and env for stdio servers, and in url and headers for HTTP servers. Keep tokens in the environment, not in a committed file.
Some credentials always read as empty in remote URLs and headers, on purpose: ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, NPM_TOKEN and a few others. If your header comes out blank, this is likely why.
An entry with a url but no type is an error. Add "type": "http".
A per-server timeout in milliseconds overrides MCP_TOOL_TIMEOUT for that server only.

Pitfall: editing the wrong file. Claude Code does not read ~/.claude/.mcp.json, ~/.claude/mcp.json, ~/.claude/config/mcp.json or %APPDATA%\Claude\mcp.json (quickstart troubleshooting). Edits to .mcp.json take effect only when a new session starts (quickstart troubleshooting). A malformed entry is skipped while the others still load, so run claude mcp list and look for the parse warning (quickstart troubleshooting).

How do you add and remove a Claude Code MCP server?

You add a Claude Code MCP server with claude mcp add (or add-json) and remove it with claude mcp remove <name>. Both commands print the file they modified (quickstart).

Add, by server type (MCP reference):

Remote HTTP: claude mcp add --transport http <name> <url>
Remote with a static token: append --header "Authorization: Bearer <token>". The docs' GitHub example is claude mcp add --transport http github https://api.githubcopilot.com/mcp/ --header "Authorization: Bearer YOUR_GITHUB_PAT".
Local stdio: claude mcp add [options] <name> -- <command> [args...], for example --env AIRTABLE_API_KEY=YOUR_KEY.
Raw JSON: claude mcp add-json <name> '<json>', which is also the only way to add a WebSocket (ws) server.
OAuth servers such as Sentry (https://mcp.sentry.dev/mcp): add the URL, start claude, run /mcp, select the server and choose Authenticate. You can also run claude mcp login <name> from the shell.

The -- separator is the most common mistake. Your own options (--transport, --env, --scope) go before it, and the docs say "Everything after -- is passed to the server untouched" (MCP reference). If claude mcp get <name> shows a different command from the one you typed, remove the server and add it again with -- in place.

Remove or pause:

claude mcp remove claude-code-docs                 # default scope
claude mcp remove claude-code-docs --scope local   # when it exists in several scopes

If the name exists in more than one scope, remove reports exists in multiple scopes, and you choose one with --scope (quickstart). To pause a server without deleting it, toggle it off in the /mcp panel. Claude Code records that choice per project in a disabledMcpServers list in ~/.claude.json (MCP reference). You can't change a server's scope in place. Remove it and add it again at the new scope (quickstart).

How do you connect Claude Code to local vs remote MCP servers?

Claude Code connects to local servers over stdio and to remote servers over HTTP. SSE still works but is deprecated, and WebSocket can only be configured through JSON (MCP reference). For unattended runs, the biggest difference between them is how each transport fails and recovers.

	Local (stdio)	Remote (HTTP)
Runs where	Subprocess on your machine	Vendor's infrastructure
Auth	Env vars (`--env`)	Header token, OAuth, or `headersHelper` script
Default idle timeout per tool call	1,800,000 ms (30 min)	300,000 ms (5 min)
If it drops mid-session	In `claude -p`, retried on the next tool call	Up to 5 retries with exponential backoff starting at 1 s
If first connect fails	Marked failed; use `/mcp` Reconnect	Transient errors retried up to 3 times
Main risk unattended	Missing runtime (Node), slow first `npx` download	Network, expired token, vendor outage

The idle timeout figures come from the environment variables reference (CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT, v2.1.187 or later). The retry behavior comes from the MCP reference. Authentication errors are different: Claude Code stops retrying and waits for you to sign in again. A cron job can't do that, which is why OAuth servers fit unattended runs poorly unless their credentials stay valid.

Quick connectivity check. For an HTTP server, run curl -I <url>. A 404 or 405 still means the server is up, because many MCP endpoints answer only POST. A 401 or 403 means you need to authenticate. No response at all points to the URL or the network (quickstart troubleshooting).

How do you use a Claude Code MCP server in a session?

You use a Claude Code MCP server by asking for the task in plain language. Claude picks the relevant tool on its own, and you approve the first call when prompted (quickstart).

Run claude mcp list and confirm the server shows ✔ Connected.
Start claude in the same project. Servers at local scope are tied to the directory where you added them.
Ask for something that needs the server, for example: Use playwright to open https://example.com and tell me the page title.
Approve the permission prompt. Each tool call is labeled with the server name, such as browser_navigate from playwright.
Run /mcp at any time to see status and tool lists, reconnect a server or authenticate.

Naming the server in your prompt isn't required. The docs do it in the walkthrough so the test definitely goes through the new server rather than a built-in tool such as web fetch.

Watch the output size. Claude Code shows a warning when an MCP tool returns more than 10,000 tokens and caps output at 25,000 tokens by default. MAX_MCP_OUTPUT_TOKENS changes the cap (MCP reference). Large database or log queries hit that cap quickly. For ways to trim them, see how to see token usage and reduce it.

Status meanings you'll see (quickstart):

! Needs authentication: the server is reachable but needs a browser sign-in or a token.
! Connected · tools fetch failed: the server connected but couldn't list its tools. Run claude mcp get <name> for details.
⏸ Pending approval: a project-scoped server you haven't approved yet.
The server connects but shows no tools: usually a missing required env var, such as an API key.

Claude Code MCP servers list: which ones are worth running?

For recurring solo-builder work, start with two or three servers that each cover a job you would otherwise script by hand. Each server you connect adds context cost and permission surface (quickstart).

Server	Transport	Good for	Not for / limitation
Claude Code docs (`https://code.claude.com/docs/mcp`)	HTTP, no auth	Your first test; looking up flags	Only searches docs, no real work
GitHub (`https://api.githubcopilot.com/mcp/`)	HTTP + PAT	Triage issues and PRs on a schedule	The token's scope is the blast radius; use a fine-grained PAT
Playwright (`@playwright/mcp`)	stdio, Node 18+	Smoke-checking a deployed page	Needs a browser on the host, heavy on a small VPS
Sentry (`https://mcp.sentry.dev/mcp`)	HTTP + OAuth	Morning error summaries	OAuth expiry can block cron runs
Filesystem / Git / Fetch / Time (reference servers)	stdio	Learning, narrow local tasks	Reference code only (see below)

Sources: the quickstart for docs, Playwright and Sentry, the MCP reference for GitHub, and the modelcontextprotocol/servers repo for the reference set.

Two things many "best MCP servers" lists get wrong as of October 2026. First, the official repo now maintains seven reference servers: Everything, Fetch, Filesystem, Git, Memory, Sequential Thinking and Time. Second, it archived thirteen others, including GitHub, PostgreSQL, Slack, Puppeteer and SQLite, in servers-archived (servers repo). If a tutorial tells you to install the old GitHub or Postgres package from that repo, look for the vendor's own server instead.

Where do you find and vet Claude Code MCP servers on GitHub?

Find Claude Code MCP servers in the official MCP Registry, Anthropic's reviewed Anthropic Directory, or vendor GitHub repos. Vet each one before an automation depends on it.

The official repo says plainly that its servers are educational examples, "not as production-ready solutions" (modelcontextprotocol/servers). Anthropic's MCP page adds: "Verify you trust each server before connecting it." It also warns that servers which fetch external content carry prompt injection risk (MCP reference).

A short vetting checklist before a server goes into a scheduled job:

Who publishes it? Prefer the vendor's own server (GitHub, Sentry, Microsoft's Playwright) over a third-party wrapper.
What does the token allow? A server can only do what its credential permits. Create a dedicated token with read-only or single-repo scope where the vendor offers it.
Is it pinned? @latest in npx args can change under a cron job overnight. Pin a version in automation configs, for example @playwright/mcp@<version>, and update it on purpose.
Does it read untrusted content? Web fetchers, issue readers and inbox readers can carry injected instructions. Pair them with narrow allow rules (next section).
Is it still maintained? A recent commit history and open-issue activity matter more than star count.

Also be careful with similar names: projects like steipete/claude-code-mcp run Claude Code itself as an MCP server for other agents (GitHub). That's the opposite direction from this guide.

How do you run Claude Code MCP servers headlessly and safely?

Give each headless job its own server file, allowlist named MCP tools, set short timeouts, and check server status in the output. A claude -p run has nobody to answer prompts.

Why your interactive setup is risky in cron

Without --bare, claude -p loads the MCP servers in the working directory's .mcp.json, and the headless docs note: "A -p session shows no workspace trust dialog and no per-server approval prompt." The approval that protects you interactively is skipped. Two flags close the gap:

--bare skips auto-discovery of MCP servers, hooks and CLAUDE.md, and connects only servers you pass with --mcp-config. The limitation: bare mode never reads OAuth or keychain credentials, so it needs ANTHROPIC_API_KEY and won't use your Pro or Max login (headless docs).
--strict-mcp-config with --mcp-config loads only the servers in that file and ignores other MCP configuration (CLI reference). Use this one if you run on a subscription. One open report says it does not override a project's disabledMcpServers list (issue #14490), so don't disable your automation's servers in /mcp.

Permission scoping

From v2.1.302, a -p run starts in Manual mode when nothing else sets a mode. Any call that needs approval is denied unless an allow rule, a PermissionRequest hook or a permission prompt tool approves it (headless docs). Use allow rules to list exactly which MCP tools the job may call:

mcp__github allows every tool from the github server.
mcp__github__get_issue allows a single tool.

The rule format is described in the permissions docs. Prefer naming single tools for jobs that write anything. Also add --permission-prompts none (v2.1.259 or later). With it, Claude is told nobody can approve, so it stops retrying denied calls, and any MCP elicitation request that no hook answers is cancelled (headless docs). For a complete rule set, see the headless permissions template.

Timeouts that fail fast

Variable	Default	What it bounds
`MCP_TIMEOUT`	30 s	Server startup; with `--mcp-config` in `-p`, the first turn waits up to this
`CLAUDE_CODE_MCP_STARTUP_WAIT_MS`	(uses the startup wait)	How long the first non-interactive turn waits for servers that are still connecting (v2.1.274+)
`CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT`	5 min network / 30 min stdio	A tool call with no response or progress
Per-server `timeout` in JSON	none	Overall tool time for that server
`CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS`	120,000 ms	When a long MCP call moves to the background

Sources: headless docs, env vars, MCP reference.

The last row matters most for cron. After 2 minutes, a long MCP call moves to the background. The headless docs say that when a -p run ends, for a call that automatic backgrounding moved, "the call's result is lost". Setting CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS=0 turns automatic backgrounding off (env vars). Also set CLAUDE_CODE_MCP_ALLOWLIST_ENV=1. Stdio servers then get only a safe baseline environment plus their configured env, instead of inheriting every secret in your shell (env vars).

What happens when a server is unreachable

The run does not crash on its own. An entry that fails validation is skipped and the run "continues and exits cleanly" (headless docs). A server that never connects means its tools are simply missing, so Claude may finish the task without them and still exit 0. Your script has to check for that. The system/init event in stream-json output lists mcp_servers (each with name and status) and mcp_server_errors. A redirected stderr shows no warning, so read these fields instead. Exit code 143 means the process got SIGTERM (headless docs).

The following example wrapper adapts documented flags. Check the status strings your version emits before relying on the filter.

#!/usr/bin/env bash
set -euo pipefail
cd /srv/myproject
export MCP_TIMEOUT=15000
export CLAUDE_CODE_MCP_TOOL_IDLE_TIMEOUT=120000
export CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS=0
export CLAUDE_CODE_MCP_ALLOWLIST_ENV=1

timeout 20m claude -p "Summarize issues labeled bug opened in the last 24h" \
  --mcp-config ./automation.mcp.json --strict-mcp-config \
  --allowedTools "mcp__github__list_issues,mcp__github__get_issue" \
  --permission-prompts none \
  --output-format stream-json --verbose > run.jsonl

# Fail the job if any server is not connected (example filter)
if jq -e 'select(.type=="system" and .subtype=="init")
  | (.mcp_server_errors // []) + [.mcp_servers[] | select(.status!="connected")]
  | length > 0' run.jsonl >/dev/null; then
  echo "MCP server missing; result not trusted" >&2; exit 2
fi

Scheduling, retries and alerting belong in your cron layer. See the cron job setup guide and catching silent failures. For every -p flag and the auth trade-offs, see the headless mode guide.

Next steps

Do this today:

Run claude mcp add --transport http claude-code-docs https://code.claude.com/docs/mcp, then claude mcp list. That takes about two minutes and confirms your setup works.
Create automation.mcp.json containing only the servers your cron job needs, with pinned versions and ${VAR} tokens.
Run the wrapper above once by hand and inspect the system/init line in run.jsonl before you schedule it.

If you'd rather not run servers yourself: use vendor-hosted remote servers from the Anthropic Directory, so the vendor handles uptime and updates. You still configure scopes and tokens. Claude Code itself requires a paid plan: Pro is $20/month billed monthly ($17/month on annual billing), and Max starts at $100/month (as of October 2026; check the official pricing page). Pro suits one person running a few scheduled jobs. It's a poor fit if you need --bare everywhere, because bare mode needs an API key billed separately through the Claude Console (headless docs).

Related guides on this site:

FAQ

What happens if an MCP server becomes unreachable mid-run in Claude Code?

For HTTP servers, Claude Code retries up to 5 times with exponential backoff starting at 1 second. For stdio servers in claude -p, it retries on the next tool call (MCP reference). If the server stays down, that server's tools fail but the run can still exit 0. Check mcp_servers in the system/init event and treat a missing server as a failed job.

Can Claude Code use multiple MCP servers at once?

Yes. List as many as you need under mcpServers in one file. Each server's tool names and instructions load into the context window, though, so remove ones you don't use (quickstart). Tool search reduces that cost when you have many tools (MCP reference).

Where is the Claude Code MCP servers config file on each OS?

Local and user scopes live in ~/.claude.json on macOS and Linux, and in %USERPROFILE%\.claude.json on Windows. Project scope is .mcp.json in the repo root. Run claude mcp get <name> to see which scope holds a server (quickstart).

Is it safe to run Claude Code MCP servers in a cron job without review?

It's reasonably safe only when the job loads a fixed server list (--strict-mcp-config or --bare), allows named MCP tools only, and uses narrowly scoped tokens. Without those, a -p run connects whatever .mcp.json the directory contains, with no approval prompt (headless docs). Servers that read web pages or issues can also carry prompt injection.

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

Claude Code Sandbox Mode: Exact Config and What It Blocks

Aoba Works — Wed, 28 Oct 2026 18:18:30 +0000

Originally published at Unattended.

Claude Code sandbox mode is a built-in, OS-enforced boundary around the Bash commands Claude runs. You turn it on with /sandbox or "sandbox": {"enabled": true} in settings.json. Anthropic's docs describe it as "a boundary that the operating system enforces around the shell commands Claude runs on your machine". It limits where those commands can write and which hosts they can reach. It does not cover Claude's own file tools, hooks or MCP servers. That gap matters most if you plan to leave an agent running unattended.

Which option is right for you?

On macOS, Linux or WSL2, turn on the built-in sandbox in auto-allow mode. Use a container or VM only if you need stronger isolation than that.

Your situation	What to pick first	First page to open
Beginner on macOS	`/sandbox` → auto-allow mode. Seatbelt is built in, so there's nothing to install (docs)	Sandboxing docs
Daily work on Linux or WSL2	Install `bubblewrap` and `socat`, then enable the sandbox with `failIfUnavailable: true`	Sandboxing docs, Linux setup
Native Windows (no WSL)	Move Claude Code into a WSL2 distribution. Native Windows runs commands unsandboxed	Sandboxing docs
Cron jobs or headless runs on a VPS	Sandbox plus `allowUnsandboxedCommands: false` plus deny rules	/claude-code-permissions-best-practices-headless-template/
Zero budget	The sandbox costs nothing extra. It ships with Claude Code	Permission modes docs
You want to skip every prompt (YOLO)	Use `bypassPermissions` only inside a container or VM	Permission modes docs

Why these picks: on macOS the sandbox needs no setup. On Linux the most common failure is missing dependencies, and failIfUnavailable makes that failure loud. Windows needs WSL2 before any sandbox setting does anything.

What is Claude Code sandbox mode?

Claude Code sandbox mode wraps every Bash command Claude runs in an OS-level jail that controls filesystem writes and outbound network access. The point is to let Claude run commands without asking you each time, while capping the damage a bad command can do.

It's built on standard OS tools. On macOS it uses the built-in Seatbelt framework. On Linux and WSL2 it needs the bubblewrap and socat packages (Claude Code docs). Anthropic published the sandbox runtime as open source in the @anthropic-ai/sandbox-runtime package (GitHub).

When Anthropic introduced it on October 20, 2025, the engineering team wrote: "In our internal usage, we've found that sandboxing safely reduces permission prompts by 84%." That number comes from Anthropic's own internal use. It is not a benchmark.

Here is what the sandbox allows by default (docs):

Writes: the current working directory, the per-user temp directory, and any directory added with --add-dir, /add-dir or permissions.additionalDirectories.
Reads: most of the machine, credential files like ~/.ssh and ~/.aws/credentials included.
Network: no direct route out. Traffic goes through a local proxy that checks each host against allowedDomains, and that list starts empty.

Pitfall: many people assume a sandbox means "can't see my secrets." By default the sandbox restricts writes, not reads. If you want credential files hidden, you must deny them explicitly (see the config section below).

Does Claude Code have sandbox mode?

Yes. Claude Code has a built-in sandbox that you enable with the /sandbox command or the sandbox.enabled key in settings.json. It works on macOS, Linux and WSL2, but not on native Windows.

There are three official ways to turn it on (docs):

In a session: type /sandbox to open the sandbox panel.
Persistently: add "sandbox": {"enabled": true} to ~/.claude/settings.json or the project's .claude/settings.json.
For one run: pass --settings '{"sandbox": {"enabled": true}}' when you start claude.

Some third-party guides mention a claude --sandbox startup flag. The official sandboxing page lists no such flag, so use --settings for one-off runs and scripts.

Once it's on, you choose one of two modes. In auto-allow mode, "Claude Code approves a command automatically, with no prompt, when the command runs inside the sandbox". In regular permissions mode, every Bash command still goes through the normal approval flow, but it runs inside the boundary. The filesystem and network limits are the same in both modes. The only difference is whether you get prompted.

Claude Code auto mode vs sandbox mode: what's the difference?

Claude Code auto mode decides whether an action needs your approval. Sandbox mode limits what a command can reach once it runs. They solve different problems, and you can use them together.

The permission modes page puts it this way: "the Bash sandbox and outer isolation boundaries decide what an action can reach once it runs." Permission modes only decide whether Claude asks first.

A common mix-up is treating auto mode and "YOLO mode" as the same thing. They aren't. Auto mode sends actions to a separate classifier model, Claude Sonnet 5 by default, which blocks things like curl | bash, force pushes and terraform destroy. YOLO is bypassPermissions (alias --dangerously-skip-permissions), which approves everything (permission modes docs).

	Auto mode	bypassPermissions ("YOLO")	Sandbox (auto-allow)
What it controls	Whether to prompt	Whether to prompt	What Bash can touch
How it decides	Classifier model reviews each action	Approves everything	OS rules (Seatbelt / bubblewrap)
Covers file edit tools	Yes	Yes	No, Bash only
Hard guarantee?	No: "does not guarantee safety"	No	Yes, within its scope
Official "best for"	Long tasks, fewer prompts	Isolated containers and VMs only	Bash-heavy work on macOS/Linux/WSL2
Extra requirements	Supported model (e.g. Opus 4.6+, Sonnet 4.6+)	None	`bubblewrap` and `socat` on Linux

The docs say plainly: "Auto mode reduces permission prompts but does not guarantee safety." Auto mode also pauses and goes back to prompting after 3 blocks in a row or 20 blocks in total in a session (same page). For unattended runs, that means a "stuck" job can actually be a job waiting at a prompt. See why Claude Code stops mid-automation.

How they combine. With auto mode and the sandbox both on, sandboxed Bash commands run inside the boundary. If a command fails because of the sandbox and Claude retries it with dangerouslyDisableSandbox, the classifier reviews the retry. In bypassPermissions mode, that unsandboxed retry runs with no prompt at all (sandboxing docs). So if you combine YOLO with the sandbox, also set allowUnsandboxedCommands: false. Otherwise the sandbox has an exit you never see.

Which to choose: for daily interactive work, use the sandbox in auto-allow mode plus auto mode. For unattended jobs, use the sandbox in strict mode plus explicit permission rules. Don't rely on a classifier in a job nobody is watching.

How do you run Claude Code in sandbox mode?

Type /sandbox in a session and choose auto-allow mode. For a persistent setup, enable sandbox in settings.json, then check that a forbidden command fails.

Step-by-step

Linux/WSL2 only: install the dependencies. Run sudo apt-get install bubblewrap socat on Ubuntu/Debian, or sudo dnf install bubblewrap socat on Fedora (docs).
Ubuntu 24.04 and later only: run sysctl kernel.apparmor_restrict_unprivileged_userns. If it returns 1, AppArmor is blocking the user namespaces bubblewrap needs. Add the bwrap AppArmor profile shown on the docs page and run sudo systemctl reload apparmor.
Start Claude Code and type /sandbox. The panel has a Mode tab (auto-allow or regular permissions), an Overrides tab (strict sandbox mode) and a Config tab showing the resolved settings. On Linux, a Dependencies tab appears if the optional seccomp filter is missing.
Choose auto-allow on the Mode tab. Claude Code saves this choice to .claude/settings.local.json.
For unattended use, open Overrides and turn on strict sandbox mode. This sets allowUnsandboxedCommands: false.
Verify. Ask Claude to run the two test commands from the docs:

Test command	Expected result inside the sandbox
`touch ~/sandbox-probe`	`Operation not permitted` (macOS) or `Read-only file system` (Linux/WSL2)
`curl --noproxy '*' https://example.com`	`Could not resolve host`

If either command succeeds, the sandbox isn't active. Check /sandbox → Config.

Example settings.json for an unattended project

This example combines keys from the official sandboxing page. The domain names and paths are sample values to replace with your own:

{
  "sandbox": {
    "enabled": true,
    "autoAllowBashIfSandboxed": true,
    "allowUnsandboxedCommands": false,
    "failIfUnavailable": true,
    "filesystem": {
      "allowWrite": ["/tmp/build"],
      "denyRead": ["~/private"]
    },
    "network": {
      "allowedDomains": ["github.com", "*.npmjs.org"],
      "strictAllowlist": true
    },
    "credentials": {
      "files": [
        { "path": "~/.aws/credentials", "mode": "deny" },
        { "path": "~/.ssh", "mode": "deny" }
      ],
      "envVars": [
        { "name": "GITHUB_TOKEN", "mode": "deny" }
      ]
    }
  }
}

What each piece does:

failIfUnavailable: true stops Claude Code at startup if the sandbox can't run. Without it, a missing bubblewrap on a fresh VPS can leave you running with no sandbox.
allowUnsandboxedCommands: false removes the retry escape. The docs say Claude Code then "ignores the dangerouslyDisableSandbox parameter."
strictAllowlist: true blocks any host outside the allowlist instead of prompting. Per the docs, this only takes effect in user, managed or CLI --settings settings, so a project file alone won't do it.
allowedDomains entries without a port allow every port on that host. Wildcards like *.example.com work.

Settings precedence, from highest to lowest, is: CLI --settings, managed, user (~/.claude/settings.json), project (.claude/settings.json), local project. Filesystem arrays from different scopes are merged, not replaced (docs). For cron and claude -p runs, keep the sandbox block in user settings or pass it with --settings. The headless guide covers the rest of the flags: /claude-code-headless-mode-flags-auth-and-cost-2026/.

Common mistakes and fixes

docker fails inside the sandbox. The docs call docker incompatible. Fix it with a narrow exclusion such as "excludedCommands": ["docker compose *"].
git push over SSH fails on macOS even when the host is allowed. Switch the remote to HTTPS, or exclude git fetch *, git pull * and git push *.
A database client can't connect to an allowed host. Tools that ignore proxy environment variables can't get out. Exclude that specific command.
A local dev server can't be reached. On macOS, set network.allowLocalBinding: true. On Linux/WSL2 that setting has no effect, so use excludedCommands instead.

Does Claude Code sandbox mode work on Windows?

Not natively. On native Windows, Claude Code runs commands unsandboxed, so you need to run Claude Code inside WSL2 to get the sandbox.

The docs say it directly: "Claude Code runs commands unsandboxed. To use the sandbox on a Windows machine, run Claude Code inside a WSL2 distribution". Inside WSL2 the setup is the same as on Linux: install bubblewrap and socat, and apply the AppArmor fix if your distro is Ubuntu 24.04 or newer.

There's one WSL2-specific catch. Setting network.allowAllUnixSockets: true opens every Unix socket to sandboxed commands. On WSL2 that includes the socket WSL uses to launch Windows binaries, so a sandboxed command could start a Windows program outside the boundary (docs). Leave that setting off unless you know which socket you need.

Pitfall: turning on sandbox.enabled in native Windows PowerShell gives no error and no protection. Add failIfUnavailable: true so a misconfigured machine fails loudly instead of quietly running everything unsandboxed.

What does Claude Code sandbox mode actually block, and what slips through?

Claude Code sandbox mode blocks Bash writes outside allowed paths and network traffic to hosts not on the allowlist. Built-in tools, hooks, MCP servers, excluded commands and default reads all fall outside it.

This is the part most guides skip. The docs list everything the sandbox does not wrap: built-in tools like Read, Edit, Write, WebFetch and WebSearch "follow permission rules instead," and "a denyRead entry doesn't stop the Read tool, and allowedDomains doesn't limit WebFetch" (sandboxing docs). Command hooks, local MCP servers, plugin monitors, LSP servers, your status line command and apiKeyHelper all "run with your full access."

Scenario	Blocked by sandbox?	What actually controls it
`rm -rf ~/Documents` via Bash	Yes (write outside allowed paths)	OS sandbox
`curl` to a host not in `allowedDomains`	Yes (or a prompt, depending on mode)	Sandbox proxy
`cat ~/.ssh/id_ed25519` via Bash	No by default (reads are open)	Add `credentials.files` deny or `denyRead`
Claude's Read tool opening `~/.aws/credentials`	No	Permission deny rules
Edit/Write tool changing files	No	Permission rules / permission mode
WebFetch to any domain	No	`WebFetch(domain:...)` permission rules
A command hook script	No, full access	Your own script review
Local MCP server	No, full access	MCP config / permission rules
A command in `excludedCommands`	No, full access	Keep patterns narrow
Writes to `.claude/`, `.bashrc`, `.git/hooks`, `.mcp.json`	Yes (protected paths)	Built-in sandbox protection
Exfiltration to an allowed domain	No, hostname-only check	`tlsTerminate` (experimental) or a tighter allowlist

A few rows need more explanation.

Excluded commands are fully outside. "An excluded command runs with your full access." A broad pattern like docker * hands Claude everything Docker can do, and that includes mounting your home directory into a container. Write the narrowest pattern that works, such as docker compose up * rather than docker *.

The proxy only checks the hostname. By default the proxy "does not terminate or inspect TLS traffic" (docs). If you allow github.com, a command can push anything to any repo on github.com. The experimental network.tlsTerminate setting lets the proxy inspect traffic, and credential masking depends on it.

Some settings weaken the boundary. allowNestedUserNamespaces (Linux/WSL2) lets code end up outside the sandbox. allowAppleEvents (macOS) lets commands launch other apps unsandboxed. enableWeakerNestedSandbox exposes process information. Only turn these on for a specific tool that needs them.

What it does protect well. Writes to protected paths are denied even inside your project. That covers .claude settings, .mcp.json, .gitconfig, .bashrc, .zshrc, .vscode, .idea, and Git internals like HEAD, refs and .git/hooks. If a symlink appears at a protected settings file's path mid-session, writes to its target are denied too (docs). So a prompt-injected command can't quietly rewrite your settings or plant a Git hook.

Example of a full closure (sample values): to keep a nightly job from reading your AWS keys through any route, you need two things. A sandbox credentials.files entry with "mode": "deny" stops Bash. A permission deny rule for the Read tool on that path stops the Read tool. You can copy deny-rule templates from /ai-agent-guardrails-example-copy-paste-claude-code-rules/.

How do Claude Code hooks fit with sandbox mode?

Claude Code hooks are scripts that run at lifecycle events like PreToolUse. They can block tool calls the sandbox ignores, but they run outside the sandbox themselves.

Hooks are defined in settings files (~/.claude/settings.json, .claude/settings.json, .claude/settings.local.json or managed policy). The hooks reference lists events such as SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PostToolUseFailure and Stop. PreToolUse runs "before a tool call executes" and can block it. You block either by exiting with code 2 or by returning permissionDecision: "deny" with a reason.

That makes hooks the right tool for the gaps in the table above. The sandbox can't stop the Edit tool from touching migrations/, but a PreToolUse hook matched to Edit|Write can. It can also add dynamic logic that static config can't express, such as "deny pushes after 6 p.m." or "deny writes when the test suite is red."

To add one:

Save a script, for example ~/.claude/hooks/guard.sh, that reads the JSON on stdin and exits 2 to block.
Make it executable: chmod +x ~/.claude/hooks/guard.sh.
Add it under hooks.PreToolUse with a matcher (for example "Edit|Write") in settings.json.
Run /hooks and confirm it's listed under PreToolUse (costs docs use the same check).

The tradeoff: command hooks "run with your full access" (sandboxing docs). A hook that downloads something or calls an API isn't held back by your allowedDomains. Keep hook scripts short, review them like production code, and store them where the sandbox denies writes. .claude/ is a protected path, so Claude can't rewrite hooks you keep there. For ready-made scripts, see /claude-code-hooks-examples-7-copy-paste-scripts/.

How do you check Claude Code usage?

You check Claude Code usage with the /usage command in a session. Authoritative billing lives in the Claude Console for API users and on claude.ai for subscribers.

Sandboxing makes Claude do more without stopping, which can also mean more tokens. So check usage after you turn on auto-allow. Your options, per the Claude Code costs docs:

/usage in a session. The Session block shows total cost, API duration and tokens by model. Claude Code computes the dollar figure locally at list price, so it's an estimate. It resets when you run /clear (since v2.1.211).
Plan breakdown (Pro, Max, Team, Enterprise). The same screen shows usage bars, attribution to skills, subagents and MCP servers, and recommendations. Press d or w to switch between 24 hours and 7 days.
API billing. The Console usage page is the authoritative source. The first time you authenticate, a "Claude Code" workspace is created automatically, and you can set workspace spend limits there.
Headless runs. Cap a single run with --max-budget-usd.

For scale, Anthropic reports that across enterprise deployments, the average cost is "around $13 per developer per active day" and $150–250 per developer per month (as of October 2026; costs docs). Background features use a small amount even when idle, typically under $0.04 per session. More detail is in /claude-code-how-to-see-token-usage-and-reduce-it/.

Next steps

Start with the three DIY steps below. All of them work on any plan.

DIY today:

Run /sandbox, choose auto-allow, then run the two test commands (touch ~/sandbox-probe and curl --noproxy '*' https://example.com). Both should fail.
Add failIfUnavailable: true and allowUnsandboxedCommands: false to ~/.claude/settings.json on any machine that runs jobs unattended.
Close the read gap: deny ~/.ssh and ~/.aws/credentials in sandbox.credentials, and add matching Read deny rules.

If you'd rather not build this yourself: Claude Code is included with Anthropic's Pro plan at $20/month and Max from $100/month (as of October 2026; check the official pricing page). Paying doesn't configure the sandbox for you; it buys more usage. Team plans ($25/seat monthly for Standard, per the official pricing page) add central administration, so an admin can enforce the sandbox for everyone through managed settings with failIfUnavailable: true (sandboxing docs). That suits a small team where one person owns security. It's overkill for a solo builder, who gets the same sandbox for free in user settings. If you need isolation that also covers the Read and Edit tools, a dev container or VM is the official recommendation for bypassPermissions. The cost of that is extra setup and upkeep.

Related guides on this site:

FAQ

What does Reddit say about Claude Code sandbox mode?

Community threads mostly cover practical friction, such as Docker, SSH git remotes and database clients failing inside the sandbox. All of these are documented with fixes on the official sandboxing page. The usual fix is a narrow excludedCommands entry or switching git to HTTPS. Check any forum advice against the docs, because settings like allowNestedUserNamespaces quietly weaken the boundary.

Can I use Claude Code sandbox and auto mode together?

Yes. They work independently and combine. The sandbox limits what Bash can reach, and auto mode's classifier reviews other actions, including unsandboxed retries (permission modes docs). Auto mode requires a supported model, such as Opus 4.6+ or Sonnet 4.6+ on the Anthropic API.

Is it safe to use Claude Code YOLO mode with the sandbox?

Only partly. --dangerously-skip-permissions still lets the Edit, Write and Read tools act outside the sandbox. It also auto-approves unsandboxed retries unless you set allowUnsandboxedCommands: false. Anthropic says to use bypass mode only in containers or VMs, ideally without internet access (docs).

Is there a Claude Code CLI flag for sandbox mode?

The official sandboxing page lists no dedicated flag. For one run, pass claude --settings '{"sandbox": {"enabled": true}}'. For every run, put the sandbox block in ~/.claude/settings.json (docs).

How do I check Claude Code usage after enabling the sandbox?

Run /usage in the session for an estimated cost and token breakdown. Use the Console usage page for authoritative API billing. Subscribers see plan usage bars on the same /usage screen (costs docs).

Sources

This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.

AI Agent Evaluation Metrics: Pass Rate Across Repeat Runs

Aoba Works — Tue, 27 Oct 2026 16:45:32 +0000

Originally published at Unattended.


Track three AI agent evaluation metrics for Claude Code runs that nobody watches. The first is **pass@k**: can the agent solve the task at all within k tries? The second is **pass^k**, or pass rate across repeat runs: does it succeed every time? The third is **graded-score drift**: is the quality score from your grader slowly falling on input that hasn't changed? Run each task 10–20 times in a clean environment and log a pass/fail result plus a score for every run. Then compute these numbers before you let a cron job run the agent alone.

Anthropic defines the second metric as one that "measures the probability that all k trials succeed" ([Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)). That's the question an overnight automation actually asks. The [pillar guide on the eval loop](https://en.aobaworks.com/ai-agent-evaluation-framework-claude-code-eval-loop-steps/) names these metrics. This guide shows how to choose and compute them.

## Which option is right for you?

If you're starting out, use a bash loop around `claude -p` plus a short Python script. Move to a framework only once you have more than a handful of tasks.

| Your situation | What to pick first | First page to open |
|---|---|---|
| Beginner, one or two recurring tasks | Bash loop + `claude -p --output-format json` + pass rate over 10 reruns | [Claude Code headless docs](https://code.claude.com/docs/en/headless) |
| Daily work use, cron jobs already running | pass^k on a fixed set of 20–50 tasks, gated before every prompt/model change | [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) |
| Many tasks, want a results table and assertions | Promptfoo with `--repeat` and the `anthropic:claude-agent-sdk` provider | [Promptfoo: Evaluate Coding Agents](https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/) |
| Tight budget | Code-based graders only (tests, file checks); skip LLM judges at first | [Promptfoo command line docs](https://www.promptfoo.dev/docs/usage/command-line/) |
| Output is prose or reviews, not code | Add an LLM-judge score and track its drift | Section on graded-score drift below |

Here's why each pick fits. A bash loop teaches you what the numbers mean before a tool hides them. pass^k is the metric that matches "runs without me". Promptfoo saves you from writing your own repeat-and-cache plumbing. Code graders cost nothing per run. And prose can't be checked by `npm test`.

## What Are Evaluation Metrics in AI Agent Testing?

In Claude Code testing, an evaluation metric is a rule that turns each run's result into a pass/fail or a score. That lets you compare runs instead of eyeballing transcripts.

Anthropic's evals post uses four words you'll need throughout this guide:

- A **task** is "[a single test with defined inputs and success criteria](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)".
- A **trial** is one attempt at a task. You run several, "[because model outputs vary between runs](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)".
- A **grader** is the logic that scores the trial.
- The **outcome** is the final state of the environment when the trial ends.

A metric sits on top of these. It's what you get when you add up grader results across trials and tasks.

For example, say your nightly job asks Claude Code to "bump the patch version, update CHANGELOG.md, and make the test suite pass". The task is that prompt plus a frozen copy of the repo. A grader might check three things: `npm test` exits 0, `package.json` has the expected version, and `CHANGELOG.md` has a new top entry. One trial gives one pass/fail. Ten trials give a pass rate. That pass rate is the metric.

**Pitfalls:**

- **Treating the exit code as the metric.** Claude Code exits with code 0 on success ([headless docs](https://code.claude.com/docs/en/headless)). That means the run ended normally, not that the work is right. A run can exit 0 after editing the wrong file. Always grade the outcome (tests, files, diff) separately from the exit status.
- **Scoring the transcript text.** A confident summary like "All tests pass!" isn't evidence. Run the tests yourself in the grader.
- **No reference solution.** Anthropic recommends tasks that are unambiguous, with a known solution that proves they can be solved. If you can't write the passing answer yourself, the metric can't tell a model failure from a broken task.

## What Is Evaluation Metrics in Machine Learning (and How Agent Evals Differ)?

Classic machine learning metrics score one fixed prediction against a fixed label. Agent evals for tools like Claude Code score multi-step runs that can succeed in different ways or fail quietly partway through.

scikit-learn describes [accuracy](https://scikit-learn.org/stable/modules/model_evaluation.html) as "the fraction (default) or the count (normalize=False) of correct predictions". Precision, recall and F1 refine that for imbalanced classes. These metrics assume something that agent runs break: the same input always produces the same output, and there's one correct label to compare against.

| Property | Classic ML metrics (accuracy, F1) | Coding-agent evals (Claude Code) |
|---|---|---|
| Unit scored | One prediction | A whole trial: tool calls, edits, final repo state |
| Determinism | A fixed model gives the same prediction | Same prompt, different paths and results per run |
| Ground truth | A label | A checkable end state (tests pass, file exists, no stray changes) |
| Valid answers | Usually one | Many diffs can be correct |
| Repeats needed | Once per example | Multiple trials per task to estimate a rate |
| Silent failure mode | Wrong label | Plausible summary over a broken or partial change |

The practical difference is the repeats. A classifier's accuracy on a test set is fixed once you freeze the model. A Claude Code task's pass rate is a probability, and you can only estimate it by sampling.

That's why the Codex paper that made pass@k popular reported two numbers. Its model solved 28.8% of HumanEval problems with 1 sample per problem ([Chen et al.](https://arxiv.org/abs/2107.03374)), and "[70.2% of our problems with 100 samples per problem](https://arxiv.org/abs/2107.03374)". Same model, same problems. The only difference was how many tries it got.

**Pitfall:** Don't borrow precision/recall unless your agent really is classifying something. A triage agent that labels issues fits that frame. An agent that writes code needs pass rates.

## What Is Evaluation in AI (Agent Context)?

For Claude Code, evaluation means rerunning a task many times and checking the final state it leaves. A plausible-looking output isn't enough; the outcome has to match what you intended.

Anthropic's rule for this is to "[grade what the agent produced, not the path it took](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)". If Claude fixes a bug by editing `utils.ts` when you expected `auth.ts`, and the tests pass with no unrelated changes, that counts as a pass. Grading the exact steps penalizes valid alternatives and makes your metric noisier.

The same post warns that "[unnecessary shared state between runs (leftover files, cached data, resource exhaustion) can cause correlated failures](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)". That's why a single trial needs a clean environment.

Steps for one evaluation trial:

1. **Reset the environment.** Make a fresh checkout or worktree of a pinned commit. Don't reuse the last trial's folder.
2. **Run the agent headless.** Use `claude -p "<task prompt>" --output-format json` so the result and metadata come back as JSON ([headless docs](https://code.claude.com/docs/en/headless)).
3. **Grade the outcome.** Run tests, check files, and run `git diff --stat` to spot changes outside the expected paths.
4. **Record one row.** Save the task id, trial number, pass/fail, score if any, cost and duration.
5. **Read some transcripts.** Anthropic notes that 0% pass@100 usually points to a broken task spec, not a hopeless agent ([Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)). You only find that out by reading what happened.

If you already run Claude Code from cron, this is the same command your job uses, pointed at a frozen copy of the repo. See [Claude Code Headless Mode: Flags, Auth and Cost](https://en.aobaworks.com/claude-code-headless-mode-flags-auth-and-cost-2026/) for the flags and auth.

## What Are the Core AI Agent Performance Metrics?

The core metrics are pass@k, pass^k, success rate, cost, latency and turns per task. Compute all of them on a fixed task set.

Here are the definitions and the numbers to compute:

| Metric | What it answers | How to compute from n trials with c passes | Where the data comes from |
|---|---|---|---|
| pass@1 (success rate) | How often does one run succeed? | c / n | Your grader |
| pass@k | Can it solve this at all within k tries? | 1 − C(n−c, k) / C(n, k) | Your grader |
| pass^k | Will k runs in a row all succeed? | C(c, k) / C(n, k) | Your grader |
| Cost per task | What does one trial cost? | Mean of `total_cost_usd` | [`--output-format json`](https://code.claude.com/docs/en/headless) |
| Latency per task | How long does a trial take? | Mean or p90 of `duration_ms` | [SDK result message](https://code.claude.com/docs/en/agent-sdk/typescript) |
| Turns per task | How much back-and-forth did it need? | Mean of `num_turns` | [SDK result message](https://code.claude.com/docs/en/agent-sdk/typescript) |
| Cost per successful task | What do you pay per good result? | Total cost / c | Both |

The pass@k formula is the unbiased estimator from the Codex paper: pass@k := E[1 − C(n−c, k)/C(n, k)]. The paper calls the shortcut 1 − (1 − p̂)^k "[biased](https://arxiv.org/html/2107.03374)", and its authors drew n=200 samples per problem for k ≤ 100 ([Chen et al., section 2.1](https://arxiv.org/html/2107.03374)).

pass^k comes from τ-bench, which introduced it as "[a new metric to evaluate the reliability of agent behavior over multiple trials](https://arxiv.org/abs/2406.12045)". The C(c,k)/C(n,k) form above uses the same combinatorial idea: the share of k-sized subsets of your trials in which every trial passed.

The two metrics move in opposite directions as k grows. Anthropic's example uses a 75% per-trial success rate, where the chance that all 3 of 3 trials pass is about 42% (0.75³) ([Anthropic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)). As k rises, pass@k heads toward 100% and pass^k falls. τ-bench found leading function-calling agents were "[quite inconsistent (pass^8 <25% in retail)](https://arxiv.org/abs/2406.12045)".

To see how this works out, here's an example calculation (illustrative numbers, not a measurement). You run a task 10 times and 7 pass.

- pass@1 = 0.70
- pass@3 = 1 − C(3,3)/C(10,3) = 1 − 1/120 ≈ 0.99
- pass^3 = C(7,3)/C(10,3) = 35/120 ≈ 0.29

"It basically always works" (pass@3) and "three nights in a row, it fails at least once about 70% of the time" (pass^3) describe the same 10 runs. For unattended work, believe pass^k.

**Which k should you use?** Match k to how you actually run the job. If a human picks the best of three drafts, pass@3 is fine. If cron runs it daily and nobody checks, use pass^k with k equal to how many runs you'd let happen before you look. For example, k=7 if you review weekly.

## How Do AI Agent Performance Metrics Work in Amazon Connect?

Amazon Connect's agent metrics measure contact-center agents and AI assistants, not code correctness. Only the idea of "a rate over many interactions" carries over to Claude Code.

AWS defines metrics like these in the [Amazon Connect metric definitions](https://docs.aws.amazon.com/connect/latest/adminguide/metrics-definitions.html):

| Amazon Connect metric (API id) | What AWS says it measures | Nearest Claude Code equivalent | Does it transfer? |
|---|---|---|---|
| Average handle time (`AVG_HANDLE_TIME`) | Agent interaction, hold and after-contact work time | Mean `duration_ms` per trial | Partly: latency only |
| AI handoff rate (`AI_HANDOFF_RATE`) | Share of AI-handled contacts escalated to humans or extra support | Share of runs that need your manual fix | Yes, as an idea |
| AI agent invocation success rate (`AI_AGENT_INVOCATION_SUCCESS_RATE`) | Share of invocations that "[executed successfully without technical failures](https://docs.aws.amazon.com/connect/latest/adminguide/metrics-definitions.html)" | Share of runs with exit code 0 | Only as a health check |
| Agent answer rate (`AGENT_ANSWER_RATE`) | (Contacts accepted / total routing attempts) × 100 | None | No |

If you came here from a search about Connect, the useful lesson is the gap between the last two rows. "Invocation success" in Connect, and exit code 0 in Claude Code, both tell you that nothing crashed. Neither tells you the answer was right.

**Pitfall:** Reporting invocation success as your quality metric. A coding agent can have 100% clean exits and a 60% pass rate. Track both. The pass rate is the one that matters, and clean exits are only a health signal.

**Fix:** Pair any health metric with an outcome grader. In Connect terms, that's like pairing invocation success with handoff rate. In Claude Code, pair exit status with your test-based grader.

## What Are the Common AI Agent Evaluation Metrics for Coding Agents?

Three metrics matter most: pass rate across N reruns, run-to-run variance, and graded-score drift. This holds especially for solo builders who run Claude Code repeatedly.

Most top-ranking pages on this query list categories: task completion, tool-call accuracy, latency, business KPIs. They rarely show how to separate a flaky task from a reliable one. That's the part that decides whether a cron job is safe.

### Pass rate across N reruns

Run each task n times (10 is a sensible minimum, 20 if the task is cheap). Record c passes per task and compute pass@1, pass@k and pass^k as shown above. Anthropic suggests starting with "[20-50 simple tasks](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)" taken from real failures, so you don't need a big benchmark.

### Variance across reruns (flakiness)

Flakiness is simple to detect. A task is flaky when 0 < c < n: it passed sometimes and failed sometimes on the same input. The harder question is how sure you can be about its pass rate. A pass rate p from n trials has a standard error of about √(p(1−p)/n). Here's an example: at p=0.7 and n=10, that's √(0.021) ≈ 0.145, so roughly ±14 percentage points. That's arithmetic, not a measurement.

So with 10 reruns, you can't reliably tell a 70% task from an 85% task. Two rules follow:

1. **Count flaky tasks, not just the average.** "18 of 20 tasks are 10/10, two are 4/10" is far more useful than "88% overall". The two flaky tasks are where your overnight failures will come from.
2. **Rerun before you conclude.** If a prompt change moves a task from 7/10 to 9/10, run another 10 before you believe it.

Promptfoo's coding-agent guide puts the next step bluntly: "[If a prompt fails 50% of the time, the prompt is ambiguous.](https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/)" Its advice is to fix the instructions rather than add retries. Retries raise pass@k but leave pass^k where it was.

### Graded-score drift

Drift is a slow fall in a 1–5 or 0–100 quality score on input that hasn't changed. It matters for output that tests can't fully check, like PR review comments, changelogs or docs. You track it by keeping a baseline and comparing each new batch against it.

Steps:

1. Freeze 10–20 inputs, and have an LLM judge score each output with a fixed rubric (for example, 1–5 for "covers every changed function").
2. **Measure judge noise first.** Grade the *same* saved transcripts twice. Anthropic notes model-based graders are "[non-deterministic](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)", so part of any movement is the judge, not the agent.
3. Set the baseline as the mean score of your first full batch.
4. On each later batch, flag drift when the mean drops by more than about twice the standard error of the batch mean, or by an absolute threshold you choose (for example, 0.3 on a 5-point scale). These numbers are illustrative.
5. When drift fires, diff the transcripts from a high-scoring baseline run and a low-scoring new run. Look at what changed, such as a new model, a CLAUDE.md edit or an MCP server.

**Pitfall:** Changing the judge prompt or judge model mid-series. Every judge change resets the baseline. Keep a `judge_version` column and never compare scores across versions.

## How Do You Set Up Metrics Visualization for an AI Agent Evaluation Framework?

A minimal Claude Code framework appends one JSON line per trial to a file and then plots pass rate and score per task over time.

Here's the logging loop. It's an example script, so adapt the paths and the grader to your project. It uses only flags from the [headless docs](https://code.claude.com/docs/en/headless): `-p`, `--output-format json`, `--allowedTools`, and `--bare`. Bare mode skips local hooks, plugins, MCP servers and CLAUDE.md, so every trial starts the same way. It needs `ANTHROPIC_API_KEY`, because bare mode "doesn't use your subscription login" ([headless docs](https://code.claude.com/docs/en/headless)).

bash

!/usr/bin/env bash

eval.sh (example)

set -u
TASK="$1"; PROMPT=$(cat "$2"); RUNS="${3:-10}"
for i in $(seq 1 "$RUNS"); do
DIR=$(mktemp -d)
git worktree add --detach "$DIR" eval-base >/dev/null 2>&1
OUT=$(cd "$DIR" && claude --bare -p "$PROMPT" \
--allowedTools "Read,Edit,Bash" --output-format json)
EXIT=$?
(cd "$DIR" && npm test >/dev/null 2>&1) && PASS=1 || PASS=0
echo "$OUT" | jq -c --arg t "$TASK" --argjson i "$i" \
--argjson p "$PASS" --argjson e "$EXIT" \
'{task:$t, trial:$i, pass:$p, exit:$e, cost:.total_cost_usd,
ts:(now|todate)}' >> results.jsonl
git worktree remove --force "$DIR"
done


Then summarize and plot the results (also an example):

python
import json, math, collections
import matplotlib.pyplot as plt
rows = [json.loads(l) for l in open("results.jsonl")]
by = collections.defaultdict(list)
for r in rows: by[r["task"]].append(r["pass"])
def pass_at(n,c,k): return 1 - math.comb(n-c,k)/math.comb(n,k)
def pass_hat(n,c,k): return math.comb(c,k)/math.comb(n,k)
for t, v in by.items():
n, c = len(v), sum(v)
print(t, f"p1={c/n:.2f}", f"pass@3={pass_at(n,c,3):.2f}",
f"pass^3={pass_hat(n,c,3):.2f}", "FLAKY" if 0<c<n else "")
tasks = list(by)
plt.bar(tasks, [sum(by[t])/len(by[t]) for t in tasks])
plt.ylabel("pass rate"); plt.ylim(0,1); plt.savefig("pass_rate.png")




What to chart, in order of usefulness:

1. **Bar chart of pass rate per task**, with flaky tasks highlighted. This is your daily view.
2. **Line chart of pass^k for the whole suite**, one point per eval batch, with batches labeled by model or prompt version. This shows regressions.
3. **Line chart of mean judge score** with a band of ± two standard errors. This shows drift.
4. **Cost per successful task** per batch. Retries and long loops show up here before they show up on your bill. The guide to [seeing and reducing token usage in Claude Code](https://en.aobaworks.com/claude-code-how-to-see-token-usage-and-reduce-it/) covers where the tokens go.

Two caveats from the docs apply. First, `total_cost_usd` is a client-side estimate that "can differ from your actual bill" ([headless docs](https://code.claude.com/docs/en/headless)). Second, if a run ends without valid structured output, the JSON has no `structured_output` field, so check the field exists before your grader reads it.

**When to switch to Promptfoo instead:** use it when you have dozens of tasks and want a results table without writing your own. `promptfoo eval --repeat 3` runs each test 3 times ([command line docs](https://www.promptfoo.dev/docs/usage/command-line/)). Each repeat gets its own cache namespace, and the docs say to add `--no-cache` when you want fresh calls every time ([command line docs](https://www.promptfoo.dev/docs/usage/command-line/)). The Claude Agent SDK provider id is `anthropic:claude-agent-sdk` ([coding agents guide](https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/)).

Promptfoo isn't for you if you have two tasks and want zero new dependencies. It also has a limitation for this guide: it reports per-test pass/fail, and you still compute pass^k from those results yourself.

## How Do You Build an AI Agent Evaluation Framework? A Metrics Checklist

Pick fixed tasks, grade them, rerun them, and block changes that fall below a threshold. In between, compute pass@k and pass^k for every task.

Follow this order:

1. **Pick 20–50 tasks from real work.** Use failures from your own cron logs first. The [silent-failure monitoring guide](https://en.aobaworks.com/ai-agent-monitoring-in-claude-code-catch-silent-failures/) explains how to spot them.
2. **Write a reference solution for each task**, so you know the task can be solved.
3. **Write a code-based grader for each task**: tests, file checks, and an allow-list of paths the diff may touch. Add an LLM judge only for output tests can't check.
4. **Pin the environment.** Tag a commit (`eval-base` in the script above), use `--bare`, and give every trial a fresh worktree.
5. **Run n=10 trials per task** (20 for tasks you'll run unattended daily).
6. **Compute per task:** pass@1, pass^k (k = runs between your reviews), a flaky flag, mean cost, and judge mean if you use one.
7. **Set gates.** For example, every unattended task must hit pass^k ≥ 0.9 with no new flaky tasks compared with the last batch. These thresholds are illustrative, so pick ones that match how costly a bad night is for you.
8. **Rerun the full suite after every change**: model, prompt, CLAUDE.md, hooks, MCP servers, or a Claude Code upgrade. Anthropic calls these regression evals, which ask "[Does the agent still handle all the tasks it used to?](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)"
9. **Read transcripts for every task at 0% or 100%** at least once. A 0% usually means a broken task. A long run of 100% means the task is saturated and no longer tells you anything.
10. **Only then schedule it.** Follow the [cron job setup guide](https://en.aobaworks.com/claude-code-cron-job-setup-run-automations-overnight/) and keep the same flags as your eval.

Common mistakes and fixes:

| Mistake | What it does to the metric | Fix |
|---|---|---|
| Reusing the same folder across trials | Correlated failures; pass rate looks worse (or better) than it is | Fresh worktree per trial |
| Running without `--bare` | A local hook or MCP server changes behavior between machines | `--bare`, and pass context with flags |
| Promptfoo repeats served from cache | Every repeat is identical, so variance looks like 0 | Add `--no-cache` |
| Adding retries to "fix" a flaky task | pass@k rises, pass^k stays flat | Clarify the prompt or the task, then rerun |
| Judging with the same model and prompt you're testing | Judge shares the agent's blind spots | Use a fixed rubric and check it against your own labels on a few samples |
| Comparing 7/10 vs 8/10 as a real change | Inside noise (±14 pts at n=10) | Rerun; compare only large or repeated shifts |

If a batch regresses after a change, roll back the prompt or config first, then investigate. The [checkpoints and rollback guide](https://en.aobaworks.com/claude-code-checkpoints-how-rollback-works-in-automations/) explains what Claude Code can and can't undo.

## Next steps

Do these today:

1. Pick the one cron task that worries you most. Freeze a commit, and run `eval.sh` on it with `RUNS=10`.
2. Compute pass@1 and pass^7 with the Python snippet. If pass^7 is below what you'd accept for a week unattended, the task isn't ready to run alone.
3. If any task is flaky (0 < c < 10), rewrite its prompt to remove the ambiguity and rerun 10 more before changing anything else.

If you'd rather not build this yourself, here are some options:

- **A Claude subscription that includes Claude Code**: Pro is "$20 if billed monthly" ([Claude pricing](https://claude.com/pricing)), and Max starts "From $100 Per month" ([Claude pricing](https://claude.com/pricing)) (as of October 2026; check the official page). It suits builders who run evals interactively and often. It's not the right fit for the `--bare` script above, which bills to an `ANTHROPIC_API_KEY` on the Anthropic API instead of your subscription.
- **Promptfoo**: a ready-made runner with repeats, assertions and a provider for the Claude Agent SDK ([guide](https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/)). It suits people with many tasks. It's not worth it for one or two tasks, and you still compute pass^k yourself.
- **Anthropic's evals post** is a free reference if you want to design graders more carefully before you automate anything ([read it](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)).

Related guides on this site:

- [AI Agent Evaluation Framework: Claude Code Eval Loop Steps](https://en.aobaworks.com/ai-agent-evaluation-framework-claude-code-eval-loop-steps/), the overview this guide expands on
- [AI Agent Monitoring in Claude Code: Catch Silent Failures](https://en.aobaworks.com/ai-agent-monitoring-in-claude-code-catch-silent-failures/)
- [Claude Code Headless Mode: Flags, Auth and Cost (2026)](https://en.aobaworks.com/claude-code-headless-mode-flags-auth-and-cost-2026/)
- [Claude Code Cron Job Setup: Run Automations Overnight](https://en.aobaworks.com/claude-code-cron-job-setup-run-automations-overnight/)

## FAQ

### What is evaluation metrics in machine learning?

Evaluation metrics in machine learning are formulas that score a model's predictions against known labels, such as accuracy, precision, recall and F1 in [scikit-learn](https://scikit-learn.org/stable/modules/model_evaluation.html). They assume a fixed model gives the same prediction for the same input. AI agents break that assumption, so agent evals add repeated trials and rates like pass@k.

### What is evaluation in AI?

In the AI agent context, evaluation means running a defined task, grading the outcome with code or a model, and repeating it enough times to estimate how often the agent succeeds. For coding agents like Claude Code, you grade the final repo state (tests, files, diff), not the explanation the agent writes.

### What are ai agent performance metrics in Amazon Connect?

In Amazon Connect they're contact-center metrics such as average handle time, AI handoff rate and AI agent invocation success rate, defined in the [AWS metric definitions](https://docs.aws.amazon.com/connect/latest/adminguide/metrics-definitions.html). They measure customer conversations, not code correctness. Invocation success only means no technical failure occurred.

### What are common ai agent evaluation metrics?

The common ones are task success rate (pass@1), pass@k, pass^k, tool-call accuracy, cost per task, latency and turns per task. For unattended Claude Code jobs, pass^k and the number of flaky tasks are the most useful, because they predict failures across consecutive runs.

### How does ai agent evaluation framework metrics visualization work?

You log one record per trial (task, pass/fail, score, cost, timestamp, version) and chart it. A per-task pass-rate bar chart shows flaky tasks. A pass^k line across batches shows regressions after changes, and a judge-score line with an error band shows drift.

## Sources

- [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
- [Claude Code docs: Run Claude Code programmatically (headless)](https://code.claude.com/docs/en/headless)
- [Claude Agent SDK TypeScript reference (result message fields)](https://code.claude.com/docs/en/agent-sdk/typescript)
- [Chen et al., Evaluating Large Language Models Trained on Code (arXiv:2107.03374)](https://arxiv.org/abs/2107.03374) and [HTML version, section 2.1](https://arxiv.org/html/2107.03374)
- [Yao et al., τ-bench (arXiv:2406.12045)](https://arxiv.org/abs/2406.12045)
- [scikit-learn: Metrics and scoring](https://scikit-learn.org/stable/modules/model_evaluation.html)
- [AWS: Amazon Connect metric definitions](https://docs.aws.amazon.com/connect/latest/adminguide/metrics-definitions.html)
- [Promptfoo: Evaluate Coding Agents](https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/)
- [Promptfoo: Command line](https://www.promptfoo.dev/docs/usage/command-line/)
- [Claude pricing](https://claude.com/pricing)
- [SWE-bench evaluation guide](https://www.swebench.com/SWE-bench/guides/evaluation/)
- Top results reviewed for the query: [Amplitude](https://amplitude.com/explore/analytics/ai-agent-evaluation-metrics), [Weights & Biases](https://wandb.ai/onlineinference/genai-research/reports/AI-agent-evaluation-Metrics-strategies-and-best-practices--VmlldzoxMjM0NjQzMQ), [IBM](https://www.ibm.com/think/topics/ai-agent-evaluation)

---

*This article was created with the help of AI. Sources are linked inline; check the official documentation for the latest details.*