DEV Community

Cover image for Claude Code's Safety Telemetry Event Lived Six Days. Audit Your Agents With Git
Gabriel Anhaia
Gabriel Anhaia

Posted on Edited on AI-assisted

Claude Code's Safety Telemetry Event Lived Six Days. Audit Your Agents With Git

#ai #claudecode #opentelemetry #discuss

claude_code.safety_event existed for six days. Claude Code 2.1.307 added it on October 21, a telemetry event a security team would want, logged "for each tool call a permission rule, auto mode, a safety check, a hook or the sandbox stopped, prompted or sent back" (changelog).

Version 2.1.312 carried one line about it on October 27: "Changed Claude Code to stop sending the claude_code.safety_event OpenTelemetry event" (same changelog).

If you wired a dashboard or an alert to that event last week, it went quiet on the day your developers updated. Nothing errors when an event stops arriving. The panel shows zero, and zero looks like good news.

Vendor telemetry from an agent CLI is useful, and you should collect it. But it is a schema that moves every few days, so it can't be the only record of what your agents did. The record that holds still is the one you own: git, and hooks you wrote.

A CLI that ships 13 times in 14 days

Look at the release cadence first. The Claude Code changelog lists 2.1.300 on October 14 and 2.1.312 on October 27, with a release on most days in between (changelog). Thirteen versions in fourteen days.

Many of those touched the OpenTelemetry output. In that same window, the changelog records:

  • 2.1.300 split cache writes into cache_creation_5m_tokens and cache_creation_1h_tokens on the api_request event, and added GenAI semantic-convention attributes to trace spans.
  • 2.1.301 added classifier_verdict and permission_mode to tool_decision events.
  • 2.1.302 fixed new tool_decision attributes "pushing out tool_use_id, tool_source and tool_parameters" when OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT is low.
  • 2.1.312 changed mcp_server.name to report real names for some plugin servers "instead of custom", and stopped sending safety_event.

Every one of those is a reasonable change, and most are improvements. Anthropic writes them down, which is more than many vendors do. But read 2.1.302 again: for a while, on some configurations, adding attributes could push out the ID you would join on. And after 2.1.312, a query grouping MCP calls by mcp_server.name can split what used to be one custom bucket into several names. A query written on October 20 and the same query on October 28 are not measuring the same thing.

The docs say as much about one corner of it. On joining OTel data to transcripts, the monitoring page warns that the transcript format "changes between versions, so a pipeline that joins on these fields can break on any release; treat the joins as version-specific rather than a stable contract."

A security dashboard panel reading flat zero while the pipe feeding it has been quietly disconnected upstream
An alert on a missing event doesn't fire. It just reads zero.

The docs still point at the event

Here is the part that makes it easy to miss. As of October 29, the monitoring page still has a full "Safety event" section for claude_code.safety_event, with its category, decision, decision_source and reason_code attributes, marked "Requires Claude Code v2.1.307 or later." It does not say the event stopped in 2.1.312.

It goes further. The tool_decision event's source attribute says "config" without telling you which rule matched, and the page's answer is: "To find which one refused a call, look for the Safety event with the same tool_use_id and read its category and rule_source attributes." The security signals table on the same page lists safety_event as the event for "Which check refused a tool call or required a permission prompt for it."

So a careful engineer who reads the docs today, builds the query the docs describe, and points it at a current Claude Code gets nothing back. Docs and code drift, and a team shipping that fast will drift more. It is exactly why you don't make a fast-moving vendor stream your only source of truth.

To be fair to the telemetry: tool_decision is still documented, with decision set to "accept" or "reject" and a source that separates config, hooks and the user's own answers. Keep collecting it. Just treat it like a third-party API: pin the Claude Code version you validated, alert when the event volume for a type drops to zero, and re-read the changelog before you upgrade.

What doesn't move: hooks you own

Claude Code's hooks are a contract you control. A command hook gets a JSON payload on stdin with session_id, cwd, permission_mode and hook_event_name, plus tool_name, tool_input and tool_use_id for tool events like the three below. The fields can still change between versions, but the file they land in, its location and its retention are yours.

Three events get you most of the way back. PreToolUse fires before every tool call. PermissionRequest runs "when Claude Code is about to ask you for permission to use a tool." PermissionDenied runs when auto mode denies a call, with a reason field, and only then: the docs say it doesn't run for a manual denial, a PreToolUse hook block or a matching deny rule (hooks docs). So a call a deny rule blocked without a prompt shows up in your log as a PreToolUse line with no outcome. For the outcome, pair that line with tool_decision on the same tool_use_id.

A logger that appends each payload as one JSON line:

#!/bin/sh
# .claude/hooks/audit.sh
# Append the hook payload to a JSONL audit log.
log="$CLAUDE_PROJECT_DIR/.claude/audit/tool-calls.jsonl"
mkdir -p "$(dirname "$log")"
jq -c '. + {logged_at: (now | todate)}' >> "$log"
exit 0
Enter fullscreen mode Exit fullscreen mode

It needs jq and chmod +x. Register it for all three events in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [{ "matcher": "*", "hooks": [
      { "type": "command",
        "command":
          "\"${CLAUDE_PROJECT_DIR}\"/.claude/hooks/audit.sh" }
    ]}],
    "PermissionRequest": [{ "matcher": "*", "hooks": [
      { "type": "command",
        "command":
          "\"${CLAUDE_PROJECT_DIR}\"/.claude/hooks/audit.sh" }
    ]}],
    "PermissionDenied": [{ "matcher": "*", "hooks": [
      { "type": "command",
        "command":
          "\"${CLAUDE_PROJECT_DIR}\"/.claude/hooks/audit.sh" }
    ]}]
  }
}
Enter fullscreen mode Exit fullscreen mode

The script prints nothing and exits 0. The docs are explicit about what that means for PreToolUse: "Exit code 0 with no output means the hook has no decision to report, so the tool call continues through the normal permission flow." The hook records; it doesn't approve. Commit the hook and the settings so every clone gets them, and add .claude/audit/ to .gitignore if you'd rather not commit the log itself.

If a future release renames a field, your log still holds the raw payload, so you can read it and adjust. And if a hook stops firing, you notice from a file you own going stale, which is a check you can write today.

And git, which no agent vendor owns

Hooks tell you what the agent tried. Git tells you what it actually changed, and git doesn't change when your agent ships a new version.

git diff, git log -p and git diff main...agent-branch show every line an agent left behind, whatever the agent's version, whatever its vendor. When the question is "did the agent touch the CI workflow" or "did a lockfile change", the answer is in the diff. Point your alerts at the things that matter in the repo: a pre-commit hook or a CI job that fails when .github/workflows/ or a lockfile changes on an agent branch keeps working whether the agent sent an event or not.

This is also how I build Hermes IDE, so weigh this section with that in mind. Hermes runs Claude Code, Codex and other agents each in its own terminal and its own worktree. Its README says it "observes what agents report on purpose and what git shows," and it "never answers a permission prompt for you and never types into a terminal on its own." Every agent turn is saved, so you can diff or restore any single turn, with undo. The Review Desk (⌘G) shows the changes by turn or by file, with risk flags (README).

The Hermes Review Desk in By turn view: two turns of a Claude Code task, turn 1's diff of src/db.ts, the changed files and a drafted commit message
The Review Desk (⌘G) in By turn view for a Claude Code task: two turns, turn 1's diff of src/db.ts, the changed files and a commit message drafted from the turns.

None of that depends on which telemetry event a vendor sent this week. If Claude Code renames an OTel attribute tomorrow, the turn diff looks the same, because it comes from git.

What to do this week

If you collect Claude Code telemetry, search your dashboards and alert rules for safety_event. Anything built on it has been empty since your fleet moved to 2.1.312 or later. Rebuild what you can on tool_decision, and add a "this event type went silent" alert for every event you depend on.

Then put the durable layer under it: a hook log you own, and a CI check that fails when an agent branch touches .github/workflows/ or a lockfile.

If this was useful

The problem here isn't specific to Claude Code: any LLM system you watch through someone else's event stream can change shape under your alerts. Observability for LLM Applications is about building tracing, evals and cost tracking that survive that, with your own instrumentation at the core and vendor data as one input.

If you run several coding agents, Hermes IDE is source-available under BSL 1.1 and runs on macOS, Windows and Linux. The code is on GitHub.

Observability for LLM Applications — the book

Top comments (2)

Collapse
 
stefan_07d23f966225156b50 profile image
Stefan •

The logger has a smaller version of the same silent-failure problem: it exits 0 even if mkdir or jq fails. I'd want a separate alert when the audit write fails, even if the policy is to let the tool call continue. Owning the log helps, but we still need to know when it's stopped recording.

Collapse
 
kaj_ilands profile image
Kaj •

Durable doesn't mean safe. tool_input lands in that JSONL raw, and for a Bash call tool_input is the command line: curl -H "Authorization: Bearer $TOKEN", git remote add origin https://user:ghp_xxxx@github.com/org/repo, docker login -p .... The log you own forever, sitting on a laptop, becomes a credential store, and it's the one file nobody reads in review because none of it looks like source.

The fix is not a redaction list. The list is the bug: any channel you didn't name goes through untouched, and a test that walks the list can never see it. Drop one sentinel string into every channel at once (command, tool_input, cwd, whatever else reaches the hook), let the hook write, then read the file back cold and grep for the sentinel. What comes back is a live leak. What doesn't is the only thing you have actually tested.

So: does your redaction key on field name or on pattern? A field-name list loses to the next release, and this article is about exactly how fast the next release comes.