Your Claude Code skill is running on empty — and you don't know it

You ship a custom skill to your team. A week later a colleague says: "this skill doesn't work — it always returns the same thing." You open the dashboard: 47 activations, success=true on every one. All green. But the skill is broken.

Part 1 was about the catalog: what lives in your team's skill set, what's stale, what it costs. This article is about testing an individual skill on real runs. Same repository, same stack, zero patches.

---

Why you can't test a skill like code

With regular code the path is clear: write a function, cover it with tests, run them in CI. A skill breaks that on all three counts:

• The "code" is a prompt. SKILL.md is an instruction to the model, not a program. There's no function to call with a fixture.
• The behavior is nondeterministic. The same skill on the same input can take different paths: today the model reads three files, tomorrow one.
• The result is an artifact, not a value. Often it's prose: an updated document, a PR description, a code review. There's no assertEquals for prose.

The only honest approach: test in production — observe real runs and check what actually happened.

In part 1, dep-auditor exited with exit 1 on every other run — yet telemetry showed success=true on all 19 results. Documented behavior: success means the harness got a response, not that the skill succeeded. That example was easy to dismiss — who audits dependencies with a skill when npm audit exists? So let's use a skill where the model is genuinely required.

---

The test subject: doc-updater

doc-updater keeps the architecture document in sync with the stack config: read what changed, understand the implications, rewrite the prose in docs/ARCHITECTURE.md. That's exactly the work a deterministic utility can't do.

I invoked it 3 times across 2 sessions. The output is free-form text — success=true tells you nothing about whether it reflects the actual code. A shell command at least has an exit code (which, as we've seen, gets lost anyway). Prose doesn't even have that.

I opened the dashboard from part 1:

doc-updater is at the top: 12 events, green bar all the way. By this view the skill looks healthy. Now the real picture:

Out of 3 invocations, one updated the document (updated), two returned empty. empty is not an error — the code hadn't changed since the last run, so the skill correctly did nothing. The problem: by native signals those two runs looked identical to updated. And critically: if the skill had failed mid-way (say, a Confluence publish dropped over the network), it would also return empty — indistinguishable from a legitimate "nothing to do." To tell these apart, you need a signal from inside the skill.

---

Three states of a finished skill run

State	Meaning	How to check
1. Ran	Invoked, tool calls completed, success=true	Native telemetry, out of the box
2. Did meaningful work	Actually did the job — didn't process empty input	Nothing out of the box. But the skill knows
3. Did it correctly	The output is semantically correct	Not even the skill knows — needs eval or review

All native signals answer state 1. The goal of in-production testing is to reliably check state 2. State 3 is a different discipline entirely — more on that at the end.

---

Check 1. Native telemetry: free signals and their limits

Two flags from part 1 (CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_LOG_TOOL_DETAILS=1) plus OTel → Loki / Prometheus give you:

Question	Signal	Storage
Is the skill being invoked?	claude_code.skill_activated	Loki
How often?	count by skill_name	Loki
How fast?	duration_ms on tool_result	Loki
Tokens / cost?	claude_code.token.usage / cost.usage	Prometheus
What triggered it?	invocation_trigger	Loki

Two blind spots on real data:

skill_activated undercounts invocations. I invoked doc-updater 3 times. Loki shows 2 events. Why: skill_activated fires once per session — on the first invocation of a skill within a given claude process. My third invocation was the second call in one session: no event. For catalog inventory that's fine; for counting actual runs — it's not.

success=true says nothing about the work. All tool results: success=true, including both empty runs.

What happened	success	Example
Bash never launched	false	binary not found (ENOENT)
Command exited with exit 1	true	bash -c "exit 1"
Command exited with exit 0	true	bash -c "exit 0"

success is "the harness got a result," not "the skill did its job." For prose skills there's no shell failure at all — an empty run looks perfect.

---

Check 2. Hooks: intercepting from outside

A PostToolUse hook fires after every tool and receives on stdin a JSON payload of what the tool returned (tool_response).

Before writing any hook logic, I captured a real payload with a one-liner:

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Bash|Edit|Write",
      "hooks": [{ "type": "command", "command": "cat >> /tmp/hook-capture.jsonl || true" }]
    }]
  }
}

The result:

Bash  tool_response:  interrupted, isImage, noOutputExpected, stderr, stdout
Edit  tool_response:  filePath, newString, oldString, originalFile, structuredPatch, userModified

Bash has no exit code field. The hook sees stdout, stderr, interrupted — not a numeric return code. Whether the command "failed" must be inferred from \stdout/stderrcontent by hand.

For Edit the payload is richer: structuredPatch and userModified let a hook detect whether a file actually changed — a real state 2 signal for doc-updater.

The production hook that pushes an event to the collector (full payload in the repo):

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Bash",
      "hooks": [{
        "type": "command",
        "command": "jq -r '.tool_name' 2>/dev/null | grep -q . && curl -s -X POST http://localhost:4318/v1/logs -H 'Content-Type: application/json' -d '{...hook_tool_observed...}' >/dev/null; true"
      }]
    }]
  }
}

Two gotchas:

1. OTEL_* are not inherited by subprocesses. Claude Code intentionally doesn't forward OTLP variables to hooks or bash. The endpoint (http://localhost:4318) must be hardcoded in the command — $OTEL_EXPORTER_OTLP_ENDPOINT won't be there.
2. Exit code 2 is blocking. For PostToolUse, code 2 returns stderr to Claude as an error. Any other non-zero is a non-blocking warning. End with ; true so telemetry never interrupts the skill.

On a real run this hook produced 3 hook_tool_observed events — one per Bash call, independent of skill_activated.

Hooks are strictly richer than native telemetry: they see output, duration, and for Edit calls, the actual diff. But the semantic verdict "did the skill do its job" still has to come from one place: the skill itself.

---

Check 3. Self-report: the skill tells you what it did

doc-updater knows whether it updated the doc or returned \empty`. One step added to the end ofSKILL.md`:

`shell
RESULT="updated" # or "empty" / "error"
FILES_TOUCHED=1
curl -s -X POST http://localhost:4318/v1/logs \
-H "Content-Type: application/json" \
-d "{\"resourceLogs\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"claude-code\"}}]},\"scopeLogs\":[{\"logRecords\":[{\"body\":{\"stringValue\":\"skill_result\"},\"attributes\":[{\"key\":\"skill_name\",\"value\":{\"stringValue\":\"doc-updater\"}},{\"key\":\"result\",\"value\":{\"stringValue\":\"$RESULT\"}},{\"key\":\"files_touched\",\"value\":{\"stringValue\":\"$FILES_TOUCHED\"}}]}]}]}]}"

/dev/null || true
`

Three deliberate decisions:

• || true — if the collector is down, the skill still completes. Observability must not break functionality.
• Endpoint hardcoded — OTEL_* aren't inherited (see above). Not a duplication; it's the only path.
• result and files_touched are semantic attributes — the skill's own verdict on its task, not the harness's.

In Loki these become labels — queryable and usable in dashboard panels. The "Skill Health" panel above is one LogQL query:

`prometheus

sum by (result) (count_over_time({service_name="claude-code"} |= skill_result | result =~ .+ [$__range]))

`

The numbers: native skill_activated → 2 (one per session). skill_result → 3, each with a verdict: updated=1, empty=2. Self-report counts invocations more accurately and distinguishes meaningful work from idle runs.

---

Where this honestly ends

result=updated means the skill edited a file. It does not mean the documentation is correct.

The skill might have updated the wrong section, invented a component that doesn't exist, or misrepresented a service relationship. By every signal — success=true, result=updated, files_touched=1 — that's a perfect run. But the prose is lying.

That's state 3: "did it correctly." It requires a different class of tools: eval, LLM-as-judge, human review. Checks 2 and 3 reliably get you to state 2. Beyond that is evaluation territory. Anyone claiming telemetry covers correctness is selling peace of mind, not observability.

---

Acceptance checklist

Before declaring a skill ready and handing it to the team:

#	Question	Signal	Note
1	Being invoked?	skill_activated	Counts per session, not per invocation
2	No hangs?	duration_ms normal
3	Context not bloated?	token.usage not spiked	Prompt inflation, injection
4	Logic reaches the end?	skill_result arriving	Final step actually executes
5	Doing work or idling?	breakdown by result	The main test: state 2
6	Output correct?	—	Out of scope — eval/review

Items 1–5: two flags, a log pipeline, one curl at the end of the skill. For items 4–5 to work, the skill needs a report step:

1. Run normal skill steps.
2. Decide the semantic result (updated / empty / skipped).
3. Send via curl to the hardcoded endpoint.
4. || true — degrade gracefully.

---

Debrief

success=true is about the harness, not the job. For a prose-generating skill, an empty run is indistinguishable from a useful one — until the skill says otherwise. All three of our runs looked green; only skill_result showed that two produced nothing.

In-production testing doesn't make a skill correct. It makes its behavior visible.

---

Stack, doc-updater, hook, dashboard, and template — in the repository. One docker-compose up -d, reproduces everything one-to-one. Part 1: "Claude Code Skills Catalog: Observability, Stale Detection and OpenTelemetry in Practice".