"Is Claude nerfed" is unanswerable from vibes, and a 20-task nightly canary that costs about $27.60 a month on Sonnet 5.5, or $1.38 on Haiku 5.5, answers it with numbers. The Hacker News thread "Livenerf: Has Opus 5.5 been nerfed yet?" reached 924 points and 394 comments (item 49901736), and the loudest voices on both sides had no baseline. This post builds one: 20 fixed tasks, a pinned model and effort level, graders that run code instead of reading tone, a drift threshold, and a GitHub Action that runs it at 02:17 UTC every night. Then it shows how to tell a real regression from drift you caused yourself.
Why "it feels dumber" can't be tested
Every complaint in that thread is a comparison against a memory. Commenter jug wrote that "people sense nerfs more often than they happen and that it's often about honeymoon effects." Commenter stbenjam put it at "90% AI psychosis." On the other side, user3939382 ran a nightly fixed transcription task and reported that token counts "stay within 0.1%" while quota usage differed sharply, and added that "when it toggles it does so for days."
Both camps are describing real observations. Neither has the one thing that settles it: the same input, graded the same way, on a schedule.
Anthropic's own record is more specific than the Reddit version of events. Its April 23 postmortem lists three causes. Claude Code's default effort dropped from high to medium between March 4 and April 7. A caching change cleared old reasoning on every turn instead of once after an hour idle, March 26 to April 10. A verbosity instruction in the system prompt hurt coding quality, April 16 to April 20. The post says "We never intentionally degrade our models" and that the API "was not impacted."
Read that again as an engineer. All three regressions lived in the harness: defaults, caching, prompts. A canary that pins the harness version and the effort level would have separated them from model behaviour on day one.
What Anthropic actually promises
The models overview page states the contract plainly: "Every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on" (platform.claude.com, models overview). Sonnet 5.5 is claude-sonnet-5-5, Haiku 5.5 is claude-haiku-5-5, and the retirement floor for Sonnet 5.5 is September 28, 2027.
That is a claim you can test. If the snapshot is fixed and your canary moves, one of four things changed: the CLI, your prompt, your tools, or the serving stack. The canary tells you which by what else moved on the same night.
One limit: the CLI reference I checked has no temperature flag for claude -p. You pin what the CLI exposes, which is --model and --effort, and you absorb sampling noise with repeats and a threshold. More on that below.
Design the 20 tasks
Rules that keep the canary honest:
Every task has a machine grader. Exit code 0 means pass. No rubric, no LLM judge, no "looks right."
Every prompt is frozen text in git. Never edit a prompt without resetting the baseline.
Tasks are small enough to finish in under 8 agent turns.
Half the tasks should be ones the model passes reliably, so a drop is a signal. A task at 50% pass rate is a coin flip, not a sensor.
Tasks come from your real work, scrubbed. A canary on LeetCode tells you about LeetCode.
| Category | Tasks | Grader |
|---|
| Fix a failing unit test | 6 | Test runner exits 0 |
| Type-safe refactor | 4 | tsc --noEmit exits 0, tests still pass |
| Structured extraction | 4 | JSON validates against a schema, field values match exactly |
| Shell and regex one-liners | 3 | Output diff against a golden file |
| Long-context retrieval | 2 | Exact string match on a planted fact |
| Instruction adherence | 1 | Output contains no forbidden token, line count within limit |
Layout on disk:
canary/
run.sh
drift.sh
tasks/
01-fix-date-parse/
prompt.txt
fixture/ # starting files, copied to a temp dir per run
check.sh # runs inside the temp dir, exit 0 = pass
02-.../
results.csv
The runner
The flags below are all in the CLI reference: -p, --bare, --model, --effort, --max-turns, --max-budget-usd, --allowedTools, --no-session-persistence and --output-format json. The headless docs say --bare skips hooks, skills, plugins, MCP servers, auto memory and CLAUDE.md, and needs ANTHROPIC_API_KEY because it never reads your login. That is what you want. The model sees the task and nothing you accumulated.
#!/usr/bin/env bash
# canary/run.sh - usage: ./run.sh claude-sonnet-5-5
set -u
MODEL="$1"
ROOT="$(pwd)"
DATE="$(date -u +%F)"
CLI="$(claude --version | head -n1)"
mkdir -p "$ROOT/raw/$DATE"
for dir in "$ROOT"/tasks/*/; do
id="$(basename "$dir")"
for trial in 1 2 3; do
work="$(mktemp -d)"
cp -R "$dir/fixture/." "$work/"
( cd "$work" && claude --bare -p "$(cat "$dir/prompt.txt")" \
--model "$MODEL" --effort high --max-turns 8 \
--max-budget-usd 0.50 --allowedTools "Read,Edit,Bash" \
--no-session-persistence --output-format json \
> "$ROOT/raw/$DATE/$id-$trial.json" 2>/dev/null )
if ( cd "$work" && bash "$dir/check.sh" >/dev/null 2>&1 ); then ok=1; else ok=0; fi
cost="$(jq -r '.total_cost_usd // 0' "$ROOT/raw/$DATE/$id-$trial.json" 2>/dev/null || echo 0)"
echo "$DATE,$MODEL,\"$CLI\",$id,$trial,$ok,$cost" >> "$ROOT/results.csv"
rm -rf "$work"
done
done
Three things to notice. The CLI version goes into every row, because the April postmortem says harness changes were the culprit. Each task runs three times, which turns a 5-point swing per flipped task into a 1.7-point swing per flipped trial. And --max-budget-usd 0.50 caps a runaway loop; the docs warn that spend can pass the cap, so treat it as a brake, not a guarantee.
For the second arm, run the same loop without --bare from inside your real repo, so your CLAUDE.md, hooks and MCP servers load. Log it with a different label. That arm is your own configuration, and it is the one that usually moves.
A drift threshold that won't cry wolf
Take the first 14 nights as the baseline. With 20 tasks and 3 trials you get 60 pass/fail results a night. Say the baseline mean is 52 of 60, or 86.7%.
#!/usr/bin/env bash
# canary/drift.sh - exit 1 when the 3-night mean falls 8+ points under baseline
MODEL="$1"
awk -F, -v m="$MODEL" '
$2 == m { sum[$1] += $6; n[$1]++ }
END {
k = 0
for (d in sum) { dates[++k] = d }
# sort dates ascending
for (i = 1; i <= k; i++) for (j = i + 1; j <= k; j++)
if (dates[j] < dates[i]) { t = dates[i]; dates[i] = dates[j]; dates[j] = t }
if (k < 17) { print "baseline not ready: " k " nights"; exit 0 }
for (i = 1; i <= 14; i++) { bs += sum[dates[i]]; bn += n[dates[i]] }
for (i = k - 2; i <= k; i++) { rs += sum[dates[i]]; rn += n[dates[i]] }
base = 100 * bs / bn; recent = 100 * rs / rn
printf "baseline %.1f%% last3 %.1f%%\n", base, recent
if (recent < base - 8) exit 1
}' "$(dirname "$0")/results.csv"
Why 8 points over 3 nights? One night of 60 trials has a standard error near 4.4 points at an 87% pass rate, and three nights pooled shrink that to about 2.5. Eight points is roughly three standard errors. That is a heuristic, not a statistical guarantee, and sampling noise is real. Tune it after a month of your own data. If your baseline sits near 95%, tighten it.
The nightly GitHub Action
name: claude-canary
on:
schedule:
- cron: '17 2 * * *'
workflow_dispatch:
permissions:
contents: write
jobs:
canary:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- run: npm install -g @anthropic-ai/claude-code
- run: sudo apt-get install -y jq
- name: Run canary
working-directory: canary
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: ./run.sh claude-haiku-5-5
- name: Commit results
run: |
git config user.name "canary-bot"
git config user.email "canary@users.noreply.github.com"
git add canary/results.csv
git commit -m "canary: $(date -u +%F)" || true
git push
- name: Drift check
working-directory: canary
run: ./drift.sh claude-haiku-5-5
A failing drift check fails the workflow, and GitHub emails you. No extra alerting stack.
What it costs at October 2026 prices
Per the pricing page, Sonnet 5.5 is $2 per million input tokens and $10 per million output. Haiku 5.5 is $0.10 and $0.50 for prompts up to 100,000 tokens. My working assumption for a bare-mode task of up to 8 turns: 8,000 input tokens and 3,000 output tokens per run, no caching credit. Measure your own with the total_cost_usd column, and run a draft prompt through the token counter first.
| Setup | Tokens per night | Per night | 30 nights |
|---|
| Sonnet 5.5, 20 tasks, 1 trial | 160K in, 60K out | $0.92 | $27.60 |
| Haiku 5.5, 20 tasks, 1 trial | 160K in, 60K out | $0.046 | $1.38 |
| Sonnet 5.5, 3 trials each | 480K in, 180K out | $2.76 | $82.80 |
| Haiku 5.5, 3 trials each | 480K in, 180K out | $0.138 | $4.14 |
Run Haiku nightly and Sonnet weekly. Haiku is the cheap smoke detector. The AI model cost calculator handles the arithmetic for other mixes. For spend limits on the agent side of your bill, see the Haiku 5.5 100K-token rule.
Real regression or your own drift
The HN thread had its own version of this fight. Commenter throwitaway222 said apparent usage increases came from larger accumulated context, not a model change. Commenter phoghed read a Codex drop as a harness change because input tokens fell at the same time. Both are arguments about what else moved. Your two arms settle it.
| Bare arm | Your-config arm | CLI version | Likely cause |
|---|
| Flat | Dropping | Same | Your CLAUDE.md, hooks or MCP tools grew. Audit them. |
| Dropping | Dropping | Changed | Harness change. Pin the previous version and compare. |
| Dropping | Dropping | Same | Candidate model-side or serving-side regression. Rerun twice before filing. |
| Dropping | Flat | Any | Suspect the canary: a flaky grader or expired fixture. |
The first row is the common one. A CLAUDE.md that doubled in a quarter costs you tokens and attention every turn. Paste yours into the prompt bloat analyzer and see what is dead weight; the CLAUDE.md production rules show what a lean file looks like, and the Claude Code production pack covers settings and hooks that stay stable. Effort level is the other usual suspect: the effort parameter post explains why a changed default reads as a smarter or dumber model.
What to do with the data
Pin the CLI in production scripts and CI.
claude installaccepts a version number, per the CLI reference.When the drift check fires, attach the raw JSON for the failing tasks, the CLI version and the baseline numbers to your support report. A report with two passing runs and eight failing ones gets read. "It feels worse" does not.
Raise
--effortone level as a stopgap and watch whether the pass rate recovers; that tells you if the gap is reasoning budget.Do not edit prompts to chase one bad night. Reset the baseline only when you change tasks on purpose.
Add a task every time a real production failure teaches you something. Ship the failing case as task 21 and retire the flakiest old one.
Check the thread's own trackers (livenerf, Nerf Bench, Marginlab) as a cross-check, not a source of truth. Commenters there disputed each other's methods.
Quick answers
Does Anthropic change a model's weights without a new version?
Anthropic's models page says every Claude model ID is a pinned snapshot. Its April postmortem says "We never intentionally degrade our models" and attributes the March to April drops to Claude Code defaults, a caching bug and a system prompt edit, with the API unaffected.
Why use bare mode for the canary?
--bare skips CLAUDE.md, hooks, plugins, MCP servers and auto memory, so the run is the same on every machine. Your own configuration goes in the second arm.
Can I pin temperature?
Not through the CLI flags I checked. Pin --model and --effort, repeat each task three times, and set the drift threshold wide enough to absorb sampling noise.
How many tasks do I need?
Twenty detects a drop of about 8 points over three nights. Fewer than 12 and one flaky task moves the average more than most real regressions would.
Can an LLM judge replace the exact checks?
Not for a canary. A judge model can drift too, and then you cannot tell which model moved. Keep graders deterministic.
What did the METR evaluation say?
I could not find a published METR report for Opus 5.5 while writing this, only a secondary mention of pre-release testing, so nothing here relies on it.
Originally published at wowhow.cloud
Top comments (0)