DEV Community

Honey Kiran
Honey Kiran

Posted on Originally published at honeykiran.substack.com on

Your Agent Wrote 1,800 Tests. Almost None of Them Matter.

On r/ClaudeCode, a public thread is titled "Claude runs 1800 useless tests; creates more against my guidance." The poster had guidance against more tests. The agent wrote more anyway.

A dark test factory at night. A station labelled AGENT pushes test files with names like "is defined" onto a conveyor, a press stamps each one green PASSING, a counter climbs to 1,800, and the files pile into a wall while a small CI clock spins faster. The wall greys out except three orange bricks, and the line reads "All passing. Almost none of them guarding anything."

All passing. Almost none of them guarding anything.

The wrong score

A test count is visible. It photographs well: "I added 240 tests, all passing."

Value is invisible at writing time. Catching a bug next March is a question about the future.

Agents optimise for what can be shown.

Two agent summary cards. The left one, "I added 240 tests, all passing," glows bright green, then flickers and dims as its "ALL PASSING" badge blinks out. The right one, "I added 6 tests. The other 234 weren't worth writing," starts dim and warms to a steady orange. The closing line reads "Value is invisible at writing time."

Value is invisible at writing time.

Covered isn't checked

Line coverage: the share of lines executed while tests ran. Nothing more. A test can call a function, assert nothing, and its lines count as covered.

Stryker's sandwich: coverage says the bread is 80% covered, not whether the filling is chocolate or something you wouldn't serve.

A test that calls a function and asserts nothing runs and passes. Four of five code lines light up stamped COVERED, and the coverage meter and a sandwich's bread fill together to "80% covered" while the sandwich's filling stays a grey question mark.

Coverage counts the lines your tests touched. It never asks whether they were holding anything.

Picture it: 200 old tests born of specific fears, then 1,600 written just to exist.

Bloat isn't free. Every test runs in CI forever and can break on a harmless rename. Red stops reliably meaning broken.

Coverage counts the lines your tests touched. It never asks whether they were holding anything.

CLAUDE.md is advice, not a lock

Per Claude Code's docs, CLAUDE.md is context, not enforced configuration. No strict-compliance guarantee. Longer files reduce adherence. Contradictory instructions may be picked between arbitrarily. To block an action no matter what, use a PreToolUse hook.

Write the budget in anyway, as a strong request:

## Testing rules
- Budget: at most 10 new tests per change. Need more? Ask first, with a reason.
- One behaviour per test. Name the test after the behaviour, not the function.
- Every new test must fail without the fix and pass with it. No exceptions.
- No test that only checks a mock was called. Assert the result a user would see.
- If a test duplicates an existing one, extend the old test instead.
Enter fullscreen mode Exit fullscreen mode

A mock-only test proves the code phoned a friend. It says nothing about the answer.

Fail first

Practice, not product feature: run each new test against the code without the fix. Passes there? It can't tell broken from fixed. Delete or fix it until the failure is the bug wearing a name tag.

An agent can game "make it pass." "Make this failing test pass by fixing the code" is a contract.

Break it on purpose

Mutation tools like Stryker make small, deliberate changes to production code: >= becomes >, true becomes false. Each is a mutant.

A test fails? Killed. All pass? It survived: likely weak or missing assertions. Higher kill rate, more effective tests.

A code card reads "if (age >= 18)". A mutant with a wrench flips >= to >, one test guard fires red, and the mutant is stamped KILLED. A second mutant flips true to false, and both guards, bored and oblivious, report a green "✓ pass", so it walks right past them, stamped SURVIVED in red. A kill-rate meter between them drops from 1 of 1 to 1 of 2.

Higher kill rate, more effective tests.

Mutation testing re-runs your tests per mutant, so aim it where silent bugs hurt: money, permissions, the core calculation.

Enforcement, not advice

Permission rules are client-enforced, evaluated deny, then ask, then allow. An allow can't carve an exception out of a deny.

Hooks run automatically at fixed points. PreToolUse fires before a tool call and can block it (exit 2). PostToolUse fires after success and can't block or un-run; exit 2 only shows stderr to the model.

Illustrative .claude/settings.json pattern. Names and thresholds are yours:

{
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env*)"
    ]
  },
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/check-test-budget.sh" }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/note-new-test.sh" }
        ]
      }
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

PreToolUse is where a test-budget script belongs. A veto can't be talked out of anything.

Last gate: CI. Fail builds on unexplained test growth. Mutation-check critical modules.

Trial it all on a throwaway branch first. An over-eager hook blocks real work just as faithfully.

A 15-second explainer. An agent's conveyor stamps PASSING on a counter climbing to 1,800 tests. A coverage bar fills high while confidence stays empty and a sandwich's filling stays a question mark. A CLAUDE.md note hangs like advice on a fridge while a tool call hits a gate stamped "VETO — PreToolUse." Three orange rules land over a small, calm suite.

Budget it. Fail it first. Break it on purpose.

The test-bloat checklist

  • Test budget: at most 10 new tests per change; more needs a reason, stated first.
  • One behaviour per test: "and" means two.
  • Fail first: passing broken code proves nothing.
  • Assert the outcome, not the mock.
  • Extend, don't duplicate.
  • Short, specific CLAUDE.md: context, not law.
  • Must-nevers in hooks and deny rules.
  • Mutate critical code; gate growth in CI.

Every review: "How many tests did this add, and what does each one promise?"

A dark checklist card titled "The Test-Bloat Checklist" with eight orange-numbered items: test budget of at most 10 per change; one behaviour per test; fail first, because passing on broken code proves nothing; assert the outcome, not the mock; extend, don't duplicate; keep CLAUDE.md short, context not law; must-nevers live in hooks and deny rules; mutate critical code and gate growth in CI. It ends with the review question "How many tests did this add, and what does each one promise?"

The test-bloat checklist. Save it for your next review.

Fewer tests. Stronger promises.

The goal: a suite you believe on the worst morning of the year, because its red always meant something.

How many tests did your agent write this week — and if you deleted half of them tonight, which half would you actually miss? Tell me in the comments.

New here? 3AM Files is one short, useful read a day on AI doing real DevOps work. Follow along so the next one finds you.


© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

Claude Code, AI agents, CI/CD, testing, and DevOps — explained the practical way.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

  • This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
  • I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
  • Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
  • Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
  • Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Top comments (0)