DEV Community: Paul Newell

Coverage Lied. Mutation Testing Told the Truth.

Paul Newell — Tue, 30 Jun 2026 12:30:00 +0000

When AI writes both the code and the tests, coverage only proves the same blind spot was executed twice.

Every test passed. Nothing was tested.

AI wrote 20 tests for a Laravel controller.

Every endpoint was covered. Every test passed. The coverage report looked healthy.

Then I ran mutation testing.

95 mutants generated. 68 killed. 27 survived.

The tests ran the code. They did not test it.

The new failure mode: shared blind spots

Here's the old setup: a developer writes the code, developer/QA writes the tests. Bugs live in the gap between two mental models, and that gap is doing useful work. Two people, two sets of assumptions, and the disagreements are where the bugs surface.

That's changing. More and more, AI writes both the code and the tests, and the more of your workflow you hand it, the more this bites. Same model, same assumptions, same missed cases.

This is the real failure mode, and it's worse than "AI writes bad tests." The agent writes tests that agree with its own misunderstanding. Writing code forces you into the weeds, where you trip over the blind spots. Reading code skims past them, so the reviewer inherits the agent's assumptions without ever discovering they were assumptions.

The test suite agrees with the bug.

Mutation testing is fifty years old (Lipton 1971, DeMillo–Lipton–Sayward 1978) and there are good primers on the basics elsewhere. This post isn't one. It's about what the technique reveals now that it couldn't before: not human laziness, but agent overconfidence.

The experiment

I'm using PestPHP in a Laravel backend as it is what my current test application is written in but Mutation testing exists in every major language: mutmut for Python, Stryker for JS/TS, PIT for Java. The lesson and the survivor categories generalise.

A real Laravel 12 legacy controller. Cyclomatic complexity 13 (CC 13). The agent generated 16 Feature tests and 4 Unit tests, all passing. Then:

--covered-only restricts mutations to lines the tests actually run. So a survivor is a covered line whose assertions wouldn't notice it breaking, not a line no test ever reached. That separates the two failures coverage smears together: code you never exercised, and code you exercised without checking. This score only judges the second. It assumes coverage is already high and asks the harder question on top.

95 mutants generated. 68 killed. 27 survived.

Pest prints each mutation line by line with the mutator that generated it. A tick means the test suite noticed the change. A cross means it survived.

Each row is one transformation applied to the source. TernaryNegated flips a ternary's condition, IfNegated inverts a conditional, SmallerToSmallerOrEqual swaps < for <=. The survivor on line 25, where < became <= and no test failed, is exactly the date-equality blind spot below.

What survived

Category	Count	%
Response shape (`RemoveArrayItem`)	11	41%
Boundary equality (`<=` / `<` / `>` / `>=`)	7	26%
Logic inversion (negations, boolean and/or, `===`/`!==`)	6	22%
Loop control flow (break/continue/foreach)	3	11%

Before running it I made a prediction. The survivors would cluster in the hidden closure branches phpmetrics caught and I'd missed. (phpmetrics counted CC 13. I'd hand-counted 10. The three branches I missed were a ?? default, a short-circuit, and a closure inside ->map().)

I was wrong. The closure had two survivors. Out of twenty-seven.

The real gaps were boring: response bodies, date equality, loop cardinality. Places I'd skimmed because they looked routine, which is exactly where the agent skimmed when it wrote the tests.

What the survivors teach

1. Status code assertions are not contract tests.
assertStatus(422) passes whether the body is ['error' => '...'] or []. Use assertExactJson or assertJsonStructure on error responses. This single change kills 11 of 27 mutants here for free.

2. Date logic needs equality boundaries.
The agent wrote past, future, and "clearly active now" cases. It never wrote $end == $now. Equality on continuous values feels artificial, and that's exactly why it gets missed.

3. A loop with one fixture is not really tested.
With N≤1, break and continue are indistinguishable. Every foreach whose order matters needs a fixture with at least two items, ordered to discriminate.

The agentic feedback loop

Coverage becomes smoke. It tells you the agent didn't forget to write tests. Nothing more.

Mutation testing becomes the feedback loop. Run it on the diff, before review. The agent isn't bad at writing tests, it's bad at knowing which tests matter. Mutation testing tells it, mechanically, which ones do.

So I asked the agent to fix it. Same controller, same prompt, but this time with the surviving mutant list as input.

Metric	Before	After
Tests in suite	20	40
Mutants generated	95	106
Killed	68	95
Survivors	27	11
Mutation score	71.58%	89.62%

Doubling the test count took survivors from 27 to 11. Note the mutant count went up: the extra fixtures covered more lines, so Pest generated 106 mutants this run instead of 95. A bigger surface, killed harder. Every category dropped except the corners.

Where QA judgement still matters

The remaining 11 were not all worth chasing.

Seven of the eleven I judged to be equivalent mutants: syntactic changes that don't actually alter behaviour. Removing the 'date' rule from a Laravel validator that already has 'after_or_equal:today' is one of them. Non-dates still fail, because after_or_equal:today needs a parseable date too. Detecting equivalence is undecidable in general, so the tool can't rule these out for you. You read each one and make the call. You flag it and ignore it.

The other four are fixture-cost prohibitive. Killing them is technically possible but expensive. One survivor changes $start > $now to $start >= $now inside a loop. The only test that distinguishes them is a violation starting at exactly now(): frozen time, a multi-item fixture ordered so this case fires first, fragile under clock or timezone drift. Fifteen lines of setup for a one-character mutation that triggers at one specific second.

That's the QA judgement call. Mutation testing surfaces the gap. It does not decide whether the gap is worth closing.

90% is the honest stopping point. High enough to trust the suite, low enough to admit which mutants are intentionally left alive.

A practical `/mutate` command

You want this to run between the agent's first test pass and your PR review. Split it into two pieces: the work, and the stopping condition.

The work is a slash command. It describes a single pass:

Run pest --mutate on the changed classes.
Categorise survivors. Add a test for each.
Print the score and surviving mutant list.
Do not weaken the production code to kill mutants.

The last line is doing real work. The easiest way to kill a mutant is to delete the code it lives in, so the command isn't allowed to touch production code, only tests.

But a procedure can't teach taste. "Add a test for each survivor" is the step. How you kill the mutant is the part that matters. A response-shape survivor wants assertExactJson, not another assertStatus. A loop survivor wants a two-item ordered fixture, not a second assertion on the same one. I keep that knowledge in a pest-code-writer skill, so every pass writes tests that kill mutants the right way, not the easy way.

That split is what makes the loop worth trusting. The skill knows how to write the test, /mutate runs the pass and holds the guardrails.

The condition belongs in /goal, which exists for exactly this. You give it a completion condition and the model checks after every turn whether it's met:

/goal mutation score ≥ 85% or only equivalent mutants remain
/mutate

The catch: the evaluator only sees the transcript. That's why the command prints the score and the survivor list, if the number isn't on screen, the goal can't see it.

Closing

Mutation testing pressures the tests, not the spec. Every mutant you kill makes the suite a tighter net around today's behaviour, and says nothing about whether today's behaviour was ever right. Kill every mutant on a misread requirement and you just get the wrong answer. That gap needs a human in the loop reading the requirement, not a higher score.

But in a world where AI writes both sides of the contract, "would anyone notice if this changed?" is the question coverage was pretending to answer and never could.

Coverage proves the code ran. Mutation testing proves it matters.

43 Years Later: Finishing My BBC Micro Game in Assembly

Paul Newell — Wed, 10 Jun 2026 12:30:00 +0000

In 1983, the first program I ever wrote appeared in the September issue of Computer & Video Games which was the pre-eminent UK games magazine of the day.

Computer & Video Games issue 23, September 1983

Inside that issue, in the type-in listings section, was my game:

Caterpillar

Like most type-in games of the era, it was pretty awful. Mine was doubly so as it was my first fumbling attempt at programming, written in BBC BASIC.

And like a lot of coders back then, I told myself I'd rewrite it properly in assembly one day.

This is that rewrite.

It's only 43 years late.

📖 The original

Back in the early 80s, publishing a game didn't mean uploading to GitHub or pushing to Steam.

It meant seeing your code printed across pages of a magazine and hoping someone, somewhere, would type it in correctly. And that was no easy feat, as anyone who ever tackled a multi-page listing will remember. One mistyped DATA statement and the whole thing fell over.

The Caterpillar type-in listing as printed in C&VG

The premise was simple: guide a caterpillar through a maze, collecting food and avoiding the poisonous mushrooms. About 110 lines of BASIC. It ran, it was playable-ish, and it was slow and I mean SLOW ...

The 1983 original in play. Jerky movement and painfully slow in BBC BASIC

💾 The constraints

The BBC Micro Model B gave you 32K of RAM, a 2 MHz 6502, and not much else. The original Caterpillar lived inside that, and the limits showed:

BASIC was the bottleneck. BBC BASIC was truly one of the best versions of the language but still pretty hopeless for action games.
Performance. Scrolling was very slow even using hardware calls.
Collision detection. I remember struggling to get this right

⚙️ The promise

Even as a kid, I knew what the "proper" version would need to be coded in:

6502 Assembly.

I just could not get to grips with it in those early days.

I did eventually become competent in 6502 a couple of years later but that was with the Commodore 64 (C64) and its hardware sprites which were way easier to get moving.

I never did get back to the BBC game.

🧠 43 years later

I knew my original game was hosted in the BBC Micro Archive and it always nagged me that I was never able to make the game I wanted back in the day.

So this wasn't just a translation. It was adding all the things I wanted to do on the first version but could not fit in.

6502 assembly handles the game. Sprites, scrolling, collision and the season cycle, in MODE 2 (16 colours), running on a vsync-locked 50Hz loop.

BASIC runs CALL &1900 to hand control to the engine; the engine plays the game, then returns a result code and the score in zero page for BASIC to display.

🔍 1983 vs 2026

Although they look similar, they are quite different under the hood.

Hardware scrolling, kept. Both versions scroll the easy way: a VDU 30, 11 at the top of the screen makes the MOS nudge the CRTC display-start registers, so the hardware does the work for almost free. The 1983 version already got this right, so the rewrite kept it. I did consider writing directly to the CRTC's R12/R13 registers itself instead of going through the MOS, games like Repton did this, but I backed out of that pretty quickly due to the added complexity.
Direct sprite blits. Each sprite is pre-encoded and written straight to screen memory, four bytes per scanline. No slow BASIC, no OS character calls in the path.
Ring-buffer collision. Items are tracked in a 24-slot ring as they scroll from the top of the screen down to the caterpillar's row, so a "hit" is just a position lookup, with no per-pixel scan.
Deterministic by design. The original just splatted random mushrooms on the screen but this one has a proper map like any other speed runner.
Score Attack. The new version finishes at 4 seasons. Score is everything - grab what you can.
Game mechanics. Hot streaks, bonus rounds and an easter egg 🥚

What I could not grasp in 1983 was that MODE 2 memory is laid out in character cells of 8 bytes. So stepping down one pixel row is usually just +1, until you fall off the bottom of a cell and have to leap +633 bytes to land on the next character row down:

.advance_scanline
    INC temp0
    LDA temp0 : AND #7        ; still inside this character cell?
    BNE as_same_row           ; yes next byte is just +1
    CLC                       ; no cross to the next character row
    LDA scr_addr_lo : ADC #LO(633) : STA scr_addr_lo
    LDA scr_addr_hi : ADC #HI(633) : STA scr_addr_hi
.as_same_row
    INC scr_addr_lo

That one odd number, 633, is the whole quirk of BBC screen memory in a nutshell. No wonder I struggled with this at the time and why I found C64 sprites so much easier.

The original was ~110 lines of BASIC. The rewrite is around 1,250 lines of lightly commented 6502, which assembles down to just 3 KB of machine code.

🎮 Seeing it run again

There's something uniquely satisfying about seeing old code come back to life.

Not really modernised. Just finished, the way I always meant to write it.

The 2026 rewrite — the same game in native 6502, fast and fluid sprites

🏆 Score to beat

Will Newell - 4605

⏳ Why this mattered

This was never really about optimisation.

It was about finishing something.

If I say I'll do something, I'll do it.

Some things just take a little longer than expected 😄

🔗 Links

BBC Micro Archive (the 1983 original): https://www.bbcmicro.co.uk/game.php?id=1066
GitHub: https://github.com/newell-paul/caterpillar-assembler
Play in the browser: https://newell-paul.github.io/caterpillar-assembler/

Claude Code Has a Power Meter. You Just Have to Wire It Up

Paul Newell — Tue, 02 Jun 2026 12:00:00 +0000

One bash script. Your session cost in dollars, live, so you stop burning Opus on work that doesn't need it.

I kept leaving sessions on Opus long after the hard thinking was done. Claude Code does warn you, but the context and rate-limit notices flash past and you dismiss them without reading. Discipline loses to a number you can't see.

TL;DR

Grab statusline-hud.sh from github.com/newell-paul/statusline-hud, drop it in ~/.claude/, chmod +x it.
Add a statusLine block to ~/.claude/settings.json (snippet below).

Wire it up

Prerequisites

jq on your PATH. The script hard-exits with ⚠ jq missing without it.
A terminal with 256-colour and UTF-8 support, and a font that renders 🔥 ⚡ ↺ ↩ ✗ ░ █ ▏▎▍▌▋▊▉. Any modern terminal with a Nerd Font, or the macOS/iTerm2/GNOME defaults, will do.
macOS or Linux. Native Windows isn't supported; the script is bash and uses POSIX tools (WSL should work, untested).

Install

Copy statusline-hud.sh to ~/.claude/ and make it executable.
Add this to ~/.claude/settings.json:

{
  "statusLine": {
    "type": "command",
    "command": "~/.claude/statusline-hud.sh"
  }
}

refreshInterval is optional. Leave it out and the bar redraws on Claude Code events: each new assistant message, after /compact, and on permission-mode or vim-mode changes. Add "refreshInterval": 5 if you want it to also tick on a timer, for example to keep the countdown accurate while background subagents are running.

The repo has 80 bats tests and a hardened git_safe() wrapper that disables core.fsmonitor and core.hooksPath, so a hostile .git/config can't execute code via the bar.

Settings reload; the bar appears on your next interaction.

The JSON hiding in the statusline

Claude Code's statusline is just a shell command: JSON in on stdin, one line of text out. Cost, effort, fast mode, cache stats, both rate-limit windows are all already in there, emitted on every render (debounced 300ms). Nothing's missing. There's just nothing showing it. (Tap the pipe with tee to see the full payload; PAYG (pay-as-you-go) and API users will find rate_limits absent.)

What the bar shows

With everything switched on:

Directory + git status (↑N↓N, ✗ if dirty)
Model name with effort badge (⚡Lo / ⚡Med / ⚡Hi / ⚡xHi / ⚡Max) and fast-mode rocket 🚀 when /fast is active
Context-window bar: 5 cells using eighths (▏▎▍▌▋▊▉█) so the bar moves smoothly within a tier, with the tier colour stepping green → yellow → orange → red (see the README for the exact ctx and rate-limit breakpoints; they differ)
5-hour and 7-day rate-limit bars (both shown by default; comment either out in SEGMENTS). A reset countdown (↺2h14m) attaches to whichever limit is more constrained, and only once it crosses 60%
Cache hit ratio (↩97%), only after >5k input tokens
🔥 cumulative session spend in USD (or input tokens; see toggle below)

Everything is configurable from the CONFIG block at the top of the script.

The SEGMENTS array is your control panel. Comment out a line to hide that segment, reorder lines to rearrange the bar. Want git status on the right and the flame on the left? Move the lines.

Other knobs you'll probably touch:

TURN_UNIT: usd (default) shows the flame in dollars; flip to tokens for input-token count instead. In tokens mode the value is the live context window (see the flame section below), so its thresholds are fractions of a 200k context, not session totals.
TIER_COLOR, BAR_CTX, BAR_LINEAR: bar palette and the tier thresholds at which each bar flips colour.
TURN_HI_USD / TURN_MED_USD: USD thresholds for the flame's amber/red (TURN_HI_TOK / TURN_MED_TOK for tokens mode).

One jq call and a few git invocations per render. Cheap enough to ignore.

The flame: a session-scale spend gauge

🔥 $5.64 is the cumulative session spend, read straight from cost.total_cost_usd. Green under $5, amber $5–$20, red ≥ $20, tuned for Max users where a long Opus session runs that high in estimated spend. Red doesn't mean stop. It means the next routine task is the one to drop to Sonnet for, or /clear and start fresh.

PAYG users paying list price should dial down: try TURN_MED_USD=0.50 / TURN_HI_USD=2.00.

The discomfort isn't financial, it's visible. Same psychology as a power meter: you don't read the number, you notice when it pegs into the red.

Green means I haven't checked the bar in a while and don't need to. Amber means the session has grown legs, fine if I meant it, worth noticing if I didn't. Red means I'm running Opus on a long, heavy context, and it's time to make a choice.

Acting on the red

Opus for architecture, complex debugging, ugly refactors. Sonnet for most everyday work. Haiku for routine commands.

On Pro and Max, cost.total_cost_usd is an estimate, not your bill: API list rates applied client-side, which can differ from what you're actually billed. It's still the best signal for what the flame answers, which is how much have I put through the model so far.

If you'd rather see input tokens, flip TURN_UNIT=tokens in the CONFIG block, but be aware that as of v2.1.132 total_input_tokens reflects current context window, not cumulative session totals, and can drop after a /compact. The default token thresholds (TURN_MED_TOK / TURN_HI_TOK) are tuned to that: fractions of a 200k context, so red means "context is filling up," not "expensive session."

For historical totals, /usage gives you the deeper view. The bar nudges you in the moment; /usage is the rear-view mirror.

Once the bar is in front of you, two habits move the needle:

Pin routine slash commands to model: haiku. Commit, lint, review-diff: none of these need Opus.
Use subagents for delegation. They get a fresh, isolated context window, not your full history.

When a session has done its job, /clear before the next task so you're not dragging a stale context forward; if you want to keep the thread, /compact reclaims the space instead. Then swap to the model the next task actually needs.

The data was always there. Now you can see it.

The repo is at github.com/newell-paul/statusline-hud. Issues and PRs welcome.

I Programmed an AI in 6502 Assembly - It Worked

Paul Newell — Wed, 06 May 2026 12:21:00 +0000

In 1975 the 6502 processed 8-bit values through memory and control flow. A Claude Code skill now uses the same mnemonics to process forge tickets through triage loops.

Paul Newell · Apr 2026 · 8 min read

I'll be honest, this started as a joke. I wanted to see if I could get Claude Code to understand 6502 assembly language.

The 6502 powered the Apple II, the Commodore 64, the BBC Micro, the NES. It had 56 opcodes, 64KB of address space and absolutely no business anywhere near a large language model.

So I built a Claude Code skill that maps its instruction set onto a modern triage-and-fix workflow. You write .s files. Claude executes them. Its response is also assembly. There's a zero-page. There's a stack. BRK means "commit and halt."

Same Verbs, Bigger Nouns

Forty years later I'm writing LDA to load issue 42, which arrives with a description, acceptance criteria, a set of labels, screenshots Claude can actually see with vision and a graph of linked PRs.

The verbs haven't changed. The nouns got about six orders of magnitude bigger.

The Smallest Useful Program

peek.s - 6502 looking very out of place in colour

Six opcodes, three flags in play. This is the heart of what the whole thing does:

It's not a real 6502, of course. I just borrowed the mnemonics.

If the tests pass (C=1), commit and halt. If they fail (C=0), branch to skip and return without committing. That's it. The BCC/BCS pattern carries over directly from the real 6502. Carry set means success, carry clear means failure.

The Trace Is the Reply

An 800-token prose response ("Let me analyse this issue. I'll start by examining the file at...") becomes a handful of lines of assembly trace.

Issue fetched, file fixed, tests passed, diff reviewed, committed, PR opened. No "Let me now...", no "I'll proceed to...", no "Here's what I did...". Just the trace.

That trace is structured. Diffable. Greppable. Re-runnable. Whether it costs fewer tokens than prose turned out to be more complicated than I expected, but the structural value is still there.

One Shot, One Fix

oneshot.s is the next step up: fetch an issue, analyze it, fix it, test it, self-review, and commit, with a single retry branch if the first attempt fails.

If the first fix passes tests and review, BRK commits immediately. If not, the retry branch gives it one more shot. If that also fails, RTS leaves the branch uncommitted for a human to pick up. The carry flag is doing exactly what it did on the original chip, routing control flow on a binary result.

oneshot.s - that's better, much easier on the eyes

How It Works: 15 Opcodes and 9 Vectors

The skill is called opcode and it defines a semantic DSL that borrows the 6502's mnemonics but maps them onto issue triage. The core ISA is just 15 opcodes. You can genuinely hold the entire thing in your head.

LDA loads an issue. STA persists to a slot. LDX/INX/CPX handle loops. JSR calls one of nine I/O vectors: FETCH, FIX, TEST, LINT, REVIEW, PUSH, PULL, CLONE, ANALYZE. BCC/BCS branch on test results. BRK commits and halts. PHA/PLA push and pop todos from a stack that maps directly to Claude Code's task list.

There's a zero-page with named slots. $00 is the current issue ID, $01 is the branch name, $02 is the current file, $05 is the commit message. The fetched issue queue lives at $10-$1F. The stack page at $0100-$01FF is the Claude Code task list.

Opcode Oriented Programming (OOP, obviously 🙂)

6502 mnemonics. Modern AI execution. Workflows as programs.

It talks to GitHub or GitLab through a forge abstraction layer so JSR PUSH opens a PR on GitHub or an MR on GitLab. Nobody has to compromise their vocabulary.

A Real Triage Session

This program loops through every issue labelled bug, tries to fix each one and only commits the ones where tests pass and the self-review is clean. Failed fixes get skipped. At the end it opens PRs for everything that worked. The PHA/PLA cycle maps directly onto Claude Code's task list so each pushed issue becomes a real tracked todo.

drain-the-swamp.s - in its preferred output format; complete with historically accurate code review comments

The Constraint Is the Feature

The 6502 aesthetic isn't just nostalgic decoration, it's a forcing function. When your output format is assembly you commit to a verb sequence before execution instead of wandering through chat mode. The resulting traces scan way faster than prose. They're diffable, greppable and versionable. You can commit a .s file alongside your code and re-run it on the next batch of issues.

The temptation to add "just one sentence of context" is the whole failure mode. Resist it absolutely.

The skill is pretty militant about this. Claude's response must be a valid .s program. No greetings, no sign-offs, no "Let me", no markdown headers. If it doesn't fit as an instruction, a directive or a semicolon comment it doesn't get said. There are escape hatches like .ASK for questions (≤60 chars), .NOTE for observations (≤80 chars) and .ERR for errors. But these are pressure-release valves, not invitations to prose.

The Extended ISA: For Completeness and Nostalgia

There's a full extended layer (~40 more mnemonics) you can opt into with .EXTENDED ON. Shifts become priority promotions (ASL A = "promote priority"). Logic ops become label-filter algebra (AND #imm = "intersect label mask"). BVS branches on merge conflict. Most of these are tagged as metaphor so they get narrated in the trace but not actually executed.

And for completeness there are six illegal opcodes behind .UNSAFE ON. The real undocumented 6502 ops like LAX, SAX and DCP. Pure flavour. Narrated, never executed, not even with the directive set. They exist simply because I could not resist adding them.

What about tokens?

I expected the tight assembly output to translate to substantial cost savings versus prose Claude. The story turned out more nuanced.

The reason is asymmetric overhead. For small, deterministic tasks, the orchestration overhead dominates the work — like running a CI pipeline to compile a single file. Opcode comes out ahead on larger programs with more decision points (multi-issue queue walks, longer reasoning chains where prose Claude would actually ramble).

Prose prompts invite interpretation. Assembly doesn't.

So Is This Actually Useful?

Maybe. if you're running the same triage pattern across many issues like scanning a backlog, fixing straightforward bugs and opening PRs then the token savings start making sense and the structured output is way tighter to read than endless TL;DR prose.

The .s files are reviewable artifacts.

The traces are audit logs.

And the constraint of planning your workflow as a sequence of opcodes before execution turns out to be surprisingly good discipline.

BRK

At the end of the day the question is the same one it's always been.

Are you a quiche eater or a Real Programmer?

Real Programmers don't need prose.

They need opcodes, a carry flag and a BRK at the end.

Commit the work. Halt the machine 🙂

Somewhere between JSR FETCH and BRK this stopped feeling entirely satirical

But if this article raised a smile or jogged some long-distant memories then please comment below.

The code

Full skill and example programs: github.com/newell-paul/opcode