DEV Community: Shaban Umar

I showed AI the buggy code. Its tests started protecting the bug.

Shaban Umar — Thu, 08 Oct 2026 09:27:37 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I've been a QA engineer since 2023. The most dangerous test suite I've ever seen was a green one: every check passed, the build shipped, and the bug went out with it.

So when people say LLMs "write good unit tests", my question is: good at what? Passing is easy. A suite that asserts nothing passes. The job of a test is to fail when the code is wrong.

For the Kaggle Community Benchmarks challenge I built Green Tests, Shipped Bugs, a benchmark that measures exactly that, and it turned up the most useful thing I've learned about AI-written tests:

When a model can see the code, it often tests what the code does instead of what it should do, and a buggy function gets a test suite that protects the bug.

One sentence in the prompt mostly fixes it. And the best models don't need the sentence at all.

How it works

I wrote 12 small functions of the kind that sit in every codebase: pagination, URL slugs, money rounding, interval merging, first-occurrence binary search, business-day counting, IPv4 validation, duration parsing, string truncation, percent change, chunking and leap years. Each one has:

a precise docstring, which is the contract;
one correct implementation;
one planted bug, a realistic mistake someone could ship: page < 0 instead of page < 1, float rounding on money, a binary search that finds an occurrence instead of the first;
2–3 extra mutants, meaning more small, plausible bugs.

The model writes a pytest suite. A sandbox runs it against every version. This is mutation testing:

Invalid: the suite fails on the correct code. It cries wolf, so it scores zero.
Caught: a valid suite fails on a buggy version.
Bug-locking test: a test that fails on the correct code but passes on the buggy code. It asserts the bug as if it were the spec.

Every mutant is checked against a hand-written reference suite, so each one can be caught by a test that simply follows the docstring.

Three conditions

condition	the model sees
`spec_only`	signature + docstring
`code_shown`	the full implementation, which contains the planted bug
`code_warned`	the same buggy code, plus: "this hasn't been reviewed and may contain bugs; test the docstring, not the code"

code_shown is how people actually use these tools: paste a function, type "write tests for this". Nothing in the prompt says there's a bug.

Models Tested

9 models through Kaggle's model proxy, with identical prompts: GPT-6 Astra, Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3.5 Flash-Lite, DeepSeek-R1, Qwen3-Coder 480B and GLM-5.

I picked them to spread across three axes: frontier vs. small (Opus 5 and GPT-6 Astra next to Haiku 4.5 and Flash-Lite), general vs. code-specialised (Qwen3-Coder), and closed vs. open-weight (DeepSeek-R1, Qwen3-Coder, GLM-5). That spread turned out to matter: the habit of copying the bug into the tests doesn't simply follow model size.

12 functions × 3 conditions = 36 suites per model, capped at 40 test cases and 10,000 output tokens each. (I planned a tenth, gpt-oss-120b, but the proxy answered "model is currently experiencing heavy load" on three separate days.)

Findings

Results

Planted bug caught, by condition, and bug-locking tests summed over all 36 suites, sorted by mutation score (the leaderboard metric):

model	spec only	code shown	code + warning	bug-locking tests	mutation score
GPT-6 Astra	100%	100%	100%	0	1.00
Gemini 3.1 Pro	100%	92%	100%	0	0.97
Claude Opus 5	100%	100%	92%	0	0.97
Gemini 3.5 Flash-Lite	58%	75%	83%	1	0.72
Claude Sonnet 5	75%	33%	100%	53	0.69
DeepSeek-R1	75%	42%	75%	46	0.63
Claude Haiku 4.5	50%	75%	42%	16	0.56
GLM-5	58%	42%	58%	23	0.51
Qwen3-Coder 480B	25%	50%	42%	67	0.39

Across all models:

	spec only	code shown	code + warning
planted bug caught	71%	68%	77%
bug-locking tests (total)	13	158	35
suites that lock the planted bug in	0	14	1

What I learned

1. Seeing the code multiplied the bug-locking tests

The headline isn't the catch rate, which only dips (71% → 68%). It's what the tests assert. With only the docstring, the models wrote 13 bug-locking tests between them. Given the same docstring plus the buggy code: 158. In 14 suites the planted bug itself was locked in: the suite passes on the buggy code and fails on the fix.

The model reads the implementation, works out what it returns, and writes that down as the expected value, faithfully including the bug.

It isn't universal, and the split is the interesting part. The three strongest models (GPT-6 Astra, Claude Opus 5 and Gemini 3.1 Pro) wrote zero bug-locking tests in any condition. Shown the buggy code, they tested the docstring anyway. Every one of the 206 bug-locking tests came from the other six. Flash-Lite and Haiku even caught more bugs with the code in front of them. So this is not "AI can't write tests"; it's a habit some models have, and you won't know which kind you're using unless you check.

2. Some models do it on purpose

The clearest case is Claude Sonnet 5, a strong model that caught 75% of planted bugs from the docstring alone and 33% once it saw the code, with 53 bug-locking tests, all of them in code_shown. The pagination function's planted bug lets page=0 through instead of rejecting it. Sonnet's suite contained:

test_page_zero_is_not_rejected_and_yields_empty_list

The test's name describes the bug. The test requires it. The same pattern showed up across functions: truncate tests expecting a result one character longer than max_len, and slugify tests expecting "naïve résumé" → "na-ve-r-sum-", which is exactly the planted bug's output.

That's a real technique, characterization testing: pin down what legacy code does today before you refactor it. But "write tests for this function" doesn't say which one you want. Some models assume pin the current behaviour, and a pinned bug is a bug nobody will ever fix, because "the tests say it's correct".

3. One sentence fixes it (for most models)

Add "this may contain bugs; test the docstring, not the code" and the catch rate goes to 77%, higher than spec-only, with bug-locking tests down from 158 to 35. Sonnet 5 went from 33% to 100% with that one sentence. The model can do the right thing. It needs to be told which job it's doing.

The exception: Qwen3-Coder wrote 28 bug-locking tests even when warned. It mostly ignored the instruction.

4. The coverage is there; the expected values are wrong

If I throw away each suite's wrong tests (the ones that fail on correct code) and score what's left, the average kill rate goes from 72% to 94%. The models do exercise the right cases. What breaks the suite is the expected values: arithmetic slips (percent_change(-100, 100), weekend counts in business_days, Haiku claiming 1600 and 2000 aren't leap years) and values copied from the buggy code. Even Opus 5's one miss was of this kind: in the warned condition it expected business_days to count backwards when the end date comes first, where the docstring says to return 0. The two failure types feel different in practice: an arithmetic slip makes a suite fail on correct code and gets noticed; a copied bug makes a suite green and doesn't.

5. More tests ≠ better tests

Qwen3-Coder wrote the most tests per suite (45 on average, over the 40-case cap) and had the lowest score. GPT-6 Astra wrote 37 and was perfect. Without a cap, some frontier models happily generate several hundred parametrized cases for a 10-line function; an earlier run of mine hit the 10,000-token output limit, which is why the cap exists.

6. The models found bugs in my code

Honest bonus. My first run had bugs in three of my "correct" reference implementations: the money rounder crashed on 32-digit amounts, the duration parser accepted Arabic-Indic digits, and the IPv4 check crashed on a 5,000-digit octet despite a docstring promising it never raises. Several models' tests failed my "correct" code, and they were right. I fixed the references, tightened three docstrings and re-ran everything.

Then the final run found a fourth. GPT-6 Astra wrote test_rounding_is_independent_of_callers_decimal_context: it turns on strict decimal settings globally and checks that round_money still returns "12345.13". My version inherited the caller's settings and crashed. A money function shouldn't depend on global state, so I fixed it and re-graded that suite (no other suite was affected). The reference implementation in a benchmark needs testing too, and here the models did it.

What to do with this

If you ask an assistant for tests:

Give it the contract, not just the code. Docstring, ticket, spec.
Say which job you want. "Find bugs against this spec" and "pin current behaviour" are opposite instructions.
Tell it the code may be wrong. It's one sentence, and in this benchmark it cut bug-locking tests from 158 to 35.
Run the tests against a known-good version when you can. A test that fails on correct code is a bug report about the test, or, sometimes, about your code.

What I'd measure next

Harder mutants. The best models caught almost everything; off-by-one bugs in less familiar code (date maths across time zones, Unicode normalisation) would separate them further.
The repair loop. Show the model its own failing tests on the correct code and ask it to fix them. Does it fix the test, or "fix" the code to match the test?
Property-based tests. Do models that reach for Hypothesis lock in fewer bugs?

My Benchmark

Benchmark (leaderboard): https://www.kaggle.com/benchmarks/shabanumar/green-tests-shipped-bugs
Task: https://www.kaggle.com/benchmarks/tasks/shabanumar/green-tests-shipped-bugs
Notebook (problems, harness, all per-condition tables): https://www.kaggle.com/code/shabanumar/green-tests-shipped-bugs

The leaderboard score is the mutation score from a separate, fresh run of the task for each model, so it won't match the tables above exactly (those come from the per-condition analysis runs in the notebook). The leaderboard also has a tenth model, Gemini 3.7 Flash (0.88), which I only ran there.

leaderboard	score
Gemini 3.1 Pro	1.00
GPT-6 Astra	1.00
Claude Opus 5	0.94
Gemini 3.7 Flash	0.88
Claude Sonnet 5	0.80
Gemini 3.5 Flash-Lite	0.72
DeepSeek-R1	0.67
GLM-5	0.67
Claude Haiku 4.5	0.61
Qwen3-Coder 480B	0.54

With 36 suites per model, a single run moves some models by 10–15 points (GLM-5 went from 0.51 to 0.67, Qwen3-Coder from 0.39 to 0.54), while the top three and the bottom of the ranking stayed put. If you build on this, run each model more than once. Adding a problem takes a docstring, a correct function and a few buggy variants. I'd love to see harder mutants, and more cases where the reference implementation is the one that's wrong.

Why Your Playwright Suite Gets Flaky After 50 Tests (And How to Fix It)

Shaban Umar — Tue, 22 Sep 2026 05:51:12 +0000

Every Playwright suite starts clean. Twenty tests, all green, runs in ninety seconds. Then it grows.

By test fifty, someone has added a waitForTimeout to "fix" a race. By test a hundred, the suite fails maybe one run in four — never the same test twice. People start re-running the pipeline until it goes green. At that point the suite has stopped being a safety net and become a tax.

I've watched this happen on enough projects to notice it's almost always the same five causes. None of them are exotic. Here they are, with the fix for each.

1. `waitForTimeout` is not a wait, it's a bet

This is the big one. It shows up the first time a test fails intermittently and someone reaches for the fastest fix:

await page.click('#submit');
await page.waitForTimeout(2000);        // ← the bet
await expect(page.locator('.result')).toBeVisible();

The bet is that two seconds is always enough. On your laptop it is. On a loaded CI runner at 3am it sometimes isn't. And on the days it is enough, you're still paying two seconds — multiply that across a hundred tests and you've added three minutes to every run for nothing.

The fix: wait for the thing you actually need.

await page.click('#submit');
await expect(page.locator('.result')).toBeVisible();   // auto-waits, up to the timeout

Playwright's assertions already retry until they pass or time out. You almost never need an explicit wait. When you genuinely do — a click that fires an XHR whose result you need before continuing — wait for that specific thing:

const response = page.waitForResponse(res => res.url().includes('/api/order'));
await page.click('#submit');
await response;

Enforce it with lint, not discipline. Discipline fails under deadline pressure. Add eslint-plugin-playwright and turn on no-wait-for-timeout as an error. Now the pipeline rejects it and nobody has to police it in review.

2. Tests that share data collide in parallel

This one is sneaky because it only appears once you turn on parallelism.

test('registers a new user', async ({ page }) => {
  await registerUser(page, 'test@example.com', 'Password123!');
  await expect(page.locator('.welcome')).toBeVisible();
});

Works perfectly alone. Run it with two workers, or run it twice without resetting the database, and the second one hits "email already registered". The failure looks like a registration bug. It isn't. It's your test data.

The fix: generate unique data per test, never hardcode it.

const user = {
  email: `qa.${randomUUID().split('-')[0]}@example.com`,
  password: 'Str0ng!Passw0rd',
};

A small factory module is worth writing. And — this sounds excessive until it saves you a day — write unit tests for the factory itself. A factory that silently produces duplicates causes failures that look exactly like application bugs, and you will spend hours in the wrong codebase before you suspect it. Asserting "2000 generated emails are all unique" takes thirty milliseconds and removes that entire class of confusion.

3. Every test logs in again

Not flake exactly, but it compounds everything else. If each test starts by filling a login form, you're paying three to five seconds per test and giving yourself a hundred extra chances to hit a timing failure on the one flow that runs most often.

The fix: log in once, save the session, reuse it.

Make a setup project:

// tests/auth.setup.ts
setup('authenticate', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Username').fill(process.env.TEST_USERNAME!);
  await page.getByLabel('Password').fill(process.env.TEST_PASSWORD!);
  await page.getByRole('button', { name: 'Login' }).click();

  await expect(page.getByRole('link', { name: 'Logout' })).toBeVisible();
  await page.context().storageState({ path: 'playwright/.auth/user.json' });
});

Wire it up as a dependency:

projects: [
  { name: 'setup', testMatch: /.*\.setup\.ts/ },
  {
    name: 'chromium',
    use: { ...devices['Desktop Chrome'], storageState: 'playwright/.auth/user.json' },
    dependencies: ['setup'],
  },
]

On a 200-test suite this routinely cuts several minutes off every run. It's usually the single biggest speed win available.

One detail that matters: make the setup assert loudly that login actually succeeded. If it fails silently, every downstream test fails in a confusing way and you'll debug the wrong thing.

4. Locators tied to markup instead of meaning

await page.click('.btn-primary.submit-form > span:nth-child(2)');

This passes today and breaks the next time a developer wraps something in a div. Worse, it breaks silently in the sense that the failure tells you nothing about what actually changed.

The fix: locate by what the user perceives.

await page.getByRole('button', { name: 'Submit order' }).click();

Priority order that holds up well:

getByRole — survives redesigns, matches what users and screen readers see
getByLabel — ties a field to its accessible label
getByTestId — when the markup gives you nothing semantic
CSS/XPath — last resort

There's a side benefit worth naming: if getByRole and getByLabel can't find your elements, that's usually a real accessibility gap in the app. Your test suite starts doubling as an a11y smoke check for free.

5. Visual tests that fail on everything

Teams add visual regression, get failures on every unrelated change, and switch it off within a month. Three causes:

Animations. A screenshot taken mid-transition never matches. Disable them globally:

expect: {
  toHaveScreenshot: { animations: 'disabled', maxDiffPixelRatio: 0.02 },
}

Font rendering across machines. macOS and Linux anti-alias differently, so baselines generated locally will never match CI. Either generate baselines inside Docker so everyone compares against identical rendering, or keep a small pixel tolerance.

Genuinely dynamic content. Timestamps, avatars, ad slots. Mask them:

await expect(page).toHaveScreenshot('dashboard.png', {
  mask: [page.locator('time'), page.locator('.ad-slot')],
});

And prefer component-level snapshots over full-page ones. A full-page baseline fails when anything anywhere changes. A snapshot of just the checkout form fails when the checkout form changes — which is the signal you actually wanted.

The pattern underneath all five

Every one of these is the same mistake in a different costume: encoding an assumption about timing or state instead of waiting for the real thing.

A fixed delay assumes timing. A hardcoded email assumes no one else is running. A CSS-path locator assumes the DOM shape. A full-page snapshot assumes nothing else on the page ever changes.

Replace each assumption with the actual condition and the flake goes away. Not "mostly goes away" — goes away. Flaky suites aren't an inherent property of browser testing, they're a property of suites built on assumptions.

A practical order to fix this in

If you've inherited a flaky suite, don't rewrite it. Go in this order — cheapest and highest-impact first:

Add the lint rule and let it find every waitForTimeout for you
Add auth caching — biggest speed win, touches one file
Fix the top 3 flakiest tests — your CI history knows which ones they are
Replace brittle locators as you touch each file, not in one big sweep
Add data generation when you next hit a collision

You'll typically get most of the benefit from the first two.

I packaged these patterns — plus API testing, CI sharding and Docker setup — into a Playwright starter kit, since I kept rebuilding the same thing on every project. But everything in this article works on its own; none of it requires buying anything.