DEV Community: nagaya-shion

The Bug Report Your Developers Will Actually Thank You For

nagaya-shion — Sat, 17 Oct 2026 02:06:46 +0000

I've spent 5 years in QA — 3 of them at Japan's largest software testing firm, managing test teams for major game publishers, and now as a QA director. In that time I've read thousands of bug reports.

Maybe 10% of them were actually useful on first read.

The other 90% triggered the same ritual: a developer reads the report, can't reproduce it, writes "works on my machine", and the bug ping-pongs between QA and dev for three days. The bug didn't cost three days. The report did.

Here's the template I enforce on every team I run, and the reasoning behind each field.

The template

## Title
[Area] One-line summary: what breaks, under which condition

## Environment
Device / OS / Browser + version / App build or commit
Account type used (if relevant)

## Steps to Reproduce
1. Numbered.
2. Exact.
3. From a clean state.

## Expected
What should happen (and WHY you believe that — spec link, design, or convention)

## Actual
What happens instead. Paste exact error text, never paraphrase it.

## Reproduction rate
n / m attempts (e.g. 4/5). "Sometimes" is not a number.

## Severity + why
High / Medium / Low — with one sentence of business impact.

## Evidence
Screenshot (annotated), screen recording, console/network log.

Nothing exotic. The value is in the details most reports skip.

The three fields everyone gets wrong

1. Reproduction rate: "sometimes" is not a number

"It sometimes fails" tells a developer nothing. "4/5 attempts with network throttled to Slow 3G, 2/5 on fast connection" tells them it's a race condition — before they've opened the code.

When I find a flaky-looking bug, I run it five more times and count. Those 10 minutes routinely save the developer half a day of guessing. The repro-rate line is the single highest-ROI sentence in the entire report.

2. Expected behavior needs a "because"

Weak: "Expected: works correctly."

Strong: "Expected: the second click is ignored, because the button should disable on first submission (and a double POST here means a double charge on checkout)."

The "because" does two jobs: it tells the developer where the bar is, and it tells the triage meeting why the bug matters. Severity debates usually die the moment the expected behavior has a reason attached.

3. Severity is a business statement, not a feeling

My rule of thumb:

High: blocks a core flow, loses data, or costs money (double charges, lost form submissions, broken checkout)
Medium: core flow survives, but a real user segment is hurt (an error message screen readers never announce, a layout broken on one device class)
Low: cosmetic, or an edge case with an easy workaround

Note that who is affected changes severity. A broken layout at 717px sounds Low — until you know 717px is exactly the half-open posture of foldable phones, a growing share of premium Android devices. (I test on a Motorola Razr for precisely this reason. The number of bugs living between Tailwind's sm and md breakpoints is remarkable.)

What "no bugs found" should look like

The most undervalued deliverable in QA: the honest empty report.

If I test a flow and find nothing critical, the client still gets a list of what was tested, how, on which devices, and what looks solid — plus any usability observations that aren't bugs. "We checked and it's fine" is informational. "We checked X, Y, Z in these 4 environments and found nothing" is evidence.

Teams that only reward found-bug counts end up with inflated Low-severity noise and no idea what's actually been covered. Coverage transparency beats bug-count theater.

A 30-second checklist before you hit send

[ ] Could a developer reproduce this without asking me anything?
[ ] Is the repro rate a number?
[ ] Does "Expected" contain a because?
[ ] Does severity state the business impact?
[ ] Is there at least one piece of visual evidence?

If all five are checked, your bug report is already in the top 10%.

I'm a QA director in Japan (ex-SHIFT, Japan's largest QA firm) and I offer manual QA & bug hunting as a freelance service — exploratory testing on real devices (Windows 11 / iPhone / foldable Android) with reports exactly like the template above. You can find me on Fiverr or at s-nagaya.vercel.app. My E2E automation work is over here: AI-driven test automation with Claude Code + Playwright.

I Let Claude Code Write My Playwright Tests for a Month. Here's What Actually Worked

nagaya-shion — Sat, 17 Oct 2026 02:03:33 +0000

As a QA director at a Japanese IT company, I spent the last months building E2E test suites with an unusual pair programmer: Claude Code, Anthropic's AI coding agent. Not for a toy project — for production QA pipelines.

Here's an honest breakdown of what AI is genuinely good at in test automation, where it fails, and the workflow that ended up sticking.

TL;DR

Claude Code is excellent at generating Playwright test skeletons from plain-language flow descriptions — roughly 70% of the boilerplate work disappears.
It is terrible (without guardrails) at selector strategy and flaky-wait hygiene. You need conventions it must follow.
The sweet spot: AI writes, human stabilizes. My rule: every AI-generated test gets run 5× locally and once in CI before it counts as "done".

The setup

Nothing exotic:

your-app-e2e/
├── playwright.config.ts   # 3 projects: desktop / mobile / foldable
├── tests/
│   ├── pages/             # Page Object Model
│   │   ├── ContactPage.ts
│   │   └── DashboardPage.ts
│   └── e2e/
│       ├── contact-form.spec.ts
│       └── dashboard.spec.ts
└── ci/github-actions-e2e.yml

The trick is in what you tell Claude Code before asking for tests. I keep a CLAUDE.md in the repo with three hard rules:

# Test conventions (non-negotiable)
1. Selectors: getByRole / getByLabel / [name=] attributes only.
   NEVER CSS class chains.
2. No waitForTimeout. Playwright auto-waiting + expect() polling only.
3. Every test must pass in isolation AND in parallel (no shared state).

With these three lines, the quality of generated tests jumps dramatically. Without them, you get page.locator('.css-1kxonj9 > div:nth-child(3)') and waitForTimeout(3000) — the stuff flaky suites are made of.

What AI-generated tests look like (real example)

I asked Claude Code: "Write tests for the contact form: success, server error, pending state. Never hit the real API."

It produced a Page Object with network stubbing:

/** Stub the contact API so tests never send real notifications. */
async mockContactApi(status: number, delayMs = 0): Promise<void> {
  await this.page.route("**/api/contact", async (route) => {
    if (delayMs > 0) {
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    }
    await route.fulfill({
      status,
      contentType: "application/json",
      body:
        status === 200
          ? JSON.stringify({ ok: true })
          : JSON.stringify({ error: "Submission failed." }),
    });
  });
}

And a pending-state test I honestly wouldn't have bothered writing by hand:

test("disables the submit button while the request is in flight", async () => {
  // 1.5s of artificial latency makes the pending state observable.
  await contact.mockContactApi(200, 1500);
  await contact.fillValidForm();
  await contact.submit();

  await expect(contact.submitButton).toBeDisabled();
  await expect(contact.successHeading).toBeVisible({ timeout: 5000 });
});

That last test caught a real bug class I see constantly in manual QA: double submission. The button only disabled after a React re-render — two fast clicks under slow network = two POST requests. On a contact form that's an annoyance; on a checkout it's a double charge.

Where the AI fails (and how I patched it)

1. It trusts the DOM too much. Generated selectors matched what exists today, including auto-generated class names. Fix: the conventions file above, plus review of every selector during stabilization.

2. It doesn't know what's worth testing. Ask for "tests for the dashboard" and you get 15 shallow assertions about text being visible. The test plan still has to come from a human who understands risk: payment flows, auth edge cases, state transitions. I write the plan; AI writes the code.

3. It can't feel flakiness. An AI-generated test that passes once isn't done. My stabilization loop: run 5× locally (--repeat-each=5), once in CI, review traces for silent retries. About 1 in 10 generated tests needs a rework at this stage.

An underrated trick: test the viewports nobody tests

My config includes a project most suites never have:

{
  // Foldable half-open posture — where responsive bugs hide.
  name: "foldable-half-open",
  use: { viewport: { width: 717, height: 1172 }, isMobile: true, hasTouch: true },
}

717px sits between Tailwind's sm and md breakpoints. I own a foldable (Motorola Razr), and the number of layout bugs living exactly in that gap is remarkable. Costs one config block, catches a bug class your users on premium Android devices hit daily.

Results after a month

Test authoring time: roughly −60%. The plan takes as long as before; the code doesn't.
Coverage went up, because tedious-to-write tests (pending states, error paths, viewport edge cases) stopped being skipped.
Maintenance stayed manageable — but only because of the selector conventions. I've seen AI-generated suites without conventions, and they rot in weeks.

The honest summary: Claude Code didn't replace QA judgment. It replaced QA typing.

I'm a QA director in Japan (ex-SHIFT, Japan's largest QA firm) and I build Playwright + Claude Code test suites as a freelance service — async, text-first, from JST. If your team wants a suite like this, you can find me on Fiverr or see more at s-nagaya.vercel.app. Questions about the workflow are welcome in the comments — happy to share the conventions file.