Paste failing Playwright into the agent, not into Slack

Last week a checkout e2e failed on main. Someone screenshot the red CI job, dropped it in Slack, and three of us spent twenty minutes guessing which locator drifted. Then I pasted the raw Playwright error — stack, expected vs received, the page URL from the trace — into the agent and asked it to propose a one-file fix. Ten minutes later we had a patch and a short note on why the locator was brittle.

Failing Playwright output is dense, boring, and perfect for an agent. Slack is for decisions. Mixing those two jobs is how flaky tests stay flaky.

What the agent actually needs from a failure

A red badge and “checkout is broken” is not a prompt. Agents fix tests when you give them the same artifacts you’d open yourself:

• The exact assertion message (expected / received / timeout)
• The stack pointing at the test file and line
• The page URL (or route) at failure time
• Any trace / screenshot path if CI uploaded one
• The command that ran (pnpm test:e2e --grep checkout)

Playwright’s default reporter already prints most of that. Copy the failure block. Don’t paraphrase it into “the submit button didn’t work.”

I keep a tiny paste template so I don’t forget pieces:

## Playwright failure (paste raw)

Command: pnpm exec playwright test e2e/checkout.spec.ts --project=chromium
CI job: main / e2e / #4821

<<< paste the failure block here, including Expected/Received and stack >>>

Trace zip (if any): artifacts/trace-checkout.zip
Goal: smallest fix in the test OR one related page object — do not rewrite the suite
Out of scope: changing product behavior, deleting assertions "to make green"

That last line matters. Agents love to “fix” a red test by weakening it.

Why Slack threads make flaky tests worse

Slack is great for “is this a real regression or CI flake?” It’s bad for debugging locators:

• Screenshots lose the selector string and timeout numbers
• People reply with hunches (“maybe the spinner?”) before anyone reads the stack
• The original error scrolls away; the agent never sees it
• You get five parallel theories and zero pasted artifacts

I still post a one-liner in Slack after I have a plan: “checkout e2e: locator for pay CTA drifted after the redesign — patch in PR #…”. The debugging happens with the agent and the raw log, not in the channel.

A workflow that sticks

1. Reproduce locally once if CI is flaky-suspicious — same project, headed if needed. If it only fails under load, say that in the prompt.
2. Paste the failure + the test file path into the agent. Attach the relevant page-object file if the stack points there.
3. Constrain the edit: “one test file + at most one page object; keep assertion intent; prefer getByRole over CSS.”
4. Ask for a hypothesis first, then the patch. I want to read “the button text changed from Pay to Place order” before I accept a locator rewrite.
5. Re-run the single test and paste the green (or still-red) output back. Don’t merge on vibes.

Example constraint I reuse:

You are fixing ONE Playwright failure. Do not expand coverage.

Rules:
- Prefer getByRole / getByLabel over CSS / XPath
- Do not remove waits by deleting the assertion
- Do not add `waitForTimeout`
- If the app copy changed, update the locator and note the copy change in the PR body
- If the failure looks like a real product bug, STOP and say so — do not paper over it in the test

That stop rule is the whole point. Green tests that hide broken checkout are worse than a red CI job.

What “good” agent output looks like

I reject patches that only say “updated selector.” I want:

• Root cause in one sentence (copy change, timing, wrong frame, strict-mode dual match)
• Diff limited to the failing path
• Same assertion meaning (still checking the user can place an order)
• Optional: a note if the test was asserting implementation detail (class names) and should move to role/label

If the agent rewrites three unrelated specs “while here,” I throw the patch away and tighten the prompt.

Flakes vs bugs — make the agent choose

Paste this when you’re unsure:

Classify first:
A) Product regression — leave the test failing; describe what to verify in the app
B) Test drift — locator/copy/timing; propose the minimal test fix
C) Infra flake — suggest a stable wait (network idle / attached) without sleep

Do not mix A and B in one PR.

Agents default to B because editing tests is easy. Forcing the classification catches the “we made CI green and checkout is still broken” failure mode.

Traces without dumping a novel into chat

If you have a Playwright trace, don’t paste the whole JSON. Point the agent at:

• the failing action name from the trace UI
• the snapshot HTML path if it’s small
• or a short description: “click Pay → expect order confirmation; timed out waiting for #confirmation”

Better: upload the trace as a CI artifact and paste the URL + the failing step title. The agent still needs the assertion text; the trace is supporting evidence.

What I stopped doing

• Dropping only screenshots into Slack and hoping someone “knows that page”
• Letting the agent add waitForTimeout(2000) as a “fix”
• Accepting suite-wide locator refactors for one red test
• Merging “fixed flaky test” PRs with no root-cause sentence

Paste the failure. Constrain the blast radius. Make the agent classify bug vs drift. Then tell Slack what you decided — not how you got there.