It passes on my machine and fails in CI: a practical checklist (and where AI actually helps)

If you work in test automation long enough, you get a very specific kind of message from your pipeline. Red build. You pull the branch, run the same test locally, and it's green. You run it again. Green. You run it ten times. Still green.

Then CI runs it again and it fails.

This week I asked other testers for their strangest "only fails in CI" bugs, and the replies were a mix of comfort and horror. One person described a Selenium failure on a VM that devs, DevOps and testers had been chasing for over a year. Nobody had cracked it.

So this isn't a "five easy fixes" post. Some of these are genuinely hard. But most CI-only failures I've seen fall into a handful of buckets, and knowing the buckets makes the search a lot shorter.

The pattern behind most of them

CI usually isn't broken. Your machine is quietly giving the test something for free, and CI isn't.

Once you think about it that way, the question changes from "why is CI flaky?" to "what does my laptop have that the runner doesn't?"

1. Speed

Your laptop is probably faster than the shared runner, and nothing else is competing for its CPU.

So any check that doesn't wait will pass locally by luck and fail in CI. The classic one in Playwright:

// Checks once, right now. No waiting, no retry.
if (await page.getByText('Saved').isVisible()) { ... }

// Retries until it passes or times out.
await expect(page.getByText('Saved')).toBeVisible();

Same goes for reading text straight after a click, or a fixed waitForTimeout(2000) that happens to be long enough on your machine.

Quick check: run the test many times locally, ideally while your machine is busy with something else. If it starts failing, you've found your bucket.

npx playwright test my.spec.ts --repeat-each=20

2. Leftover state

Locally, you've been logged in to that test account for weeks. The test data already exists. Some feature flag got switched on months ago and nobody remembers.

CI starts from nothing.

If your test only passes because something already happened before it ran, it'll pass on your machine forever and fail on a clean runner. The fix is boring but reliable: every test sets up what it needs (seed data, log in through an API, saved storage state) and doesn't assume anything else.

3. Environment differences

The small stuff:

• A .env file that only exists on your machine
• Timezone: runners are very often on UTC. If your machine isn't, "today" and "yesterday" can land on different dates and assertions break
• Locale: number and date formatting
• Screen size: a smaller default window in CI can push elements off-screen or under a sticky header
• Browser version drifting between local and CI

In Playwright you can pin a lot of this in the config, so local and CI share it by default instead of hoping they match:

// playwright.config.ts
export default defineConfig({
  use: {
    locale: 'en-GB',
    timezoneId: 'Europe/London',
    viewport: { width: 1280, height: 720 },
    trace: 'on-first-retry',
  },
});

One of the replies I got put it well: the most useful habit is reproducing the CI environment as closely as possible (same OS, headless, browser version, locale, timezone, clean state) before you start guessing. If your CI runs in a container, run that same container locally.

4. The network path

This one is the most invisible, and someone in the thread raised it better than I would have.

Your laptop may be on a corporate VPN, behind a proxy, using split DNS, or trusting a company certificate authority. The CI image may have none of those.

And here's the trap: a curl or Node health check passing in CI doesn't prove the browser is using the same proxy settings. They can take completely different routes.

To tell a network problem apart from a slow element:

• Check DNS resolution from inside the runner
• Check the browser's proxy settings, not just the shell's
• Find the first failed network request in the trace, not the last error message

And please don't "fix" a certificate error by turning on ignoreHTTPSErrors and moving on. The test goes green, but you've hidden exactly the difference you were looking for. Get the CA into the CI image properly.

5. Parallel runs

Tests that pass alone and fail together. Two workers editing the same user, the same basket, the same row in a table. Locally you're often running just the one spec you're debugging, so you never see the collision.

Quick check: run the failing test on its own in CI. If it passes alone and fails in the full run, look for shared data.

Before you add a retry

Retries have their place. But as one reply put it: retries can hide the cause, but they rarely explain it.

My order of operations now:

1. Get the trace, console log, network log and screenshot from the failed CI run
2. Run locally with the same inputs and settings, and get the same artefacts
3. Compare them side by side. Where do they first differ?
4. Only then decide if it's a real bug, a test bug or an environment problem

npx playwright show-trace path/to/trace.zip

One caution: traces capture network requests, headers and page content. Check them for tokens and internal hostnames before attaching one to a public ticket.

Where AI actually helps (and where it doesn't)

I use AI assistants for this kind of debugging most days now, and they've changed how fast I get through the boring parts. But they're better at some bits than others.

Where it helps:

• Reading the wall of log output. Paste in a long CI log and ask "what's the first real error here?" It's good at spotting that the timeout at the bottom was caused by a failed request 200 lines earlier.
• Explaining unfamiliar errors. Certificate errors, DNS failures, odd browser launch messages. You get a plain-English explanation and a list of likely causes in seconds.
• Comparing two runs. Give it the passing and failing output and ask what's different.
• Writing the boring setup. Seed scripts, API login helpers, a config that pins locale and timezone.
• Git and PRs, if that's not your strong side yet. It can create the branch and raise the PR, but read the diff yourself and ask it to explain each command. Your name is on that PR.

Where it struggles:

• It can't see your environment. It doesn't know your VPN, your proxy or what's baked into your CI image unless you tell it. A lot of CI-only bugs live exactly there.
• Its first fix is often a band-aid. Ask an assistant to "make this test pass" and you'll quite often get a longer timeout, a retry, or force: true on a click. That makes the build green without explaining anything. Ask it "why does this fail?" before "fix this".
• It sounds just as confident when it's guessing. Treat its answer as a hypothesis, then prove it with the trace.

The way I think about it: AI shortens the time between "the build is red" and "I have three good theories". Proving which theory is right is still your job, and the trace is still the evidence.

Over to you

The year-long Selenium VM mystery is still bugging me. What's the strangest CI-only failure you've tracked down, and what turned out to be the cause? Drop it in the comments. I'm collecting them.

---

I'm building Locator League, browser-based Playwright challenges for practising this kind of thing (waiting properly, mocking, network checks) without setting up a project.

Comments

[Alex Shev]

The practical checklist is valuable because CI failures are often environment mismatches disguised as code failures. I would log the runtime image, dependency lockfile, timezone, locale, and any generated inputs alongside the failing job. AI can help summarize diffs or suggest hypotheses, but a reproducible artifact is what turns the fix from guesswork into evidence.