DEV Community

Cover image for Fails only in CI, part 2: the causes readers taught me
Babatunde Hammed
Babatunde Hammed

Posted on

Fails only in CI, part 2: the causes readers taught me

Last week I wrote about tests that pass on your machine and fail in CI. I listed five usual suspects: speed, leftover state, environment, the network path and parallel runs.

Then the comments arrived, here and on Reddit, LinkedIn and the Ministry of Testing Club. Testers shared failures that took them a day, a week, and in one case over a year to solve. Most of them weren't on my list.

So this is part 2: the causes I learned from you, each with the fix that worked, and then a small CI setup that makes the next mystery quicker to solve.

1. The mouse pointer was in the wrong place

One reader had tests that always passed locally and always failed in CI. The logs and screenshots showed an empty field. After a day of debugging: the mouse pointer.

On the CI runner, the pointer happened to sit over an element that opened a popup menu. The menu covered the field the test was filling, so the text went nowhere. The field was optional, so nothing failed there. The test blew up much later, at a step that had nothing to do with the cause.

Two habits would have saved most of that day:

// Make the starting state explicit instead of "wherever the pointer is"
await page.mouse.move(0, 0);

await page.getByLabel('Reference').fill('ABC-123');
// Assert right after you act, so the failure points at the real step
await expect(page.getByLabel('Reference')).toHaveValue('ABC-123');
Enter fullscreen mode Exit fullscreen mode

Lesson: assert where you act. A failure three steps later sends you looking in the wrong place.

2. File and glob order

fs.readdir, glob and friends don't promise any order. The order you get depends on the filesystem. Locally it looks alphabetical every time, so nobody notices that the test depends on it. On the runner, it isn't.

Any test that takes "the first file" or compares a list without sorting is waiting to break.

const files = (await fs.promises.readdir(dir)).sort();
Enter fullscreen mode Exit fullscreen mode

The same applies to database queries without an ORDER BY, API results without an explicit sort order, and rows in a table that your test reads as a list. Locally they usually come back in insertion order by coincidence; a fresh CI database with a different disk layout or query plan can flip it.

3. Shallow clones

Most CI checkouts fetch only the latest commit and no tags. Anything that calls git describe, reads the last tag for a version number, or diffs against main works on your machine and fails, or quietly returns nonsense, in CI.

In GitHub Actions, fetch full history for the one job that needs it, not every job:

- uses: actions/checkout@v4
  with:
    fetch-depth: 0   # full history and tags
Enter fullscreen mode Exit fullscreen mode

4. Dependency drift: both builds green, and broken anyway

This one was the reverse of the usual story. The build succeeded locally and in CI, but one component broke at runtime on a developer's machine.

The cause: CI couldn't read the lockfile in the repo, so it regenerated it on every run. CI and local ended up with different versions of a worker package. The team followed what CI did step by step and saw no difference, because the difference was in what got installed, not in the steps.

Two fixes make this kind of drift fail loudly at install time instead of at runtime:

  • Pin the Node version, for example in .nvmrc or the engines field of package.json
  • Use npm ci in CI, not npm install. It installs exactly what's in the lockfile and fails if the lockfile is out of sync with package.json

5. Docker isn't the whole environment

A reader writing unit tests for kernel flags had dockerized builds and still got caught. Containers share the host's kernel. The newer kernel on their laptop supported flags that the CI runner's kernel didn't.

  • Print uname -r at the start of the CI job, so the kernel version is in every log
  • Check for the feature at runtime and skip with a clear reason, instead of assuming it's there
  • If the feature is required, pin the runner OS so the kernel is known

6. Shared data gives false greens, not just false reds

I wrote last week that tests sharing data can fail when run in parallel. A reader on X pointed out the scarier half: they can also pass when they shouldn't.

Two workers share a basket. One test passes because it finds the item the other test added. A false green is worse than a failure, because nobody looks at it.

test('adds an item to the basket', async ({ page }, testInfo) => {
  // Each test gets its own basket, so nothing else can make it pass
  const basketName = `basket-${testInfo.testId}`;
  // ...create basketName, add the item, then assert on that basket only
});
Enter fullscreen mode Exit fullscreen mode

Assert on the specific record you created, not on "an item exists".

7. "CI quietly decides it lives in UTC"

Timezone was in part 1, but readers added the sneakiest version: a test that fails for only an hour or two each day, when local time and UTC fall on different dates. Everyone assumes it's flaky until someone notices it always fails late in the evening.

Set the timezone explicitly, and control the clock in tests that care about dates:

// playwright.config.ts
use: { timezoneId: 'Europe/London', locale: 'en-GB' }

// in a test that depends on "today"
await page.clock.setFixedTime(new Date('2026-11-09T23:30:00Z'));
Enter fullscreen mode Exit fullscreen mode

The same goes for localeCompare and number or date formatting. Minimal CI images often have different locale data, so pass an explicit locale in tests.

A small CI setup that makes the next one easier

Several replies came back to the same idea, and @alexshev put it best: CI failures are often environment mismatches disguised as code failures. The fix is evidence, not guesswork.

1. Print the environment at the start of every job. It costs seconds, and it means local versus CI becomes two lists you can compare instead of two memories:

- name: Environment
  run: |
    node -v
    npx playwright --version
    uname -r
    date
    echo "TZ=$TZ LANG=$LANG"
Enter fullscreen mode Exit fullscreen mode

2. Keep the evidence when a run fails. Save the Playwright report and traces as artifacts, even when the tests fail:

- uses: actions/upload-artifact@v4
  if: ${{ !cancelled() }}
  with:
    name: playwright-report
    path: playwright-report/
    retention-days: 14
Enter fullscreen mode Exit fullscreen mode

3. Save the generated input, not just the seed. This was @alexshev's point, and it changed how I think about test data. He records the seed, generator version, locale, timezone and fixture version on every run. For a failed CI run, he also keeps the actual generated input (scrubbed if needed) alongside the trace for a short time. In his words, a seed alone is not enough once a dependency, clock, locale or generator behaviour changes. Once the bug is fixed, that exact input becomes a regression fixture.

In Playwright, attaching the input to the failing test takes one line:

await testInfo.attach('generated-input', {
  body: JSON.stringify(input, null, 2),
  contentType: 'application/json',
});
Enter fullscreen mode Exit fullscreen mode

4. Decide who looks first. A question I got on Reddit: when tests break after a merge, is it the tester's job or the SDETs'? It depends on the team, but a common default works well. If the tests fail on your pull request, you look first. If something breaks after merge, whoever's change caused it investigates, and whoever owns the pipeline handles runner and config problems. The worst option is a red build that everyone assumes someone else is fixing.

Where AI fits in

Same view as part 1, sharpened by the comments. AI assistants are good at reading a long CI log, explaining an unfamiliar error and suggesting theories. Give one the environment printout from a passing local run and a failing CI run, and it will often spot the difference faster than you.

What it can't do is see your runner. Nobody's AI would have guessed the mouse pointer or the kernel version without that information in front of it. The evidence from the setup above is what makes its suggestions worth anything.

Keep them coming

Twelve causes so far between the two posts, and I suspect we're nowhere near the end. What's yours? I'll keep collecting them, and I plan to turn the best ones into hands-on practice challenges.

Thank you to everyone who shared a story, and to @alexshev for letting me quote him.


I'm building Locator League, browser-based Playwright challenges where you write the test and it runs for real. I plan to turn some of the stories above into a "Works on my machine" practice track.

Top comments (4)

Collapse
 
sebastian_gedda_08bcd28f8 profile image
Sebastian Gedda •

Really useful follow-up, the false-green point in #6 is the one I'll be quoting. One more cause from my side: live third-party or staging APIs. Locally they answer fast and with the data you expect; on the runner they're slower, rate-limited, or the seed data changed overnight, and the test fails somewhere unrelated. What fixed it for us was recording real responses once and replaying them in CI, so "the network path" becomes a file in the repo you can diff when something changes.

Collapse
 
2unde profile image
Babatunde Hammed •

Glad #6 landed, a false green is the scariest one. Record and replay is a great fix, and I love "the network path becomes a file you can diff". In Playwright, routeFromHAR() does exactly that: record once with update: true, replay in CI. How do you decide when to re-record?

Collapse
 
sebastian_gedda_08bcd28f8 profile image
Sebastian Gedda •

Good question. I re-record when a drift check fails: a CI job hits the real API, compares the response with the stored mock, and goes red if the shape changed. Then I re-record, review the diff, and accept it. Never a silent overwrite.

routeFromHAR is great inside Playwright, but it stops at the browser: no Node, Jest/Vitest or React Native calls, and one big HAR that's hard to diff per request. That's the gap I'm building Mockifyer for (disclosure: I'm the author). It records at the fetch/axios level and keeps one file per request, so every re-record is a clean diff.

Collapse
 
marcusykim profile image
Marcus Kim •

The shared basket example makes a false green concrete. I'd have each worker assert the basket ID and the specific line item it just added, then attach those IDs to the trace with the test input. That makes it possible to tell a real persistence failure from a different worker's data showing up in the same screen.