DEV Community: Mihir Shinde

How to Fix Flaky Tests in GitHub Actions

Mihir Shinde — Fri, 17 Apr 2026 00:36:33 +0000

You know the drill: CI goes red, you check the logs, the failure looks unrelated to your changes. You hit re-run. It passes. You merge. And the cycle repeats tomorrow.

This guide covers the six most common patterns behind flaky tests in GitHub Actions and gives you concrete fixes for each. Not theories — actual code changes and configuration updates you can apply today.

Before you start fixing

The first step is knowing which tests are flaky and how often they fail. If you're guessing based on Slack complaints, you're working blind. Kleore analyzes your CI history and ranks every flaky test by failure rate and cost — so you fix the worst ones first.

1. Timing & race conditions

Symptom: Test passes locally, fails intermittently in CI. Often involves UI tests, async operations, or anything that waits for a condition to become true.

Root cause: GitHub Actions runners have variable performance. A 2-core runner under load is slower than your M3 MacBook. Tests that assume operations complete within a specific window break when the runner is under pressure.

The fix: Replace fixed waits with condition-based polling. For E2E tests with Playwright or Cypress, use their built-in auto-waiting mechanisms instead of explicit sleeps. For backend tests, poll with exponential backoff rather than sleeping.

2. Shared mutable state

Symptom: Test passes in isolation (it.only) but fails when run with the full suite.

Root cause: Tests share a database, in-memory store, filesystem, or global variable. Test A writes data that Test B doesn't expect, or Test A forgets to clean up.

The fix: Isolate test state completely. If you're using a shared test database, consider running each test file in its own database schema or using containers.

3. External service dependencies

Symptom: Tests fail with network timeouts, 503 errors, or rate-limit responses.

Root cause: Your tests make real HTTP calls to APIs you don't control.

The fix: Mock at the HTTP boundary, not the function level. Use MSW or similar tools to intercept HTTP at the network level.

4. Environment differences

Symptom: Tests pass on macOS, fail on Linux. Or pass with Node 20, fail with Node 22.

Root cause: Assumptions baked into tests about the OS, timezone, locale, filesystem behavior, or available system resources.

The fix: Pin your CI environment explicitly. Always set TZ=UTC, use a .node-version file, and normalize filesystem paths.

5. Port & resource conflicts

Symptom: EADDRINUSE errors, database connection failures, or file lock errors — especially when tests run in parallel.

The fix: Use dynamic port allocation and unique database names per test worker.

6. Test order dependency

Symptom: Tests pass when run in the default order, fail when randomized.

Root cause: Test A sets up state that Test B implicitly depends on.

The fix: Enable test randomization (--randomize in Jest, sequence.shuffle in Vitest) to catch these issues early. Make every test self-contained.

The meta-fix: Retry as a bandaid, not a cure

Retry logic hides the problem. A test that fails 30% of the time and gets retried 3 times will appear to pass 99.7% of the time — while still costing you 3x the CI minutes and masking the underlying issue.

Retry to unblock your team today. Fix the root cause this sprint.

How to prioritize which tests to fix first

Prioritize by failure frequency, blast radius, cost per failure, and fix complexity.

Or let Kleore do the prioritization for you. Kleore analyzes your GitHub Actions history and ranks every flaky test by failure rate, cost, and impact. You get a prioritized list with dollar amounts — so you know exactly where to start.

Scan my repos — free

Also read: What Are Flaky Tests? | How Much Do Flaky Tests Actually Cost?

How Much Do Flaky Tests Actually Cost?

Mihir Shinde — Fri, 17 Apr 2026 00:10:34 +0000

When teams talk about the cost of flaky tests, they usually start with CI minutes. That's the visible part — the line item on your GitHub bill. But CI compute is maybe 10% of the real cost. The other 90% is human time, delayed shipping, and the slow erosion of engineering culture.

Let's break it down with real numbers.

Layer 1: CI compute

This is the easy math. Every time a flaky test causes a re-run, you're paying for the same CI job twice.

Metric	Example team
Average CI run duration	12 minutes
Flaky-caused re-runs per week	40
Wasted CI minutes per week	480 minutes
GitHub Actions cost per minute	$0.008
Monthly CI waste	~$60/month

$60 a month? That's nothing, right? That's the trap. CI compute is cheap enough that nobody escalates it. But it's the tip of the iceberg.

Layer 2: Developer time

This is where the real money goes. Every flaky failure triggers a human response:

Developer sees red CI badge on their PR
Opens CI logs, scrolls through output
Tries to figure out if the failure is real or flaky
Decides to re-run (or asks a teammate)
Waits for the re-run to finish
Resumes their previous work — but the context switch already happened

Research on context switching shows it takes an average of 23 minutes to regain deep focus after an interruption. Even if the investigation itself takes only 5 minutes, the true cost per interruption is closer to 30 minutes of productive time.

Metric	Example team
Flaky interruptions per week	40
Context-switch cost per interruption	30 min
Total developer hours lost per week	20 hours
Average fully-loaded eng cost	$85/hour
Monthly developer time waste	~$6,800/month

That's over 100x the CI compute cost. And this is for a modest team with a moderate flaky test problem. A team of 30 engineers with a bad flaky test culture can easily burn $20,000+/month in lost productivity.

Layer 3: Shipping velocity

Flaky tests don't just waste time — they slow down how fast you ship.

PRs stay open longer. A PR that gets a flaky red build sits in review limbo.
Merge conflicts compound. Longer PR lifetimes mean more merge conflicts.
Deploys batch up. When teams can't merge quickly, changes pile up into larger, riskier deploys.

Layer 4: Trust erosion

This is the most dangerous cost because it's invisible until it's catastrophic.

When tests are unreliable, developers develop a reflex: "It's probably just flaky." This is rational behavior given unreliable signals. But it means real failures get ignored too.

Phase	Behavior
Phase 1	Team re-runs flaky tests and reports them in Slack
Phase 2	Team re-runs without reporting — it's just background noise
Phase 3	Team merges with red CI, assuming flakiness
Phase 4	A real bug slips through. "We thought it was flaky."
Phase 5	Production incident. Post-mortem identifies eroded CI trust as root cause.

The total picture

Cost layer	Monthly cost	Visibility
CI compute	$60	On your bill
Developer time	$6,800	Hidden
Shipping velocity	$???	Invisible
Trust erosion	$???	Invisible until incident
Total	$7,000–$25,000+/month

The irony: the cost that shows up on your bill (CI minutes) is the smallest component.

What can you actually do about it?

Step one is visibility. You can't fix what you can't see. Most teams have no idea how many flaky tests they have, which ones are the worst, or what they cost.

That's the gap Kleore fills. It connects to your GitHub repos, analyzes your CI run history, and gives you a ranked list of every flaky test — with dollar costs attached. No configuration, no test framework changes, no new CLI tools. Just the data you need to start making decisions.

Scan my repos — free →

GitHub Actions Slow? What's Actually Wasting Your Time

Mihir Shinde — Thu, 16 Apr 2026 23:02:01 +0000

Your GitHub Actions pipeline takes 20 minutes. Your team runs it 50 times a day. That's 16 hours of CI compute daily — and most of it is waste. Developers context-switch while waiting, merge queues back up, and by the end of the week your team has lost an entire engineer's worth of productive time to a slow pipeline.

The fix isn't "buy bigger runners." It's eliminating the waste that's already in your pipeline. Here are the five biggest time wasters and how to fix each one.

The hidden cost of slow CI

Slow CI doesn't just waste compute. It creates a cascade of productivity losses that compound across your team:

Developer wait time: A developer waiting 20 minutes for CI is not coding. They're checking Slack, reading Hacker News, or starting a second task that creates costly context-switching when CI finishes.
Context switching: Studies show it takes 23 minutes to fully refocus after a context switch. A 20-minute CI wait often creates a 43-minute productivity gap.
Merge queue bottlenecks: When CI takes 20 minutes, your merge queue can process 3 PRs per hour at most (serially). With a team of 10 developers, PRs stack up and block each other.
Deployment velocity: Slow CI means fewer deployments per day, which means larger batch sizes, which means more risk per deploy. It's a vicious cycle.

The math is simple: if your CI takes 20 minutes and you have 10 developers, optimizing it to 8 minutes saves 2 hours of developer wait time per day. At $150/hour loaded engineering cost, that's $300/day or $78,000/year.

Time waster #1: Flaky test reruns

This is the single biggest source of CI waste, and it's the one most teams underestimate. When a flaky test fails, developers re-run the entire pipeline. That re-run wastes 100% of the compute — you're running the same tests again just to get a different roll of the dice.

The numbers are staggering. In our analysis of 10,000 GitHub Actions workflow runs, we found that 15-25% of CI compute is wasted on flaky test reruns. That means if you spend $10,000/month on GitHub Actions, $1,500 to $2,500 is literally burned on re-running tests that aren't actually broken.

The fix: Identify and quarantine flaky tests. You can't fix what you can't measure. Start by identifying which tests are flaky, then quarantine them so they don't block CI while you fix the root causes.

Time waster #2: No dependency caching

Every CI run that starts with npm install or pip install -r requirements.txt from scratch is downloading the same packages over and over. For a typical Node.js project, this wastes 1-3 minutes per run. Multiply that by 50 runs/day and you're losing 1-2.5 hours daily.

The fix: Use actions/cache or built-in caching. Pro tip: npm ci is faster than npm install in CI because it skips the lockfile resolution step.

Time waster #3: Running all tests on every PR

If a PR only changes a README file, there's no reason to run your entire test suite. Yet most teams configure their pipeline to run everything on every push. For large monorepos, this wastes enormous amounts of compute.

The fix: Use path filters and affected test detection.

Time waster #4: Sequential jobs that could be parallel

Many teams structure their pipeline as a linear chain: lint → type-check → unit tests → integration tests → e2e tests. Lint and tests don't depend on each other — they can run simultaneously.

The fix: Parallelize independent jobs. With parallel sharding, you can cut total wall time from 19 minutes to ~4 minutes. You pay for more compute-minutes, but your developers get feedback 5x faster.

Time waster #5: Oversized Docker images

If your CI builds Docker images, the image size directly impacts build time, push time, and pull time. A 2GB image takes minutes to push and pull. Most of that size is build dependencies that aren't needed at runtime.

The fix: Multi-stage builds with slim base images.

Quick wins checklist

✅ Enable dependency caching — 5 minutes to set up, saves 1-3 minutes per run
✅ Parallelize lint/typecheck/test — 15 minutes to restructure your workflow
✅ Add path filters — 10 minutes to add paths to your workflow trigger
✅ Shard your test suite — 20 minutes to set up matrix strategy
✅ Identify and quarantine flaky tests — 5 minutes to install Kleore
✅ Use multi-stage Docker builds — 30 minutes, cuts image size 50-90%

Originally published at kleore.com/blog/github-actions-ci-optimization