DEV Community

oleksoleks07
oleksoleks07

Posted on AI-assisted

Your CI Pipeline Is Green. So Why Is Everyone Afraid to Release?

A green CI badge is supposed to mean the team can merge with confidence.

But ask the people who maintain the test suite a different question:

Would you trust this pipeline enough to release on a Friday afternoon?

If the answer is “it depends,” your automation may be passing without actually providing confidence.

A pipeline can be green while developers are rerunning tests, ignoring known failures, waiting too long for feedback, or quietly avoiding parts of the suite. The dashboard says success; the team’s behavior says otherwise.

That gap is where QA automation health matters.

Green is a status, not a quality signal

A passing build tells you that one execution completed successfully under one set of conditions. It does not tell you whether the result is repeatable, whether failures are diagnosable, or whether the suite protects the behavior that matters to users.

I usually look for four warning signs:

  • Retries are normal. A test fails, passes on retry, and the build is called successful. The failure disappears from the headline metric, but the instability remains.
  • The suite is slow enough to change behavior. Developers batch changes, postpone running tests, or stop waiting for the full pipeline.
  • Failures are hard to explain. A red build gives the team a symptom, not a useful route to the cause.
  • People work around the suite. They rerun jobs, skip tests, or ask someone who “knows the flaky ones” whether a failure can be ignored.

None of these problems is solved by adding more tests.

1. Measure reliability, not just pass rate

Pass rate is useful, but it is an incomplete measure when retries and quarantines are involved.

Track a small set of signals together:

Signal What it tells you
First-run failure rate How often the suite fails before retries
Retry rate How often a test needs another attempt
Flaky-test recurrence Whether the same tests keep failing inconsistently
Median and P95 pipeline duration Typical feedback time and the slow tail
Time to diagnose a failure How quickly the team can identify the cause
Quarantined tests and age How much coverage is currently outside the normal gate

The important part is to keep the definitions stable. If a retry turns a failure into a pass, preserve both facts in your reporting. Otherwise, the dashboard can make a noisy suite look healthy.

Also segment the numbers. A single overall pass rate can hide one browser, environment, or test group that is responsible for most of the pain.

2. Fix isolation before increasing test volume

Flaky tests often get blamed on the test runner. Sometimes the runner is involved, but recurring instability commonly comes from shared state and weak test boundaries.

Look for:

  • shared accounts or records that tests modify concurrently;
  • fixtures that leak state between tests;
  • selectors coupled to layout or styling details;
  • fixed sleeps where the test should wait for a meaningful condition;
  • tests that depend on execution order;
  • cleanup that runs only on the happy path.

For example, a fixed wait in Playwright is usually a warning sign:

await page.waitForTimeout(3000);
Enter fullscreen mode Exit fullscreen mode

It says, “I hope the page is ready after three seconds.”

Prefer waiting for the condition the test actually needs:

await expect(page.getByRole("heading", { name: "Dashboard" }))
  .toBeVisible();
Enter fullscreen mode Exit fullscreen mode

This is not a universal cure for flaky tests. The application may still have race conditions, unstable data, or asynchronous behavior that needs to be fixed. But condition-based waits make the test’s expectation explicit and remove timing guesses.

A useful rule: make each test responsible for its own setup, data, and cleanup wherever practical. Parallel execution should reveal isolation problems, not create mysterious dependencies.

3. Make failures actionable

A failure report should help an engineer answer three questions:

  1. What failed?
  2. What was the system doing at the time?
  3. What should I inspect next?

For browser automation, that often means retaining the right evidence: error output, screenshots, traces, browser and environment details, and enough information to reproduce the run.

Avoid collecting everything forever. Keep evidence that helps diagnose failures, set retention limits, and make sure secrets or personal data are not accidentally included in artifacts.

A useful failure message is not merely:

Expected true, received false
Enter fullscreen mode Exit fullscreen mode

It should include the failed expectation, the relevant state, and a link to the trace or screenshot when available.

The goal is to reduce the time between “the pipeline is red” and “we know why.”

4. Treat CI duration as an engineering constraint

A slow pipeline changes how people work.

Start by measuring median and P95 duration for the whole pipeline and its major stages. Then identify where time is actually spent:

  • dependency installation;
  • environment provisioning;
  • test startup and teardown;
  • browser or service startup;
  • repeated work that could be cached safely;
  • tests that can run independently in parallel.

Be careful with parallelism. Increasing workers can make a suite slower if tests compete for the same database, accounts, CPU, or external services. Measure the effect rather than assuming more workers means faster feedback.

A practical improvement is to separate fast feedback from broader confidence:

  • run linting, unit tests, and a small critical smoke suite on every change;
  • run broader integration and end-to-end coverage at an appropriate stage;
  • keep scheduled or pre-release suites for checks that are too expensive for every commit.

The exact split depends on the product. The principle is to give developers useful feedback quickly without pretending that a small smoke suite replaces full regression coverage.

5. A four-week path to a more trustworthy suite

You do not need to rebuild your automation framework to make progress. A focused sequence is often more effective.

Week 1 — Establish a baseline

  • Measure first-run failures, retries, flaky recurrence, and pipeline duration.
  • Identify the tests responsible for the most repeated failures.
  • Record which tests are quarantined and why.

Week 2 — Remove the biggest sources of instability

  • Fix shared test data and fixture leakage.
  • Replace arbitrary sleeps with condition-based waits.
  • Make cleanup reliable, including after failures.
  • Assign an owner to recurring flaky tests.

Week 3 — Improve diagnosis and feedback

  • Add or tune traces, screenshots, and useful logs.
  • Make failure reports easy to find from the CI job.
  • Split fast checks from slower regression coverage where it makes sense.

Week 4 — Make ownership sustainable

  • Agree on what qualifies as flaky and when a test may be quarantined.
  • Set an owner and review date for every quarantined test.
  • Review the baseline with the team and choose the next two or three improvements.

The objective is not a perfect dashboard. It is a suite that gives the team evidence they can act on.

Before buying another testing tool, inspect the system you already have

When a team is struggling with automation, it is tempting to add another framework, reporting service, or orchestration layer.

Sometimes a new tool is the right answer. But first, understand the current system: test architecture, data isolation, CI configuration, failure reporting, and ownership.

That is the thinking behind QA Health Check, a service for smaller SaaS and product engineering teams. It reviews one automation repository and one main CI pipeline, then provides a prioritized view of risks, practical quick wins, and a 30-day improvement plan. The aim is to help a team decide what to fix first—not to sell a framework rewrite.

Disclosure: I'm associated with QA Health Check. I’m mentioning it because it focuses on the same automation reliability and CI problems discussed in this article.

The real question

A mature automation suite is not the one with the most tests or the greenest badge. It is the one developers trust enough to use when making a release decision.

So, what is the biggest obstacle in your current setup: flaky tests, slow pipelines, hard-to-diagnose failures, or a lack of clear ownership?

I’d be interested to hear what has helped your team move from “the build passed” to “we trust this result.”

Top comments (0)