DEV Community: Jan Duris

Your Tests Passed. That Still Doesn’t Mean You Should Release.

Jan Duris — Tue, 25 Aug 2026 04:42:05 +0000

Your tests passed.

CI is green.

The build succeeded.

So you release.

Seems reasonable, right?

Not necessarily.

One of the most dangerous assumptions in modern software delivery is that a successful test suite automatically means a release is ready for production.

It doesn't.

Passing tests tell us that the checks we wrote passed.

A release decision asks a much larger question:

Do we have enough evidence to accept the risk of shipping this change?

Those are not the same thing.

What does a passing test actually prove?

Every automated test checks a specific expectation.

A unit test might verify that a function returns the expected result.

An API test might verify that an endpoint responds with 200.

An end-to-end test might confirm that a user can complete checkout.

All of these are useful.

But every test is created around assumptions.

Someone decided:

what should be tested,
which scenarios matter,
which data should be used,
which environments should be covered,
and which failure modes were worth checking.

So when a test passes, what we really know is:

Under the conditions we tested, the expected behavior occurred.

That is valuable evidence.

But it is not the same as:

This release is safe.

A release contains more risk than your test suite can see

Imagine this release:

Unit tests PASS
Integration tests PASS
E2E tests PASS
Lint PASS
Build PASS

Everything looks perfect.

But the same release also includes:

Database migration HIGH IMPACT
Authentication changes YES
Dependency upgrade YES
Rollback recently tested NO
Observability coverage PARTIAL
Deployment blast radius HIGH

None of those conditions necessarily need to make a test fail.

Yet every one of them can influence whether releasing is a good idea.

This is where we often confuse two different activities:

verification and decision-making.

Tests verify behavior.

Release decisions evaluate risk.

Green CI is a signal

A common delivery model effectively looks like this:

Tests pass
+
Build succeeds
+
Artifact exists
=
Deploy

For small systems and simple changes, this can work remarkably well.

But the larger the system becomes, the weaker that assumption gets.

A release can have a perfectly healthy CI pipeline and still be operationally risky.

For example:

green pipeline
+
critical service change
+
large migration
+
weak rollback confidence
+

limited observability

???

The individual checks are green.

The combined risk might not be.

Tests should be evidence, not the final decision

I think a better mental model is this:

             RELEASE DECISION
                    ▲
                    │
    ┌───────────────┼───────────────┐
    │               │               │
 Tests           Changes        Operations
    │               │               │
 Security       Dependencies     Rollback
    │               │               │
    └──────────── Evidence ─────────┘

Tests remain one of the strongest signals we have.

But they are one signal among several.

A release decision may also need to consider:

change size,
affected services,
security findings,
dependency changes,
database migrations,
production incident history,
deployment timing,
rollback readiness,
observability,
required approvals.

Not every team needs every signal.

The important part is the distinction:

Test results are evidence presented to a release decision. They are not the release decision itself.

This gets more important with AI-generated code

AI coding tools are making software creation dramatically faster.

They can generate implementation code.

They can generate tests.

They can fix failing tests.

And they can do all of that very quickly.

That creates an interesting situation:

More code
More tests
More automation
More green checks

But potentially:

Less human understanding of every individual change

An AI-generated implementation can satisfy the tests while still introducing:

a new architectural assumption,
an unexpected dependency,
different runtime behavior,
operational complexity,
or a failure mode nobody encoded into the tests.

The solution is not to stop using AI.

And it isn't simply to generate even more tests.

The solution is to become better at evaluating evidence around the release.

CI systems are excellent at execution

Modern CI/CD systems are incredibly capable.

They can:

run thousands of tests,
scan dependencies,
build containers,
validate infrastructure,
generate artifacts,
and deploy applications.

But there is one question many pipelines are surprisingly bad at answering:

Why are we comfortable releasing this version?

Ask that question after a deployment.

You may get answers like:

"Everything was green."

or:

"The PR was approved."

or:

"The pipeline allowed it."

Those answers describe what happened.

They don't necessarily explain the actual release decision.

What would a release decision layer look like?

Instead of:

CI green → deploy

we could think about delivery like this:

Code change
↓
CI / QA / Security / Operational signals
↓
Evidence collection
↓
Risk evaluation
↓
Policy
↓
Release decision
↓
Deploy

That decision does not always need to be binary.

For example:

The evidence is sufficient and the risk is acceptable.

CONDITIONAL GO

The release can proceed, but only with an approval, mitigation, feature flag, additional monitoring, or another condition.

NO GO

The available evidence or risk does not justify releasing yet.

This is much closer to how experienced engineers already think.

The difference is making that reasoning explicit instead of leaving it inside someone's head.

Green should have a precise meaning

There is nothing wrong with green CI.

We want green builds.

We want passing tests.

We want fast feedback.

The problem begins when we give those signals more meaning than they actually contain.

A green pipeline should mean:

The checks we defined passed.

A release decision should mean:

Given the available evidence and our risk policy, we are willing to ship this change.

Those statements are related.

But they are not identical.

And as software development becomes faster, more distributed, and increasingly AI-assisted, I think that distinction is going to matter much more.

How does your team decide that something is ready to release?

Is a green pipeline enough?

Or is there another decision layer after CI?

CI Is Green. What Could Still Go Wrong?

Jan Duris — Fri, 21 Aug 2026 13:41:28 +0000

A green CI pipeline is a nice feeling.

The build passed.
Tests are green.
Linting is clean.
Nothing obvious is on fire.

For a few seconds, it feels like the release question has already been answered.

It hasn't.

A green pipeline tells you something useful: the checks you configured have passed under the conditions in which they were executed.

That is not the same thing as saying the release is safe.

And the gap between those two ideas is where a lot of production problems begin.

I touched on this in my previous article about why a green CI pipeline can create a false sense of security. But there is another question that deserves more attention:

If CI is green, what can still go wrong?

Quite a lot, actually.

CI only knows what you taught it to check

This is probably the most important limitation.

CI is not intelligent by default.

It does not understand your product, your users, your business risk, or the weird operational history of your system.

It runs instructions.

If those instructions cover the right risks, great.

If they don't, the pipeline can be perfectly green while the release is still dangerous.

This is not really a failure of CI.

It is a failure of interpretation.

We often treat CI as if it has answered a bigger question than it actually has.

The pipeline says:

"The checks I know about passed."

Humans often hear:

"This release is safe."

Those are very different statements.

The code can be correct and the release can still fail

Imagine a change that passes every unit test and every integration test.

The code itself is fine.

Then it reaches production and fails because of configuration.

Wrong environment variable.

Missing secret.

Different feature flag state.

A service points to the wrong endpoint.

A timeout is configured differently in production.

A permission exists in staging but not in production.

None of those problems necessarily mean your tests were bad.

They mean the release depended on conditions outside the scope of those tests.

This is one of the reasons I think we should stop treating release readiness as purely a testing problem.

Testing is part of it.

It is not the whole thing.

Database migrations are a good example

Database changes are one of those areas where "green" can be misleading.

The migration itself may be syntactically correct.

Tests may pass against a fresh database.

Everything can look completely normal.

But production has something your test environment often does not have:

history.

Real data.

Old records.

Unexpected null values.

Large tables.

Long-running transactions.

Different indexes.

Different traffic patterns.

A migration that takes two seconds in CI can behave very differently against a production dataset that has existed for six years.

The pipeline can be green because nothing it executed was technically wrong.

The release can still be risky because the environment changed the meaning of that change.

Dependencies can change the picture

Modern software rarely runs in isolation.

A release can depend on:

another internal service
a third-party API
a package update
a cloud service
a message queue
a payment provider
a database
an authentication system

Your own repository may be completely healthy.

The dependency chain around it may not be.

Maybe another service has not deployed the expected API yet.

Maybe a backwards-compatibility assumption is wrong.

Maybe one dependency is already degraded.

Maybe the release introduces a combination of versions that nobody has actually tested together.

Again, CI may remain perfectly green.

It tested what it could see.

The release depends on more than that.

Passing tests do not mean the right tests ran

This one is uncomfortable, because it sounds obvious once you say it.

A test suite can have a 100% pass rate while still telling you very little about the change you are about to release.

The interesting question is not:

How many tests passed?

It is:

Did we test the things that matter for this release?

Those are not equivalent.

If somebody changes a critical authentication flow and your pipeline runs 3,000 tests mostly covering catalog functionality, the number "3,000 passed" looks impressive.

But what does it actually tell you about authentication risk?

Not much.

This is why I find the concept of risk coverage more useful than simply looking at test coverage or test count.

What changed?

What can that change affect?

Which checks give us evidence about those areas?

That context matters more than a big green number.

Feature flags do not magically remove risk

Feature flags are useful.

Very useful.

But sometimes they become another source of false confidence.

The thinking becomes:

"It's behind a flag, so the release is safe."

Maybe.

But the code still shipped.

The flag configuration itself can be wrong.

The disabled path may affect initialization.

The new code may introduce dependencies before the flag is evaluated.

The wrong audience may receive the feature.

The rollback path may not behave the way you expect.

Feature flags reduce certain kinds of risk.

They do not eliminate release risk.

That distinction matters.

Rollback is not a guarantee either

Another sentence I hear in software teams is some version of:

"If something goes wrong, we'll just roll it back."

That can be a perfectly reasonable risk strategy.

But "rollback exists" and "rollback is safe" are not the same thing.

What happens if the release includes a destructive migration?

What happens if new data has already been written in a new format?

What happens if another service starts depending on the new behavior?

What happens if the release changes an external contract?

Rollback sounds simple when we imagine software as a single binary moving backwards.

Real systems are messier.

Sometimes the release creates state that cannot be cleanly reversed.

A green CI pipeline usually does not tell you whether your rollback strategy actually works.

Production traffic is different

There is also a class of failures that only appears under real usage.

Concurrency.

Traffic spikes.

Unusual request sequences.

Large payloads.

Rare account states.

Slow upstream services.

Network instability.

Real users are extremely good at discovering combinations that test environments never produce.

This is not an argument against automation.

It is an argument for understanding what automation can and cannot prove.

A test suite can give you evidence.

It cannot give you certainty.

Observability matters before the release, not only after it

One thing I think teams often underestimate is operational readiness.

Suppose the release does contain a problem.

Will you notice?

How quickly?

Do you have the right metrics?

Do you have logs around the changed area?

Will an alert actually fire?

Can you distinguish expected noise from a real regression?

A release with some uncertainty but excellent observability can sometimes be safer than a release with more passing tests but poor visibility.

Because release safety is not only about preventing every possible failure.

It is also about limiting the damage when a failure happens.

That changes the release decision.

Human context still matters

Not every release risk is technical.

Sometimes the biggest risk is simply that nobody has the full picture.

QA knows about one unresolved issue.

A developer knows the fix was rushed.

Operations knows a dependent system has been unstable all afternoon.

Product knows a feature absolutely must not reach one customer segment yet.

Security raised a concern that is technically "non-blocking."

Each piece of information exists.

But it exists in different places, owned by different people.

Then the pipeline turns green.

And everyone assumes somebody else must have checked the rest.

This is where release processes become dangerous.

Not because there was no information.

Because the information never became part of the decision.

So what does green CI actually mean?

I still want green CI.

Obviously.

A red pipeline is useful information too, but I would much rather start from green.

The problem is not the green status.

The problem is giving that status more authority than it deserves.

For me, a green pipeline should mean something closer to:

The automated checks we currently require did not detect a blocking problem.

That is a strong signal.

It is not yet a release decision.

A real release decision needs more context.

Depending on the system, I would want to know things like:

What changed?
How risky is that area?
Which relevant tests actually ran?
Are there unresolved defects?
Are dependencies healthy and compatible?
Are migrations involved?
Are security checks clean?
Do we have sufficient observability?
Can we roll back safely?
Is there any known operational or business risk?

Only then does the question become interesting:

Do we have enough evidence to release?

Maybe the problem is the way we frame CI

We built CI to automate verification.

Over time, many teams started using it as a proxy for release readiness.

That shortcut works surprisingly often.

Which is exactly why it is dangerous.

Most releases with a green pipeline are probably fine.

Then one isn't.

And when you investigate what happened, the answer is often not:

"CI failed."

It is:

"CI never knew that this was something it needed to evaluate."

That is the gap I keep coming back to.

The future of release engineering, in my opinion, is not just more tests and faster pipelines.

It is better decision-making.

More context.

Better evidence.

A clearer understanding of what we know before we put a change in front of users.

Green CI should absolutely make us more confident.

It just shouldn't make us stop asking questions.

CI is green. Your release may still be risky.

Jan Duris — Sun, 12 Jul 2026 20:15:10 +0000

Every team has a moment like this: CI passes, the PR gets merged, the deploy goes out — and two hours later someone's asking why checkout is throwing 500s in production.

Nobody skipped a step. The pipeline was green the whole time.

The false comfort of green CI
Green CI feels like permission. It's a checkmark, it's fast, and it's automated — so it's easy to treat it as the final word on whether a release is safe. For a lot of teams, "tests passed" and "safe to ship" have quietly become the same sentence.

They're not.

What CI actually checks
Most CI pipelines are built to catch a specific, narrow slice of problems:

Does the code compile / build?
Do unit tests pass?
Does lint pass?
Maybe: does a small integration suite pass?

That's genuinely useful. It catches typos, broken imports, obvious logic errors, regressions in covered code paths. It's necessary.

It's just not the same thing as "this release is safe to put in front of users."

What CI doesn't tell you
CI has no opinion on:

Downstream dependencies — does the third-party API you call still behave the way your integration test mocked it to?
Manual steps — did someone actually run the DB migration on staging before merging, or is that still "to do"?
Deploy-time risk — blue-green cutover, cache invalidation, feature flag state — none of that is exercised by a test suite that runs against a branch.
Human process gaps — did the person who approved this PR actually understand the blast radius, or did they approve because the CI badge was green?

None of this shows up as a failing check. It shows up as an incident.

Release risk is a different question than test pass/fail
"Did the tests pass" is a yes/no question about code. "Is this release safe to ship" is a probabilistic question about a system: what changed, how big is the diff, does it touch a critical path, is there a rollback plan, has this kind of change caused problems before.

Most teams answer that second question with vibes — a Slack message, a shrug, "looks fine to me." That works fine until it doesn't, and by the time it doesn't, you're writing a postmortem instead of a checklist.

What a release risk gate actually looks like
The teams that don't get burned by this usually have some version of the same thing, whether they call it that or not:

A policy layer that blocks release on concrete conditions (required approvals, coverage thresholds, freeze windows) — not vibes, actual enforced rules.
A signal aggregator that pulls test results, error rates, and coverage into one release health view instead of five open tabs.
An audit trail — an immutable record of what was actually in a release, so when something breaks you're debugging from evidence, not memory.

None of this requires an enterprise platform. Small teams can build a lightweight version of this with GitHub Actions, a policy file, and some discipline. The point isn't the tooling — it's separating "code compiles" from "release is safe" as two distinct questions with two distinct answers.

Where to start if you don't have this yet
You don't need to build a whole release control system this week. Start smaller:

Write down the 3–5 things that, if skipped, have caused an incident before. Turn them into a checklist that's actually enforced, not just documented.
Add one non-negotiable gate to your PR template — coverage on touched files, or a manual QA sign-off for anything touching payments/auth.
Capture a snapshot of what shipped in each release (commit list, who approved, what tests ran) so your next postmortem starts with facts instead of "let me check Slack."

If you want a starting point instead of building it from scratch, I put together a Release Checklist Pack and a GitHub QA Gates Pack — practical templates for exactly this, no platform lock-in required.

I'm building Dulvarn, a release control system for GitHub-first teams, plus a set of standalone QA/release tools and templates. This post reflects my own experience shipping software, not any employer's views or confidential information.