DEV Community: Mykhailo Krasnovskyi

API Contract Testing vs Integration Testing: What's the Difference?

Mykhailo Krasnovskyi — Fri, 25 Sep 2026 15:02:31 +0000

A team we worked with ran 400 integration tests before every release. Full green suite. Deploy on Friday.

Payments broke in production two hours later.

The cause? A field got renamed in the response body. Every integration test ran on a shared staging environment where both services happened to be on compatible versions that week. The contract had already drifted underneath them.

Nothing local caught it. Nothing local was actually checking the contract. It was checking whether two specific deployed versions got along that particular day.

This mix-up costs teams real money. It comes from treating contract testing and integration testing like the same thing, wearing two different names. They're not. They catch different bugs, run at different speeds, and belong in different parts of your pipeline.

The One-Sentence Version

Integration testing checks whether real, running services work together correctly, end-to-end.

Contract testing verifies that two services agree on the shape of their communication without either running the other's actual code.

That's the whole split. Everything else is detail.

What Integration Testing Actually Tests

Integration tests spin up real dependencies. Real database. Real message queue. Real downstream API, or at least a faithful stand-in for one.

Then they run an actual scenario through the stack and check what comes out the other end.

A typical integration test for an order flow:

Create an order through the API
Confirm it lands in the database with the right status
Confirm the payment service gets called and returns a charge ID
Confirm the notification service sends a confirmation email
Confirm the order status updates to "confirmed"

This is valuable work. It's the only way to catch bugs living in the interaction itself, not just the interface.

Race conditions. Transaction rollbacks. Timing issues. Real business logic tangled across services. If your discount logic breaks when a coupon code and a loyalty tier stack in a specific order, only a real integration test finds that. A contract doesn't know what a coupon is.

But integration tests carry a cost that gets uglier as your architecture grows:

They need everything running at once. Every dependent service, healthy, in sync.
They're slow. Minutes, not seconds, once you're past a trivial suite. -** They're environment-fragile.** A test passes locally, then fails in CI because staging runs a different service version. -** Failures stay vague.** Something broke. Rarely what, or where, without real digging.

At three services, none of these bite hard. At fifteen, it starts eating whole afternoons.

What Contract Testing Actually Tests

Contract testing skips the "does this whole thing work" question. It asks a narrower one instead: Does this API still look the way its consumers expect it to look?

No live dependency chain. No shared staging environment.

The consumer service — say, checkout — writes a test against a mock of the provider, in this case, payments. It states exactly what request it sends and what response shape it expects in return. That expectation gets saved as a contract, usually a JSON file.

The provider team then runs that contract against their real, running service, in isolation, with zero consumer code involved.

If the provider's real response doesn't match the contract, verification fails. Right there. In the provider's own build. Before anything ships anywhere.

Here's what a Pact consumer test looks like in practice:

provider
.given('account 123 exists')
.uponReceiving('a request for account details')
.withRequest({
method: 'GET',
path: '/accounts/123',
})
.willRespondWith({
status: 200,
body: {
id: '123',
balance: like(500.00),
},
});

Notice what's missing. No real payment service. No database. No network call to anything live. Just a statement of expectations, turned into a check that actually runs.

Contract tests are fast — seconds, because there's nothing to spin up. They're precise about failure, because a broken contract points straight at the field or endpoint that changed.

And they scale in a way integration tests never do. Add a sixteenth service, and you get one more contract. Not a combinatorial mess of environments to keep in sync.

What they can't do: tell you the business logic is correct. A contract test confirms that balance is a number. It has zero opinion on whether that number is the right one.

Side-by-Side Comparison

Integration Testing

What runs: Real services, real dependencies
What it catches: Business logic bugs, timing issues, real end-to-end failures
Speed: Slow — minutes per run
Environment needs: Every dependent service running and healthy
Failure clarity: Vague — something broke somewhere in the chain
Scales with service count: Badly — more services means more coordination
Coverage of user journeys: Yes, full flows
Coverage of API shape drift: Only if the test happens to touch the changed field

Contract Testing

What runs: Mocked provider on the consumer side, real provider in isolation on the verification side
What it catches: Shape mismatches, breaking API changes, drift between what's expected and what's shipped
Speed: Fast — seconds
Environment needs: None — no shared environment at all
Failure clarity: Precise — this field, this endpoint, this contract
Scales with service count: Well — more services means more contracts, not more coordination
Coverage of user journeys: No — single interactions only
Coverage of API shape drift: Yes, by design

Where Contract Testing Wins Outright

Independent deployments. Payments wants to ship three times a day. Checkout ships weekly. Integration tests force coordination, or they run against stale versions and lie to you. Contract tests let payments verify against every active consumer contract on every build. Checkout verifies its own expectations without waiting for payments to deploy anywhere.

Fast feedback in CI. A contract test suite for one service boundary runs in seconds. You get an answer before your coffee's done. An integration suite covering the same boundary, with real dependencies, easily runs ten or twenty times longer.

Deployment gating. Most teams miss this part entirely. Tools like Pact support a can-i-deploy check. Before a service ships, it asks a broker whether its current version has been verified against every contract still in force. No means the deploy stops. Integration testing doesn't give you anything like this — it doesn't produce a portable, queryable compatibility record. Just a pass or fail for one run, in one environment, at one moment.

Debugging speed. A contract test fails, you know exactly what changed. An integration test fails, you're digging through logs across four services trying to figure out which one lied.

Where Integration Testing Wins Outright

Real bugs in real interactions. Contract testing checks shape. It has no idea whether your refund logic correctly triggers a retention workflow for enterprise annual accounts, but skips it for monthly ones. That's business logic playing out across services. Only a real integration test, hitting real code with a real scenario, catches that.

Full user journeys. Checkout isn't one API call. It's a cart, inventory check, payment, notification, order confirmation — in sequence, with state carried through the whole thing. Contract tests check each boundary on its own. They say nothing about whether the full journey holds together.

Data consistency across writes. Need to confirm a write to one service correctly triggers a downstream read in another, with real data flowing through? Contract testing can't help you there. That's an integration test's job. Full stop.

Catching timing and race conditions. Contract tests are stateless snapshots of expected shape. They don't run concurrently against a shared state. They can't surface the kind of race condition that only shows up when two requests hit a database in the wrong order.

The Mistake Teams Keep Making

The common failure isn't picking the wrong tool. It's picking one tool and dropping the other entirely.

Teams that go all-in on integration testing end up with slow pipelines, flaky environments, and breaking changes slipping through. The test that would've caught it wasn't run that day because it required a service that was down for maintenance.

Teams that go all-in on contract testing get fast, precise pipelines that miss real bugs in business logic. Nobody's running a full order through the system anymore. Everyone's happy their contracts are green while checkout quietly breaks in a way no contract could ever catch.

Neither approach alone is a real strategy. They're complementary layers, not competitors. That's the part that gets lost whenever someone tries to crown a winner.

How to Actually Split the Work

A reasonable line most teams land on, after enough pain:

Contract tests for every service-to-service API boundary. Every consumer states what it needs. Every provider verifies it on every build.
A smaller set of integration tests for the handful of user journeys that actually matter to the business. Checkout, signup — the flows that lose you money or customers if they break.
Deployment gates built on contract verification, not on waiting for a full integration suite to go green across every environment.
Integration tests running less often — nightly, or before major releases — instead of on every single commit. They're expensive, and running them 40 times a day buys you almost nothing extra.

This won't fit every team exactly. A payments-heavy fintech product probably needs more integration coverage on money-moving flows than a content platform does. But the shape of the split — contracts for boundaries, integration for journeys — holds up across most architectures we've seen.

A Quick Gut Check

Not sure which one you need for a given test? Ask:

Am I checking whether two services agree on a shape? Contract test.
Am I checking whether** a real scenario produces the right outcome? **Integration test.
Do I need this to run in seconds, on every commit? Contract test.
Do I need this to catch a bug that only shows up when real code runs against real code? Integration test.
Am I trying to stop a bad deployment before it ships? Contract test, with a gate.
Am I trying to prove the checkout flow works end to end? Integration test.

Most teams don't need to pick one. They need both, sized right, each doing the job it's actually good at.

Where to Go Deeper

Running microservices and breaking changes keep slipping through to production? The fix usually isn't more integration tests. It's contract testing at the boundaries, wired into your pipeline as a deployment gate.

Our guide on API contract testing for microservices walks through exactly how to set that up — including the Pact broker workflow and the can-i-deploy check that stops a broken contract before it ever reaches a real user.

How to Reduce Flaky Tests in CI/CD Without Rewriting Your Whole Test Suite

Mykhailo Krasnovskyi — Fri, 07 Aug 2026 21:26:37 +0000

Flaky tests rarely spread evenly across a suite. In most pipelines we audit, a small group of specs produces the majority of unreliable failures. So you can usually fix the pipeline without a rewrite.

This is a practical guide to stabilizing automation you already own. No definitions of flakiness, no "rewrite it in Playwright" advice. Just triage, root cause work, quarantine rules, and the metrics that tell you whether it worked.

Examples cover Playwright, Cypress, Selenium, GitHub Actions, and GitLab CI.

How can I reduce flaky tests in CI/CD without rewriting the entire test suite?

Find the small set of specs causing most failures, classify why each one fails, fix by category instead of one test at a time, and quarantine the rest so the pipeline stays honest.

A rewrite is tempting because the suite feels untrustworthy. But a rewrite moves the same design mistakes into new syntax. Hard-coded sleeps and shared test data will reappear in any framework. Fix the causes first. Then decide if you still need new tooling.

What causes flaky tests in CI/CD pipelines?

Almost every flaky test traces back to a small number of root causes. Classify before you fix, because the remedy differs by category.

The Selenium documentation is blunt about the first row. Race conditions between browser state and driver commands are "one of the primary causes of flaky tests." That is where most teams should start.

Why do tests fail in CI but pass locally?

This question comes up in every audit, and the answer is usually one of 6 things.

Speed. CI machines are often slower and more loaded than your laptop, so implicit timing assumptions break.
Parallelism. Local runs are frequently single-threaded. CI runs four or eight workers that fight over the same test data.
Headless rendering. Different viewport, no GPU, fonts missing, elements positioned differently.
Clean state. Your local browser has cookies, cached auth, and a warm database. CI starts empty.
Locale and timezone. Runners default to UTC. Date assertions written in your timezone drift.
Network shape. Local hits a dev server on localhost. CI crosses a network boundary with real latency.

As a fast diagnostic, run your suite locally with the same worker count and a fresh browser profile. If it fails there too, the problem is your tests, not the runner.

How should QA teams prioritize flaky tests in an existing automation suite?

Do not fix flaky tests in the order you find them. Rank them. Start by calculating a flake rate per spec over the last 30 days:

flake rate = (runs that failed then passed on retry) / total runs

Then score each test on three factors: flake rate, how often the spec runs, and whether it blocks merges. A test that fails 4% of the time on every pull request costs far more engineering hours than one failing 30% of the time in a nightly job.

Flaky test triage checklist

Run this on each candidate before writing a fix:

Does it fail in isolation, or only in a full parallel run?
Does it fail on a specific worker, browser, or shard?
Is there a hard-coded sleep or fixed timeout in the failure path?
Does it create or reuse data that another test touches?
Does it depend on a previous test, leaving the app in a certain state?
Does it call a third-party service that is not stubbed?
Do the failure timestamps cluster around deploys, backups, or cron jobs?
Which root cause row from the table above matches?
Fix now, quarantine, or delete?

Keep the last question open. Some flaky tests cover logic already tested at a lower level, so deleting them is a legitimate outcome.

How do you fix flaky automated tests in CI/CD?

Fix by category. The next 3 categories cover most of the work.

1. Replace sleeps with condition-based waits

A fixed sleep is a guess about timing. It fails when the app is slower and wastes time when it is faster.

Playwright assertions retry until they pass or time out, so this is usually enough:

// Fragile: assumes the row renders within 3 seconds

await page.waitForTimeout(3000);

await expect(page.locator('.order-row')).toHaveCount(1);

// Stable: retries the condition itself

await expect(page.locator('.order-row')).toHaveCount(1, { timeout: 10_000 });

In Cypress, wait on the request rather than the clock:

cy.intercept('POST', '/api/orders').as('createOrder');

cy.get('[data-cy=submit]').click();

cy.wait('@createOrder').its('response.statusCode').should('eq', 201);

cy.get('[data-cy=order-row]').should('have.length', 1);

For Selenium, use explicit waits and avoid one specific trap. The Selenium waits documentation warns against mixing implicit and explicit waits, because the combination produces unpredictable timeouts:

from selenium.common import NoSuchElementException, ElementNotInteractableException

from selenium.webdriver.support.wait import WebDriverWait

errors = [NoSuchElementException, ElementNotInteractableException]

wait = WebDriverWait(driver, timeout=10, poll_frequency=0.2, ignored_exceptions=errors)

wait.until(lambda d: d.find_element(By.CSS_SELECTOR, ".order-row").is_displayed())

Pick one strategy per project. Explicit waits give you control per interaction.

2. Give every test its own data

Shared fixtures are the most common cause of parallel-only failures. Generate data per test and seed it through the API instead of the UI:

test('user can cancel an order', async ({ page, request }) => {

const email = qa+${crypto.randomUUID()}@example.com;

const { id } = await (await request.post('/api/test/orders', {

data: { email, status: 'pending' },

})).json();

await page.goto(/orders/${id});

await page.getByRole('button', { name: 'Cancel order' }).click();

await expect(page.getByText('Order cancelled')).toBeVisible();

});

API setup removes a long UI path from the test and cuts the number of steps that can fail for reasons unrelated to what you are testing.

3. Control the network

Stub third-party calls in functional tests. Keep a separate, small set of contract tests that hit the real service on a schedule, so an outage at a payment provider does not block merges.

How can I stabilize Playwright, Cypress, or Selenium tests?

Framework-specific moves that pay off quickly.

Playwright. Use role and label locators over CSS chains. Enable trace: 'on-first-retry' so you get a full timeline for the failure without storing traces for every run. Keep tests isolated rather than reaching for serial mode, which the docs recommend for the same reason.

// playwright.config.ts

export default defineConfig({

retries: process.env.CI ? 2 : 0,

use: { trace: 'on-first-retry', video: 'retain-on-failure' },

});

Cypress. Turn on test isolation, use cy.session() for cached login, and replace cy.wait(ms) with aliased intercepts. Set retries only in run mode:

// cypress.config.js

retries: { runMode: 2, openMode: 0 }

Selenium. Centralize waits in one helper so timeouts are consistent, pin browser and driver versions in CI, and run in a container so local and pipeline environments match.

Across all three, disable CSS animations in your test build. It removes an entire class of timing failures in a single change.

Should engineering teams use retries for flaky tests?

Retries are a detection tool, not a cure. Used well, they keep the pipeline moving while you fix causes. Used badly, they hide real bugs.

The rule we apply: retries are allowed, but a passing-on-retry result must be recorded as flaky, not green. Playwright does this by default, sorting results into passed, flaky, and failed. Its Playwright test retries docs also expose testInfo.retry, which is useful for clearing server state before a second attempt:

test('checkout completes', async ({ page }, testInfo) => {

if (testInfo.retry) await resetCartState();

// ...

});

Two limits to set. Cap retries at two, since a test needing three attempts is broken rather than flaky. And never retry the whole pipeline job to get a green build, because that erases the signal you need.

In GitLab CI, scope job-level retries to infrastructure problems only:

e2e:

script: npx playwright test

retry:

max: 2

when:

  - runner_system_failure

  - stuck_or_timeout_failure

artifacts:

when: always

paths: [playwright-report/]

reports:

  junit: results.xml

That way runner crashes get retried and genuine test failures do not.

When should flaky tests be quarantined?

Quarantine when a test is unreliable enough to erode trust but too valuable to delete, and you cannot fix it this sprint. It keeps the main pipeline meaningful while the test stays visible.

Quarantine works only with an expiry date. Without one, the quarantine list becomes a graveyard.

Quarantine policy example

1. Eligibility: flake rate above 2% over 30 days, or 3+ false failures in a week.

2. Action: tag the test @quarantine and move it out of the blocking job.

**3. Ownership: **the owning team is assigned within 24 hours. No owner, no quarantine.

4. Time limit: 14 days. Fixed, deleted, or escalated at expiry.

**5. Cap: **quarantine holds no more than 2% of the suite. At the cap, stabilization work takes priority over new test development.

6. Visibility: quarantined tests run nightly and report to the team channel.

In GitHub Actions, run quarantined tests in a separate non-blocking job:

jobs:

e2e:

runs-on: ubuntu-latest

steps:

  - uses: actions/checkout@v4

  - run: npx playwright test --grep-invert @quarantine

  - uses: actions/upload-artifact@v4

    if: always()

    with:

      name: playwright-report

      path: playwright-report/

quarantined:

runs-on: ubuntu-latest

continue-on-error: true

steps:

  - uses: actions/checkout@v4

  - run: npx playwright test --grep @quarantine

The continue-on-error: true flag keeps these results reported without blocking the merge.

Key metrics teams should monitor

Stabilization work needs numbers, or you cannot tell progress from luck.

Track flake rate per spec alongside the suite-wide number. Averages hide the handful of tests doing the damage.

Do and Don't

Do:

Classify root cause before writing a fix
Rank by flake rate multiplied by run frequency
Seed data through the API, unique per test
Store traces and video on first retry
Report retry passes as flaky, never as green
Put an expiry date on every quarantined test
Pin browser and driver versions in CI
Review the flakiest five specs in your weekly QA sync

Don't:

Rewrite the suite before diagnosing it
Add sleeps to "make it stable"
Raise global timeouts as a blanket fix
Retry entire pipeline jobs to get green
Mix implicit and explicit waits in Selenium
Share user accounts or records across parallel tests
Skip tests silently with no owner or date
Judge progress on suite-wide averages

Where to start on Monday

Pull the last 30 days of CI results. Rank specs by flake rate times run frequency. Take the top five, classify each against the root cause table, and fix by category. Quarantine anything you cannot fix in two weeks, with an owner and a date.

In the audits we run, pipeline trust usually recovers from this alone, before anyone touches framework choice. If you would rather have an outside read on which tests to fix, quarantine, or delete, that is what test automation consulting services cover. In this case, experts audit the existing framework, stabilize what is worth keeping, and hand back the flake metrics and quarantine policy your team runs afterwards.

By the way, if you have a stabilization tactic that works on your pipeline, drop it in the comments, it would be nice to read your experience.