DEV Community: Sushant Joshi

What We Learned Generating 12,000 API Tests from OpenAPI Specs

Sushant Joshi — Wed, 15 Jul 2026 14:04:55 +0000

We thought generating API tests from an OpenAPI specification would eliminate the repetitive work of writing test cases. It did. What we didn't expect was how quickly it exposed problems in our API specifications instead of our APIs.

After generating more than 12,000 API tests from OpenAPI specifications across dozens of projects, one thing became obvious:

The biggest bottleneck wasn't the test generator.

It was the API specification itself.

We expected to spend most of our time improving generation algorithms.

Instead, we spent most of it fixing incomplete schemas, ambiguous contracts, and inconsistent specifications.

If you're planning to generate tests from OpenAPI, here's what surprised us the most.

The Promise of OpenAPI Test Generation

Most API teams already have an OpenAPI specification.

Unfortunately, many teams treat it as documentation instead of a contract.

That's a missed opportunity.

A good specification can generate:

Request validation
Response validation
Schema assertions
Negative test cases
Authentication scenarios
Mock servers
SDKs
Documentation

And, of course, automated API tests.

Generating tests directly from the specification means developers don't have to manually recreate the contract inside Postman collections or code-based frameworks.

The specification becomes the single source of truth.

At least, that's the theory.

What We Actually Generated

Over several months we processed thousands of endpoints from internal projects, public APIs, and customer specifications.

Across all of them we generated approximately:

12,000+ API test cases
Hundreds of OpenAPI specifications
Thousands of request/response assertions
Countless schema validations

The APIs themselves weren't the interesting part.

The specifications were.

Lesson 1: Most OpenAPI Specs Are Only 80% Complete

One pattern appeared immediately.

Developers document the happy path.

Everything else gets skipped.

Typical examples included:

Missing error responses:

responses:
  200:
    description: Success

But no:

The generator can only create tests for scenarios that exist in the contract.

If the specification ignores validation failures, your generated suite will too.

Lesson 2: Example Values Matter More Than Expected

OpenAPI supports examples.

Many teams ignore them.

Instead of:

name:
  type: string

provide:

name:
  type: string
  example: John Doe

This improves generated:

Requests
Mock responses
Documentation
Test readability

Good examples dramatically improve generated tests.

Lesson 3: Nullable Doesn't Mean Optional

This was responsible for an astonishing number of incorrect assertions.

These are different:

nullable: true

and

required: false

One means:

"The property exists but may contain null."

The other means:

"The property might not exist at all."

Generated tests need to distinguish between both.

Otherwise perfectly valid responses fail incorrectly.

Lesson 4: Authentication Is Frequently Under-Specified

Many APIs defined:

security:
  - bearerAuth: []

Globally.

Then individual endpoints silently behaved differently.

Some required API keys.

Others ignored authentication.

The specification never reflected reality.

Test generation immediately exposed those inconsistencies.

Lesson 5: Schemas Drift Faster Than Teams Realize

One endpoint returned:

{
  "userId": 123
}

Another returned:

{
  "id": 123
}

Both represented the same resource.

Both passed manual testing.

Neither matched the documented contract consistently.

Generated tests highlighted schema drift that had accumulated over years.

What the Generator Could Infer

Modern generators are surprisingly capable.

They can infer:

Required request bodies
Path parameters
Query parameters
Status codes
JSON schema assertions
Authentication requirements

From a single specification.

But they cannot infer business rules.

What Still Requires Human Input

For example:

Should:

discount = 110%

be valid?

The schema cannot answer that.

Should creating an order reduce inventory?

The schema doesn't know.

Should duplicate email addresses return 409 or 422?

Only the business can define that.

Generated tests complement—not replace—human-designed scenarios.

The Biggest Surprise

We expected developers to ask:

"How accurate is the generator?"

Instead they asked:

"Why is the generator failing on my specification?"

In most cases the answer was simple.

The specification wasn't actually valid.

Or it was incomplete.

The tool simply surfaced issues that had always been there.

The Generator Became a Spec Validator

One unexpected outcome was that teams started fixing specifications before generating tests.

The generator effectively became a quality gate.

Instead of discovering contract issues in production, developers discovered them during generation.

That feedback loop was much cheaper.

Common Specification Problems We Found

Across hundreds of specifications, the same issues appeared repeatedly:

Missing required properties
Incorrect data types
Invalid examples
Inconsistent naming
Missing response schemas
Undefined authentication
Duplicate models
Ambiguous nullable fields

Most of these weren't testing problems.

They were specification problems.

What We Changed in Our Workflow

Today our process looks like this:

Design API

↓

Write OpenAPI Specification

↓

Validate Specification

↓

Generate Tests

↓

Add Business Assertions

↓

Run in CI

Notice what's missing?

Manual recreation of test cases.

The contract drives almost everything.

Where Human Testing Still Wins

Automatic generation is fantastic for:

Schema validation
Required fields
Authentication
Parameter combinations
Status codes

Humans still write tests for:

Business rules
Complex workflows
Performance
Security
User journeys

The two approaches work best together.

The Biggest Productivity Gain

The biggest benefit wasn't writing fewer tests.

It was eliminating duplicated work.

Previously developers described the API three times:

OpenAPI specification
Postman collection
Automated tests

Now the specification generates much of the repetitive work automatically.

Teams spend more time testing behavior instead of recreating contracts.

What We'd Recommend to Teams Starting Today

If you're adopting OpenAPI-driven testing:

Treat the specification as code.
Validate it continuously.
Keep examples realistic.
Separate nullable from optional fields.
Add business assertions after generation.
Run generated tests in every CI pipeline.

The better the specification becomes, the better the generated tests become.

Final Thoughts

Generating 12,000 API tests taught us something unexpected.

The hardest part wasn't automation.

It was creating clear, accurate API contracts.

OpenAPI isn't just documentation.

Done well, it becomes the foundation for documentation, SDK generation, mock servers, contract validation, and automated testing.

The quality of your generated tests will almost always reflect the quality of your specification.

If you're interested in this approach, we have documented more about OpenAPI test automation

The best API tests don't start with a testing framework.

They start with a well-written contract.

The Test ID Pattern That Finally Killed Our Flake

Sushant Joshi — Mon, 29 Jun 2026 13:51:06 +0000

Our API test suite went from 4–6% flake to 0.3% the week we changed how we generated entity IDs in test fixtures.

I wish I could tell you we solved our flaky test problem with a fancy testing framework or a massive infrastructure rewrite.

We didn't.

The fix was embarrassingly simple.

We changed how we generated IDs.

That's it.

For months, our CI dashboard looked like this:

Run #1  ✅
Run #2  ❌
Run #3  ✅
Run #4  ❌
Run #5  ✅

Same code.

Same tests.

Different results.

Eventually, we measured the problem.

Our API test suite had a 4–6% flake rate.

That meant roughly one out of every twenty pipeline runs failed for no legitimate reason.

Developers stopped trusting the pipeline.

Failed builds were retried.

Warnings were ignored.

And eventually, real bugs started hiding among false positives.

The culprit?

Randomly generated IDs.

The Flake Source: Random ID Collisions

A typical test looked like this:

const customerId =
  Math.floor(Math.random() * 1000000);

await createCustomer({
  id: customerId
});

Seems harmless.

Until your suite grows.

Eventually:

Hundreds of tests run.
Multiple pipelines run simultaneously.
Parallel workers execute the same code.
Test retries create additional requests.

Sooner or later:

Customer 48291 already exists

appears.

A retry passes.

Nobody investigates.

The flake count grows.

The Problem Wasn't Just Collisions

Random IDs caused other issues too.

Suppose a test failed.

You found this in the database:

Customer: 839174

Questions immediately followed:

Which test created it?
Which pipeline run?
Which branch?
Is it safe to delete?

Nobody knew.

The data had no meaning.

It was just another random number.

Randomness Is the Enemy of Debugging

Random values make systems harder to reason about.

If a failure cannot be reproduced easily, it becomes expensive to investigate.

We wanted IDs that were:

Predictable
Searchable
Namespaced
Easy to clean up

So we introduced a new pattern.

The New Pattern

Instead of:

84738291

we started generating IDs like this:

test-orders-create-001

Or:

pr-482-users-login-003

Or:

ci-198-payments-refund-001

The format became:

<prefix>-<test-name>-<counter>

Every ID suddenly became self-describing.

Why Deterministic IDs Help So Much

Now when a failure happened:

Customer already exists:
pr-482-users-create-001

we immediately knew:

Which PR created it
Which test created it
Which feature it belonged to

Debugging became dramatically easier.

Example Generator

function buildTestId(
  namespace,
  testName,
  sequence
) {
  return `${namespace}-${testName}-${sequence}`;
}

Usage:

const customerId =
  buildTestId(
    'pr-482',
    'create-user',
    1
  );

Result:

pr-482-create-user-1

Simple.

Readable.

Deterministic.

The Biggest Improvement: Test Isolation

This change dramatically improved test isolation.

Previously:

Test A
↓
Creates customer 123

Test B
↓
Creates customer 123

Failure.

Now:

pr-482-test-a-001
pr-482-test-b-001

No collision.

No interference.

No flake.

Parallel API Tests Become Much Safer

Modern CI systems run tests in parallel.

Our pipelines used:

Multiple workers
Multiple containers
Multiple retry attempts

Random IDs weren't enough.

We needed namespaces.

Namespaced IDs

Every test run now receives a namespace.

Examples:

pr-482
build-991
worker-3

IDs become:

pr-482-worker-1-orders-001
pr-482-worker-2-orders-001
pr-482-worker-3-orders-001

Each worker owns its own space.

Parallel execution becomes dramatically safer.

Why Namespaces Matter

Without namespaces:

worker-1 → customer-001
worker-2 → customer-001

Collision.

With namespaces:

worker-1-customer-001
worker-2-customer-001

No collision.

This single change removed a huge percentage of our flaky failures.

The Unexpected Benefit: Easier Debugging

We started seeing logs like:

DELETE customer:
pr-512-login-tests-001

Immediately we knew:

PR number
Test suite
Test case

This sounds small.

It's actually enormous when debugging CI failures.

The Database Cleanup Hook

Deterministic IDs made cleanup incredibly easy.

Previously:

deleteAllTestCustomers();

Terrifying.

What if production-like data exists?

What if another team is using the environment?

Instead we now do:

deleteCustomersByPrefix(
  'pr-482'
);

Or:

deleteOrdersByPrefix(
  'worker-2'
);

Cleanup Became Surgical

Instead of:

Delete everything.

we do:

Delete only records created
by this test run.

This dramatically reduced accidental interference.

Example Cleanup Hook

afterAll(async () => {
  await deleteByPrefix(
    process.env.TEST_NAMESPACE
  );
});

Simple.

Fast.

Safe.

This Also Helps With Retries

Suppose a pipeline fails halfway through.

The cleanup never executes.

Previously:

Random test data remains forever.

Now:

DELETE
WHERE id LIKE 'pr-482%'

Done.

Entire test runs can be cleaned up with one query.

Our Flake Numbers

Before:

4–6% flaky failures

After:

0.3%

The change wasn't perfect.

But it was transformative.

The One Case Where This Pattern Still Loses

Deterministic IDs don't solve every problem.

They still struggle with:

Truly Global Resources

Examples:

Unique email addresses
Shared queues
Third-party systems
Rate-limited APIs

Even with namespaced IDs:

pr-482-john@example.com

might still violate constraints.

Our Fallback Strategy

For these scenarios, we combine:

Namespace
+
Timestamp
+
Random Suffix

Example:

pr-482-user-1721728172-x8a3

This preserves:

Searchability
Cleanup capabilities
Collision resistance

while handling globally unique requirements.

The Pattern We Use Today

Our current generator looks like this:

<namespace>
-
<test-suite>
-
<sequence>

Examples:

pr-811-orders-001
pr-811-orders-002
pr-811-payments-001
worker-3-users-004

Every piece of test data becomes:

Traceable
Isolated
Easy to clean up
Easy to debug

Why This Matters More Than People Think

Flaky tests aren't just annoying.

They slowly destroy trust.

Eventually developers start saying:

Just rerun the pipeline.

That's dangerous.

Because one day:

A real failure looks exactly like another flaky one.

And nobody notices.

Final Thoughts

I spent a long time assuming flaky tests were caused by:

Infrastructure
Networks
Containers
CI providers

Sometimes they are.

But surprisingly often, they're caused by something much simpler:

Bad test data management.

Changing our ID generation strategy didn't eliminate every flaky test.

But it removed an enormous category of failures almost overnight.

And it made our tests easier to debug, easier to clean up, and safer to run in parallel.

If you're trying to improve reliability in your own suites, spend some time looking at your test data strategy.

You might discover that your biggest source of flake isn't your infrastructure.

It's your IDs.

If you're curious about broader testing metrics beyond flaky failures, here's what we measure for API test coverage:

Because sometimes the biggest improvements come from changing something as small as a string format.

The Only 3 Types of Assertions You Need for REST API Tests

Sushant Joshi — Fri, 26 Jun 2026 13:36:29 +0000

I went through 2,400 of our team's API test assertions last month. 91% of them fall into three categories.*

That number surprised me.

Not because it was low.

Because it was so high.

I expected to find dozens of assertion patterns:

Header assertions
Pagination assertions
Security assertions
Performance assertions
Database validations
Custom business rules

Instead, almost everything we had written could be grouped into just three buckets.

When I removed duplicate patterns and categorized the assertions, 91% of them fit into:

Schema Assertions
Identity Assertions
Side-Effect Assertions

The remaining 9%?

Most of them probably shouldn't exist.

If you're building or maintaining REST API tests, understanding these three categories will dramatically simplify how you think about testing.

Why Most API Test Suites Become Hard to Maintain

A lot of test suites grow organically.

A developer writes:

expect(response.status).toBe(200);

Another adds:

expect(response.body.name).toBe('John');

Someone else adds:

expect(response.body.items.length).toBe(3);

Eventually the suite contains thousands of assertions.

Many of them:

Duplicate each other
Validate implementation details
Add maintenance without adding confidence

The goal isn't to write more assertions.

The goal is to write the assertions that actually matter.

1. Schema Assertions (The One Most Tests Skip)

This is the most undervalued type of API assertion.

A schema assertion answers:

Does the response still match the contract?

Suppose your endpoint returns:

{
  "id": 123,
  "name": "John Smith",
  "email": "john@example.com"
}

Tomorrow someone changes it to:

{
  "id": "123",
  "fullName": "John Smith"
}

The endpoint still returns:

200 OK

But consumers may immediately break.

This is why schema assertions matter.

What Schema Assertions Validate

Required fields exist
Data types are correct
Fields have not disappeared
Arrays contain the right structure
Response contracts remain compatible

Example JSON Schema Assertion

expect(response.body).toMatchSchema({
  type: 'object',
  required: ['id', 'name'],
  properties: {
    id: {
      type: 'integer'
    },
    name: {
      type: 'string'
    }
  }
});

Why Teams Skip This

Because checking:

expect(response.status)
  .toBe(200);

feels sufficient.

It isn't.

The biggest API regressions I see are contract changes that still return successful responses.

This is why json schema assertion techniques provide so much value.

Copy-Paste Template: Schema Assertion

expect(response.body)
  .toMatchSchema(schema);

Or:

expect(response.body.id)
  .toEqual(expect.any(Number));

2. Identity Assertions (The Value You Actually Care About)

This is the category most people think of when they hear "API testing."

An identity assertion answers:

Did the API return the correct business value?

Example:

{
  "discount": 20
}

The contract may be valid.

The endpoint may return 200.

But if the expected discount is:

{
  "discount": 50
}

the API is still broken.

Identity Assertions Validate

Business calculations
Field values
Sorting
Filtering
Authorization decisions
Domain rules

Example

expect(response.body.discount)
  .toBe(20);

Or:

expect(response.body.status)
  .toBe('ACTIVE');

Why Identity Assertions Matter

Customers care about values.

They do not care that:

{
  "discount": {
    "type": "integer"
  }
}

is valid.

They care that the discount is correct.

Copy-Paste Template: Identity Assertion

expect(response.body.field)
  .toBe(expectedValue);

Or:

expect(response.body)
  .toEqual(expectedObject);

3. Side-Effect Assertions (The Ones That Prove the Work Happened)

This category gets overlooked surprisingly often.

An API can return:

200 OK

and still fail completely.

Consider:

POST /orders

The endpoint returns success.

But:

The database row wasn't created.
The message wasn't published.
The email wasn't sent.

The business process failed.

Side-Effect Assertions Validate

Database writes
Queue messages
Event publication
Emails
Audit logs
Third-party integrations

Example Database Assertion

expect(orderRepository.find(orderId))
  .not.toBeNull();

Example Queue Assertion

expect(queue.contains(orderCreatedEvent))
  .toBe(true);

Why Side Effects Matter

Many APIs exist solely to trigger something else.

The response itself is often the least important part.

For example:

POST /payments

Nobody cares about:

{
  "success": true
}

What matters is:

Was the payment captured?
Was the invoice generated?
Was the receipt sent?

Copy-Paste Template: Side-Effect Assertion

expect(databaseRecord)
  .toExist();

Or:

expect(publishedEvent)
  .toBeDefined();

The 9% That Didn't Fit

After categorizing our assertions, around 9% remained.

Examples included:

expect(response.body.items.length)
  .toBe(5);

expect(response.body.createdAt)
  .toBe('2026-07-01');

expect(response.body.version)
  .toBe('1.2.8');

Many of these were:

Brittle
Overly specific
Tied to implementation details

Why Most of Them Should Be Deleted

Ask yourself:

If this assertion failed tomorrow, would users actually notice?

If the answer is:

Probably not.

Delete it.

A surprising amount of maintenance comes from assertions that don't provide meaningful confidence.

Bad Assertions

expect(responseTime)
  .toBe(183);

expect(itemCount)
  .toBe(17);

expect(timestamp)
  .toEqual('2026-07-28T08:00:00Z');

These tend to break constantly.

Better Assertions

expect(responseTime)
  .toBeLessThan(500);

expect(itemCount)
  .toBeGreaterThan(0);

expect(timestamp)
  .toBeDefined();

These are more resilient.

The Assertion Pyramid I Recommend

Whenever I write a new API test, I ask three questions.

First

Does the response still match the contract?

→ Schema Assertion.

Second

Did the API return the correct business value?

→ Identity Assertion.

Third

Did the system actually perform the work?

→ Side-Effect Assertion.

Most useful tests contain at least one of these categories.

Many contain all three.

Example:

expect(response.body)
  .toMatchSchema(userSchema);

expect(response.body.name)
  .toBe('John');

expect(databaseUser)
  .toExist();

Three assertions.

Three different guarantees.

High confidence.

Low maintenance.

Final Thoughts

When we reduced our thousands of assertions down to categories, we realized something important:

Most API testing is simpler than we make it.

The majority of valuable assertions answer only three questions:

Is the contract still valid?
Is the business value correct?
Did the side effect happen?

Everything else should justify its existence.

If an assertion doesn't increase confidence or protect against meaningful regressions, it may not belong in the suite.

That's why I now start every API test by deciding which of these three categories I'm actually trying to validate.

If you'd like to go deeper into building maintainable API suites, I highly recommend the REST API testing best practices guide:

Because the best API tests aren't the ones with the most assertions.

They're the ones with the right assertions.

API Versioning Tests: 3 Cases Almost Every Team Misses

Sushant Joshi — Thu, 25 Jun 2026 13:28:32 +0000

Our v2 API broke v1 clients in production on a Wednesday because nobody had tested the case where a v1 client received a v2-shaped response after a load-balancer routing rule was tweaked.*

The outage lasted less than an hour.

The damage lasted much longer.

A mobile application started failing silently because it expected a field named:

{
  "customerName": "John Smith"
}

but suddenly received:

{
  "fullName": "John Smith",
  "customerStatus": "Active"
}

Nothing was technically wrong with the v2 API.

The issue was that some v1 clients unexpectedly started receiving v2 responses.

The routing change looked harmless during deployment.

No tests covered that scenario.

This experience taught us an important lesson:

Versioning an API isn't just about creating /v2.

It's about ensuring every version continues behaving exactly as its consumers expect.

Most teams write happy-path tests for each version independently. Very few teams write tests that validate interactions between versions, backward compatibility guarantees, and accidental cross-version behavior.

These are the three categories of API versioning tests that almost every team misses.

Why Versioning Tests Matter More Than Ever

Modern APIs evolve constantly.

New versions introduce:

Additional fields
Renamed properties
New authentication requirements
Performance improvements
Business logic changes

None of these are inherently bad.

The risk comes from assuming:

"Since v2 works, v1 must still work too."

That assumption causes many production incidents.

Consumers don't upgrade immediately.

Some clients may continue using older versions for months or years.

A single breaking change can impact:

Mobile applications
Partner integrations
Third-party SDKs
Internal microservices
Reporting systems

This is why API version testing requires more than endpoint-level validation.

It requires contract validation across versions.

1. The Dual-Version Contract Test

This is the most valuable test we added after our routing incident.

The concept is simple.

Every request that exists in both versions should be tested against both versions simultaneously.

For example:

GET /customers/123

is executed against:

/v1/customers/123
/v2/customers/123

The goal isn't necessarily to make responses identical.

The goal is to validate contractual guarantees.

What the Test Checks

The test should answer:

Does v1 still return every promised field?
Are field types unchanged?
Are required properties still present?
Are deprecated properties still available?
Are status codes consistent?

This catches situations where implementation changes accidentally impact previous versions.

The Cross-Version Routing Test

Now comes the scenario most teams never test.

What happens if:

v1 request → v2 service

v2 request → v1 service

This sounds unlikely.

In distributed systems, it happens more often than people expect.

Examples include:

Load-balancer misconfigurations
Canary deployments
Service discovery issues
Proxy rules
API gateway changes

Your system should either:

Continue functioning safely.
Return a clear version error.

It should never fail unpredictably.

Example Test Matrix

Request	Expected Response
v1 → v1	Success
v2 → v2	Success
v1 → v2	Graceful handling
v2 → v1	Graceful handling

Most teams only test the first two rows.

The last two are where incidents happen.

2. Deprecated-Field Assertions (Yes, You Should Assert They Exist)

This recommendation often surprises people.

Many engineers believe deprecated fields should gradually disappear from tests.

In reality, the opposite is usually true.

If a field is marked as deprecated but remains part of the contract, your tests should explicitly assert its existence.

A Real Example

Version 1 returns:

{
  "customerName": "John Smith"
}

Version 2 introduces:

{
  "fullName": "John Smith"
}

The team marks:

customerName

as deprecated.

Everything seems fine.

Six months later, an engineer removes:

customerName

assuming nobody uses it anymore.

Unfortunately:

Mobile apps still depend on it.
A partner integration still parses it.
Several reports still consume it.

Production breaks.

The Better Approach

Your test should explicitly verify:

expect(response.customerName)
  .toBeDefined();

Even though the field is deprecated.

Deprecation means:

"Planned for removal."

It does not mean:

"Safe to remove today."

Add Expiration Dates

The best practice is to combine assertions with metadata.

For example:

deprecated:
  field: customerName
  remove_after: 2027-01-01

Now tests can remind teams:

Field still exists.
Deprecation window is active.
Removal date is approaching.

This creates a controlled migration process.

3. The Header-Based vs URL-Based Version Test Pattern

API versioning strategies vary significantly.

The two most common approaches are:

URL Versioning

/api/v1/orders
/api/v2/orders

Header Versioning

Accept: application/vnd.company.v1+json
Accept: application/vnd.company.v2+json

Both approaches require different testing strategies.

URL-Based Testing

This approach is relatively straightforward.

Test suites can easily target:

/v1
/v2
/v3

Contract separation is clear.

However, routing issues become important.

You need tests that verify:

URL rewrites
Gateway behavior
Redirect handling

Header-Based Testing

Header-based versioning creates additional complexity.

The same endpoint:

GET /orders

can return completely different responses depending on headers.

Testing must validate:

Missing version headers
Invalid versions
Default versions
Unsupported versions
Version negotiation behavior

A Test Pattern We Like

For every endpoint:

No header
v1 header
v2 header
invalid version header
future version header

This often reveals unexpected behavior.

Examples include:

Silent fallbacks
Incorrect defaults
Cached responses
Improper content negotiation

Snapshot Diffs Between v1 and v2 Responses

One of the simplest and most effective versioning techniques is snapshot comparison.

For a representative request:

GET /customers/123

Capture:

v1_response.json
v2_response.json

Then generate structural diffs.

Example Diff

{
- "customerName": "John Smith",
+ "fullName": "John Smith",

+ "customerStatus": "Active"
}

This provides immediate visibility into:

Added fields
Removed fields
Type changes
Structural changes

Why Snapshots Matter

Humans often miss small changes during code review.

Examples:

- "123"
+ 123

This appears minor.

For some consumers, it's a breaking change.

Snapshot testing catches these differences immediately.

Contract Snapshots Are Better Than Payload Snapshots

Avoid comparing exact values whenever possible.

Compare:

Field names
Types
Required properties
Structure

This reduces noise while preserving meaningful change detection.

The CI Gate That Catches Accidental Breaking Changes

The most effective versioning tests don't run only during releases.

They run continuously.

Every pull request should answer one question:

Does this change accidentally break a previous contract?

What the CI Gate Should Check

Schema Changes

Required fields removed
Data types changed
Enum values changed

Response Changes

Deprecated fields removed
Status codes changed
Error structures modified

Version Behavior

Routing changes
Header handling
Content negotiation

Example CI Pipeline

Build
↓
Run version contract tests
↓
Generate snapshots
↓
Compare against baseline
↓
Fail if breaking changes detected

The process is simple.

The impact is enormous.

The Breaking Changes You Want to Catch

Examples include:

- customerId: integer
+ customerId: string

- status: Active
+ state: Active

- customerName

These changes may look harmless to developers.

For API consumers, they can be catastrophic.

Building a Sustainable Version Testing Strategy

A practical strategy usually includes four layers.

Layer 1: Endpoint Tests

Validate each version independently.

Layer 2: Contract Tests

Validate compatibility between versions.

Layer 3: Snapshot Diffs

Highlight structural changes.

Layer 4: CI Gates

Prevent accidental regressions from reaching production.

This layered approach dramatically reduces version-related incidents.

It also provides confidence when introducing new versions.

Why Teams Still Miss These Tests

Versioning issues often occur because teams think in terms of endpoints.

Consumers think in terms of contracts.

Those perspectives are different.

An endpoint can function perfectly while still breaking consumers.

The goal of version testing is not simply:

"Does the endpoint return 200?"

The goal is:

"Can every supported consumer continue working safely?"

That question requires broader testing.

Final Thoughts

The most painful API incidents I've seen were not caused by servers crashing or databases failing.

They were caused by small compatibility assumptions.

A renamed field.

A removed property.

A routing rule.

A header negotiation bug.

These changes rarely trigger alarms immediately.

Instead, they quietly break integrations that nobody remembered to test.

Strong backward compatibility tests and thoughtful deprecation testing dramatically reduce those risks.

Versioning isn't just about introducing new APIs.

It's about preserving trust with every client that still depends on older ones.

If your team is building or maintaining multiple API versions, adopting the contract-testing approach we use for versioned APIs can provide a much safer foundation for change:

Because in versioned systems, the bugs that hurt the most are usually the ones nobody thought to test.

Building a Self-Healing Test Suite ~ My Honest Version

Sushant Joshi — Thu, 18 Jun 2026 11:59:56 +0000

"Self-healing" is one of those phrases that means three different things depending on which vendor's homepage you read last.*

In one product, it means updating locators when a button moves.

In another, it means regenerating entire test cases from production traffic.

In yet another, it means using AI to automatically rewrite assertions whenever tests fail.

The problem isn't that these claims are entirely wrong.

The problem is that "self-healing" has become a catch-all marketing term that often creates unrealistic expectations.

Teams hear "self-healing tests" and imagine a future where test failures magically disappear while quality remains intact.

Reality is more nuanced.

Modern self-healing technology can dramatically reduce maintenance effort, especially for API and integration testing. However, there are clear boundaries between what can be safely repaired and what still requires human judgment.

This article explores the practical reality of building a self-healing test suite, including what it can fix, what it cannot fix, and where automation should stop and ask for approval.

What Self-Healing Actually Covers (And the 3 Things It Can't Fix)

Before discussing implementations, it's important to define what self-healing means in practice.

At its core, a self-healing system attempts to determine whether a test failed because:

The application changed legitimately.
The test became outdated.
The application is actually broken.

The goal is to automatically repair tests only in scenario #2.

The challenge is distinguishing between all three.

What Self-Healing Can Usually Fix

The most effective self-healing systems focus on structural changes.

Examples include:

Renamed JSON fields
Additional optional response properties
Endpoint path updates
Schema version changes
Authentication token format changes
Header name changes
Parameter renaming

These changes often represent intentional application evolution rather than defects.

Because they are structural, they can frequently be analyzed and corrected automatically.

What Self-Healing Cannot Reliably Fix

There are three categories that remain extremely difficult to automate safely.

1. Business Logic Changes

Consider an API that calculates discounts.

Yesterday:

{
  "discount": 20
}

Today:

{
  "discount": 10
}

Did the business rule change?

Or is there a bug?

The test failure alone cannot answer that question.

A healing engine should never guess.

2. Missing Business Outcomes

Imagine a checkout API that suddenly stops creating orders.

The response format remains identical.

All fields still exist.

Yet the core business outcome is gone.

No amount of structural healing can identify the intended business behavior.

3. Security-Related Failures

Authentication, authorization, and access-control failures should never be auto-corrected.

If an API suddenly returns sensitive data to unauthorized users, automatic healing could accidentally hide a critical security issue.

Security failures require investigation, not repair.

A Worked Example: A Renamed JSON Field

Let's look at a realistic scenario.

An API originally returned:

{
  "customerId": 123,
  "customerName": "John Smith",
  "status": "Active"
}

The test asserted:

expect(response.customerName).toEqual("John Smith");

Several months later, the API team introduces a naming convention update.

The response becomes:

{
  "customerId": 123,
  "fullName": "John Smith",
  "status": "Active"
}

The test now fails.

Traditional automation requires:

Failure investigation
Root cause analysis
Test update
Code review
Commit
Redeployment

For a single field rename, that's a surprisingly expensive workflow.

What a Self-Healing Engine Sees

A modern healing engine analyzes several signals.

It notices:

Response schema remains largely unchanged
Field value still exists
Data type matches
Object structure remains identical
Similar semantic meaning between names

It may calculate:

Signal	Confidence
Data type match	100%
Position similarity	100%
Value pattern match	100%
Semantic similarity	92%
Overall confidence	96%

The system now has strong evidence that:

customerName

became

fullName

rather than the application breaking.

The Diff the Healer Proposed and the Diff I Accepted

One of the biggest misconceptions about auto-fix API tests is that they should operate silently.

In reality, silent modifications can become dangerous very quickly.

A better approach is proposing changes first.

The failing assertion looked like this:

expect(response.customerName)
  .toEqual("John Smith");

The healer proposed:

- expect(response.customerName)
+ expect(response.fullName)
    .toEqual("John Smith");

At first glance, this appears obvious.

However, the review process still matters.

The engineer can quickly verify:

Was the field intentionally renamed?
Is the value equivalent?
Does the business meaning remain unchanged?

In this case, the answer was yes.

The change was accepted.

The test passed immediately.

Total maintenance effort: less than one minute.

Why Human Approval Still Matters

Now imagine this proposed change:

- accountBalance
+ availableCredit

Those fields may look similar.

They are not the same thing.

Automatically accepting that modification could introduce serious defects into the test suite.

Self-healing should reduce human work, not eliminate human oversight.

Confidence Thresholds — When to Auto-Apply vs Ask

The most effective resilient test suite implementations use confidence scoring.

Not every proposed repair deserves the same level of trust.

A useful model might look like this:

Auto-Apply (95–100%)

Examples:

Added optional response fields
Header renames
Query parameter aliases
Non-breaking schema extensions

Risk is extremely low.

Automation can safely proceed.

Request Approval (75–95%)

Examples:

Field renames
Schema restructures
Endpoint migrations
Nested object movement

These changes are usually safe but deserve a quick review.

A human can validate them in seconds.

Require Manual Investigation (<75%)

Examples:

Value changes
Business rule differences
Calculation differences
Authorization changes

At this point, the system lacks enough confidence.

Automatic repair becomes risky.

Why Confidence Matters More Than AI

Many discussions focus on whether healing uses AI, machine learning, or rules.

The more important question is:

How confident is the system in the proposed repair?

Even sophisticated models make mistakes.

A strong healing framework acknowledges uncertainty and surfaces it rather than hiding it.

The goal isn't to appear intelligent.

The goal is to avoid masking defects.

The Category of Failure Where You Should Never Auto-Heal

If there is one principle every engineering team should adopt, it's this:

Never auto-heal business assertions.

Let's examine why.

Suppose a tax calculation API should return:

{
  "tax": 15.25
}

A deployment causes the API to return:

{
  "tax": 12.75
}

The test fails.

A dangerous healing engine might decide:

"The value changed. Let's update the expected result."

The test now passes.

The bug survives.

The entire purpose of testing has been defeated.

The Cost of False Positives

Many organizations focus on reducing false failures.

That's important.

However, the larger risk is introducing false success.

A false failure wastes time.

A false success ships defects.

Given the choice, every mature QA organization should prefer a small amount of investigation over silently hiding a production issue.

Where Self-Repair Works Best

The strongest use cases for test self repair involve maintenance-heavy changes that provide little business value.

Examples include:

Schema evolution
Endpoint versioning
Contract updates
Response restructuring
Naming convention changes

These changes generate noise rather than insight.

Removing that noise allows engineers to focus on genuine quality risks.

Building a Practical Self-Healing Strategy

Organizations often ask whether every test should be self-healing.

The answer is no.

A layered approach works far better.

Layer 1: Contract Tests

Allow healing.

These tests validate structure.

They're ideal candidates for automated repair.

Layer 2: Integration Tests

Allow limited healing with approval.

These tests validate interactions between services.

Some repairs are safe.

Others require review.

Layer 3: Business Validation Tests

No automatic healing.

These tests exist specifically to detect behavioral changes.

Their assertions should remain under human control.

Layer 4: Security Tests

Never heal automatically.

Security failures should always trigger investigation.

Final Thoughts

The promise of self-healing tests isn't that failures disappear.

The real value is that teams spend less time fixing tests that were never providing meaningful feedback in the first place.

A renamed field should not consume hours of engineering effort.

A schema evolution should not trigger dozens of manual pull requests.

A version upgrade should not create a maintenance backlog.

Modern self-healing systems can eliminate much of that friction while preserving confidence in the test suite.

The key is understanding where automation helps and where human judgment remains essential.

If a healing engine is modifying assertions tied to business outcomes, it's probably going too far.

If it's repairing structural changes while providing transparency and confidence scoring, it's likely delivering real value.

For a deeper technical explanation of how self-healing actually works under the hood

The most effective self-healing strategy isn't about making tests smarter.

It's about making maintenance quieter while keeping quality signals loud.