DEV Community: Smeet Gohel

API Error Codes: A Test Suite Pattern I Stole from Stripe

Smeet Gohel — Fri, 26 Jun 2026 13:39:06 +0000

Read Stripe's API reference for an hour and you'll notice every endpoint has a complete enumerated list of error codes with example payloads. Then look at your own API.

The contrast is hard to ignore.

Stripe's API documentation treats errors as first-class citizens. Every endpoint clearly documents not only the happy path but also every expected failure, complete with structured error codes, descriptions, HTTP status codes, and example responses.

Now compare that to many APIs in production.

You might find a generic list of HTTP status codes somewhere in the documentation, but business-specific errors are often buried inside controller logic, scattered across wiki pages, or simply undocumented. The test suite isn't much better—there are dozens of happy-path tests, but only a handful of negative scenarios.

That imbalance creates problems for everyone involved:

Developers don't know which errors are expected.
Frontend teams can't reliably handle failures.
QA engineers miss important negative cases.
Refactoring accidentally changes error responses without anyone noticing.

A few years ago, I borrowed a simple idea from Stripe's documentation and turned it into a testing strategy.

Instead of treating error responses as exceptions, we created an error-code catalog and made it the foundation of our negative test suite.

The result wasn't just better API error code testing—it also improved documentation, simplified maintenance, and made API contracts far more consistent.

Here's how the pattern works.

Why Error Responses Are Part of the API Contract

When people think about API testing, they naturally focus on successful responses.

Typical assertions include:

HTTP 200 OK
HTTP 201 Created
Correct JSON payload
Required fields
Business calculations

Negative testing often gets much less attention.

Maybe there are a few tests for:

Invalid authentication
Missing required fields
Unknown resources

Beyond that, many APIs rely on manual testing or hope that the framework handles everything correctly.

The problem is that real users encounter failures just as often as successful requests.

Examples include:

Customer account locked
Payment declined
Coupon expired
Inventory unavailable
Duplicate registration
Subscription canceled
Rate limit exceeded

These aren't exceptional scenarios.

They're expected business outcomes.

Treating them as first-class API contracts changes how you design both documentation and tests.

The Error-Code Catalog as a Test Input

The first step is creating a centralized catalog of every business error the API can intentionally return.

A simplified example might look like this:

errors:
  USER_NOT_FOUND:
    httpStatus: 404
    message: User not found

  EMAIL_ALREADY_EXISTS:
    httpStatus: 409
    message: Email already exists

  INVALID_TOKEN:
    httpStatus: 401
    message: Invalid authentication token

  PAYMENT_DECLINED:
    httpStatus: 402
    message: Payment declined

  ORDER_ALREADY_SHIPPED:
    httpStatus: 409
    message: Order cannot be modified

This catalog becomes far more than documentation.

It becomes an executable specification.

Instead of asking:

"What errors should this endpoint return?"

the answer already exists in one authoritative location.

Every new business error must be added here before it reaches production.

That single requirement dramatically improves consistency.

Why a Catalog Helps

Without a catalog:

Documentation drifts.
Tests become incomplete.
Frontend teams discover errors by accident.
Reviewers overlook breaking changes.

With a catalog:

Every error is documented.
Every error becomes testable.
Every API consumer sees the same contract.

The catalog becomes the foundation for automation.

One Test per Error Code, Generated from the Catalog

Once the catalog exists, generating negative tests becomes surprisingly straightforward.

Rather than manually writing dozens of repetitive tests, a generator simply iterates through every defined error.

Conceptually:

for (const errorCode of catalog) {
    generateNegativeTest(errorCode);
}

Each generated test validates four things:

The expected HTTP status
The error code
The error message
The response schema

Consider EMAIL_ALREADY_EXISTS.

The generated scenario might:

Create a user.
Attempt to create the same user again.
Verify the response:

{
  "code": "EMAIL_ALREADY_EXISTS",
  "message": "Email already exists"
}

The implementation differs depending on the framework, but the testing philosophy remains the same:

Every documented error deserves exactly one corresponding test.

As new error codes are introduced, new tests appear automatically.

No engineer has to remember to write them.

Why This Scales Better

Imagine your API exposes:

150 endpoints
90 business error codes

Maintaining those manually quickly becomes tedious.

Generation solves two maintenance problems simultaneously:

Missing tests
Duplicate effort

Instead of asking developers to remember every negative case, the catalog guarantees baseline coverage.

Engineers can then focus on more complex business workflows rather than repetitive validation tests.

The Shape Assertion That Prevents Silent Error Drift

One lesson we learned very early was this:

Checking only the HTTP status is almost useless.

Imagine an endpoint originally returns:

{
  "code": "USER_NOT_FOUND",
  "message": "User not found",
  "requestId": "abc123"
}

Months later, someone refactors the global exception handler.

The response becomes:

{
  "error": "User not found"
}

The HTTP status is still:

Many tests still pass.

But every client expecting the original response contract is now broken.

This is known as silent error drift.

Nothing appears wrong until consumers start failing.

The Solution: Shape Assertions

Every negative test also validates the response structure.

Example:

expect(response.body).toEqual({
    code: expect.any(String),
    message: expect.any(String),
    requestId: expect.any(String)
});

Notice that we're not only validating values.

We're validating the schema itself.

That single assertion protects every API consumer from accidental response changes.

Why This Matters

Consumers often depend on:

Error codes
Localization keys
Correlation IDs
Documentation URLs

Removing any of these fields can become a breaking API change even though the HTTP status remains correct.

Schema validation catches those problems immediately.

Keeping the Catalog in Sync with the Code (Code Generation)

The obvious concern is maintenance.

If engineers must manually update both:

Source code
Error catalog

the catalog eventually becomes outdated.

The solution is code generation.

Most applications already define errors centrally.

For example:

export enum ErrorCode {
    USER_NOT_FOUND,
    INVALID_TOKEN,
    PAYMENT_DECLINED,
    EMAIL_ALREADY_EXISTS
}

A simple generation step can produce:

API documentation
OpenAPI components
Markdown reference tables
Test inputs
SDK constants

All from the same source.

Now there's only one place where error definitions live.

Everything else is generated automatically.

Benefits of Codegen

This approach creates several advantages:

Documentation Never Falls Behind

As soon as a new error appears in code, documentation updates automatically.

Generated Tests Stay Current

No manual synchronization required.

API Consumers Stay Aligned

Client SDKs can reference the same constants used by the server.

Code Reviews Become Easier

Adding a new business error becomes highly visible because it affects generated documentation and tests.

The Two Error Codes We Deliberately Don't Test (And Why)

Although our negative suite covers nearly every business error, there are two categories we intentionally exclude.

1. Generic Internal Server Errors

Example:

500 Internal Server Error

These represent unexpected failures.

They're not part of normal business behavior.

Rather than intentionally triggering every possible internal exception, we verify:

Sensitive details aren't exposed
Generic messages are returned
Correlation IDs exist
Logging occurs correctly

Testing every possible server failure adds little value.

Testing the response contract provides much greater return.

2. Infrastructure Failures

Examples include:

Database unavailable
Network partition
DNS outage
Message broker failure
Cloud storage unavailable

These failures belong to resilience testing rather than standard API automation.

They are better validated using:

Chaos engineering
Fault injection
Infrastructure testing
Disaster recovery exercises

Mixing infrastructure scenarios into routine API negative tests usually creates unstable pipelines.

Keeping them separate results in cleaner and more reliable automation.

Additional Benefits We Didn't Expect

Once the catalog became part of our development process, several unexpected improvements appeared.

More Consistent APIs

Every endpoint used the same response format.

Better Frontend Development

Frontend teams no longer guessed which errors could occur.

Simpler Documentation

Error references stayed synchronized automatically.

Cleaner Pull Requests

Adding a new error became an explicit design decision rather than an implementation detail.

Better QA Coverage

Negative scenarios became just as visible as successful ones.

Final Thoughts

Most engineering teams invest heavily in testing successful requests while treating failures as secondary concerns.

Stripe demonstrates a different philosophy.

Errors are documented, standardized, and treated as an integral part of the public API contract.

Building an error-code catalog allowed us to adopt that same mindset.

Instead of manually maintaining dozens of repetitive error response testing scenarios, we generated them from a single source of truth.

Combined with response schema validation and code generation, the approach dramatically reduced maintenance while increasing confidence that every documented failure behaved exactly as expected.

If your API already has a growing collection of business errors, consider creating a centralized catalog before the list becomes unmanageable.

The investment is relatively small, but the payoff in documentation quality, test coverage, and long-term maintainability is substantial.

If you'd like to explore how automated API testing can support this approach, you can spin up a free trial to try this catalog pattern and see how generated negative tests, schema validation, and API contracts work together in practice.

5 OpenAPI Mistakes That Break Every Test Generator I've Tried

Smeet Gohel — Thu, 25 Jun 2026 13:28:37 +0000

I fed the same OpenAPI spec into four different test generators last weekend. All four failed on the same five things.

That wasn't supposed to happen.

The generators came from different vendors. They used different parsing engines, different test-generation strategies, and different AI capabilities. Some focused on contract testing. Others emphasized intelligent test creation and edge-case discovery.

Yet all four stumbled on the exact same sections of the specification.

The surprising part wasn't that the tools failed.

The surprising part was that every failure could be traced back to issues inside the OpenAPI document itself.

Over the years, I've learned that most API test generation problems aren't actually tool problems. They're specification quality problems.

A clean OpenAPI specification can generate hundreds of useful tests automatically.

A flawed specification can confuse even the smartest generator.

If you're using OpenAPI to generate tests, SDKs, mocks, documentation, or validation suites, these are the five most common OpenAPI mistakes I've seen repeatedly—and why they cause so much trouble.

1. Missing Required Fields on Nested `$ref` Objects

This is easily one of the most overlooked issues in OpenAPI specifications.

Consider the following schema:

Customer:
  type: object
  properties:
    id:
      type: integer
    address:
      $ref: '#/components/schemas/Address'

The referenced schema looks like:

Address:
  type: object
  properties:
    street:
      type: string
    city:
      type: string

Looks harmless.

Now imagine the actual API requires:

{
  "street": "Main Street",
  "city": "London"
}

but the schema never specifies:

required:
  - street
  - city

The generator now assumes both fields are optional.

Why Test Generators Struggle

When generating positive and negative test cases, the tool needs to know:

Which fields must exist
Which fields may be omitted
Which omissions should trigger validation failures

Without required declarations, generators often create:

{
  "address": {}
}

and treat the payload as valid.

The resulting tests provide little value because they don't reflect real application behavior.

Best Practice

Always define required fields explicitly at every schema level.

Even when schemas are reused via $ref, each referenced object should clearly identify mandatory properties.

Never assume the generator will infer business intent.

Schemas should be unambiguous.

2. `anyOf` With No Discriminator (The One That Hurts Most)

If I had to choose the single most problematic specification pattern, this would be it.

Consider:

Pet:
  anyOf:
    - $ref: '#/components/schemas/Cat'
    - $ref: '#/components/schemas/Dog'

This tells the generator:

"The payload may match Cat or Dog."

Sounds reasonable.

The problem is that the generator has no reliable way to determine which schema should be used for a specific test case.

Why It Breaks Generation

Imagine:

Cat:
  properties:
    name:
      type: string

Dog:
  properties:
    name:
      type: string

Now the payload:

{
  "name": "Buddy"
}

matches both schemas.

The generator faces several questions:

Is this a Cat?
Is this a Dog?
Should both test paths be created?
Which validation rules apply?

Different tools make different assumptions.

Most assumptions are wrong.

The Fix

Use discriminators whenever possible.

Example:

Pet:
  oneOf:
    - $ref: '#/components/schemas/Cat'
    - $ref: '#/components/schemas/Dog'
  discriminator:
    propertyName: petType

Now responses become:

{
  "petType": "Dog",
  "name": "Buddy"
}

The ambiguity disappears.

Test generation becomes deterministic.

This single change often improves generated coverage dramatically.

3. Untyped Query Parameters (String vs Integer Ambiguity)

Another surprisingly common issue involves query parameters.

Consider:

parameters:
  - name: page
    in: query

Looks simple.

Unfortunately, the schema never specifies the data type.

Why This Creates Problems

A generator needs type information to create meaningful tests.

Should it generate:

?page=1

?page=abc

?page=true

Without explicit typing, every option becomes theoretically valid.

Different generators handle this differently:

Some assume strings
Some guess based on naming
Some generate everything
Some skip validation entirely

None of these approaches are ideal.

The Hidden Impact

This issue affects more than positive tests.

It also impacts:

Boundary testing
Negative testing
Fuzz testing
Schema validation

For example:

schema:
  type: integer
  minimum: 1
  maximum: 100

enables generators to automatically create:

Without typing, those valuable edge cases disappear.

Best Practice

Always define parameter schemas completely.

Example:

parameters:
  - name: page
    in: query
    schema:
      type: integer
      minimum: 1

The more constraints provided, the better the generated tests become.

4. Example Values That Don't Match the Schema

This problem creates some of the most confusing failures.

Imagine:

type: integer
example: "123"

type: boolean
example: "true"

The examples look correct at first glance.

But they violate the schema definition.

Why This Matters

Most generators rely heavily on example values.

Examples help generate:

Request payloads
Mock data
Positive test cases
Sample assertions

When examples contradict schemas, the generator receives conflicting instructions.

The schema says:

type: integer

The example says:

"123"

Which one should the tool trust?

What Usually Happens

Different generators respond differently:

Generator A

Uses schema definition.

Generator B

Uses example value.

Generator C

Attempts to coerce data types.

Generator D

Fails validation completely.

The result is inconsistent behavior across platforms.

Best Practice

Treat examples as executable documentation.

Every example should validate successfully against its schema.

A useful review process is:

Validate schema
Validate examples
Validate generated payloads

All three should agree.

5. Auth Scheme Defined Globally but Overridden Per-Path

Authentication definitions often create subtle specification issues.

Consider:

security:
  - bearerAuth: []

This applies globally.

Everything looks fine.

Later, a specific endpoint introduces:

paths:
  /public-data:
    get:
      security: []

This explicitly removes authentication.

Still valid.

The trouble begins when specifications contain dozens or hundreds of endpoints.

Why Test Generators Fail Here

Generators must determine:

Which endpoints require authentication
Which endpoints are public
Which credentials to attach
Which negative auth scenarios to generate

Conflicting security definitions make this surprisingly difficult.

I've seen generators:

Add tokens to public endpoints
Skip tokens on protected endpoints
Generate invalid authentication tests
Ignore overrides completely

The Real Problem

Many teams inherit APIs over several years.

Security definitions evolve.

Documentation evolves.

Endpoints move between versions.

Eventually, the specification contains multiple overlapping security patterns.

Humans can usually understand the intent.

Generators often cannot.

Best Practice

Maintain a clear authentication strategy.

Review:

Global security settings
Path-level overrides
Operation-level overrides

Whenever possible, minimize exceptions.

The fewer special cases you create, the easier automated tooling becomes.

Why These Mistakes Keep Appearing

What's interesting is that none of these issues are technically invalid OpenAPI.

Many specifications containing these patterns will still:

Render documentation correctly
Generate SDKs successfully
Pass schema validation

The problems emerge only when advanced automation enters the picture.

Test generation requires much higher precision than documentation generation.

Documentation can tolerate ambiguity.

Automated testing cannot.

A Simple Validation Checklist

Before feeding an OpenAPI document into any generator, review the following:

Schema Definitions

Are required fields defined?
Are nested references complete?
Are enums constrained properly?

Polymorphism

Does every anyOf or oneOf include a discriminator?
Can payload types be identified deterministically?

Parameters

Are query parameters typed?
Are constraints defined?

Examples

Do examples validate against schemas?
Are example payloads realistic?

Authentication

Are security rules consistent?
Are overrides intentional?

This five-minute review can save hours of debugging later.

Final Thoughts

After testing multiple generators against the same API specification, one conclusion became obvious:

Most test-generation failures begin long before the generator runs.

They begin when the specification is written.

The better your OpenAPI document, the better your generated tests, mocks, SDKs, documentation, and validation suites become.

Tools continue to improve every year.

AI-assisted generation is becoming increasingly capable.

Yet even the most advanced platform struggles when the specification contains ambiguity, conflicting definitions, or incomplete schema information.

If you're planning to generate automated API tests from your OpenAPI specification, spending time on specification quality is often the highest-return investment you can make.

For a deeper walkthrough on how to generate API tests from OpenAPI without these traps, visit:

https://totalshiftleft.ai/blog/how-to-generate-api-tests-from-openapi

A clean specification doesn't just improve documentation.

It becomes the foundation for every automation layer built on top of it.

DEV Community: Smeet Gohel

API Error Codes: A Test Suite Pattern I Stole from Stripe

Why Error Responses Are Part of the API Contract

The Error-Code Catalog as a Test Input

Why a Catalog Helps

One Test per Error Code, Generated from the Catalog

Why This Scales Better

The Shape Assertion That Prevents Silent Error Drift

The Solution: Shape Assertions

Why This Matters

Keeping the Catalog in Sync with the Code (Code Generation)

Benefits of Codegen

Documentation Never Falls Behind

Generated Tests Stay Current

API Consumers Stay Aligned

Code Reviews Become Easier

The Two Error Codes We Deliberately Don't Test (And Why)

1. Generic Internal Server Errors

2. Infrastructure Failures

Additional Benefits We Didn't Expect

More Consistent APIs

Better Frontend Development

Simpler Documentation

Cleaner Pull Requests

Better QA Coverage

Final Thoughts

5 OpenAPI Mistakes That Break Every Test Generator I've Tried

1. Missing Required Fields on Nested $ref Objects

Why Test Generators Struggle

Best Practice

2. anyOf With No Discriminator (The One That Hurts Most)

Why It Breaks Generation

The Fix

3. Untyped Query Parameters (String vs Integer Ambiguity)

Why This Creates Problems

The Hidden Impact

Best Practice

4. Example Values That Don't Match the Schema

Why This Matters

What Usually Happens

Generator A

Generator B

Generator C

Generator D

Best Practice

5. Auth Scheme Defined Globally but Overridden Per-Path

Why Test Generators Fail Here

The Real Problem

Best Practice

Why These Mistakes Keep Appearing

A Simple Validation Checklist

Schema Definitions

Polymorphism

Parameters

Examples

Authentication

Final Thoughts

1. Missing Required Fields on Nested `$ref` Objects

2. `anyOf` With No Discriminator (The One That Hurts Most)