DEV Community

Cover image for How to Diff OpenAPI Specs and Block Breaking Changes in CI
Hassann
Hassann

Posted on • Originally published at apidog.com

How to Diff OpenAPI Specs and Block Breaking Changes in CI

A pull request changes openapi.yaml. CI passes. The spec is valid, linting is clean, and reviewers approve it. Three days later, a mobile client starts crashing because a response field it depended on disappeared. The field was not intentionally removed; it was renamed during a refactor, and nothing in review caught the compatibility break.

Try Apidog today

That is the gap a plain OpenAPI validator cannot close. A spec can be syntactically valid and still break consumers. To catch that class of problem, compare the proposed spec against the version it replaces and fail the merge when the change is backward-incompatible. That comparison is an OpenAPI diff.

What an OpenAPI diff compares

An OpenAPI diff compares two specs:

  • Base spec: the version on the target branch, usually what is already live.
  • Head spec: the version proposed by the pull request.

Unlike git diff, a structural OpenAPI diff understands paths, operations, parameters, schemas, request bodies, and responses. That lets it classify changes by compatibility instead of showing raw line changes.

Additive changes are usually safe:

  • Adding a new optional request parameter
  • Adding a new response field
  • Adding a new endpoint
  • Adding a new enum value to a request body

Existing clients can continue sending the same requests and reading the same fields.

Backward-incompatible changes are the ones you want to block:

  • Removing a response field
  • Renaming a property
  • Making an optional parameter required
  • Narrowing a type, for example string to integer
  • Removing an enum value a client might send
  • Deleting an endpoint or HTTP method

A structural diff should surface these as breaking changes and point to the affected path, operation, schema, or parameter.

If you need the compatibility model behind these decisions, see how to version and deprecate APIs at scale.

Option 1: Use oasdiff

oasdiff is a widely used open-source OpenAPI diff tool. It is distributed as a Go binary, supports OpenAPI 3.0 and 3.1, and has commands for full diffs, changelogs, and breaking-change detection.

The commands you will use most often are:

oasdiff diff base-openapi.yaml head-openapi.yaml
Enter fullscreen mode Exit fullscreen mode 
oasdiff breaking base-openapi.yaml head-openapi.yaml
Enter fullscreen mode Exit fullscreen mode 
oasdiff changelog base-openapi.yaml head-openapi.yaml
Enter fullscreen mode Exit fullscreen mode

For CI, use breaking with a fail threshold:

oasdiff breaking base-openapi.yaml head-openapi.yaml --fail-on ERR
Enter fullscreen mode Exit fullscreen mode

Here:

  • base-openapi.yaml is the target branch version.
  • head-openapi.yaml is the pull request version.
  • --fail-on ERR exits non-zero when oasdiff finds a definite breaking change.

That non-zero exit code is what turns the diff into a merge gate.

oasdiff uses severity levels:

  • ERR: definite backward-incompatible changes
  • WARN: potentially incompatible changes
  • INFO: informational changes

A practical default is:

oasdiff breaking base-openapi.yaml head-openapi.yaml --fail-on ERR
Enter fullscreen mode Exit fullscreen mode

If your API has stricter compatibility requirements, fail on warnings too:

oasdiff breaking base-openapi.yaml head-openapi.yaml --fail-on WARN
Enter fullscreen mode Exit fullscreen mode

For a human-readable report, use:

oasdiff changelog base-openapi.yaml head-openapi.yaml
Enter fullscreen mode Exit fullscreen mode

oasdiff can also output formats such as HTML, JSON, YAML, and Markdown, which makes it useful for PR comments, artifacts, and generated changelogs.

Option 2: Use openapi-diff

If your build stack is already JVM-based, OpenAPITools/openapi-diff is another good option. It compares OpenAPI 3.x specs and can render output as HTML, Markdown, AsciiDoc, JSON, or console text.

The CI-oriented command is:

openapi-diff old-openapi.yaml new-openapi.yaml --fail-on-incompatible
Enter fullscreen mode Exit fullscreen mode

The important flag is:

--fail-on-incompatible
Enter fullscreen mode Exit fullscreen mode

It exits non-zero only when the new spec is not backward-compatible with the old one.

Other useful modes include:

openapi-diff old-openapi.yaml new-openapi.yaml --fail-on-changed
Enter fullscreen mode Exit fullscreen mode

Use this if you want CI to fail on any detected change.

openapi-diff old-openapi.yaml new-openapi.yaml --state
Enter fullscreen mode Exit fullscreen mode

This prints a script-friendly state such as:

no_changes
compatible
incompatible
Enter fullscreen mode Exit fullscreen mode

Use openapi-diff if the JVM dependency is already acceptable in your pipeline. Use oasdiff if you want a lightweight standalone binary.

Add an OpenAPI diff gate to GitHub Actions

The important CI detail is that you need both versions of the spec:

  • The pull request checkout gives you the head version.
  • git show can extract the base version from the target branch.

Here is a minimal GitHub Actions workflow using oasdiff:

name: openapi-diff

on:
  pull_request:
    paths:
      - "openapi.yaml"

jobs:
  breaking-changes:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get base spec
        run: git show origin/${{ github.base_ref }}:openapi.yaml > base-openapi.yaml

      - name: Install oasdiff
        run: |
          curl -fsSL https://raw.githubusercontent.com/oasdiff/oasdiff/main/install.sh | sh

      - name: Diff for breaking changes
        run: oasdiff breaking base-openapi.yaml openapi.yaml --fail-on ERR
Enter fullscreen mode Exit fullscreen mode

Key implementation details:

  • fetch-depth: 0 ensures the base branch is available.
  • git show origin/${{ github.base_ref }}:openapi.yaml extracts the base spec without a second checkout.
  • paths prevents the workflow from running on unrelated pull requests.
  • oasdiff breaking ... --fail-on ERR fails the job when a definite breaking change is detected.

When the job fails, the PR author gets feedback before merge, while the change is still cheap to fix.

Not every breaking change is accidental. Sometimes you intentionally ship a major version. In that case, keep the gate enabled by default and require an explicit override, such as:

  • A PR label
  • A version bump in info.version
  • A separate approval workflow
  • A dedicated release branch for major versions

The goal is not to make breaking changes impossible. The goal is to make them deliberate. For deciding when a break requires a new major version, see the API versioning strategy guide.

What an OpenAPI diff cannot catch

A diff compares two spec files. It can tell you whether the new contract is backward-compatible with the old contract.

It cannot tell you whether your running service actually conforms to either contract.

For example:

  • The spec says a response includes created_at, but the service no longer returns it.
  • The spec says an endpoint returns 200, but the implementation returns 500 for a common case.
  • The spec marks a field as required, but the API sometimes omits it.

The diff is clean because both spec versions agree. The implementation is wrong because the live API does not match the contract.

To catch that, you need contract tests that run against the actual service. Generate or maintain tests from the OpenAPI contract, execute them against a deployed environment, and assert that real responses match documented status codes, required fields, and schemas.

That layer is API contract testing.

Add contract testing with Apidog and the Apidog CLI

Apidog can be used alongside an OpenAPI diff gate. The diff checks whether the spec changed safely. Apidog checks whether the running API still matches the spec.

A practical workflow is:

  1. Import or sync your OpenAPI spec into an Apidog project.
  2. Generate or maintain test scenarios from the spec.
  3. Add assertions for status codes, required fields, schemas, and response types.
  4. Run those scenarios in CI against a staging or preview environment.

Apidog can generate test scenarios from OpenAPI specs, so you do not have to maintain a separate test suite that drifts away from the contract.

You can also download Apidog and import an existing spec to try the workflow on your own API. If you are still organizing your spec in Git, see the guide to version-controlling an OpenAPI spec with Git.

The Apidog CLI runs those scenarios headlessly in CI.

Install it with npm:

npm install -g apidog-cli
Enter fullscreen mode Exit fullscreen mode

Run a scenario:

apidog run \
  --access-token $APIDOG_ACCESS_TOKEN \
  -t <scenarioId> \
  -e <environmentId> \
  -r junit,cli \
  --out-dir ./apidog-reports
Enter fullscreen mode Exit fullscreen mode

The flags are:

  • --access-token: authenticates the run; store it as a CI secret.
  • -t: selects the scenario.
  • -e: selects the environment.
  • -r junit,cli: emits JUnit XML plus terminal output.
  • --out-dir: writes reports to a directory that CI can upload.

You can copy the full command, including the real scenario and environment IDs, from the scenario’s CI/CD tab in Apidog.

For all available options, see the complete Apidog CLI guide or run:

apidog run --help
Enter fullscreen mode Exit fullscreen mode

Like the diff gate, the CLI uses exit codes. If an assertion fails because the live API no longer matches the contract, apidog run exits non-zero. CI marks the step failed and blocks the merge.

Full pre-merge workflow

Use two separate checks:

  1. OpenAPI diff: blocks backward-incompatible spec changes.
  2. Contract conformance: blocks implementations that do not match the spec.

Example GitHub Actions workflow:

jobs:
  breaking-changes:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get base spec
        run: git show origin/${{ github.base_ref }}:openapi.yaml > base-openapi.yaml

      - name: Install oasdiff
        run: curl -fsSL https://raw.githubusercontent.com/oasdiff/oasdiff/main/install.sh | sh

      - name: Check for breaking OpenAPI changes
        run: oasdiff breaking base-openapi.yaml openapi.yaml --fail-on ERR

  contract-conformance:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install Apidog CLI
        run: npm install -g apidog-cli

      - name: Run contract tests
        run: |
          apidog run \
            --access-token "$APIDOG_ACCESS_TOKEN" \
            -t 605067 \
            -e 1629989 \
            -r junit,cli \
            --out-dir ./apidog-reports
        env:
          APIDOG_ACCESS_TOKEN: ${{ secrets.APIDOG_ACCESS_TOKEN }}

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: apidog-report
          path: ./apidog-reports
Enter fullscreen mode Exit fullscreen mode

The two jobs protect different failure modes:

  • breaking-changes catches unsafe contract changes before they merge.
  • contract-conformance catches drift between the documented contract and the running service.

Run the conformance job against a deployed staging, preview, or test environment. Keep if: always() on the report upload so you can inspect failures even when the test step fails.

If either job fails, block the PR.

For more CI examples, see the Apidog CLI GitHub Actions guide and the CI/CD pipeline walkthrough.

Top comments (0)