DEV Community

ITFabers
ITFabers

Posted on

Your OpenAPI spec is lying

#api #python #testing #openapi

Most OpenAPI files describe the API somebody intended to build. The server has been changing since. Nothing fails when the two drift apart, because nothing in the pipeline reads the spec as a test.

The cheap fix is to turn the spec into assertions and run them against the real server. Each operation gets one test for the documented success status. Each secured operation gets a second test that sends no credentials and expects 401 or 403. Wherever the spec and the server disagree, a test goes red.

I built a generator that does this and ran it on the public Swagger Petstore OpenAPI 3 spec. The output is in a repo you can clone.

Run it 

git clone https://github.com/arsenhambardzumyan/openapi-pytest-suite
cd openapi-pytest-suite/example
pip install pytest
python3 -m pytest tests -q
Enter fullscreen mode Exit fullscreen mode

Expected output:

.s....s..s.s.s..s.....s.s..s   [100%]
19 passed, 9 skipped
Enter fullscreen mode Exit fullscreen mode

That is a dry run with no network. It proves the generated suite is consistent with the spec. It does not prove anything about a server yet. The 9 skips are the missing-credentials checks, which only make sense against a live API.

What a generated test looks like

This is tests/test_getOrderById.py for GET /store/order/{orderId}:

def test_getOrderById():
    """Happy path: GET /store/order/1 must return HTTP 200."""
    dry_run = os.environ.get("ATG_DRY_RUN", "1") != "0"
    base = os.environ.get("ATG_BASE_URL", "") or "https://CHANGE-ME.example.com"
    client = Client(base, dry_run=dry_run)
    resp = client.request("GET", "/store/order/1", headers={},
                          expected_dry_status=200, drop_credentials=False)
    assert resp.status_code == 200, "unexpected status: %s" % resp.status_code
Enter fullscreen mode Exit fullscreen mode

And the secured-endpoint check, from tests/test_deletePet.py:

def test_deletePet_rejects_missing_credentials():
    """Contract: DELETE /pet/{petId} must reject requests without credentials."""
    ...
    assert resp.status_code in (401, 403), (
        "missing credentials must return 401/403, got: %s" % resp.status_code
    )
Enter fullscreen mode Exit fullscreen mode

The client is plain standard library, so the only install is pytest. Tokens come from ATG_API_TOKEN or ATG_API_KEY, never from the files.

Point it at your own API 

ATG_BASE_URL=https://api.yourcompany.com \
ATG_API_TOKEN=... \
ATG_DRY_RUN=0 python3 -m pytest tests -q
Enter fullscreen mode Exit fullscreen mode

Expect some red on the first live run. The path parameters are filled with sample values, so a 404 on /pet/1 can mean the sample ID does not exist rather than that your spec is wrong. Read each failure and decide whether the spec, the server or the sample value is at fault. That triage is the useful part, since it tells you where the documentation has stopped matching reality.

Limits

  • It only covers documented behaviour: success status and the missing-credentials rejection. It knows nothing about your business logic.
  • It checks the status code. It does not validate response bodies against the schema.
  • The Petstore example has no server URL, so I have not run it against a live API.

If you would rather not set it up

I do this as a fixed-price service: send me your OpenAPI 3.x or Swagger 2.0 spec and within 24 hours you get the zip (one test module per operation, the client, a README). It is $99, one-time. Do not send secrets; the spec is all I need.

Order here: https://itfabers.lemonsqueezy.com/checkout/buy/3ad36b82-d00d-42d5-84a4-42a36104c397

Landing page with the details: https://arsenhambardzumyan.github.io/openapi-pytest-suite/

I'm Arsen, an independent developer in Yerevan. This offer is new, so there are no customer stories to quote yet.

Top comments (0)