DEV Community

DAYAN ELVIS JAHUIRA PILCO
DAYAN ELVIS JAHUIRA PILCO

Posted on

API Testing Frameworks Compared — and a Real Suite with Pytest + Requests

#testing #python #api #qa

TL;DR: I compared the most popular API testing frameworks (Postman/Newman, REST Assured, Karate, Supertest, pytest+requests), then built a real 14-test suite against a live public API with pytest + requests, covering status codes, JSON schema, error handling, response time and CRUD — all automated in GitHub Actions with a daily scheduled run. Full code: GitHub repo →

Why test APIs directly?

UI tests are slow and fragile; unit tests can't tell you if your service actually talks to the world correctly. API tests sit in the sweet spot of the testing pyramid: fast enough to run on every commit, realistic enough to catch integration bugs — wrong status codes, broken schemas, slow endpoints — before your users do.

The frameworks, compared

There are many tools in this space (Postman's blog lists dozens). Here's an honest comparison of the big five:

Framework Language Style Best for Watch out for
Postman + Newman GUI + JS Collections + scripts Manual exploration, teams with non-coders Test logic hidden in JSON exports; hard to code-review
REST Assured Java Fluent DSL Java shops, Spring ecosystems Verbose; needs Maven/Gradle setup
Karate Gherkin DSL BDD feature files Readable specs, mixing API+UI+perf Its own language to learn; JVM required
Supertest + Jest JavaScript Code Node.js teams testing their own Express apps Tied to the JS ecosystem
pytest + requests Python Code Anyone who knows Python; CI-first teams You assemble the pieces yourself

My pick for this article is pytest + requests, for three reasons: tests are plain Python (reviewable in any PR like normal code), pytest's fixtures and parametrization eliminate boilerplate, and the ecosystem (pytest-xdist for parallelism, jsonschema for contract validation) grows with you.

The target: a real, live API

JSONPlaceholder is a free fake REST API with realistic resources (posts, comments, users...). Perfect to demonstrate every kind of assertion against a real HTTP service — no mocks.

The test suite

The full suite is 14 tests in one file. Let's walk through the interesting parts.

Setup: one session for all tests 

import os

import pytest
import requests

BASE_URL = os.environ.get("API_BASE_URL", "https://jsonplaceholder.typicode.com")
TIMEOUT = 10


@pytest.fixture(scope="session")
def session():
    """Reuse one HTTP session for all tests (faster: keep-alive)."""
    with requests.Session() as s:
        s.headers.update({"Accept": "application/json"})
        yield s
Enter fullscreen mode Exit fullscreen mode

Two production-grade details here: the base URL comes from an environment variable, so the same suite runs against dev, staging or production; and a session-scoped fixture reuses one TCP connection for all tests.

Status codes, schema, and the unhappy path 

def test_get_single_post_has_expected_schema(session):
    resp = session.get(f"{BASE_URL}/posts/1", timeout=TIMEOUT)
    assert resp.status_code == 200
    post = resp.json()
    # schema validation: keys and types
    assert set(post.keys()) == {"userId", "id", "title", "body"}
    assert isinstance(post["userId"], int)
    assert post["id"] == 1


def test_get_nonexistent_post_returns_404(session):
    resp = session.get(f"{BASE_URL}/posts/9999", timeout=TIMEOUT)
    assert resp.status_code == 404
Enter fullscreen mode Exit fullscreen mode

Testing the error path is where most real bugs hide — an API that returns 200 with an empty body for a missing resource will silently corrupt every client that consumes it.

Non-functional checks: performance and content type 

def test_response_time_under_2_seconds(session):
    resp = session.get(f"{BASE_URL}/posts", timeout=TIMEOUT)
    assert resp.elapsed.total_seconds() < 2.0


def test_content_type_is_json(session):
    resp = session.get(f"{BASE_URL}/posts/1", timeout=TIMEOUT)
    assert "application/json" in resp.headers["Content-Type"]
Enter fullscreen mode Exit fullscreen mode

A latency budget as a test means a performance regression fails CI like any other bug.

Writing: POST, PUT, DELETE 

def test_create_post_returns_201_and_echoes_body(session):
    payload = {"title": "SAST and API testing", "body": "hello", "userId": 7}
    resp = session.post(f"{BASE_URL}/posts", json=payload, timeout=TIMEOUT)
    assert resp.status_code == 201
    created = resp.json()
    assert created["title"] == payload["title"]
    assert "id" in created
Enter fullscreen mode Exit fullscreen mode

Note the assertion on 201 Created — not just "2xx". Precise status codes are part of your API's contract.

Parametrization: one test, five cases 

@pytest.mark.parametrize("resource,expected_count", [
    ("posts", 100),
    ("comments", 500),
    ("albums", 100),
    ("todos", 200),
    ("users", 10),
])
def test_collection_sizes(session, resource, expected_count):
    resp = session.get(f"{BASE_URL}/{resource}", timeout=TIMEOUT)
    assert resp.status_code == 200
    assert len(resp.json()) == expected_count
Enter fullscreen mode Exit fullscreen mode

This is where pytest beats collection-based tools: five endpoints covered in eight lines, and each case reports individually.

Running it 

pip install pytest requests
pytest test_api.py -v
Enter fullscreen mode Exit fullscreen mode 
test_api.py::test_get_all_posts_returns_200_and_100_items PASSED         [  7%]
test_api.py::test_get_single_post_has_expected_schema PASSED             [ 14%]
test_api.py::test_get_nonexistent_post_returns_404 PASSED                [ 21%]
test_api.py::test_response_time_under_2_seconds PASSED                   [ 28%]
test_api.py::test_content_type_is_json PASSED                            [ 35%]
test_api.py::test_filter_posts_by_user PASSED                            [ 42%]
test_api.py::test_create_post_returns_201_and_echoes_body PASSED         [ 50%]
test_api.py::test_update_post_returns_200 PASSED                         [ 57%]
test_api.py::test_delete_post_returns_200 PASSED                         [ 64%]
test_api.py::test_collection_sizes[posts-100] PASSED                     [ 71%]
test_api.py::test_collection_sizes[comments-500] PASSED                  [ 78%]
test_api.py::test_collection_sizes[albums-100] PASSED                    [ 85%]
test_api.py::test_collection_sizes[todos-200] PASSED                     [ 92%]
test_api.py::test_collection_sizes[users-10] PASSED                      [100%]

============================== 14 passed ==============================
Enter fullscreen mode Exit fullscreen mode

Automating with GitHub Actions — including a daily run

API tests have a property unit tests don't: they can break without any commit, because the API on the other side changes. That's why my workflow adds a schedule trigger on top of push/PR:

name: API Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: "0 6 * * *" # daily run: catch API regressions even without commits

jobs:
  api-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Run API test suite
        run: pytest test_api.py -v --junitxml=report.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: api-test-report
          path: report.xml
Enter fullscreen mode Exit fullscreen mode

If the API breaks its contract overnight, tomorrow at 06:00 UTC this workflow turns red and I get an email — before any user files a bug.

Takeaways

Framework choice matters less than people think — what matters is what you assert: exact status codes (including errors), response schema, latency budgets, and the full CRUD cycle. Code-based frameworks like pytest+requests make those assertions reviewable, versionable, and parametrizable, which is why they scale better in CI than GUI-first tools. And because APIs change under you, a scheduled run is as important as the per-commit one.

Full demo code + workflow: github.com/Dayan-18/api-testing-demo

Which API testing framework does your team use, and why? Tell me in the comments! 👇

Previous articles in this series: SAST for Python with Bandit · IaC scanning with Checkov

References: pytest documentation · requests documentation · JSONPlaceholder

Top comments (0)