Your CI Pipeline Never Re-Runs the Request That Proved the Bug

You already have the requests. You ran them once, found a bug, fixed it, and moved on. The gap in most CI pipelines is not detection — it is that nobody ever re-runs the request that proved the bug in the first place.

This post is about a small harness that fixes that: take the requests your testing produced, replay them against a second tenant's session, and fail the build when the response says "yes" to a question it should have said "no" to.

The shape of the test

A cross-tenant authorization bug has a signature that is easy to express as a test:

1. An object exists for tenant A (invoice 4821, project 91, report 17).
2. Tenant B holds a valid session.
3. Tenant B requests A's object by ID.
4. The response is 200 with A's data instead of 403/404.

Steps 1–3 are the hard part to write by hand and the trivial part to automate. If you can capture a request from one session, you can replay it with another session's cookie in about fifteen lines of Python.

import requests

def replay(request: dict, as_session: str) -> requests.Response:
    """Re-send a captured request with a different session token."""
    headers = {k: v for k, v in request["headers"].items()
               if k.lower() not in ("cookie", "authorization", "content-length")}
    headers["Authorization"] = f"Bearer {as_session}"
    return requests.request(
        request["method"],
        request["url"],
        headers=headers,
        data=request.get("body"),
        timeout=10,
        allow_redirects=False,
    )

The harness itself is uninteresting. What matters is what you do with the three responses you can now compare.

Compare, don't just assert

The naive test asserts response.status_code == 403. That assertion fails the moment a developer returns 404 to avoid leaking existence — a defensible choice, and your test now cries wolf.

Assert on the property you actually care about instead: does the response contain data belonging to the other tenant?

def leaks(response: requests.Response, marker: str) -> bool:
    return response.status_code == 200 and marker in response.text

# marker = a unique string seeded into tenant A's object at fixture time
assert not leaks(replay(captured, tenant_b_token), marker="INV-4821-A7F3")

That single assertion survives refactors, catches silent 200s, and produces a failure message an engineer can act on: this exact request, with this token, returned another tenant's invoice.

Where this fits in a pipeline

Run the replay set as a separate CI job, not inside the unit test suite:

• Fixture job: seed two tenants and one object per endpoint you care about. Ten minutes of setup, reusable across runs.
• Replay job: iterate over the captured request corpus. One process, parallel by endpoint. Fails fast on the first leak.
• Report job: emit the failing request as a file artefact. The request is the bug report.

The corpus grows the way a regression suite grows: every real finding you fix becomes a permanent entry. After six months you have a few hundred requests that encode every authorization decision your team has ever got wrong — which is a far better use of the corpus than a scanner's four hundred unreproducible alerts.

Three failure modes worth pre-empting

State-dependent endpoints. A POST /invoices/4821/approve will mutate data on replay. Scope the corpus to read paths first, or run replays against a throwaway environment with seeded data.

Token expiry. A captured request with a two-minute token is a flaky test. Mint fresh tokens per run from your identity provider's test client, and keep the captured request shape rather than the captured credential.

Over-broad markers. If your marker string appears anywhere in shared fixtures, every response "leaks" and the test is worthless. Generate per-tenant, per-run UUIDs.

What this does not catch

Replay testing finds authorization regressions in endpoints you already know about. It does not find endpoints you never captured, and it says nothing about business-logic abuse that stays inside one tenant's data — for example, applying a discount code twice, or approving your own expense claim.

Those need the other half: exploratory, attacker-shaped testing that generates new requests rather than replaying old ones. That is the part automation is genuinely improving right now, but it is also the part where a reproducible artifact matters most — because a finding you cannot replay is a finding you cannot fix, then verify.

I work with HACKBOX PTE. LTD., a Singapore-based company that runs an AI-driven automated penetration testing platform as SaaS, with flexible pricing on a results-based or subscription model. The platform's outputs are shaped for exactly this loop: each finding arrives with the request that reproduces it, so it can be turned into a permanent regression test rather than a ticket that closes on a promise.