Have you ever run an API test, watched it fail, and then run it again without changing anything—only to see it pass?
A changing dataset can cause exactly that problem with pagination.
Suppose you fetch every record from an endpoint twice: first with five records per page, then with eleven. If both traversals cover the same dataset in the same order, you should get the same sequence of IDs.
But what happens if someone inserts a record between requests?
Now a difference between the two results might be a pagination bug. Or it might be a perfectly valid consequence of reading changing data.
I explored this problem through a controlled Python experiment. My question was:
Can we reduce misleading pagination failures without hiding genuine defects?
A simple example of the problem
Imagine an endpoint returns records ordered by ID:
Page 1: 10, 20, 30
Before you request the next page, another client inserts record 5.
The collection now looks like this:
5, 10, 20, 30, 40, 50, ...
If your next request uses offset=3, it can return:
Page 2: 30, 40, 50
Record 30 appears twice.
That looks suspicious, but whether it is a defect depends on the API’s contract. A live offset-based endpoint may allow this behaviour when the collection changes. An endpoint promising a consistent snapshot should not.
The test needs to know what the API promises before deciding that a difference is a failure.
Testing without knowing the exact answer
The approach I used is called metamorphic testing.
Instead of writing down the exact expected response for every request, you check a relationship between related executions.
For pagination, one useful relationship is:
Changing the page size should preserve the complete ordered result when both traversals read the same collection state.
In simplified Python, the comparison looks like this:
small_pages = collect_all_ids(page_size=5)
large_pages = collect_all_ids(page_size=11)
assert small_pages == large_pages
This is an illustration of the comparison, not the full experimental checker. A real implementation also needs to handle incomplete traversals, timeouts, duplicate IDs, and the endpoint’s consistency guarantees.
The difficult part is the assumption hidden inside that assertion: did both traversals read the same state?
What I built
I worked on a small Python HTTP service backed by SQLite. It supported:
- Offset pagination.
- Keyset pagination.
- Deterministic ordering.
- Synthetic records.
- Controlled insertions and deletions between page requests.
- Live-data and snapshot contracts.
I also included deliberately faulty implementations, such as skipping records, repeating boundaries, stopping too early, and breaking snapshot promises.
A separate list-based specification checked the expected behaviour against the controller’s recorded history. The checker itself only received the public HTTP interface—it could not inspect hidden fault labels.
The main experiment contained 15,600 runs, using 20 seeds, seven consistency/update scenarios, and two request budgets.
Those numbers need context: this was one purpose-built system with two pagination configurations. It was not a study of 15,600 independent APIs.
Five ways to handle the comparison
I compared five policies:
| Policy | What it does |
|---|---|
| Direct checking | Compares results without first establishing that live data stayed stable. |
| Bounded retries | Repeats a failing comparison and looks for confirmation, within the request budget. |
| Consistency guards | Uses public consistency evidence to decide whether a comparison is justified. |
| Contract-only abstention | Avoids live-data comparisons when consistency is unknown; still checks promised snapshots. |
| Frozen-data reference | Stops updates before testing, creating a stable environment. |
The frozen-data reference deserves special attention. Freezing the data changes the environment. Its benefits cannot be credited to a smarter checker.
What the guards checked
In the versioned live-data scenarios, the API exposed a collection-wide version counter.
The guarded checker read the counter before and after a traversal pair and checked the versions reported by individual pages. If that evidence did not establish a stable view, it treated the comparison as inconclusive.
This only works under the documented assumptions: the counter must cover the collection, increase with every relevant change, and agree with atomic page reads.
A matching record count would not provide the same evidence. Neither would an ETag that applies only to one resource.
Snapshot promises were handled differently. If the API promised an immutable snapshot and then exposed inconsistent versions, that was a reportable contract violation—not a reason to quietly discard the comparison.
Fewer false alarms came with a trade-off
At the primary budget of 96 requests per run:
| Method | Correct runs with a false alarm | Faulty runs with a confirmed detection |
|---|---|---|
| Direct checking | 100 / 280 | 860 / 1,280 |
| Bounded retries | 20 / 280 | 780 / 1,280 |
| Consistency guards | 0 / 280 | 740 / 1,280 |
| Contract-only abstention | 0 / 280 | 380 / 1,280 |
| Frozen-data reference | 0 / 280 | 840 / 1,280 |
The denominators matter. These are run-level outcomes, not counts of unique bugs. Faulty-run totals include cases where a policy abstained or the environment prevented a fault from being triggered.
The guarded policy produced no false alarms in the evaluated correct runs. But it also detected fewer faulty runs than direct checking.
The guards did not make every test better. They traded some testing opportunities for better-justified verdicts.
That is why the simple abstention baseline was useful. A checker can look reassuringly quiet simply because it avoids making comparisons. Zero false alarms alone is not enough evidence of usefulness.
When retries helped
Retries worked well when the data changed briefly and then settled.
An insertion could invalidate the first comparison, while a later attempt ran against stable data. In those cases, retrying was enough to avoid some misleading reports.
Continuous changes were harder. Another attempt did not necessarily create a valid comparison.
Retries also introduced a different problem: requiring repeated confirmation could suppress a genuine snapshot failure that appeared only during the first attempt.
So “retry failed tests” was helpful in some scenarios, but it was not a complete consistency strategy.
One defect every method missed
One seeded defect omitted the same record from both traversals.
The two sequences still matched.
That exposed a basic limitation of the equality relation:
Two results can agree and still both be wrong.
In this experiment, that shared omission escaped all five policies. Detecting it requires another source of evidence—for example, a known fixture or an independent completeness check.
Adding retries does not fix an oracle that checks the wrong property.
Freezing data can hide snapshot problems
Stable fixtures are useful for checking ordinary pagination behaviour.
But freezing updates also removes the conditions needed to expose certain snapshot defects. If nothing changes, an implementation that accidentally reads live data may look indistinguishable from one that correctly preserves a snapshot.
For me, that was an important practical lesson: a useful test suite needs both stable-data checks and tests that actively challenge consistency promises.
A scoring mistake worth discussing
During code review while the main experiment was running, I found a weakness in the original scoring rule.
It could give a checker detection credit when it reported a difference during an invalid comparison, simply because a genuine fault also occurred during that attempt.
Those two events were not enough to establish a valid detection.
The corrected analysis required the report to have both a specification-confirmed violation and an applicable comparison context.
This correction happened after the protocol was frozen. The original scores were retained, and the manuscript reports both interpretations. The experiment was not selectively rerun to improve the results.
It changed the detection totals, which is why I consider it part of the findings rather than a detail to leave out.
What this study does—and does not—show
The experiment illustrates how verdict policies behave under explicitly controlled conditions.
It does not establish that guards will achieve zero false alarms in production.
The service was small, the faults were deliberately seeded, and the update schedules were deterministic. The 20 seeds varied record assignments; they did not create 20 independently developed systems or cover arbitrary production workloads.
The subject, specification, and tests were also developed within the same AI-assisted effort. Their algorithmic separation helped with checking, but it does not replace independent review or replication.
What I would take into a real API test suite
Before adding more assertions or retries, I would ask:
- What does this endpoint promise? Live reads, stable ordering, and snapshots are different guarantees.
- What makes this comparison valid? Similar-looking responses are not proof of a shared state.
- What happens when the evidence is missing? Record an inconclusive outcome rather than silently counting a pass.
- What does the relation fail to detect? Equality cannot establish completeness by itself.
- Are consistency promises being exercised? A snapshot test needs changes that could reveal a broken snapshot.
The most useful outcome was a clearer distinction between a failing comparison and a demonstrated API defect.
Code and data
The benchmark, validation tests, raw experimental data, analysis scripts, and generated results are available here:
Download the research package on Zenodo
This article describes a controlled research project. It has not been peer reviewed. Metamorphic pagination checks, retries, and consistency mechanisms are established ideas; the project compares their behaviour under a documented benchmark rather than claiming to invent them.
When testing paginated APIs, how do you distinguish a genuine pagination bug from inconsistencies caused by data changing between requests?
I'd love to hear how others approach this in real-world testing. If I've missed anything, made a mistake, or if there's a better way to handle this, please feel free to correct me or share your suggestions. I'm still learning and always happy to improve!
Top comments (1)
This is the cleanest example of a test that is actually testing the world's stability instead of the code. If the dataset can change between page one and page two, then offset pagination gives you no consistent snapshot to assert against — the failure is the API contract showing through the test, not flakiness to be retried away. The practical fix I have seen work is to assert properties that survive change — no duplicates across pages, monotonic ordering within a stated snapshot — or to test against a seeded, frozen dataset when you need exact contents. Cursor pagination exists precisely because this experiment keeps getting the same result.