137Foundry

Posted on Jul 11

How to Build API Pagination That Survives Rate Limits and Client Retries

#webdev #api #backend #programming

A pagination implementation that works fine in a demo can fall apart the moment a real client starts retrying failed requests, backing off, and resuming where it left off. If your cursor or page token isn't resilient to retries, you end up with duplicate processing, skipped records, or client-side bugs that only show up under real production network conditions, usually discovered by whichever customer has the flakiest network connection. Here's how to build it so it holds up under exactly those conditions.

Step 1: Make the cursor idempotent, not session-based

The most common mistake is tying a pagination cursor to server-side session state, like an in-memory query result cached against a session ID or a temporary server-side result set. The moment a client retries after a timeout, or a load balancer routes the retry to a different server instance than the one that generated the cursor, that session state is gone and the request fails or returns garbage instead of the expected next page.

Instead, make the cursor fully self-describing. Encode everything needed to resume the query directly in the cursor itself: the sort column value and a unique tiebreaker, typically the primary key.

import base64
import json

def encode_cursor(sort_value: str, row_id: int) -> str:
    payload = json.dumps({"sort_value": sort_value, "id": row_id})
    return base64.urlsafe_b64encode(payload.encode()).decode()

Any server instance, at any time, can decode this cursor and resume the exact same query from the exact same point. No session state required anywhere in the stack, which also makes horizontal scaling of your API servers trivial since no instance needs to remember anything about a client's previous requests.

Step 2: Return rate limit headers on every paginated response

If your API enforces rate limits, and most public APIs handling meaningful traffic should, include standard headers so clients can back off intelligently instead of guessing when it's safe to retry:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 43
X-RateLimit-Reset: 1720713600

There's no single universal standard for these exact header names, but this pattern is common enough across major APIs that most client libraries already know how to read it out of the box. Document your exact header names clearly in your API reference, since a client that can't parse your rate limit headers will retry blindly on a fixed interval and make the underlying problem worse rather than better.

Step 3: Make retries safe by design, not by convention

A client that times out waiting for page 3 and retries with the exact same cursor should get the exact same page 3 back, not a different set of rows. Because cursor-based pagination is deterministic (same cursor, same query, same result, assuming no underlying data changed in the meantime), this is naturally idempotent as long as you're not accidentally injecting randomness like ORDER BY RANDOM() or a non-deterministic tiebreaker somewhere in the query. Double-check your sort clause always includes a unique column so ties resolve the same way on every single execution.

Step 4: Implement exponential backoff on the client side

If you're consuming a paginated API rather than building one, implement backoff with jitter rather than fixed-interval retries, especially when pulling large datasets across many sequential pages:

import time
import random

def fetch_with_backoff(fetch_fn, max_retries=5):
    for attempt in range(max_retries):
        try:
            return fetch_fn()
        except RateLimitError:
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
    raise Exception("Max retries exceeded")

Jitter matters because without it, many clients hitting a rate limit at the same moment will all retry in lockstep, creating a thundering herd effect that makes the rate limit problem measurably worse instead of resolving it. The Python requests library documentation covers session-level retry configuration if you'd rather not hand-roll this logic yourself.

Step 5: Set a reasonable page size ceiling to reduce total request volume

Smaller default page sizes mean more total requests needed to page through a large dataset, which means more chances to hit a rate limit somewhere in the middle of a long pull. If you know a client is doing a bulk export or a full data sync, a larger page size, within your API's hard maximum, reduces both the number of round trips and the number of opportunities for a transient failure to interrupt the process partway through.

Step 6: Log cursor state on failure for debugging and safe resumption

When a paginated pull fails partway through, log the last successful cursor so the operation can resume from that exact point rather than restarting from page 1. This matters enormously for large data syncs where restarting from scratch could mean re-processing millions of already-handled rows, which is both slow and risks duplicate side effects if the downstream processing isn't itself idempotent.

Step 7: Version the cursor format so future changes don't break old clients

Once your API has real clients holding onto cursors, whether that's a bookmarked "load more" position or a long-running batch job resuming after a failure, changing the cursor's internal shape can silently break anyone with an old cursor still in hand. Include a version marker in the encoded payload so your decoder can detect an outdated format and respond with a clear, actionable error rather than a confusing failure deep in a query:

def decode_cursor(cursor: str) -> dict:
    payload = json.loads(base64.urlsafe_b64decode(cursor.encode()))
    if payload.get("v") != 1:
        raise ValueError("unsupported cursor version, please request a fresh page")
    return payload

This is a small amount of upfront work that saves a confusing debugging session later, whenever the schema underneath the cursor inevitably needs to change to support a new sort option or filter combination.

Step 8: Handle partial failures in bulk consumers gracefully

If a client is using pagination to pull a large dataset for a batch job, sync, or export, design the consumer so a failure on page 400 doesn't discard the 399 pages already successfully processed. This sounds obvious written out, but it's a surprisingly common failure mode in hastily written sync scripts that wrap the entire pagination loop in a single try/except block and throw away all progress on any exception, including transient ones that a retry would have resolved cleanly.

def sync_all_pages(fetch_fn):
    cursor = None
    processed = 0
    while True:
        try:
            page = fetch_with_backoff(lambda: fetch_fn(cursor))
        except Exception as e:
            log_failure(last_cursor=cursor, error=e, processed_so_far=processed)
            raise
        process_records(page["data"])
        processed += len(page["data"])
        if not page["has_more"]:
            break
        cursor = page["next_cursor"]
    return processed

The key detail is logging the last successful cursor before re-raising, which ties directly back into step 6 and gives whoever restarts the job a clean resumption point instead of a guess.

Putting it together

None of these six steps depend heavily on each other, so they can be implemented incrementally and tested one at a time. The highest-leverage change, by far, is step 1: making the cursor self-describing rather than session-dependent. Everything else builds resilience on top of that foundation, and none of it matters much if the cursor itself can't survive a request landing on a different server instance. If your API's pagination breaks under retries today, that's almost always where the root cause actually lives, not in the retry logic itself.

For teams building or auditing API infrastructure like this, 137Foundry's web development service covers exactly this kind of backend design work, from initial architecture through production hardening and load testing.

DEV Community