API docs said pagination worked. It didn't.

Was integrating this ecommerce API last week for product data. Docs say pagination works with offset and limit parameters. Seemed simple enough.

What the docs showed

response = requests.get(
    'https://api.example.com/products',
    params={'offset': 0, 'limit': 100}
)

Perfect right? Just loop and increment offset by 100 each time.

The problem

Worked fine for first 200 products maybe 300. Then started getting duplicates. Then missing products entirely. Offset 400 returned same stuff as offset 300. Offset 1000 gave me products from offset 200.

Thought my code was broken. Checked logs for 2 hours. Nothing wrong on my end.

Turns out

API was sorting by relevance not ID. Every time I hit it the order changed slightly because products got sold or restocked. So offset 400 wasnt actually the 400th product anymore, it was whatever happened to be 400th at that exact moment.

Docs never mentioned this. Support didnt know either when I asked.

What actually worked

Had to switch to cursor based pagination. Most APIs support this but dont document it well.

import requests

url = 'https://api.example.com/products'
all_products = []
cursor = None

while True:
    params = {'limit': 100}
    if cursor:
        params['cursor'] = cursor

    response = requests.get(url, params=params)
    data = response.json()

    all_products.extend(data['products'])

    cursor = data.get('next_cursor')
    if not cursor:
        break

print(f'Got {len(all_products)} products total')

Cursor is basically a bookmark. API returns next_cursor in response and you pass it back in next request. Guarantees you get unique results even if data changes between requests.

If you see duplicate items across pages your offset pagination is probably broken. Check response for next_cursor or continuation_token or page_token. Most REST APIs have this but call it different things. Sometimes just trying cursor as a param works even if docs dont mention it.

Wasted 4 hours on this because docs looked fine. Still annoyed their docs showed offset as the main example tho.