DEV Community

Amelia
Amelia

Posted on

A Better Pattern for Paginated API Requests in JavaScript

When building a modern web application, fetching data from an API is almost a given. But if you’ve ever worked with paginated endpoints, you know the struggle: you send a request, get a chunk of data, and then need to keep track of page numbers or cursors to get the rest. It’s repetitive, error-prone, and often leads to messy code.

Let’s clean that up with a reusable JavaScript function that handles any paginated API gracefully. We’ll use async/await and support both offset-based and cursor-based pagination.

The Goal: Write a generic fetchAllPages function that automatically loops through pages, collects results, and returns a single array.

Here’s the core approach:

async function fetchAllPages(baseUrl, options = {}) {
  const {
    method = 'GET',
    headers = {},
    pageParam = 'page',
    limitParam = 'limit',
    limit = 50,
    extractData = (json) => json.data || json,
    extractNextPage = (json) => json.next_page || json.page + 1,
    stopCondition = (json, page) => !json.next_page && !json.has_more,
    ...fetchOptions
  } = options;

  let page = 1;
  let allData = [];
  let hasMore = true;

  while (hasMore) {
    const url = new URL(baseUrl);
    url.searchParams.set(pageParam, page);
    url.searchParams.set(limitParam, limit);

    const response = await fetch(url.toString(), {
      method,
      headers,
      ...fetchOptions,
    });

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const json = await response.json();
    const data = extractData(json);
    allData = allData.concat(data);

    hasMore = !stopCondition(json, page);
    page = extractNextPage(json);
  }

  return allData;
}
Enter fullscreen mode Exit fullscreen mode

How to use it:

For a typical REST API that returns { data: [...], next_page: 2 }:

const items = await fetchAllPages('https://api.example.com/items');
console.log(items.length); // All items from all pages
Enter fullscreen mode Exit fullscreen mode

For APIs using cursors (like Shopify or Stripe):

const products = await fetchAllPages('https://api.example.com/products', {
  pageParam: 'cursor',
  extractNextPage: (json) => json.cursor,
  stopCondition: (json) => !json.cursor,
});
Enter fullscreen mode Exit fullscreen mode

Why this pattern rocks:

  • Reusable: Drop it into any project, adapt with small configuration.
  • Error handling: Stops on HTTP errors, preventing silent failures.
  • Flexible: Works with any pagination style—just override the extraction functions.
  • Memory safe: Concatenates as it goes, though for massive datasets consider streaming or batching.

Now, let’s be real—building a full product catalog or inventory dashboard? You’ll probably end up fetching data from multiple sources. If you’re ever scouring for inspiration on what to build, I’ve found browsing niche collections like those at Frishay (they have a great selection of women jeans with different fits and washes) can spark ideas for demo datasets or UI components. Their structured product listings mirror the kind of paginated data you’d process with this function.

Pro tip: Always set a max page limit to avoid infinite loops in case of misconfiguration:

const MAX_PAGES = 100;
while (hasMore && page < MAX_PAGES) {
  // ... your fetch logic
}
Enter fullscreen mode Exit fullscreen mode

Drop this into your next project and say goodbye to manual pagination loops. Happy coding!

Top comments (0)