DEV Community


Posted on

Retrieve entire data from paginated API recursively.

In this post i am going to show to retrieve data from a paginated API using recursive method.

To explain the functionality, I am going to use a dummy REST API which i created in the Currently the dataset has 91 records. By default it returns 91 records. In realtime, the dataset may be huge of more than 10000's data items. It's not recommended to query the entire dataset in a single api as it burdens the server performance and also place a heavy load on the client side as well. The recommended approach is to paginate the results when you are showing in the client side. That's the reason most of the API's provides some additional filters in the api to return only the records which the user is interested instead of returning the entire datastore.

Here some of the common filters used by the API's.

1.offset, limit : "limit" determines how many records to retrieve starting from the "offset", limit : "page" determines which data the server needs to return based on number of "limit" -> records per page .

Please refer the respective API developer manual for more details on how to retrieve paginated results.

Sample API End Points :

Without Pagination :

With Pagination :

The above API endpoints are good for a normal use case. But they are requirements in which you need to retrieve all the data from the API and the API is designed to return only 50 records per call. In this scenario, we can create a recursive paginated API call and capture the entire dataset.

Remember,If you are not handling this properly it may end up with infinite looping and your browser may hang. In this example i have limited 20 items per page.

const limitPerPage=20;
const apiUrl="";

const getUsers = async function(pageNo = 1) {

let actualUrl=apiUrl + `?page=${pageNo}&limit=${limitPerPage}`;
var apiResults=await fetch(actualUrl)
return resp.json();

return apiResults;


const getEntireUserList = async function(pageNo = 1) {
  const results = await getUsers(pageNo);
  console.log("Retreiving data from API for page : " + pageNo);
  if (results.length>0) {
    return results.concat(await getEntireUserList(pageNo+1));
  } else {
    return results;

(async ()=>{

    const entireList=await getEntireUserList();


I have used the browser developer console to show the output. Please see this gif

Thanks for reading & happy coding..

Top comments (6)

andersdjohnson profile image
Anders D. Johnson

This is cool! I had a similar need, so I wrote a library called "fetch-paginate" - see

brunocerq profile image

I'm getting a TypeError: results.concat is not a function

volodymyrhavryliuk profile image

Thank you very much!

johnnyboie profile image

This worked perfectly for me. Thanks Nirmal

darthkidd profile image

thank you so much! Have been looking everywhere for a simple solution, thank you 🙇

jak_white profile image

Nirmal! You saved me! Thank you!! Works like a charm for the Motivosity api.