DEV Community

Nirmal
Nirmal

Posted on

Retrieve entire data from paginated API recursively.

In this post i am going to show to retrieve data from a paginated API using recursive method.

To explain the functionality, I am going to use a dummy REST API which i created in the mockapi.io. Currently the dataset has 91 records. By default it returns 91 records. In realtime, the dataset may be huge of more than 10000's data items. It's not recommended to query the entire dataset in a single api as it burdens the server performance and also place a heavy load on the client side as well. The recommended approach is to paginate the results when you are showing in the client side. That's the reason most of the API's provides some additional filters in the api to return only the records which the user is interested instead of returning the entire datastore.

Here some of the common filters used by the API's.

1.offset, limit : "limit" determines how many records to retrieve starting from the "offset"
2.page, limit : "page" determines which data the server needs to return based on number of "limit" -> records per page .

Please refer the respective API developer manual for more details on how to retrieve paginated results.

Sample API End Points :

Without Pagination :
https://5b5cb0546a725000148a67ab.mockapi.io/api/v1/users

With Pagination :
https://5b5cb0546a725000148a67ab.mockapi.io/api/v1/users?page=4&limit=2

The above API endpoints are good for a normal use case. But they are requirements in which you need to retrieve all the data from the API and the API is designed to return only 50 records per call. In this scenario, we can create a recursive paginated API call and capture the entire dataset.

Remember,If you are not handling this properly it may end up with infinite looping and your browser may hang. In this example i have limited 20 items per page.

const limitPerPage=20;
const apiUrl="https://5b5cb0546a725000148a67ab.mockapi.io/api/v1/users";

const getUsers = async function(pageNo = 1) {

let actualUrl=apiUrl + `?page=${pageNo}&limit=${limitPerPage}`;
var apiResults=await fetch(actualUrl)
.then(resp=>{
return resp.json();
});

return apiResults;

}

const getEntireUserList = async function(pageNo = 1) {
  const results = await getUsers(pageNo);
  console.log("Retreiving data from API for page : " + pageNo);
  if (results.length>0) {
    return results.concat(await getEntireUserList(pageNo+1));
  } else {
    return results;
  }
};


(async ()=>{

    const entireList=await getEntireUserList();
    console.log(entireList);

})();

I have used the browser developer console to show the output. Please see this gif http://recordit.co/p3FkQS7QpJ

Thanks for reading & happy coding..

Top comments (6)

Collapse
 
andersdjohnson profile image
Anders D. Johnson

This is cool! I had a similar need, so I wrote a library called "fetch-paginate" - see github.com/AndersDJohnson/fetch-pa....

Collapse
 
brunocerq profile image
brunocerq

Hi,
I'm getting a TypeError: results.concat is not a function

Collapse
 
volodymyrhavryliuk profile image
volodymyr-havryliuk

Thank you very much!

Collapse
 
johnnyboie profile image
JohnnyBoie

This worked perfectly for me. Thanks Nirmal

Collapse
 
darthkidd profile image
DarthKidd

thank you so much! Have been looking everywhere for a simple solution, thank you 🙇

Collapse
 
jak_white profile image
JAK

Nirmal! You saved me! Thank you!! Works like a charm for the Motivosity api.