Paging a http endpoint by recursion

Paging the results of REST API calls is a good opportunity to show recursion and how it differs from just using loops.

Recursion requires 3 things:

• a stop condition (also called a base case)
• an accumulator (some number or value which changes) which is used as input into the stop condition
• a method which can be called with the accumulator

Here’s an example where the goal is to get REST resources from a HTTP API. We have a client which has a method get(int offset, int limit) which gets the resource from the API and returns a Java object representation of the response. Here’s how to use that client recursively:

public List<Object> getResources() {
    List<Object> resources = new ArrayList();
    return retrievePagedResource(resources, 0);
}

private List<Object> retrievePagedResource(List<Object> resources, int offset) {
    ResourceResponse response = client.get(offset, limit);
    resources.addAll(response.getItems());
    if (response.getTotal > limit + offset) {
        retrievePagedResource(resources, limit + offset);
    }    
}

Let’s see how the above implementation of paging satisfies the requirements for recursion. The stop condition is in the implicit else of the if statement - recursion will stop when total results are less than or equal to the limit plus offset. We use the combination of the limit and offset as the accumulator and we use it as an input in the stop condition. Because the original method takes no method arguments we need a second method which takes the accumulator as an argument. This second method can then be called from the original function with a default value, 0, indicating we want to start paging from the 0th item in the available resources.

Using loops, this same example would look like:

public List<Object> getResources() {
    int limit = 50;
    int offset = 0;
    int totalResults;
    List<Object> resources = new ArrayList();
    ResourceResponse response = client.get(offset, limit);
    resources.addAll(response.getItems());
    totalResults = response.getTotal;
    while (totalResults > limit + offset) {
        offset = offset + limit;
        ResourceResponse response = client.get(offset, limit);
        resources.addAll(response.getItems());
    }
    return resources;
}

There are some clear differences - the loop implementation relies heavily on modifying variables and so since the variables are always changing it’s harder to reason about. The recursive example can be made purer by not adding items to an existing list but creating a new list every time which is concatenated with another list.

When paging, be wary of storing every object in-memory, in case there’s a lot of objects. If you are filtering the returned list, it is best to filter as each page comes in rather than gather a huge list and filter after everything has been returned.