Stop Writing Boilerplate API Pagination Loops: Meet lazy-api-paginator

#javascript #webdev #node #typescript

We have all written this exact chunk of boilerplate code at least a dozen times:

let nextUrl = 'https://example.com';
const allItems = [];

while (nextUrl) {
  const response = await fetch(nextUrl);
  const json = await response.json();
  allItems.push(...json.items);
  nextUrl = json.next_page_url; 
}

It looks simple, but this naive implementation breaks instantly in production.

What happens when the API hits you with a 429 Too Many Requests error? Your script crashes. What if the dataset contains 500,000 records? Your server runs out of RAM, triggers a memory leak, and throws an Out of Memory panic. What if the API requires cursor, keyset, or Link-header pagination? You are back to rewiring custom loops from scratch.

To solve this once and for all, I built lazy-api-paginator—a lightweight, zero-dependency TypeScript engine designed to stream paginated API data seamlessly using async generators.

🚀 The Core Solution: Lazy Async Data Streams

Instead of pulling thousands of items into local memory before processing them, lazy-api-paginator fetches pages dynamically on-demand. It yields items one-by-one using native JavaScript for await...of loops.

Here is how simple a basic integration looks:

import { createPaginator } from 'lazy-api-paginator';

const paginator = createPaginator<ApiResponse, User>({
  initialUrl: 'https://example.com',
  extractItems: (res) => res.data,
  getNextPageUrl: (res) => res.nextCursor ? `https://example.com?cursor=${res.nextCursor}` : null,
});

// Memory consumption remains constant, no matter how large the API dataset is!
for await (const user of paginator) {
  console.log(`Processing: ${user.name}`);
}

🛠️ Production-Ready Features Built-In

This isn't just a wrapper around fetch. It includes the mission-critical safeguards that production environments demand:

1. Smart Rate Limiting & 429 Handling

The paginator automatically throttles outward requests to match a target rate limit. If it encounters a 429 status code, it intelligently parses standard Retry-After, X-RateLimit-*, or RateLimit-* headers to pause execution before gracefully retrying.

rateLimit: {
  requestsPerSecond: 10,       // Throttles requests safely
  respectRetryAfter: true,     // Automatically honors API headers
  maxRateLimitDelay: 60000     // Maximum wait cap
}

2. Exponential Backoff with Jitter

Network blips happen. The library includes built-in exponential backoff configuration complete with configurable randomness (jitter) to ensure your microservices do not accidentally DDoS your target API endpoints when recovering from a downtime window.

3. Server-Side Request Forgery (SSRF) Protection

If you are passing user-submitted URLs into your backend API integrations, you are exposing yourself to internal network scanning vulnerabilities. lazy-api-paginator provides an optional, pluggable ssrfProtection layer to lock down server-to-server calls away from internal cloud metadata instances (169.254.169.254) and local subnets.

📦 Zero Boilerplate Templates for Common APIs

Different platforms use completely different pagination mental models. The package ships with ready-to-use structural strategies to eliminate manual token plumbing:

Cursor-based: Out-of-the-box configurations mapping straight to the data schemas of Stripe, Slack, and Notion.
Link Headers: Automatic header extraction custom-tailored for the GitHub REST API.
Page number-based: Perfect for applications consuming default frameworks like Laravel or Django.
Offset/Keyset Seek: Highly optimized for traditional database pagination setups.

import { createPaginator, strategies } from 'lazy-api-paginator';

// Zero boilerplate pagination for GitHub Issues
const githubPaginator = createPaginator({
  initialUrl: 'https://github.com',
  ...strategies.linkHeader({ dataPath: '' }),
});

🌟 Open Source & Open to Feedback

lazy-api-paginator works out of the box in both modern ECMAScript Modules (ESM) and legacy CommonJS (CJS) environments with full, native TypeScript autocomplete types.

If you are tired of writing fragile, memory-heavy while loops every time you connect to an external API service, give it a spin: