5 Real-Life Problems Solved with Async Generators in JavaScript

While I was looking over some data fetching code the other day, I came across a pattern that immediately caught my attention. A colleague was fetching paginated results from an API, and the code looked something like this massive recursive function with callbacks nested inside callbacks. It worked, but boy was it hard to follow.

That's when I remembered async generators—a feature I had learned about years ago but rarely used. Little did I know that pulling this tool out of my toolbox would completely transform how I think about handling streams of asynchronous data.

If you're like me and have heard of async generators but never quite found the right moment to use them, this post is for you. We're going to look at 5 real-world problems where async generators shine, and I promise you'll walk away with practical patterns you can use today.

What Are Async Generators?

Before we dive into the problems, let's quickly cover what async generators actually are. An async generator is a function that combines two powerful JavaScript features:

async function* myAsyncGenerator() {
  yield await fetchSomething()
  yield await fetchSomethingElse()
}

The async function* syntax tells JavaScript: "This function can pause, yield values, and also await promises." You consume it with for await...of:

for await (const value of myAsyncGenerator()) {
  console.log(value)
}

Here's what makes this so powerful: when you call an async generator function, it doesn't execute the body immediately. Instead, it returns an async iterator—an object with a next() method that returns a promise. Each time you call next() (or iterate with for await), the function runs until it hits a yield, pauses, and hands you the yielded value. The next call resumes right where it left off.

This "pause and resume" behavior is what enables all the elegant patterns we're about to explore.

Now let's see how this elegant pattern solves real problems.

1. Paginated API Fetching

This is probably the most common use case, and it's the one that opened my eyes to the power of async generators.

The Problem:

You need to fetch all results from an API that uses cursor-based pagination. The typical approach looks something like this:

// The "before" approach - messy and hard to follow
async function fetchAllUsers() {
  const allUsers = []
  let cursor = null

  do {
    const response = await fetch(`/api/users?cursor=${cursor || ''}`)
    const data = await response.json()
    allUsers.push(...data.users)
    cursor = data.nextCursor
  } while (cursor)

  return allUsers
}

// Usage - you have to wait for ALL users before processing any
const users = await fetchAllUsers()
users.forEach((user) => processUser(user))

The problem? You're loading everything into memory before you can do anything with it. If you have 10,000 users, you're holding all 10,000 in memory before processing a single one. Even worse, if you only needed the first 50 users that match some criteria, you've wasted time and bandwidth fetching all of them.

The Async Generator Solution:

async function* fetchUsers() {
  let cursor = null

  do {
    const response = await fetch(`/api/users?cursor=${cursor || ''}`)
    const data = await response.json()

    for (const user of data.users) {
      yield user
    }

    cursor = data.nextCursor
  } while (cursor)
}

// Usage - process users as they arrive!
for await (const user of fetchUsers()) {
  await processUser(user)
  // Memory efficient: only one user in memory at a time
}

Why This Works:

The magic here is in the yield statement inside the inner for loop. Each time we yield user, the generator pauses execution entirely. It doesn't fetch the next page until the consumer asks for more users. This creates a beautiful "pull-based" system—data only flows when you request it.

Here's what's happening step by step:

1. You start iterating with for await
2. The generator fetches the first page
3. It yields the first user and pauses
4. You process that user
5. The loop asks for the next user
6. The generator resumes, yields the second user, pauses again
7. When the page is exhausted, it fetches the next page
8. And so on...

The difference is night and day. With the async generator, you process each user as it arrives. If you need to stop early (maybe you found the user you were looking for), you can just break out of the loop. The generator immediately stops—no more API calls, no wasted memory.

for await (const user of fetchUsers()) {
  if (user.email === targetEmail) {
    console.log('Found them!')
    break // Generator stops here - no more fetching!
  }
}

I cannot stress this enough—once you start thinking in terms of async generators for pagination, you'll never go back.

2. Processing Large Files Line by Line

Here's a scenario that comes up all the time in Node.js: you need to process a large log file, CSV, or JSON Lines file.

The Problem:

// The naive approach - loads entire file into memory
const fs = require('fs').promises

async function processLogFile(filepath) {
  const content = await fs.readFile(filepath, 'utf-8')
  const lines = content.split('\n')

  for (const line of lines) {
    await processLine(line)
  }
}

If your log file is 2GB, you just loaded 2GB into memory. Your server will not be happy. In fact, on a typical Node.js process with default memory limits, this will crash with a heap out of memory error. And even if it doesn't crash, while you're waiting for the entire file to load, your application is doing nothing useful.

The Async Generator Solution:

const fs = require('fs')
const readline = require('readline')

async function* readLines(filepath) {
  const fileStream = fs.createReadStream(filepath)
  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity,
  })

  for await (const line of rl) {
    yield line
  }
}

// Usage - memory stays flat regardless of file size
for await (const line of readLines('./massive-log-file.log')) {
  if (line.includes('ERROR')) {
    await alertTeam(line)
  }
}

Why This Works:

Under the hood, fs.createReadStream reads the file in small chunks (typically 64KB at a time). The readline interface takes those chunks and intelligently splits them into lines, handling the messy details of lines that span multiple chunks.

By wrapping this in an async generator, we get a clean abstraction: "give me lines one at a time." The file is never fully loaded into memory—we're processing a stream. Whether your file is 1KB or 100GB, your memory usage stays roughly constant.

The crlfDelay: Infinity option, by the way, ensures that we correctly handle Windows-style line endings (\r\n) by treating \r followed by \n as a single line break, no matter how much time passes between reading them.

This pattern is beautiful because it composes so well. Want to filter lines? Add another async generator:

async function* filterLines(lines, predicate) {
  for await (const line of lines) {
    if (predicate(line)) {
      yield line
    }
  }
}

// Compose them together
const lines = readLines('./server.log')
const errors = filterLines(lines, (line) => line.includes('ERROR'))

for await (const error of errors) {
  console.log('Found error:', error)
}

Notice how we're not storing any intermediate arrays. The filterLines generator consumes from readLines one item at a time, checks the predicate, and only yields matching lines. It's like a pipeline of transformations, but with zero memory overhead for buffering.

Trust me, when you start composing async generators like this, it's noticeably different in a positive way. Your code becomes declarative pipelines instead of imperative loops.

3. Real-Time Event Streams

Modern applications often need to handle real-time data—WebSocket messages, Server-Sent Events, or even database change streams.

The Problem:

// Traditional callback approach
const ws = new WebSocket('wss://api.example.com/stream')

ws.onmessage = (event) => {
  const data = JSON.parse(event.data)
  // Hard to compose, hard to test, hard to reason about
  processMessage(data)
}

ws.onerror = (error) => {
  // Error handling is separate from data handling
  handleError(error)
}

Callbacks are fine for simple cases, but they don't compose well. What if you want to batch messages? Filter them? Transform them? You end up with spaghetti code. And testing becomes a nightmare—how do you simulate a sequence of messages arriving over time?

The fundamental issue is that callbacks are "push-based"—the WebSocket pushes data to you whenever it wants. You have no control over the pace. This is the opposite of what we want for composition.

The Async Generator Solution:

async function* websocketMessages(url) {
  const ws = new WebSocket(url)
  const messageQueue = []
  let resolve = null

  ws.onmessage = (event) => {
    const data = JSON.parse(event.data)
    if (resolve) {
      // Someone is waiting for a message - give it to them directly
      resolve(data)
      resolve = null
    } else {
      // No one waiting - queue it for later
      messageQueue.push(data)
    }
  }

  ws.onerror = (error) => {
    throw error
  }

  // Wait for connection before yielding anything
  await new Promise((res) => (ws.onopen = res))

  try {
    while (ws.readyState === WebSocket.OPEN) {
      if (messageQueue.length > 0) {
        // Messages waiting in queue - yield immediately
        yield messageQueue.shift()
      } else {
        // No messages - wait for the next one
        yield await new Promise((res) => (resolve = res))
      }
    }
  } finally {
    // Cleanup: always close the connection when done
    ws.close()
  }
}

// Now you can use it like any other async iterable!
for await (const message of websocketMessages('wss://api.example.com/stream')) {
  console.log('Received:', message)

  if (message.type === 'shutdown') {
    break // Clean shutdown, finally block closes connection
  }
}

Why This Works:

This pattern converts a "push-based" API (WebSocket callbacks) into a "pull-based" API (async iteration). The key insight is the messageQueue and resolve variables working together:

1. When a WebSocket message arrives and someone is already waiting (via for await), we resolve their promise immediately with the data.
2. When a message arrives but no one is waiting, we queue it.
3. When the consumer asks for the next message and the queue has data, we yield immediately.
4. When the consumer asks but the queue is empty, we create a promise that will be resolved when the next message arrives.

This creates backpressure—if your consumer is slow, messages queue up rather than being dropped or causing the callback to block. And the finally block ensures the WebSocket is always properly closed, even if you break out of the loop or an error occurs.

The beauty here is that your WebSocket handling now looks just like any other data processing code. You can compose it with the same filter, map, and batch generators we built earlier. You can write unit tests that just yield mock messages. The mental model becomes unified.

4. Rate-Limited API Calls

Here's a problem I've run into countless times: you need to call an external API for each item in a list, but you can't hammer the API or you'll get rate limited.

The Problem:

// This will get you rate limited fast
async function enrichAllUsers(userIds) {
  const results = await Promise.all(
    userIds.map((id) => fetch(`/api/external/user/${id}`))
  )
  return results
}

If you have 1,000 user IDs, this fires off 1,000 concurrent requests instantly. Most APIs will reject the vast majority of them with 429 (Too Many Requests) errors.

Or the other extreme:

// This is painfully slow - one at a time
async function enrichAllUsers(userIds) {
  const results = []
  for (const id of userIds) {
    const result = await fetch(`/api/external/user/${id}`)
    results.push(result)
    await sleep(100) // Manual rate limiting
  }
  return results
}

This works but has two problems: the rate limiting is mixed in with the business logic, and you've hard-coded the timing. What if the API allows bursts? What if different endpoints have different limits?

The Async Generator Solution:

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms))
}

async function* rateLimited(items, requestsPerSecond) {
  const delay = 1000 / requestsPerSecond
  let lastRequest = 0

  for (const item of items) {
    const now = Date.now()
    const timeSinceLastRequest = now - lastRequest

    if (timeSinceLastRequest < delay) {
      await sleep(delay - timeSinceLastRequest)
    }

    yield item
    lastRequest = Date.now()
  }
}

async function* enrichUsers(userIds) {
  for await (const id of rateLimited(userIds, 10)) {
    // 10 requests per second
    const response = await fetch(`/api/external/user/${id}`)
    yield await response.json()
  }
}

// Usage
for await (const enrichedUser of enrichUsers(userIds)) {
  await saveToDatabase(enrichedUser)
  // Results stream in at a controlled pace
}

Why This Works:

The rateLimited generator is a reusable "throttle valve" that you can wrap around any iterable. It tracks the timestamp of the last yielded item and waits if necessary before yielding the next one. The beauty is that this logic is completely decoupled from what you're actually doing with the items.

Let's break down the timing logic:

1. We calculate the minimum delay between items (for 10 requests/second, that's 100ms)
2. Before yielding each item, we check how long it's been since the last yield
3. If not enough time has passed, we sleep for the remainder
4. We yield the item and record the current timestamp

The key insight is that we only wait when necessary. If the consumer is slow (maybe saveToDatabase takes 200ms), we don't add extra delay—we're already under the rate limit. The throttling only kicks in when the consumer would otherwise be too fast.

What I love about this approach is that the rate limiting logic is completely separate from the business logic. You could swap in a different rate limiting strategy without changing your enrichment code:

// Token bucket for bursty traffic
async function* tokenBucket(items, tokensPerSecond, maxTokens) {
  let tokens = maxTokens
  let lastRefill = Date.now()

  for (const item of items) {
    // Refill tokens based on time passed
    const now = Date.now()
    const elapsed = now - lastRefill
    tokens = Math.min(maxTokens, tokens + (elapsed / 1000) * tokensPerSecond)
    lastRefill = now

    // Wait if no tokens available
    while (tokens < 1) {
      await sleep(100)
      const now = Date.now()
      tokens += ((now - lastRefill) / 1000) * tokensPerSecond
      lastRefill = now
    }

    tokens -= 1
    yield item
  }
}

Same interface, different strategy. Your business logic doesn't change at all.

5. Database Cursor Iteration

When you're dealing with large datasets in a database, loading everything into memory is a recipe for disaster. Most databases support cursors for this exact reason.

The Problem:

// MongoDB example - the memory-hungry way
async function processAllOrders() {
  const orders = await Order.find({ status: 'pending' }).exec()

  // If you have 100,000 pending orders, you're in trouble
  for (const order of orders) {
    await processOrder(order)
  }
}

When you call .exec() on a Mongoose query, it fetches ALL matching documents into memory at once. With 100,000 orders, each containing nested products and customer info, you might be looking at several gigabytes of data. This is a common source of production outages—the query works fine in development with 100 records, then crashes in production with 100,000.

The Async Generator Solution:

async function* findOrdersWithCursor(query) {
  const cursor = Order.find(query).cursor()

  for await (const order of cursor) {
    yield order
  }
}

// Usage - memory-efficient iteration over millions of records
for await (const order of findOrdersWithCursor({ status: 'pending' })) {
  await processOrder(order)

  // Can stop early if needed
  if (reachedDailyLimit()) {
    break
  }
}

Why This Works:

When you call .cursor() instead of .exec(), MongoDB uses a server-side cursor. Instead of sending all documents at once, the database sends them in batches (typically 101 documents at a time by default). When you exhaust one batch, it automatically fetches the next.

The Mongoose cursor implements the async iterable protocol, so we can use for await directly. By wrapping it in our own generator, we get a clean abstraction that hides the MongoDB-specific details.

In other words, the database sends you records one at a time (or in small batches), and you process them as they arrive. Your memory usage stays flat regardless of how many records match your query.

Here's an even more powerful pattern—combining cursor iteration with batching:

async function* batch(asyncIterable, size) {
  let currentBatch = []

  for await (const item of asyncIterable) {
    currentBatch.push(item)

    if (currentBatch.length >= size) {
      yield currentBatch
      currentBatch = []
    }
  }

  // Don't forget the final partial batch!
  if (currentBatch.length > 0) {
    yield currentBatch
  }
}

// Process orders in batches of 100
const orders = findOrdersWithCursor({ status: 'pending' })

for await (const orderBatch of batch(orders, 100)) {
  // Bulk operations are often more efficient
  await Order.bulkWrite(
    orderBatch.map((order) => ({
      updateOne: {
        filter: { _id: order._id },
        update: { $set: { processedAt: new Date() } },
      },
    }))
  )

  console.log(`Processed batch of ${orderBatch.length} orders`)
}

Why batching matters: While processing one-at-a-time is memory-efficient, it's often not the most efficient for throughput. Database operations have overhead (network round-trips, transaction handling), and bulk operations amortize that overhead across many records. The batch generator gives us the best of both worlds: we never hold more than 100 orders in memory, but we get the efficiency of bulk writes.

This is a done deal for handling large datasets efficiently.

The Composability Superpower

If there's one thing I want you to take away from this post, it's that async generators compose beautifully. Let me show you what I mean:

// These are all reusable building blocks
async function* map(iterable, fn) {
  for await (const item of iterable) {
    yield await fn(item)
  }
}

async function* filter(iterable, predicate) {
  for await (const item of iterable) {
    if (await predicate(item)) {
      yield item
    }
  }
}

async function* take(iterable, count) {
  let taken = 0
  for await (const item of iterable) {
    if (taken >= count) break
    yield item
    taken++
  }
}

// Now compose them into a pipeline
const pipeline = take(
  filter(
    map(fetchUsers(), async (user) => ({
      ...user,
      enriched: await enrichUser(user.id),
    })),
    async (user) => user.enriched.isActive
  ),
  100 // Only need first 100 active users
)

for await (const user of pipeline) {
  console.log(user)
}

What's happening in this pipeline:

1. fetchUsers() yields users one at a time from the paginated API
2. map enriches each user by calling another API
3. filter only passes through users where isActive is true
4. take stops after 100 users (which stops the entire pipeline!)

The profound thing here is lazy evaluation. If only 100 of the first 150 fetched users are active, we only fetch 150 users. If the 100th active user is on page 3, we stop fetching after page 3. The take(100) at the end propagates "stop" signals all the way back through the pipeline.

Luckily, we can build these utilities once and reuse them everywhere. The declarative nature makes the code self-documenting—you can see exactly what transformations are being applied just by reading the pipeline.

When NOT to Use Async Generators

I'd be doing you a disservice if I didn't mention when async generators might not be the right choice:

1. Simple, one-time fetches: If you're just fetching one page of data, a regular async function is simpler. Don't add abstraction when it doesn't help.

2. When you need all data upfront: Some operations genuinely need all the data before they can proceed (like sorting or calculating totals). You can't sort a stream—you need the complete dataset. In these cases, just collect everything into an array first.

3. Highly parallel operations: Async generators are inherently sequential—you process one item, then the next. If you need to process 100 items in parallel, Promise.all with p-limit for concurrency control is a better fit.

4. Simple event handlers: For simple click handlers or one-off events, callbacks are perfectly fine. Not every event stream needs to be an async iterable.

5. When performance is critical: The generator machinery does add a small amount of overhead. For extremely hot paths processing millions of items per second, a tight loop might be faster. Profile first, optimize second.

Conclusion

Async generators are one of those features that seem unnecessary until you need them—and then they're absolutely indispensable. They give you:

• Memory efficiency: Process data as it arrives instead of loading everything into memory
• Composability: Build pipelines of transformations that are easy to read and maintain
• Control: Pause, resume, or cancel iteration at any point
• Elegance: Turn callback-based APIs into clean, linear code
• Lazy evaluation: Only do work when the consumer asks for more data

The mental model shift is from "collect everything, then process" to "process as it flows." Once that clicks, you'll start seeing opportunities to use async generators everywhere.

The next time you find yourself dealing with paginated APIs, large files, real-time streams, rate limiting, or database cursors, give async generators a try. I think you'll be pleasantly surprised.

And that concludes the end of this post! I hope you found this valuable and look out for more in the future!