DEV Community

Usman Khan
Usman Khan

Posted on Originally published at ctousman.com

Server-Sent Events (SSE) Backpressure Breaks: Memory Leaks and Socket Starvation in High-Throughput LLM Pipelines

#performance #javascript #node #ai

Streaming LLM completions over Server-Sent Events (SSE) looks trivial in sandbox demos, but unhandled backpressure under real-world network conditions can destroy backend application servers. When slow clients fail to consume token chunks as fast as upstream model APIs generate them, unbounded Node.js internal stream buffers trigger memory leaks and OOM process crashes. Here is an architectural deep dive into enforcing stream backpressure, managing drain events, and protecting socket thread pools.

Integrating generative AI features into enterprise SaaS interfaces relies heavily on streaming response tokens via Server-Sent Events (SSE) to lower Time-To-First-Token (TTFT). However, naïve stream piping implementations assume the client's network connection can consume data at the same rate the upstream model provider (e.g., OpenAI or Anthropic) yields tokens. Under real-world network conditions, this assumption breaks down catastrophically.

1. The Mechanics of Unhandled Backpressure in Event Streams

In Node.js and similar asynchronous event-driven runtimes, HTTP responses are writable streams backed by internal memory buffers. When an upstream LLM API yields token deltas at high velocity (e.g., 80 to 120 tokens per second), the application server forwards these chunks directly into the client's HTTP response socket.

How Slow Clients Trigger Node.js Heap Exhaustion

  • Producer-Consumer Velocity Mismatch: An upstream LLM provider generates text rapidly, while a client on a high-latency mobile connection or throttled browser tab consumes data slowly.
  • Unbounded Internal Buffering: When res.write(chunk) is called faster than the TCP socket can flush frames over the wire, Node.js buffers the overflow chunks in process memory.
  • Cascading Out-Of-Memory (OOM) Crashes: Under high concurrency (e.g., 500 parallel streaming sessions), unbounded buffer growth consumes gigabytes of memory, ending in JavaScript heap out of memory process crashes across backend instances.

2. The Flaw in Naive SSE Controller Implementations

A standard anti-pattern in Node.js/Express controllers involves directly listening to LLM token events and pushing them into the response without inspecting return values or handling stream backpressure:

// NAÏVE ANTI-PATTERN: Zero Backpressure Control
export async function streamLLMResponseNaive(req: Request, res: Response) {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  const upstreamStream = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: req.body.prompt }],
    stream: true,
  });

  // FAILURE VECTOR: If client socket is slow, res.write returns FALSE
  // Naïve code ignores this signal, buffering unbounded chunks in RAM!
  for await (const chunk of upstreamStream) {
    const token = chunk.choices[0]?.delta?.content || '';
    res.write(`data: ${JSON.stringify({ token })}\n\n`);
  }

  res.write('data: [DONE]\n\n');
  res.end();
}
Enter fullscreen mode Exit fullscreen mode

When res.write() returns false, it explicitly signals that the stream's internal highWaterMark threshold has been breached. Continuing to push data ignores this backpressure boundary.

3. Production-Ready Backpressure-Aware SSE Pipeline

To safely stream LLM tokens at scale, backend services must inspect res.write() signals. If a write returns false, the worker must pause the upstream reader until the underlying TCP socket fires a 'drain' event. The drain wait must also be cancellable: if the client disconnects while the buffer is full, 'drain' never fires, and an un-cancellable wait becomes its own memory leak.

// PRODUCTION PATTERN: Explicit Backpressure & Abort Handling
import { Request, Response } from 'express';
import { once } from 'events';

export async function streamLLMResponseResilient(req: Request, res: Response) {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('X-Accel-Buffering', 'no'); // Disable Nginx proxy buffering

  const abortController = new AbortController();

  // Clean up upstream request if client abruptly disconnects.
  // Listen on res, not req: req 'close' fires as soon as the request body is read.
  res.on('close', () => {
    if (!res.writableEnded) abortController.abort();
  });

  try {
    const upstreamStream = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [{ role: 'user', content: req.body.prompt }],
      stream: true,
    }, { signal: abortController.signal });

    for await (const chunk of upstreamStream) {
      if (abortController.signal.aborted) break;

      const token = chunk.choices[0]?.delta?.content || '';
      if (!token) continue;

      const payload = `data: ${JSON.stringify({ token })}\n\n`;

      // 1. Write chunk and inspect backpressure buffer capacity
      const canContinue = res.write(payload);

      // 2. If internal buffer is full, pause until the socket drains
      //    (or the client disconnects, which rejects with AbortError)
      if (!canContinue) {
        await once(res, 'drain', { signal: abortController.signal });
      }
    }

    if (!abortController.signal.aborted) {
      res.write('data: [DONE]\n\n');
      res.end();
    }
  } catch (error: any) {
    if (error.name === 'AbortError') {
      console.log('[SSE_STREAM] Client disconnected mid-stream. Resource freed.');
    } else {
      console.error('[SSE_STREAM_ERROR]', error);
      // Headers are already sent once streaming starts, so just close the stream
      if (!res.headersSent) res.status(500);
      res.end();
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

4. Reverse Proxy Tuning: Nginx and AWS ALB Buffering Traps

Backpressure handling in application code is rendered useless if reverse proxies in your network path silently buffer event streams in memory.

  • Nginx Proxy Buffering: By default, Nginx buffers upstream HTTP responses before forwarding them to clients. This breaks real-time token delivery and obscures backpressure signals. Always pass the X-Accel-Buffering: no header or set proxy_buffering off; in location blocks.
  • AWS ALB Idle Timeouts: Ensure your AWS Application Load Balancer idle timeout exceeds the maximum expected LLM completion duration (e.g., set to 120+ seconds) to prevent premature 504 Gateway Timeouts on long reasoning chains.

5. Executive Summary & Systems Rules

  • Respect res.write() Return Values: Never write to an HTTP response stream blindly. Pause upstream generation when res.write() returns false and await the 'drain' event, with an abort signal so the wait can't hang forever.
  • Attach Abort Controllers to Client Sockets: Always listen for res.on('close') to terminate upstream LLM API connections immediately when users navigate away or close tab sockets.
  • Disable Proxy Buffering: Configure edge proxies and gateways explicitly to allow immediate, unbuffered chunk passthrough for text/event-stream content types.

Are your LLM streaming endpoints leaking memory or dropping connections under load? I help SaaS teams build production-grade AI streaming pipelines with proper backpressure, abort handling, and proxy configuration. Book a consultation →

Originally published on ctousman.com.

Top comments (1)

Collapse
 
dev_in_the_fog profile image
Jason Y. (dev_in_the_fog) •

Insightful breakdown! The architectural considerations for deterministic outputs and cost control were spot on.