Implementing AI Streaming Responses with JSON Lines Chunked Communication Instead of SSE

Background

When streaming AI chat responses, Server-Sent Events (SSE) are commonly used. They are also adopted by APIs from OpenAI and Anthropic, as well as by MCP server responses.

In fact, I implemented several AI chat projects that modified responses from AI platforms while streaming them to the browser. In doing so, I encountered an issue where SSE did not work because of certain intermediary proxies and load balancers, such as AWS App Runner.

After taking a closer look at the SSE specification, I no longer felt that using SSE was right when the purpose was not actually event notification. The API's block data itself is JSON. This is also the same format as the structured logging sent to services such as CloudWatch Logs today (I had already been working on structuring application logs as JSON).

Moving from SSE to JSON Lines Chunked Communication

What I came up with was a combination of Transfer-Encoding: chunked and Content-Type: application/jsonl (which is not defined by the IAEA). With this approach, even if a proxy or load balancer buffers the response and returns it as a single body rather than chunks, only streaming is lost; the final complete data remains unchanged. Because it is JSON Lines (NDJSON), all you need to do is split on line feeds (LF) and JSON-parse each line. It is also easy to inspect in browser developer tools.

However, implementing this from scratch every time is a bit of work, so I implemented and published jsonl-webstream, an npm library of stream utilities for browsers and servers (Node.js). The library has zero dependencies.

tilfin / jsonl-webstream

Lightweight library for JSON Lines web stream between browsers and Node.js environments

jsonl-webstream

Lightweight library for JSON Lines web stream between browsers and Node.js environments

Overview

This library provides utilities for processing JSON Lines formatted data through the Web Streams API It enables efficient streaming of JSON Lines data with minimal memory overhead across browsers and Node.js environments.

Installation

npm install jsonl-webstream

Usage

Reading JSON Lines

import { createJsonLinesReceiver } from 'jsonl-webstream';

// With fetch API
async function processJsonLines() {
  const response = await fetch('https://example.com/stream');
  const reader = response.body.getReader();
  const jsonlStream = createJsonLinesReceiver(reader);

  for await (const jsonData of jsonlStream) {
    // Process each JSON object
    console.log(jsonData);
  }
}

Writing JSON Lines

import { createJsonLinesSender } from 'jsonl-webstream';
function handleRequest(reply) {
  // Create a JSON Lines writer and its associated stream
  const {

…

View on GitHub

I named the library webstream to emphasize that it uses the Web Streams API, rather than Node.js's traditional Stream. At present, the two inevitably tend to coexist because of library support, but I would like to move clearly toward Web Streams.

How to Use jsonl-webstream

Server utility function: createJsonLinesSender

When you call createJsonLinesSender() with no arguments, it returns a stream and a writer. The stream is a ReadableStream itself and can be used as an API response. The writer instance lets you send data with write(plainData), mainly while processing responses from an AI platform. Finally, just call close(). You can set a callback handler with onCancel(Callback) for when the connection is closed on the client side (in the browser).

Client utility function: createJsonLinesReceiver

Simply call it as stream = createJsonLinesReceiver(response.body.getReader()) after response = fetch(...).
This stream is also a ReadableStream, so it handles the server response transparently. Inside a for await (const plainData of stream) loop, append the contents of each data object to the UI.

Implementation

Server

import { createJsonLinesSender } from "jsonl-webstream";

app.post("/api/chat", async (c) => {
  const { messages } = await c.req.json();

  const upstreamController = new AbortController();
  const { stream, writer } = createJsonLinesSender();

  // If the browser calls reader.cancel(), abort the upstream generation request too.
  writer.onCancel(() => upstreamController.abort());

  // Return the response first, then write chunks as they arrive from upstream.
  void streamCompletion(messages, upstreamController.signal, writer);

  return new Response(stream, {
    headers: { "Content-Type": "application/jsonl; charset=utf-8" },
  });
});

async function streamCompletion(messages, signal, writer) {
  try {
    const client = new OpenAI();
    const completion = await client.chat.completions.create(
      { model: "gpt-x", messages, stream: true },
      { signal },
    );

    for await (const chunk of completion) {
      const text = chunk.choices[0]?.delta?.content;
      if (text) writer.write({ text });
    }
  } catch (error) {
    writer.abort(error);
  } finally {
    writer.close();
  }
}

Immediately return the stream from createJsonLinesSender() as a Response, then call writer.write({ text }) from the separate AI invocation process. Convert each AI response chunk into the data you need and send it.

Client

import { createJsonLinesReceiver } from "jsonl-webstream";

let reader = null;

async function sendMessage(messages, render) {
  const response = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
  });
  if (!response.ok || !response.body) {
    throw new Error(await response.text() || `HTTP ${response.status}`);
  }

  reader = response.body.getReader();
  const stream = createJsonLinesReceiver(reader);
  let receivedText = "";

  try {
    for await (const chunk of stream) {
      receivedText += chunk.text;
      render(receivedText); // Update the screen every time data is received.
    }
  } finally {
    reader = null;
  }
}

stopButton.addEventListener("click", () => {
  void reader?.cancel();
});

Pass response.body.getReader() to createJsonLinesReceiver(). When you read the returned ReadableStream with for await...of, each JSONL line is returned as an object, so you can add the received delta directly to the UI. When the stop button calls cancel() on the reader, writer.onCancel() is called on the server.

Supplementary Material

Comparing Raw Data: SSE vs. JSON Lines Chunked Communication

The key-value structure is removed, making the data one level flatter, more compact, and free of blank lines in between.

text/event-Stream

event: message
{"type":"text","delta":"Hello"}

event: message
data: {"type":"text","delta":" world"}

application/jsonl

{"type":"text","delta":"Hello"}
{"type":"text","delta":" world"}