Progressive JSON in TypeScript: Loading and Processing Data Incrementally

Progressive JSON transmits and processes JSON in fragments, so important parts of the data can be displayed before the full payload arrives — similar to progressive images. Below is an overview with TypeScript examples for client and server.

The Progressive JSON Concept

Progressive JSON is like a Progressive JPEG: information loads in "layers." Instead of sending a complete JSON all at once, the server distributes data in chunks so the client can start processing and rendering immediately. That reduces perceived latency, especially in dynamic dashboards, social networks, and e-commerce apps.

For example, an API that returns a large JSON with metadata, article content, and comments can render the header and main body first, while comments load later.

Use Cases

Dashboards and monitoring systems — critical data first, details in the background.
Social networks — posts and main text immediately; comments and reactions progressively.
E-commerce — basic product info first; reviews or recommendations later.
Content systems — main article text without waiting for high-resolution images or multimedia.

This improves end-user experience and spreads load on client and server when data volume is large.

How It Works: Sending and Consuming Data in Fragments

The server can use Transfer-Encoding: chunked to send data in parts. Each fragment may be a partial JSON or contain placeholders replaced as new data arrives.

On the client, streams enable incremental reading. With Fetch and ReadableStream, decode each chunk and process data as it arrives.

TypeScript client example:

class ProgressiveJSONClient {
  async fetchProgressive(url: string, onProgress: (data: any) => void): Promise<any> {
    const response = await fetch(url);
    const reader = response.body?.getReader();
    const decoder = new TextDecoder();
    let buffer = '';

    if (!reader) throw new Error("Stream não disponível");

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      buffer += decoder.decode(value, { stream: true });
      const lines = buffer.split('\n');
      buffer = lines.pop() || ''; // Preserva dados incompletos

      for (const line of lines) {
        if (line.trim()) {
          try {
            const data = JSON.parse(line);
            onProgress(data);
          } catch (e) {
            console.error("Erro no parsing do trecho:", e);
          }
        }
      }
    }

    // Processa o que sobrou no buffer
    if (buffer.trim()) {
      try {
        return JSON.parse(buffer);
      } catch (e) {
        console.error("Erro final de parsing:", e);
        return null;
      }
    }
  }
}

Each line from the stream is decoded and processed. onProgress runs for each complete JSON object so the UI can update incrementally.

Server Implementation Example with Node.js

With Express, set the appropriate headers and send data in fragments:

const express = require('express');
const app = express();

app.get('/api/progressive', (req, res) => {
  res.setHeader('Content-Type', 'application/json');
  res.setHeader('Transfer-Encoding', 'chunked');

  // Primeiro chunk: Conteúdo inicial com placeholders
  res.write(JSON.stringify({ header: "$1", content: "$2" }) + '\n');

  // Simulação de delay para enviar dados progressivamente
  setTimeout(() => {
    res.write('/* $1 */ "Bem-vindo ao blog"\n');
    res.write('/* $2 */ { "text": "Este é o conteúdo do artigo", "comments": "$3" }\n');

    setTimeout(() => {
      res.write('/* $3 */ ["Comentário 1", "Comentário 2"]\n');
      res.end();
    }, 1000);
  }, 500);
});

app.listen(3000, () => console.log('Servidor rodando na porta 3000'));

The server starts with placeholders, then fills them in with simulated delays so the client can render header and main content without waiting for the full response.

Libraries and Tools for Progressive Parsing

stream-json

Processes large JSON files in streams — ideal when the JSON is too large to load entirely into memory.

import { parser } from 'stream-json';
import { streamValues } from 'stream-json/streamers/StreamValues';
import * as fs from 'fs';

const pipeline = fs.createReadStream('large-file.json')
  .pipe(parser())
  .pipe(streamValues());

pipeline.on('data', ({ value }) => {
  console.log('Valor lido:', value);
});

pipeline.on('end', () => {
  console.log('Processamento concluído.');
});

JSONStream

Lightweight library to filter and transform JSON in real time for stream-based systems:

import * as JSONStream from 'JSONStream';
import * as fs from 'fs';

const stream = fs.createReadStream('large-file.json')
  .pipe(JSONStream.parse('items.*'));

stream.on('data', (data) => {
  console.log('Item:', data);
});

stream.on('end', () => {
  console.log('Processamento do JSON concluído.');
});

fast-json-parse and jsonparse

fast-json-parse focuses on performance and safety (including schema validation). jsonparse is minimalist and event-based, useful for handling data as it arrives.

import { parse } from 'fast-json-parse';

const jsonString = '{"name": "John", "age": 30}';
const result = parse(jsonString);

if (result.err) {
  console.error('Erro no parsing:', result.err);
} else {
  console.log('JSON parseado:', result.value);
}

These tools help build data pipelines with lower memory use and faster response times.

Best Practices for Progressive JSON

Use streams for large files or data flows — don't load everything into memory.
Validate received JSON, especially from external or untrusted sources (e.g. AJV for schemas).
Define TypeScript interfaces and types to keep data integrity.
Wrap parsing in try-catch and handle incomplete or corrupted chunks.
Use placeholders and fragment identifiers to reconcile partial data between server and client.

Final Considerations

Progressive JSON helps when latency and perceived performance matter: render critical UI parts without waiting for the full payload. Libraries like stream-json and JSONStream make this practical in TypeScript/Node.js. Pair them with validation and solid error handling so the pipeline stays reliable under high data volume.

Worth trying if you work with dashboards, real-time apps, or any system that ships large JSON responses.

DEV Community