DEV Community

王磊
王磊

Posted on

Parsing multi-gigabyte Outlook PST files in the browser

#javascript #webdev #email #performance

Disclosure up front: I build Mailward, a browser-based PST/OST/OLM/MBOX/EML viewer. These are the implementation notes.

A PST file is not a folder of emails. It is a single-file B-tree database — Microsoft documents the layout as [MS-PST] — and the mail inside exists only as sets of typed properties. To show one message you walk the B-trees, resolve the property set, and reconstruct an RFC 5322 message from scratch.

Now do that in a browser tab, for a 4 GB file, without uploading it anywhere and without loading it into memory.

The constraint that shapes everything

The whole point of a browser-side viewer is privacy: the file never leaves the machine. That rules out "stream it to a server". And FileReader.readAsArrayBuffer on the whole file rules itself out the moment archives pass a gigabyte — mobile Safari will simply kill the tab.

So the parser must do random access over a File it never fully reads.

One method to intercept

We build on pst-extractor, a faithful JS port of java-libpst. It has one architectural gift: every byte it reads funnels through a single method, PSTFile.readSync(buffer, length, position).

That means you can subclass PSTFile and reroute all I/O without touching parser internals:

import { PSTFile } from "pst-extractor";

let pendingReader: RandomAccessReader | null = null;

class ReaderBackedPSTFile extends PSTFile {
  archiveReader!: RandomAccessReader;

  constructor(reader: RandomAccessReader) {
    pendingReader = reader; // super() reads the 514-byte header via readSync
    try {
      super(Buffer.alloc(0));
    } finally {
      pendingReader = null;
    }
    this.archiveReader = reader;
  }
}

(ReaderBackedPSTFile.prototype as any).readSync = function (
  buffer: Buffer, length: number, position: number
): number {
  return (this.archiveReader ?? pendingReader).read(buffer, length, position);
};
Enter fullscreen mode Exit fullscreen mode

Two quirks worth explaining:

  • The override lives on the prototype, outside the class body, because PSTFile's constructor immediately reads the 514-byte header — so the replacement must already be installed while super() runs.
  • Instance fields don't exist yet at that point, hence the module-scoped pendingReader handoff. Workers are single-threaded, so this is safe.

The reader: 64 KB pages, LRU-capped

RandomAccessReader is a paged view of the file:

  • 64 KB pages, loaded on demand.
  • LRU cache capped at 1024 pages — 64 MB worst case, regardless of file size.
  • Reads that span page boundaries are assembled from multiple pages.

Inside a Web Worker the page loader is almost embarrassingly small, because FileReaderSync gives you synchronous random access to a user-selected File:

class FileSlicePageLoader {
  private reader = new FileReaderSync();
  constructor(private file: Blob) {}

  loadPage(start: number, length: number): Uint8Array {
    return new Uint8Array(
      this.reader.readAsArrayBuffer(this.file.slice(start, start + length))
    );
  }
}
Enter fullscreen mode Exit fullscreen mode

file.slice() doesn't copy — it's a view. We measured a 4 KB slice read at ~200 µs in Chrome, which makes B-tree walks feel instant. The same RandomAccessReader interface is backed by a plain Buffer in Node, so the entire parser is testable outside the browser.

If you're in Node, you don't need any of this

pst-extractor's constructor accepts either a Buffer or a filename. The 2 GB failures people report almost always come from fs.readFileSync → Buffer. Pass the path instead and the library reads through a file descriptor with no Buffer ceiling:

const pst = new PSTFile("/path/to/large.pst"); // descriptor-backed
Enter fullscreen mode Exit fullscreen mode

What this doesn't solve

Honesty section: none of this repairs a corrupted PST — if scanpst can't open it, a reader can't either. And ANSI-era PSTs (Outlook 97–2002, the 2 GB-capped variant) parse through the same pipeline in theory, but genuine ANSI files written by 20-year-old Outlook are nearly impossible to test against, so we claim "likely works, fails loudly" rather than "supported".

More format field notes (OST's account lock, OLM's timezone-less UTC timestamps, MBOX From-munging) live in this repo: email-archive-formats.

Top comments (0)