Foretaste

Posted on Oct 7

Listing a 40 MB ZIP Over HTTP by Reading Only 67 KB of It

#webdev #programming #javascript #performance

Normally, if you want to see what is inside a remote .zip file on the web, you have to download the whole file to your disk or upload it to a cloud tool. If the archive is 40 MB or 1 GB and you only need to check one file name or grab a single text file, downloading the whole thing is slow and wastes bandwidth.

Here is how you can read the entire directory listing of a remote ZIP by fetching only ~67 KB over HTTP using byte range requests, without downloading the full archive.

Key Metric / Real Measurement

Tested against a remote server logging all request payloads:

Archive total size: 40,013,838 bytes (~40 MB) with 10 files.
Bytes fetched to list all files: ~67 KB (0.17% of total archive size).
Overhead to extract a single file inside it: 97 bytes (local header + entry data).

Why this is possible

The ZIP format specification places the table of contents (called the Central Directory) at the very end of the file, not at the beginning:

[ local header + file data ] ... [ Central Directory ] [ EOCD (22 bytes) ]
                                 ^                     ^
                                 Table of contents     End of Central Directory

The last record in the file is the End of Central Directory (EOCD). It is 22 bytes long and contains:

Total number of entries in the ZIP
Byte size of the Central Directory
Byte offset where the Central Directory starts

(Note: A ZIP can have an optional trailing comment up to 65,535 bytes, so the EOCD is located somewhere within the last 22 + 65,535 = 65,557 bytes).

If the remote web server supports HTTP Range requests (Accept-Ranges: bytes), we can request just the tail of the file.

How it works step-by-step

Step 1: Fetch the tail (EOCD) with a Range request

Send an HTTP Range request for the end of the file. Most archives have no comment, so a 4 KB read is usually enough. If no EOCD signature is found, fall back to 65,557 bytes:

// Request the last 4096 bytes first
const FIRST_TAIL = 4096;
const LONGEST_TAIL = 22 + 0xffff; // 65557 bytes

let from = Math.max(0, size - FIRST_TAIL);
let tail = await fetchRange(url, from, size);
let at = findEOCD(tail); // scans backward for signature 0x06054b50

if (at < 0 && from > 0) {
  // Trailing comment present: fetch the full 65k tail
  from = Math.max(0, size - LONGEST_TAIL);
  tail = await fetchRange(url, from, size);
  at = findEOCD(tail);
}

Note: Always scan backwards for 0x06054b50 because that 4-byte sequence could randomly exist inside compressed payload bytes.

Step 2: Read the Central Directory

From the 22-byte EOCD record, pull the count, size, and offset:

const end = new DataView(tail.buffer, tail.byteOffset + at, 22);
const count    = end.getUint16(10, true);
const cdSize   = end.getUint32(12, true);
const cdOffset = end.getUint32(16, true);

Then send a second range request for cdOffset to cdOffset + cdSize (or slice it directly from memory if it was already included in the tail you fetched):

// Each record starts with 0x02014b50
while (p + 46 <= cd.length && view.getUint32(p, true) === 0x02014b50) {
  const flags      = view.getUint16(p + 8, true);
  const nameLength = view.getUint16(p + 28, true);
  const name       = new TextDecoder(flags & 0x0800 ? 'utf-8' : 'latin1')
                       .decode(cd.subarray(p + 46, p + 46 + nameLength));

  entries.push({
    name,
    packedSize: view.getUint32(p + 20, true),
    uncompressedSize: view.getUint32(p + 24, true),
    offset: view.getUint32(p + 42, true)
  });

  p += 46 + nameLength + view.getUint16(p + 30, true) + view.getUint16(p + 32, true);
}

This gives you the full file tree, file sizes, and offsets.

Step 3: Extract a single file on demand

When the user clicks on a specific file:

Fetch the 30-byte local header at entry.offset to determine the start of the compressed data.
Fetch only the compressed byte range (entry.packedSize).
Decompress in the browser using the native Streams API (no external libraries needed):

const stream = new Blob([packedBytes]).stream()
  .pipeThrough(new DecompressionStream('deflate-raw'));
const fileBytes = new Uint8Array(await new Response(stream).arrayBuffer());

Quick test with curl (Try it yourself)

You can verify this in your terminal on any remote ZIP:

# 1. Check if the server supports ranges (look for Accept-Ranges: bytes)
curl -sI https://example.com/large.zip | grep -i accept-ranges

# 2. Download only the last 65 KB of the archive
curl -s -r -65557 https://example.com/large.zip -o tail.bin

# 3. unzip will read the central directory directly from the tail
unzip -l tail.bin

Edge cases and limitations

Small files (< 100 KB): Two HTTP round trips to fetch 64 KB of an 80 KB archive take longer than a single GET request for the whole file.
Servers ignoring Range headers: Some servers or CDNs do not honor byte ranges and respond with 200 OK sending the whole file anyway.
ZIP64: Archives over 4 GB or with more than 65,535 files store offsets in a separate PK\x06\x06 record.
Tarballs (.tar.gz): Does not work on tar.gz archives because gzip streams do not have an index at the end and must be decoded sequentially from byte 0.

About the project

I implemented this technique in Foretaste, a lightweight browser extension that lets you right-click any file link to inspect ZIP contents, read documents (PDF, DOCX, XLSX), or view media directly in your browser without downloading.

DEV Community