Building a Local Search Index for a 200MB X Archive

--
title: "Building a Local Search Index for a 200MB X Archive"
description: "The archive arrives as a ZIP and tweets.js inside it often runs to a few hundred megabytes. Finding the one address you mentioned three years ago is not a text-editor job. A short script turns the archive into a searchable local index, plus why memory blows up when you stream the file the naive way."
tags: ["javascript", "node", "privacy", "datasette"]

canonical_url: https://digital-footprint-health.shop/blog/offline-archive-search-indexing

Unpack an X archive and the post text sits in a single file called tweets.js. It opens with an assignment, then one very large JSON array. Nothing is encrypted and nothing is indexed either, and the thing you want is usually one detail: a house number, an email address, the name of a hotel from one trip.

Reading it without installing anything is a problem already solved elsewhere and it works fine for a one-off question. This is the other direction: turn the archive into a local index you build once and query in milliseconds from then on.

Why loading it directly runs out of memory

Start with the numbers, since they decide how the script has to be written.

Object	Typical size	Note
Archive ZIP	150 to 400 MB	Several js files plus media folders
tweets.js	120 to 300 MB	UTF-8, one enormous line
Parsed object array	3 to 5x the source file	Per-object overhead in V8 dwarfs the text
Index file, stopwords dropped	15 to 40 MB	Depends on post count and token granularity

The third row is where things break. Parsing a couple of hundred megabytes of JSON in a single call, then holding the deduplicated strings resident, reliably hits the default heap ceiling and surfaces as a vague heap out of memory error. Raising the limit works, but there is no reason to: an index wants the text, not the object graph.

Streaming the structure away

The shape of the fix is to drop the assignment prefix, split the body at top level array boundaries, process each element and release it before moving on. Memory then scales with the largest individual post instead of with the total number of posts.

import fs from 'node:fs';
import readline from 'node:readline';

const src = process.argv[2];
const out = fs.createWriteStream('index.ndjson');

const rl = readline.createInterface({
  input: fs.createReadStream(src, { encoding: 'utf8' }),
  crlfDelay: Infinity,
});

let buf = '';
let written = 0;

rl.on('line', (line) => {
  buf += line;
  for (;;) {
    const end = findObjectEnd(buf, 0);
    if (end < 0) break;
    const chunk = buf.slice(0, end + 1);
    buf = buf.slice(end + 1);
    try {
      const obj = JSON.parse(chunk);
      const t = obj.tweet || obj;
      out.write(JSON.stringify({
        id: t.id_str,
        t: t.full_text || t.text,
        d: t.created_at,
        l: t.lang,
      }) + '\n');
      written++;
    } catch (e) {
      // a split boundary landed inside a string, keep accumulating
    }
  }
});

rl.on('close', () => {
  out.end();
  console.error('records=' + written);
});

The findObjectEnd helper walks the buffer tracking brace depth while respecting string state and escapes, and returns -1 when it reaches the end of the buffer without closing the current object. That is what makes an element-per-line streaming pass possible without loading the array.

Two details that bite

The prefix is not always the same. Older exports open with window.YTD.tweet.part0 =, newer ones use window.YTD.tweets.part0 =. Grep the first line before assuming, or strip everything up to the first [ and let the parser deal with the rest.

HTML entities survive the parse. Tweet text arrives with escaped ampersands, angle brackets and quotes intact. Run a single decode pass over the extracted text field before writing, otherwise searches for a URL containing a query string will miss.

Querying it

With an NDJSON index and one record per line, grep is a legitimate search engine for a file this size, and it costs nothing to build.

grep -i -n 'hotel name' index.ndjson | head -20

When you want ranked results rather than exact matches, load the NDJSON into any local full-text engine. DuckDB reads it directly without an import step, and a single CREATE TABLE over the file gives you case-insensitive matching and counts:

SELECT id, d, t
FROM read_ndjson_auto('index.ndjson')
WHERE lower(t) LIKE '%hotel name%'
ORDER BY d DESC
LIMIT 20;

Rebuilding versus appending

You do not need to rebuild when the archive refreshes. Post identifiers increase monotonically, so take the largest id in the existing index, process only records above it, and append. That also gives you a resumable state: the high water mark is a number you can store and check.

The property worth remembering is that the archive is a snapshot. Anything you deleted after the export still exists inside it. Which makes the local index useful as a before picture, and a reasonable way to see which items you already removed and which ones still need attention elsewhere.

Comments

[Dev Support]

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‍‌‍