Why I Replaced Everything With a Homegrown MFT Search Engine (Memory: Hundreds of MB a Few MB)

I build yyzTools — a free, local-first Windows productivity suite that folds 40+ tools (command palette, clipboard history, OCR, file preview, batch processing) into one installer. This post is about the 1.0.5 release, where I did something that sounds insane: I removed my dependency on Everything — the gold standard of Windows file search — and wrote my own engine.

---

The setup

For about two years, file search in yyzTools was powered by Everything. And let me be clear: Everything is phenomenal. Sub-second full-disk search, a mature query syntax (ext:pdf size:>100mb dm:today), battle-tested by millions of users. Integrating it was one of the best decisions I made early on — instant world-class search for free.

But there was a catch that grew more annoying with every release: Everything is designed to be used standalone. Its index lives in its own service process. On a machine with a lot of files, that index holds hundreds of MB of RAM, resident from boot to shutdown.

For a standalone search tool, that's a perfectly reasonable price. But yyzTools is a resident suite — the whole point is that it sits quietly in your tray until you hit Alt+Space. Every MB of resident memory lowers the threshold at which users uninstall you. And search is the heaviest feature in the suite.

So the conflict was structural: the dimension Everything optimizes least for (being embedded in someone else's memory-sensitive app) was exactly my core requirement.

In 1.0.5, I replaced it. Index memory went from hundreds of MB to a few MB. Query syntax stayed Everything-compatible. Here's how.

Wrong turn #1: the "official" API is a trap

When you start researching full-disk enumeration on Windows, you find FSCTL_ENUM_USN_DATA. Documented, blessed, gives you every file's FRN (file reference number) and parent FRN. Seems like the obvious choice.

I implemented it. 1.34 million files, 120 seconds to enumerate. Rejected.

The problem: USN records are a change journal, not a catalog. They don't carry file size or timestamps — so for every single file you have to call back with FSCTL_GET_NTFS_FILE_RECORD to fetch the full MFT record. A million random reads by FRN. That's not slow code; that's a wrong-shaped data channel.

The Everything way: read $MFT directly

Everything and WizFile don't use that API for full enumeration. They open the volume device (\\.\C:, read-only) and read the $MFT byte stream directly. The Master File Table is NTFS's census book: every file and directory is one fixed 1024-byte record, stored sequentially, containing name, size, and timestamps. Scan it linearly and you get everything in one pass of sequential IO:

// 1. Parse the $DATA run list of $MFT's own record (VCN→LCN mapping)
static bool ParseMftRuns(BYTE* mftRec, int len, std::vector<int64_t>& lcns, ...);

// 2. ReadFile the volume handle in chunks, per run
// 3. Fix up each 1024-byte record, extract $FILE_NAME / $DATA attributes

Two details worth knowing if you go this route:

• Hard links: one MFT record can carry multiple $FILE_NAME attributes (one per link, each pointing at its own parent directory). Each one must become a separate row in your index, or link results silently go missing.
• Permissions: reading the volume device needs more than a standard user token. That's why the engine runs as a separate process backed by a Windows service — the service holds the volume handle, the main app consumes queries over IPC, and the GUI never elevates.

Where the memory went: columnar storage

The full-enumeration part was honestly the easier half. The interesting question was: you now have 1.34 million rows of metadata. How do you hold them in RAM for a few MB instead of a hundred-plus?

The naive layout is an array of objects:

struct FileEntry {
    uint64_t frn, parentFrn, size;   // 24 B
    int64_t  mtime;                 // 8 B
    std::string name;               // 32 B + heap allocation per file
    bool isDir;                     // 1 B + 7 B padding
};

Padding, per-string heap allocations, allocator headers, terrible cache locality. That's your hundred-plus MB right there — not because anyone wrote bad code, but because the layout itself is that size.

The engine stores rows columnar instead — separate tight arrays, one per attribute, plus an interned name pool:

std::vector<uint64_t> frns;      // full 64-bit FRNs (sequence<<48 | index)
std::vector<uint64_t> sizes;
std::vector<uint32_t> mtimes;     // Unix seconds — 1970..2106 fits in 4 bytes
std::vector<uint32_t> nameOffs;   // offset into the blob
std::vector<uint32_t> nameLens;
std::vector<uint8_t>  isDirs;
std::vector<char>     nameBlob;   // UTF-8 pool, no NUL terminators, deduplicated

Every frontend developer knows this instinctively: it's the difference between [{name, size, mtime}, ...] and one TypedArray per column. SoA beats AoS when you scan.

The merciless details that stack up:

1. Interned names. "New folder", "package.json", "index.js" appear tens of thousands of times each. One copy in the blob; every row is just an 8-byte offset+length slice.
2. uint32 Unix-seconds mtimes. Halves the time column; 1970–2106 is plenty.
3. Lowercase names that aren't there. Case-insensitive search needs a lowercased name — but most filenames are already lowercase. Each row carries a flag bit: if the folded name is byte-identical to the original, the lowercase view just reads the original blob. Saves nearly an entire second name pool.
4. Derived columns aren't persisted. The parent FRN you need for path reconstruction is derivable from the parent-row column at query time, so the snapshot simply doesn't contain it.

Same data, different layout: hundreds of MB becomes a few MB. Structure, not optimization.

Staying fresh: USN journal catch-up — and a subtle trap

Full builds are one thing; filesystems keep changing. NTFS ships a change journal (USN records, monotonically increasing). The engine stores an anchor (JournalId, NextUsn) and periodically replays new records.

The trap I hit so you don't have to: the anchor must be taken before enumeration starts, not after it finishes.

My first version grabbed the current USN after the full scan — logically airtight, right? Wrong. The scan takes seconds. Save a file during the scan and it may not be on disk when your read passes its directory, yet its USN is already below your anchor. The journal replay skips it forever. The index is silently missing a file, no error, no recovery.

Store the pre-enumeration anchor. Replay may redundantly re-apply a change that the scan already caught — idempotent, harmless. Missing is not.

Also: journals get trimmed and recreated. The anchor's JournalId detects that; on mismatch, abandon incremental catch-up and do a full rebuild. Correctness always outranks cleverness.

Persistence: mmap snapshots

Rebuilding on every boot is wasteful, so the index snapshots to disk per volume and loads via mmap — O(1) open, column pointers point straight at mapped pages, zero copies. The query-accelerating structure (an ASCII bigram inverted index, VByte-delta compressed) is materialized into the snapshot, so loading requires zero rebuilding.

Two Windows-specific footnotes for anyone doing this:

• After mmap-loading, call PrefetchVirtualMemory. Otherwise the first full scan pulls tens of MB in random 4 KB page faults (measured: ~6 s on my C: volume) while the OS pages things in lazily.
• A file you have mmap'd cannot be replaced. Atomic snapshot replacement (MoveFileEx over the old path) fails with gle=5 while your own process still holds a mapping of the old file — even though you're "just" swapping the path. Release your own mapping first, and guard the column-pointer swap with a reader-writer lock so in-flight queries don't race the swap.

The scorecard

	Bundled Everything	Homegrown (1.0.5)
Index memory	Hundreds of MB (external service)	A few MB
External dependency	Third-party install/bundle	None
Query syntax	Everything's	Compatible (ext:pdf, size:>100mb, dm:today)
Control	Black box	End-to-end

Was it worth it? For this product, yes — search is the palette's core experience, and resident memory is the metric that decides whether a tray app survives on someone's machine. The rule of thumb I'd offer from this round:

Don't build a wheel where the existing one is already optimized for you. Build it when your core requirement is exactly the dimension the dependency doesn't optimize for.

Everything is perfectly optimized for being Everything. It was never going to be optimized for being inside something else.

This shipped in yyzTools 1.0.5 — free, no accounts, no telemetry, Windows 10/11, at yyztools.com. The same release also added a quick OCR-translate pipeline (screenshot → recognize → translate, all offline via the local RapidOCR engine). Happy to dig deeper into any part of this in the comments — the parallel-scan worker pool has a genuinely fun stack-lifetime crash story if anyone wants it.