Thou Shall Not 'ls -la'

#devops #software #datascience #opensource

submit to towards data science ## 1. The Incident

It's 2:47 PM on a Tuesday. Somewhere on your team, a junior engineer wants to know how much disk space last quarter's render output is taking up. They SSH into the NAS, cd into the shared mount, and type the four most dangerous characters in a storage admin's vocabulary:

find / -mtime +90

or maybe

ls -laR

Thirty seconds later, Slack lights up. Someone's editing session on the array just froze. The metadata server's CPU is pinned. IOPS that were supposed to be serving production reads are instead being burned walking a directory tree with 50 million entries. For the next twenty minutes, your team experiences what can only be described as a storage brownout — not because anyone did anything malicious, but because someone asked a question the file system was never built to answer quickly.

If "thou shalt not ls -laR" is an actual rule at your organization — written down, whispered as folklore, or learned the hard way — that's not a people problem. It's a sign your architecture has outgrown your file system. The fix isn't "train people not to run find." The fix is to stop using a file system as a database, because it was never designed to be one.

2. The Problem with POSIX and File Systems

File systems like ext4, XFS, and NTFS trace their design lineage back to the 1970s. Their job, fundamentally, is to answer one question well: given a path, where are the bytes? They do this through inodes and directory trees — a hierarchical structure where every "ls" of a directory means opening it, reading its entries, and for anything richer than a name (size, mtime, permissions), stat-ing each one individually.

That design is excellent for what it was built for: reading and writing files. It is a terrible design for the question storage teams actually ask as they scale: "which files haven't been touched in 90 days?" or "how much are we storing in `/projects//rushes`?"*

There's no index for that. There's no WHERE clause. To answer it, the OS has no choice but to traverse the entire tree, open every directory, and stat every inode along the way — an O(n) operation where n is your total file count. At a few thousand files, you don't notice. At 50 million files spread across a shared NFS mount, that traversal doesn't just take a while — it competes for the exact same metadata I/O that production workloads depend on, and it can bring a storage array to its knees.

The real issue isn't that recursive scans are slow. It's that as storage scales into hundreds of terabytes or petabytes, the actual payload data stops being the bottleneck. The metadata — the information about the data — becomes the thing you're constantly fighting.

3. Enter the Data Catalog

A data catalog solves this by doing something conceptually simple: it decouples the knowledge of your data from the physical storage of your data. Instead of a directory tree you have to walk, you get a structured, indexed database — usually SQL or a fast key-value store — that already knows the answer before you ask.

The mechanics look like this:

Rather than reading the disk to discover what files exist, applications and users query the catalog directly.
The catalog indexes the metadata you'd expect from POSIX — size, timestamps, permissions — but also stores extended context a file system has no concept of: checksums, version history, logical groupings, custom tags.
The catalog stays in sync with reality either through filesystem event listeners (think inotify / fanotify) or by updating itself at the moment data is ingested — so it reflects the true state of your storage without ever needing to re-scan it.

The net effect: a question that used to mean "walk a 50-million-node tree" becomes "run an indexed query." Same question, radically different cost.

4. Why Catalogs Are a Massive Improvement

1. Query speed — milliseconds instead of hours. Finding every 50GB video file older than a year stops being a recursive filesystem crawl and becomes a SELECT. It runs in milliseconds and generates zero I/O against your actual storage array.

2. Abstraction of physical location. A file system assumes your file's bytes live at a specific block on a specific disk, full stop. A catalog doesn't have to make that assumption. It can record that a file logically exists while its physical bytes are sitting in an S3 bucket, on a cold SMR drive, or offline on an LTO tape sitting on a shelf — and the catalog is the only thing that needs to know which.

3. Rich, enforceable policy. You can't attach a business rule to an inode. You can attach compliance_retention = 7_years or tiering_policy = aggressive to a row in a catalog, and then let a policy engine act on that column instead of parsing filenames or crawling directories to guess at intent.

Here's the contrast that makes it click. The bash version of "what's old":

find /mnt/storage -mtime +90 -type f

...which has to touch every inode under the search path. The catalog version of the same question:

SELECT file_path
FROM catalog
WHERE last_accessed < date('now', '-90 days');

...which touches an index, not your disks.

5. Bringing It Home: The HuskHoard Catalog & Tiering Engine

Full-blown enterprise data catalogs exist — Collibra, Alation, and friends — but they're built for data lakes and databases, priced accordingly, and overkill if what you actually want is to stop your NVMe array from filling up with cold files nobody's touched since 2023.

This is the gap HuskHoard is built for: an open-source, AGPL-licensed, Rust-based tiering engine for Linux that treats the catalog — not the file system — as the ground source of truth for where your data actually lives.

The catalog is a SQLite brain, not a directory walk. HuskHoard keeps a husk_catalog.db file that tracks, per archived object, the exact physical volume UUID and byte offset it lives at, its version history, its BLAKE3 checksum, and its original POSIX metadata. When the system needs to know where a file's bytes are, it queries this database — it never has to scan a tape, a disk, or a cloud bucket to find out.

Files stay visible without occupying space. HuskHoard doesn't use FUSE. It hooks into the Linux fanotify kernel API to watch for file access in real time. When cold data is archived, the file on your hot tier shrinks to occupy effectively zero disk blocks while its logical size — the size ls reports — stays exactly the same. The moment an application actually opens that file, the Interceptor blocks the read for a moment, pulls the real bytes back from wherever the catalog says they are, and resumes the process like nothing happened.

The Janitor queries SQL, not your NVMe drives. The policy engine that decides what's cold enough to archive runs against the catalog's indexes — by age, extension, or directory rule — instead of stat-ing your way through a live filesystem tree. When it finds a candidate, the Archive Worker compresses it into independent Zstd frames, writes it out to whichever tier you've configured (a flat disk image, a physical LTO-5 through LTO-9 tape drive, or a cloud bucket via rclone), and updates the catalog's offset columns. Nothing about that process ever requires a recursive scan of your storage.

Why "Zstd frame format" matters more than it sounds like it should. Compression algorithms normally chain their dictionary state across an entire file, which means seeking into the middle of a compressed archive forces you to decompress everything before it just to rebuild that state. For a 200GB dataset on tape, that's not a rounding error — it's the difference between an instant partial read and stalling a pipeline for an hour. HuskHoard breaks that chain deliberately: files are sliced into independent 16MB frames, with the dictionary reset at every boundary, and a small TLV (type-length-value) index embedded directly in the file's own header maps uncompressed offsets to their compressed position on the volume. A request for a specific byte range binary-searches that index, seeks straight to the right frame, and decompresses only the ~16MB it actually needs — whether the data is sitting on disk, in the cloud, or on a physical tape cartridge. Pre-compressed formats like MP4 skip the compression path entirely and get true O(1) seeks, because logical and physical offsets line up 1-to-1.

The result is a system where the catalog answers "where is it" instantly, and the frame format answers "how do I get just the part I need" without ever reading the whole object.

6. Conclusion

File systems are excellent at one job: holding bytes reliably on a disk. They are not, and were never meant to be, databases — and the moment you start asking them database questions at scale, you find that out the hard way, usually during business hours, usually while someone's editing session freezes.

If you want to manage data efficiently, cut your electricity bill, and automate tiering without putting your array through a brownout every time someone wants a list of old files, the fix is architectural: decouple the knowledge of your data from the storage of your data.

So the next time someone reaches for find / -mtime +90 across a shared mount with tens of millions of files, stop them — and point them at a catalog instead.