Why Is Only 40% of Your Data Compressed? The Tarball Trap and How to Escape It

The Uncompressed Elephant in the Room

Open a shell on your primary storage array. Run du -sh on the directories nobody has touched since 2022. Now look at your cloud bill. There's a good chance you're paying full price to store data that could be sitting in less than half the space.

Industry analysts have been saying the same thing for years: somewhere around 80% of enterprise data is unstructured — logs, CSVs, old project exports, documents, media, backups of backups. And unstructured data is exactly the kind of data that modern compression algorithms are good at. Zstd routinely gets 2:1 to 3:1 on typical unstructured payloads, which means 50-66% of that storage footprint doesn't need to exist.

So here's the question worth sitting with: if the algorithms are this good, and the financial case is this obvious, why does so much cold data sit completely uncompressed on expensive primary storage?

The answer isn't ignorance. It's fear — a very rational fear, rooted in how compression tools actually behave once you use them.

Why We Don't Compress: The Tarball Trap

Everyone has run tar czf archive.tar.gz project/ at some point, watched a directory of ten thousand files collapse into a single blob, and felt the small dread of knowing that directory is now effectively gone.

This is the core usability problem with traditional compression:

• Loss of visibility. Once files are inside a tarball or zip, they vanish from the filesystem's view. No more find, no more grep -r, no more casually browsing a folder in your file manager. The files still exist, technically, but they're locked behind an opaque container.
• The "I might need that" price. Someone eventually needs one 5MB config file out of that archive. To get it, they have to pull down and decompress the entire 100GB tarball — CPU time, disk I/O, and a coffee break, just to retrieve a file a fraction of a percent the size of the container.

Faced with that trade-off, most teams make the rational choice: leave it uncompressed. Storage is "cheap enough," and a working filesystem beats a compressed black box you're afraid to touch. We don't skip compression because we don't understand its value — we skip it because traditional tooling makes accessibility and compression mutually exclusive.

The Financial Cost of That Fear

Choosing not to compress isn't a one-time decision that costs you some disk space. It's a decision that gets multiplied across your entire infrastructure stack, every day, forever.

• Backups. Every backup job now moves and stores the full uncompressed volume — 100TB instead of 40TB, on every cycle.
• Replication. Cloud egress fees and inter-site bandwidth are spent transmitting bytes that were mostly redundant text or empty padding to begin with.
• Power and cooling. Uncompressed data means more spinning drives, drawing wattage around the clock, to hold data nobody has read in three years.

None of these costs show up as a single line item labeled "we didn't compress our archive." They show up quietly, spread across every budget line that touches storage — which is exactly why they're so easy to ignore and so expensive to sustain.

The Solution: Compression That Doesn't Change the User Experience

The real fix isn't a better zip format. It's removing the trade-off entirely: what if data could be compressed at rest, but still look and behave like a normal, uncompressed file to everything above the storage layer?

That's the specific problem HuskHoard is built around. It's an open-source (AGPL v3), Rust-based tiering engine for Linux that automatically moves cold data off expensive NVMe/SSD storage onto disk, cloud buckets, or physical LTO tape — while leaving a normal-looking file behind in your directory tree.

The mechanism that makes this work is worth calling out, because it's not FUSE. FUSE-based transparent filesystems exist, but they route every single read through a userspace filesystem driver, which adds latency and a layer you now depend on for correctness. HuskHoard instead uses the Linux fanotify kernel API. When a process opens a file that's been archived, the kernel itself pauses that process, HuskHoard's daemon recalls the data, and the process resumes — with zero awareness that anything happened. ls, find, and your file manager all keep working normally, because as far as the filesystem is concerned, the file is still there.

The Technical Layer: How the Catalog and Frames Actually Work

This is the part worth understanding if you're evaluating whether this approach holds up under real workloads.

The Catalog fixes visibility. Instead of hiding archived files inside a monolithic container, HuskHoard logs every file's metadata — path, version history, byte offset on physical media, BLAKE3 hash — into a SQLite database it calls the Catalog. The filesystem keeps a lightweight stub in place of the original file, so directory listings, find, and search all keep working exactly as before. You never lose sight of what you have or where it lives; the Catalog is queryable in milliseconds even across a multi-petabyte archive, without waking up a single tape drive.

Independent Zstd frames fix the "give me 5MB out of 100GB" problem. A standard Zstd or gzip stream builds a single continuous compression window across the whole file — to read byte 90,000,000 you generally have to decompress everything before it. HuskHoard avoids this by packing archived data into independent 16MB Zstd frames when writing to cloud storage via rclone. Each frame is a self-contained compression boundary, so decompressing frame 400 doesn't require touching frames 1 through 399. This is also what keeps cloud costs down on the write side — batching into 16MB units minimizes the number of PUT requests, which matters when providers bill per-request as well as per-byte.

TLV headers carry the metadata that makes recovery possible without a working database. Every file HuskHoard archives is preceded by a 4,096-byte object header. The first 136 bytes hold the essentials — UUID, POSIX mode, compressed size, BLAKE3 hash. The remaining ~3,960 bytes are packed using Type-Length-Value (TLV) encoding, which stores POSIX extended attributes (xattrs) directly alongside the payload. That detail matters more than it sounds: if a parser encounters a TLV type it doesn't recognize, it just reads the length and skips it, which is what makes the format tolerant of future changes. It also means the archive is self-describing — if the Catalog database is ever lost, HuskHoard can rebuild it from scratch by scanning the tape or bucket and reading every object header back out.

Frame indexing is what makes partial reads actually fast. Combine the Catalog's byte-offset tracking with independent frame boundaries, and you get real range access instead of full-file extraction. HuskHoard's StreamGate HTTP gateway uses this to let you seek into massive files without downloading them wholesale — want to scrub through a 4K video sitting on S3 or a physical LTO tape, in Plex or mpv? StreamGate resolves the byte range you're asking for, fetches only the relevant frame (an HTTP Range request for cloud, a SCSI tape seek for LTO), and decompresses just that piece. Tape gets its own physics-aware handling on top of this — writes land in 256KB SCSI-aligned blocks with filemarks, specifically to avoid "shoe-shining" the drive, and the Catalog indexes those blocks the same way it indexes cloud frames.

You can see the shape of this in practice — after installing and pointing the daemon at a test volume, dropping a file into the hot tier and waiting a few seconds is enough to watch it happen:

# Drop a file into the watched hot tier
dd if=/dev/urandom of=hot_tier/dummy_data.bin bs=1M count=12

# after the janitor interval elapses...
ls -ls hot_tier        # allocated size drops to ~0, logical size unchanged
du -h hot_tier         # it's become a "husk"

# open the file normally — fanotify intercepts the read and recalls it

And streaming straight off tape or cloud without ever touching local disk:

./target/release/huskhoard cat --file-path /media/movies/scifi.mp4 | mpv -

Nothing about that command looks like archive recovery. That's the point.

Have Your Cake and Compress It Too

The underlying idea here isn't really about Zstd, or SQLite, or fanotify specifically — it's that compression shouldn't be a choice between "save money" and "keep my data usable." Those two goals only feel opposed because most tooling makes them opposed. Once compression happens at the infrastructure layer instead of as a manual, user-facing chore, the trade-off disappears.

If you want to see how the frame layout and TLV headers work in more detail, the HuskHoard GitHub repo has the full source (it's Rust, MIT-adjacent AGPL v3, and genuinely readable), and the project blog has deeper dives into specific pieces like the Catalog and the tape format.

In the meantime, it's worth an honest audit: how much of your own cold, unstructured data is sitting around uncompressed right now — not because compression wouldn't help, but because the tools made it too painful to bother?