DEV Community: Joichiro Mitaka

Why Is Only 40% of Your Data Compressed? The Tarball Trap and How to Escape It

Joichiro Mitaka — Mon, 06 Jul 2026 07:06:25 +0000

The Uncompressed Elephant in the Room

Open a shell on your primary storage array. Run du -sh on the directories nobody has touched since 2022. Now look at your cloud bill. There's a good chance you're paying full price to store data that could be sitting in less than half the space.

Industry analysts have been saying the same thing for years: somewhere around 80% of enterprise data is unstructured — logs, CSVs, old project exports, documents, media, backups of backups. And unstructured data is exactly the kind of data that modern compression algorithms are good at. Zstd routinely gets 2:1 to 3:1 on typical unstructured payloads, which means 50-66% of that storage footprint doesn't need to exist.

So here's the question worth sitting with: if the algorithms are this good, and the financial case is this obvious, why does so much cold data sit completely uncompressed on expensive primary storage?

The answer isn't ignorance. It's fear — a very rational fear, rooted in how compression tools actually behave once you use them.

Why We Don't Compress: The Tarball Trap

Everyone has run tar czf archive.tar.gz project/ at some point, watched a directory of ten thousand files collapse into a single blob, and felt the small dread of knowing that directory is now effectively gone.

This is the core usability problem with traditional compression:

Loss of visibility. Once files are inside a tarball or zip, they vanish from the filesystem's view. No more find, no more grep -r, no more casually browsing a folder in your file manager. The files still exist, technically, but they're locked behind an opaque container.
The "I might need that" price. Someone eventually needs one 5MB config file out of that archive. To get it, they have to pull down and decompress the entire 100GB tarball — CPU time, disk I/O, and a coffee break, just to retrieve a file a fraction of a percent the size of the container.

Faced with that trade-off, most teams make the rational choice: leave it uncompressed. Storage is "cheap enough," and a working filesystem beats a compressed black box you're afraid to touch. We don't skip compression because we don't understand its value — we skip it because traditional tooling makes accessibility and compression mutually exclusive.

The Financial Cost of That Fear

Choosing not to compress isn't a one-time decision that costs you some disk space. It's a decision that gets multiplied across your entire infrastructure stack, every day, forever.

Backups. Every backup job now moves and stores the full uncompressed volume — 100TB instead of 40TB, on every cycle.
Replication. Cloud egress fees and inter-site bandwidth are spent transmitting bytes that were mostly redundant text or empty padding to begin with.
Power and cooling. Uncompressed data means more spinning drives, drawing wattage around the clock, to hold data nobody has read in three years.

None of these costs show up as a single line item labeled "we didn't compress our archive." They show up quietly, spread across every budget line that touches storage — which is exactly why they're so easy to ignore and so expensive to sustain.

The Solution: Compression That Doesn't Change the User Experience

The real fix isn't a better zip format. It's removing the trade-off entirely: what if data could be compressed at rest, but still look and behave like a normal, uncompressed file to everything above the storage layer?

That's the specific problem HuskHoard is built around. It's an open-source (AGPL v3), Rust-based tiering engine for Linux that automatically moves cold data off expensive NVMe/SSD storage onto disk, cloud buckets, or physical LTO tape — while leaving a normal-looking file behind in your directory tree.

The mechanism that makes this work is worth calling out, because it's not FUSE. FUSE-based transparent filesystems exist, but they route every single read through a userspace filesystem driver, which adds latency and a layer you now depend on for correctness. HuskHoard instead uses the Linux fanotify kernel API. When a process opens a file that's been archived, the kernel itself pauses that process, HuskHoard's daemon recalls the data, and the process resumes — with zero awareness that anything happened. ls, find, and your file manager all keep working normally, because as far as the filesystem is concerned, the file is still there.

The Technical Layer: How the Catalog and Frames Actually Work

This is the part worth understanding if you're evaluating whether this approach holds up under real workloads.

The Catalog fixes visibility. Instead of hiding archived files inside a monolithic container, HuskHoard logs every file's metadata — path, version history, byte offset on physical media, BLAKE3 hash — into a SQLite database it calls the Catalog. The filesystem keeps a lightweight stub in place of the original file, so directory listings, find, and search all keep working exactly as before. You never lose sight of what you have or where it lives; the Catalog is queryable in milliseconds even across a multi-petabyte archive, without waking up a single tape drive.

Independent Zstd frames fix the "give me 5MB out of 100GB" problem. A standard Zstd or gzip stream builds a single continuous compression window across the whole file — to read byte 90,000,000 you generally have to decompress everything before it. HuskHoard avoids this by packing archived data into independent 16MB Zstd frames when writing to cloud storage via rclone. Each frame is a self-contained compression boundary, so decompressing frame 400 doesn't require touching frames 1 through 399. This is also what keeps cloud costs down on the write side — batching into 16MB units minimizes the number of PUT requests, which matters when providers bill per-request as well as per-byte.

TLV headers carry the metadata that makes recovery possible without a working database. Every file HuskHoard archives is preceded by a 4,096-byte object header. The first 136 bytes hold the essentials — UUID, POSIX mode, compressed size, BLAKE3 hash. The remaining ~3,960 bytes are packed using Type-Length-Value (TLV) encoding, which stores POSIX extended attributes (xattrs) directly alongside the payload. That detail matters more than it sounds: if a parser encounters a TLV type it doesn't recognize, it just reads the length and skips it, which is what makes the format tolerant of future changes. It also means the archive is self-describing — if the Catalog database is ever lost, HuskHoard can rebuild it from scratch by scanning the tape or bucket and reading every object header back out.

Frame indexing is what makes partial reads actually fast. Combine the Catalog's byte-offset tracking with independent frame boundaries, and you get real range access instead of full-file extraction. HuskHoard's StreamGate HTTP gateway uses this to let you seek into massive files without downloading them wholesale — want to scrub through a 4K video sitting on S3 or a physical LTO tape, in Plex or mpv? StreamGate resolves the byte range you're asking for, fetches only the relevant frame (an HTTP Range request for cloud, a SCSI tape seek for LTO), and decompresses just that piece. Tape gets its own physics-aware handling on top of this — writes land in 256KB SCSI-aligned blocks with filemarks, specifically to avoid "shoe-shining" the drive, and the Catalog indexes those blocks the same way it indexes cloud frames.

You can see the shape of this in practice — after installing and pointing the daemon at a test volume, dropping a file into the hot tier and waiting a few seconds is enough to watch it happen:

# Drop a file into the watched hot tier
dd if=/dev/urandom of=hot_tier/dummy_data.bin bs=1M count=12

# after the janitor interval elapses...
ls -ls hot_tier        # allocated size drops to ~0, logical size unchanged
du -h hot_tier         # it's become a "husk"

# open the file normally — fanotify intercepts the read and recalls it

And streaming straight off tape or cloud without ever touching local disk:

./target/release/huskhoard cat --file-path /media/movies/scifi.mp4 | mpv -

Nothing about that command looks like archive recovery. That's the point.

Have Your Cake and Compress It Too

The underlying idea here isn't really about Zstd, or SQLite, or fanotify specifically — it's that compression shouldn't be a choice between "save money" and "keep my data usable." Those two goals only feel opposed because most tooling makes them opposed. Once compression happens at the infrastructure layer instead of as a manual, user-facing chore, the trade-off disappears.

If you want to see how the frame layout and TLV headers work in more detail, the HuskHoard GitHub repo has the full source (it's Rust, MIT-adjacent AGPL v3, and genuinely readable), and the project blog has deeper dives into specific pieces like the Catalog and the tape format.

In the meantime, it's worth an honest audit: how much of your own cold, unstructured data is sitting around uncompressed right now — not because compression wouldn't help, but because the tools made it too painful to bother?

50% of Cloud Hard Drives are Now SMR: Why Hyperscalers Love What Consumers Hate

Joichiro Mitaka — Tue, 30 Jun 2026 05:39:23 +0000

If you hang around r/DataHoarder or home lab communities, you’ve likely heard the golden rule of buying hard drives: "Avoid SMR at all costs."

The general consensus is that Shingled Magnetic Recording (SMR) drives have abysmal write speeds, fail early, and will crash your RAID arrays. Yet, if you look at the enterprise storage market today, a massive paradox emerges.

(Caption: Data from Western Digital’s Quarterly Earnings showing UltraSMR crossing 50% of nearline exabyte shipments in 2023)

The numbers don't lie. During recent earnings calls, Western Digital confirmed a massive milestone: their UltraSMR technology now makes up over 50% of their total nearline exabyte shipments. Cloud giants like Dropbox rely on SMR for over 90% of their entire storage fleet. In fact, production for high-capacity SMR drives is essentially sold out as AI and data centers buy them by the truckload.

So, SMR offers higher capacity and a lower price per terabyte. Why does everyone in the consumer and small enterprise space avoid them, and more importantly, how can you start using them like the cloud giants do?

Let's demystify SMR technology, look at how the hyperscalers bypassed its limitations, and explore how you can leverage it in your own home lab or small enterprise using modern open-source tools.

The Root of the Stigma: Drive-Managed SMR and the Physics of the Write Head

To understand the hate, you have to understand the physics of the technology.

Hard drive manufacturers eventually hit a physical wall: the magnetic write head of a hard drive is physically wider than the read head. You need a certain amount of magnetic energy to flip bits on the platter, meaning the write head can only be shrunk so much before it fails to work. However, the read head can be made incredibly tiny.

Conventional Magnetic Recording (CMR) accommodates this bulky write head by writing wide data tracks and leaving tiny gaps between them so they don't overlap.

SMR, on the other hand, exploits the tiny read head. It writes a wide track, and then writes the next track partially overlapping the first—like shingles on a roof. This leaves just enough of the previous track exposed for the smaller read head to read, allowing manufacturers to squeeze up to 25% more data onto the same physical platters.

The catch? You cannot randomly overwrite data. Because the tracks overlap, modifying a single block of data requires the drive to read a massive "zone" of data into a cache, modify it, and rewrite the entire zone sequentially.

The stigma started around 2020 when manufacturers silently slipped Drive-Managed SMR into consumer NAS drives. These drives pretended to be normal CMR drives to the host operating system, attempting to manage all that complex "read-modify-write" garbage collection internally. Under heavy, random-write NAS workloads, their caches would fill up, performance would tank to dial-up speeds, and RAID controllers would assume the drives were dead and drop them from arrays.

Consumers rightly felt burned. But the cloud providers took a completely different approach.

The Cloud's Secret Weapon: Host-Managed SMR, Aggressive Tiering, and Pin 3

Hyperscalers like Dropbox and Google didn't try to force SMR drives to act like traditional CMR drives. Instead, they completely rebuilt their storage infrastructure around three core concepts.

1. Host-Managed Zoned Storage

In a Host-Managed setup, the hard drive doesn't try to hide its shingled nature. It exposes its architecture to the operating system as a Zoned Block Device (ZBD). The drive essentially says to the host: "Here are my zones. You can only write to them sequentially. If you want to delete or overwrite something, you must reset the entire zone and start from the beginning." By shifting the responsibility from the drive's underpowered internal processor to the host machine's software, the performance penalty vanishes.

2. Aggressive Telemetry and Tiering

Rewriting the file system is only half the magic. Cloud services have immense intelligence about your files and access patterns. When you upload a file, it rarely lands directly on a spinning disk; it hits a blazing-fast NVMe cache (the "hot" tier).

However, the moment their algorithms determine that a file is no longer actively being used—say, a cloud backup from three months ago, or an old photo album—they aggressively tier that data. It is migrated off the expensive flash and funneled sequentially, block by tightly-packed block, into the zones of a high-capacity SMR drive (the "cold" tier).

3. Pin 3 Power Management (Letting the Disks Go Cold)

Once that data is packed into an SMR drive, cloud providers take it a step further: they literally pull the plug.

If you've ever "shucked" an external hard drive for your home lab, you might have run into the infamous "Pin 3 Power Disable" (PWDIS) issue, where you had to put Kapton tape over the 3rd SATA pin just to get the drive to spin up. While consumers were taping over this pin out of frustration, hyperscalers were demanding this feature by design.

The PWDIS feature allows data centers to remotely hard-reset or cut power to individual drives. When an SMR drive is full of cold data, it doesn't just sit there spinning, consuming electricity, and generating heat. It is commanded to spin down. It sits entirely dormant, acting as a high-density, rapidly accessible tape replacement. It only wakes up when an API call requests a specific dormant file—spinning up, serving the file with a few seconds of acceptable latency, and going back to sleep.

The Power of Zoned.io

If you want to dive deep into how the foundation of this works on a technical level, the ultimate resource is Zoned.io. Backed by major storage players, it documents the Linux kernel's implementation of Zoned Block Devices. Zoned Storage is becoming critical not just for SMR hard drives, but for next-generation NVMe SSDs (ZNS) that use the exact same sequential-write paradigms to reduce write amplification and extend flash memory life.

Bringing SMR to the Home Lab and Small Enterprise

You don't need to be AWS or Dropbox to take advantage of SMR pricing and density. You just need to pay attention to your workload requirements and use software designed to treat storage sequentially.

SMR is an absolute powerhouse for WORM (Write Once, Read Many) workloads:

Media archiving and Plex servers
Immutable backup targets
Log aggregation and long-term surveillance video storage
Data hoarding and cold storage

If you are using ZFS, you can utilize sequential-friendly configurations, but to truly unlock the potential of Host-Managed SMR in a small enterprise or home lab, you need purpose-built archiving tools.

The Do's and Don'ts of SMR Archiving

SMR is an absolute powerhouse for WORM (Write Once, Read Many) workloads, but only if you follow the rules.

The Don'ts: RAID, ZFS, and Standard File Systems

The biggest mistake you can make is blindly handing an SMR drive over to your operating system or RAID controller.

Don't use hardware RAID or standard ZFS. RAID arrays (and ZFS RAIDZ) rely on parity calculations and constant metadata updates. If an array needs to rebuild or "resilver", it blasts the drives with random, fragmented writes. The SMR cache will choke, write speeds will drop to practically zero, and the controller will drop the drive entirely.
Don't just format it to ext4 or NTFS. Traditional file systems are designed for random I/O. Even if you are just sequentially copying a large video file, the file system is constantly darting its write head back and forth across the platter to update the journal, timestamps, and file allocation tables. This micro-random I/O ruins SMR performance.

The Do's: Buffer, Batch, and Stream

Instead of fighting the drive with random I/O, treat it exactly like a modern tape drive.

Do use an SSD staging tier. Borrow the cloud provider's playbook. Buffer your incoming files on an NVMe or standard SSD first. Once you have accumulated a large batch of cold data, flush it to the SMR drive in one massive, continuous stream.
Do use Zoned-aware storage layers. The Linux kernel has made massive strides here. For example, btrfs now has a dedicated "Zoned Mode" explicitly designed to work with Zoned Block Devices. It forces all writes to be strictly sequential, keeping metadata organized and eliminating the read-modify-write penalty.

If you want to go deeper into SMR this is a good resource(Stop Murdering your SMR Drives), but if you want to take advantage of SMR pricing and density today, you must completely rethink your storage architecture.

Conclusion

The storage industry is moving toward Zoned Storage, whether consumers like it or not. With the massive shift toward UltraSMR in the enterprise space, the era of ignoring the technology is over.

It's time we stop universally hating SMR drives and start treating them like the specialized, high-capacity archiving tools they are. If you stop handing them directly to standard file systems and instead provide them with the buffered, sequential workloads they crave, they will reward you with massive capacity, incredible cost savings, and a much lower power bill.

Check out the documentation at Zoned.io to understand the kernel-level magic, and if you are looking for an out-of-the-box archiving solution that loves SMR drives, check out AGPL archiving at Huskhord.com.

Addressing the global namespace

Joichiro Mitaka — Thu, 25 Jun 2026 06:20:12 +0000

There is a moment in every data hoarder's life — and in every small media shop's IT history — when the first archive drive fills up. You slot in a second tape, plug in another external drive, configure a new cloud bucket. Problem solved.

Except now you have a different problem: you have no idea where anything is.

Not in the "I lost the file" sense. More in the "I know I have that file somewhere across these six volumes, but I don't know which one, and I'm not sure I haven't archived it twice, and that third copy might be the old version" sense. The data is there. The knowledge of where it is, and which copy is canonical, is not.

This is the multi-volume namespace problem. It's been lurking in storage management since reel-to-reel tape in the 1960s, and the solutions to it span from "I have a spreadsheet" to "I have a $200,000 enterprise storage cluster." Most people end up somewhere in the uncomfortable middle.

Let's look at the problem properly, put some numbers on it, and walk through what people actually do — and why each approach eventually runs out of road.

The Numbers First

"Multiple volumes" sounds modest. It isn't, once you start measuring.

LTO tape capacity by generation (native / 2.5:1 compressed):

Generation	Native	Compressed	$/TB (cartridge only, approx.)
LTO-7	6 TB	15 TB	~$0.02/GB
LTO-8	12 TB	30 TB	~$0.015/GB
LTO-9	18 TB	45 TB	~$0.012/GB
LTO-10	40 TB	100 TB	(2026, available now)

A serious homelab data hoarder with a modest LTO-8 drive will fill a 12 TB cartridge in 6–18 months depending on what they're collecting. A four-tape rotation is a realistic minimum for anyone who takes offline copies seriously. That's 48 TB native — multiple volumes by definition.

Scale up to a small production company, a university research group, or a regional broadcaster. Now you have dozens of tapes, offline drives at an off-site location, and a cloud backend for disaster recovery. Each one a separate island of storage.

Hard drive archives have the same problem. A rotating set of 14–20 TB USB drives for cold backup is cheap and practical. It's also, over five years, eight or ten drives with no shared index.

The problem isn't finding the storage. The problem is that the more storage you buy, the worse your ability to reason about it gets — unless you build something to manage the namespace across all of it.

What "Namespace" Means in This Context

A namespace is simply a unified view of your files. On a single volume, it's trivial: mount the drive, run ls, see everything. The file system is the namespace.

The moment you have two volumes, the file system isn't enough. A file might be on either one. If both are mounted simultaneously, you have two separate trees. If one is offline, its files are invisible. If the same logical file has been archived to both volumes at different points in time, you have two physical copies with no built-in way to determine which is current.

The ideal global namespace would let you:

List every file you've ever archived, regardless of which physical volume holds it
Know immediately which volume a file is on, and what its current version is
Know that you haven't accidentally archived the same content twice on different volumes
Retrieve the right version without checking six different drives one by one

No single standard delivers all four of these, which is why the solutions below exist — and why each one is only a partial answer.

Approach 1: LTFS — A File System on Every Tape

The closest thing to a proper multi-generation standard for tape namespace management is LTFS (Linear Tape File System), adopted by the LTO Consortium in 2010 and standardized by SNIA. If you're running LTO-5 or later, you can format any cartridge as LTFS and mount it exactly like a USB drive.

The way it works is elegant. LTFS divides a tape cartridge into two partitions. Partition 0 is the index — an XML document describing your directory tree, file names, timestamps, and the exact tape block where each file's data begins. Partition 1 is the data itself. When you insert an LTFS tape and run the mount command, the drive reads Partition 0 into memory, and your OS presents the tape as a browsable volume.

ltfs -o devname=0 /mnt/tape
ls /mnt/tape/projects/alpha/
# rushes/  finals/  audio/

You can drag and drop. You can cp. Files are written sequentially to the end of the data partition and the index is updated. It's a genuine open standard — an LTFS tape written on Linux can be read on macOS or Windows with compatible software, with no proprietary software in the middle.

For interchange and manual archiving, LTFS is excellent. If you're sending a tape to a collaborator or to a facility you've never worked with before, LTFS is the right choice. It is genuinely self-describing: the metadata travels with the media.

Where LTFS Stops

The problem is that LTFS solves the per-tape namespace and stops there. Each tape is its own island.

When a tape is offline — sitting on a shelf — its files are invisible. There is no LTFS spanning standard that lets you query across multiple cartridges simultaneously. If you have four tapes and need to find a file from 2022, you are mounting tapes until you find it, or maintaining a separate external record of what's where.

LTFS also inherits tape's fundamental constraint: files can only be appended, never overwritten. If you delete a file in the LTFS index, the data blocks on tape are simply marked unavailable — the space is not recovered until you reformat the entire cartridge. More importantly for long-lived archives: LTFS has no native versioning. The spec doesn't define what a version is or how to track one. If you archive final.mov, re-edit it, and archive it again to the same tape, the second write is a separate file entry in the same index. Version control becomes your problem.

The index partition itself has a ceiling — roughly 5–10% of tape capacity — which can become a bottleneck for archives with millions of small files.

For many users, these aren't dealbreakers for a single-tape workflow. But the moment you're managing a tape library, you need something sitting above LTFS to provide the cross-volume view. LTFS gives you a great building block. It doesn't give you a global namespace.

Approach 2: The Spreadsheet (And Its Cousins)

The most common approach in homelabs and small shops isn't LTFS or enterprise software. It's a spreadsheet. Or a text file. Or a Notion database. Or a Python script someone wrote in a weekend that generates a CSV of filenames and tape labels.

This approach is completely understandable and immediately comprehensible to everyone on the team. "Tape 003 has the 2023 raw footage. Tape 007 has the 2024 project deliverables. Check the sheet."

It also has a half-life of about six months.

The spreadsheet tracks what you intended to put on each tape at the moment you archived it. It doesn't automatically update when files change. It doesn't track versions — when you re-archive an updated file, do you add a new row? Update the existing one? Add a note? Different people make different decisions, and over time the spreadsheet becomes an archaeological record of past intentions rather than a live index.

Deduplication is purely manual. Nothing in the spreadsheet warns you that client_deliverable_FINAL_v3.mov on Tape 003 and client_deliverable_FINAL_v3_2.mov on Tape 009 are the same underlying content, archived twice by different team members six weeks apart.

Search is grep over a CSV or a Notion filter — workable for hundreds of files, painful for tens of thousands, and broken for millions.

The spreadsheet approach is a record of what someone thinks is on the tapes. The tapes themselves know what's actually on them. These two sources of truth diverge over time, and the divergence accelerates as the archive grows.

Some teams move from spreadsheets to proper databases — SQLite or Postgres — and write custom tooling to maintain the index. This is a genuine improvement. The index is now queryable, consistent, and maintainable. But custom tooling requires ownership: someone has to build it, update it when formats change, and ensure it stays in sync with the physical media. Most teams don't have that person, or lose them.

Approach 3: Enterprise HSM

The "correct" industrial answer to this problem is a Hierarchical Storage Management (HSM) system. HSM has existed since IBM implemented it for mainframes in 1978. The idea is straightforward: files on fast, expensive storage automatically migrate to slower, cheaper storage based on access patterns, and transparent stub files replace them so users never notice the transition.

Modern commercial HSM products — IBM Spectrum Protect, HPE Data Management Framework, Quantum StorNext — provide exactly the global namespace you need. You have a single directory tree. Files that haven't been touched in 90 days silently move to tape. Open one, and the system recalls it transparently. The catalog tracks every file, every version, every volume.

They work. They work very well. They're also built for organizations with dedicated storage administrators, SAN infrastructure, enterprise support contracts, and storage budgets in the low-to-mid six figures.

Enterprise HSM is designed for the assumption that you have a team, a budget, and a vendor relationship. The software itself can cost more than a homelab enthusiast's entire NAS. The hardware requirements (dedicated tape libraries, Fibre Channel HBAs, redundant metadata servers) further narrow the addressable audience.

For a small production company with four LTO drives and a NAS, enterprise HSM is a missile to kill a mosquito. And there is no meaningful open-source equivalent in this space — the gap between "hobbyist script" and "enterprise HSM" has historically been a cliff with nothing in the middle.

Approach 4: Backup Software (The Wrong Tool)

A common misconception: "I'll just use my backup software to manage my archive." Backup software — Veeam, Bacula, Duplicati, Restic, Amanda — is excellent at what it does, which is taking point-in-time snapshots and storing them in a recoverable format.

It is not designed for an archive, and the distinction matters.

A backup is a recovery mechanism. You restore from it when something goes wrong. The interface is: "restore system/file to state as of [date]." Backup software maintains its own catalog of backup sets, not a live representation of your file namespace.

An archive is a primary store. Files live there permanently, and you access them directly. The interface should be: "open this file" — and it should just work, regardless of which volume holds the bytes.

Backup software can hold your archival data. It will not give you a unified browsable namespace across multiple tapes. It will not transparently serve a file from tape when an application requests it. It will not tell you whether a file is already archived before you try to archive it again. These are the functions a real archive system needs to provide, and backup software isn't designed to provide them.

What the Ideal Solution Would Look Like

At this point the shape of the problem is clear, and you can start to describe what a complete solution would need to provide.

Volume identity that outlives device paths. If your LTO drive moves from /dev/nst0 to /dev/nst1 because you rearranged your HBA, every restore should still work. Volume identity needs to be based on something burned into the media at format time — a UUID — not a device path or a human-assigned label.

A catalog that is separate from any single volume. You need a central index that can answer "where is this file?" without mounting every volume in your library. The catalog holds the address (volume UUID + byte offset) for every archived file.

But also: volumes that are self-describing. The catalog is the fast path. But if you lose the catalog database, you should be able to reconstruct it from the physical media — because each volume should carry enough metadata to identify every object it contains. The catalog is an optimization, not a dependency.

A namespace that is always visible, regardless of volume state. Files that are on offline tapes should still appear in your OS. You should be able to see the file, read its metadata, and — when you try to open it — get a useful message about which volume to insert, not a confusing "file not found" error.

Hash-based deduplication, not path-based. The system should know whether a file has already been archived by its content, not just its name. Renaming a file shouldn't trigger a redundant archive.

Versioning as a first-class concept. When a file changes and is re-archived, both the old and new versions should be tracked and retrievable. The system should know that version 3 is on Tape C and version 1 is on Tape A, and serve the right one by default while making rollback possible on request.

None of the approaches above deliver all six of these properties. LTFS gets you two or three. Spreadsheets get you one on a good day. Enterprise HSM gets you all six, but only if you have an enterprise budget.

An Open-Source Answer

This gap — between hobbyist scripts and enterprise HSM — is where a relatively new open-source project called HuskHoard is trying to operate.

HuskHoard is a data-tiering engine built in Rust for Linux. It uses the fanotify kernel API to transparently stub cold files — replacing them with zero-byte placeholders while keeping them visible in your file system — and automatically recalls the data from the appropriate volume when an application opens the file. It supports LTO tape drives natively (via the SCSI tape driver), flat image files, external drives, and cloud backends via rclone.

The catalog is a SQLite database that tracks every archived file by its content hash, its version history, and its exact byte offset on a specific volume UUID — not a device path. Plug a drive into a different port, and the catalog heals the mapping automatically via a rescan. Volumes carry their own self-describing headers, so the catalog can be fully rebuilt from raw media if it's ever lost.

For multi-volume management specifically: when a needed volume is offline, HuskHoard reports the volume's UUID and the human-readable location_hint you've assigned it ("Shelf A, Slot 2 — LTO-9 Vol 001"), then waits for you to insert it. It won't re-archive a file that's already in the catalog with a matching content hash, regardless of which volume holds the original. Each volume is formatted once with a UUID, and that identity persists for the life of the media.

It's not enterprise HSM. The robotic tape library automation, the Fibre Channel fabric, the dedicated metadata servers — those aren't there. But for a homelab operator, a small production company, or any team that has more data than fits on one thing and needs a coherent answer to "where is that file?", it addresses all six requirements above as open-source software, free under AGPL v3.

The architecture notes and a deeper dive into how the catalog handles the multi-volume problem are on the HuskHoard blog if you want to go further.

The multi-volume namespace problem isn't glamorous. It doesn't show up in benchmarks or feature comparison tables. It shows up the third time you're manually mounting tapes trying to find a project file from 18 months ago, or the moment you realize you've been archiving the same directory to two different drives for six months and have no idea which copies are current.

Every solution in this space involves a catalog — some record of what is where. The question is whether that catalog is a spreadsheet you maintain manually, a per-tape index that goes dark when the tape goes offline, an enterprise product with an enterprise price tag, or something designed to sit in the middle and make this problem manageable for the rest of us.

Using Zstd Frames to Egress Partial Parquet Files

Joichiro Mitaka — Wed, 24 Jun 2026 15:26:22 +0000

Jump Tables, TLV Footers, and the Real Cost of Reading What You Don't Need

You're paying for bytes you never read.

A data engineer on a busy pipeline touches dozens of Parquet files a day: schema discovery, predicate pushdown, column pruning, metadata scrapes for a data catalog sync. In each case, the application needs maybe 200 KB of context from a file that is 4 GB on disk. Without a seekable archive format and a jump table to find the right frame, your HTTP client fetches the whole thing, and your cloud egress invoice reflects every unnecessary gigabyte.

This post quantifies the problem, then walks through how HuskHoard uses seekable Zstd frames, a per-volume jump table, and TLV-encoded footer metadata to make partial egress a first-class citizen across multi-volume archives — disk, cloud, and LTO tape alike.

The Problem, In Dollars

S3 standard egress runs $0.09/GB. GCS is $0.08/GB. Even Cloudflare R2, which is free for egress from R2 to the internet, still costs you in latency and API call count when you cannot bound the range of bytes you need.

Here is a representative read pattern for a cold analytics archive:

Operation	Bytes Needed	Bytes Fetched (naïve)	Ratio
Schema discovery	~50 KB (Parquet footer)	1–8 GB (full file)	~1:16,000
Single column scan	~200 MB (one column chunk)	4 GB (full row group)	1:20
Data catalog sync (1M files)	~50 GB (footers only)	~4 PB (full files)	1:80,000
Selective restore (1 row group)	~128 MB	4 GB	1:32

On 100 TB of cold Parquet data with $0.09/GB egress:

Full read for schema sync: 100 TB × $0.09 = $9,216
Partial read (footers only, avg 100 KB/file, 1M files): ~100 GB × $0.09 = $9.00
Savings per catalog sync: $9,207 — 99.9% reduction

Even a conservative column-scan scenario (pulling 15% of each file's bytes) cuts a $9,216 monthly read bill to $1,382. The ceiling on savings is determined entirely by how precisely you can address the bytes you actually need.

That precision is what frames and jump tables buy you.

Zstd Frames: What They Are and Why They Matter

A single .zst file produced by the standard zstd CLI is one frame. Everything inside is a single compressed stream. You have to start decompression at byte 0 to reach any byte inside.

But the Zstd spec allows a concatenation of independent frames. Each frame is a complete, self-contained unit:

[Frame 0][Frame 1][Frame 2]...[Frame N]
 ^         ^         ^           ^
 16 MB     16 MB     16 MB       partial

Every frame has a known compressed_size and decompressed_size. If you know those sizes in advance (stored in a jump table), you can seek directly to Frame N by summing the compressed sizes of frames 0 through N-1. You never decompress anything you don't need. Frame N is fetched with a single HTTP Range request, decompressed independently, and the relevant bytes are piped downstream.

This is the architectural core of HuskHoard's egress model, and it maps cleanly onto how the Parquet format itself carves up a file.

The Parquet Parallel: Row Groups as Frames

Parquet is deliberately designed for partial reads. A Parquet file contains:

Row groups — horizontal partitions of the data, each independently readable
Column chunks — vertical slices within a row group
Page headers — per-page metadata within each column chunk
Footer — the FileMetaData Thrift struct at the end of the file, containing the schema, row group offsets, column statistics, and key-value metadata. Preceded by a 4-byte footer length and terminated with the magic bytes PAR1.

A reader that wants only the footer performs two range requests: one to get the last 8 bytes (magic + footer length), and one to get the footer itself. Everything else stays on the remote. A reader that wants one column from one row group consults the footer to find the column chunk's byte offset and length, then fires a single range request.

HuskHoard's frame model mirrors this exactly, but at the archive level rather than within a single Parquet file.

HuskHoard's Implementation: Frames, the Catalog, and the Jump Table

When HuskHoard archives a file to any backend — a flat image file acting as a tape volume, a physical LTO cartridge, or an rclone cloud remote — it writes the payload as a sequence of 16 MB Zstd frames. For each frame, it records the mapping between uncompressed byte position and compressed byte position on the volume.

That mapping is the object_frames table in husk_catalog.db:

CREATE TABLE IF NOT EXISTS object_frames (
    file_path           TEXT    NOT NULL,
    version             INTEGER NOT NULL,
    uncompressed_offset INTEGER NOT NULL,   -- where this frame starts in the original file
    compressed_offset   INTEGER NOT NULL,   -- where this frame starts on the storage volume
    compressed_size     INTEGER NOT NULL    -- how many bytes to fetch from the volume
);

CREATE INDEX IF NOT EXISTS idx_frames
    ON object_frames (file_path, version);

This is the jump table. Given a byte range request for bytes=2147483648-2281701376 (a 128 MB window starting at the 2 GB mark), the gateway does:

SELECT compressed_offset, compressed_size
FROM   object_frames
WHERE  file_path = '/warehouse/events/2024-01-01.parquet'
  AND  version   = (SELECT MAX(version) FROM object_frames WHERE file_path = ...)
  AND  uncompressed_offset <= 2147483648
ORDER  BY uncompressed_offset DESC
LIMIT  1;

One row. One seek. One range request against the volume. Everything else stays dark.

The HTTP gateway loop in StreamGate:

HTTP Range request arrives (bytes=X-Y)
        │
        ▼
Query object_frames → nearest frame boundary ≤ X
        │
        ▼
Seek to compressed_offset on volume (tape block, S3 range, local seek)
        │
        ▼
Decompress forward to exact byte X, stream through Y
        │
        ▼
Client receives exactly what it asked for

For a 4K video file seeking to the 2-hour mark, this is why mpv can start playing from tape or S3 in under a second instead of waiting for a multi-gigabyte download.

TLV Footers: Turning the Frame Header Into a Parquet-Style Catalog Entry

Every file archived by HuskHoard is preceded on the storage volume by a strict 4,096-byte ObjectHeader. The first 136 bytes carry the fixed-width mechanics: a magic string (USTDHUSK), the file's UUID, POSIX permissions, BLAKE3 hash, compressed and uncompressed sizes, and a CRC32 of the header itself.

The remaining 3,960 bytes are dedicated to TLV (Type-Length-Value) encoded metadata — the same binary framing used in X.509 certificates, SNMP, and dozens of wire protocols chosen specifically because unknown type codes can be safely skipped by any forward-compatible parser.

Byte  0 –  7: Magic "USTDHUSK"
Byte  8 – 23: Volume UUID (16 bytes)
Byte 24 – 55: BLAKE3 hash (32 bytes)
Byte 56 – 63: Uncompressed payload size
Byte 64 – 71: Compressed payload size
Byte 72 – 79: Original mtime
Byte 80 – 83: POSIX mode
Byte 84 – 87: Header CRC32
Byte 88 –135: File path (null-terminated, 48 bytes max inline)
Byte 136–4095: TLV region (3,960 bytes)

A TLV tag entry looks like:

[Type: u8][Key-Length: u16][Key: bytes][Value-Length: u32][Value: bytes]

This is where the Parquet footer analogy becomes structural rather than metaphorical. For a Parquet file being archived, HuskHoard can embed the Parquet FileMetaData statistics directly into this TLV region:

TLV Type	Key	Value
`0x02`	`parquet.schema`	Serialized Thrift schema (JSON or binary)
`0x02`	`parquet.row_count`	Total row count as little-endian u64
`0x02`	`parquet.col.event_ts.min`	Minimum value of `event_ts` column
`0x02`	`parquet.col.event_ts.max`	Maximum value of `event_ts` column
`0x02`	`parquet.col.user_id.null_count`	Null count for `user_id`
`0x02`	`parquet.row_group.count`	Number of row groups
`0x01`	`workflow.pipeline`	`"ingest_v3"` — POSIX xattr from source
`0x01`	`workflow.owner`	`"data-eng-team"`

These statistics travel physically bonded to the data on every storage medium — disk image, tape cartridge, S3 object. If the SQLite catalog is lost, husk rebuild walks the volume, reads every 4 KB header, and reconstructs the catalog complete with all column statistics. The tape is entirely self-describing.

But the real payoff is what this enables while the catalog is present.

Multi-Volume Catalog Queries: Pruning at the Volume Level

A production HuskHoard deployment might span several volumes:

Volume A (tape, 12 TB) — archive 2022–2023
Volume B (tape, 12 TB) — archive 2023–2024
Volume C (NVMe image, 2 TB) — archive 2024–present
Volume D (S3:us-east-1, 50 TB) — cloud replica

The catalog table records which volume holds each archived version of each file:

SELECT
    c.file_path,
    c.tape_uuid,          -- identifies the volume
    c.tape_offset,        -- byte offset of the ObjectHeader on that volume
    c.payload_size,
    c.compressed_size,
    c.custom_metadata     -- mirrors the TLV tags as JSON
FROM   catalog c
WHERE  json_extract(c.custom_metadata, '$.parquet.col.event_ts.min') >= '2024-01-01'
  AND  json_extract(c.custom_metadata, '$.parquet.col.event_ts.max') <= '2024-03-31'
  AND  json_extract(c.custom_metadata, '$.parquet.row_count')        > 0;

This query executes in milliseconds against the SQLite catalog on your SSD. The tape drives stay spun down. S3 is never contacted. You get back a list of (file_path, tape_uuid, tape_offset) tuples — the exact volumes and positions to touch.

Then, per file, for each column you actually need:

SELECT compressed_offset, compressed_size
FROM   object_frames
WHERE  file_path           = '/warehouse/events/2024-01-15.parquet'
  AND  version             = 3
  AND  uncompressed_offset BETWEEN :col_chunk_start AND :col_chunk_end
ORDER  BY uncompressed_offset;

You issue a range request for only those frames. For a 4 GB Parquet file where you need one 200 MB column chunk:

Step	Data Transferred
Catalog query (SQLite, local)	0 bytes egress
object_frames lookup (SQLite, local)	0 bytes egress
HTTP Range to S3 (compressed frame bytes)	~85 MB (at 2.4:1 Zstd ratio)
Total vs naïve full-file fetch	85 MB vs 1.7 GB

That is a 95% reduction on a per-query basis.

Putting Numbers to the Savings

Let's use a concrete scenario: a data team maintains a 10 TB cold Parquet archive on S3, with an average file size of 4 GB. They run three workloads:

Workload A — Nightly catalog sync (schema + statistics only)

Files: 2,500 Parquet files
Data needed per file: footer only (~150 KB each)
Total needed: ~375 MB
Full-file cost: 10 TB × $0.09 = $921.60/month
Partial-frame cost: 375 MB × $0.09 = $0.03/month
Monthly savings: $921.57

Workload B — Ad-hoc column scan (one column across 20% of files)

Files queried: 500 (selected by TLV statistics predicate)
Column chunk per file: ~200 MB uncompressed → ~85 MB compressed frames
Total fetched: ~42.5 GB
Full-file cost: 500 × 4 GB × $0.09 = $180.00/query
Partial-frame cost: 42.5 GB × $0.09 = $3.83/query
Per-query savings: $176.17 (97.9%)

Workload C — Point-in-time restore of a single row group

1 file × 1 row group = 128 MB uncompressed → ~54 MB compressed
Full-file cost: 4 GB × $0.09 = $0.36
Partial-frame cost: 54 MB × $0.09 = $0.005
Per-restore savings: $0.355 (98.6%)

At scale, Workload A alone — a nightly catalog sync that most teams run without thinking about the bill — generates ~$11,000/year in unnecessary egress on a 10 TB archive. The frame-indexed approach reduces that to under $1/year.

The Self-Describing Volume: Your Catalog Backup Is on the Tape

One underappreciated consequence of storing TLV column statistics in every ObjectHeader is that the volume itself becomes a data catalog. After a complete disaster recovery (catalog database lost, fresh server), husk rebuild walks the storage volume 4 KB at a time, reads every ObjectHeader, validates the CRC32, and inserts a new catalog row including all TLV-encoded metadata — column statistics, schema, pipeline tags, everything.

The catalog is not a separate system that the archive depends on. The catalog is a cache that accelerates access to information already encoded in the archive itself. This is the same philosophical commitment Parquet makes: the footer is not a separate sidecar file; it is part of the format.

For teams integrating with external data catalogs (Apache Atlas, Hive Metastore, Unity Catalog), this means HuskHoard can emit catalog events on husk rebuild just as well as on initial archive — the metadata survives the worst failure scenario, format-native.

Wiring It Up: What a Partial Read Looks Like End-to-End

A data engineer's dbt model lands at the StreamGate HTTP gateway with a Range: bytes=536870912-671088640 request (512 MB – 640 MB, pulling a specific row group from a 4 GB Parquet file on S3):

1. GET http://localhost:8080/v1/stream/warehouse/events/2024-01-15.parquet
   Range: bytes=536870912-671088640

2. Gateway queries object_frames:
   → nearest frame boundary ≤ 536870912 is at uncompressed_offset=536870912
   → compressed_offset=225,978,112 on Volume D (S3:us-east-1)
   → 6 frames needed, compressed total = 56.3 MB

3. Gateway fires:
   GET s3://huskhoard-cold/volume-d.img
   Range: bytes=225978112-285884415

4. Gateway decompresses frames on the fly, streams bytes 536870912–671088640
   to the client.

5. Total egress from S3: 56.3 MB
   Total egress if client had fetched the full file: 1.71 GB
   Savings: 96.7%
   Time to first byte (LAN): ~180ms vs ~14s for full-file download

The client — dbt, Spark, DuckDB, curl, whatever — receives a standard HTTP 206 Partial Content response. No special client library. No SDK. Just the HTTP Range spec, universally supported.

Practical Takeaways for Data Engineers

1. Frame size is a tuning knob. HuskHoard defaults to 16 MB frames, optimized for cloud PUT cost (fewer, larger requests) and Zstd compression ratio. For workloads with very fine-grained access patterns (column-level reads in narrow schemas), smaller frames (1–4 MB) reduce the minimum fetch size at the cost of more catalog rows and higher PUT count. Benchmark against your actual access patterns.

2. TLV statistics are opt-in per file type. For video files you probably don't store column min/max values. For Parquet, CSV, and Arrow IPC files it's worth paying the archiver CPU time to extract and embed statistics at archive time — you pay once and recoup every time a catalog query avoids a volume read.

3. The catalog query is your explain plan. Before a restore or a scan, husk catalog query --path "/warehouse/events/*.parquet" --filter "parquet.col.event_ts.min >= 2024-01-01" shows you which volumes and frame ranges will be touched. Run it first. If the egress estimate is unexpected, the TLV coverage on those files probably needs improving.

4. Multi-volume means cross-volume pruning is free. A query that touches two volumes and skips three is doing volume-level predicate pushdown before any I/O. The catalog does this automatically based on the tape_uuid in each matching row.

5. Egress savings compound with replication. HuskHoard replicates to multiple volumes simultaneously. If your primary volume is on S3 ($0.09/GB egress) and your replica is on Cloudflare R2 ($0.00 egress), the gateway can route the range request to whichever backend minimizes cost. Partial reads from R2 are free. You still benefit from the jump table because API call count and latency still matter.

# Why I Bypassed FUSE: Building a Transparent DataTiering Engine in Rust

Joichiro Mitaka — Fri, 19 Jun 2026 05:51:33 +0000

If you run a home lab or manage large datasets, you’ve hit this wall: NVMe drives are fast but too expensive to hoard data on. Hard drives or cloud buckets are cheap, but they are slow and a pain to manage manually.

The enterprise world solves this with HSM Hierarchical Storage Management automatically shuffling cold data to slow storage while keeping a transparent stub on the fast drive. But enterprise HSMs cost thousands of dollars and lock your data in proprietary black boxes.

I wanted this for Linux, for free. So, I started building HuskHoard, an opensource data tiering engine.

My first thought, like almost every Linux developer building a virtual filesystem, was to use FUSE. But I quickly realized FUSE was the wrong tool for the job. Here is why I abandoned it, and how I used the Linux fanotify API and Rust to build a transparent, zero overhead archiving engine.

The Problem with FUSE

FUSE is fantastic for creating custom filesystems like SSHFS or mounting an S3 bucket. But for an HSM, it creates a massive bottleneck.

When you use FUSE, every single read and write has to go through a context switch:
Application > Kernel > FUSE Daemon Userspace > Kernel > Physical Drive.

If 90% of your data is Hot actively being used on your fast NVMe, forcing it through FUSE overhead completely defeats the purpose of buying expensive NVMe drives in the first place. You sacrifice native I/O performance just to manage the 10% of Cold data.

I needed a solution where the Hot data ran at native speed, touching nothing but the XFS/Ext4 kernel drivers.

The Solution: Enter fanotify

Instead of intercepting every transaction via FUSE, I realized I only needed to intervene in one specific scenario: When a user tries to open a file that has been archived.

Linux has a kernel API called fanotify originally designed for antivirus scanners. It allows a userspace program to monitor a mount point and, crucially, block an application from opening a file until the daemon says it’s okay.

Here is how HuskHoard uses fanotify to create transparent tiering:

The Janitor: A background Rust thread scans my NVMe drive. When it finds a file that hasnt been touched in 30 days, it compresses it Zstd and moves the payload to a cheap HDD, LTO Tape, or S3 bucket.
The Husk Stub: It leaves the original file on the NVMe drive but truncates its allocated size to 0 bytes creating a sparse file. To the OS and the user, the file still looks like it’s 50GB and sits in /home/movies.
The Interceptor: This is where fanotify shines. The HuskHoard daemon listens for FAN_ACCESS_PERM events. If VLC media player tries to open that Husk file, fanotify pauses VLCs execution in the kernel.
The Recall: HuskHoard intercepts the request, streams the 50GB payload from the tape/S3 bucket back into the sparse file on the NVMe, and then tells fanotify to allow VLC to proceed.

VLC thinks it just opened a local file. It has no idea the data was fetched from an S3 bucket 50 milliseconds ago.

The Rust Implementation

Rust was the obvious choice for this. When you are blocking kernellevel I/O requests, memory safety and predictable latency are nonnegotiable.

Handling the fanotify loop requires a few specific Linux capabilities specifically CAP_SYS_ADMIN, but Rust allows us to safely manage the multithreaded heavy lifting of the Archive Worker.

pub fn run_interceptor(config: Arc<HuskConfig>, use_direct_io: bool) -> std::io::Result<()> {
    let watch_dir = &config.hot_tier;
    let db_path = &config.db_path;
    info!("\n[Daemon] Starting fanotify interceptor on '{}'...", watch_dir);
    let abs_dir = std::fs::canonicalize(watch_dir)?;

    let fan_fd = unsafe {
        libc::fanotify_init(libc::FAN_CLASS_PRE_CONTENT, libc::O_RDWR as u32)
    };
    if fan_fd < 0 { 
        let err = std::io::Error::last_os_error();
        error!(" fanotify_init failed: {}. Missing Root or Capabilities!", err);
        return Err(err); 
    }


        let mark_mask = libc::FAN_ACCESS_PERM | libc::FAN_CLOSE_WRITE | libc::FAN_EVENT_ON_CHILD;

        // 1. Recursively mark the root watch directory and all current subdirectories
        info!("[Daemon]  Scanning and attaching listeners to all subdirectories...");
        mark_directory_recursive(fan_fd, &abs_dir, mark_mask, &config);

Escaping Vendor Lockin The Easy Exit Promise

One of the biggest issues with commercial HSMs is that if the daemon dies, your data is gone, trapped in proprietary metadata.

Because I was building this for the opensource community, I enforced a strict Easy Exit architecture:

Payload data is stored in standard Zstd streams verified by BLAKE3.
The catalog metadata the Brain tracking where the cold bytes live is an SQLite database.
You can natively export the entire catalog to Apache Parquet.

This means if you decide to stop using HuskHoard, you dont need my software to get your data back. You can query your catalog with DuckDB or Python and manually extract your Zstd archives.

Whats Next?

Building HuskHoard has been a massive deepdive into Linux kernel APIs and SCSI Tape drivers. Yes, it natively supports physical LTO drives via /dev/nstX to prevent tape shoeshining.

The engine currently supports automated replication across local drives, tapes, and rclone supported cloud buckets.

If you are a Rust developer, a HomeLab data hoarder, or just interested in Linux storage architecture, Id love your feedback or code reviews. It is fully AGPL v3 licensed.

Check out the repo here: [GitHub HuskHoard]https://github.com/huskhoard/huskhoard
More architecture details: [HuskHoard Blog]https://www.huskhoard.com/blog.html