DEV Community: Joichiro Mitaka

Point-in-Time Recovery Without the Enterprise Price Tag

Joichiro Mitaka — Sat, 25 Jul 2026 07:28:28 +0000

description: A deep dive into HuskHoard's open-source PITR engine — how the catalog tracks version history and byte offsets across physical media, and how you roll back a single file without touching the rest of the volume.

Point-in-Time Recovery Without the Enterprise Price Tag

If you've priced out Veeam, Commvault, or a Zerto license recently, you already know the punchline: granular point-in-time recovery is one of the most aggressively monetized features in enterprise storage. Vendors charge for it because it genuinely solves a hard problem — and because buyers are often in enough pain that they'll pay whatever it costs.

HuskHoard is an open source data tiering archive for Linux that ships PITR as a first-class feature, built directly into its catalog engine, free under AGPL v3. This post is the deep dive that the project's docs don't yet cover: how versioning actually works under the hood, how the catalog tracks byte offsets across physical media (disk, LTO tape, and S3), and how you roll back a single file without touching anything else on the volume.

Versioning vs Backup: Stop Using the Words Interchangeably

Before getting into mechanics, it's worth being precise about terminology, because these two words describe fundamentally different operations with different tradeoffs.

A backup is a complete redundant copy of a dataset at a point in time. The goal is full-loss recovery. If your server catches fire, you restore the backup. The cost is that you're duplicating data — 10 TB of active data requires at least 20 TB of total storage to have one backup copy.

Versioning (or PITR at the object level) is a historical record of how a specific file changed over time. The goal is granular rollback — recovering from an overwrite, a corruption, or a ransomware event that targeted specific files. It doesn't guarantee you can recover from total hardware loss unless your versions live on separate media. But it lets you say "give me /etc/nginx/nginx.conf as it was three weeks ago" without restoring your entire backup set.

The confusion is expensive. People who treat their archive as a backup get burned when the archive goes down (there's no redundant copy). People who treat their backup as a versioning system get burned when they need to roll back a single file and discover they have to restore 2 TB of snapshot data to get at it.

A solid DR strategy uses both:

Scenario	What You Want
Server catches fire / disk fails completely	Backup — full restore from a redundant copy
Developer overwrites critical config file	PITR — roll back that specific file in seconds
Ransomware encrypts 10,000 files	PITR — roll back affected files to pre-infection version
VM guest OS corrupts itself	Backup — restore the whole VM image
VM config file or snapshot is accidentally deleted	PITR — roll back just that object

HuskHoard handles versioning (PITR at the file/object level) natively. For full redundancy, pair it with N-way replication across multiple volumes, which HuskHoard also handles.

How HuskHoard's Versioning Engine Works

The Append-Only Storage Model

HuskHoard's architecture is built on a core constraint borrowed from tape: all writes are append-only and sequential. Even if you're archiving to a spinning disk image or an S3 bucket, HuskHoard writes to the volume as though it were a tape — from the current write head forward, never punching holes or overwriting previous data.

This constraint, which exists primarily to avoid the "write wall" problem on SMR drives and to stay compatible with physical LTO hardware, turns out to give you versioning entirely for free. When a file is re-archived, the old payload stays exactly where it was on the volume. The new version gets appended after everything else. Nothing is overwritten.

The catalog is where this becomes useful.

The Catalog Schema — The Version Column

Here is the actual schema from database.rs, as discussed in The Catalog post:

CREATE TABLE IF NOT EXISTS catalog (
    id               INTEGER PRIMARY KEY AUTOINCREMENT,
    file_path        TEXT    NOT NULL,
    version          INTEGER NOT NULL,
    tape_uuid        TEXT    NOT NULL,
    tape_offset      INTEGER NOT NULL,
    payload_size     INTEGER NOT NULL,  -- uncompressed byte count
    compressed_size  INTEGER DEFAULT 0,
    compression_type INTEGER DEFAULT 0, -- 0=None, 1=Zstd
    uid              INTEGER NOT NULL,
    gid              INTEGER NOT NULL,
    posix_mode       INTEGER NOT NULL,
    archived_at      DATETIME DEFAULT CURRENT_TIMESTAMP,
    original_mtime   INTEGER NOT NULL,
    blake3_hash      TEXT    NOT NULL,
    custom_metadata  TEXT
);

CREATE INDEX IF NOT EXISTS idx_path_version ON catalog (file_path, version);

The key is the (file_path, version) pair. Every time the Janitor archives a file that has been modified since its last archive event, it does not overwrite the previous row. It inserts a new row with version = previous_version + 1. The old row — pointing to the old payload at its old tape_offset — stays in the catalog permanently (until pruned by max_versions).

Because the old payload is still physically present on the append-only volume, that old row is a complete, restorable snapshot of the file at the moment it was archived.

file_path                        | version | tape_uuid | tape_offset | archived_at
---------------------------------+---------+-----------+-------------+---------------------
/data/projects/config.yaml       |       1 | a1b2c3... |        4096 | 2026-04-01 09:00:00
/data/projects/config.yaml       |       2 | a1b2c3... |    52432896 | 2026-04-15 14:22:11
/data/projects/config.yaml       |       3 | a1b2c3... |   104857600 | 2026-06-01 09:30:45

Three archive events. Three catalog rows. Three independently restorable versions. Zero additional tooling required.

Rolling Back a Specific Version

From the CLI:

# Restore the latest version (default behavior)
huskhoard restore --file /data/projects/config.yaml

# Roll back to a specific version
huskhoard restore --file /data/projects/config.yaml --version 2

# List all available versions for a file
huskhoard versions --file /data/projects/config.yaml

The versions subcommand shows you what's available:

/data/projects/config.yaml
  v1  2026-04-01 09:00:00  blake3: a3f1...  44 KB   [tape: a1b2c3..., offset: 4096]
  v2  2026-04-15 14:22:11  blake3: 7c90...  47 KB   [tape: a1b2c3..., offset: 52432896]
  v3  2026-06-01 09:30:45  blake3: d81e...  51 KB   [tape: a1b2c3..., offset: 104857600]  ← current

Under the Hood: The Mechanics of a Byte-Level Rollback

This is the part that separates HuskHoard's approach from snapshot-based alternatives — and it's worth understanding in detail.

Step 1: Catalog Lookup

When you run huskhoard restore --file /path/to/file --version 2, the first thing that happens is a single indexed SQL query:

SELECT tape_uuid, tape_offset, payload_size, compressed_size,
       compression_type, blake3_hash, uid, gid, posix_mode, original_mtime
FROM catalog
WHERE file_path = '/data/projects/config.yaml'
  AND version = 2;

The (file_path, version) composite index makes this microsecond-fast regardless of how many millions of files and versions are in the catalog.

This query returns a tape_uuid and a tape_offset — a complete physical address for the version's payload.

Step 2: Volume Resolution

The tape_uuid isn't a device path. It's a 16-byte UUID burned into the volume's header block when it was first formatted with huskhoard format. This is critical: it means the restore doesn't care that your LTO drive moved from /dev/nst0 to /dev/nst1, or that your disk image was renamed, or that your rclone remote was reconfigured. The daemon resolves the current physical location of any UUID by scanning tapes table entries against live hardware on startup.

SELECT device_path, backend_type
FROM tapes
WHERE tape_uuid = 'a1b2c3...';
-- Returns: /dev/nst0, local

Step 3: Seek to the Exact Byte Offset

Once the device is located, HuskHoard seeks directly to tape_offset. For disk images, this is a standard lseek(). For physical LTO tape, this uses hardware SCSI positioning commands to reach the correct block without reading through intervening data.

Volume [a1b2c3...]
 ┌─────────────────────────────────────────────────────────────────────┐
 │ Header │ v1 payload │ other files... │ v2 payload │ v3 payload │... │
 │  4KB   │  offset:   │               │  offset:   │  offset:   │    │
 │        │  4,096     │               │  52,432,896│ 104,857,600│    │
 └─────────────────────────────────────────────────────────────────────┘
                                             ↑
                               seek here for version 2

No other version's data is read. No other file's data is read. The restore is surgical.

Step 4: Decompress and Verify

The payload at that offset is a Zstd-compressed stream. HuskHoard decompresses it and feeds the output bytes through a BLAKE3 hasher simultaneously. When decompression completes, the computed hash is compared against the blake3_hash stored in the catalog at archive time. If they don't match, the restore is aborted and the corrupted block is flagged.

This means every version restore is automatically integrity-verified. You're not just getting the bytes that were there — you're getting cryptographic confirmation that those bytes are exactly what was archived.

Step 5: Atomic Placement

The restored bytes are written to a temporary file, ownership and permissions are set from the catalog's uid, gid, and posix_mode columns, and then the file is atomically renamed into place at the target path. The previous stub (or the current version, if one existed) is replaced atomically — no partially-written state is ever visible to the OS.

The `object_frames` Table: Partial-Content Rollback Without Full Recall

For large files, there's an additional optimization. When HuskHoard archives a file, it writes the Zstd stream in discrete frames (typically 16 MB each) and records the mapping of uncompressed byte ranges to their compressed positions in the object_frames table:

CREATE TABLE IF NOT EXISTS object_frames (
    file_path           TEXT    NOT NULL,
    version             INTEGER NOT NULL,
    uncompressed_offset INTEGER NOT NULL,
    compressed_offset   INTEGER NOT NULL,
    compressed_size     INTEGER NOT NULL
);

This is primarily what powers the StreamGate HTTP gateway (letting Plex or VLC seek into a 100 GB video on tape without full recall), but it also means that targeted restores of large files don't require reading the entire payload from media. If you need to verify the state of bytes 4,000,000,000 through 4,016,000,000 of a 2 TB database dump at version 5, HuskHoard can seek directly to the relevant frame, decompress only that 16 MB chunk, and verify just the portion you need.

This is fundamentally different from snapshot-based PITR in backup tools, which typically require mounting an entire snapshot volume before you can access any individual file within it.

Real-World Strategies

Strategy 1: File Versioning for Granular DR

The most common use case. You have a home lab or a small team environment where files change regularly — code, configs, media project files, database exports. You want the ability to say "give me that file as it was two weeks ago."

Setup: Point HuskHoard's hot tier at your working directories. Set a max_age_days appropriate to your workflow (7 or 14 days is typical). HuskHoard will archive files that haven't been touched since the threshold, and re-archive modified files, building up a version history automatically.

# husk_config.toml
hot_tier = "/home/user/projects"
primary_volumes = ["/mnt/archive/main.img"]
max_age_days = 14
janitor_interval_secs = 3600
max_versions = 10  # keep the 10 most recent versions per file

Recovery scenario: Your web app's database.conf got overwritten by a botched deployment script at 3 PM on a Tuesday. You want the version from Monday.

# See what versions exist
huskhoard versions --file /home/user/projects/myapp/database.conf

# Output:
#   v1  2026-05-20 08:14:00  blake3: 9a2f...  1.2 KB
#   v2  2026-05-27 11:55:33  blake3: 4d8c...  1.3 KB   ← Monday's version
#   v3  2026-06-03 15:01:44  blake3: f1a0...  1.3 KB   ← broken deployment

# Restore Monday's version directly to path
huskhoard restore --file /home/user/projects/myapp/database.conf --version 2

Total recovery time: seconds. You don't need to find last night's backup tape, mount a snapshot, or navigate a backup UI. You query the catalog, seek to a byte offset, decompress, verify. Done.

Strategy 2: VM Backup + File Versioning for Faster DR

This is where the versioning-vs-backup distinction pays off in a real operational context.

If you're running KVM or Proxmox, your VM disk images (.qcow2 or raw) are just files. You can absolutely archive them with HuskHoard, and every re-archive creates a new versioned copy that you can roll back to. But a full restore of a 500 GB VM image is still going to take time, even with direct seeks to the right offset on tape.

A better hybrid strategy:

Back up the full VM image periodically (weekly or on major changes). This lives as a versioned archive in HuskHoard — your safety net for total VM loss.

Version the individual files that change frequently and are small enough to restore in seconds: VM configs, cloud-init seeds, network configuration files, snapshot manifests, Terraform state files, Ansible inventories.

# Archive the full VM disk image (becomes version 1, then 2, then N)
# HuskHoard handles this automatically when the Janitor runs
ls -lh /var/lib/libvirt/images/production-web.qcow2
# 487G production-web.qcow2

# But for the config files alongside it:
huskhoard versions --file /etc/libvirt/qemu/production-web.xml
#   v1  2026-04-01  blake3: ...  12 KB
#   v2  2026-05-15  blake3: ...  13 KB
#   v3  2026-06-10  blake3: ...  14 KB  ← current

# VM XML config accidentally deleted or corrupted?
huskhoard restore --file /etc/libvirt/qemu/production-web.xml --version 2
# Restored in < 1 second

The result: when something breaks, you reach for the right tool at the right granularity. Config went wrong? PITR the config file. VM guest OS corrupted itself? Restore the full image from the latest VM image archive. You're never force-multiplying a small problem into a large recovery operation.

DR time comparison for "someone edited the wrong network bridge config":

Approach	Recovery Time
Restore full VM from last night's backup	45 minutes to 2 hours
Mount last night's VM snapshot, extract file	10–20 minutes
HuskHoard PITR the specific XML file	< 5 seconds

Strategy 3: Defending Against Ransomware

Ransomware targets active files. A ransomware event that encrypts your working data will create a wave of "modified" files that, if your archive system isn't careful, will get archived as new versions — burying the pre-encryption versions under the corrupted ones.

HuskHoard's max_versions config is your safety net here. As long as max_versions is set high enough that the pre-infection versions haven't been pruned, you can roll back every affected file individually.

# After discovering ransomware encrypted files on June 10th at 14:00

# Find all files modified after that timestamp that have prior versions
huskhoard versions --file /data/documents/proposal.docx

#   v1  2026-05-01 10:00:00  blake3: 3c8a...  45 KB  ← clean
#   v2  2026-06-10 14:03:22  blake3: 9f1b...  45 KB  ← encrypted

# Roll back to clean version
huskhoard restore --file /data/documents/proposal.docx --version 1

The append-only storage model is what makes this safe. The encrypted version was appended to the volume — it didn't overwrite the clean version. Both are present, both restorable.

For a large-scale ransomware event affecting thousands of files, you'd script the rollback:

# Roll back all files in /data/documents/ to their version 1
huskhoard restore --directory /data/documents/ --before "2026-06-10 14:00:00"

Pruning Old Versions

Versions accumulate on append-only media. The max_versions config key controls how many versions per file are retained in the catalog (and eligible for restore):

max_versions = 10  # Default: keep the 10 most recent versions per file

When a file exceeds max_versions, the oldest version's catalog row is deleted. The underlying payload bytes on the storage volume are not immediately freed — they become "dark" space that gets reclaimed the next time huskhoard repack is run on that volume (the garbage collection pass).

This means you always have control over the disk-vs-history tradeoff. More versions = more history = more storage consumed on the archive tier over time. For most file versioning use cases, 5–10 versions is sufficient.

What Enterprise Vendors Charge for This

Just for context. Veeam's granular file recovery requires their Enterprise Plus tier. Commvault's IntelliSnap point-in-time recovery is licensed per agent. Zerto's journal-based PITR (which tracks continuous block-level changes for arbitrary-timestamp recovery) starts at thousands of dollars per month for production workloads. IBM Spectrum Protect (formerly TSM) charges per terabyte of managed data.

HuskHoard's PITR engine — the catalog, the version rows, the byte-offset seeks, the BLAKE3 verification — is in the open-source core, licensed AGPL v3. You can read every line of the implementation in src/database.rs and src/daemon.rs.

The tradeoff is that HuskHoard's versioning is event-driven (archive events) rather than continuous (block-level journal). You get snapshots at the moments the Janitor ran, not arbitrary-timestamp recovery for every write in between. For most use cases — homelab, small teams, media workflows, development environments — that's entirely sufficient. For continuous block-level CDP of a high-transaction OLTP database, you need a dedicated database tool for that layer.

Getting Started

# Clone and build
git clone https://github.com/huskhoard/huskhoard.git && cd huskhoard
cargo build --release

# Grant capabilities (no root daemon needed)
sudo setcap cap_sys_admin,cap_dac_read_search+ep target/release/huskhoard

# Format an archive volume and start the engine
./target/release/huskhoard format --tape-dev my_archive.img
./target/release/huskhoard daemon

# After files are archived, list versions
./target/release/huskhoard versions --file /path/to/your/file

# Roll back
./target/release/huskhoard restore --file /path/to/your/file --version 2

Full documentation is on GitHub. The architecture series on HuskHoard.com/blog.html covers the catalog schema, file stubbing, the fanotify interceptor, and SMR drive handling in depth — good reading if you want to understand the full system before deploying it.

Summary

Versioning and backup are different tools for different failure modes. Use both, not one instead of the other.
HuskHoard's append-only storage model gives you versioning for free — old payloads are never overwritten, so every archive event is inherently a new restorable version.
The catalog is the version index. A (file_path, version) row with a tape_uuid + tape_offset physical address is all you need to restore any specific version of any file in microseconds.
Rolling back a single file is a single seek. No snapshot volumes to mount, no backup sets to restore, no staging areas. Seek to the offset, decompress, verify the BLAKE3 hash, place atomically.
The object_frames jump table means even large files can be partially restored without reading the entire payload from media.
The whole engine is open source, free under AGPL v3, built in Rust, and runs in user-space on any Linux kernel 5.1+.

If you're evaluating backup or archiving software and PITR is on your requirements list, it's worth spinning up HuskHoard before you sign any enterprise contracts.

HuskHoard is an open source data tiering archive. Source code at github.com/HuskHoard/HuskHoard. Project site and architecture blog at huskhoard.com.

Why You Can't Just Mount an S3 Bucket and Start Editing

Joichiro Mitaka — Fri, 17 Jul 2026 07:01:45 +0000

Everyone who's tried pointing an NLE at a mounted S3 bucket has hit the same wall. Browsing folders is slow. Scrubbing stutters or hangs. The usual fix people reach for is downloading the whole file before touching it, which defeats the entire point of using cheap, scalable object storage in the first place.

The reason is simple once you see it: object storage — S3, MinIO, Ceph RGW, TrueNAS's S3-compatible gateway — isn't a filesystem. There's no directory tree, no stat(), no partial reads in the POSIX sense. Everything is a flat key with a GET and a PUT. Any workflow that wants to browse and edit media stored this way has to build a filesystem-shaped illusion on top of an object-shaped reality.

Tools like LucidLink made that illusion feel seamless enough that people forget it's an illusion at all. But the trick isn't magic, and it isn't proprietary to any one product — it's a handful of well-understood pieces working together. Worth breaking those apart before looking at any specific implementation.

The naive approach: mount and download

The simplest way to make an S3 bucket look like a drive is a FUSE mount (s3fs, goofys, rclone mount, etc.). These translate filesystem calls into API calls in real time. That works fine for sequential access — copying a file, reading a log — but post-production workloads are not sequential. Scrubbing a timeline, seeking to a mark, or letting a codec read its index atoms means constant, unpredictable jumps around a file. Each of those can turn into a separate HTTP request under a naive FUSE translation layer, and latency stacks up fast.

The common workaround is to just download the whole file before handing it to the application — effectively a cache-on-open. It works, but it reintroduces the thing object storage was supposed to let you avoid: waiting for a full copy before you can touch a frame.

The basic building block: transparent stubbing

Before getting to how bytes get fetched, it's worth being explicit about what the filesystem shows the application in the meantime — this is the part most write-ups skip, and it's the actual foundation the rest of the system sits on.

A stub (also called a placeholder, or "smart" file in the OneDrive/Dropbox world) is a filesystem entry that reports correct, fully-formed metadata — file size, timestamps, permissions — without the underlying data being present locally. ls, stat, project panels, media browsers, and autosave routines all see what looks like a complete, correctly-sized file. Only when something actually tries to read content does the system need to do anything.

This is the same underlying idea as OneDrive Files On-Demand or Dropbox Smart Sync, but post-production adds a wrinkle: those consumer implementations generally hydrate a file in full on first touch. A stub used for editorial has to support partial hydration — serving byte range 40MB–41MB of a 40GB file without pulling the other 39.999GB. That distinction is what separates a "smart sync" folder from something you can scrub a 4K timeline against.

Two things a stubbing layer needs to get right:

Correct size reporting from creation. If an NLE or transcoder trusts stat() size and the stub lies about it, seeks past the "real" data will behave unpredictably.
Read interception at the right layer. Something has to sit between "application calls read()" and "bytes are actually available," decide what's missing, fetch it, and hand it back — ideally without the application knowing this happened at all.

That second point is where implementations diverge.

Intercepting reads: FUSE vs. fanotify

FUSE handles this by owning the entire filesystem — every syscall against the mount routes through a userspace daemon. It's flexible but has real overhead, since even reads that could be served straight from a local page cache round-trip through the FUSE protocol.

An alternative is Linux's fanotify API, which lets a userspace process subscribe to filesystem events — opens, accesses, and, with permission events, the ability to block a read until the daemon signals it's safe to proceed. Instead of intermediating the entire filesystem, fanotify sits over a real (or virtual) filesystem and only intervenes when it needs to: on open, or on access to a range that isn't populated yet. Once data is present, subsequent reads can be served by the kernel's normal page cache with no daemon involvement.

Practically, that means:

Application opens a file → daemon sees the event, resolves it against the object store, ensures a stub exists locally.
Application reads offset X → if that range isn't hydrated, the daemon fetches exactly that byte range from the backend (an S3 GET with a Range header, or equivalent) and writes it into the local stub before releasing the read.
Application reads an already-hydrated offset → served locally, no network round trip.

This is closer to how a block device with a remote backing store behaves than a download manager. It also means media stays in its native container/codec — no re-chunking or transcoding is required to make this work, which matters if you don't want to convert an existing archive to participate.

The trade-off is real: fanotify permission events typically need elevated privileges, and the very first touch of a cold region always costs a network round trip no matter how the interception is done. Nothing eliminates the physics of "the bytes have to come from somewhere the first time."

The other bottleneck: just listing files

Byte-range fetching solves reading data. It doesn't solve finding out what data exists. Object stores don't have real directories — list-objects calls against a prefix are how "folders" get faked, and at scale (media libraries commonly run into the millions of objects) those list calls get slow, especially with deep, nested project structures. This is the part of cloud-mount tools that often feels worse than the streaming itself: opening a folder shouldn't take longer than opening the file inside it.

The usual fix is a local metadata catalog — commonly something as simple as SQLite — that mirrors the object store's namespace and metadata (paths, sizes, timestamps, hydration state) locally. Directory listings, stat() calls, and search then hit the local database instead of the network. The catalog needs to stay in sync with the backend (via polling, event notifications, or explicit indexing passes), but once it does, browsing a library feels local regardless of library size, because it is local — only the file contents are remote.

Putting the layers together

Strip away branding and this pattern generally has three layers:

A local metadata catalog — fast namespace and stat() info, decoupled from the backend's list performance.
Transparent, partially-hydratable stub files — so applications see a complete, correctly-sized tree immediately.
Demand-driven byte-range fetching — via fanotify, FUSE, or another interception method — that pulls only the ranges actually requested, on the fly, in native format.

None of these is exotic on its own. The interesting engineering is in making them consistent with each other under real editorial load: concurrent readers, scrubbing, proxy generation, and multiple users hitting the same objects from different machines.

Where HuskHoard fits

HuskHoard is a project that implements this pattern, aimed at people who want it running against storage they already control rather than a hosted service. It uses a local SQLite catalog for directory/metadata performance, fanotify-based interception for read handling, and treats the backend as swappable — S3, MinIO, Ceph RGW, or TrueNAS's S3 gateway all work the same way from the client side, since the client only needs an S3-compatible API to talk to.

It's not only cloud

The three-layer pattern above doesn't actually require the slow, non-randomly-addressable backend to be a network API. It applies just as well when the backend is a local device with the same basic problem: sequential, high-latency access that punishes naive random reads.

LTO tape is the obvious case — real seek latency, strictly sequential media, no benefit from re-reading small ranges the way a disk cache would help. SMR (shingled) hard drives have their own version of it: random writes into an already-shingled zone are drastically slower than sequential writes, because the drive has to read-modify-rewrite an entire shingle band. Both are cheap-per-terabyte storage that behaves badly under an editorial access pattern, which is exactly the problem the catalog/stub/interceptor stack was built to solve in the first place.

HuskHoard runs that same stack against local media directly, rather than only against S3-style APIs:

A native SCSI tape driver handles LTO-5 through LTO-9, tracking filemark positions and 256KB block alignment so the catalog can seek a file to within seconds instead of doing a linear tape rewind (and avoiding "shoe-shining" — the repeated back-and-forth tape motion that happens when a drive is fed data slower than its streaming speed).
SMR drives get a strict log-structured write path, so data is only ever appended sequentially and nothing writes back into an already-written shingle zone.
A local HTTP gateway (StreamGate) exposes the same indexed byte-range seeking whether the file lives on tape, an SMR archive drive, or S3 — so a player like mpv, ffmpeg, Plex, or Jellyfin can scrub into a multi-terabyte file on a tape shelf the same way it would scrub a file on cloud storage, without staging the whole thing to disk first.

That's the actual differentiator worth calling out: this isn't "S3 but with a nicer client." The stubbing-and-intercept model isn't tied to object storage as a concept — it works anywhere data can be indexed and fetched by range, which turns out to include a tape autoloader as much as it includes a cloud bucket. It's the LucidLink-style instant-access experience pointed at whatever's actually holding the cold data, network-attached or not.

It's not doing anything conceptually different from the LucidLink-style tools people on this forum already know — the value proposition is that it's self-hosted against infrastructure you own, rather than a subscription tied to a vendor's storage backend. Worth a look if you're already running S3-compatible storage in-house and want the streaming-editorial workflow without adding a hosted middleman

GitHub.com/huskhoard/huskhoard

Tape Has Come Back From the Dead (It Was Never Dead)

Joichiro Mitaka — Tue, 14 Jul 2026 06:44:37 +0000

Every few years someone declares tape dead. Earlier this year it was DOGE staffers reportedly mocking the Library of Congress for still using tape. Before that it was the cloud. Before that it was cheap spinning disk. Tape has been "dying" for roughly thirty years now.

It isn't dying. The numbers say the opposite, and the reason most people miss this is that they're still picturing tape the way it worked in 2005: write a reel, label it by hand, put it on a shelf, forget what's on it. That workflow is dying. The medium underneath it is having its best decade ever — but only if you stop treating it like a glorified USB stick and start treating it like what it actually is: a storage tier that needs the same indexing, metadata, and search infrastructure you'd expect from any modern filesystem.

This post covers three things: why the "tape is dead" narrative is wrong by the numbers, why the common ways people do use tape (pure backup target, raw LTFS) leave most of its value on the table, and what a real solution looks like, a global namespace, a persistent catalog, and (looking forward a bit) AI generated descriptive metadata that makes a multi-petabyte cold archive as searchable as your local filesystem.

The numbers don't lie

In 2024, the LTO Program — the joint body run by HPE, IBM, and Quantum that governs the Linear Tape-Open standard — shipped a record 176.5 exabytes of compressed tape capacity. That's the fourth consecutive year of growth, and a 15.4% jump over 2023.

A couple of things stand out in that chart. First, the 2020 dip is the pandemic — new data center buildouts paused and tape shipments fell 8% before snapping back with 40% growth in 2021. Second, the climb since then hasn't slowed down at all. The LTO Program's own attribution is blunt about why: enterprises adopting AI/ML workloads are generating enormous volumes of unstructured data, and tape is where a meaningful chunk of it ends up once it goes cold.

To put 176.5 exabytes in perspective against the rest of the storage industry: in early 2025 alone, hard drive vendors shipped roughly 361 exabytes in a single quarter, putting annualized HDD shipments somewhere north of 1,400 exabytes. So tape isn't winning a head-to-head capacity war against disk — it was never going to, and that's not the point. Tape occupies a specific niche (cold, durable, offline, cheap) inside a much larger storage market that IDC pegs as growing toward hundreds of zettabytes of new data created annually by the back half of this decade. The interesting story isn't "tape vs. disk." It's that the cold tier of that exploding datasphere is, by a wide margin, choosing tape — and the market backs that up too: analysts size the global tape storage market at somewhere around $5–6 billion in 2026, growing at a 7–8% CAGR through the early 2030s.

Why now, specifically

A few forces are converging at once:

HDD prices have gone the wrong way. AI demand for nearline drives has pushed enterprise HDD prices up — some consumer-facing drives are reportedly up an average of 46% since last September. For the first time in a decade, the "just buy more disk" answer is getting noticeably more expensive.
Ransomware made the air gap matter again. Roughly 90% of organizations worldwide have experienced a ransomware attack, by IDC's count, and a tape sitting in a drawer with no network or USB path is fundamentally un-attackable in a way that a "cold" but still-attached HDD is not.
AI/ML training and inference data is enormous and mostly cold. You don't need fast random access to last year's training corpus. You need it to exist cheaply, durably, and verifiably for a long time — which is exactly tape's value proposition.

None of this is hypothetical vendor spin. It's the LTO Program's own technology providers and IDC analysts saying it on the record, and the shipment numbers back it up year over year.

The actual cost case, including the part everyone leaves out

Here's a comparison most articles on this topic don't make: tape vs. HDD vs. SSD, and tape vs. cloud cold storage — because that last one is where the real surprise is.

At 2026 retail pricing, an 18TB-native LTO-9 cartridge works out to roughly $0.005/GB. A 20TB enterprise HDD lands around $0.030/GB. A 4TB NAS SSD is closer to $0.075/GB. All three of those are one-time costs — you pay once and the bytes are yours.

Cloud cold storage looks irresistibly cheap by comparison on a per-month basis. AWS S3 Glacier Deep Archive, for instance, is published at $0.00099/GB-month — a fraction of a cent. But "per month" is the catch: that's a recurring charge for as long as the data exists, not a purchase.

Run the math out and the crossover points are almost comically close. Cloud cold storage costs more, cumulatively, than an LTO-9 cartridge's entire one-time purchase price after about five months of retention. It passes the cost of an equivalent amount of enterprise HDD storage at around two and a half years. And this is before counting retrieval fees, request fees, or egress — Glacier Deep Archive in particular charges per-GB to get your own data back out, on top of a mandatory 180-day minimum storage commitment. For data you're keeping for 5, 10, or 30 years (LTO media carries a 30-year archival rating), the cloud's "low monthly price" framing quietly becomes the most expensive option on the table.

This isn't an argument against cloud storage in general — it's excellent for hot and warm data with unpredictable access patterns. It's specifically an argument that "cold archive" and "cloud" are not synonyms, and that the economics tilt hard toward owned, offline media the longer your retention window gets.

Where most tape setups go wrong

So tape is cheap, durable, and having a renaissance. Given that, why does it still have a reputation as something only enterprise IT departments bother with?

Because the tooling around it never grew up. Walk into most home labs or small shops running tape and you'll find a workflow that looks like this: write data to a cartridge, label it by hand with a Sharpie, put it on a shelf. Need a file back? Find the right tape (maybe), restore the entire archive set, then go looking for your file inside it.

Two specific approaches dominate, and both have a structural weakness that doesn't show up until you're a year or two into using them.

Tape as a pure backup target. This is the "3-2-1 rule" use case — tape as your offsite, air-gapped third copy. It's a legitimate and important use of tape, but it's also where tape's reputation problem comes from: a backup set is something you restore wholesale, in a disaster, not something you browse day to day. If tape is only ever your backup target, you've correctly captured its durability and its air-gap security, but you've left its cost-per-GB advantage completely untapped for anything except disaster recovery. You're paying for the cheapest storage medium that exists and using it like an expensive one you're afraid to touch.

LTFS as a primary archive. LTFS (Linear Tape File System) was genuinely useful when it landed — for the first time, you could mount a tape cartridge and have it behave like a USB drive, no proprietary restore software required. For an interchange format between studios or broadcast houses, it's still great. But LTFS was designed for a "write a reel, hand it to someone, they mount it" workflow. It is a transport format, not a living archive, and the cracks show fast once you try to use it as your primary cold tier:

There's no catalog. You cannot search the contents of an ejected tape without physically mounting it first.
Every cartridge is its own isolated volume. A 50-tape archive is fifty disconnected filesystems, not one coherent library.
Finding a file means knowing — or guessing — which physical cartridge it lives on.
There's no demand-loading. Applications can't transparently request a file; a human has to manually restore it first.
Metadata lives only on the tape itself. Lose or damage the cartridge, and you lose the index along with the data.

Both patterns share the same root cause: there's no persistent, off-media index that knows what's on your tapes. Without that, tape stays stuck as either a write-only insurance policy or a manual, file-by-file restore chore — and most people understandably give up and just buy more disk.

The fix: a global namespace and a catalog

The fix looks less like "better tape software" and more like "treat your tape library the way a search engine treats the web" — index everything centrally, keep the index separate from the thing it describes, and let the physical location of any given object become an implementation detail.

Concretely, that means two things working together: a global namespace (your entire collection of tapes presents as one coherent filesystem, not fifty islands) and a catalog (a persistent, queryable index of every object you've ever archived, regardless of whether the media holding it is currently online).

In a system like this, every archive operation writes a row into a small database — something like:

CREATE TABLE catalog (
    id              INTEGER PRIMARY KEY,
    file_path       TEXT NOT NULL,
    version         INTEGER NOT NULL,
    volume_uuid     TEXT NOT NULL,   -- which cartridge, identified by UUID, not /dev/nst0
    volume_offset   INTEGER NOT NULL, -- exact byte position on that volume
    payload_size    INTEGER NOT NULL,
    blake3_hash     TEXT NOT NULL,
    archived_at     DATETIME DEFAULT CURRENT_TIMESTAMP,
    custom_metadata TEXT             -- more on this below
);

A few design choices here matter more than they look like they do:

Identify volumes by UUID, not device path. /dev/nst0 changes when you move a drive to a different port. A UUID burned into the volume's header at format time doesn't. This is what lets a catalog survive hardware shuffling without breaking every restore path.

Version instead of overwrite. Tape is sequential and append-only — you literally cannot punch a hole in the middle of a cartridge to update a file in place. Treat that as a feature: every re-archive of a file becomes a new row with an incremented version, which gives you point-in-time rollback essentially for free.

Make the volumes self-describing, independent of the catalog. If every object is preceded by a small header containing its own path, hash, and metadata, the catalog becomes an index, not a dependency. Lose the database — drive failure, rm -rf, whatever — and you can rebuild it by scanning the raw tape for header magic bytes. This is the property that makes the difference between "we lost our archive" and "we lost an afternoon re-indexing it."

With that in place, search stops requiring a mounted tape drive at all:

$ archive find "project_alpha_final.mov"

Found 1 result:
  /archive/video/2024/project_alpha/project_alpha_final.mov
  Volume: 83ad72b7 | Size: 47.2 GB | Written: 2024-11-14
  Status: OFFLINE — cartridge not currently mounted

You get a full answer instantly, with zero tape movement, even though the cartridge is sitting on a shelf in a box somewhere. That's the part LTFS structurally cannot do — its index lives only on the medium it's describing.

The other half of "global namespace" is transparent access: instead of a restore being a separate job a human has to remember to run, the filesystem layer can intercept an open() call (on Linux, fanotify is the relevant kernel interface) for a file that's currently offline, pause the requesting process, and prompt for the right cartridge by UUID. The application doesn't know a tape was ever involved — from its point of view, the file was just slow to open. This is conceptually identical to enterprise Hierarchical Storage Management (HSM), minus the enterprise price tag and the dedicated hardware requirement.

Taking it further: AI-generated descriptions as searchable metadata

A catalog that indexes paths, sizes, hashes, and timestamps solves "where is this file." It doesn't solve "what's actually in this file" — and for a multi-petabyte cold archive accumulated over years, that second question is often the one you actually have.

POSIX extended attributes (xattrs) are an underused piece of infrastructure here. Linux, macOS, and most Unix filesystems let you attach arbitrary key-value metadata directly to a file's inode, independent of its contents — user.camera.operator, user.project.codename, whatever your workflow needs. If your archiving pipeline already scrapes xattrs at write time (which it should, since editorial and scientific tools frequently rely on them), then attaching one more attribute costs nothing architecturally:

# Conceptual sketch — generate a description, attach it as an xattr,
# let the existing archive pipeline pick it up like any other tag.
description = vision_model.describe(video_path)   # e.g. a multimodal model
xattr.set(video_path, "user.ai.description", description.encode())
xattr.set(video_path, "user.ai.tags", ",".join(description.tags).encode())

That's it — no new pipeline, no separate database to keep in sync. Whatever process already pulls xattrs into the catalog's custom_metadata column at archive time picks up the AI-generated description the same way it picks up a manually-set tag. A vision-language model can describe what's in a video frame; an ASR model can transcribe spoken audio into the same field; an embedding model can index the description for semantic search instead of just exact-match. None of this requires touching the underlying archive format — it's additive metadata riding along on a mechanism the system already needs for ordinary tags.

The payoff is a search experience that goes well beyond filenames:

$ archive search --semantic "drone footage of a mountain, no people visible"

Found 3 results across 2 volumes (0 tapes mounted):
  /archive/footage/2023/alps_b-roll_07.mov   (Volume a1f9, OFFLINE)
  /archive/footage/2024/denali_aerial_02.mov (Volume 83ad, OFFLINE)
  /archive/footage/2024/denali_aerial_04.mov (Volume 83ad, OFFLINE)

For media production, scientific archives, surveillance footage, or just a chaotic personal NAS, that's the difference between "I know it's in here somewhere" and actually finding it. And critically, the search itself never touches a tape drive — it's a query against the hot, SSD-backed catalog database. The tape only spins up once you actually want the bytes.

Putting it together

None of these pieces are individually exotic — xattrs have existed since the early 2000s, SQLite is older than most of us, and fanotify has been in the kernel for over a decade. What changes the picture is using them together, deliberately, instead of treating tape as an isolated dumb-storage device that occasionally gets a file copied onto it.

The pattern is: write data to the cheapest durable medium that exists, index it centrally and redundantly (catalog plus self-describing volumes), make retrieval transparent instead of a manual restore job, and use cheap, off-the-shelf AI to make the contents — not just the filenames — searchable. Tape's economics have always been good. What's been missing is software that takes that seriously enough to build real infrastructure around it instead of a Sharpie and a shelf.

You can find most of these features in an open-source, single-binary archive system in Rust called HuskHoard — and most of the concrete examples above (the catalog schema, the demand-load mechanism, the xattr capture pipeline) come directly out of that project. It's AGPL v3. Other catalogs include open metadata and Amundsen Tape is still relevant in 2026 if we apply the tools we have available in 2026 not those that were used in 2006.

Sources: LTO Program Technology Provider Companies (HPE, IBM, Quantum) annual media shipment reports, 2020–2025; IDC Global DataSphere research; AWS S3 published pricing (verified 2026); Tom's Hardware, Blocks & Files, and TechTarget reporting on tape and HDD shipment trends. Cost figures reflect 2026 retail media pricing and exclude drive amortization, power, and (for cloud) retrieval/egress fees.

Why Is Only 40% of Your Data Compressed? The Tarball Trap and How to Escape It

Joichiro Mitaka — Mon, 06 Jul 2026 07:06:25 +0000

The Uncompressed Elephant in the Room

Open a shell on your primary storage array. Run du -sh on the directories nobody has touched since 2022. Now look at your cloud bill. There's a good chance you're paying full price to store data that could be sitting in less than half the space.

Industry analysts have been saying the same thing for years: somewhere around 80% of enterprise data is unstructured — logs, CSVs, old project exports, documents, media, backups of backups. And unstructured data is exactly the kind of data that modern compression algorithms are good at. Zstd routinely gets 2:1 to 3:1 on typical unstructured payloads, which means 50-66% of that storage footprint doesn't need to exist.

So here's the question worth sitting with: if the algorithms are this good, and the financial case is this obvious, why does so much cold data sit completely uncompressed on expensive primary storage?

The answer isn't ignorance. It's fear — a very rational fear, rooted in how compression tools actually behave once you use them.

Why We Don't Compress: The Tarball Trap

Everyone has run tar czf archive.tar.gz project/ at some point, watched a directory of ten thousand files collapse into a single blob, and felt the small dread of knowing that directory is now effectively gone.

This is the core usability problem with traditional compression:

Loss of visibility. Once files are inside a tarball or zip, they vanish from the filesystem's view. No more find, no more grep -r, no more casually browsing a folder in your file manager. The files still exist, technically, but they're locked behind an opaque container.
The "I might need that" price. Someone eventually needs one 5MB config file out of that archive. To get it, they have to pull down and decompress the entire 100GB tarball — CPU time, disk I/O, and a coffee break, just to retrieve a file a fraction of a percent the size of the container.

Faced with that trade-off, most teams make the rational choice: leave it uncompressed. Storage is "cheap enough," and a working filesystem beats a compressed black box you're afraid to touch. We don't skip compression because we don't understand its value — we skip it because traditional tooling makes accessibility and compression mutually exclusive.

The Financial Cost of That Fear

Choosing not to compress isn't a one-time decision that costs you some disk space. It's a decision that gets multiplied across your entire infrastructure stack, every day, forever.

Backups. Every backup job now moves and stores the full uncompressed volume — 100TB instead of 40TB, on every cycle.
Replication. Cloud egress fees and inter-site bandwidth are spent transmitting bytes that were mostly redundant text or empty padding to begin with.
Power and cooling. Uncompressed data means more spinning drives, drawing wattage around the clock, to hold data nobody has read in three years.

None of these costs show up as a single line item labeled "we didn't compress our archive." They show up quietly, spread across every budget line that touches storage — which is exactly why they're so easy to ignore and so expensive to sustain.

The Solution: Compression That Doesn't Change the User Experience

The real fix isn't a better zip format. It's removing the trade-off entirely: what if data could be compressed at rest, but still look and behave like a normal, uncompressed file to everything above the storage layer?

That's the specific problem HuskHoard is built around. It's an open-source (AGPL v3), Rust-based tiering engine for Linux that automatically moves cold data off expensive NVMe/SSD storage onto disk, cloud buckets, or physical LTO tape — while leaving a normal-looking file behind in your directory tree.

The mechanism that makes this work is worth calling out, because it's not FUSE. FUSE-based transparent filesystems exist, but they route every single read through a userspace filesystem driver, which adds latency and a layer you now depend on for correctness. HuskHoard instead uses the Linux fanotify kernel API. When a process opens a file that's been archived, the kernel itself pauses that process, HuskHoard's daemon recalls the data, and the process resumes — with zero awareness that anything happened. ls, find, and your file manager all keep working normally, because as far as the filesystem is concerned, the file is still there.

The Technical Layer: How the Catalog and Frames Actually Work

This is the part worth understanding if you're evaluating whether this approach holds up under real workloads.

The Catalog fixes visibility. Instead of hiding archived files inside a monolithic container, HuskHoard logs every file's metadata — path, version history, byte offset on physical media, BLAKE3 hash — into a SQLite database it calls the Catalog. The filesystem keeps a lightweight stub in place of the original file, so directory listings, find, and search all keep working exactly as before. You never lose sight of what you have or where it lives; the Catalog is queryable in milliseconds even across a multi-petabyte archive, without waking up a single tape drive.

Independent Zstd frames fix the "give me 5MB out of 100GB" problem. A standard Zstd or gzip stream builds a single continuous compression window across the whole file — to read byte 90,000,000 you generally have to decompress everything before it. HuskHoard avoids this by packing archived data into independent 16MB Zstd frames when writing to cloud storage via rclone. Each frame is a self-contained compression boundary, so decompressing frame 400 doesn't require touching frames 1 through 399. This is also what keeps cloud costs down on the write side — batching into 16MB units minimizes the number of PUT requests, which matters when providers bill per-request as well as per-byte.

TLV headers carry the metadata that makes recovery possible without a working database. Every file HuskHoard archives is preceded by a 4,096-byte object header. The first 136 bytes hold the essentials — UUID, POSIX mode, compressed size, BLAKE3 hash. The remaining ~3,960 bytes are packed using Type-Length-Value (TLV) encoding, which stores POSIX extended attributes (xattrs) directly alongside the payload. That detail matters more than it sounds: if a parser encounters a TLV type it doesn't recognize, it just reads the length and skips it, which is what makes the format tolerant of future changes. It also means the archive is self-describing — if the Catalog database is ever lost, HuskHoard can rebuild it from scratch by scanning the tape or bucket and reading every object header back out.

Frame indexing is what makes partial reads actually fast. Combine the Catalog's byte-offset tracking with independent frame boundaries, and you get real range access instead of full-file extraction. HuskHoard's StreamGate HTTP gateway uses this to let you seek into massive files without downloading them wholesale — want to scrub through a 4K video sitting on S3 or a physical LTO tape, in Plex or mpv? StreamGate resolves the byte range you're asking for, fetches only the relevant frame (an HTTP Range request for cloud, a SCSI tape seek for LTO), and decompresses just that piece. Tape gets its own physics-aware handling on top of this — writes land in 256KB SCSI-aligned blocks with filemarks, specifically to avoid "shoe-shining" the drive, and the Catalog indexes those blocks the same way it indexes cloud frames.

You can see the shape of this in practice — after installing and pointing the daemon at a test volume, dropping a file into the hot tier and waiting a few seconds is enough to watch it happen:

# Drop a file into the watched hot tier
dd if=/dev/urandom of=hot_tier/dummy_data.bin bs=1M count=12

# after the janitor interval elapses...
ls -ls hot_tier        # allocated size drops to ~0, logical size unchanged
du -h hot_tier         # it's become a "husk"

# open the file normally — fanotify intercepts the read and recalls it

And streaming straight off tape or cloud without ever touching local disk:

./target/release/huskhoard cat --file-path /media/movies/scifi.mp4 | mpv -

Nothing about that command looks like archive recovery. That's the point.

Have Your Cake and Compress It Too

The underlying idea here isn't really about Zstd, or SQLite, or fanotify specifically — it's that compression shouldn't be a choice between "save money" and "keep my data usable." Those two goals only feel opposed because most tooling makes them opposed. Once compression happens at the infrastructure layer instead of as a manual, user-facing chore, the trade-off disappears.

If you want to see how the frame layout and TLV headers work in more detail, the HuskHoard GitHub repo has the full source (it's Rust, MIT-adjacent AGPL v3, and genuinely readable), and the project blog has deeper dives into specific pieces like the Catalog and the tape format.

In the meantime, it's worth an honest audit: how much of your own cold, unstructured data is sitting around uncompressed right now — not because compression wouldn't help, but because the tools made it too painful to bother?

50% of Cloud Hard Drives are Now SMR: Why Hyperscalers Love What Consumers Hate

Joichiro Mitaka — Tue, 30 Jun 2026 05:39:23 +0000

If you hang around r/DataHoarder or home lab communities, you’ve likely heard the golden rule of buying hard drives: "Avoid SMR at all costs."

The general consensus is that Shingled Magnetic Recording (SMR) drives have abysmal write speeds, fail early, and will crash your RAID arrays. Yet, if you look at the enterprise storage market today, a massive paradox emerges.

(Caption: Data from Western Digital’s Quarterly Earnings showing UltraSMR crossing 50% of nearline exabyte shipments in 2023)

The numbers don't lie. During recent earnings calls, Western Digital confirmed a massive milestone: their UltraSMR technology now makes up over 50% of their total nearline exabyte shipments. Cloud giants like Dropbox rely on SMR for over 90% of their entire storage fleet. In fact, production for high-capacity SMR drives is essentially sold out as AI and data centers buy them by the truckload.

So, SMR offers higher capacity and a lower price per terabyte. Why does everyone in the consumer and small enterprise space avoid them, and more importantly, how can you start using them like the cloud giants do?

Let's demystify SMR technology, look at how the hyperscalers bypassed its limitations, and explore how you can leverage it in your own home lab or small enterprise using modern open-source tools.

The Root of the Stigma: Drive-Managed SMR and the Physics of the Write Head

To understand the hate, you have to understand the physics of the technology.

Hard drive manufacturers eventually hit a physical wall: the magnetic write head of a hard drive is physically wider than the read head. You need a certain amount of magnetic energy to flip bits on the platter, meaning the write head can only be shrunk so much before it fails to work. However, the read head can be made incredibly tiny.

Conventional Magnetic Recording (CMR) accommodates this bulky write head by writing wide data tracks and leaving tiny gaps between them so they don't overlap.

SMR, on the other hand, exploits the tiny read head. It writes a wide track, and then writes the next track partially overlapping the first—like shingles on a roof. This leaves just enough of the previous track exposed for the smaller read head to read, allowing manufacturers to squeeze up to 25% more data onto the same physical platters.

The catch? You cannot randomly overwrite data. Because the tracks overlap, modifying a single block of data requires the drive to read a massive "zone" of data into a cache, modify it, and rewrite the entire zone sequentially.

The stigma started around 2020 when manufacturers silently slipped Drive-Managed SMR into consumer NAS drives. These drives pretended to be normal CMR drives to the host operating system, attempting to manage all that complex "read-modify-write" garbage collection internally. Under heavy, random-write NAS workloads, their caches would fill up, performance would tank to dial-up speeds, and RAID controllers would assume the drives were dead and drop them from arrays.

Consumers rightly felt burned. But the cloud providers took a completely different approach.

The Cloud's Secret Weapon: Host-Managed SMR, Aggressive Tiering, and Pin 3

Hyperscalers like Dropbox and Google didn't try to force SMR drives to act like traditional CMR drives. Instead, they completely rebuilt their storage infrastructure around three core concepts.

1. Host-Managed Zoned Storage

In a Host-Managed setup, the hard drive doesn't try to hide its shingled nature. It exposes its architecture to the operating system as a Zoned Block Device (ZBD). The drive essentially says to the host: "Here are my zones. You can only write to them sequentially. If you want to delete or overwrite something, you must reset the entire zone and start from the beginning." By shifting the responsibility from the drive's underpowered internal processor to the host machine's software, the performance penalty vanishes.

2. Aggressive Telemetry and Tiering

Rewriting the file system is only half the magic. Cloud services have immense intelligence about your files and access patterns. When you upload a file, it rarely lands directly on a spinning disk; it hits a blazing-fast NVMe cache (the "hot" tier).

However, the moment their algorithms determine that a file is no longer actively being used—say, a cloud backup from three months ago, or an old photo album—they aggressively tier that data. It is migrated off the expensive flash and funneled sequentially, block by tightly-packed block, into the zones of a high-capacity SMR drive (the "cold" tier).

3. Pin 3 Power Management (Letting the Disks Go Cold)

Once that data is packed into an SMR drive, cloud providers take it a step further: they literally pull the plug.

If you've ever "shucked" an external hard drive for your home lab, you might have run into the infamous "Pin 3 Power Disable" (PWDIS) issue, where you had to put Kapton tape over the 3rd SATA pin just to get the drive to spin up. While consumers were taping over this pin out of frustration, hyperscalers were demanding this feature by design.

The PWDIS feature allows data centers to remotely hard-reset or cut power to individual drives. When an SMR drive is full of cold data, it doesn't just sit there spinning, consuming electricity, and generating heat. It is commanded to spin down. It sits entirely dormant, acting as a high-density, rapidly accessible tape replacement. It only wakes up when an API call requests a specific dormant file—spinning up, serving the file with a few seconds of acceptable latency, and going back to sleep.

The Power of Zoned.io

If you want to dive deep into how the foundation of this works on a technical level, the ultimate resource is Zoned.io. Backed by major storage players, it documents the Linux kernel's implementation of Zoned Block Devices. Zoned Storage is becoming critical not just for SMR hard drives, but for next-generation NVMe SSDs (ZNS) that use the exact same sequential-write paradigms to reduce write amplification and extend flash memory life.

Bringing SMR to the Home Lab and Small Enterprise

You don't need to be AWS or Dropbox to take advantage of SMR pricing and density. You just need to pay attention to your workload requirements and use software designed to treat storage sequentially.

SMR is an absolute powerhouse for WORM (Write Once, Read Many) workloads:

Media archiving and Plex servers
Immutable backup targets
Log aggregation and long-term surveillance video storage
Data hoarding and cold storage

If you are using ZFS, you can utilize sequential-friendly configurations, but to truly unlock the potential of Host-Managed SMR in a small enterprise or home lab, you need purpose-built archiving tools.

The Do's and Don'ts of SMR Archiving

SMR is an absolute powerhouse for WORM (Write Once, Read Many) workloads, but only if you follow the rules.

The Don'ts: RAID, ZFS, and Standard File Systems

The biggest mistake you can make is blindly handing an SMR drive over to your operating system or RAID controller.

Don't use hardware RAID or standard ZFS. RAID arrays (and ZFS RAIDZ) rely on parity calculations and constant metadata updates. If an array needs to rebuild or "resilver", it blasts the drives with random, fragmented writes. The SMR cache will choke, write speeds will drop to practically zero, and the controller will drop the drive entirely.
Don't just format it to ext4 or NTFS. Traditional file systems are designed for random I/O. Even if you are just sequentially copying a large video file, the file system is constantly darting its write head back and forth across the platter to update the journal, timestamps, and file allocation tables. This micro-random I/O ruins SMR performance.

The Do's: Buffer, Batch, and Stream

Instead of fighting the drive with random I/O, treat it exactly like a modern tape drive.

Do use an SSD staging tier. Borrow the cloud provider's playbook. Buffer your incoming files on an NVMe or standard SSD first. Once you have accumulated a large batch of cold data, flush it to the SMR drive in one massive, continuous stream.
Do use Zoned-aware storage layers. The Linux kernel has made massive strides here. For example, btrfs now has a dedicated "Zoned Mode" explicitly designed to work with Zoned Block Devices. It forces all writes to be strictly sequential, keeping metadata organized and eliminating the read-modify-write penalty.

If you want to go deeper into SMR this is a good resource(Stop Murdering your SMR Drives), but if you want to take advantage of SMR pricing and density today, you must completely rethink your storage architecture.

Conclusion

The storage industry is moving toward Zoned Storage, whether consumers like it or not. With the massive shift toward UltraSMR in the enterprise space, the era of ignoring the technology is over.

It's time we stop universally hating SMR drives and start treating them like the specialized, high-capacity archiving tools they are. If you stop handing them directly to standard file systems and instead provide them with the buffered, sequential workloads they crave, they will reward you with massive capacity, incredible cost savings, and a much lower power bill.

Check out the documentation at Zoned.io to understand the kernel-level magic, and if you are looking for an out-of-the-box archiving solution that loves SMR drives, check out AGPL archiving at Huskhord.com.

Addressing the global namespace

Joichiro Mitaka — Thu, 25 Jun 2026 06:20:12 +0000

There is a moment in every data hoarder's life — and in every small media shop's IT history — when the first archive drive fills up. You slot in a second tape, plug in another external drive, configure a new cloud bucket. Problem solved.

Except now you have a different problem: you have no idea where anything is.

Not in the "I lost the file" sense. More in the "I know I have that file somewhere across these six volumes, but I don't know which one, and I'm not sure I haven't archived it twice, and that third copy might be the old version" sense. The data is there. The knowledge of where it is, and which copy is canonical, is not.

This is the multi-volume namespace problem. It's been lurking in storage management since reel-to-reel tape in the 1960s, and the solutions to it span from "I have a spreadsheet" to "I have a $200,000 enterprise storage cluster." Most people end up somewhere in the uncomfortable middle.

Let's look at the problem properly, put some numbers on it, and walk through what people actually do — and why each approach eventually runs out of road.

The Numbers First

"Multiple volumes" sounds modest. It isn't, once you start measuring.

LTO tape capacity by generation (native / 2.5:1 compressed):

Generation	Native	Compressed	$/TB (cartridge only, approx.)
LTO-7	6 TB	15 TB	~$0.02/GB
LTO-8	12 TB	30 TB	~$0.015/GB
LTO-9	18 TB	45 TB	~$0.012/GB
LTO-10	40 TB	100 TB	(2026, available now)

A serious homelab data hoarder with a modest LTO-8 drive will fill a 12 TB cartridge in 6–18 months depending on what they're collecting. A four-tape rotation is a realistic minimum for anyone who takes offline copies seriously. That's 48 TB native — multiple volumes by definition.

Scale up to a small production company, a university research group, or a regional broadcaster. Now you have dozens of tapes, offline drives at an off-site location, and a cloud backend for disaster recovery. Each one a separate island of storage.

Hard drive archives have the same problem. A rotating set of 14–20 TB USB drives for cold backup is cheap and practical. It's also, over five years, eight or ten drives with no shared index.

The problem isn't finding the storage. The problem is that the more storage you buy, the worse your ability to reason about it gets — unless you build something to manage the namespace across all of it.

What "Namespace" Means in This Context

A namespace is simply a unified view of your files. On a single volume, it's trivial: mount the drive, run ls, see everything. The file system is the namespace.

The moment you have two volumes, the file system isn't enough. A file might be on either one. If both are mounted simultaneously, you have two separate trees. If one is offline, its files are invisible. If the same logical file has been archived to both volumes at different points in time, you have two physical copies with no built-in way to determine which is current.

The ideal global namespace would let you:

List every file you've ever archived, regardless of which physical volume holds it
Know immediately which volume a file is on, and what its current version is
Know that you haven't accidentally archived the same content twice on different volumes
Retrieve the right version without checking six different drives one by one

No single standard delivers all four of these, which is why the solutions below exist — and why each one is only a partial answer.

Approach 1: LTFS — A File System on Every Tape

The closest thing to a proper multi-generation standard for tape namespace management is LTFS (Linear Tape File System), adopted by the LTO Consortium in 2010 and standardized by SNIA. If you're running LTO-5 or later, you can format any cartridge as LTFS and mount it exactly like a USB drive.

The way it works is elegant. LTFS divides a tape cartridge into two partitions. Partition 0 is the index — an XML document describing your directory tree, file names, timestamps, and the exact tape block where each file's data begins. Partition 1 is the data itself. When you insert an LTFS tape and run the mount command, the drive reads Partition 0 into memory, and your OS presents the tape as a browsable volume.

ltfs -o devname=0 /mnt/tape
ls /mnt/tape/projects/alpha/
# rushes/  finals/  audio/

You can drag and drop. You can cp. Files are written sequentially to the end of the data partition and the index is updated. It's a genuine open standard — an LTFS tape written on Linux can be read on macOS or Windows with compatible software, with no proprietary software in the middle.

For interchange and manual archiving, LTFS is excellent. If you're sending a tape to a collaborator or to a facility you've never worked with before, LTFS is the right choice. It is genuinely self-describing: the metadata travels with the media.

Where LTFS Stops

The problem is that LTFS solves the per-tape namespace and stops there. Each tape is its own island.

When a tape is offline — sitting on a shelf — its files are invisible. There is no LTFS spanning standard that lets you query across multiple cartridges simultaneously. If you have four tapes and need to find a file from 2022, you are mounting tapes until you find it, or maintaining a separate external record of what's where.

LTFS also inherits tape's fundamental constraint: files can only be appended, never overwritten. If you delete a file in the LTFS index, the data blocks on tape are simply marked unavailable — the space is not recovered until you reformat the entire cartridge. More importantly for long-lived archives: LTFS has no native versioning. The spec doesn't define what a version is or how to track one. If you archive final.mov, re-edit it, and archive it again to the same tape, the second write is a separate file entry in the same index. Version control becomes your problem.

The index partition itself has a ceiling — roughly 5–10% of tape capacity — which can become a bottleneck for archives with millions of small files.

For many users, these aren't dealbreakers for a single-tape workflow. But the moment you're managing a tape library, you need something sitting above LTFS to provide the cross-volume view. LTFS gives you a great building block. It doesn't give you a global namespace.

Approach 2: The Spreadsheet (And Its Cousins)

The most common approach in homelabs and small shops isn't LTFS or enterprise software. It's a spreadsheet. Or a text file. Or a Notion database. Or a Python script someone wrote in a weekend that generates a CSV of filenames and tape labels.

This approach is completely understandable and immediately comprehensible to everyone on the team. "Tape 003 has the 2023 raw footage. Tape 007 has the 2024 project deliverables. Check the sheet."

It also has a half-life of about six months.

The spreadsheet tracks what you intended to put on each tape at the moment you archived it. It doesn't automatically update when files change. It doesn't track versions — when you re-archive an updated file, do you add a new row? Update the existing one? Add a note? Different people make different decisions, and over time the spreadsheet becomes an archaeological record of past intentions rather than a live index.

Deduplication is purely manual. Nothing in the spreadsheet warns you that client_deliverable_FINAL_v3.mov on Tape 003 and client_deliverable_FINAL_v3_2.mov on Tape 009 are the same underlying content, archived twice by different team members six weeks apart.

Search is grep over a CSV or a Notion filter — workable for hundreds of files, painful for tens of thousands, and broken for millions.

The spreadsheet approach is a record of what someone thinks is on the tapes. The tapes themselves know what's actually on them. These two sources of truth diverge over time, and the divergence accelerates as the archive grows.

Some teams move from spreadsheets to proper databases — SQLite or Postgres — and write custom tooling to maintain the index. This is a genuine improvement. The index is now queryable, consistent, and maintainable. But custom tooling requires ownership: someone has to build it, update it when formats change, and ensure it stays in sync with the physical media. Most teams don't have that person, or lose them.

Approach 3: Enterprise HSM

The "correct" industrial answer to this problem is a Hierarchical Storage Management (HSM) system. HSM has existed since IBM implemented it for mainframes in 1978. The idea is straightforward: files on fast, expensive storage automatically migrate to slower, cheaper storage based on access patterns, and transparent stub files replace them so users never notice the transition.

Modern commercial HSM products — IBM Spectrum Protect, HPE Data Management Framework, Quantum StorNext — provide exactly the global namespace you need. You have a single directory tree. Files that haven't been touched in 90 days silently move to tape. Open one, and the system recalls it transparently. The catalog tracks every file, every version, every volume.

They work. They work very well. They're also built for organizations with dedicated storage administrators, SAN infrastructure, enterprise support contracts, and storage budgets in the low-to-mid six figures.

Enterprise HSM is designed for the assumption that you have a team, a budget, and a vendor relationship. The software itself can cost more than a homelab enthusiast's entire NAS. The hardware requirements (dedicated tape libraries, Fibre Channel HBAs, redundant metadata servers) further narrow the addressable audience.

For a small production company with four LTO drives and a NAS, enterprise HSM is a missile to kill a mosquito. And there is no meaningful open-source equivalent in this space — the gap between "hobbyist script" and "enterprise HSM" has historically been a cliff with nothing in the middle.

Approach 4: Backup Software (The Wrong Tool)

A common misconception: "I'll just use my backup software to manage my archive." Backup software — Veeam, Bacula, Duplicati, Restic, Amanda — is excellent at what it does, which is taking point-in-time snapshots and storing them in a recoverable format.

It is not designed for an archive, and the distinction matters.

A backup is a recovery mechanism. You restore from it when something goes wrong. The interface is: "restore system/file to state as of [date]." Backup software maintains its own catalog of backup sets, not a live representation of your file namespace.

An archive is a primary store. Files live there permanently, and you access them directly. The interface should be: "open this file" — and it should just work, regardless of which volume holds the bytes.

Backup software can hold your archival data. It will not give you a unified browsable namespace across multiple tapes. It will not transparently serve a file from tape when an application requests it. It will not tell you whether a file is already archived before you try to archive it again. These are the functions a real archive system needs to provide, and backup software isn't designed to provide them.

What the Ideal Solution Would Look Like

At this point the shape of the problem is clear, and you can start to describe what a complete solution would need to provide.

Volume identity that outlives device paths. If your LTO drive moves from /dev/nst0 to /dev/nst1 because you rearranged your HBA, every restore should still work. Volume identity needs to be based on something burned into the media at format time — a UUID — not a device path or a human-assigned label.

A catalog that is separate from any single volume. You need a central index that can answer "where is this file?" without mounting every volume in your library. The catalog holds the address (volume UUID + byte offset) for every archived file.

But also: volumes that are self-describing. The catalog is the fast path. But if you lose the catalog database, you should be able to reconstruct it from the physical media — because each volume should carry enough metadata to identify every object it contains. The catalog is an optimization, not a dependency.

A namespace that is always visible, regardless of volume state. Files that are on offline tapes should still appear in your OS. You should be able to see the file, read its metadata, and — when you try to open it — get a useful message about which volume to insert, not a confusing "file not found" error.

Hash-based deduplication, not path-based. The system should know whether a file has already been archived by its content, not just its name. Renaming a file shouldn't trigger a redundant archive.

Versioning as a first-class concept. When a file changes and is re-archived, both the old and new versions should be tracked and retrievable. The system should know that version 3 is on Tape C and version 1 is on Tape A, and serve the right one by default while making rollback possible on request.

None of the approaches above deliver all six of these properties. LTFS gets you two or three. Spreadsheets get you one on a good day. Enterprise HSM gets you all six, but only if you have an enterprise budget.

An Open-Source Answer

This gap — between hobbyist scripts and enterprise HSM — is where a relatively new open-source project called HuskHoard is trying to operate.

HuskHoard is a data-tiering engine built in Rust for Linux. It uses the fanotify kernel API to transparently stub cold files — replacing them with zero-byte placeholders while keeping them visible in your file system — and automatically recalls the data from the appropriate volume when an application opens the file. It supports LTO tape drives natively (via the SCSI tape driver), flat image files, external drives, and cloud backends via rclone.

The catalog is a SQLite database that tracks every archived file by its content hash, its version history, and its exact byte offset on a specific volume UUID — not a device path. Plug a drive into a different port, and the catalog heals the mapping automatically via a rescan. Volumes carry their own self-describing headers, so the catalog can be fully rebuilt from raw media if it's ever lost.

For multi-volume management specifically: when a needed volume is offline, HuskHoard reports the volume's UUID and the human-readable location_hint you've assigned it ("Shelf A, Slot 2 — LTO-9 Vol 001"), then waits for you to insert it. It won't re-archive a file that's already in the catalog with a matching content hash, regardless of which volume holds the original. Each volume is formatted once with a UUID, and that identity persists for the life of the media.

It's not enterprise HSM. The robotic tape library automation, the Fibre Channel fabric, the dedicated metadata servers — those aren't there. But for a homelab operator, a small production company, or any team that has more data than fits on one thing and needs a coherent answer to "where is that file?", it addresses all six requirements above as open-source software, free under AGPL v3.

The architecture notes and a deeper dive into how the catalog handles the multi-volume problem are on the HuskHoard blog if you want to go further.

The multi-volume namespace problem isn't glamorous. It doesn't show up in benchmarks or feature comparison tables. It shows up the third time you're manually mounting tapes trying to find a project file from 18 months ago, or the moment you realize you've been archiving the same directory to two different drives for six months and have no idea which copies are current.

Every solution in this space involves a catalog — some record of what is where. The question is whether that catalog is a spreadsheet you maintain manually, a per-tape index that goes dark when the tape goes offline, an enterprise product with an enterprise price tag, or something designed to sit in the middle and make this problem manageable for the rest of us.

Using Zstd Frames to Egress Partial Parquet Files

Joichiro Mitaka — Wed, 24 Jun 2026 15:26:22 +0000

Jump Tables, TLV Footers, and the Real Cost of Reading What You Don't Need

You're paying for bytes you never read.

A data engineer on a busy pipeline touches dozens of Parquet files a day: schema discovery, predicate pushdown, column pruning, metadata scrapes for a data catalog sync. In each case, the application needs maybe 200 KB of context from a file that is 4 GB on disk. Without a seekable archive format and a jump table to find the right frame, your HTTP client fetches the whole thing, and your cloud egress invoice reflects every unnecessary gigabyte.

This post quantifies the problem, then walks through how HuskHoard uses seekable Zstd frames, a per-volume jump table, and TLV-encoded footer metadata to make partial egress a first-class citizen across multi-volume archives — disk, cloud, and LTO tape alike.

The Problem, In Dollars

S3 standard egress runs $0.09/GB. GCS is $0.08/GB. Even Cloudflare R2, which is free for egress from R2 to the internet, still costs you in latency and API call count when you cannot bound the range of bytes you need.

Here is a representative read pattern for a cold analytics archive:

Operation	Bytes Needed	Bytes Fetched (naïve)	Ratio
Schema discovery	~50 KB (Parquet footer)	1–8 GB (full file)	~1:16,000
Single column scan	~200 MB (one column chunk)	4 GB (full row group)	1:20
Data catalog sync (1M files)	~50 GB (footers only)	~4 PB (full files)	1:80,000
Selective restore (1 row group)	~128 MB	4 GB	1:32

On 100 TB of cold Parquet data with $0.09/GB egress:

Full read for schema sync: 100 TB × $0.09 = $9,216
Partial read (footers only, avg 100 KB/file, 1M files): ~100 GB × $0.09 = $9.00
Savings per catalog sync: $9,207 — 99.9% reduction

Even a conservative column-scan scenario (pulling 15% of each file's bytes) cuts a $9,216 monthly read bill to $1,382. The ceiling on savings is determined entirely by how precisely you can address the bytes you actually need.

That precision is what frames and jump tables buy you.

Zstd Frames: What They Are and Why They Matter

A single .zst file produced by the standard zstd CLI is one frame. Everything inside is a single compressed stream. You have to start decompression at byte 0 to reach any byte inside.

But the Zstd spec allows a concatenation of independent frames. Each frame is a complete, self-contained unit:

[Frame 0][Frame 1][Frame 2]...[Frame N]
 ^         ^         ^           ^
 16 MB     16 MB     16 MB       partial

Every frame has a known compressed_size and decompressed_size. If you know those sizes in advance (stored in a jump table), you can seek directly to Frame N by summing the compressed sizes of frames 0 through N-1. You never decompress anything you don't need. Frame N is fetched with a single HTTP Range request, decompressed independently, and the relevant bytes are piped downstream.

This is the architectural core of HuskHoard's egress model, and it maps cleanly onto how the Parquet format itself carves up a file.

The Parquet Parallel: Row Groups as Frames

Parquet is deliberately designed for partial reads. A Parquet file contains:

Row groups — horizontal partitions of the data, each independently readable
Column chunks — vertical slices within a row group
Page headers — per-page metadata within each column chunk
Footer — the FileMetaData Thrift struct at the end of the file, containing the schema, row group offsets, column statistics, and key-value metadata. Preceded by a 4-byte footer length and terminated with the magic bytes PAR1.

A reader that wants only the footer performs two range requests: one to get the last 8 bytes (magic + footer length), and one to get the footer itself. Everything else stays on the remote. A reader that wants one column from one row group consults the footer to find the column chunk's byte offset and length, then fires a single range request.

HuskHoard's frame model mirrors this exactly, but at the archive level rather than within a single Parquet file.

HuskHoard's Implementation: Frames, the Catalog, and the Jump Table

When HuskHoard archives a file to any backend — a flat image file acting as a tape volume, a physical LTO cartridge, or an rclone cloud remote — it writes the payload as a sequence of 16 MB Zstd frames. For each frame, it records the mapping between uncompressed byte position and compressed byte position on the volume.

That mapping is the object_frames table in husk_catalog.db:

CREATE TABLE IF NOT EXISTS object_frames (
    file_path           TEXT    NOT NULL,
    version             INTEGER NOT NULL,
    uncompressed_offset INTEGER NOT NULL,   -- where this frame starts in the original file
    compressed_offset   INTEGER NOT NULL,   -- where this frame starts on the storage volume
    compressed_size     INTEGER NOT NULL    -- how many bytes to fetch from the volume
);

CREATE INDEX IF NOT EXISTS idx_frames
    ON object_frames (file_path, version);

This is the jump table. Given a byte range request for bytes=2147483648-2281701376 (a 128 MB window starting at the 2 GB mark), the gateway does:

SELECT compressed_offset, compressed_size
FROM   object_frames
WHERE  file_path = '/warehouse/events/2024-01-01.parquet'
  AND  version   = (SELECT MAX(version) FROM object_frames WHERE file_path = ...)
  AND  uncompressed_offset <= 2147483648
ORDER  BY uncompressed_offset DESC
LIMIT  1;

One row. One seek. One range request against the volume. Everything else stays dark.

The HTTP gateway loop in StreamGate:

HTTP Range request arrives (bytes=X-Y)
        │
        ▼
Query object_frames → nearest frame boundary ≤ X
        │
        ▼
Seek to compressed_offset on volume (tape block, S3 range, local seek)
        │
        ▼
Decompress forward to exact byte X, stream through Y
        │
        ▼
Client receives exactly what it asked for

For a 4K video file seeking to the 2-hour mark, this is why mpv can start playing from tape or S3 in under a second instead of waiting for a multi-gigabyte download.

TLV Footers: Turning the Frame Header Into a Parquet-Style Catalog Entry

Every file archived by HuskHoard is preceded on the storage volume by a strict 4,096-byte ObjectHeader. The first 136 bytes carry the fixed-width mechanics: a magic string (USTDHUSK), the file's UUID, POSIX permissions, BLAKE3 hash, compressed and uncompressed sizes, and a CRC32 of the header itself.

The remaining 3,960 bytes are dedicated to TLV (Type-Length-Value) encoded metadata — the same binary framing used in X.509 certificates, SNMP, and dozens of wire protocols chosen specifically because unknown type codes can be safely skipped by any forward-compatible parser.

Byte  0 –  7: Magic "USTDHUSK"
Byte  8 – 23: Volume UUID (16 bytes)
Byte 24 – 55: BLAKE3 hash (32 bytes)
Byte 56 – 63: Uncompressed payload size
Byte 64 – 71: Compressed payload size
Byte 72 – 79: Original mtime
Byte 80 – 83: POSIX mode
Byte 84 – 87: Header CRC32
Byte 88 –135: File path (null-terminated, 48 bytes max inline)
Byte 136–4095: TLV region (3,960 bytes)

A TLV tag entry looks like:

[Type: u8][Key-Length: u16][Key: bytes][Value-Length: u32][Value: bytes]

This is where the Parquet footer analogy becomes structural rather than metaphorical. For a Parquet file being archived, HuskHoard can embed the Parquet FileMetaData statistics directly into this TLV region:

TLV Type	Key	Value
`0x02`	`parquet.schema`	Serialized Thrift schema (JSON or binary)
`0x02`	`parquet.row_count`	Total row count as little-endian u64
`0x02`	`parquet.col.event_ts.min`	Minimum value of `event_ts` column
`0x02`	`parquet.col.event_ts.max`	Maximum value of `event_ts` column
`0x02`	`parquet.col.user_id.null_count`	Null count for `user_id`
`0x02`	`parquet.row_group.count`	Number of row groups
`0x01`	`workflow.pipeline`	`"ingest_v3"` — POSIX xattr from source
`0x01`	`workflow.owner`	`"data-eng-team"`

These statistics travel physically bonded to the data on every storage medium — disk image, tape cartridge, S3 object. If the SQLite catalog is lost, husk rebuild walks the volume, reads every 4 KB header, and reconstructs the catalog complete with all column statistics. The tape is entirely self-describing.

But the real payoff is what this enables while the catalog is present.

Multi-Volume Catalog Queries: Pruning at the Volume Level

A production HuskHoard deployment might span several volumes:

Volume A (tape, 12 TB) — archive 2022–2023
Volume B (tape, 12 TB) — archive 2023–2024
Volume C (NVMe image, 2 TB) — archive 2024–present
Volume D (S3:us-east-1, 50 TB) — cloud replica

The catalog table records which volume holds each archived version of each file:

SELECT
    c.file_path,
    c.tape_uuid,          -- identifies the volume
    c.tape_offset,        -- byte offset of the ObjectHeader on that volume
    c.payload_size,
    c.compressed_size,
    c.custom_metadata     -- mirrors the TLV tags as JSON
FROM   catalog c
WHERE  json_extract(c.custom_metadata, '$.parquet.col.event_ts.min') >= '2024-01-01'
  AND  json_extract(c.custom_metadata, '$.parquet.col.event_ts.max') <= '2024-03-31'
  AND  json_extract(c.custom_metadata, '$.parquet.row_count')        > 0;

This query executes in milliseconds against the SQLite catalog on your SSD. The tape drives stay spun down. S3 is never contacted. You get back a list of (file_path, tape_uuid, tape_offset) tuples — the exact volumes and positions to touch.

Then, per file, for each column you actually need:

SELECT compressed_offset, compressed_size
FROM   object_frames
WHERE  file_path           = '/warehouse/events/2024-01-15.parquet'
  AND  version             = 3
  AND  uncompressed_offset BETWEEN :col_chunk_start AND :col_chunk_end
ORDER  BY uncompressed_offset;

You issue a range request for only those frames. For a 4 GB Parquet file where you need one 200 MB column chunk:

Step	Data Transferred
Catalog query (SQLite, local)	0 bytes egress
object_frames lookup (SQLite, local)	0 bytes egress
HTTP Range to S3 (compressed frame bytes)	~85 MB (at 2.4:1 Zstd ratio)
Total vs naïve full-file fetch	85 MB vs 1.7 GB

That is a 95% reduction on a per-query basis.

Putting Numbers to the Savings

Let's use a concrete scenario: a data team maintains a 10 TB cold Parquet archive on S3, with an average file size of 4 GB. They run three workloads:

Workload A — Nightly catalog sync (schema + statistics only)

Files: 2,500 Parquet files
Data needed per file: footer only (~150 KB each)
Total needed: ~375 MB
Full-file cost: 10 TB × $0.09 = $921.60/month
Partial-frame cost: 375 MB × $0.09 = $0.03/month
Monthly savings: $921.57

Workload B — Ad-hoc column scan (one column across 20% of files)

Files queried: 500 (selected by TLV statistics predicate)
Column chunk per file: ~200 MB uncompressed → ~85 MB compressed frames
Total fetched: ~42.5 GB
Full-file cost: 500 × 4 GB × $0.09 = $180.00/query
Partial-frame cost: 42.5 GB × $0.09 = $3.83/query
Per-query savings: $176.17 (97.9%)

Workload C — Point-in-time restore of a single row group

1 file × 1 row group = 128 MB uncompressed → ~54 MB compressed
Full-file cost: 4 GB × $0.09 = $0.36
Partial-frame cost: 54 MB × $0.09 = $0.005
Per-restore savings: $0.355 (98.6%)

At scale, Workload A alone — a nightly catalog sync that most teams run without thinking about the bill — generates ~$11,000/year in unnecessary egress on a 10 TB archive. The frame-indexed approach reduces that to under $1/year.

The Self-Describing Volume: Your Catalog Backup Is on the Tape

One underappreciated consequence of storing TLV column statistics in every ObjectHeader is that the volume itself becomes a data catalog. After a complete disaster recovery (catalog database lost, fresh server), husk rebuild walks the storage volume 4 KB at a time, reads every ObjectHeader, validates the CRC32, and inserts a new catalog row including all TLV-encoded metadata — column statistics, schema, pipeline tags, everything.

The catalog is not a separate system that the archive depends on. The catalog is a cache that accelerates access to information already encoded in the archive itself. This is the same philosophical commitment Parquet makes: the footer is not a separate sidecar file; it is part of the format.

For teams integrating with external data catalogs (Apache Atlas, Hive Metastore, Unity Catalog), this means HuskHoard can emit catalog events on husk rebuild just as well as on initial archive — the metadata survives the worst failure scenario, format-native.

Wiring It Up: What a Partial Read Looks Like End-to-End

A data engineer's dbt model lands at the StreamGate HTTP gateway with a Range: bytes=536870912-671088640 request (512 MB – 640 MB, pulling a specific row group from a 4 GB Parquet file on S3):

1. GET http://localhost:8080/v1/stream/warehouse/events/2024-01-15.parquet
   Range: bytes=536870912-671088640

2. Gateway queries object_frames:
   → nearest frame boundary ≤ 536870912 is at uncompressed_offset=536870912
   → compressed_offset=225,978,112 on Volume D (S3:us-east-1)
   → 6 frames needed, compressed total = 56.3 MB

3. Gateway fires:
   GET s3://huskhoard-cold/volume-d.img
   Range: bytes=225978112-285884415

4. Gateway decompresses frames on the fly, streams bytes 536870912–671088640
   to the client.

5. Total egress from S3: 56.3 MB
   Total egress if client had fetched the full file: 1.71 GB
   Savings: 96.7%
   Time to first byte (LAN): ~180ms vs ~14s for full-file download

The client — dbt, Spark, DuckDB, curl, whatever — receives a standard HTTP 206 Partial Content response. No special client library. No SDK. Just the HTTP Range spec, universally supported.

Practical Takeaways for Data Engineers

1. Frame size is a tuning knob. HuskHoard defaults to 16 MB frames, optimized for cloud PUT cost (fewer, larger requests) and Zstd compression ratio. For workloads with very fine-grained access patterns (column-level reads in narrow schemas), smaller frames (1–4 MB) reduce the minimum fetch size at the cost of more catalog rows and higher PUT count. Benchmark against your actual access patterns.

2. TLV statistics are opt-in per file type. For video files you probably don't store column min/max values. For Parquet, CSV, and Arrow IPC files it's worth paying the archiver CPU time to extract and embed statistics at archive time — you pay once and recoup every time a catalog query avoids a volume read.

3. The catalog query is your explain plan. Before a restore or a scan, husk catalog query --path "/warehouse/events/*.parquet" --filter "parquet.col.event_ts.min >= 2024-01-01" shows you which volumes and frame ranges will be touched. Run it first. If the egress estimate is unexpected, the TLV coverage on those files probably needs improving.

4. Multi-volume means cross-volume pruning is free. A query that touches two volumes and skips three is doing volume-level predicate pushdown before any I/O. The catalog does this automatically based on the tape_uuid in each matching row.

5. Egress savings compound with replication. HuskHoard replicates to multiple volumes simultaneously. If your primary volume is on S3 ($0.09/GB egress) and your replica is on Cloudflare R2 ($0.00 egress), the gateway can route the range request to whichever backend minimizes cost. Partial reads from R2 are free. You still benefit from the jump table because API call count and latency still matter.

# Why I Bypassed FUSE: Building a Transparent DataTiering Engine in Rust

Joichiro Mitaka — Fri, 19 Jun 2026 05:51:33 +0000

If you run a home lab or manage large datasets, you’ve hit this wall: NVMe drives are fast but too expensive to hoard data on. Hard drives or cloud buckets are cheap, but they are slow and a pain to manage manually.

The enterprise world solves this with HSM Hierarchical Storage Management automatically shuffling cold data to slow storage while keeping a transparent stub on the fast drive. But enterprise HSMs cost thousands of dollars and lock your data in proprietary black boxes.

I wanted this for Linux, for free. So, I started building HuskHoard, an opensource data tiering engine.

My first thought, like almost every Linux developer building a virtual filesystem, was to use FUSE. But I quickly realized FUSE was the wrong tool for the job. Here is why I abandoned it, and how I used the Linux fanotify API and Rust to build a transparent, zero overhead archiving engine.

The Problem with FUSE

FUSE is fantastic for creating custom filesystems like SSHFS or mounting an S3 bucket. But for an HSM, it creates a massive bottleneck.

When you use FUSE, every single read and write has to go through a context switch:
Application > Kernel > FUSE Daemon Userspace > Kernel > Physical Drive.

If 90% of your data is Hot actively being used on your fast NVMe, forcing it through FUSE overhead completely defeats the purpose of buying expensive NVMe drives in the first place. You sacrifice native I/O performance just to manage the 10% of Cold data.

I needed a solution where the Hot data ran at native speed, touching nothing but the XFS/Ext4 kernel drivers.

The Solution: Enter fanotify

Instead of intercepting every transaction via FUSE, I realized I only needed to intervene in one specific scenario: When a user tries to open a file that has been archived.

Linux has a kernel API called fanotify originally designed for antivirus scanners. It allows a userspace program to monitor a mount point and, crucially, block an application from opening a file until the daemon says it’s okay.

Here is how HuskHoard uses fanotify to create transparent tiering:

The Janitor: A background Rust thread scans my NVMe drive. When it finds a file that hasnt been touched in 30 days, it compresses it Zstd and moves the payload to a cheap HDD, LTO Tape, or S3 bucket.
The Husk Stub: It leaves the original file on the NVMe drive but truncates its allocated size to 0 bytes creating a sparse file. To the OS and the user, the file still looks like it’s 50GB and sits in /home/movies.
The Interceptor: This is where fanotify shines. The HuskHoard daemon listens for FAN_ACCESS_PERM events. If VLC media player tries to open that Husk file, fanotify pauses VLCs execution in the kernel.
The Recall: HuskHoard intercepts the request, streams the 50GB payload from the tape/S3 bucket back into the sparse file on the NVMe, and then tells fanotify to allow VLC to proceed.

VLC thinks it just opened a local file. It has no idea the data was fetched from an S3 bucket 50 milliseconds ago.

The Rust Implementation

Rust was the obvious choice for this. When you are blocking kernellevel I/O requests, memory safety and predictable latency are nonnegotiable.

Handling the fanotify loop requires a few specific Linux capabilities specifically CAP_SYS_ADMIN, but Rust allows us to safely manage the multithreaded heavy lifting of the Archive Worker.

pub fn run_interceptor(config: Arc<HuskConfig>, use_direct_io: bool) -> std::io::Result<()> {
    let watch_dir = &config.hot_tier;
    let db_path = &config.db_path;
    info!("\n[Daemon] Starting fanotify interceptor on '{}'...", watch_dir);
    let abs_dir = std::fs::canonicalize(watch_dir)?;

    let fan_fd = unsafe {
        libc::fanotify_init(libc::FAN_CLASS_PRE_CONTENT, libc::O_RDWR as u32)
    };
    if fan_fd < 0 { 
        let err = std::io::Error::last_os_error();
        error!(" fanotify_init failed: {}. Missing Root or Capabilities!", err);
        return Err(err); 
    }


        let mark_mask = libc::FAN_ACCESS_PERM | libc::FAN_CLOSE_WRITE | libc::FAN_EVENT_ON_CHILD;

        // 1. Recursively mark the root watch directory and all current subdirectories
        info!("[Daemon]  Scanning and attaching listeners to all subdirectories...");
        mark_directory_recursive(fan_fd, &abs_dir, mark_mask, &config);

Escaping Vendor Lockin The Easy Exit Promise

One of the biggest issues with commercial HSMs is that if the daemon dies, your data is gone, trapped in proprietary metadata.

Because I was building this for the opensource community, I enforced a strict Easy Exit architecture:

Payload data is stored in standard Zstd streams verified by BLAKE3.
The catalog metadata the Brain tracking where the cold bytes live is an SQLite database.
You can natively export the entire catalog to Apache Parquet.

This means if you decide to stop using HuskHoard, you dont need my software to get your data back. You can query your catalog with DuckDB or Python and manually extract your Zstd archives.

Whats Next?

Building HuskHoard has been a massive deepdive into Linux kernel APIs and SCSI Tape drivers. Yes, it natively supports physical LTO drives via /dev/nstX to prevent tape shoeshining.

The engine currently supports automated replication across local drives, tapes, and rclone supported cloud buckets.

If you are a Rust developer, a HomeLab data hoarder, or just interested in Linux storage architecture, Id love your feedback or code reviews. It is fully AGPL v3 licensed.

Check out the repo here: [GitHub HuskHoard]https://github.com/huskhoard/huskhoard
More architecture details: [HuskHoard Blog]https://www.huskhoard.com/blog.html

DEV Community: Joichiro Mitaka

Point-in-Time Recovery Without the Enterprise Price Tag

Point-in-Time Recovery Without the Enterprise Price Tag

Versioning vs Backup: Stop Using the Words Interchangeably

How HuskHoard's Versioning Engine Works

The Append-Only Storage Model

The Catalog Schema — The Version Column

Rolling Back a Specific Version

Under the Hood: The Mechanics of a Byte-Level Rollback

Step 1: Catalog Lookup

Step 2: Volume Resolution

Step 3: Seek to the Exact Byte Offset

Step 4: Decompress and Verify

Step 5: Atomic Placement

The object_frames Table: Partial-Content Rollback Without Full Recall

Real-World Strategies

Strategy 1: File Versioning for Granular DR

Strategy 2: VM Backup + File Versioning for Faster DR

Strategy 3: Defending Against Ransomware

Pruning Old Versions

What Enterprise Vendors Charge for This

Getting Started

Summary

Why You Can't Just Mount an S3 Bucket and Start Editing

The naive approach: mount and download

The basic building block: transparent stubbing

Intercepting reads: FUSE vs. fanotify

The other bottleneck: just listing files

Putting the layers together

Where HuskHoard fits

It's not only cloud

Tape Has Come Back From the Dead (It Was Never Dead)

The numbers don't lie

Why now, specifically

The actual cost case, including the part everyone leaves out

Where most tape setups go wrong

The fix: a global namespace and a catalog

Taking it further: AI-generated descriptions as searchable metadata

Putting it together

Why Is Only 40% of Your Data Compressed? The Tarball Trap and How to Escape It

The Uncompressed Elephant in the Room

Why We Don't Compress: The Tarball Trap

The Financial Cost of That Fear

The Solution: Compression That Doesn't Change the User Experience

The Technical Layer: How the Catalog and Frames Actually Work

Have Your Cake and Compress It Too

50% of Cloud Hard Drives are Now SMR: Why Hyperscalers Love What Consumers Hate

The Root of the Stigma: Drive-Managed SMR and the Physics of the Write Head

The Cloud's Secret Weapon: Host-Managed SMR, Aggressive Tiering, and Pin 3

1. Host-Managed Zoned Storage

2. Aggressive Telemetry and Tiering

3. Pin 3 Power Management (Letting the Disks Go Cold)

The Power of Zoned.io

Bringing SMR to the Home Lab and Small Enterprise

The Do's and Don'ts of SMR Archiving

The Don'ts: RAID, ZFS, and Standard File Systems

The Do's: Buffer, Batch, and Stream

Conclusion

Addressing the global namespace

The Numbers First

What "Namespace" Means in This Context

Approach 1: LTFS — A File System on Every Tape

Where LTFS Stops

Approach 2: The Spreadsheet (And Its Cousins)

Approach 3: Enterprise HSM

Approach 4: Backup Software (The Wrong Tool)

What the Ideal Solution Would Look Like

An Open-Source Answer

Using Zstd Frames to Egress Partial Parquet Files

Jump Tables, TLV Footers, and the Real Cost of Reading What You Don't Need

The Problem, In Dollars

Zstd Frames: What They Are and Why They Matter

The Parquet Parallel: Row Groups as Frames

HuskHoard's Implementation: Frames, the Catalog, and the Jump Table

TLV Footers: Turning the Frame Header Into a Parquet-Style Catalog Entry

Multi-Volume Catalog Queries: Pruning at the Volume Level

Putting Numbers to the Savings

The Self-Describing Volume: Your Catalog Backup Is on the Tape

Wiring It Up: What a Partial Read Looks Like End-to-End

Practical Takeaways for Data Engineers

The `object_frames` Table: Partial-Content Rollback Without Full Recall