Why You Can't Just Mount an S3 Bucket and Start Editing

Everyone who's tried pointing an NLE at a mounted S3 bucket has hit the same wall. Browsing folders is slow. Scrubbing stutters or hangs. The usual fix people reach for is downloading the whole file before touching it, which defeats the entire point of using cheap, scalable object storage in the first place.

The reason is simple once you see it: object storage — S3, MinIO, Ceph RGW, TrueNAS's S3-compatible gateway — isn't a filesystem. There's no directory tree, no stat(), no partial reads in the POSIX sense. Everything is a flat key with a GET and a PUT. Any workflow that wants to browse and edit media stored this way has to build a filesystem-shaped illusion on top of an object-shaped reality.

Tools like LucidLink made that illusion feel seamless enough that people forget it's an illusion at all. But the trick isn't magic, and it isn't proprietary to any one product — it's a handful of well-understood pieces working together. Worth breaking those apart before looking at any specific implementation.

The naive approach: mount and download

The simplest way to make an S3 bucket look like a drive is a FUSE mount (s3fs, goofys, rclone mount, etc.). These translate filesystem calls into API calls in real time. That works fine for sequential access — copying a file, reading a log — but post-production workloads are not sequential. Scrubbing a timeline, seeking to a mark, or letting a codec read its index atoms means constant, unpredictable jumps around a file. Each of those can turn into a separate HTTP request under a naive FUSE translation layer, and latency stacks up fast.

The common workaround is to just download the whole file before handing it to the application — effectively a cache-on-open. It works, but it reintroduces the thing object storage was supposed to let you avoid: waiting for a full copy before you can touch a frame.

The basic building block: transparent stubbing

Before getting to how bytes get fetched, it's worth being explicit about what the filesystem shows the application in the meantime — this is the part most write-ups skip, and it's the actual foundation the rest of the system sits on.

A stub (also called a placeholder, or "smart" file in the OneDrive/Dropbox world) is a filesystem entry that reports correct, fully-formed metadata — file size, timestamps, permissions — without the underlying data being present locally. ls, stat, project panels, media browsers, and autosave routines all see what looks like a complete, correctly-sized file. Only when something actually tries to read content does the system need to do anything.

This is the same underlying idea as OneDrive Files On-Demand or Dropbox Smart Sync, but post-production adds a wrinkle: those consumer implementations generally hydrate a file in full on first touch. A stub used for editorial has to support partial hydration — serving byte range 40MB–41MB of a 40GB file without pulling the other 39.999GB. That distinction is what separates a "smart sync" folder from something you can scrub a 4K timeline against.

Two things a stubbing layer needs to get right:

• Correct size reporting from creation. If an NLE or transcoder trusts stat() size and the stub lies about it, seeks past the "real" data will behave unpredictably.
• Read interception at the right layer. Something has to sit between "application calls read()" and "bytes are actually available," decide what's missing, fetch it, and hand it back — ideally without the application knowing this happened at all.

That second point is where implementations diverge.

Intercepting reads: FUSE vs. fanotify

FUSE handles this by owning the entire filesystem — every syscall against the mount routes through a userspace daemon. It's flexible but has real overhead, since even reads that could be served straight from a local page cache round-trip through the FUSE protocol.

An alternative is Linux's fanotify API, which lets a userspace process subscribe to filesystem events — opens, accesses, and, with permission events, the ability to block a read until the daemon signals it's safe to proceed. Instead of intermediating the entire filesystem, fanotify sits over a real (or virtual) filesystem and only intervenes when it needs to: on open, or on access to a range that isn't populated yet. Once data is present, subsequent reads can be served by the kernel's normal page cache with no daemon involvement.

Practically, that means:

1. Application opens a file → daemon sees the event, resolves it against the object store, ensures a stub exists locally.
2. Application reads offset X → if that range isn't hydrated, the daemon fetches exactly that byte range from the backend (an S3 GET with a Range header, or equivalent) and writes it into the local stub before releasing the read.
3. Application reads an already-hydrated offset → served locally, no network round trip.

This is closer to how a block device with a remote backing store behaves than a download manager. It also means media stays in its native container/codec — no re-chunking or transcoding is required to make this work, which matters if you don't want to convert an existing archive to participate.

The trade-off is real: fanotify permission events typically need elevated privileges, and the very first touch of a cold region always costs a network round trip no matter how the interception is done. Nothing eliminates the physics of "the bytes have to come from somewhere the first time."

The other bottleneck: just listing files

Byte-range fetching solves reading data. It doesn't solve finding out what data exists. Object stores don't have real directories — list-objects calls against a prefix are how "folders" get faked, and at scale (media libraries commonly run into the millions of objects) those list calls get slow, especially with deep, nested project structures. This is the part of cloud-mount tools that often feels worse than the streaming itself: opening a folder shouldn't take longer than opening the file inside it.

The usual fix is a local metadata catalog — commonly something as simple as SQLite — that mirrors the object store's namespace and metadata (paths, sizes, timestamps, hydration state) locally. Directory listings, stat() calls, and search then hit the local database instead of the network. The catalog needs to stay in sync with the backend (via polling, event notifications, or explicit indexing passes), but once it does, browsing a library feels local regardless of library size, because it is local — only the file contents are remote.

Putting the layers together

Strip away branding and this pattern generally has three layers:

1. A local metadata catalog — fast namespace and stat() info, decoupled from the backend's list performance.
2. Transparent, partially-hydratable stub files — so applications see a complete, correctly-sized tree immediately.
3. Demand-driven byte-range fetching — via fanotify, FUSE, or another interception method — that pulls only the ranges actually requested, on the fly, in native format.

None of these is exotic on its own. The interesting engineering is in making them consistent with each other under real editorial load: concurrent readers, scrubbing, proxy generation, and multiple users hitting the same objects from different machines.

Where HuskHoard fits

HuskHoard is a project that implements this pattern, aimed at people who want it running against storage they already control rather than a hosted service. It uses a local SQLite catalog for directory/metadata performance, fanotify-based interception for read handling, and treats the backend as swappable — S3, MinIO, Ceph RGW, or TrueNAS's S3 gateway all work the same way from the client side, since the client only needs an S3-compatible API to talk to.

It's not only cloud

The three-layer pattern above doesn't actually require the slow, non-randomly-addressable backend to be a network API. It applies just as well when the backend is a local device with the same basic problem: sequential, high-latency access that punishes naive random reads.

LTO tape is the obvious case — real seek latency, strictly sequential media, no benefit from re-reading small ranges the way a disk cache would help. SMR (shingled) hard drives have their own version of it: random writes into an already-shingled zone are drastically slower than sequential writes, because the drive has to read-modify-rewrite an entire shingle band. Both are cheap-per-terabyte storage that behaves badly under an editorial access pattern, which is exactly the problem the catalog/stub/interceptor stack was built to solve in the first place.

HuskHoard runs that same stack against local media directly, rather than only against S3-style APIs:

A native SCSI tape driver handles LTO-5 through LTO-9, tracking filemark positions and 256KB block alignment so the catalog can seek a file to within seconds instead of doing a linear tape rewind (and avoiding "shoe-shining" — the repeated back-and-forth tape motion that happens when a drive is fed data slower than its streaming speed).
SMR drives get a strict log-structured write path, so data is only ever appended sequentially and nothing writes back into an already-written shingle zone.
A local HTTP gateway (StreamGate) exposes the same indexed byte-range seeking whether the file lives on tape, an SMR archive drive, or S3 — so a player like mpv, ffmpeg, Plex, or Jellyfin can scrub into a multi-terabyte file on a tape shelf the same way it would scrub a file on cloud storage, without staging the whole thing to disk first.

That's the actual differentiator worth calling out: this isn't "S3 but with a nicer client." The stubbing-and-intercept model isn't tied to object storage as a concept — it works anywhere data can be indexed and fetched by range, which turns out to include a tape autoloader as much as it includes a cloud bucket. It's the LucidLink-style instant-access experience pointed at whatever's actually holding the cold data, network-attached or not.

It's not doing anything conceptually different from the LucidLink-style tools people on this forum already know — the value proposition is that it's self-hosted against infrastructure you own, rather than a subscription tied to a vendor's storage backend. Worth a look if you're already running S3-compatible storage in-house and want the streaming-editorial workflow without adding a hosted middleman

GitHub.com/huskhoard/huskhoard