How Local Image Libraries Really Work: File Layout, Metadata Sidecars, and the Export Pipeline

If you collect images professionally — for design, ecommerce, or content — browser bookmarks stop working after a few hundred items. Tabs get closed, CDN links rot, thumbnails expire, and a "saved" image is often just a URL you no longer trust. That's why serious collectors graduate to local asset management tools like Eagle or Billfish. This post looks at what actually happens inside those tools, and why getting images into them from a browser is a harder engineering problem than it sounds.

Part 1: Inside a local asset library

A digital asset manager (DAM) for images is, at its core, a file system plus a metadata index. Take Eagle's layout as a concrete example: the library is a folder (typically eagle.library), and inside it the organization is deliberately simple.

Blob store. Every imported image is copied into a files/ directory, renamed to a generated ID (something like 20231018-abcdef123456.png). The tool owns the filename; the human's original filename is stored as metadata, not on disk.

Metadata sidecars. Alongside each blob, a small JSON document records the useful facts — original filename, source URL, tags, annotations, star rating, timestamp. Storing metadata beside the file, rather than only inside a database, makes the library portable and resilient: copy the folder and everything travels with it, and a corrupt database doesn't destroy the assets.

Thumbnails and previews. Separate thumbnail files are generated at import so the UI can render grid views without decoding full-resolution JPEGs hundreds of times.

Search index. A local database aggregates the sidecar metadata for fast tag/filename queries.

Why this design? A single giant database is a single point of failure and hostile to syncing. File-plus-sidecar means every asset is self-describing, diffable, and recoverable even if the index is lost. The same pattern appears in Billfish and in photo managers like DigiKam — the file is the source of truth, metadata rides along.

Part 2: The export pipeline problem

The hard part isn't the library format — it's getting images from a web page into that format correctly. A naive pipeline (grab every <img src>, download, dump into a folder) fails in four predictable ways:

1. Lazy loading. Modern pages hide real URLs in data-src, data-original, or srcset and only load them on scroll. A crawler that reads only src gets placeholders and tracking pixels.
2. CDN thumbnails. Image CDNs (Aliyun OSS, Tencent COS, Qiniu) serve resized thumbnails with processing parameters in the URL — ?x-oss-process=image/resize,w_200, imageMogr2/thumbnail/200x200, imageView2/2/w/200. Saving the thumbnail defeats the purpose; the original is often one parameter-strip away, but writing that recovery logic for three different CDN syntaxes is fiddly.
3. Duplicates. The same hero image appears on 40 article pages. Without detection you import 40 copies, waste disk space, and pollute the library's search-by-color/tag.
4. Collisions and filenames. Two sites can both ship logo.png, and Windows-incompatible characters (, : " * ?) silently break exports.

So the correct pipeline has stages: discover (walk the DOM, union all URL sources, normalize), fetch (download with dedup by content hash or pHash), transform (recover originals from CDN parameters), and commit (write blob + sidecar JSON, generate thumbnails, update the index). Each stage is its own project — which is why most people never build it and settle for right-click-save, one image at a time.

Part 3: Scale is where it breaks

Add scale and a fifth failure appears: memory. Collecting a serious reference board might mean 2,000+ images. A tool that buffers every file in RAM and then zips them will freeze the page or crash the tab. The fix is streaming: pipe each response body through a transform stream straight into the archive writer, never materializing the whole byte array, and let backpressure pause downloads when the pipeline is saturated. With a streaming ZIP, 2,000 images export without the UI ever blocking — but implementing streaming ZIP writers, CDN recovery, and pHash dedup correctly takes weeks of edge-case handling.

Part 4: What a finished pipeline looks like

All of the above — discover, dedup, recover originals, stream, and write into the library — is exactly the plumbing that OmniPic AI Studio Pro bundles into a single browser extension. One click inspects the entire site: images are crawled from lazy-load attributes and srcset, classified by type, deduplicated with pHash, and CDN thumbnails are resolved back to their originals where the CDN rules allow it. The results export directly to Eagle or Billfish with proper metadata sidecars, or as a streaming ZIP of 2,000+ images without locking up the tab. Everything runs 100% locally — nothing is uploaded, which matters when you're collecting mood boards from a client's private site.

Available on Chrome, Edge, and Firefox:

Get OmniPic AI Studio Pro: https://chromewebstore.google.com/detail/omnipic-studio-pro/cokoeaaaglgcljahaffjeijomhmkljlc