My disk hit 100% full on a Tuesday night while pack_product.py was writing the 39th product zip. The script died mid-write, the zip was corrupt, and I had to delete 2 GB of raw render previews just to get the shell back. That was the moment I stopped treating backups as an afterthought and built a real archive routine. Here is what actually works when your asset library is bigger than the disk it lives on.
Zip first, copy later
The obvious fix is "buy a bigger disk" or "sync to cloud". Both cost money I did not want to spend, and both fail the same way: they copy everything, including the 300 MB of intermediate renders nobody needs. The fix was to zip each finished product folder before it leaves the machine. A finished pack of 30 vectors at around 22 MB compresses to roughly 18 MB, but the real win is that a zip is one file with one checksum. When a cloud sync or USB copy dies halfway, you know it immediately, instead of discovering a truncated JPG when a buyer complains.
The kit's pack_product.py already writes my-product.zip plus a 1280×720 cover and a 600×600 thumbnail into a _system folder. That zip is the unit of backup. One product, one file, one line in a manifest.
Never trust a bare copy
I used to cp -r packs/ /mnt/usb/ and call it done. Then I found a pack folder that had been copied while pack_product.py was still writing it. The zip inside was fine, the cover was a zero-byte file. So now every archive gets a checksum written next to it:
sha256sum _system/my-product.zip >> manifests/my-product.sha256
Verifying a backup is then one command per drive, not a manual hunt. This matters more the bigger the library gets. My full output was 856 MB across 39 products, small by stock-library standards, but even that size produced two silent corruptions before I added checksums. At thousands of assets you will not find those by eye.
Zip streaming on a small box
The whole pipeline runs on 1 vCPU and 2 GB RAM, no GPU, so compression settings matter. zipfile with ZIP_DEFLATED at default level is fine for text-heavy products like templates and PDFs. For JPGs and PNGs, do not bother: those formats are already compressed and deflating them burns CPU for maybe 2% savings. My rule is simple, deflate everything, accept that image packs barely shrink, and get the checksum benefit either way.
The other small-box habit that carries over from the render side: one process at a time. Zipping 39 archives in a loop is fine. Fanning out parallel zip jobs on one core just thrashes. I also call gc.collect() between products when the loop also does any image work, and I check peak memory with resource.getrusage(resource.RUSAGE_SELF).ru_maxrss before and after any change to the routine. Average usage lies, peak RSS tells you when the OOM killer is coming.
Keep the raw material, archive the finished thing
The register note in my own docs says it: keep research/ forever. Your notes, scrape logs, and keyword lists are the raw material for the next product. What you archive off the main disk is the finished layer: the packs/ folder and its _system outputs. My layout looks like this:
project/
assets/ kept on disk, regenerable
output/ kept on disk, regenerable
packs/ zipped, checksummed, archived
research/ never deleted, backed up twice
scripts/ tiny, backed up with research
The regenerable layers stay on the small disk. If I lose output/, I rerun the pipeline. If I lose packs/, I lose sold products, so that is what gets the careful treatment.
The restore test
A backup you have never restored is a hope, not a backup. Every month I pick one zip at random, copy it back, run sha256sum -c on the manifest, and unzip it into a scratch folder. Takes two minutes. The one time this caught a real problem, a USB drive had started throwing read errors, I found it on a scheduled restore test instead of during a customer refund. That test is the cheapest insurance in the whole pipeline.
None of this is clever. It is a zip per product, a checksum per zip, a manifest per drive, and a monthly restore. But it is the difference between a library that survives a full disk and one that loses a product you already sold.
Everything above is built into the packaging layer of the pipeline starter kit, the same four scripts that packaged those 39 products on a 1 vCPU, 2 GB box, including pack_product.py and the folder layout and RAM habits described here.
Building digital products? Free guides & kits for digital product sellers.
Top comments (0)