How I Keep Upload Ledgers And Metadata CSVs Safe When Storage Runs Out

Last February my 2 GB server filled up mid-batch. The disk hit 100 percent while make_metadata.py was writing a CSV for 240 Adobe assets, and the write truncated. I did not notice. I uploaded the truncated CSV to Adobe Stock and got the standard instant rejection: bad header row. Then I re-ran the batch to "fix" it, forgot the ledger, and double-posted 60 files to Vecteezy. That was the week I learned that storage pressure and ledger integrity are the same problem, because the moment you start deleting things to free space, you delete the wrong things first.

Here is what I do now.

The ledger is the first thing you protect, not the last

upload_tracker.py is a plain text ledger in the standard library, no dependencies. It records every asset marked done with one command:

python upload_tracker.py done "asset-001.jpg"

The whole point is that a re-run never double-posts. Stock platforms treat duplicate submissions as a spam signal, and repeated duplicates are the fastest way I know to get an account flagged. So the ledger file lives in research/, the one folder I never delete, and I back it up before any cleanup pass. It is a few kilobytes. The 856 MB of finished output this pipeline produced is replaceable from the raw assets. The ledger is not, because it encodes which uploads already happened on which platform.

Storage cleanup has a fixed order

When the disk fills, most people delete the biggest folders first. That is exactly backwards. My order now:

1. Raw AI image output that has already been packaged into packs/. The zip plus cover and thumbnail are the product. The intermediates are not.
2. Preview renders. I inspect at 60 dpi instead of 300, which cuts preview size by roughly 25x and lets me see what I need without the full resolution file existing.
3. Old CSVs. Metadata is regenerated in seconds from the asset folder, so a stale CSV has zero value.

What I never delete: research/, the ledger, and assets/ for anything not yet submitted. My scrape logs and notes are the raw material for future products. I made the mistake of clearing them once to "clean up" and lost a week reconstructing context I had already written down.

CSV headers fail instantly, so regenerate, do not edit

Adobe wants a 5-column header, Vecteezy wants 4 columns, Dreamstime wants 15. Get one column wrong and the platform rejects the upload before it ever reaches a human reviewer. Hand-editing a CSV under storage pressure, on a truncated file, is how the February failure happened.

make_metadata.py writes the exact header each channel demands and nothing else:

python make_metadata.py ./my-assets --platform adobe --out metadata.csv --ext .jpg

It reads the folder, builds the file, done. If the disk filled and truncated the output, I delete the CSV and regenerate it. Two seconds. The CSV is a derived artifact, always. The source of truth is the asset folder plus titles.json. Once you treat metadata as disposable output rather than a document you maintain, the "did my CSV get corrupted" fear disappears.

Measure the ceiling before you fill it

The whole pipeline runs on 1 vCPU, 2 GB RAM, no GPU. The habits that keep it alive:

• One process at a time. Never fan out image jobs in parallel on a single core.
• gc.collect() after each asset.
• resource.getrusage(RUSAGE_SELF).ru_maxrss for peak RSS, not average, because the peak is what kills you.
• One file per subprocess for heavy renders, so memory goes back to the OS.

Disk is the same discipline. Check free space before a batch, not after the rejection email. A 240-asset batch at full resolution is measured in gigabytes. If I have less headroom than that, I split the batch, run it, package it, and clear the intermediates before the next one.

The folder layout makes cleanup safe

project/
  assets/       raw output
  output/       ready-to-submit files + metadata.csv
  packs/        finished zips + covers
  research/     notes, logs, ledger backups
  scripts/      the tools

Because everything in output/ and packs/ is derived, cleanup is mechanical: anything downstream of assets/ can be rebuilt. Anything in research/ stays forever. That single rule is what turned storage pressure from a panic into a checklist.

Where the tools live

Everything above is not theory. These are the same four scripts that packaged 39 products from one prompt engine: make_metadata.py, upload_tracker.py, md2pdf.py, and pack_product.py, backed by about 19,000 lines of pipeline code and 300+ pages of research, all validated on that same 1 vCPU, 2 GB box. They are standalone Python, no framework, no cloud account, and the two metadata tools need nothing but the standard library.

I put them together as the vector prompt pack, on Gumroad. It is the honest version of this post: the working tools, not a description of them.

Get vector prompt pack

---

Building digital products? Free guides & kits for digital product sellers.