Error Handling Lessons From Batching 39 Asset Products On A 2 GB Server

I ran a batch of 39 products through my pipeline on a server with 1 vCPU and 2 GB of RAM. Around product 14, the whole thing died mid-render with a MemoryError and no partial output. I lost about three hours of work because I had assumed that if a script worked once, it would work 39 times. It would not. Here is what broke and what I changed.

The error was never the image, it was the metadata CSV

The first failure that showed up repeatedly was not rendering at all. Adobe's bulk upload CSV has a strict 5-column header, Vecteezy's has 4 columns, Dreamstime's has 15. If the header is wrong by even one column, the platform rejects the file instantly at review. My first version of the metadata script built the header by hand, and I got it wrong on Dreamstime twice before I caught it.

The fix was boring: hardcode the exact header per platform and write nothing else. make_metadata.py now takes --platform adobe, --platform vecteezy, or --platform dreamstime and emits the exact column count that platform demands. No configuration, no templates. The header is not a place to be clever.

The second metadata lesson: titles. Without a titles file, the script falls back to title-casing the filename. That is fine for a first pass, but submitting it means review rejects or weak discoverability. I now always pass --titles titles.json with real titles and keywords before anything goes live.

Double-posting was the error I could not undo

The scariest failure mode was not a crash, it was a successful re-run. Running the batch twice means double-posting assets, and that is the fastest way to get a contributor account flagged. A crash-and-retry loop that re-executes from the top is a liability, not a safety net.

So I built upload_tracker.py as a plain ledger. Before uploading, I run:

python upload_tracker.py pending items.txt

That shows what is not done yet. After each successful post:

python upload_tracker.py done "asset-001.jpg"

It uses only the Python standard library. The rule is simple: the ledger is the source of truth, not my memory of what I ran last night. A re-run becomes harmless by construction.

Memory errors on a 2 GB box are predictable, so plan for them

The 3 a.m. MemoryError taught me to measure, not guess. Average memory usage told me nothing. What mattered was peak RSS, which I check with:

resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

Once I measured, the fixes were obvious:

• One process at a time. Fanning out image jobs in parallel on a 1-core box just thrashes.
• Call gc.collect() after each asset so freed objects actually release between items.
• Run heavy render steps as one file per subprocess, so the OS reclaims the memory when the child exits instead of it lingering in the parent.
• Inspect previews at 60 dpi instead of rendering at 300. I was paying full render cost just to check whether an image looked right.

With those habits, the same batch that crashed at product 14 completed all 39 products, about 856 MB of output, on the same 2 GB machine. No GPU, no upgrade.

Crash recovery needs a folder layout, not willpower

Half of my early failures were self-inflicted mess: raw renders mixed with finished zips, no way to tell what was submitted. The layout that fixed it:

project/
  assets/     raw output
  output/     ready-to-submit files + metadata.csv
  packs/      finished zips + covers
  research/   notes and scrape logs
  scripts/    the tools

One rule I learned the hard way: never delete research/. I cleaned it out once to save disk space, then needed those scrape logs a month later to rebuild a product variant. Keeping it forever costs almost nothing and has already paid for itself.

Packaging errors show up as silence, not crashes

The last class of failure was quiet. A product with a bare zip and no cover just does not sell, and nothing errors out to tell you. So packaging became a script, pack_product.py, that renders a 1280×720 cover and a 600×600 thumbnail alongside the zip in one command:

python pack_product.py my-product "My Product" "A short subtitle" 9

Output lands in a _system folder, redirectable with --dir or PACK_ROOT. Covers use Pillow, and that is one of only two third-party dependencies in the whole toolkit. The other is reportlab for md2pdf.py, which turns a Markdown doc into a styled PDF for products that are documents rather than asset packs.

What I would tell myself at hour zero

Every failure in this batch fell into four buckets: a wrong CSV header, a double post, a memory ceiling, or a missing cover. None of them were about making the art. All four are now enforced by scripts instead of by me remembering things at midnight. The whole pipeline ran on a 2 GB server, and the error handling is what made that possible, not raw power.

I packaged the four scripts exactly as they ran that batch into the pipeline starter kit, along with the folder layout and the RAM notes. It is the tool built from this work, nothing more.

Get pipeline starter kit

---

Building digital products? Free guides & kits for digital product sellers.