Last March I kicked off a 300-file metadata run before bed. Adobe CSV, 297 JPGs, everything staged in output/. Around file 84 the script hit a zero-byte JPG that a render subprocess had left behind, threw on Image.open(), and died. I woke up to a CSV with 83 rows and no error log worth reading. The fix took twenty minutes. Finding the actual line that failed took an hour, because the traceback pointed at the CSV writer, not the broken file.
That night taught me the rule I now build every script around: on a batch run, the job is not "process 300 files", it's "process every file that can be processed, and tell me about the rest".
Fail per file, not per run
The mistake was a single loop with no try block. One bad file killed the run. Now every file goes through its own try/except, and a failure writes one line to errors.log with the filename and the reason, then moves on. The run finishes. I get a summary: 294 rows written, 6 skipped, reasons listed.
This matters more than it sounds because of how the platforms actually review. Adobe's CSV is a 5-column header and they reject the whole upload for a wrong header or a malformed row. Vecteezy uses a 4-column header. Dreamstime wants 15 columns. If a crash truncates your CSV mid-run, you might not notice and upload a partial file anyway, which reads to a reviewer as sloppy metadata, and repeated sloppy uploads are how accounts get flagged. A clean CSV with only valid rows is safer than a complete CSV with one poisoned row.
Validate before you write, not after
The second version of the script wrote rows first and hoped. Now it checks each file before a row gets built: does the file exist, is it non-zero bytes, does the extension match what I passed with --ext, can the title be derived. A zero-byte file fails the size check and never touches the CSV. A filename like IMG_20240311_002.jpg still gets title-cased into a usable title, but I know that's a first-pass title, and the kit's docs say to replace it before submitting, because keyword-stuffing filenames is one of the things reviewers notice.
The check costs microseconds per file. On a 300-file run it's nothing. Without it, the cost is a truncated CSV and a re-run.
The re-run problem is worse than the crash problem
Here's the trap nobody warns you about: after a crash, you fix the bad file and run again. Now your tracker has stale state, or worse, you already uploaded the first 83 files manually and the re-run generates rows for all 300. Double-posting is the fastest way to get an account flagged, and it happens exactly this way, through partial runs and re-runs.
That's why make_metadata.py never works alone in my flow. upload_tracker.py runs beside it. It's a plain ledger: done "asset-001.jpg" marks a file, pending items.txt shows what's left, status gives the count. Before any upload session I check pending. After the crash in March, pending is the first thing I look at, not the CSV.
Both scripts use only the standard library. No Pillow, no reportlab, no install step. That's deliberate. The metadata layer should have zero dependencies, because dependencies are where runs break on servers that aren't your laptop.
Small server reality
This all runs on 1 vCPU and 2 GB RAM, no GPU. The metadata script itself barely uses memory, it's text processing. But the render steps upstream of it don't, and a crashed render subprocess is exactly what produced the zero-byte file that started this story. So the habits stack: one process at a time, gc.collect() after each asset, one file per subprocess for heavy renders so memory returns to the OS, and measure peak RSS with resource.getrusage(RUSAGE_SELF).ru_maxrss instead of guessing. A file that fails to render fully gets caught by the size check downstream. The layers cover each other.
I also stopped rendering previews at 300 dpi just to inspect them. 60 dpi tells you if a vector export came out right, and it's a fraction of the memory. The 300 dpi pass only runs for final output.
What I'd tell past me
Write the error log first. Wrap the per-file loop. Validate file size and extension before building the row. Keep the tracker in the same command as the workflow, not as an afterthought. And never assume a run finished because the process exited, check the row count against the file count every time. 294 out of 300 is a success if you know about the 6. 294 out of 300 that you believe is 300 is how accounts get flagged.
The whole pipeline, including this metadata script, the tracker, the PDF tool, and the packager, is now 39 finished products and about 856 MB of output, built from roughly 19,000 lines of pipeline code. I packaged the four scripts that do the packaging layer into the vector prompt pack, because these were the pieces I kept rewriting from scratch before I got them right. If you're building the same kind of pipeline, it's the honest starting point: four standalone Python scripts, the folder layout, and the small-server notes, nothing you don't need.
Building digital products? Free guides & kits for digital product sellers.
Top comments (0)