I lost about forty minutes of work one Tuesday night because a script died on asset 61 of 120 and I had no idea which files had finished. The batch was generating metadata rows for an Adobe upload. Asset 61 was a file with a weird filename, the script threw an exception, and everything before it was fine but nothing told me that. I re-ran the whole thing, got duplicate rows, and had to diff the CSV against the folder by hand before submitting, because Adobe rejects a CSV with duplicate filenames and a wrong header instantly, so you do not get to be sloppy on that step twice.
That night I wrote the retry-and-log pattern I still use. It runs inside upload_tracker.py and the metadata scripts in my pipeline, and it has since processed the batches that became 39 finished products, around 856 MB of output, on a server with 1 vCPU, 2 GB RAM, and no GPU. Here is what actually works.
A ledger file beats a log file
A log tells you what happened. A ledger tells you what is done. Those are different questions and re-runs only care about the second one.
upload_tracker.py keeps a plain text ledger with three commands:
python upload_tracker.py pending items.txt # shows what is NOT done yet
python upload_tracker.py done "asset-001.jpg"
python upload_tracker.py status
The rule is simple: mark an asset done only after the write succeeds, not after the attempt starts. If you mark it at the start of processing, a crash mid-write leaves you with a "done" entry and a missing row. If you mark it after, a crash leaves the asset pending and the re-run picks it up. This is the difference between a re-run being safe and a re-run double-posting. Double-posting is not just annoying, it is the fastest way to get an account flagged on stock platforms, so I treat the ledger as the source of truth and never trust my memory of what ran.
Retry with a cap, and log the reason
The failures you get in batch processing are mostly transient: a network hiccup, a subprocess that got killed, a memory spike on a 2 GB box. My pattern is three attempts per asset with a short sleep between them, and after the third failure the asset goes into a failures list with the exception text attached. Then the script moves on.
The part people skip is logging the reason per asset. When the run finishes I want one file that says "asset-061.jpg: FileNotFoundError after 3 attempts" and not a 4,000-line traceback wall. When I fixed asset 61 that night, the fix took two minutes. Finding it took forty.
One process at a time on a small box
It is tempting to fan out with multiprocessing. On a 1-core, 2 GB server it is a mistake. I run one process at a time, call gc.collect() after each asset to free memory, and for heavy render steps I spawn one subprocess per file so memory actually returns to the OS instead of accumulating in the parent.
Two measuring habits that saved me:
- Check
resource.getrusage(RUSAGE_SELF).ru_maxrssfor peak RSS, not average usage. Average tells you nothing about whether you will get OOM-killed at asset 90. - Render at the resolution you need. Inspecting a preview at 60 dpi instead of 300 is the difference between fitting in RAM and not, and the output looks identical for the check you are doing.
The retry loop and the memory discipline interact: if a retry happens because of a memory kill, retrying in the same bloated parent process just fails again. The subprocess-per-file approach is what makes retries meaningful.
Idempotent writes, so a re-run never corrupts anything
Every script in the pipeline is written so that running it twice on the same input produces the same result. make_metadata.py derives the title from the filename (title-cased) unless you pass --titles titles.json, so a re-run on the same folder rebuilds the same rows instead of appending. The platform headers are fixed and exact: Adobe is 5 columns, Vecteezy is 4, Dreamstime is 15, and all three reject a wrong header on upload. The script writes the header they demand and nothing else, so a partial run followed by a re-run produces a valid file, not a file with two header rows.
If your script appends instead of rebuilds, your retry pattern is broken no matter how good the retry logic is. Idempotency is what makes "just run it again" a safe instruction.
The whole pattern in five lines of philosophy
- Mark done after success, never before.
- Retry three times, then record the failure and move on.
- Log one line per failure with the reason.
- One process at a time on a small server,
gc.collect()between items. - Make every script re-runnable so the retry is always safe.
None of this is clever. That is the point. The night I lost forty minutes was not caused by a hard problem, it was caused by a script with no ledger and no failure record. The fix was boring and it has held up across every batch since.
Where this lives now
This pattern is baked into the pipeline starter kit, the four standalone scripts that packaged those 39 products: make_metadata.py for the Adobe, Vecteezy, and Dreamstime CSVs, upload_tracker.py for the ledger, md2pdf.py for turning Markdown into a styled PDF, and pack_product.py for zips with a 1280×720 cover and 600×600 thumbnail. Two of the scripts need nothing but the standard library, the other two need only Pillow and reportlab. Everything was validated on 1 vCPU / 2 GB RAM / no GPU.
Building digital products? Free guides & kits for digital product sellers.
Top comments (0)