I found the double-post at 11pm on a Tuesday. I was reviewing my Adobe Stock contributor dashboard and noticed the same batch of vector assets sitting in review twice, forty files submitted twice in one week because I re-ran my upload script after a partial failure and had no record of what had already gone out. Nothing dramatic happened to the account, but that was luck. Running a batch twice is the fastest way to get a contributor account flagged, and I had just done it on the biggest platform I sell through.
The fix took an afternoon and it is now the least glamorous, most valuable file in my pipeline. Here is what I learned.
Why double-posting happens so easily
If you batch your uploads the way I do, the flow looks like this: generate a folder of assets, build a metadata CSV, submit through the contributor portal. The failure mode is that step two and step three are not atomic. Adobe Stock's CSV template is a strict five-column header, Vecteezy wants four columns, Dreamstime wants fifteen. If any header is wrong the channel rejects the file instantly, and my instinct the first few times was to fix the CSV and re-run the whole batch. That re-run is where duplicates come from. The script had no memory. It happily rebuilt entries for files that were already submitted, because "already submitted" was a fact that lived only in the contributor dashboard, not on my disk.
Partial failures make it worse. If the run dies at file thirty of forty, you do not want to resubmit the first thirty. But without a ledger, you have no way to know where it stopped, so you either resubmit everything or you hand-audit the dashboard, which is exactly the kind of tedious work a pipeline is supposed to remove.
The tracker is one file and no dependencies
The solution is embarrassingly small. upload_tracker.py is a single Python script, standard library only, no pip install. It keeps a plain ledger of what is done and answers three questions:
python upload_tracker.py pending items.txt # what is NOT done yet
python upload_tracker.py done "asset-001.jpg"
python upload_tracker.py status
The workflow that replaced my Tuesday-night disaster: before any submission run, generate items.txt from the folder of files. Run pending first. Only submit what pending shows. Mark each file done as it goes out. If the run dies at file thirty, re-run pending and it shows you the last ten, and only those. The re-run can no longer double-post because the ledger, not my memory, decides what goes out.
I want to stress the no-dependency part, because it matters more than it sounds. My whole pipeline runs on a 1 vCPU, 2 GB RAM server with no GPU. Only two scripts in my toolkit need third-party packages at all (Pillow for covers, reportlab for PDFs). The metadata builder and the tracker need nothing but Python 3.9+, which means they run anywhere, including a box this small, without a virtualenv argument about it.
The metadata half of the same problem
The tracker only helps if the CSV is right the first time. make_metadata.py writes the exact header each channel demands and nothing else:
python make_metadata.py ./my-assets --platform adobe --out metadata.csv --ext .jpg
python make_metadata.py ./my-assets --platform vecteezy --out vecteezy.csv --ext .jpg
python make_metadata.py ./my-assets --platform dreamstime --out dreamstime.csv --ext .jpg
It can pull real titles and keywords from a titles.json if you supply one. Without it, the filename gets title-cased and used as the title, which is fine for a first pass but should be replaced before you submit. Rejection at the CSV stage is what triggers the re-run impulse in the first place, so fixing the header problem and fixing the duplicate problem are really the same fix: make the pipeline deterministic so you never feel the urge to "just run it again".
The habits that keep this safe
Three things I now do every batch:
- One process at a time. Never fan out image jobs in parallel on a 1-core box. Sequential is slower per item and faster overall because nothing dies.
- Free memory between items with
gc.collect(), and measure peak RSS withresource.getrusage(RUSAGE_SELF).ru_maxrss, not average. Average memory tells you nothing about the ceiling. - One file per subprocess for heavy render steps, so memory actually returns to the OS between assets.
These came out of running the full pipeline, roughly 19,000 lines of it, to package 39 products, about 856 MB of finished output. The tracker is the piece I would rebuild first if I lost everything else.
Where to get it
I packaged the tracker, the metadata builder, and two other scripts (a Markdown-to-PDF converter and a product packager that zips files and renders a 1280x720 cover plus a 600x600 thumbnail) into the Pipeline Starter Kit. It is the same tooling that produced my vector prompt pack and the rest of the catalogue, running on hardware that costs a few dollars a month. If you are building a similar pipeline, it will save you the Tuesday night I lost.
Building digital products? Free guides & kits for digital product sellers.
Top comments (0)