Last March I ran my batch job twice in one evening. Not on purpose. The first run looked like it had stalled, so I killed it, fixed what I thought was the bug, and ran it again. It had not stalled. The result was 60 duplicate submissions sitting in three contributor queues, and on one of those platforms a duplicate batch is exactly the kind of thing that gets a reviewer to look at your whole account more carefully.
That night I wrote upload_tracker.py, and it has been part of every batch since. Here is what I learned about why double-posting happens and what actually stops it.
The platforms will not save you
None of the big three stock platforms deduplicate on upload in a way that helps you. Adobe Stock gives you a five-column CSV header (Filename, Title, Keywords, Category, Releases) and it rejects the file instantly if that header is wrong, but it will happily accept the same JPG twice under the same name. Vecteezy uses a four-column header and behaves the same way. Dreamstime wants fifteen columns and is the strictest about metadata formatting, but strictness about the CSV is not the same as a check against your upload history.
So the deduplication burden is entirely on you. That surprised me. I assumed there would be a hash check or at least a warning. There is not. If you post the same file twice, it goes through, sits in review, and wastes a review slot that one of your real submissions could have used.
A ledger beats a memory
The fix is embarrassingly simple. One flat text file, one line per asset, appended only when you actually finish the upload for that asset. The tool has three commands:
python upload_tracker.py pending items.txt # what is NOT done yet
python upload_tracker.py done "asset-001.jpg"
python upload_tracker.py status
The key design decision is that done is a separate, manual step. The batch script does not mark things done automatically when it queues them, because queuing is not uploading. I learned that the hard way too: an early version marked assets complete when the CSV was generated, which meant a crash mid-upload left the ledger lying to me. Now the ledger only records what a human confirmed went through.
Because it is append-only plain text, I can read it with grep, diff it between machines, and recover it from any backup. No database, no dependencies. Both upload_tracker.py and make_metadata.py run on the Python standard library alone. You do not even need a virtualenv for them.
The same ledger checks your CSVs before they cost you anything
The other half of the problem is that a re-run regenerates metadata files that look identical to new ones. So before make_metadata.py writes a row for Adobe (5 columns), Vecteezy (4 columns) or Dreamstime (15 columns), it is trivial to cross-check the filename against the ledger. A filename that is already marked done gets skipped, not silently rewritten. That single check is what made my second-run disaster impossible to repeat.
The header rule matters here too. All three platforms reject a wrong CSV header on upload, instantly. The script writes the exact header each platform demands and nothing else, because I got tired of finding out about a bad header after a 40-file upload failed.
Batch hygiene on a small box
My whole pipeline runs on 1 vCPU with 2 GB RAM and no GPU, so the ledger also has to be cheap. It is: appending a line to a text file costs nothing. The habits that keep the rest of the batch safe on that box apply here too:
- One process at a time. Never fan out image jobs in parallel on a single core.
- Call
gc.collect()after each asset to free memory between items. - Measure peak RSS with
resource.getrusage(RUSAGE_SELF).ru_maxrss, not averages, because averages hide the spike that kills the run. - One file per subprocess for heavy render steps so memory actually returns to the OS.
A crashed batch is what causes most double-posting in the first place, so stability and deduplication are the same problem. If your run survives, you do not re-run it.
What this looks like in practice
Since the ledger went in, I have run the pipeline for 39 finished products, about 856 MB of output, across the three platforms, with zero duplicate submissions. The full parent project is around 19,000 lines of pipeline code, and the ledger is maybe 80 of them. It is the 80 lines I would rescue first from a fire.
The workflow is now: generate assets, build the per-platform CSVs with make_metadata.py, check pending before every upload session, mark done after each confirmed upload, run status at the end of the session to confirm the counts line up. Thirty seconds of bookkeeping versus an afternoon of deleting duplicates and an account review I did not want.
If you want the same setup, the four scripts (metadata builder, upload ledger, Markdown to PDF, and a product packager that renders a 1280x720 cover and 600x600 thumbnail) are bundled as the Pipeline Starter Kit, which grew out of the same work as my vector prompt pack. The ledger and metadata builder need nothing but Python 3.9+.
Building digital products? Free guides & kits for digital product sellers.
Top comments (0)