How Upload_Tracker.Py Stops Double-Posting When You Re-Run A Batch

I run my whole asset pipeline on a 1 vCPU, 2 GB RAM box with no GPU. A few months ago I had a batch of 40 sticker assets ready for Adobe Stock, Vecteezy, and Dreamstime. The CSV was built, the files were named, everything looked done. Then the SSH session dropped halfway through my upload queue.

I reconnected, re-ran the batch script from the top, and went to make coffee. When I came back I had 14 duplicate submissions sitting in my Adobe contributor queue. Adobe's review checks flag repeated identical content, and the fastest way to get an account flagged is running your batch twice. I had just done exactly that, by hand, in the most predictable way possible.

That afternoon I wrote upload_tracker.py. Here is what it does, why it works, and what I learned about idempotency the hard way.

A ledger, not a lock

The script is a plain-text ledger. Three commands:

python upload_tracker.py pending items.txt   # shows what is NOT done yet
python upload_tracker.py done "asset-001.jpg"
python upload_tracker.py status

It uses the standard library alone, no pip install. The design decision that matters: the ledger is written after the upload succeeds, not before. If the process dies mid-upload, the asset is still "pending" on the next run, and the re-run picks it up. If I had marked items done at the start of the attempt, a crash would silently skip anything unfinished.

The pending command takes your full list (I keep an items.txt per batch) and subtracts the ledger. So the re-run loop is always "upload everything in pending", which is a no-op when nothing is pending. That is the whole trick. The batch script does not need to know whether it is the first run or the fifth.

Why duplicates are worse than they look

I used to think a duplicate submission was just noise a reviewer would skip. It is not. On Adobe, identical content submitted twice counts against you in review, and repeated identical submissions are one of the listed reasons contributors get flagged. Vecteezy's queue is smaller and a duplicate is a visible waste of a reviewer's time. Dreamstime's 15-column CSV gives you more metadata room, which means a duplicate there carries two full metadata records for one file, and mismatches between them look sloppier than the duplicate itself.

One duplicate is survivable. A pattern of them, across batches, is what gets an account looked at closely. When you are a solo seller, your account is the business. I do not risk it on memory anymore.

The failure mode I did not expect

The first version of my tracker stored the ledger in the same folder as the assets. Then I reorganized the project, moved output/ around, and the ledger path broke silently. The script happily started a fresh empty ledger and I nearly double-posted again.

Now the ledger lives in a fixed place relative to the project root, and the folder layout is fixed too:

project/
  assets/            raw output (one file per asset)
  output/            ready-to-submit files + metadata.csv
  packs/             finished product zips + covers
  research/          your notes and sources
  scripts/           these tools

If you keep the structure stable, the ledger stays valid. If you restructure, check status before the next batch. It prints a count of done and pending items, so an empty "done" count on a project you know you have posted to is an immediate red flag.

How it fits with the metadata step

upload_tracker.py does not know about CSVs. That separation is deliberate. make_metadata.py builds the Adobe 5-column, Vecteezy 4-column, and Dreamstime 15-column headers from a folder of files, and it is idempotent by nature: re-running it over the same folder produces the same CSV. The tracker covers the other half, the side that talks to a platform queue and cannot be safely re-run.

So my per-batch loop is now:

1. Generate files into output/.
2. Build the CSVs for each platform.
3. Read pending, upload those items.
4. Mark each one done after the upload confirms.
5. Re-run the whole script any time, at any point, without fear.

Step 4 is manual on purpose. The upload happens in a browser on the platform's site, so no script can know it succeeded. I mark items done immediately after each confirmation, not in a batch at the end. That way a crash loses at most one item's state.

The boring lesson

Most pipeline failures are not exotic. Mine was: a dropped SSH session plus a script with no memory. The fix is not clever either, it is a text file that remembers what happened. The whole thing is maybe 80 lines of standard-library Python, and it has prevented every double-post since. I re-run batches freely now, mid-batch, after crashes, weeks later, and pending always tells me the truth.

This tracker is one of four scripts in the pipeline starter kit I built while packaging 39 products (856 MB of output, ~19,000 lines of pipeline code, validated on that same 1 vCPU / 2 GB RAM box). The full system, including the prompt engine and the packaging layer, is written up in the ai sticker factory playbook. It is the same honest, no-framework approach: two dependencies, standard library where possible, and rules that exist because I broke them first.

Get ai sticker factory playbook

---

Building digital products? Free guides & kits for digital product sellers.