Two Packages, Four Scripts, One 2 GB Server: Building an Upload Pipeline That Never Double-Posts

I still remember the night I found out I had posted the same 14 vectors twice.

It was 1 a.m. I had run my batch, saw it finish, changed one keyword, and ran it again. Two days later I got a polite email from a marketplace asking me to explain the duplicates. That email cost me a week of cleaning up, and it taught me something that shaped every script I have written since: the pipeline must know what it has already done.

Everything below comes from one real project. Thirty nine finished products, 856 MB of output, about 19,000 lines of pipeline code, and 300 plus pages of research behind the rules the scripts enforce. All of it running on a single core, 2 GB of RAM, and no GPU.

The metadata rules are not suggestions

The three channels I upload to each want a different CSV, and each one rejects a wrong header instantly.

Adobe wants 5 columns. Vecteezy wants 4. Dreamstime wants 15. Miss one, or put them in the wrong order, and the upload fails before it starts. That is actually good news, because it means the header is a spec, not a guess. My first script, make_metadata.py, does exactly one job: it reads a folder of files and writes the header each platform demands, nothing extra.

The trap is titles. If you do not pass a titles.json, the script title cases the filename and uses that. That is fine for a first pass, but I have seen people submit filenames as titles because they forgot to replace them. Build the JSON, then submit.

The ledger that saved my account

The fastest way to get an account flagged is running your batch twice. upload_tracker.py is a small ledger, standard library only, that records every asset you mark done. Before any upload run, you ask it what is pending:

python upload_tracker.py pending items.txt

It returns only the files that are not in the ledger. Mark one done, and it is gone from the next pending list. No database, no cloud account, just a file that remembers what you have already posted. That one command is the whole reason I do not live in fear of a re-run anymore.

Turning documents into products

The other two scripts exist because the fun part is the art, and the stalls happen everywhere else.

md2pdf.py turns a Markdown file into a styled PDF with headings, tables, lists, and working links. One command:

python md2pdf.py content.md product.pdf "My Product Title"

pack_product.py does the packaging. It zips the files and renders a 1280 by 720 cover plus a 600 by 600 thumbnail into a _system folder next to where you run it. Point it elsewhere with --dir or PACK_ROOT, no editing required:

python pack_product.py my-product "My Product" "A short subtitle" 9

A zip with a real cover sells. A bare file does not. That is the whole philosophy of the kit.

Two packages, and only two

I want to be precise here, because dependency bloat is where small projects die. The kit uses exactly two third party packages: Pillow for the covers in pack_product.py, and reportlab for the PDFs in md2pdf.py. The other two scripts, the metadata builder and the tracker, use the standard library alone. If those are all you need, you skip the install entirely:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

That requirement file contains two names. Nothing else.

Running the whole thing on 2 GB

This pipeline ran on 1 vCPU, 2 GB of RAM, and no GPU. It works because of a few habits, not because of clever code:

• One process at a time. Never fan out image jobs in parallel on a single core box.
• Free memory between items. Call gc.collect() after each asset.
• Measure peak RSS, not average. resource.getrusage(RUSAGE_SELF).ru_maxrss tells you the real ceiling, and the average will lie to you.
• Process one file per subprocess for heavy render steps, so memory returns to the OS.
• Render at the resolution you need. Inspecting a preview at 60 dpi beats rendering it at 300.

None of these are exotic. They are just the difference between a pipeline that survives a small server and one that gets killed halfway through a batch.

The folder layout that keeps a catalogue from becoming a junk drawer

The kit assumes one fixed structure:

project/
  assets/     raw output, one file per asset
  output/     ready-to-submit files plus metadata.csv
  packs/      finished zips and covers
  research/   your notes and sources
  scripts/    these tools

One rule matters more than the rest: keep research/ forever. Your scrape logs and notes are raw material for future products. I deleted mine once to clean up, and I will not repeat that mistake.

What this actually is

Four standalone Python scripts, no framework, no cloud account. They are the packaging layer of a larger free stack, and they are the part that stopped me from double-posting, wrong-header rejections, and half-baked zips.

If you want the scripts exactly as they ran here, they ship as the compliance metadata kit.

Get compliance metadata kit

---

Building digital products? Free guides & kits for digital product sellers.