I killed my pipeline at 2:40 in the morning watching a batch re-run post the same 40 files to Adobe Stock twice. The upload had timed out halfway, I restarted the script, and my ledger was a text file I had forgotten to update. That is the fastest way I know to get an account flagged, and I did it to myself.
That failure is why I now run everything on a ledger instead of memory, and why the whole 39-product pipeline runs on a server with 1 vCPU and 2 GB of RAM. No cloud account, no GPU, no monthly bill. Here is what actually matters if you want to do the same.
Metadata is where submissions die, not the art
All three major stock channels validate the CSV header before they look at a single file. Adobe expects 5 columns, Vecteezy expects 4, Dreamstime expects 15. Get one column wrong and the upload is rejected instantly, before any human or model sees your work.
So I stopped hand-editing spreadsheets. make_metadata.py reads a folder of files and writes the exact header each platform demands, nothing more. Adobe gets 5 columns. Vecteezy gets 4. Dreamstime gets 15. If the filename is vintage-compass-rose.jpg, it becomes the title-cased title by default, which is fine for a first pass. I pass a titles.json when I want real keywords, and I always replace the default before submitting, because a filename is not a keyword strategy.
The point is not that the script is clever. The point is that the header is a fixed contract, so a script should own it, not your memory at the end of a long session.
Double-posting is the failure mode nobody plans for
My 2 a.m. mistake came from one gap: nothing remembered what had already been uploaded. Batch scripts are stateless by default, and any re-run after a timeout or a crash replays everything.
upload_tracker.py is a small ledger built around three commands: pending shows what has not been posted, done marks an item, and status prints the count. It uses only the Python standard library, so there is nothing to install and nothing to break. Every batch script I run now checks the ledger before it acts. A re-run finds an empty pending list and stops. That single habit is worth more than any upload speedup.
2 GB of RAM is enough if you run one thing at a time
This is the part I got wrong for months. On a 1-core box, fanning out image jobs in parallel does not speed anything up, it just makes the process get killed by the OOM killer halfway through a render.
The habits that made it work:
- One process at a time. No parallel image jobs, ever.
- Call
gc.collect()after each asset so memory does not creep across a 200-item batch. - Measure peak RSS, not average.
resource.getrusage(RUSAGE_SELF).ru_maxrssgives you the real ceiling. Average usage told me I was fine; peak usage told me the truth. - Run heavy render steps as one file per subprocess, so the OS reclaims the memory when the process exits.
- Render at the resolution you need. Inspecting a preview at 60 dpi instead of 300 cut one step's memory use dramatically, and I could not tell the difference in the output.
The whole thing, 39 finished products and 856 MB of packaged output, ran inside those limits. The parent project behind it is around 19,000 lines of pipeline code, and none of it needs a GPU.
Turning content into a product is two commands, not a weekend
I had Markdown documents full of useful material sitting in a research/ folder doing nothing. The gap between "I have content" and "I have a product" was always packaging.
Two scripts close it. md2pdf.py takes a Markdown file and renders a styled PDF with headings, tables, lists, and links, in one command. pack_product.py takes a folder and produces the zip, a 1280×720 cover, and a 600×600 thumbnail. A zip with a real cover sells. A bare file does not, and I have the sales numbers to prove it on my own catalogue.
Between the two scripts they pull in exactly two third-party packages: Pillow for the covers and reportlab for the PDF. Everything else is standard library. On a small server, dependency count is RAM budget.
The folder layout that saved me from myself
A catalogue becomes a junk drawer without a fixed structure. Mine is five folders: assets/ for raw output, output/ for submission-ready files plus the metadata CSVs, packs/ for finished zips and covers, research/ for notes and sources, and scripts/ for the tools.
One rule I learned the hard way: never delete research/. I once cleaned it out to "tidy up" and lost the raw material for three products I had planned. Scrape logs and old notes are inventory, not clutter.
Where the tools live
Everything above came out of building this pipeline for real: the scripts that packaged 39 products, backed by 300+ pages of research behind the rules they enforce, validated on that same 1 vCPU / 2 GB RAM box. I bundled the four scripts, the folder layout, and the RAM notes into the Pipeline Starter Kit because I kept rewriting the same answers for other makers in my inbox. It is not a course, it is the working tools with no framework and no cloud account required. If it saves you one flagged account or one 2 a.m. double-post, it did its job.
Building digital products? Free guides & kits for digital product sellers.
Top comments (0)