I killed my pipeline with a single &.
It was 1am, I had about 40 images queued for cover rendering, and I thought, why not run these in parallel? It's a modern server, right? Wrong. It's a 1 vCPU box with 2 GB of RAM and no GPU. Ten minutes later the OOM killer had taken down the render process, half my batch was in an unknown state, and I had no idea which covers had finished. I spent the next two hours figuring out which files were complete, and I double-uploaded three assets to Adobe Stock because my tracking was a text file I'd forgotten to update.
That night taught me more about running a one-person asset pipeline than any tutorial. Here's what actually works on tiny hardware.
The box is small, so act like it
The whole pipeline that produced 39 finished products, 856 MB of output, ran on 1 vCPU and 2 GB of RAM. That is not much. Pillow opening a large source image can eat 300 to 500 MB by itself depending on dimensions, and if you fan out four of those in parallel you're done before you start.
The habits that keep it alive:
- One process at a time. Never parallelize image jobs on a single core. You gain nothing and lose everything.
- Call
gc.collect()after each asset. Python holds references longer than you expect. - Measure peak RSS, not average.
resource.getrusage(RUSAGE_SELF).ru_maxrssgives you the real ceiling. Average memory tells you nothing about the spike that kills you. - Run heavy render steps as one file per subprocess, so the OS reclaims the memory when it exits instead of your long-lived Python process hoarding it.
- Render at the resolution you need. I used to render everything at 300 dpi just to inspect it. Inspecting a preview at 60 dpi is the same decision for a fraction of the memory.
None of this is clever. It's just respecting the hardware you actually have.
The CSV header is a wall, not a suggestion
The fastest way to lose a day is a rejected upload batch. All three major channels validate the CSV header before they look at a single file. Adobe Stock wants its 5-column format, Vecteezy its 4-column, Dreamstime its 15-column. Get one column wrong and the whole batch bounces, instantly, with an error that doesn't tell you what you actually broke.
I stopped hand-editing CSVs after the third rejection. Now a script builds the header each platform demands and nothing else, straight from the folder of finished files. The title comes from the filename, title-cased, unless I pass a JSON file with real titles and keywords. That fallback is fine for a first pass, but replace it before you submit, because filename titles read like filename titles and reviewers notice.
Double-posting is the silent account killer
This is the one nobody warns you about. Run your upload batch twice, even accidentally, even months apart, and you've submitted duplicates. Duplicates are one of the fastest ways to get flagged on stock platforms, and if you're building an automated pipeline, running things twice is not an edge case. It's a certainty. Cron jobs overlap, you re-run after a partial failure, you forget what you did last Tuesday.
The fix is boring and it works: a ledger. Before each upload, check the ledger. After each upload, mark it done. A re-run now skips everything already posted instead of resubmitting it. Two commands, a plain text store, no database. My 1am disaster would have been a non-event with this in place.
Packaging is where a file becomes a product
I had a folder of good assets for weeks before I sold anything, because "ready to sell" is a lie you tell yourself. A bare file doesn't sell. A zip with a real cover at 1280×720 and a thumbnail at 600×600 does, and producing those by hand for every product is the kind of work you'll skip, and then your listing looks like everyone's abandoned listing.
Same for documents. I had research and guides sitting in Markdown, and turning them into a styled PDF by hand took an hour each. One command with reportlab does the headings, tables, lists, and links in seconds. On the tiny server this is where the subprocess rule matters most: render the PDF, let the process exit, move on.
Keep the dependency list at two
The full pipeline is around 19,000 lines and sits on top of a free stack, but the packaging layer deliberately uses only two third-party packages: Pillow for covers, reportlab for PDFs. The metadata builder and the upload ledger run on the standard library alone, nothing to install. On a 2 GB box, every dependency is surface area for a memory surprise or a version conflict, and for text and CSV work Python's standard library is genuinely enough.
One more habit from the register of mistakes I've made: keep your research folder forever. My scrape logs and notes are the raw material for future products. I deleted them once during a "cleanup" and rebuilt from scratch later. Don't.
The tool that came out of this
Everything above is now four standalone scripts, no framework, no cloud account: the metadata builder for Adobe, Vecteezy, and Dreamstime, the upload ledger, the Markdown to PDF converter, and the packager that zips files and renders the cover and thumbnail. They're the same ones that packaged those 39 products on the 1 vCPU, 2 GB server, backed by 300+ pages of research into the platform rules they enforce. If you want the working tools instead of the theory, that's what they are.
Building digital products? Free guides & kits for digital product sellers.
Top comments (0)