Last November I ran a batch of 40 asset uploads at 11pm after my day job, on my phone, from a parking lot. Halfway through I lost the connection, restarted the script in the morning, and double-posted 17 files. Adobe flags accounts for duplicate submissions, and cleaning that up took more time than the original batch. That was the night I decided the scheduling had to be boring, automatic, and impossible to fumble by hand.
Here is what actually works for me now, running weekly batches on a 1 vCPU, 2 GB RAM server with no GPU, alongside a full time job.
One cron window, one process
I schedule one weekly window, Sunday evening, when the whole batch runs end to end. Not ad hoc uploads whenever I find ten minutes. The reason is simple: a batch you start and stop is the batch you double-run. The orchestrator (Hermes Agent, free) kicks off the job, and inside the job there is exactly one process at a time. On a 1-core box, fanning out image jobs in parallel does not make anything faster, it just gets you OOM kills. Everything is sequential: generate, render, package, write metadata, update the ledger.
If a step fails, the orchestrator retries that step, not the whole batch. That distinction is what saved me from a repeat of the parking lot incident.
The ledger is the whole safety net
The single most important script in my pipeline is not the fancy one. It is upload_tracker.py, a flat ledger that records every asset marked done. Before any upload step runs, the batch asks the ledger what is pending. Anything already recorded is skipped, no matter how many times the job re-runs.
python upload_tracker.py pending items.txt
python upload_tracker.py done "asset-001.jpg"
python upload_tracker.py status
It uses the Python standard library alone, no install. The fastest way to get a stock account flagged is running your batch twice, and a 40-line ledger makes that impossible. I stopped trusting my own memory at around asset 200 and never looked back.
Metadata headers are pass or fail, not style
All three of my channels reject a wrong CSV header instantly. Adobe wants the 5-column format, Vecteezy wants 4 columns, Dreamstime wants 15. I got these wrong by hand more than once, and each rejection cost a review cycle, which on Dreamstime can mean days of waiting. Now make_metadata.py writes the exact header each platform demands and nothing else:
python make_metadata.py ./my-assets --platform adobe --out metadata.csv --ext .jpg
python make_metadata.py ./my-assets --platform vecteezy --out vecteezy.csv --ext .jpg
By default it title-cases the filename as the title, which is fine for a first pass. I replace titles and keywords from a titles.json before anything is submitted, because filename titles are the fastest way to look like spam in review.
Memory discipline on 2 GB
The whole pipeline, image generation through packaging, runs inside 2 GB of RAM. That only works because of habits, not hardware:
- One file per subprocess for heavy render steps, so memory is actually returned to the OS when the step ends.
-
gc.collect()after every asset. Sounds superstitious until you watch RSS climb across a 40-asset batch and kill the job at asset 30. - Measure peak, not average, with
resource.getrusage(RUSAGE_SELF).ru_maxrss. Average tells you nothing about the ceiling that kills you. - Render previews at 60 dpi when inspecting, 300 only for final output.
The AI image generation itself I do in the same window, and the cost per image is small enough that the real bottleneck was never money, it was my evening hours. Automating the window is what fixed that.
Packaging is where products actually get made
A zip with a real cover sells, a bare file does not. Once a batch passes the ledger and metadata checks, pack_product.py bundles each product: it zips the files and renders a 1280x720 cover and a 600x600 thumbnail in one command. Covers come from Pillow, and PDF products come from md2pdf.py, which turns a Markdown file into a styled PDF using reportlab. Those are the only two third-party packages in the entire pipeline.
The folder layout is fixed and boring: assets/ for raw output, output/ for submission-ready files plus CSVs, packs/ for finished zips and covers, research/ for notes and logs. Keep research/ forever. I deleted mine once to "clean up" and lost the raw material for three products I never rebuilt.
What a week looks like now
During the week, nothing. I dump new assets into assets/ whenever they are ready. Sunday, the cron window runs: generate anything queued, render, package, build the three CSVs, mark the ledger, and stage everything for me to review Monday morning with coffee. Review and submit takes me 20 minutes, manual, because I want human eyes on titles before a platform reviewer sees them. This setup packaged 39 products, about 856 MB of output, on that tiny server, backed by roughly 19,000 lines of pipeline code and 300+ pages of research behind the rules the scripts enforce.
The four scripts are the packaging layer I wish I had on that night in the parking lot. I bundled them, unchanged from what I run, as the pipeline starter kit. If you want the exact ledger, metadata builders, PDF converter, and packager instead of rebuilding them from this post, that is where they are.
Building digital products? Free guides & kits for digital product sellers.
Top comments (0)