DEV Community: Ryan - building OwnStack

You baked the model into the image. One env var can silently un-bake it

Ryan - building OwnStack — Sat, 01 Aug 2026 02:05:53 +0000

On serverless GPU, cold start is image pull plus model load. The single biggest win is usually to stop downloading weights at boot: download them at build time so they ship inside the image and load from local disk instead.

The setup is two lines. Point the cache at a path inside the image, then fetch during the build:

ENV HF_HOME=/opt/huggingface
RUN python -c "from huggingface_hub import snapshot_download; snapshot_download('<repo>', revision='<pinned>')"

snapshot_download writes into HF_HOME, that directory becomes an image layer, and at runtime the loader finds everything already on disk.

The failure that doesn't look like a failure

Here's the part worth knowing. The loader resolves weights through HF_HOME at runtime. So if anything in your runtime environment sets HF_HOME to a different path — a network volume, a mount, a leftover template variable — the loader looks there, finds an empty cache, and quietly falls back to downloading from the network.

Nothing errors. Your image still contains the weights, sitting in a directory nobody reads anymore. You just paid to bake them, and you're paying the download on every cold start anyway. The only symptom is that cold starts went back to what they were before you did the work, which is easy to blame on the platform.

How to make it hard to get wrong

Attach no volume at all. If the worker has no volume, there's no tempting path to point the cache at. In my setup the endpoints run with volume size zero for exactly this reason: everything the worker needs is in the image, so a volume is not an optimization, it's a way to reintroduce the network.

Verify at boot, not by feel. Watch a cold worker's log. If you see download progress for weights, the bake is being bypassed, full stop. A baked image reaching the network for weights is always a misconfiguration, never a normal path.

Treat the cache path as part of the build contract. The Dockerfile that baked the weights and the runtime env that reads them have to agree. That agreement is invisible in both files individually, so write it down where whoever edits the template env will see it.

Worth saying out loud

Baking has real costs: the image gets large, builds take much longer, and updating the model means rebuilding and redistributing the whole thing. If your weights are small or your traffic keeps workers warm, a volume or a runtime fetch may genuinely be the better trade. This is about the case where you already chose to bake — because then paying the build cost and the download cost is the worst of both.

RunPod serverless: jobs stuck in queue forever after changing MODEL_ID

Ryan - building OwnStack — Sat, 01 Aug 2026 01:42:25 +0000

Symptom: you submit a job, it sits in IN_QUEUE and never moves. The endpoint looks healthy in the console. Workers keep starting and dying, so it reads like a queue problem rather than a crash.

The usual advice is "check your handler logs," which is right but doesn't tell you what to look for. Here is a specific failure mode I lost time to and haven't seen written down.

The setup that causes it

If you bake model weights into your image, you probably also pin a revision so an upstream change can't silently swap them under you. Reasonable. The trap is that the repo id and the revision usually end up managed in two different places:

the repo id comes from an env var on the RunPod template
the revision is pinned in code, next to the loader

Change only the repo id, say you point MODEL_ID at the original repo instead of the quantized mirror you actually baked, and you have created a combination that does not exist: a commit hash that only lives on the mirror, requested against a different repo.

What actually happens at boot

The load misses the local cache (that repo was never baked), falls back to the network, asks for a revision that repo never had, and 404s during startup. The worker dies before your handler is ever reached.

From the outside this looks nothing like a crash. The endpoint reports healthy, workers cycle, and the job just sits in the queue.

How to check

Open a worker's log, not the endpoint status. Look for a fetch attempt at boot: if a baked image is reaching the network for weights at all, something already broke. Then compare the repo id in the template env against the exact repo and revision your Dockerfile downloaded.

The takeaway

If weights are baked, treat repo id and revision as one unit. Pin both in the same place, or set neither at runtime. Splitting them across a template env and a code constant is how you end up asking for a combination that has never existed anywhere.

Character consistency isn't a seed trick: a 2-stage image pipeline that actually locks the face

Ryan - building OwnStack — Thu, 23 Jul 2026 04:00:36 +0000

If you're building an app that generates the same character across many scenes, you've probably hit the wall already: seeds drift, LoRA training is heavy and slow, and "same character, new pose" prompting quietly changes the face. The approach that actually holds up in production is a 2-stage pipeline — generate one canonical base image, then edit from that base as the reference for every new scene. Consistency comes from the reference, not the seed.

Below: why the common approaches drift, how the 2-stage pipeline works, the async job queue that makes it deployable, and the serverless-GPU setup that keeps it affordable. There's a free, runnable slice of the whole transport layer at the end.

Why the obvious approaches drift

Seeds. A seed pins the noise, not the identity. Re-use a seed with the same prompt and you get the same image — but that's reproduction, not consistency. The moment you change the prompt ("now she's in a café"), the denoising path changes and the face re-rolls with it. Seeds give you determinism for identical inputs; they give you nothing for new scenes.

Prompt-only ("the same woman as before"). The model has no memory. Every generation is a fresh sample from the distribution your words describe. "Same face as last time" isn't in the prompt vocabulary — there is no last time.

LoRA per character. This one actually works — that's why everyone suggests it — but look at what it costs in an app context: curate 15–40 images per character, run a training job per character, store and load adapter weights per character, and repeat all of it whenever a user creates someone new. For a personal project, fine. For an app where users create characters on demand, you just signed up to run a training farm.

The 2-stage pipeline

The fix is embarrassingly direct once you see it:

Stage 1 — CAST          Stage 2 — RE-SCENE (repeat forever)
text-to-image           image-edit model
"describe character" →  base image + "put them in a café" → scene 1
     = base image       base image + "walking in the rain" → scene 2
                        base image + "reading by a window"  → scene 3

Stage 1 runs a text-to-image model once (I use Z-Image-Turbo) to cast the character — one clean, canonical image. This is the identity. Store it.

Stage 2 runs an image-edit model (Qwen-Image-Edit-2511) for every scene after that. The crucial part: the input isn't a text description of the character — it's the base image itself, plus an instruction describing only what should change (pose, setting, lighting). The model's job is no longer "imagine this person" but "keep this person, change the scene."

Consistency stops being a property you hope the sampler preserves and becomes a property of the conditioning. The reference image is in the input every single time, so the face holds — across twenty scenes, fifty scenes, whatever.

Two details that matter in practice:

Describe only the change in Stage 2. If your scene prompt re-describes the character ("a woman with long dark hair…"), you're inviting the edit model to re-interpret identity. Prompt the delta: location, pose, lighting. Identity comes from the pixels, not the words.
This isn't people-only. The reference-edit mechanic doesn't care what the subject is — the same pipeline holds a consistent dog, robot, or anime character. Anything you can cast in Stage 1, you can re-scene in Stage 2.

The part nobody tells you: you can't call a GPU from a web request

Here's where most "it works in my notebook" projects die in deployment. A generation takes 10–40 seconds. An HTTP request does not want to live that long — serverless platforms cap request duration, browsers give up, and users retry and double-charge themselves.

The pattern that survives production:

client → POST /api/generate
             └── create Job row (status=queued) → return job id immediately
worker   picks up job → calls the GPU endpoint → writes image to storage
             └── Job row: status=completed, resultUrl
client → GET /api/status/:id   (poll)
             └── completed? render the image

The request that starts the job returns in milliseconds. The GPU work happens out-of-band. The client polls — a webhook or SSE is the usual alternative, but polling needs zero extra infrastructure (no public URL, identical local and deployed), so that's what I use. A few hard-won rules:

The job row is the source of truth, not the GPU vendor's job id. Vendors lose webhooks; your DB shouldn't care.
Make completion idempotent. Two racing polls — or a retry after a timeout — will happily complete the same job twice unless the finalize path claims the row atomically.
Store results in object storage and serve URLs, not bytes through your app server.

Hosted API or self-host? (honest version)

Let me be straight before the cost section, because "self-hosting is cheaper" as a blanket claim is false and you'd be right to call it out.

For most low-volume projects, a hosted image API is the correct choice. You skip GPU ops entirely, you pay per image, and at a few hundred generations a month the math favors the API — my setup would lose that comparison. If that's you, wrap the API and ship.

Self-hosting wins in three specific situations:

Volume. A hosted API's per-image price has the vendor's margin baked in — that margin is your cost floor, forever, on every image. Self-hosted, your unit cost is raw GPU-seconds; it doesn't inflate as you grow. Somewhere in the thousands-of-images-a-month range the lines cross, and past it the gap only widens.
Control. The vendor picks your model, your price, your rate limits, and what content gets filtered — and can change any of them under you. Self-hosted, an upstream deprecation or price hike is someone else's news.
Idle cost. Serverless GPU that scales to zero means a quiet app costs ~nothing to keep alive, which is what makes a side project survivable.

The trade is real: you take on packaging, cold starts, and ops. The rest of this section is how to make that trade cheap. (A follow-up post works through the actual break-even math with public prices — API per-image rates vs GPU-seconds — so you can find your own crossover point.)

Keeping it cheap: serverless GPU, scale-to-zero, bake the model

Self-hosting the pipeline on serverless GPU (I run RunPod) means you pay GPU-seconds only while a job runs, and workers scale to zero when idle.

The catch is cold start. When a worker wakes up, it has to get the model into VRAM before it can do anything. Three things cut that down hard:

Bake the model into the container image. Downloading tens of GB of weights at boot is where cold starts go to die. Weights in the image = pull once, cached on the host.
Slim the runtime. Build toolchains and dev dependencies don't belong in an inference image. Every GB you remove is faster pulls and faster boots.
One model per endpoint. A fat "does everything" image pays everyone's cold start. Per-endpoint images only load what that endpoint runs.

None of this is exotic — it's just deliberate packaging. The difference between a hobby deployment and one you can put real users on is mostly these three decisions.

Try the transport layer (free, zero infra)

I extracted the whole async transport — queue, worker, polling, serving — into a free slice you can run on a laptop in about two minutes: SQLite instead of Postgres, a stub worker instead of the GPU, and the real RunPod dispatch code included read-only so you can see exactly how the live version differs.

git clone https://github.com/ownstackhq/ai-image-job-queue
npm install && npx prisma db push && npm run dev

Type a prompt, watch the job go queued → processing → completed while the original request has long since returned. The 2-stage engine itself (the Stage 1/Stage 2 logic above, running on real GPU) lives in the full kit at ownstackhq.com, along with the deploy scripts and the cost-tuning chapter.

I'd genuinely like to hear where this approach breaks for your use case — especially if you've pushed reference-edit consistency further than twenty scenes, or found a case where per-character LoRA is still worth the training farm.

FAQ

Why not just reuse a seed?
A seed fixes the sampling noise, so the same prompt reproduces the same image. Change the prompt to get a new scene and the whole denoising path changes with it — including the face. Seeds give reproducibility for identical inputs, not identity across different ones.

Why not train a LoRA per character?
LoRA holds identity well, but it's a training job per character: dataset curation, GPU training time, and adapter storage, repeated for every character your users create. Reference-based editing needs one generated image and no training, which is what makes it viable inside an app rather than a workflow.

How do you keep GPU costs down?
Serverless GPU workers that scale to zero, model weights baked into the container image (no boot-time downloads), a slim runtime image, and one model per endpoint. Idle costs approach zero and cold starts drop dramatically.

Does this work for non-human characters?
Yes. The pipeline conditions on a reference image, not on a human-specific representation — the same cast-then-edit loop holds a consistent animal, robot, or anime character across scenes.