Symptom: you submit a job, it sits in IN_QUEUE and never moves. The endpoint looks healthy in the console. Workers keep starting and dying, so it reads like a queue problem rather than a crash.
The usual advice is "check your handler logs," which is right but doesn't tell you what to look for. Here is a specific failure mode I lost time to and haven't seen written down.
The setup that causes it
If you bake model weights into your image, you probably also pin a revision so an upstream change can't silently swap them under you. Reasonable. The trap is that the repo id and the revision usually end up managed in two different places:
- the repo id comes from an env var on the RunPod template
- the revision is pinned in code, next to the loader
Change only the repo id, say you point MODEL_ID at the original repo instead of the quantized mirror you actually baked, and you have created a combination that does not exist: a commit hash that only lives on the mirror, requested against a different repo.
What actually happens at boot
The load misses the local cache (that repo was never baked), falls back to the network, asks for a revision that repo never had, and 404s during startup. The worker dies before your handler is ever reached.
From the outside this looks nothing like a crash. The endpoint reports healthy, workers cycle, and the job just sits in the queue.
How to check
Open a worker's log, not the endpoint status. Look for a fetch attempt at boot: if a baked image is reaching the network for weights at all, something already broke. Then compare the repo id in the template env against the exact repo and revision your Dockerfile downloaded.
The takeaway
If weights are baked, treat repo id and revision as one unit. Pin both in the same place, or set neither at runtime. Splitting them across a template env and a code constant is how you end up asking for a combination that has never existed anywhere.
Top comments (0)