On serverless GPU, cold start is image pull plus model load. The single biggest win is usually to stop downloading weights at boot: download them at build time so they ship inside the image and load from local disk instead.
The setup is two lines. Point the cache at a path inside the image, then fetch during the build:
ENV HF_HOME=/opt/huggingface
RUN python -c "from huggingface_hub import snapshot_download; snapshot_download('<repo>', revision='<pinned>')"
snapshot_download writes into HF_HOME, that directory becomes an image layer, and at runtime the loader finds everything already on disk.
The failure that doesn't look like a failure
Here's the part worth knowing. The loader resolves weights through HF_HOME at runtime. So if anything in your runtime environment sets HF_HOME to a different path — a network volume, a mount, a leftover template variable — the loader looks there, finds an empty cache, and quietly falls back to downloading from the network.
Nothing errors. Your image still contains the weights, sitting in a directory nobody reads anymore. You just paid to bake them, and you're paying the download on every cold start anyway. The only symptom is that cold starts went back to what they were before you did the work, which is easy to blame on the platform.
How to make it hard to get wrong
Attach no volume at all. If the worker has no volume, there's no tempting path to point the cache at. In my setup the endpoints run with volume size zero for exactly this reason: everything the worker needs is in the image, so a volume is not an optimization, it's a way to reintroduce the network.
Verify at boot, not by feel. Watch a cold worker's log. If you see download progress for weights, the bake is being bypassed, full stop. A baked image reaching the network for weights is always a misconfiguration, never a normal path.
Treat the cache path as part of the build contract. The Dockerfile that baked the weights and the runtime env that reads them have to agree. That agreement is invisible in both files individually, so write it down where whoever edits the template env will see it.
Worth saying out loud
Baking has real costs: the image gets large, builds take much longer, and updating the model means rebuilding and redistributing the whole thing. If your weights are small or your traffic keeps workers warm, a volume or a runtime fetch may genuinely be the better trade. This is about the case where you already chose to bake — because then paying the build cost and the download cost is the worst of both.
Top comments (1)
The "attach no volume at all" advice is underrated — removing the tempting path is a much stronger guarantee than documenting the right one, because config drift beats documentation every time.
The bit I'd amplify: the failure is silent and it's cost-shaped, not error-shaped, so it survives every health check you have. The image builds green, the container boots, requests succeed — you just quietly pay the download on every cold start and blame the platform's pull times. The only honest signal is exactly what you said: watch a cold worker's log for weight-download progress.
Two guardrails we've leaned on: (1) fail-fast at boot with a
local_files_only=Trueload, so a bypassed bake throws instead of silently reaching the network — turn the invisible cost into a loud crash; (2) log a checksum/path of what actually loaded, so "did it use the baked copy" is a field you can alert on, not a vibe. Have you triedHF_HUB_OFFLINE=1as a belt-and-suspenders over the empty-volume approach?