DEV Community

Runcrate
Runcrate

Posted on

Fresh Hugging Face download vs local cache: time to first token for a 7B model on a Runcrate L40S

#huggingface #llm #gpu #python

How long does a GPU machine sit idle before a 7B model answers its first request, and how much of that is the download? I rented one L40S on Runcrate today (2026-10-29), timed every phase of starting Qwen2.5-7B-Instruct from a fresh Hugging Face download and from a copy already on the machine's disk, and deleted the machine when the numbers were saved.

Short version: from a fresh download, the first token came 48 to 55 s after the Python process started. With the weights already on local disk it came after 18 to 19 s. The download was 30 s of that (15.2 GB at about 515 MB/s), and the part a cache cannot remove was the 14 s it took to read 15 GB from disk into GPU memory.

Setup

  • GPU: one Runcrate L40S 48 GB (Des Moines, US, $0.54 per hour on demand at launch time), 12 vCPUs, 70 GB RAM.
  • Model: Qwen/Qwen2.5-7B-Instruct, bf16 safetensors, 15.24 GB on disk (4 shards).
  • Software: torch 2.8.0+cu128, transformers 4.57.1, huggingface-hub 0.36.2 with hf-xet 1.7.1 (the default download backend), Python 3.12.
  • One fresh process per measurement, timing four phases: Python imports, snapshot_download (a real download, or a cache check when the files are present), from_pretrained(..., dtype=bfloat16, device_map="cuda"), and generating one token for a short chat prompt.
  • Six starts: two fresh downloads with the defaults, one fresh download with HF_XET_HIGH_PERFORMANCE=1, one from the local cache after evicting the files from the page cache, and two from the local cache right after a previous run.

The script 

import json, os, sys, time
T0 = time.time()
import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
r = {"import_s": round(time.time() - T0, 1)}
MID = "Qwen/Qwen2.5-7B-Instruct"
t = time.time(); path = snapshot_download(MID, allow_patterns=["*.json", "*.safetensors", "*.txt"])
r["download_s"] = round(time.time() - t, 1)
t = time.time(); tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map="cuda")
torch.cuda.synchronize(); r["load_to_gpu_s"] = round(time.time() - t, 1)
t = time.time()
ids = tok.apply_chat_template([{"role": "user", "content": "Say hello in five words."}],
                              add_generation_prompt=True, return_tensors="pt").to("cuda")
model.generate(ids, max_new_tokens=1, do_sample=False)
torch.cuda.synchronize(); r["first_token_s"] = round(time.time() - t, 2)
r["total_s"] = round(time.time() - T0, 1)
print(json.dumps(r))
Enter fullscreen mode Exit fullscreen mode

A fresh start is rm -rf $HF_HOME/hub/models--Qwen--Qwen2.5-7B-Instruct before the run. To evict a cached copy from RAM without root (/proc/sys/vm/drop_caches was not writable in this container) I called os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) on every blob file.

Results

Start Python imports (s) Download (s) Load to GPU (s) First token (s) Total (s) GPU cost of the wait
Fresh download, defaults (run 1) 10.2 29.6 14.0 0.57 54.5 $0.0082
Fresh download, defaults (run 2) 3.7 29.6 13.7 0.54 47.6 $0.0071
Fresh download, HF_XET_HIGH_PERFORMANCE=1 4.0 32.5 14.1 0.50 51.1 $0.0077
Local disk cache, page cache evicted 3.7 0.1 13.9 0.54 18.3 $0.0027
Local disk cache, in page cache (run 1) 3.8 0.1 14.7 0.56 19.1 $0.0029
Local disk cache, in page cache (run 2) 3.8 0.1 14.9 0.53 19.4 $0.0029

GPU cost of the wait is the total time at the L40S's hourly price.

What the numbers say

  1. The download was 60% of a fresh start. 29.6 s in both default runs for 15.24 GB, about 515 MB/s or 4.1 Gbit/s. With the weights on disk the same phase took 0.1 s (a cache check).
  2. HF_XET_HIGH_PERFORMANCE=1 did not help here (32.5 s against 29.6 s). At 4 Gbit/s on 12 vCPUs the default settings already used what this machine's network gave.
  3. Loading to the GPU took 14 s every time, cached or not. That is 15.24 GB at about 1.1 GB/s, and dd read the largest shard at 1.0 GB/s from this disk, both right after eviction and right after a previous read. So on this machine the load is bound by disk reads, and keeping the file in RAM did not change it. I can't prove the eviction took effect, but the load time did not move either way.
  4. The first token itself took about half a second (0.50 to 0.57 s), including tokenizing the prompt and the first CUDA kernels. It is not where startup time goes.
  5. The first Python import of the session took 10 s; later ones 4 s. Bytecode compilation and cold library files cost 6 s once per machine.

What to take from it

  • If your machines start often, keep the weights on the machine (or on a disk you reattach) instead of pulling them from the Hub each time: here it cut time to first token from about 50 s to about 19 s.
  • After that, the floor is reading the weights. 15 GB at 1 GB/s is 14 s, so a 70B model in bf16 (about 140 GB) would need roughly two minutes just to load at the same rate, and the download about 4.5 minutes more at 515 MB/s. Both are extrapolations, not measured.
  • At $0.54 an hour, a fresh start's wait cost under a cent. The money matters less than the latency: a scale-to-zero endpoint that downloads on every cold start makes its first user wait close to a minute.

Caveats

  • One machine, one region, one day, one model. Network speed to the Hub varies by region and time of day; disk speed varies by machine.
  • My first script deleted the wrong cache directory (this image sets HF_HOME=/workspace/hf), so its second "fresh" run was served from cache. I reran both fresh downloads and the evicted start on the same machine before it was deleted; those are the numbers above, and the discarded runs are kept with the raw logs.
  • transformers from_pretrained only. vLLM and other servers add their own startup work (CUDA graphs, KV cache allocation) on top of the load.

Reproduce it

  1. Create an API key and add an SSH key (POST /api/v1/ssh-keys).
  2. Launch an L40S with POST /api/v1/instances ({"name": "...", "gpu_type": "L40S", "gpu_count": 1, "ssh_key_id": "...", "max_runtime_minutes": 40}), install torch and transformers==4.57.1 accelerate hf_xet with uv.
  3. Run the script above once after deleting the model's cache directory and once with it in place, then delete the instance.

The same machine also ran an embedding throughput test and a torch.compile training test; the whole run, about 12 minutes of billed L40S time, was billed $0.11.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Top comments (0)