HuggingFace's Large File Downloads Keep Stopping — Resuming with curl for Reliable Retrieval

📝 Originally published (in Japanese) at forge.workstyle.tech.

Development of a Voice Conversion App: Overcoming Seed-VC Model Download Issues

When developing a voice conversion app, I encountered an issue while trying to download the 44.1kHz Seed-VC model set (DiT weights, rmvpe, and BigVGAN vocoder) using huggingface_hub.

When attempting to download models using huggingface_hub, the transfer would stop halfway through large files (hundreds of MB) and never progress.

While small config.json files would download quickly, larger .pth or .pt files would get stuck at a certain point and freeze. This article describes how to use curl's stagnation detection and automatic resumption to ensure successful downloads, and create a custom Hugging Face cache structure to enable model reading even when HF_HUB_OFFLINE=1.

Symptoms: Only Large Files Freeze Silently

On my machine, large file transfers using hf_hub_download would stall halfway through. The suspected causes include IPv6 route unavailability and connection drops over extended periods. The issues are:

• It doesn't fail with an exception (and retry doesn't work)
• The progress bar freezes
• Small files don't reproduce the issue

This isn't a matter of a slow network, but rather large transfers dying under specific conditions, so simply increasing the retry count won't solve the problem.

Approach: Use curl for Transfers and Create a Custom HF-Compatible Structure

The appeal of huggingface_hub lies in its cache management, not just as a downloader. load_custom_model_from_hf and from_pretrained assume a specific directory structure (blob entities + symbolic links + refs) and will re-download the model if this structure is broken.

To address this, I adopted a two-step approach:

1. Use curl to handle transfers (with stagnation detection and automatic resumption)
2. Manually configure the downloaded entities in an HF cache-compatible structure (enabling subsequent references by hf_hub)

Metadata (etag, commit hash, size, and entity URL) can be retrieved from the huggingface_hub API to construct the cache structure.

Using curl: Detecting Stagnation and Automatic Resumption

The key is combining curl options:

subprocess.run(
    ["curl", "-L", "--fail", "-C", "-",
     "--retry", "50", "--retry-delay", "3", "--retry-all-errors",
     "--speed-time", "20", "--speed-limit", "2000",   # 20 seconds at 2KB/s or lower, consider transfer failed and retry
     "-o", blob, loc],
    check=True,
)

The roles of each option are as follows:

• --speed-time 20 --speed-limit 2000 — Consider transfer failed and retry if 20 seconds pass at a speed of 2KB/s or lower. This proactively cuts off silent stagnation.
• -C - — Resume download from where it left off (Range request). This avoids restarting from scratch each time.
• --retry 50 --retry-delay 3 --retry-all-errors — Retry after 3 seconds, up to 50 times. Transfers cut off by --speed-time are also retried.
• -L (follow redirects) and --fail (non-zero exit on HTTP errors).

This combination automatically loops through "stagnation → cut → resume" and eventually completes the transfer. This was the most reliable method in environments where large files would freeze.

Creating a Custom HF Cache Structure

The core of the Hugging Face cache (e.g., ~/.cache/huggingface/hub) has the following structure per repository:

models--{org}--{repo}/
├── blobs/
│   └── {etag}                      # File entity, named by etag (content hash)
├── snapshots/
│   └── {commit_hash}/
│       └── {filename}              # Symbolic link to blob
└── refs/
    └── main                        # Branch name → commit hash text

The key points are:

• Entities are stored in blobs/ with etag names. This design allows a single entity to serve multiple revisions.
• snapshots/{commit}/ contains symbolic links to blobs. User code and libraries access these human-readable file names.
• refs/main contains the commit hash. This enables resolution for revision="main" and allows hf_hub to find the correct snapshot even when HF_HUB_OFFLINE=1.

Metadata can be obtained from the huggingface_hub API:

from huggingface_hub import hf_hub_url, get_hf_file_metadata

url = hf_hub_url(repo_id, filename, revision=revision)
m = get_hf_file_metadata(url)
etag = (m.etag or "").strip('"')   # Becomes the blob file name
commit = m.commit_hash or revision # Becomes the snapshots/ directory name
size = m.size                       # Used for download completion detection
loc = m.location or url             # Entity URL (redirected)

The rest involves using curl to download the entity to blobs/{etag}, creating a relative symbolic link from snapshots/{commit}/{filename}, and writing the commit to refs/main. The relative link ensures the cache remains intact even when moved to another machine.

os.symlink(os.path.relpath(blob, os.path.dirname(snap)), snap)
with open(os.path.join(refsdir, "main"), "w") as f:
    f.write(commit)

If the size is known, checking it against the existing blob size and skipping if complete can make re-runs idempotent.

Pitfalls and Lessons Learned

Download Only Necessary Files

Some repositories, like BigVGAN, contain large unnecessary files for inference (e.g., discriminator or optimizer states). Downloading the entire repository can result in getting stuck with the largest file. Specify the necessary files explicitly to avoid this.

Create refs for Offline Use

Failing to create refs/main alongside snapshots and blobs will prevent hf_hub from resolving the correct commit when HF_HUB_OFFLINE=1, even if the cache exists. Always create the three-part set (blob, snapshot link, and refs).

HEAD Verification May Require Online Access

Even with a cache, hf_hub performs a HEAD request at startup to check for updates (without re-downloading large files). To ensure complete offline functionality, set HF_HUB_OFFLINE=1 or TRANSFORMERS_OFFLINE=1. Conversely, if you want to see updates but prevent downloads, you can leave it online and just allow HEAD requests.

Summary

• In environments where large hf_hub transfers freeze silently, use curl's --speed-time and --speed-limit to proactively cut off stagnation, and -C - with --retry for resumption.
• Configure downloaded entities in an HF cache-compatible structure (blobs/{etag} + snapshots/{commit}/ with relative symbolic links + refs/main).
• Metadata (etag, commit, size, location) can be retrieved using get_hf_file_metadata.
• Create relative symbolic links to ensure the cache remains valid even when moved.
• Explicitly exclude unnecessary large files (like discriminators or optimizers) from downloads.
• For complete offline functionality, ensure refs/main is created to enable HF_HUB_OFFLINE=1.