DEV Community

Cover image for Why more download threads make your downloads slower (and how I fixed it)
Suryansh Chaudhary
Suryansh Chaudhary

Posted on

Why more download threads make your downloads slower (and how I fixed it)

#macos #swift #opensource #showdev

There's a myth baked into almost every "download accelerator": open more connections, download faster. It's intuitive, and it's wrong for most of the modern web.

I learned this the hard way building MacGet, a free, open-source, native macOS download manager — and the fix turned into the most interesting part of the project.

The problem: parallelism looks like an attack

When you split a file into N chunks and open N HTTP-Range connections at once, a naive downloader assumes the server will happily serve all of them. Modern CDNs don't. To them, one IP suddenly opening 16 connections and pulling ranges looks exactly like leech/abuse behavior. So they fight back:

  • TCP-RST your connections after a few bytes
  • throttle your IP for a while
  • 403 new requests outright

The result is counterintuitive: crank the thread count up and your download gets slower, or fails entirely. More threads ≠ more speed.

The fix: discover each host's real capacity at runtime

Instead of trusting a fixed thread count, MacGet's engine treats parallelism as something to discover per host:

  • Adaptive up-scaling. Downloads start at 4 connections and probe upward one at a time, keeping each added connection only when aggregate throughput improves by ≥15%. So it climbs toward the host's real ceiling instead of guessing.

  • Demotion on rejection. When ≥4 chunk attempts fail without making progress inside a 10-second window the signal that the host is rejecting parallelism the engine halves its worker count, cancels the lowest-progress chunks, and carries on. It repeats until it stabilizes at a level the host actually allows.

  • Per-host memory. That learned cap is persisted (host_caps.json). The next download from the same host starts at the right level — no rediscovery tax. Caps only ratchet downward.

  • Staggered spawns. Workers start ~100ms apart, so anti-abuse middleboxes see a steady ramp instead of a SYN burst from one IP.

  • Smart retry classification. Permanent failures (401/403/404/410/451, range refusals, malformed responses) fail fast. Transient ones (mid-stream RSTs, server-side stream kills, 5xx) retry with full-jitter exponential backoff under a hard cap — and 429/503 honor the server's Retry-After.

There's also a macOS-specific gotcha: App Nap. If you switch apps, macOS will happily throttle your "background" download into the ground. MacGet holds a ProcessInfo activity assertion while any download runs, so the OS leaves it alone.

Killing the slow-chunk tail

Even when a host allows N connections, fixed N-way chunking has a long tail: split a file into N equal pieces and the whole download waits on whichever piece landed on the slowest path. MacGet slices range-capable downloads into more pieces than workers (8 MB target), and a finished worker immediately steals the next
outstanding piece. No worker sits idle while another grinds through the tail.

How it's built

The engine is modeled with Swift's actor concurrency, which made the
correctness story dramatically easier than locks-and-queues would have:

DownloadEngine (actor)
  └─ DownloadCoordinator (actor)   // one per download: probe → plan → stream → finalize
       ├─ ChunkWorker              // one HTTP-Range request, streamed via AsyncThrowingStream
       └─ FileWriter (actor)       // serializes positional writes so chunks don't race
Enter fullscreen mode Exit fullscreen mode

A few more design notes:

  • Probe first. A HEAD request (falling back to GET Range: bytes=0-0, since some servers 405 on HEAD) establishes size, Accept-Ranges, ETag, and Last-Modified.

  • HTTP/3 when offered. Requests opt into QUIC via assumesHTTP3Capable, with graceful fallback to HTTP/2 → HTTP/1.1.

  • Integrity at finalize. SHA-256 / MD5 is verified before the partial is promoted to the final filename; a mismatch fails the download and keeps the partial instead of handing you a corrupt file.

  • Sparse partials. The partial file is truncated to full size up front; APFS keeps it sparse until bytes actually land.

  • Resumable across restarts. Per-chunk byte offsets are persisted, and workers send the recorded ETag/Last-Modified as If-Range so a file that changed server-side fails fast instead of silently corrupting your partial.

  • Survives a dropped connection. An NWPathMonitor pauses active downloads on network loss ("Waiting for network…") and auto-resumes when it's back, instead of burning retries through an outage.

  • Live speed/ETA. A 3-second rolling window over (time, bytes) samples; ETA goes nil below 1 KB/s instead of lying to you.

The whole thing is SwiftUI on top, with @Observable view models draining an AsyncStream of engine events — the UI never mutates download state directly, only through the engine actor.

Beyond the engine

The latest release (1.2) builds a real download manager around that core:

  • Video & audio downloads. A conservative host classifier routes known video/audio sites to a bundled yt-dlp + ffmpeg extractor (with a quality/format picker); ordinary links still download as plain HTTP through the engine above.

  • Authenticated downloads. A Keychain-backed credential store answers Basic/Digest/NTLM challenges and remembers them per host across launches.

  • A browser extension (Chrome/Edge/Brave/Firefox) that hands off downloads you start in the browser — carrying the cookies, referrer, and user-agent — so logged-in downloads actually work.

  • Auto-sort finished files into category folders, High/Normal/Low priorities, bandwidth throttling, and signed auto-updates via Sparkle (EdDSA-verified — a tampered update is rejected).

Try it

Top comments (0)