DEV Community: Suryansh Chaudhary

Why more download threads make your downloads slower (and how I fixed it)

Suryansh Chaudhary — Mon, 29 Jun 2026 07:07:16 +0000

There's a myth baked into almost every "download accelerator": open more connections, download faster. It's intuitive, and it's wrong for most of the modern web.

I learned this the hard way building MacGet, a free, open-source, native macOS download manager — and the fix turned into the most interesting part of the project.

The problem: parallelism looks like an attack

When you split a file into N chunks and open N HTTP-Range connections at once, a naive downloader assumes the server will happily serve all of them. Modern CDNs don't. To them, one IP suddenly opening 16 connections and pulling ranges looks exactly like leech/abuse behavior. So they fight back:

TCP-RST your connections after a few bytes
throttle your IP for a while
403 new requests outright

The result is counterintuitive: crank the thread count up and your download gets slower, or fails entirely. More threads ≠ more speed.

The fix: discover each host's real capacity at runtime

Instead of trusting a fixed thread count, MacGet's engine treats parallelism as something to discover per host:

Adaptive up-scaling. Downloads start at 4 connections and probe upward one at a time, keeping each added connection only when aggregate throughput improves by ≥15%. So it climbs toward the host's real ceiling instead of guessing.
Demotion on rejection. When ≥4 chunk attempts fail without making progress inside a 10-second window the signal that the host is rejecting parallelism the engine halves its worker count, cancels the lowest-progress chunks, and carries on. It repeats until it stabilizes at a level the host actually allows.
Per-host memory. That learned cap is persisted (host_caps.json). The next download from the same host starts at the right level — no rediscovery tax. Caps only ratchet downward.
Staggered spawns. Workers start ~100ms apart, so anti-abuse middleboxes see a steady ramp instead of a SYN burst from one IP.
Smart retry classification. Permanent failures (401/403/404/410/451, range refusals, malformed responses) fail fast. Transient ones (mid-stream RSTs, server-side stream kills, 5xx) retry with full-jitter exponential backoff under a hard cap — and 429/503 honor the server's Retry-After.

There's also a macOS-specific gotcha: App Nap. If you switch apps, macOS will happily throttle your "background" download into the ground. MacGet holds a ProcessInfo activity assertion while any download runs, so the OS leaves it alone.

Killing the slow-chunk tail

Even when a host allows N connections, fixed N-way chunking has a long tail: split a file into N equal pieces and the whole download waits on whichever piece landed on the slowest path. MacGet slices range-capable downloads into more pieces than workers (8 MB target), and a finished worker immediately steals the next
outstanding piece. No worker sits idle while another grinds through the tail.

How it's built

The engine is modeled with Swift's actor concurrency, which made the
correctness story dramatically easier than locks-and-queues would have:

DownloadEngine (actor)
  └─ DownloadCoordinator (actor)   // one per download: probe → plan → stream → finalize
       ├─ ChunkWorker              // one HTTP-Range request, streamed via AsyncThrowingStream
       └─ FileWriter (actor)       // serializes positional writes so chunks don't race

A few more design notes:

Probe first. A HEAD request (falling back to GET Range: bytes=0-0, since some servers 405 on HEAD) establishes size, Accept-Ranges, ETag, and Last-Modified.
HTTP/3 when offered. Requests opt into QUIC via assumesHTTP3Capable, with graceful fallback to HTTP/2 → HTTP/1.1.
Integrity at finalize. SHA-256 / MD5 is verified before the partial is promoted to the final filename; a mismatch fails the download and keeps the partial instead of handing you a corrupt file.
Sparse partials. The partial file is truncated to full size up front; APFS keeps it sparse until bytes actually land.
Resumable across restarts. Per-chunk byte offsets are persisted, and workers send the recorded ETag/Last-Modified as If-Range so a file that changed server-side fails fast instead of silently corrupting your partial.
Survives a dropped connection. An NWPathMonitor pauses active downloads on network loss ("Waiting for network…") and auto-resumes when it's back, instead of burning retries through an outage.
Live speed/ETA. A 3-second rolling window over (time, bytes) samples; ETA goes nil below 1 KB/s instead of lying to you.

The whole thing is SwiftUI on top, with @Observable view models draining an AsyncStream of engine events — the UI never mutates download state directly, only through the engine actor.

Beyond the engine

The latest release (1.2) builds a real download manager around that core:

Video & audio downloads. A conservative host classifier routes known video/audio sites to a bundled yt-dlp + ffmpeg extractor (with a quality/format picker); ordinary links still download as plain HTTP through the engine above.
Authenticated downloads. A Keychain-backed credential store answers Basic/Digest/NTLM challenges and remembers them per host across launches.
A browser extension (Chrome/Edge/Brave/Firefox) that hands off downloads you start in the browser — carrying the cookies, referrer, and user-agent — so logged-in downloads actually work.
Auto-sort finished files into category folders, High/Normal/Low priorities, bandwidth throttling, and signed auto-updates via Sparkle (EdDSA-verified — a tampered update is rejected).

Try it

Just want to use it? Download the DMG: https://suryansh.work/macget (free, un-notarized — right-click → Open on first launch for the one-time Gatekeeper step).
Want to read or hack the code? It's MIT licensed and the engine is genuinely fun to poke at: https://github.com/Suryansh-Codes2209/Macget — clone, open Macget.xcodeproj, ⌘R. No setup.

Unleashing the Power of Django on AWS: Best Practices for Seamless Deployment!

Suryansh Chaudhary — Tue, 30 Jan 2024 04:48:26 +0000

Hey Dev Community! 💻 Excited to share some golden nuggets on deploying Django web apps on AWS. Let's make your deployment journey smoother with these best practices:
**
**AWS App Runner Magic: Dive into the simplicity of deploying and scaling Django apps on AWS App Runner. Check out this AWS blog for a quick guide.

Elastic Beanstalk Brilliance: Discover the art of deploying Django apps on Elastic Beanstalk with AWS's official guide here. It's like a magic wand for streamlined deployment.

Kubernetes Mastery: Elevate your Django deployment game by containerizing with Kubernetes. Explore the scalability and security in this DigitalOcean tutorial.

PostgreSQL Power: Level up your Django app by integrating PostgreSQL on AWS Elastic Beanstalk. Learn the ropes in this insightful Real Python tutorial.

Remember, the key to a successful deployment is a mix of AWS services tailored to your app's needs. Keep it efficient, keep it scalable! 🔗✨

Django #AWS #DevOps #DeploymentMastery

AWS Elastic Beanstalk: ⚡Simplifying Web Application Deployment and Scaling

Suryansh Chaudhary — Mon, 15 Jan 2024 11:34:02 +0000

AWS Elastic Beanstalk is a fully managed service that streamlines the deployment and scaling of web applications.

Easy Deployment: Simply upload your code, and Elastic Beanstalk takes care of the deployment process. It eliminates the need for manual intervention in tasks like capacity provisioning and configuration.

Automatic Scaling: Elastic Beanstalk dynamically scales your application based on demand. It leverages Auto Scaling, ensuring that your application can handle varying traffic loads efficiently.

Load Balancing: The service includes Elastic Load Balancing, distributing incoming traffic across multiple instances to enhance application availability and fault tolerance.

Health Monitoring: Elastic Beanstalk continuously monitors the health of your application. It automatically replaces unhealthy instances, ensuring high availability and reliability.

Traffic-Splitting Deployments: With features like traffic-splitting deployments, Elastic Beanstalk minimizes downtime by creating a new set of instances for updates while preserving the existing environment.

In summary, AWS Elastic Beanstalk abstracts the complexities of infrastructure management, allowing developers to focus on building robust applications without worrying about the intricacies of deployment, scaling, and monitoring.

Launching 🚀Your AWS Application Journey with EC2: A Student's Guide⚡

Suryansh Chaudhary — Mon, 15 Jan 2024 11:13:24 +0000

Are you a student eager to embark on your AWS application journey? Start by mastering Amazon EC2, a fundamental service that provides virtual machines in the cloud. Here's a step-by-step guide to kickstart your production-making adventure:

Set Up Your AWS Account:
Begin by creating an AWS account. Follow the official guide to ensure a seamless setup.

Launch Your First EC2 Instance:
Learn the basics of launching an EC2 instance with this tutorial. Choose an Amazon Machine Image (AMI), select an instance type, and configure your settings.

Understand Virtual Machines:
Delve into the world of virtualization. Understand how EC2 instances function as virtual machines, providing scalable computing power for your applications.

Explore CodeDeploy for Deployments:
Familiarize yourself with CodeDeploy, a service that automates application deployments. Follow this guide for deployments on EC2.

Enhance Security with Security Groups:
Master the art of managing security groups to secure your EC2 instances. Follow best practices outlined in the official documentation.