DEV Community: Suryansh Chaudhary

I deleted the smartest feature in my download manager

Suryansh Chaudhary — Thu, 30 Jul 2026 09:23:41 +0000

In 1.3, I deleted half of it. The half I was proudest of.

The feature

Downloads opened at 4 connections. Every 3 seconds the engine added one more,
measured aggregate throughput, and kept the new connection only if throughput
improved by at least 15%. Otherwise it dropped back and stopped climbing.

Read that back. It sounds like the responsible thing to do. It measures instead
of assuming. It has a control loop. It even has a tunable threshold.

The bug

"Throughput didn't improve by 15%" has two completely different causes, and the
probe couldn't tell them apart:

The host is refusing to serve more connections. → Stop climbing. Correct.
Your link is already saturated. → Also stop climbing. Catastrophically wrong.

On a 100 Mbit connection pulling from a fast CDN, four connections will happily
saturate the pipe. Connection five adds nothing measurable — not because the
server refused it, but because there's no headroom left to measure it with.
The probe reads a flat line and concludes the ceiling is 4.

Then the link frees up. Another download finishes, the VPN reconnects, whatever.
Now there is headroom — and the probe already stopped climbing. It settled low
precisely in the case where extra connections were free.

The heuristic's failure mode was invisible because it never errored. It just
quietly returned a smaller number than it should have, forever, and every
download looked "fine."

The fix: back off on evidence, not on absence of evidence

A download now opens at its full effective thread count immediately, staggered
100 ms apart so a burst of SYNs doesn't trip anti-abuse middleboxes. No ramp, no
probe.

Backing off is still there, but it now requires a positive signal instead of a
missing one. DemotionPolicy halves the worker count when:

4 chunk attempts fail without progress
inside a 10-second window
and bytes are still flowing on the other connections

That last condition is the whole fix. It's what separates a hostile host —
refusing some connections while happily serving the rest — from a dead local
link, where everything fails at once because your Wi-Fi dropped.

Without it, a 3-second network blip wrote a permanent per-host cap against a
server that had done nothing wrong. Learned caps now also expire after 7 days and
are clearable from Settings, so a cap mislearned during an outage heals itself.

The lesson I keep re-learning: a control loop that treats "I measured nothing"
as "there is nothing" will always fail toward doing less. If your only feedback
signal is absence, you don't have feedback.

Endgame chunk splitting

Related tail problem. MacGet slices range-capable downloads into more pieces than
workers (8 MB target, up to 256 pieces), so a finished worker steals the next
outstanding piece instead of idling.

That works right up until every piece is assigned. Then the freed workers have
nothing to steal, and the download waits on whichever single piece landed on the
slowest path.

So now, when there's nothing left to steal, a freed worker splits the largest
in-flight piece in half and takes the back half — BitTorrent's endgame mode,
basically, applied to HTTP ranges. Floored at 1 MB so you don't spend a fresh TCP
handshake and an HTTP round trip on a 40 KB scrap.

SlotFiller.next(...) →
  .steal(piece)   // an outstanding piece exists
  .split(piece)   // nothing outstanding: halve the largest in-flight one
  .none           // everything is too small to bother splitting

The last slow chunk no longer sets the finish time.

Book catalogs, and three APIs that didn't work

1.3 also adds a catalog browser (⇧⌘B): search a book catalog, pick a format, it
goes to the download queue like any other URL. The interesting part was that the
obvious integration path was wrong for all three sources — and I only found out
by hitting the live services.

Project Gutenberg. It publishes OPDS, so: use OPDS, right? Its search feed
returns navigation entries — one sub-feed per result — so rendering a 50-book
grid with download links would cost 50 extra requests. Those per-book feeds were
also returning 504s, and the OPDS 2.0 endpoint is a 404. Gutendex returns every
format's direct URL inline, so that's what the grid uses.

Internet Archive. IA's OPDS BookServer doesn't just return errors —
bookserver.archive.org no longer resolves at all. Replaced with
advancedsearch.php?output=json for browse plus metadata/<id>/files for the
per-item file list, fetched lazily on selection.

Standard Ebooks. Every OPDS feed returns 401 to anonymous clients; access is a
donor benefit. It ships as a built-in source that's disabled by default, which
forced a small design point: CatalogStore has to track both explicitly-enabled
and explicitly-disabled built-ins, so a shipped default can apply until the user
actually expresses a preference.

Everything still normalizes into one CatalogFeed model, so the UI never branches
on source. And AcquisitionLink.isDownloadable is a single gate — direct-download
rel, no price, http(s), known format. DRM fulfilment documents get parsed and
displayed but never fetched, because that href is a license token, not a book.
Archive.org lists LCP Encrypted EPUB right next to the free files.

One more: catalog requests use a different URLSession than downloads do. The
download session sets waitsForConnectivity = true and
timeoutIntervalForResource = .infinity, which is exactly right for a 6 GB file
and exactly wrong for a metadata fetch behind a spinner.

BitTorrent

Magnets and .torrent files now run through the same queue as everything else —
same concurrency limit, same pause/resume, same quiet hours. It's off by default
and asks once before enabling, because unlike every other download type it
uploads on your connection and puts your IP in front of a swarm.

Two things worth stealing if you ever wrap aria2:

MacGet owns the queue, aria2 doesn't. --save-session and --input-file are
deliberately absent. queue.json is the source of truth and MacGet re-adds
torrents itself on launch; if aria2 also restored its own session, every info hash
would be registered twice and aria2 fails the duplicate outright. Resume comes
from --continue=true plus the .aria2 control file.

Magnets have a metadata phase that looks exactly like success. The first GID
downloads only the metainfo — a few hundred KB — and then reports complete with
completedLength == totalLength. Indistinguishable from a finished download
unless you're looking for it. An isAwaitingMetadata flag suppresses completion
and byte reporting until followedBy hands off to the real GID. Get it wrong and
your 6 GB torrent shows "Completed" at 500 KB.

Also: bind ports as a range (6881-6890), not a single value. aria2 hard-fails
with "Errors occurred while binding port" the moment another client holds it.

Install

brew install --cask suryansh-codes2209/macget/macget

Or grab the DMG: https://macget.suryansh.work

MIT licensed, and the engine is genuinely fun to poke at — clone it, open
Macget.xcodeproj, ⌘R. No package manager step, no codegen:
https://github.com/Suryansh-Codes2209/Macget

Issues and PRs welcome. I'm especially interested in ideas for what belongs in a
download manager in 2026 — if you've got a feature you've always wanted and never
found, open an issue.

And if you've ever shipped a heuristic that looked smart and was quietly costing
you something, I'd like to hear about it. I clearly can't be trusted to spot my
own.

Why more download threads make your downloads slower (and how I fixed it)

Suryansh Chaudhary — Mon, 29 Jun 2026 07:07:16 +0000

There's a myth baked into almost every "download accelerator": open more connections, download faster. It's intuitive, and it's wrong for most of the modern web.

I learned this the hard way building MacGet, a free, open-source, native macOS download manager — and the fix turned into the most interesting part of the project.

The problem: parallelism looks like an attack

When you split a file into N chunks and open N HTTP-Range connections at once, a naive downloader assumes the server will happily serve all of them. Modern CDNs don't. To them, one IP suddenly opening 16 connections and pulling ranges looks exactly like leech/abuse behavior. So they fight back:

TCP-RST your connections after a few bytes
throttle your IP for a while
403 new requests outright

The result is counterintuitive: crank the thread count up and your download gets slower, or fails entirely. More threads ≠ more speed.

The fix: discover each host's real capacity at runtime

Instead of trusting a fixed thread count, MacGet's engine treats parallelism as something to discover per host:

Adaptive up-scaling. Downloads start at 4 connections and probe upward one at a time, keeping each added connection only when aggregate throughput improves by ≥15%. So it climbs toward the host's real ceiling instead of guessing.
Demotion on rejection. When ≥4 chunk attempts fail without making progress inside a 10-second window the signal that the host is rejecting parallelism the engine halves its worker count, cancels the lowest-progress chunks, and carries on. It repeats until it stabilizes at a level the host actually allows.
Per-host memory. That learned cap is persisted (host_caps.json). The next download from the same host starts at the right level — no rediscovery tax. Caps only ratchet downward.
Staggered spawns. Workers start ~100ms apart, so anti-abuse middleboxes see a steady ramp instead of a SYN burst from one IP.
Smart retry classification. Permanent failures (401/403/404/410/451, range refusals, malformed responses) fail fast. Transient ones (mid-stream RSTs, server-side stream kills, 5xx) retry with full-jitter exponential backoff under a hard cap — and 429/503 honor the server's Retry-After.

There's also a macOS-specific gotcha: App Nap. If you switch apps, macOS will happily throttle your "background" download into the ground. MacGet holds a ProcessInfo activity assertion while any download runs, so the OS leaves it alone.

Killing the slow-chunk tail

Even when a host allows N connections, fixed N-way chunking has a long tail: split a file into N equal pieces and the whole download waits on whichever piece landed on the slowest path. MacGet slices range-capable downloads into more pieces than workers (8 MB target), and a finished worker immediately steals the next
outstanding piece. No worker sits idle while another grinds through the tail.

How it's built

The engine is modeled with Swift's actor concurrency, which made the
correctness story dramatically easier than locks-and-queues would have:

DownloadEngine (actor)
  └─ DownloadCoordinator (actor)   // one per download: probe → plan → stream → finalize
       ├─ ChunkWorker              // one HTTP-Range request, streamed via AsyncThrowingStream
       └─ FileWriter (actor)       // serializes positional writes so chunks don't race

A few more design notes:

Probe first. A HEAD request (falling back to GET Range: bytes=0-0, since some servers 405 on HEAD) establishes size, Accept-Ranges, ETag, and Last-Modified.
HTTP/3 when offered. Requests opt into QUIC via assumesHTTP3Capable, with graceful fallback to HTTP/2 → HTTP/1.1.
Integrity at finalize. SHA-256 / MD5 is verified before the partial is promoted to the final filename; a mismatch fails the download and keeps the partial instead of handing you a corrupt file.
Sparse partials. The partial file is truncated to full size up front; APFS keeps it sparse until bytes actually land.
Resumable across restarts. Per-chunk byte offsets are persisted, and workers send the recorded ETag/Last-Modified as If-Range so a file that changed server-side fails fast instead of silently corrupting your partial.
Survives a dropped connection. An NWPathMonitor pauses active downloads on network loss ("Waiting for network…") and auto-resumes when it's back, instead of burning retries through an outage.
Live speed/ETA. A 3-second rolling window over (time, bytes) samples; ETA goes nil below 1 KB/s instead of lying to you.

The whole thing is SwiftUI on top, with @Observable view models draining an AsyncStream of engine events — the UI never mutates download state directly, only through the engine actor.

Beyond the engine

The latest release (1.2) builds a real download manager around that core:

Video & audio downloads. A conservative host classifier routes known video/audio sites to a bundled yt-dlp + ffmpeg extractor (with a quality/format picker); ordinary links still download as plain HTTP through the engine above.
Authenticated downloads. A Keychain-backed credential store answers Basic/Digest/NTLM challenges and remembers them per host across launches.
A browser extension (Chrome/Edge/Brave/Firefox) that hands off downloads you start in the browser — carrying the cookies, referrer, and user-agent — so logged-in downloads actually work.
Auto-sort finished files into category folders, High/Normal/Low priorities, bandwidth throttling, and signed auto-updates via Sparkle (EdDSA-verified — a tampered update is rejected).

Try it

Just want to use it? Download the DMG: https://suryansh.work/macget (free, un-notarized — right-click → Open on first launch for the one-time Gatekeeper step).
Want to read or hack the code? It's MIT licensed and the engine is genuinely fun to poke at: https://github.com/Suryansh-Codes2209/Macget — clone, open Macget.xcodeproj, ⌘R. No setup.

Unleashing the Power of Django on AWS: Best Practices for Seamless Deployment!

Suryansh Chaudhary — Tue, 30 Jan 2024 04:48:26 +0000

Hey Dev Community! 💻 Excited to share some golden nuggets on deploying Django web apps on AWS. Let's make your deployment journey smoother with these best practices:
**
**AWS App Runner Magic: Dive into the simplicity of deploying and scaling Django apps on AWS App Runner. Check out this AWS blog for a quick guide.

Elastic Beanstalk Brilliance: Discover the art of deploying Django apps on Elastic Beanstalk with AWS's official guide here. It's like a magic wand for streamlined deployment.

Kubernetes Mastery: Elevate your Django deployment game by containerizing with Kubernetes. Explore the scalability and security in this DigitalOcean tutorial.

PostgreSQL Power: Level up your Django app by integrating PostgreSQL on AWS Elastic Beanstalk. Learn the ropes in this insightful Real Python tutorial.

Remember, the key to a successful deployment is a mix of AWS services tailored to your app's needs. Keep it efficient, keep it scalable! 🔗✨

Django #AWS #DevOps #DeploymentMastery

AWS Elastic Beanstalk: ⚡Simplifying Web Application Deployment and Scaling

Suryansh Chaudhary — Mon, 15 Jan 2024 11:34:02 +0000

AWS Elastic Beanstalk is a fully managed service that streamlines the deployment and scaling of web applications.

Easy Deployment: Simply upload your code, and Elastic Beanstalk takes care of the deployment process. It eliminates the need for manual intervention in tasks like capacity provisioning and configuration.

Automatic Scaling: Elastic Beanstalk dynamically scales your application based on demand. It leverages Auto Scaling, ensuring that your application can handle varying traffic loads efficiently.

Load Balancing: The service includes Elastic Load Balancing, distributing incoming traffic across multiple instances to enhance application availability and fault tolerance.

Health Monitoring: Elastic Beanstalk continuously monitors the health of your application. It automatically replaces unhealthy instances, ensuring high availability and reliability.

Traffic-Splitting Deployments: With features like traffic-splitting deployments, Elastic Beanstalk minimizes downtime by creating a new set of instances for updates while preserving the existing environment.

In summary, AWS Elastic Beanstalk abstracts the complexities of infrastructure management, allowing developers to focus on building robust applications without worrying about the intricacies of deployment, scaling, and monitoring.

Launching 🚀Your AWS Application Journey with EC2: A Student's Guide⚡

Suryansh Chaudhary — Mon, 15 Jan 2024 11:13:24 +0000

Are you a student eager to embark on your AWS application journey? Start by mastering Amazon EC2, a fundamental service that provides virtual machines in the cloud. Here's a step-by-step guide to kickstart your production-making adventure:

Set Up Your AWS Account:
Begin by creating an AWS account. Follow the official guide to ensure a seamless setup.

Launch Your First EC2 Instance:
Learn the basics of launching an EC2 instance with this tutorial. Choose an Amazon Machine Image (AMI), select an instance type, and configure your settings.

Understand Virtual Machines:
Delve into the world of virtualization. Understand how EC2 instances function as virtual machines, providing scalable computing power for your applications.

Explore CodeDeploy for Deployments:
Familiarize yourself with CodeDeploy, a service that automates application deployments. Follow this guide for deployments on EC2.

Enhance Security with Security Groups:
Master the art of managing security groups to secure your EC2 instances. Follow best practices outlined in the official documentation.