In 1.3, I deleted half of it. The half I was proudest of.
The feature
Downloads opened at 4 connections. Every 3 seconds the engine added one more,
measured aggregate throughput, and kept the new connection only if throughput
improved by at least 15%. Otherwise it dropped back and stopped climbing.
Read that back. It sounds like the responsible thing to do. It measures instead
of assuming. It has a control loop. It even has a tunable threshold.
The bug
"Throughput didn't improve by 15%" has two completely different causes, and the
probe couldn't tell them apart:
- The host is refusing to serve more connections. → Stop climbing. Correct.
- Your link is already saturated. → Also stop climbing. Catastrophically wrong.
On a 100 Mbit connection pulling from a fast CDN, four connections will happily
saturate the pipe. Connection five adds nothing measurable — not because the
server refused it, but because there's no headroom left to measure it with.
The probe reads a flat line and concludes the ceiling is 4.
Then the link frees up. Another download finishes, the VPN reconnects, whatever.
Now there is headroom — and the probe already stopped climbing. It settled low
precisely in the case where extra connections were free.
The heuristic's failure mode was invisible because it never errored. It just
quietly returned a smaller number than it should have, forever, and every
download looked "fine."
The fix: back off on evidence, not on absence of evidence
A download now opens at its full effective thread count immediately, staggered
100 ms apart so a burst of SYNs doesn't trip anti-abuse middleboxes. No ramp, no
probe.
Backing off is still there, but it now requires a positive signal instead of a
missing one. DemotionPolicy halves the worker count when:
- 4 chunk attempts fail without progress
- inside a 10-second window
- and bytes are still flowing on the other connections
That last condition is the whole fix. It's what separates a hostile host —
refusing some connections while happily serving the rest — from a dead local
link, where everything fails at once because your Wi-Fi dropped.
Without it, a 3-second network blip wrote a permanent per-host cap against a
server that had done nothing wrong. Learned caps now also expire after 7 days and
are clearable from Settings, so a cap mislearned during an outage heals itself.
The lesson I keep re-learning: a control loop that treats "I measured nothing"
as "there is nothing" will always fail toward doing less. If your only feedback
signal is absence, you don't have feedback.
Endgame chunk splitting
Related tail problem. MacGet slices range-capable downloads into more pieces than
workers (8 MB target, up to 256 pieces), so a finished worker steals the next
outstanding piece instead of idling.
That works right up until every piece is assigned. Then the freed workers have
nothing to steal, and the download waits on whichever single piece landed on the
slowest path.
So now, when there's nothing left to steal, a freed worker splits the largest
in-flight piece in half and takes the back half — BitTorrent's endgame mode,
basically, applied to HTTP ranges. Floored at 1 MB so you don't spend a fresh TCP
handshake and an HTTP round trip on a 40 KB scrap.
SlotFiller.next(...) →
.steal(piece) // an outstanding piece exists
.split(piece) // nothing outstanding: halve the largest in-flight one
.none // everything is too small to bother splitting
The last slow chunk no longer sets the finish time.
Book catalogs, and three APIs that didn't work
1.3 also adds a catalog browser (⇧⌘B): search a book catalog, pick a format, it
goes to the download queue like any other URL. The interesting part was that the
obvious integration path was wrong for all three sources — and I only found out
by hitting the live services.
Project Gutenberg. It publishes OPDS, so: use OPDS, right? Its search feed
returns navigation entries — one sub-feed per result — so rendering a 50-book
grid with download links would cost 50 extra requests. Those per-book feeds were
also returning 504s, and the OPDS 2.0 endpoint is a 404. Gutendex returns every
format's direct URL inline, so that's what the grid uses.
Internet Archive. IA's OPDS BookServer doesn't just return errors —
bookserver.archive.org no longer resolves at all. Replaced with
advancedsearch.php?output=json for browse plus metadata/<id>/files for the
per-item file list, fetched lazily on selection.
Standard Ebooks. Every OPDS feed returns 401 to anonymous clients; access is a
donor benefit. It ships as a built-in source that's disabled by default, which
forced a small design point: CatalogStore has to track both explicitly-enabled
and explicitly-disabled built-ins, so a shipped default can apply until the user
actually expresses a preference.
Everything still normalizes into one CatalogFeed model, so the UI never branches
on source. And AcquisitionLink.isDownloadable is a single gate — direct-download
rel, no price, http(s), known format. DRM fulfilment documents get parsed and
displayed but never fetched, because that href is a license token, not a book.
Archive.org lists LCP Encrypted EPUB right next to the free files.
One more: catalog requests use a different URLSession than downloads do. The
download session sets waitsForConnectivity = true and
timeoutIntervalForResource = .infinity, which is exactly right for a 6 GB file
and exactly wrong for a metadata fetch behind a spinner.
BitTorrent
Magnets and .torrent files now run through the same queue as everything else —
same concurrency limit, same pause/resume, same quiet hours. It's off by default
and asks once before enabling, because unlike every other download type it
uploads on your connection and puts your IP in front of a swarm.
Two things worth stealing if you ever wrap aria2:
MacGet owns the queue, aria2 doesn't. --save-session and --input-file are
deliberately absent. queue.json is the source of truth and MacGet re-adds
torrents itself on launch; if aria2 also restored its own session, every info hash
would be registered twice and aria2 fails the duplicate outright. Resume comes
from --continue=true plus the .aria2 control file.
Magnets have a metadata phase that looks exactly like success. The first GID
downloads only the metainfo — a few hundred KB — and then reports complete with
completedLength == totalLength. Indistinguishable from a finished download
unless you're looking for it. An isAwaitingMetadata flag suppresses completion
and byte reporting until followedBy hands off to the real GID. Get it wrong and
your 6 GB torrent shows "Completed" at 500 KB.
Also: bind ports as a range (6881-6890), not a single value. aria2 hard-fails
with "Errors occurred while binding port" the moment another client holds it.
Install
brew install --cask suryansh-codes2209/macget/macget
Or grab the DMG: https://macget.suryansh.work
MIT licensed, and the engine is genuinely fun to poke at — clone it, open
Macget.xcodeproj, ⌘R. No package manager step, no codegen:
https://github.com/Suryansh-Codes2209/Macget
Issues and PRs welcome. I'm especially interested in ideas for what belongs in a
download manager in 2026 — if you've got a feature you've always wanted and never
found, open an issue.
And if you've ever shipped a heuristic that looked smart and was quietly costing
you something, I'd like to hear about it. I clearly can't be trusted to spot my
own.
Top comments (1)
My favorite line was: “Back off on evidence, not on absence of evidence.” That’s a much more general engineering principle than it first appears. Many adaptive systems fail because they treat a missing positive signal as a negative one, even though multiple underlying causes can produce the same observation. Those are often the hardest production bugs to detect because the system still appears to work.