Mirror Speed Is a Lie Until You Measure It

oncidentally, I spent last Tuesday afternoon chasing a phantom. A colleague swore our internal NuGet mirror was "slow as hell," and the build server logs backed him up—package restores that used to take 40 seconds were taking four minutes. I opened the mirror's status page, saw green health checks, and immediately suspected the network. Then I actually measured transfer speed from the mirror host to the build agent, and the number was fine: 900 Mbps sustained. The mirror itself wasn't slow. The path to the mirror was slow. That's the first lesson: mirror speed is never a single number. It's a topology of bottlenecks, and you have to measure each hop before you can fix any of them.

The Three Numbers That Actually Matter

When someone says "mirror speed," they usually mean one of three things, and conflating them causes most of the pain I see in CI pipelines.

1. Upstream sync latency — how stale the mirror is relative to the origin. If you mirror https://repo1.maven.org once a day, your "speed" is irrelevant when a developer needs a package published 20 minutes ago. I check this with a simple HTTP HEAD request to a known volatile artifact. For Maven, I use the maven-metadata.xml timestamp. For npm, I hit the registry's -/npm/v1/registry endpoint and compare last-modified.

2. Transfer throughput — the raw bytes-per-second between the mirror and the consumer. This is what most people benchmark with wget or curl, but they forget that TCP window scaling, TLS handshake overhead, and disk cache all distort the result. I always use a file that's at least 50 MB, and I force a cache miss by appending a query string with a random value.

3. Connection establishment latency — the round-trip time for a new TCP/TLS connection. This matters more than throughput for package managers that open dozens of small connections (looking at you, pip). A mirror with 200 ms RTT but 10 Gbps throughput will feel slow for pip install because every package metadata fetch pays the full RTT.

Here's the PowerShell snippet I use to get all three from a Windows build agent:

$mirror = "https://mirror.internal.corp"
$testFile = "$mirror/benchmark/large-file.bin"  # 100 MB pre-generated
$sw = [System.Diagnostics.Stopwatch]::StartNew()
$response = Invoke-WebRequest -Uri $testFile -Method Head -UseBasicParsing
$sw.Stop()
$rtt = $sw.ElapsedMilliseconds

$sw.Restart()
$tempFile = "$env:TEMP\mirror_bench_$([guid]::NewGuid()).bin"
Invoke-WebRequest -Uri $testFile -OutFile $tempFile -UseBasicParsing
$sw.Stop()
$seconds = $sw.Elapsed.TotalSeconds
$sizeMB = (Get-Item $tempFile).Length / 1MB
$throughput = $sizeMB / $seconds

Write-Host "RTT: $rtt ms"
Write-Host "Throughput: $([math]::Round($throughput,2)) MB/s"
Remove-Item $tempFile

Run that from the build agent, not from your laptop. The laptop's VPN path will lie to you.

A Real-World Workflow: Diagnosing a Slow Apt Mirror

My most recent battle was with an internal apt-cacher-ng instance. The symptom: apt-get update on Ubuntu 22.04 agents took 90 seconds instead of 8. The mirror was on a 10 GbE link, and the agents were on the same VLAN. So why so slow?

I started by checking the cache hit ratio. apt-cacher-ng exposes this at http://mirror:3142/acng-report.html. The hit ratio was 94%, which is healthy. But the miss path was the problem. When a package isn't cached, apt-cacher-ng fetches from the upstream Ubuntu archive, and that upstream connection was pegged at 15 Mbps because the corporate firewall was throttling port 80 traffic to non-whitelisted domains.

The fix wasn't to speed up the mirror. It was to change the upstream URL from http://archive.ubuntu.com to http://mirror.internal.corp/ubuntu-archive — an internal proxy that had a fast path. I edited /etc/apt-cacher-ng/acng.conf, changed Remap-deb1 and Remap-deb2 lines to point to the internal proxy, then restarted the service. apt-get update dropped to 11 seconds. The mirror was never the problem; the origin was.

Here's the Bash snippet I used to verify the upstream speed before and after the change:

# Before: direct to upstream
time curl -o /dev/null -s -w "%{speed_download}\n" \
  http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease

# After: through internal proxy
time curl -o /dev/null -s -w "%{speed_download}\n" \
  http://mirror.internal.corp/ubuntu-archive/dists/jammy/InRelease

The first command reported 1.8 MB/s. The second reported 22.4 MB/s. Same file, same agent, same network. The only difference was the route.

Failure Modes That Look Like Speed but Aren't

After a few years of this, I've learned to recognize the classic impostors.

TLS session resumption failures. If your mirror uses a certificate chain that's too deep, or if the client doesn't support the same TLS version, every connection does a full handshake. That adds 300-500 ms per connection. With 200 package requests, that's 60-100 seconds of pure overhead. The fix is usually to enable TLS 1.3 on both ends and make sure the client has the CA certificate installed. I've seen apt and pip both silently fall back to TLS 1.2 and then to 1.0, each handshake getting slower.

Disk I/O on the mirror host. A mirror that's been running for years accumulates a fragmented filesystem or runs out of inodes. The network says 10 Gbps, but the disk can only serve 50 MB/s random reads. The symptom: large files download fast, but many small files (package metadata) crawl. I check iostat -x 1 on the mirror host and look for await times over 50 ms. The fix is often just xfs_fsr or a periodic fstrim if you're on SSDs.

Client-side DNS resolution. This one is sneaky. The build agent's DNS resolver might take 2 seconds to resolve mirror.internal.corp because the DNS server is slow or the record has a long TTL and the local cache expired. I've seen pip install spend 30 seconds just resolving the hostname for each package. The fix: add a static entry in C:\Windows\System32\drivers\etc\hosts on the agents. This is a one-line change that often "fixes" a slow mirror without touching the mirror at all.

HTTP keep-alive misuse. Some package managers (older versions of nuget.exe) don't reuse connections. They open a new TCP connection per package. If the mirror is behind a load balancer that doesn't support keep-alive well, you get connection churn. I've fixed this by adding a reverse proxy with proxy_http_version 1.1 and proxy_set_header Connection "" in nginx.

Edge Case: The Mirror That's Too Fast

Here's a counterintuitive one. A mirror that's too fast can cause timeouts. I once set up a local npm mirror with an in-memory cache. It served packages in under 1 ms. But npm on the client side has a built-in timeout for the initial handshake. When the server responds faster than the client's TCP stack can process, you get sporadic ECONNRESET errorscars. The fix was to add a small artificial delay on the mirror side (10 ms) using tc qdisc add dev eth0 root netem delay 10ms. That sounds insane, but it stabilized the client. Sometimes you have to slow down to speed up.

Another edge case: mirroring to a network filesystem (NFS or SMB). The mirror process writes to a network share, and the share's latency dominates. I saw a rsync mirror to a CIFS share that took 45 minutes for a 2 GB update because every file write had a 50 ms round-trip to the file server. The fix was to sync to local disk first, then rsync the local disk to the remote share in a second pass Mend the write latency only affects the final copy, not the mirror's read path.

Measuring the Right Thing in CI

If you're running a CI farm, the most useful metric is not "mirror throughput" but "package restore time percentile." I add a step to every pipeline that times dotnet restore or pip install and logs the 95th percentile. If that number creeps up, I don't look at the mirror first. I look at the agent's CPU count, disk type, and whether the agent is sharing a VM host with a noisy neighbor.

Here's a PowerShell snippet I use to benchmark the full restore path on a Windows agent:

$sw = [System.Diagnostics.Stopwatch]::StartNew()
dotnet restore .\src\MyApp\MyApp.csproj --source https://mirror.internal.corp/v3/index.json
$sw.Stop()
$total = $sw.Elapsed.TotalSeconds
$pkgCount = (Get-ChildItem -Recurse -Filter "*.nupkg" -Path .\src | Measure-Object).Count
Write-Host "Restore took $total s for $pkgCount packages"
Write-Host "Per-package time: $([math]::Round($total / $pkgCount, 3)) s"

If that per-package time is over 0.5 seconds, something's wrong. On a good mirror, it should be under 0.1 seconds.

Related resource

If you're wrestling with mirror speed on Windows, you might also be dealing with the broader problem of managing downloaded resources efficiently. I've been using putyy res downloader 3 1 3 from GitVaults to batch-download and verify large resource files before pushing them to my mirrors. It handles resumable downloads and checksum validation, which saves me from re-downloading a corrupted 2 GB file because a flaky mirror dropped the last 5 MB. GitVaults is a solid catalog for Windows open-source tools, and their download pages are clean without the usual ad-riddled wrappers.

Caveat

This is all based on my own experience with specific networks, mirrors, and clients. Your mileage will vary. I've seen cases where the exact opposite of my advice was true—for example, a mirror that was slow because the client was on a throttled Wi-Fi link, not because of anything on the server side. If you spot an error in my methodology, or you have a better trick for isolating mirror bottlenecks, please correct me. I'm always learning from people who measure more carefully than I do.