500 Async Calls, One Number That Explains Why

I shipped a background job that fanned out to a downstream API for every row in a batch, wrapped the whole thing in Task.WhenAll, and watched it work beautifully in testing. Then someone ran it against a real batch of a few hundred rows and the downstream service started returning timeouts. Nothing in my code was slow. I'd just asked it to make every single call at the exact same instant, and the thing on the other end wasn't built for that.

The fix was one method. But before I changed anything I wanted a number that showed me what was actually happening, because "it felt like too much load" isn't something I can put in a PR description.

Measuring the thing nobody measures

Load is easy to hand-wave about and hard to see. So I built a tiny stand-in for the downstream: a class that counts how many callers are inside it at the same moment and remembers the worst it ever saw. That high-water mark is the whole story. Each "call" is just a 20 ms delay to stand in for network plus server time.

int now = Interlocked.Increment(ref _inFlight);
int seen;
while (now > (seen = Volatile.Read(ref _peak)))
    Interlocked.CompareExchange(ref _peak, now, seen);
try { await Task.Delay(20, ct); }
finally { Interlocked.Decrement(ref _inFlight); }

I fan out 500 jobs, run each strategy five times, and take the median wall-clock plus the peak concurrency observed. .NET 10, Release build, small Linux container. Not a lab. I care about the ratios, not the millisecond.

Here's the version I'd shipped:

Task.WhenAll(ids.Select(id => d.CallAsync(id, CancellationToken.None)));

It reads like exactly what I wanted: do all of these, tell me when they're done. And the numbers look great, which is the trap:

WhenAll (unbounded)              peak 500   median 20.6 ms

Twenty milliseconds. One wave. All 500 calls left the building at once and came back together. In a benchmark that's a win. In production that's 500 simultaneous connections landing on a service that was sized for maybe a dozen, and the fastest way I know to turn my batch job into someone else's incident.

Letting the runtime hold the gate

Parallel.ForEachAsync takes a MaxDegreeOfParallelism. You hand it the collection and the per-item work, and it refuses to run more than N at a time.

Parallel.ForEachAsync(
    ids,
    new ParallelOptions { MaxDegreeOfParallelism = 8 },
    async (id, ct) => await d.CallAsync(id, ct));

Same 500 jobs, cap of 8:

Parallel.ForEachAsync (DOP=8)    peak 8     median 1284.2 ms

Peak dropped from 500 to 8, exactly the ceiling I asked for. And yes, it's about sixty times slower. I want to be honest about that because it's the part people skip: bounding concurrency costs you wall-clock. Five hundred jobs through eight lanes at 20 ms each is roughly 63 waves, and that's where the 1.28 seconds comes from. There's no free lunch where you both protect the downstream and finish in one burst.

Which means the right ceiling is a real decision, not a default. Eight is not magic. If the downstream can comfortably take 50 concurrent, use 50 and get most of the speed back while still having a ceiling. The point isn't to go slow, it's to stop handing the number "500" to a dependency that never agreed to it.

When unbounded is still fine

I don't want to leave the impression that Task.WhenAll is a bug. If you're awaiting ten things, or a handful of independent calls that each hit a different service, fan them all out and move on. The gate earns its keep specifically when the count is large or unbounded and the calls pile onto one shared, limited resource: a single API, a connection pool, a database. If none of that is true, the ceiling is just ceremony.

The pattern it replaces

Before Parallel.ForEachAsync existed, the way you capped concurrency was a SemaphoreSlim you managed by hand: wait before, release in a finally, wrap the lot in Task.WhenAll.

using var gate = new SemaphoreSlim(8);
var tasks = ids.Select(async id =>
{
    await gate.WaitAsync();
    try { await d.CallAsync(id, CancellationToken.None); }
    finally { gate.Release(); }
});
await Task.WhenAll(tasks);

I measured this one too, because I half-expected it to behave differently:

SemaphoreSlim + WhenAll (DOP=8)  peak 8     median 1285.4 ms

It's a tie. Same ceiling, same time, to within noise. So the difference isn't performance, it's everything around the code: the semaphore version has a finally you can forget, a Release you can double-call, and it still builds the full list of 500 tasks up front even though only 8 ever run. My honest opinion is that Parallel.ForEachAsync wins purely on the stuff that bites you later. If I already have a semaphore doing fancier things, I'll keep it. For plain "cap the fan-out," I reach for the built-in now and don't think about releasing anything.

The whole runnable sample

I couldn't push this one to the usual repo (the sandbox blocked write access to GitHub on this run), so here's the complete program. Drop it in a project targeting net10.0 with <OutputType>Exe</OutputType> and dotnet run -c Release.

using System.Diagnostics;

sealed class Downstream
{
    private int _inFlight;
    private int _peak;
    public int Peak => _peak;

    public async Task CallAsync(int id, CancellationToken ct)
    {
        int now = Interlocked.Increment(ref _inFlight);
        int seen;
        while (now > (seen = Volatile.Read(ref _peak)))
            Interlocked.CompareExchange(ref _peak, now, seen);
        try { await Task.Delay(20, ct); }
        finally { Interlocked.Decrement(ref _inFlight); }
    }
}

static class Program
{
    const int JobCount = 500;
    const int Rounds = 5;

    static async Task Main()
    {
        int[] ids = Enumerable.Range(0, JobCount).ToArray();
        await RunUnbounded(new Downstream(), ids); // warm up

        var u = await Measure("WhenAll (unbounded)", ids, RunUnbounded);
        var b = await Measure("Parallel.ForEachAsync (DOP=8)", ids, RunBounded);
        var s = await Measure("SemaphoreSlim + WhenAll (DOP=8)", ids, RunSemaphore);

        Console.WriteLine($"\n{"strategy",-34} {"peak",5} {"median ms",10}");
        foreach (var r in new[] { u, b, s })
            Console.WriteLine($"{r.Name,-34} {r.Peak,5} {r.Median,10:0.0}");
    }

    static async Task<(string Name, int Peak, double Median)> Measure(
        string name, int[] ids, Func<Downstream, int[], Task> run)
    {
        var times = new List<double>();
        int peak = 0;
        for (int i = 0; i < Rounds; i++)
        {
            var d = new Downstream();
            var sw = Stopwatch.StartNew();
            await run(d, ids);
            sw.Stop();
            times.Add(sw.Elapsed.TotalMilliseconds);
            peak = Math.Max(peak, d.Peak);
        }
        times.Sort();
        return (name, peak, times[times.Count / 2]);
    }

    static Task RunUnbounded(Downstream d, int[] ids) =>
        Task.WhenAll(ids.Select(id => d.CallAsync(id, CancellationToken.None)));

    static Task RunBounded(Downstream d, int[] ids) =>
        Parallel.ForEachAsync(ids,
            new ParallelOptions { MaxDegreeOfParallelism = 8 },
            async (id, ct) => await d.CallAsync(id, ct));

    static async Task RunSemaphore(Downstream d, int[] ids)
    {
        using var gate = new SemaphoreSlim(8);
        var tasks = ids.Select(async id =>
        {
            await gate.WaitAsync();
            try { await d.CallAsync(id, CancellationToken.None); }
            finally { gate.Release(); }
        });
        await Task.WhenAll(tasks);
    }
}

What's your default ceiling when you fan out to a shared dependency, and how did you land on that number? I'm always curious whether people tune it or just pick 8 and move on.

Until the next time I measure something nobody asked me to,
Sukhpinder