810 blocking calls at once: what the .NET thread pool did to the first burst

I am building a cloud trade copier for MetaTrader. When a trader closes a position on the master account, the same close has to go out on every follower account, as fast as possible. For MT4 the vendor library is a .NET DLL with a synchronous API: OrderClose blocks its thread until the broker answers. In my setup that answer takes about 130 ms.

That is fine for one close. It stops being fine when 27 positions close in the same second and each of them has 30 followers.

What I measured

I measure "own time": the time from the master's callback to the moment the follower's order call is made. It excludes the broker's answer, so it is the part my code is responsible for. In a quiet minute it is 1-3 ms.

During a burst of 27 simultaneous closes (810 copies, MT4 follower accounts, a real broker with demo accounts) it looked like this, grouped by how many threads the pool had when each copy started:

thread pool threads at start	own time, median
33-36	5.5-14 ms
40-69	19 ms
73-102	45 ms
134-163	71 ms
344-486	125-190 ms (max 244 ms)

The pool had grown from 33 to 486 threads during those twelve seconds, and the delay grew with it. A staircase like that has a cause, and it is not one of the usual suspects.

What it was not

• GC. The longest pause was 6-34 ms, none above 200 ms, and the staircase was too regular to be GC.
• Channel wake-ups and fan-out. A microbenchmark writing one event to 30 channels showed the last reader waking about 20 µs later.
• Eager listing of open positions before the send. It looked suspicious, but warm it costs 2 µs.

One note on method, because I got it wrong three times: drop the first five minutes after a restart (JIT, cold caches, a small pool), and never write log lines on the path you are measuring. A log line before the send added 2-3 ms. My measuring code only stored timestamps and a background thread printed them.

What it was

Every in-flight order call holds a pool thread for the broker's whole answer. During a burst, all of them are blocked at once, and the next piece of work (the next follower's dispatcher continuation) needs a new thread. The pool creates threads gradually, and creating a thread costs about 0.2-0.5 ms. With hundreds of blocked threads and hundreds of new ones to create, those fractions of a millisecond add up to the staircase.

ThreadPool.SetMinThreads did not help the way I expected, for a dull reason: it silently rejects the whole call if either value is above the maximum. The I/O thread maximum was 1000 and I had asked for 1024, so the minimum had never taken effect and stayed at the core count (4 on my server). A warning on rejection and a log line with the effective minimum at startup fixed that. It removes the "two new threads per second" throttle, but not the cost of creating the thread.

The fix: threads that never retire

The calls that block for the broker's answer now run on their own threads, created once, kept forever, and reused:

public Task<T> Run<T>(Func<T> work, CancellationToken ct = default)
{
    var job = new Job<T>(work, ct, () => Interlocked.Decrement(ref _inFlight));
    var wanted = Interlocked.Increment(ref _inFlight);
    _queue.Add(job);

    // One thread per outstanding call, up to the cap: never fewer threads than calls to run at once.
    EnsureThreads(wanted);
    return job.Task;
}

Each worker is a plain loop over a BlockingCollection. A few details turned out to matter:

• The task's continuation must not run on these threads (TaskCreationOptions.RunContinuationsAsynchronously), or an awaiting caller can hold one.
• The in-flight counter is decremented before the task completes. Whoever continues from the task and starts the next burst then sees the true number of running calls, and no thread is created for a call that has already finished.
• A call still waiting for a thread is cancelled at once, like Task.Run, not when a thread finally gets to it.
• Past the cap, calls wait in a FIFO queue.

My first fix was not enough

I started the executor with 64 ready threads and let it grow on demand. A burst of 12 closes (360 MT4 copies) then made it create about 296 threads right in the copy path, and the own time barely improved:

	closes	own time p50 / p95 / max
before the fix, warm pool	12	2.8 / 20.7 / 25.2 ms
executor, 64 ready threads	12	25.9 / 55.3 / 61.1 ms
executor, 512 ready threads	12	1.2 / 5.0 / 7.9 ms

The cost was never the pool's. It was thread creation, and it is paid wherever it happens in the burst. The fix was to create the threads before the burst: the worst case is followers × per-follower parallelism (30 × 16 = 480 here), so that is how many threads the executor starts with. With 512 ready threads the burst created none, and the own time stayed at about 1 ms regardless of the burst size.

The three waves are different bursts at different times, not a controlled A/B, but they have the same size (12 closes), and the thread counts show the mechanism: 296 created in the second, 0 in the third.

What a ready thread costs

Sleeping threads are cheap. On Linux with .NET 10, 1024 idle threads cost about 35 MiB of resident memory (35 KiB each) and 186 ms to create at startup. The whole node stayed at 318-380 MiB resident with 452 threads, the same as before the change.

A side effect I did not prove

During the 64-thread run, my MT5 node, which has no blocking calls and was unchanged, also slowed down: p95 went from about 2 ms to 28.6 ms. The two nodes share a machine, and the MT4 node was creating 300-450 threads. With 512 ready threads the MT5 numbers went back to normal. The correlation is measured; the cause is not proven.

Takeaways

1. A synchronous call on the thread pool costs a thread for as long as it blocks. In a burst, that is a lot of threads, created exactly when you can least afford it.
2. Warming up the pool is not enough, and neither is SetMinThreads: creating a thread costs time wherever it happens. Create the threads before the burst.
3. Size them from the worst case you can compute, not from the average. Mine is one multiplication.
4. Check that SetMinThreads actually returned true.

If you are curious about the project: I build SnapCopier, a cloud trade copier for MT4 and MT5. The numbers above are from its own node logs.