DEV Community

Pedro Sakuma Travi
Pedro Sakuma Travi

Posted on

The pool that got slower the more we shared it

#dotnet #performance #opensource #ai

How a 15-minute chat with an LLM found a hidden ArrayPool<byte>.Shared
contention point that no static analyzer would ever flag — and why the fix
had nothing to do with adding a lock.

The ticket

"CPU on market-data-gateway climbed ~20% after we added a broadcast fan-out
path last sprint. No new allocations, no GC pressure, nothing obviously
wrong in the diff. Can someone take a look before we scale out more pods?"

Nobody had touched anything that looked like a synchronization primitive.
The diff that shipped the week before only touched a buffer-pooling helper —
swapping a bespoke byte-array cache for the "obviously correct" BCL default,
ArrayPool<byte>.Shared. That refactor was reviewed and merged specifically
because it removed custom code in favor of a well-known, heavily optimized
framework type. If anything, it should have been a free win.

This is the kind of regression static analysis is structurally blind to: no
analyzer rule says "using ArrayPool<T>.Shared across threads is suspicious."
It's the recommended way to pool buffers in .NET. The problem isn't in the
code you'd read in a diff — it's in a runtime behavior that only shows up
under a specific, easy-to-miss usage pattern.

The investigation, as it actually happened

This is the real value proposition of dotnet-diagnostics-mcp:
none of what follows required attaching a debugger, adding instrumentation, or
redeploying anything. It's a live conversation with an LLM that has MCP tools
wired to the running process.

Step 1 — orient.

user> Can you check what's going on with pid 41213? CPU looks high.

assistant> Let me look at the process first.
→ inspect_process(view="triage", processId=41213)
Enter fullscreen mode Exit fullscreen mode 
{
  "processId": 41213,
  "resolvedProcess": { "name": "market-data-gateway", "runtime": "net10.0" },
  "summary": "CPU 78% (host has 16 cores). No memory growth trend. No recent
               GC pause anomalies. Consider a CPU sample to see where time goes.",
  "hints": [
    "collect_sample(kind=\"cpu\", processId=41213, durationSeconds=10)"
  ]
}
Enter fullscreen mode Exit fullscreen mode

Nothing screaming "leak," nothing screaming "GC." CPU is just... busy. Time to
see where.

Step 2 — sample.

assistant> Taking a 10-second CPU sample.
→ collect_sample(kind="cpu", processId=41213, durationSeconds=10, topN=20)
Enter fullscreen mode Exit fullscreen mode 
{
  "processId": 41213,
  "totalSamples": 18545,
  "summary": "Hottest self-time method: System.Threading.Monitor.Enter_Slowpath
              (2933 exclusive, 15.8% of samples). Rank self-time with
              query_snapshot(view=\"top-methods\") or walk the call path with
              view=\"call-tree\".",
  "handle": "A0CKKWH5VVS9Q68AH3GG",
  "hints": [
    "query_snapshot(handle=\"A0CKKWH5VVS9Q68AH3GG\", view=\"caller-callee\", rootMethodFilter=\"Monitor.Enter_Slowpath\")"
  ]
}
Enter fullscreen mode Exit fullscreen mode

That's the first real surprise. 15.8% of all CPU samples land inside
Monitor.Enter_Slowpath. Nobody wrote a lock statement in this service.
grep -r "lock (" across the whole repo returns nothing relevant. So where is
a contended monitor coming from?

Step 3 — trace the lock back to its call site.

assistant> There's no explicit lock in this codebase. Let's see who's calling
into the monitor slow path.
→ query_snapshot(handle="A0CKKWH5VVS9Q68AH3GG", view="caller-callee",
                  rootMethodFilter="Monitor.Enter_Slowpath")
Enter fullscreen mode Exit fullscreen mode 
{
  "method": "System.Threading.Monitor.Enter_Slowpath(System.Object)",
  "exclusiveSamples": 2933,
  "exclusivePercent": 15.82,
  "callers": [
    {
      "method": "MarketData.BufferPool.Return(byte[])",
      "samples": 1478,
      "percent": 7.97
    },
    {
      "method": "MarketData.Broadcaster+<>c__DisplayClass.<ConsumeLoop>b__0()",
      "samples": 1455,
      "percent": 7.85
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Both callers are our code — a thin wrapper method calling .Rent() on one
side, and a per-connection consume loop calling .Return() on the other.
Not a ThreadPool internal, not GC. The lock is being taken
inside ArrayPool<byte>.Shared itself, and it's the pool, not any
business logic, that's contended.

Step 4 — read the actual lock, not a guess about it.

ArrayPool<byte>.Shared's internals are in System.Private.CoreLib — no
source, no PDB for the private implementation type. This is exactly what the
companion server, dotnet-assembly-mcp,
is for: static navigation and on-demand decompilation of compiled assemblies,
including the BCL itself.

assistant> Let's not guess. I'll look up the actual pool implementation
in the runtime binary you're running.
→ find_method(mvidOrPath="System.Private.CoreLib.dll", namePattern="^Return$")
Enter fullscreen mode Exit fullscreen mode 
{
  "summary": "5 match(es) for /^Return$/.",
  "matches": [
    { "typeFullName": "System.Buffers.SharedArrayPool`1", "methodName": "Return",
      "signature": "void System.Buffers.SharedArrayPool`1.Return(!0[], bool)",
      "handle": "m:e8c78f6b-...:0x060038A7" }
  ]
}
Enter fullscreen mode Exit fullscreen mode 
assistant> Decompiling the real Return method for this exact runtime build.
→ decompile_method(moduleVersionId="e8c78f6b-...", metadataToken="0x060038A7")
Enter fullscreen mode Exit fullscreen mode 
public override void Return(T[] array, bool clearArray = false)
{
    // ...
    SharedArrayPoolThreadLocalArray[] array2 = t_tlsBuckets ?? InitializeTlsBucketsAndTrimming();
    // ... same-thread fast-path slot swap ...
    if (array3 != null)
    {
        flag2 = (_buckets[num] ?? CreatePerCorePartitions(num)).TryPush(array3);
    }
    // ...
}
Enter fullscreen mode Exit fullscreen mode 
assistant> And the fallback path that pushes to a per-core partition:
→ find_method(mvidOrPath="System.Private.CoreLib.dll", namePattern="^TryPush$")
→ decompile_method(moduleVersionId="e8c78f6b-...", metadataToken="0x060038B3")
Enter fullscreen mode Exit fullscreen mode 
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public bool TryPush(Array array)
{
    bool result = false;
    Monitor.Enter(this);
    // ... push onto a small fixed-size array-backed stack ...
    Monitor.Exit(this);
    return result;
}
Enter fullscreen mode Exit fullscreen mode

There it is — Monitor.Enter(this), in the flesh, in the exact binary
this process is running: the actual internal type backing
ArrayPool<byte>.Shared in this runtime build is SharedArrayPool<T>, and
its per-core fallback is guarded by a plain CLR monitor. This is why the
call-tree pointed straight at Monitor.Enter_Slowpath: it's a real, ordinary
monitor, contended by every thread whose Return misses the per-thread
cache.

To be clear: dotnet/runtime's source is public on GitHub — nothing here was
hidden. The point isn't that the source was unreachable, it's the ergonomics
of the query. Getting the same answer the "normal" way means: know (or guess)
which internal type backs ArrayPool<T>.Shared today, find the right tag in
dotnet/runtime that matches the exact SDK/runtime patch this process is
actually running, navigate to the right file, and read enough surrounding
code to trust you're looking at the live behavior and not a since-changed
version. decompile_method(moduleVersionId, metadataToken) collapses all of
that into one call, scoped to the exact assembly loaded in the exact process
under investigation — no version-matching, no guessing a type name, no
context-switch out of the chat.

Checking the application code (only now, after both tools already
pointed at the exact mechanism, not before):

// One dedicated broadcaster thread — the ONLY thread that ever Rents.
while (running)
{
    var buffer = ArrayPool<byte>.Shared.Rent(FrameSize);
    FillFromMarketFeed(buffer);
    channel.Writer.TryWrite(buffer);
}

// N per-connection consumer tasks — each may Return a buffer it never Rented.
await foreach (var buffer in channel.Reader.ReadAllAsync())
{
    await connection.WriteAsync(buffer);
    ArrayPool<byte>.Shared.Return(buffer); // <-- always a different thread than Rent
}
Enter fullscreen mode Exit fullscreen mode

Exactly the pattern the tool output implied: one thread rents, many other
threads return.

Why this is invisible without live evidence

ArrayPool<T>.Shared isn't a naive shared bag — its current .NET 10
implementation, SharedArrayPool<T>, keeps a single-slot cache stored
per-thread ([ThreadStatic] buckets): if the same thread that rented a
buffer also returns it, the operation is a plain field swap, no
synchronization at all. The moment a different thread returns it, that fast
path can't apply (the slot belongs to another thread), so the buffer falls
through to a small set of per-core partitions, each guarded by a plain
Monitor. With a handful of cores and dozens of consumer threads all
returning concurrently, those few partitions become a genuine contention
point.

None of this is a bug in ArrayPool<T>. It's working exactly as designed —
optimized for the common case (rent-use-return on one thread, e.g. inside a
single request handler), and gracefully degrading, but not silently free,
for the cross-thread fan-out case. There is no compiler warning, no analyzer
rule, and no code-review heuristic for "you are about to violate an
implicit performance assumption of a BCL type you didn't write." The only way
to know, for certain, what this process, running this runtime build, is
actually doing is to watch where the CPU goes and read the exact binary it's
running.

Fix and verification

The fix isn't "add a lock" (there's nothing to protect) — it's swap the pool
implementation to one designed for the actual ownership pattern: many threads
returning items nobody-in-particular rented, i.e., a plain lock-free queue.

public sealed class CrossThreadBufferPool
{
    private readonly ConcurrentQueue<byte[]> _pool = new();
    public byte[] Rent(int size) => _pool.TryDequeue(out var buf) ? buf : new byte[size];
    public void Return(byte[] buffer) => _pool.Enqueue(buffer);
}
Enter fullscreen mode Exit fullscreen mode

Isolated cost, before/after (BenchmarkDotNet, 20,000 cross-thread
Return calls, buffers pre-rented on a background thread outside the timed
region so only Return itself is measured):

Scenario Mean time (20,000 calls) vs. contended baseline
ArrayPool.Shared.Return, cross-thread (the bug) ~4.3–5.1 ms 1x (baseline)
ConcurrentQueue<byte[]> Return, cross-thread (the fix) ~126–146 µs ~34x faster
ArrayPool.Shared full Rent+Return, same thread — theoretical best case ~115–150 µs ~34–40x faster

The last row is the ceiling: the one scenario with genuinely zero
synchronization, because the per-thread cache slot is always hit. Comparing
it to the ConcurrentQueue fix shows exactly how much overhead our own fix
still carries: ~126 µs vs. ~115 µs — roughly 10% above the theoretical
best, for a data structure built to be used safely by an arbitrary number
of unrelated threads. That's the honest cost of "general-purpose
thread-safe," and it's a rounding error next to the 34x we recovered.

A single cross-thread Return call on the unfixed pool costs more than an
entire same-thread Rent+Return round trip. That's how disproportionate the
fast-path miss is.

Live re-check after the fix — same 2-thread (1 producer / 1 consumer)
topology, same CPU sampler, before touching anything else:

Metric ArrayPool.Shared (cross-thread) Lock-free pool (cross-thread) Same-thread control
Throughput ~3.2M ops/s ~6.6M ops/s ~20M ops/s
Monitor.Enter_Slowpath in CPU sample 15.8% exclusive 0% — gone entirely n/a (no shared pool contention path) 
assistant> With the lock-free pool under the same load:
→ collect_sample(kind="cpu", processId=41213, durationSeconds=8, topN=20)
Enter fullscreen mode Exit fullscreen mode 
{
  "summary": "Hottest self-time method: System.Threading.Thread.Sleep(int32)
              (3972 exclusive, 40.6% of samples). No Monitor frames in the
              top 20 hotspots.",
  ...
}
Enter fullscreen mode Exit fullscreen mode

No Monitor.Enter_Slowpath anywhere in the profile. The contention didn't
move — it disappeared, because the pool that produced it is gone.

The obvious follow-up question: is pooling even worth it here?

ArrayPool<T>.Shared itself exists as an optimization over the naive
baseline — just calling new byte[size] and letting the GC reclaim it. A
fair objection to everything above: if the shared pool is this contended
under cross-thread Return, maybe the honest fix is to drop pooling
entirely, not swap in a different pool. Mean latency might even look
competitive, since a plain allocation has zero synchronization of any kind —
but the expectation is that Gen0/Gen1 GC pressure would show up in the tail
(P99), not the mean.

We tested it — same 2-thread producer/consumer topology, only the pool
implementation swapped for one that allocates fresh on every Rent and does
nothing on Return:

Scenario Throughput Monitor contention
ArrayPool.Shared, cross-thread ~3.2M ops/s 15.8% (Monitor)
Lock-free queue pool, cross-thread ~6.6M ops/s 0%
No pooling (new byte[size] every call) ~180K ops/s 0%

No pooling isn't a tail-latency-only regression — it's ~18x slower on raw
throughput, worse than the contended ArrayPool.Shared baseline we started
from. The reason shows up immediately in a live GC pause capture over the
same window:

assistant> Let's check what removing the pool entirely does to GC.
→ collect_events(kind="gc", processId=41213, durationSeconds=10)
Enter fullscreen mode Exit fullscreen mode 
{
  "summary": "134 collection(s), max pause 69.9ms, total pause 5118.1ms."
}
Enter fullscreen mode Exit fullscreen mode 
assistant> Breaking that down:
→ query_snapshot(handle="...", view="pauseHistogram")
Enter fullscreen mode Exit fullscreen mode 
{
  "totalCollections": 134,
  "maxPauseTime": "00:00:00.0698743",
  "buckets": [
    { "label": "<1ms",      "count": 0 },
    { "label": "1-10ms",    "count": 0 },
    { "label": "10-100ms",  "count": 134 },
    { "label": "100-1000ms","count": 0 },
    { "label": ">=1s",      "count": 0 }
  ]
}
Enter fullscreen mode Exit fullscreen mode 
assistant> And by generation:
→ query_snapshot(handle="...", view="byGeneration")
Enter fullscreen mode Exit fullscreen mode 
{
  "generations": [
    { "bucket": "gen1", "count": 134, "totalPause": "00:00:05.1180759",
      "meanPause": "00:00:00.0381945", "maxPause": "00:00:00.0698743" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

134 gen1 collections in a 10-second window — 13.4 per second, every single
one landing in the 10–100ms bucket — for 5.12 seconds of accumulated pause
time inside those 10 seconds. Over half the wall clock is GC pause, not
application throughput. That's not a tail-latency footnote; it dominates the
mean too, once allocation volume is high enough to keep pushing 4KB buffers
into gen0 fast enough to promote and collect on a ~130ms cadence.

So the honest ranking, worst to best, for this specific cross-thread
fan-out pattern is: no pooling (worst, by both mean and tail) → shared pool
with cross-thread Return (contended but bounded) → a pool built for the
actual ownership shape (best on every axis measured). ArrayPool<T>.Shared
being contended here doesn't mean pooling was the wrong call — it means the
specific pool's ownership assumption (same-thread Rent/Return) didn't match
this workload's shape. The fix was never "go back to allocating"; it was
"pool it the way this workload actually uses it."

The takeaway

Static analysis, linters, and code review are excellent at catching shapes
of bugs — a missing await, a lock around the wrong object, an O(n²)
loop. They cannot catch a case where the code is a textbook-correct call into
a well-designed BCL type, and the only thing wrong is a runtime ownership
pattern that BCL type happens to be sensitive to. You either know this
specific gotcha already, or you find it by watching the process — there is no
third option.

dotnet-diagnostics-mcp exists so that "watching the process" is a normal
step in an LLM-driven investigation, not a specialized skill gated behind
whoever on the team happens to know dotnet-trace by heart. Every dynamic
step above — inspect_process, collect_sample, query_snapshot — is a
single MCP tool call an assistant can make on its own, against a real running
.NET process, with no code changes and no redeploy. And when the trail leads
into a BCL type nobody on the team has source for, its companion server,
dotnet-assembly-mcp,
picks up exactly where live tracing stops: real decompiled source of the
exact binary the process is running, not a recollection of how some type
used to work.

The reproduction and all numbers in this post are in a small, standalone,
open repro you can run yourself — no proprietary code involved. The
investigation pattern itself (fix unrelated to this repro) is inspired by a
real production change in B3MarketDataPlatform,
a public repo, commit ac2dc9f.

dotnet-diagnostics-mcp is open source: github.com/pedrosakuma/dotnet-diagnostics.
It ships both an MCP server (for your LLM client of choice) and a standalone
CLI (dotnet-diagnostics-cli) for scripting the same investigations by hand.

Top comments (0)