We Optimized Everything. Azure Was Still Freezing Our .NET API with Gen2 Pauses

Some context

Our backend is a .NET API that a large fleet of devices talks to. The devices constantly report their status, and the API takes in those updates, processes them and serves regular user requests at the same time. On a normal day it handles about 400 requests every 15 seconds (roughly 25–30 requests per second), steadily, around the clock.

That load isn't huge, but it never stops. Because the traffic is so constant, the API has no quiet periods to catch up after a slowdown. If it stalls for even a few seconds, hundreds of requests are waiting when it comes back.

The problem

This API had a problem we couldn't get rid of. At random moments it would freeze.

Our monitoring dashboards showed the same pattern every time. The thread pool queue length suddenly jumped to around 200, the thread pool size grew sharply, and then recovery was slow and painful. While the API was frozen, requests piled up, and once it came back it had to work through the backlog. Maximum response times spiked to 20, 50, sometimes 100 seconds.

The direct cause wasn't hard to find: a large, stop-the-world Gen2 garbage collection pause. While the GC blocks every managed thread, nothing makes progress. Work keeps arriving and the queue keeps growing. The thread pool then injects new threads only gradually, which is why recovery took so long.

What we couldn't work out was why the Gen2 collections were happening.

Round 1: Fixing our own code

Our first assumption was that the problem was in our code, so we went through it carefully.

Removing blocking code

First we removed all sync-over-async code, meaning places where a thread blocks while it waits for an async operation instead of using await:

// ❌ Before: blocks a thread pool thread while waiting
var result = _client.GetDataAsync().Result;
_service.ProcessAsync(item).Wait();

// ✅ After: frees the thread while waiting
var result = await _client.GetDataAsync();
await _service.ProcessAsync(item);

Blocking calls like these are a well-known cause of thread pool starvation, so they were worth removing anyway.

Reducing memory pressure on hot paths

Next we tuned our hot paths to allocate less:

• Object pooling for frequently created objects, so they're reused instead of collected
• RecyclableMemoryStream instead of MemoryStream, to avoid repeatedly allocating large buffers that end up on the Large Object Heap

private static readonly RecyclableMemoryStreamManager StreamManager = new();

// ✅ Pooled buffers instead of a fresh allocation every time
using var stream = StreamManager.GetStream();

All of these were good improvements. None of them fixed the freezes. The Gen2 pauses kept coming.

Round 2: Looking inside the process

Next I wanted to see what the live process was doing. I got console (CMD/PowerShell) access to the Azure machine hosting the API, which is roughly the Windows equivalent of SSH-ing into the box, and used the standard .NET diagnostic tools:

# Live GC, heap and thread pool metrics
dotnet-counters monitor -p <PID> --counters System.Runtime

# Snapshot of the managed heap
dotnet-gcdump collect -p <PID>

I watched the counters and analyzed heap dumps, looking for anything unusual: Large Object Heap growth, huge collections, leaking caches.

I found nothing. Nothing stood out and nothing was leaking. After all the earlier optimization work, the allocation patterns looked healthy.

Round 3: Looking outside the process

At this point I asked a different question:

What if the problem isn't in our process, but something acting on it?

Azure keeps its own logs of what the platform does to your app. When I went through them, I found entries showing that Azure was periodically profiling our process.

I lined up the timestamps of those profiling sessions with our freezes, and they matched. Every time Azure profiled the process, the API froze and showed the same thread pool spike and long Gen2 pause.

My working theory: these profiling sessions trigger blocking, foreground Gen2 collections that pause all application threads, which produces exactly the thread pool pile-up we had been chasing.

The fix: two settings

The profiling came from two Application Insights features enabled on the app:

• [Feature 1 – e.g. Application Insights Profiler]
• [Feature 2 – e.g. Snapshot Debugger]

We weren't really using them. Our existing logging and tracing already gave us stack traces and per-method timings. So I disabled both, and the profiling entries in the Azure logs stopped.

Then I let the API run for 4+ hours and checked the dashboards.

The results

No more freezes, stable thread pool

The queue spikes stopped and the thread pool stayed flat and stable.

Before:

After:

Memory: 2.2 GB → 1.4 GB

The working set dropped from 2.2 GB to 1.4 GB. My guess is that the profiling and debugging tooling was also adding significant memory overhead.

Worst-case response times

Maximum response times that used to reach 20–100 seconds are now consistent: 99% of requests complete in under 4–5 seconds, even in the worst case. Typical response times are much lower. What changed is that the extreme outliers are gone.

Before:

After:

Background processing: under one second

The flow that processes device status updates had been slow and spiky enough that we were considering moving it out of the API entirely. It now finishes in under one second almost every time, with no peaks.

Before:

After:

Lessons learned

1. Look beyond your own code. We fixed blocking calls, pooled objects and reduced allocations. All of that was worth doing, but the real cause was the platform acting on our process from outside.
2. Correlate timestamps across every source. The breakthrough came from lining up platform logs with application metrics, not from going deeper into one tool.
3. Observability has a cost. Profilers and snapshot debuggers are useful, but they can add memory overhead and pause your application. Turn them on when you need them, not permanently "just in case."
4. Audit features you aren't using. If another tool already gives you that insight, a redundant feature is all cost and no benefit.
5. Don't accept "that's just how it is." Changing the question we were asking is what finally solved it.

---

To be clear, this is a strong correlation backed by large before/after improvements, not a proven root cause at the CLR level. Have you seen Azure profiling or snapshot features cause GC pauses in your apps? Let me know in the comments. I'd like to hear if others have run into this.