The CancellationToken That Never Propagated: A Subtle ASP.NET Core Timeout Bug I Missed in Code Review

In the early stages of my career, I believed that if an HTTP request had a timeout set, the system was safe. I would configure HttpClient.Timeout or use RequestAborted, and I felt protected.

I was wrong.

Recently, I reviewed a codebase for a .NET Core application that was experiencing mysterious latency spikes under load. The errors weren't immediate 500s; instead, the application felt sluggish, response times increased progressively, and eventually, the thread pool looked exhausted. This post breaks down that specific debugging journey and why CancellationToken propagation is more critical than simple timeouts.

The Setup

We had an ASP.NET Core API endpoint that accepted a request, validated it, and then triggered a long-running operation. To keep the user experience snappy, we didn't want the client to wait for the entire operation to complete. Instead, we used a pattern where the main request would return a "202 Accepted" quickly, and a background Task would handle the heavy lifting.

The code looked something like this:

[HttpPost("process")]
public async Task<IActionResult> ProcessAsync()
{
    // Simulate quick validation
    await Task.Delay(10); 

    // Fire and forget the heavy work
    _ = ProcessHeavyWorkAsync();

    return Accepted();
}

private async Task ProcessHeavyWorkAsync()
{
    // This calls a slow third-party API
    await _httpClient.GetAsync("https://slow-service.com/api/");
    // ... do more work ...
}

The Hidden Trap

On the surface, this looks standard. We aren't blocking the main thread. We are using async/await. What’s the problem?

The problem lies in the lack of cancellation context.

Imagine a scenario where:

1. The client sends the request.
2. The API returns 202 Accepted immediately.
3. The background task starts ProcessHeavyWorkAsync.
4. The third-party service is slow, taking 30 seconds.
5. Meanwhile, the user closes the tab, or the load balancer times out the connection.

In a traditional synchronous world, the server would kill the request. But here, the background task is detached. It is still running on the ASP.NET Core SynchronizationContext (or rather, the thread pool context). It will complete, regardless of whether the client is still listening.

If this happens once, no one notices. But under load?

The Cascade Effect

When we hit peak traffic, 1,000 requests a second were triggering these background jobs.

• Job A starts, waits 30s.
• Job B starts, waits 30s.
• ...
• Job 1000 starts, waits 30s.

The thread pool has a finite number of threads. While these tasks are awaiting the HttpClient, the threads are released back to the pool. However, the HttpClient has its own connection limits (usually 50 per host by default for HttpMessageHandler). If you have 1,000 concurrent tasks hitting the same slow service, you saturate the connection pool.

Wait, if async releases threads, why the exhaustion?

The issue wasn't just the HTTP call. ProcessHeavyWorkAsync had some CPU-bound work after the HTTP call. Because we didn't propagate a CancellationToken, the code never checked if it should stop. Under extreme load, the combination of connection pool saturation and queued CPU work caused the thread pool to grow to its maximum limit. New requests couldn't get threads to even start their background jobs. The system fell over.

The Fix: Threading the Token

The fix required two changes:

1. Propagate the token: Pass a CancellationToken from the top-level request into the background task.
2. Respect the token: Actually check it during the work.

[HttpPost("process")]
public async Task<IActionResult> ProcessAsync(CancellationToken cancellationToken)
{
    // We use cancellationToken from the DI/ActionDescriptor
    // This token is canceled if the client disconnects or the server shuts down.

    _ = ProcessHeavyWorkAsync(cancellationToken);
    return Accepted();
}

private async Task ProcessHeavyWorkAsync(CancellationToken token)
{
    // Pass the token to HttpClient
    var response = await _httpClient.GetAsync("https://slow-service.com/api/", token);

    if (token.IsCancellationRequested)
    {
        return; // Stop processing
    }

    // Check again before CPU-heavy work
    if (token.IsCancellationRequested)
    {
        return;
    }

    // Do CPU work
    await DoCPUIntensiveWorkAsync(token);
}

Note: In .NET 6+, CancellationToken can be injected directly into Action methods. Before that, you’d use HttpContext.RequestAborted.

Why This Is Harder Than It Looks

Many developers think "I just need to add a timeout." But a timeout on the HTTP client only tells the client when to give up waiting. It doesn't tell the server to stop working.

A CancellationToken is a cooperative signal. It tells every layer of your application: "The context is no longer valid. Stop doing work."

When you fail to propagate this:

1. Resource Leaks: Background jobs keep consuming memory and threads.
2. Stale Data: You might update the database after the user has already moved on, causing consistency issues.
3. Shutdown Hangs: If you don't respect the token, your application may hang when trying to shut down gracefully, because it’s waiting for these "zombie" tasks to finish.

Practical Takeaway

Always ask yourself: Who is allowed to cancel this work?

If you are doing fire-and-forget work, you need a cancellation strategy. If it’s tied to the request, use RequestAborted. If it’s tied to the application lifecycle, use the CancellationToken passed to HostedService.

In code review, look for Task.Run or async methods that take no parameters. Ask: "What happens if the client disconnects? What happens if the app shuts down?" If the answer is "it keeps running," you have a bug waiting to happen under load.

CSharp #DotNet #Debugging #Architecture