*"How I Fixed a 300ms API Latency Spike—Without Touching the Database"*

# How I Fixed a 300ms API Latency Spike—Without Touching the Database

A seemingly minor change—a new validation rule—caused a sudden 300ms latency spike in a .NET 8 API under load. No database queries or external calls were affected. The culprit? A hidden `IAsyncEnumerable` leak in the middleware pipeline, combined with `CancellationToken` misuse in a background task. Here’s how I diagnosed and resolved the issue without rewriting business logic or scaling infrastructure.

---

## The Problem: Silent Performance Degradation

After deploying a new validation rule, response times under load increased from ~80ms to ~380ms. No database or external API calls were modified. The issue was subtle but impactful:

- **Middleware was leaking `IAsyncEnumerable` streams**, causing unnecessary async operations to persist even after the request completed.
- **A background task was ignoring `CancellationToken`**, preventing timely cleanup of resources.

This resulted in a cascading effect: middleware operations lingered, background tasks ran unnecessarily, and response times degraded.

---

## Diagnosis: `dotnet-trace` to the Rescue

To identify the root cause, I followed these steps:

1. **Reproduced the issue** under load using Playwright, simulating concurrent requests.
2. Ran `dotnet-trace collect --providers Microsoft-AspNetCore.Hosting:Verbose` to capture detailed middleware execution.
3. Analyzed the trace output for:
   - Unexpected async stream operations (`IAsyncEnumerable`) lingering after request completion.
   - Background tasks that weren’t respecting `CancellationToken`.

The trace revealed:
- Middleware was not properly disposing of async streams, causing them to remain active.
- A background task was performing work even after the request was canceled, due to ignored `CancellationToken`.

---

## The Fix: Cleanup and Cancellation Respect

### 1. Fixing the Middleware Leak
The middleware component was using `IAsyncEnumerable` to process validation rules but wasn’t ensuring streams were properly disposed. I updated it to:
- Use `await foreach` with explicit disposal or ensure streams were awaited to completion.
- Example:

csharp
// Before: Potential leak
await foreach (var item in asyncEnumerable)
{
// Process item
}

// After: Ensure cleanup
await foreach (var item in asyncEnumerable.ConfigureAwait(false))
{
// Process item
}

### 2. Respecting `CancellationToken`
The background task was performing cleanup but wasn’t checking for cancellation. I modified it to:
- Pass the `CancellationToken` from the request context.
- Check for cancellation periodically.

csharp
// Before: Ignored cancellation
await Task.Run(() => PerformCleanup());

// After: Respect cancellation
await Task.Run(async () =>
{
while (!cancellationToken.IsCancellationRequested)
{
await PerformCleanup();
await Task.Delay(100, cancellationToken);
}
}, cancellationToken);

### 3. Restoring Performance
After applying these changes:
- The middleware no longer leaked streams.
- Background tasks respected cancellation and terminated promptly.
- Response times dropped back under 100ms under load.

---

## Key Takeaways

1. **Middleware leaks are silent killers**: Always validate that async streams (`IAsyncEnumerable`) are properly disposed or awaited to completion. Use `dotnet-trace` to detect lingering operations.
2. **`CancellationToken` is non-negotiable**: Background tasks must respect cancellation to avoid unnecessary work and resource leaks. Pass the token explicitly and check it periodically.
3. **Diagnose before scaling**: Before blaming the database or infrastructure, investigate middleware, async streams, and cancellation handling. Tools like `dotnet-trace` can reveal hidden bottlenecks.

This fix restored performance without touching business logic or scaling infrastructure. The lesson? Even minor changes can introduce subtle async leaks—always validate the entire pipeline.

---

---