This post was originally published on MatrixTrak.com — the production reliability toolkit for trading bot operators and .NET engineers.
When requests hang forever and recycling releases stuck work: why missing timeouts create backlog, how to add budgets safely, and the rollout plan that prevents new incidents..
Most production incidents do not start as "down." They start as waiting.
At 09:12 a dependency slows down. Your ASP.NET instances look healthy. CPU is fine. Memory is fine. But requests stop finishing. In-flight count climbs. Connection pools stop turning over. You scale out and it does not help because the new instances just join the waiting.
The cost is not subtle. Backlog grows, SLAs fail, and on-call starts recycling processes because it is the only thing that releases the stuck work. Then the incident repeats next week because nothing changed about the waiting.
This post gives you a production playbook for .NET: how to set time budgets, wire cancellation, and roll it out without triggering a new outage.
Rescuing an ASP.NET service in production? Start at the .NET Production Rescue hub.
If you only do three things
- Write down a total budget per request/job (then enforce it).
- Set per-attempt timeouts for each dependency and log
elapsedMs,timeoutMs, and the decision (retry/stop/fallback). - Propagate cancellation end-to-end so work stops (no zombie work after timeouts).
Why requests hang forever: infinite waits capture capacity
Missing timeouts are not a performance problem. They are a capacity problem.
When a call can wait forever, it will eventually wait longer than your system can afford. While it waits, it holds something your service needs to operate: a worker slot, a thread, a connection, a lock, or a request budget.
Once enough requests or jobs are holding those resources, the system stops behaving like a service and starts behaving like a queue you did not design. From the outside it looks like "everything is slow." Underneath, you are accumulating work you cannot complete.
Timeouts are not a tuning knob. They are a product decision about how long you are willing to wait before you choose a different path.
How to diagnose: find what's waiting forever without budgets
Before you change numbers, classify the waiting. This prevents the classic failure: enforcing a strict timeout everywhere and then declaring "timeouts broke production."
Do these checks in order:
- Identify the top 3 dependencies on the slow path (HTTP/SQL/vendor SDK)
- Find the longest observed durations (not averages)
- Confirm whether work stops after a timeout decision (cancellation honored) or keeps running (zombie work)
- Map retries and total time budget (timeouts + retries are one policy)
- Look for backlog signals: queue depth, oldest age, in-flight rising while completions flatten
Fast triage table (what to check first)
| Symptom | Likely cause | Confirm fast | First safe move |
|---|---|---|---|
| In-flight climbs, completions flatten, CPU looks fine | Infinite waits capturing capacity | Backlog signals (oldest request age, queue depth) rise during the slowdown | Add explicit per-attempt timeouts + a total budget; wire cancellation end-to-end |
| Requests “time out” but downstream keeps working | Zombie work (timeouts without cancellation propagation) | Work continues after client has given up; late side effects appear | Pass CancellationToken into every async call; use linked CTS with CancelAfter
|
| Retry policies make the incident worse | Timeouts + retries not treated as one budget | Attempt count rises while latency rises; total time grows unbounded | Cap attempts + add total budget; stop retrying timeouts into a slow dependency |
| One dependency dominates the slow window | Budget violated by one downstream (SQL/vendor) | Dependency logs show one name repeatedly exceeding timeoutMs
|
Add bulkhead/caps + conservative timeout; add fallback/queue path |
If you cannot answer "what is our total time budget per request or job run," that is the first gap to close.
Why infinite waits keep happening: common patterns in .NET
Infinite waits are rarely one bug. They are an emergent property of a system that is allowed to wait without limits.
These are the patterns that create repeat incidents:
- HTTP calls without an attempt budget or without cancellation propagated into the call
- SQL commands waiting behind locks, or long queries with no operator boundary
- background work with no max runtime and no heartbeat
- integrations that block on a vendor outage while you keep retrying
This is why process recycling "works." It does not fix the dependency. It discards the waiting work and frees resources temporarily.
How to fix: add budgets, fallbacks, and stop rules
Teams avoid adding timeouts because they have seen the short-term effect: more errors.
That story is usually true, and it is still the wrong conclusion.
Timeouts do not create fragility. They reveal fragility that already exists: cancellation not wired through, retries stacking waits, and no fallback path when the budget is spent.
Decide three things per request or job type:
- Total budget: how long the system can afford before it must choose a different path
- Attempt budgets: how long any single dependency call is allowed to hold resources
- Stop rules: when you stop, quarantine, or escalate instead of retrying optimistically
Fallback choices that work in the real world:
- return a clear error and stop doing work
- serve cached or stale data
- enqueue work and respond immediately
- partial response (where safe)
If your only behavior is "hang forever," you have guaranteed that dependency slowdowns will become outages.
A timeout matrix you can actually operate
You do not need perfect numbers. You need consistent budgets and an operator story.
Start with a total budget, then allocate smaller budgets inside it. Keep it boring.
Practical starting points:
User-facing web requests
- Total budget: 3-10 seconds depending on the page and fallback
- Attempt budgets: usually 1-3 seconds per dependency call
Internal service-to-service calls
- Total budget: 2-10 seconds depending on criticality
- If slow: prefer predictable failure + queue/fallback over waiting
Background jobs
- Budget must be explicit (seconds/minutes)
- Always include a max runtime guardrail and a poison path
Database calls
- Budget depends on the query type and lock profile
- "It is slow" must become "it exceeded budget X" (observable + bounded)
Implementation patterns (bounded and boring)
The goal is not clever code. The goal is to make waiting impossible without you choosing it.
Two rules prevent most repeat incidents:
- Always have a total budget (end-to-end)
- Always propagate cancellation so work stops, not just waiting
Example: enforce a budget and pass the token through the call.
public async Task
Resources
Internal:
- .NET Production Rescue
- Why your background jobs hang forever (and no one notices)
- Requests timing out but CPU normal: thread pool starvation in ASP.NET - infinite waits cause starvation
- Retries making outages worse: when resilience policies multiply failures in .NET - retries without timeouts stack waits
- Cannot trace requests across services: why correlation IDs die at boundaries in .NET - trace hung request chains
- Polly retries making outages worse: how retry storms multiply failures in .NET - bounded retries need timeouts
External:
- https://learn.microsoft.com/dotnet/fundamentals/networking/http/httpclient
- https://learn.microsoft.com/dotnet/standard/threading/cancellation-in-managed-threads
- https://learn.microsoft.com/dotnet/api/system.data.sqlclient.sqlcommand.commandtimeout
- https://github.com/App-vNext/Polly
If you want more assets like the timeout matrix (plus runbooks and logging schemas), that is what Axiom is becoming.
Join to get notified as we ship practical operational assets you can use during incidents and during rollout work, not generic tutorials.
Checklist (copy/paste)
- [ ] Each request/job has a documented total budget.
- [ ] Each dependency call has an explicit attempt timeout (HTTP/SQL/vendor SDK).
- [ ] Cancellation is propagated end-to-end (no zombie work after timeout).
- [ ] Retries + timeouts are treated as one policy (total budget + capped attempts).
- [ ] Dependency logs include: correlation/run id, dependency, attempt,
elapsedMs,timeoutMs, totalBudget, outcome, decision. - [ ] Backlog signals exist (in-flight, queue depth, oldest age) and are monitored.
- [ ] Rollout follows observe → warn → enforce and is done one dependency at a time.
- [ ] A fallback exists when the budget is spent (stop, degrade, cache, queue).
- Infinite waits capture capacity and create backlog.
- Timeouts are budgets + fallbacks, not a magic performance knob.
- Roll out safely: observe -> warn -> enforce, and wire cancellation end-to-end.
Free Tools for Trading Bot Reliability
Every article on MatrixTrak is backed by free, open-source tools you can use right now:
- Incident Runbook Builder — decision trees + escalation paths → Try it free
Originally published at matrixtrak.com.
Top comments (0)