When a Healthy Service Still Causes a 500: Understanding Stale Sockets and Resilient APIs
Distributed systems have an interesting property: a service can be completely healthy while requests to that service are still failing.
One example is the seemingly mysterious:
ECONNRESET
socket hang up
At first glance, it is tempting to assume that the downstream service crashed, restarted, or became unavailable.
But that isn't necessarily the case.
A connection can fail even when both applications are running perfectly. The issue can exist at the TCP connection layer, where the client and server have different expectations about how long an idle connection should remain open.
When this kind of networking issue is combined with an overly strict dependency on a downstream service and poor error handling, a small transient failure can turn into a user-facing 500.
This article explains how that happens and how to design systems that handle it gracefully.
Understanding ECONNRESET
ECONNRESET means that an established network connection was unexpectedly closed or reset.
A simplified flow looks like this:
Client Server
| |
|-------- TCP connection ---------->|
| |
| connection idle |
| |
| |
| closes connection
| X
| |
|-------- request ----------------->|
| |
|<--------- ECONNRESET -------------|
The important thing to understand is:
ECONNRESETdoes not automatically mean the server is down.
The server may be healthy and serving thousands of other requests.
The problem may simply be that the particular TCP connection being used by the client has already been closed.
The Stale-Socket Race
Modern HTTP clients often reuse TCP connections through keep-alive.
This avoids creating a new TCP connection for every request and can significantly improve performance.
However, connection reuse introduces another responsibility: both sides need to agree on connection lifetime.
Consider a server with an idle timeout of approximately five seconds.
Server idle timeout = 5 seconds
Now imagine the client believes an idle socket is still available:
Client:
"Socket is available. I'll reuse it."
Server:
"This socket has been idle for five seconds. I'll close it."
The client then attempts to send a request:
Client → request
|
X
socket already closed
|
↓
ECONNRESET
This is known as a stale-socket race.
Neither service necessarily has a bug in isolation.
The problem exists because their connection-lifecycle assumptions don't align.
Why Connection Pools Matter
A connection pool manages reusable connections rather than treating every request as an independent connection.
Conceptually:
Application
|
v
HTTP Client Pool
|
┌───────────┼───────────┐
| | |
v v v
Socket 1 Socket 2 Socket 3
| | |
└───────────┼───────────┘
|
v
Downstream
A properly configured pool can control:
- Maximum connections
- Idle connections
- Connection reuse
- Socket lifetime
- Connection cleanup
One particularly important setting is the idle socket timeout.
Client Timeout vs Server Timeout
Suppose:
Server idle timeout = 5 seconds
A good strategy is for the client to clean up its idle connections slightly earlier:
Client idle timeout = 4 seconds
Server idle timeout = 5 seconds
So:
0s ───────────── 4s ───────────── 5s
Client
closes socket
Server would
close socket
The client doesn't get an opportunity to reuse a socket that the server has already decided to close.
However, this value should never be blindly assumed.
The downstream server's actual timeout should be confirmed.
The general principle is:
The client should know and respect the server's connection-lifetime policy.
Timeouts Are Not Optional
Another common problem is allowing HTTP requests to run indefinitely.
Consider:
timeout = 0
This effectively means there is no request timeout.
If a downstream service accepts a request but never responds:
Request
|
├── waiting...
├── waiting...
├── waiting...
├── waiting...
└── waiting...
the caller can remain stuck indefinitely.
This can consume:
- HTTP connections
- Memory
- Event-loop resources
- Request slots
- Application capacity
Eventually, a slow downstream service can cause problems upstream.
A better approach is to define an explicit timeout:
Request
|
|------ waiting ------|
|
timeout
|
v
handle failure
The correct timeout depends on the API and business requirement, but every network dependency should have a deliberate timeout policy.
The Bigger Problem: Dependency Coupling
A networking failure becomes much more dangerous when an optional dependency is placed directly on a critical path.
Consider an API that needs to perform a core operation and also fetch some optional metadata:
Request
|
├── Core operation
|
└── Fetch optional metadata
If both are treated equally:
Optional metadata
|
X
network error
|
v
Entire request fails
|
v
500
That's usually undesirable.
The core operation shouldn't necessarily depend on the optional metadata being available.
Critical vs Non-Critical Dependencies
A useful design exercise is to classify dependencies.
Critical dependency
If it fails, the operation genuinely cannot continue.
Payment authorization
↓
Cannot complete transaction
Failure should propagate.
Non-critical dependency
The operation can still succeed without it.
Fetch recommendation
↓
Recommendation unavailable
↓
Continue without recommendation
Failure should generally be contained.
The distinction is important because not every dependency deserves the same failure semantics.
Graceful Degradation
For non-critical dependencies, use a fallback.
For example:
Fetch optional data
|
├── Success → use actual value
|
└── Failure → use fallback
This changes the behaviour from:
Dependency fails
↓
500
to:
Dependency fails
↓
Fallback
↓
Request succeeds
This is called graceful degradation.
The system isn't pretending that the dependency succeeded.
It is simply ensuring that the dependency's failure doesn't unnecessarily break the primary operation.
Retries Can Help — But Carefully
Some network failures are transient.
For example:
Request
↓
ECONNRESET
↓
Retry
↓
Success
A small number of retries can make an API more resilient to temporary connection failures.
For example:
Attempt 1 → ECONNRESET
↓
short delay
↓
Attempt 2 → success
Retries should generally have:
- A maximum retry count
- Exponential backoff
- Jitter
- Error-specific retry rules
Without limits, retries can make an incident worse.
Imagine 1,000 requests failing and all of them immediately retrying:
1,000 requests
↓
1,000 failures
↓
1,000 immediate retries
↓
more load
↓
more failures
This can create a retry storm.
Retries should therefore be treated as a recovery mechanism, not as a replacement for proper connection management.
Observability Is Part of Reliability
A system that recovers from failures but provides no useful diagnostic information is still difficult to operate.
Consider two alerts.
Alert A
Unexpected error occurred
Alert B
HTTP request failed
Error: ECONNRESET
Dependency: downstream-service
Endpoint: GET /resource
Retry attempt: 2
Timeout: 8000ms
The second alert immediately gives an engineer a direction to investigate.
Good observability should make the system explain its own failures.
Putting It All Together
A resilient HTTP client should ideally have several layers of protection:
HTTP Request
|
v
Connection Pool
|
v
Proper Timeouts
|
v
Request to downstream
|
┌────────┴────────┐
| |
Success Failure
| |
| Is it transient?
| |
| ┌─────┴─────┐
| Yes No
| | |
| Retry Handle
| |
| Success?
| |
| ┌────┴────┐
| Yes No
| | |
└──────┘ Fallback
|
v
Continue
The goal is not to eliminate every possible network failure.
That's impossible.
The goal is to ensure that failures are:
Contained → Observable → Recoverable
Practical Checklist
When building service-to-service HTTP communication, ask:
Connection management
- Are connections reused?
- Is there a connection pool?
- What is the server's idle timeout?
- What is the client's idle socket timeout?
- Can stale sockets be reused?
Timeouts
- Does every request have an explicit timeout?
- Is the timeout appropriate for the endpoint?
- Can a request remain stuck indefinitely?
Retries
- Which errors are safe to retry?
- How many retries are allowed?
- Is exponential backoff used?
- Is jitter applied?
Dependency design
- Is the downstream call actually required?
- Can the operation succeed without it?
- Is there a fallback?
- Should the dependency be moved off the critical path?
Error handling
- Are proper
Errorobjects being thrown? - Is the original stack preserved?
- Are HTTP status and response details retained?
- Do logs clearly identify the failing dependency?
Observability
- Can we identify the failing service?
- Can we identify the endpoint?
- Can we identify the error type?
- Can we distinguish timeout, reset, connection refusal, and HTTP errors?
Conclusion
A socket hang up can look like a mysterious production failure.
But the underlying issue may be surprisingly simple:
Idle connection
↓
Server closes it
↓
Client attempts reuse
↓
ECONNRESET
The real engineering challenge is what happens after that failure.
If the application:
- manages connections correctly,
- sets explicit timeouts,
- retries transient failures,
- isolates non-critical dependencies,
- provides sensible fallbacks,
- and preserves useful error information,
then a small network failure remains a small network failure.
If it doesn't, the same event can cascade into:
Network failure
↓
Dependency failure
↓
Application failure
↓
500 response
↓
Poor observability
↓
Long incident
The most resilient systems aren't the ones that assume dependencies will never fail.
They're the ones designed with the assumption that dependencies will fail — and the system will continue to work anyway.
Top comments (0)