Something failed.
You tried again.
That is the whole idea,
and most days it works.
The network stutters.
The second call goes through.
Nobody ever knows.
Then comes the day
the service you are calling
is not stuttering.
It is drowning.
And every client you wrote
responds the same way.
Again.
Again.
Again.
A retry is help
when the failure is random.
A retry is weight
when the failure is load.
Your code cannot tell
the difference,
so you have to.
Three attempts, not thirty.
Wait longer each time.
Add a little noise to the wait,
or every caller in the fleet
comes back at the same instant
like a crowd through one door.
And then stop.
Something has to stop.
The limit is not pessimism.
It is the only part of retrying
that protects anybody.
Because the worst version of this
never errors at all.
The call times out at thirty seconds.
You retry it four times.
Someone waited two minutes
for an answer
you could have given
in one.
Meanwhile a queue somewhere
is holding every request
that arrived while you were hoping.
Retries do not just repeat work.
They multiply it.
One click,
five attempts,
each fanning into three services
that each retry three times.
Do that arithmetic yourself,
before production does it for you.
And ask whether the thing
is safe to repeat at all.
A read is safe.
A charge is not.
If the first call succeeded
and only the answer got lost,
your second attempt
is a second charge
on somebody's real card.
The word for the fix
is idempotency.
It is boring
and it will save you.
Then decide what happens
when the attempts run out.
Not a stack trace.
A decision.
Fail the request cleanly.
Queue it for later.
Serve the stale value
and say plainly that it is stale.
Something honest.
Because the retry
was never the recovery plan.
It was the delay
before you needed one.
– Serguey Asael Shinder
Top comments (0)