the retry storm section is the one most engineers learn the hard way. what makes it particularly nasty is that automatic retries often have no backoff by default, so a service that's struggling gets hammered with 2-3x the original load right when it can least handle it. one thing I'd add: client-side retry budgets help here, where each client tracks total retries in flight and self-limits, rather than leaving it entirely to exponential backoff. have you found circuit breakers or client-side budgets more effective in practice for preventing the snowball?
Retry budgets, for sure. They stop things from getting out of hand in the first place by limiting how many retries clients can throw at a struggling service. Circuit breakers still matter, but they usually step in after things have already started to go sideways. So yeah, both are useful, but if you’re trying to prevent that snowball effect early, retry budgets tend to do more heavy lifting.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
the retry storm section is the one most engineers learn the hard way. what makes it particularly nasty is that automatic retries often have no backoff by default, so a service that's struggling gets hammered with 2-3x the original load right when it can least handle it. one thing I'd add: client-side retry budgets help here, where each client tracks total retries in flight and self-limits, rather than leaving it entirely to exponential backoff. have you found circuit breakers or client-side budgets more effective in practice for preventing the snowball?
Retry budgets, for sure. They stop things from getting out of hand in the first place by limiting how many retries clients can throw at a struggling service. Circuit breakers still matter, but they usually step in after things have already started to go sideways. So yeah, both are useful, but if you’re trying to prevent that snowball effect early, retry budgets tend to do more heavy lifting.