An external service started responding in 3 seconds instead of 30 milliseconds. Nothing crashed, the logs are clean, but the connection pool is full, the request queue is growing, and a minute later your service is down together with it — even though it is perfectly healthy.
Fault tolerance is a set of decisions about what to do at moments like this: whether to retry, how long to wait, what to return when there will be no answer at all. Almost all of them are made in code, not in the cluster configuration. Below is how they look in Python and Java, on ordinary tasks: calling an external API, querying a database, background processing.
Retry and exponential backoff
When a call to an external service fails over the network, it is reasonable to try again — the failure may have lasted a second. Everything else about retries is not obvious.
Python, Tenacity
Retries are easy to write by hand: a while, an attempt counter and time.sleep(). The problem is that a handwritten loop is almost always naive: a fixed pause with no spread, except Exception instead of telling error types apart, no budget, no ceiling on total time. Tenacity turns those decisions into explicit parameters: you have to make them consciously instead of skipping them by default.
import logging
import requests
from tenacity import (
retry,
stop_after_attempt,
wait_random_exponential,
retry_if_exception,
before_sleep_log,
)
logger = logging.getLogger(__name__)
def is_retryable(exc: BaseException) -> bool:
if isinstance(exc, (requests.exceptions.ConnectionError,
requests.exceptions.Timeout)):
return True
if isinstance(exc, requests.exceptions.HTTPError):
code = exc.response.status_code if exc.response is not None else None
# 429 - we are being asked to wait, 5xx - the server broke.
# Retrying 4xx other than 429 is pointless: the request won't get any more valid.
return code == 429 or (code is not None and 500 <= code < 600)
return False
@retry(
stop=stop_after_attempt(3),
# full jitter: without a random spread every client
# comes back in the very same millisecond
wait=wait_random_exponential(multiplier=0.1, max=5),
retry=retry_if_exception(is_retryable),
# without reraise the caller gets tenacity.RetryError
# instead of the original requests exception
reraise=True,
before_sleep=before_sleep_log(logger, logging.WARNING),
)
def fetch_data(url):
# the timeout is mandatory: without it retries hang instead of retrying
response = requests.get(url, timeout=(1, 3))
response.raise_for_status()
return response.json()
try:
data = fetch_data("https://api.example.com/data")
except requests.exceptions.RequestException as e:
logger.error("Failed to fetch data: %s", e)
The @retry decorator does not catch every exception, only those that pass the is_retryable check: network errors, timeouts, 429 and 5xx. A 400 Bad Request goes up on the very first attempt — retrying it is pointless, the request will not become any more valid.
The pause grows exponentially, but what is taken is not the bound itself: it is a random value between zero and that bound (the ceiling goes 0.1 → 0.2 → 0.4 … up to 5 seconds). This is full jitter. If a service went down under a thousand clients, then without a spread all of them come back at once and take it down again at the exact moment it was coming back up.
reraise=True is not cosmetic here: without it, after three failures Tenacity raises its own RetryError, the calling code catches something other than what it expects, and except requests.exceptions.RequestException does not fire at all.
And the timeout inside the request itself is mandatory. Without it an attempt does not fail — it hangs: there is nothing to retry, because the first attempt has not finished yet.
Java, Resilience4j
The standard choice ever since Hystrix went into maintenance in 2018. The idea is the same, but modular: Retry, CircuitBreaker, Bulkhead and RateLimiter are plugged in separately and combined with each other. It integrates with Spring Boot through annotations, and the retry policy itself lives in application.yml.
@Service
public class OrderService {
// The annotation says: use the Retry instance named "inventoryService"
@Retry(name = "inventoryService", fallbackMethod = "fallbackInventory")
public Inventory getInventory(int productId) {
return externalInventoryClient.getInventory(productId);
}
}
The annotation only marks the attachment point. The policy itself — how many times to retry, with what pause, and which errors count as a reason to retry at all — lives in the configuration, and the two are linked by name: name = "inventoryService" in the annotation corresponds to instances.inventoryService in the YAML.
resilience4j:
retry:
instances:
inventoryService:
max-attempts: 3
wait-duration: 200ms
enable-exponential-backoff: true
exponential-backoff-multiplier: 2
# the same jitter as in the Python example:
# without it every client retries in lockstep
enable-randomized-wait: true
randomized-wait-factor: 0.5
retry-exceptions:
- java.io.IOException
- java.util.concurrent.TimeoutException
ignore-exceptions:
- com.example.InventoryNotFoundException
retry-exceptions and ignore-exceptions are the declarative equivalent of is_retryable from the Python example. The list of exceptions is the policy: leave it unset and Resilience4j will retry everything, business logic errors included. Here InventoryNotFoundException is explicitly ignored — the item does not exist, and it will not appear on the third attempt.
The classes in retry-exceptions have to match what your client actually throws. Feign wraps network failures into FeignException, RestTemplate into ResourceAccessException, and java.io.IOException in the list will never reach them.
If the service still has not answered after all attempts, the fallback is called:
public Inventory fallbackInventory(int productId, Exception e) {
// ignore-exceptions disables retries, not the fallback:
// business exceptions land here, and they must be passed through.
if (e instanceof InventoryNotFoundException notFound) {
throw notFound;
}
log.warn("Inventory service unavailable for product {}: {}",
productId, e.getMessage());
inventoryDegraded.increment(); // metric: how often we run degraded
// Don't invent the stock level. Zero is indistinguishable from a real zero
// and reaches business logic as the fact "out of stock".
return Inventory.unknown(productId);
}
ignore-exceptions disables only the retries — the fallback is invoked anyway. That is why business exceptions have to be rethrown by hand, otherwise "item not found" turns into invented data.
Returning zero from a fallback like this is not allowed: the inventory service is merely unavailable, and because of such a stub the shop will show the entire catalogue as sold out.
From the outside none of this is visible. A system running on stubs looks healthy: no errors, and latency is even better than usual — it is not calling the service that is down. So you need a separate metric for the fact of degradation itself, otherwise you will hear about the fake answers from your users.
Retries multiply
Three services in a chain, three attempts each — one user request turns into 27 requests to the bottom service. The very one that is already down. Retries scattered across every layer "just in case" do not add up into protection, they multiply into load: traffic to the failed service grows exactly when it needs to recover.
The cure is that exactly one layer retries. Pick a layer — usually the one closest to the external dependency — and retry only there; the rest propagate the error upward. While you are at it, check that retries are not enabled by default in your HTTP client and in the service mesh: even one retry per layer is already an eightfold amplification.
But even a single layer stays dangerous as long as the number of attempts is fixed. "Three attempts per request" behaves worst precisely under a mass outage: while everything works there are almost no retries, and the moment the service goes down there are three times the normal traffic. A budget breaks that link: count the share of retries in the total number of requests over a window and stop retrying once it crosses a threshold, usually 10–20%. A one-off failure is retried as before, a mass one damps itself.
That is how it is done in gRPC (retryThrottling), in AWS SDK retry quotas and in Envoy's retry_budget. On a bare HTTP client the minimal version is a token bucket: a success deposits a fraction of a token, a retry takes a whole one. That is where "one retry per five successes" at ratio=0.2 comes from. Out of tokens — stop retrying until successes bring them back.
import threading
class RetryBudget:
SCALE = 1000
def __init__(self, ratio: float = 0.2, max_tokens: int = 10):
self._per_success = round(ratio * self.SCALE)
self._max = max_tokens * self.SCALE
# start at half: retries work right away
self._tokens = self._max // 2
self._lock = threading.Lock()
def on_success(self) -> None:
with self._lock:
self._tokens = min(self._max, self._tokens + self._per_success)
def try_retry(self) -> bool:
with self._lock:
if self._tokens >= self.SCALE:
self._tokens -= self.SCALE
return True
return False # budget exhausted
The count is kept in whole thousandths of a token rather than in floats. This is not pedantry: at ratio=0.1 ten additions of 0.1 give 0.9999999999999999, and every tenth retry is silently lost. The lock is there for the usual reason: under a thread pool the check and the decrement are not atomic without it, and two threads will spend the same token.
A budget is kept per dependency. A single one for the whole application is meaningless: a failure of one API eats the budget of the others, and retries switch off where everything was fine.
Now it has to be wired into Tenacity. is_retryable alone is no longer enough — the decision depends not only on the error type, but also on the attempt number and on the state of the budget. So instead of retry_if_exception we pass an object of our own: Tenacity accepts any callable in the retry parameter, and hands it the whole state of the attempt.
class BudgetedRetry:
def __init__(self, budget: RetryBudget, max_attempts: int, name: str):
self._budget = budget
self._max_attempts = max_attempts
self._name = name
def __call__(self, retry_state) -> bool:
exc = retry_state.outcome.exception()
if exc is None or not is_retryable(exc):
return False
# Tenacity calls the predicate after the last attempt as well,
# even though there will be no retry. No reason to spend a token on that.
if retry_state.attempt_number >= self._max_attempts:
return False
if not self._budget.try_retry():
# This event must be in the metrics: it means the dependency
# is failing en masse, not once.
logger.warning("retry budget exhausted for %s", self._name)
return False
return True
The final decorator
max_attempts = 3
inventory_budget = RetryBudget(ratio=0.2, max_tokens=10)
@retry(
stop=stop_after_attempt(max_attempts),
wait=wait_random_exponential(multiplier=0.1, max=5),
retry=BudgetedRetry(inventory_budget, max_attempts, "inventory-api"),
reraise=True,
before_sleep=before_sleep_log(logger, logging.WARNING),
)
def fetch_data(url):
response = requests.get(url, timeout=(1, 3))
response.raise_for_status()
inventory_budget.on_success() # a success refills the budget
return response.json()
on_success() is called after raise_for_status(), not before — otherwise a 500 would count as a success and the budget would be refilled exactly when it must not be.
The resulting behaviour is this: while the service answers, try_retry() almost always returns True — there are plenty of successes and tokens to spare. The moment the service goes down and the successes stop, the accumulated reserve is spent on the first few retries and is never topped up — after that retries switch themselves off. The max_tokens ceiling bounds that reserve: a service that stayed healthy for a day must not earn a day's worth of retries.
Resilience4j has no ready-made budget — its Retry can only count attempts. People build one by hand with the same counter, or push it down to the infrastructure: in Envoy and Istio retry_budget is configured in the sidecar.
Circuit Breaker
Retries help while the outage is short. If a service is down for ten minutes, retries only burn threads: every request honestly waits out its timeout only to get the same error. A circuit breaker removes that wait — after a series of failures it stops calling the troubled service altogether and fails immediately.
That does not free the threads already taken, they hang until their own timeouts. But it stops new ones from being taken, and the caller stays alive while somebody else's problem is being fixed.
There are three states, and the breaker polls nothing on its own — it counts the outcomes of ordinary calls. In Closed calls go through, each outcome is written into a window of recent calls, and when the failure rate in the window crosses the threshold the breaker moves to Open. In Open calls are not executed at all, the exception is raised instantly, and it stays that way for a fixed time — usually tens of seconds. Then Half-Open: a few trial requests are let through, the rest are still rejected. The decision is made on the failure rate among the trials, not on the first failure: with three trials and a 50% threshold one failed call will not open the circuit.
The window matters more than the threshold. Along with the threshold you must always set a minimum number of calls: without it two failed requests at startup make 100% errors and open the circuit out of nowhere. Next comes the window type — count-based or time-based. For sparse traffic a window of 20 calls stretches over tens of seconds and the breaker reacts to things long past; that is where a time-based window belongs.
Network exceptions and timeouts count as failures, obviously; business errors such as validation are better left uncounted, otherwise the circuit opens on a perfectly healthy service. HTTP status codes, however, are a trap of their own: a 500 response is, for the client, a perfectly successful network exchange. Until raise_for_status() is called in Python or recordFailurePredicate is set in Resilience4j, the breaker does not see the 500s and keeps the circuit closed while the service is completely broken.
A failure-rate threshold will not catch a slow service. It answers, but in seconds: no errors, failure rate at zero, circuit closed — while the caller holds threads waiting and goes down itself. So a slow call has to count as a failure too. Resilience4j has a separate pair of parameters for this: a duration threshold and an acceptable share of slow calls. In Python no separate mechanism is needed — with a timeout set, a slow call turns into an exception by itself and lands in the same counter.
Java, Resilience4j
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CallNotPermittedException;
@Service
public class PaymentService {
// The annotation wraps the method into a circuit breaker named "paymentService"
@CircuitBreaker(name = "paymentService", fallbackMethod = "paymentFallback")
public Receipt chargePayment(Order order) {
// The external call to the payment service
return paymentClient.charge(order);
}
// The circuit is open: the request never left, the money was definitely not taken.
// Failing here is safe.
public Receipt paymentFallback(Order order, CallNotPermittedException ex) {
log.warn("Payment circuit is open, rejecting order {} fast", order.id());
return Receipt.rejected(order, "payment temporarily unavailable");
}
// The call left and never came back. Whether the payment went through is unknown.
public Receipt paymentFallback(Order order, Throwable ex) {
log.error("Payment outcome unknown for order {}: {}", order.id(), ex.getMessage());
return Receipt.pending(order);
}
}
Resilience4j picks the fallback by exception type: the more specific one wins. There are two of them here because the two cases call for opposite answers.
CallNotPermittedException means the circuit is open and the request never went out to the network: the payment definitely did not go through, and refusing the customer is safe. Any other exception — a timeout, a dropped connection — means the opposite: the request left, there is no answer, and whether the payment went through you do not know.
A single shared fallback would glue the two cases into one answer, and whatever it returned would be wrong half the time. Declare failure — and on a timeout the customer retries and pays twice. Declare success — and the system records a payment that never happened. An open circuit is valuable precisely because it is the one situation where you know for certain that nothing happened.
The policy is set in the config under the same name.
resilience4j:
circuitbreaker:
instances:
paymentService:
sliding-window-type: COUNT_BASED
sliding-window-size: 20
minimum-number-of-calls: 10 # without this the threshold fires on two requests
failure-rate-threshold: 50
slow-call-duration-threshold: 2s
slow-call-rate-threshold: 50 # slow calls count as failures too
wait-duration-in-open-state: 30s
permitted-number-of-calls-in-half-open-state: 3
ignore-exceptions:
- com.example.PaymentDeclinedException # "card declined" is not a failure
Python, pybreaker
There are several ready-made implementations — pybreaker and circuitbreaker; the first one is below. The principle is the same: you create a breaker object and decorate the call with it.
import logging
import pybreaker
import requests
from prometheus_client import Counter
logger = logging.getLogger(__name__)
profile_degraded = Counter(
"profile_degraded_total", "Responses served from a stub instead of user-api"
)
class UserNotFound(Exception):
pass
breaker = pybreaker.CircuitBreaker(
fail_max=5, # five consecutive failures
reset_timeout=60, # after a minute it lets a trial call through
exclude=[UserNotFound], # "no such profile" is not a service failure
name="user-api",
)
@breaker
def fetch_profile(user_id):
r = requests.get(f"https://api.example.com/users/{user_id}", timeout=(1, 3))
if r.status_code == 404:
raise UserNotFound(user_id)
r.raise_for_status()
return r.json()
def _stub(user_id):
profile_degraded.inc()
return {"user_id": user_id, "name": "Unknown", "stale": True}
def get_user_profile(user_id):
try:
return fetch_profile(user_id)
except UserNotFound:
# There is no such user - that's an answer, not a failure. Pass it up as is.
raise
except pybreaker.CircuitBreakerError:
# The circuit is open: we never even reached the service
logger.warning("user-api circuit is open, serving stub")
return _stub(user_id)
except requests.exceptions.RequestException as e:
# The service did not answer; the breaker has already counted this outcome
logger.warning("user-api call failed (%s), serving stub", e)
return _stub(user_id)
UserNotFound stands apart. It is passed to exclude, so it does not increment the failure counter: the service is alive and answered on the merits, there is nothing to open the circuit for. And it goes upward as is — "there is no such user" is an answer, not a degradation, and replacing it with a stub would be a lie.
fail_max in pybreaker means consecutive failures, not a failure rate over a window. One success resets the counter: four failures, a success and four more failures will not open the circuit. The model is simpler than Resilience4j's, but also cruder — a service that steadily fails every second request will never open such a breaker. The advice to "open the circuit if 50% of the last 20 requests failed" belongs to Resilience4j; in pybreaker you cannot express it that way.
In Half-Open pybreaker lets a single trial call through: it passes — the circuit closes, it fails — the circuit reopens for another reset_timeout. Here the "first failure decides" model is correct, unlike Resilience4j, where the decision is made on the failure rate among several trial calls.
The breaker lives inside the process
In pybreaker and in Resilience4j alike, the state is kept in process memory by default. If the service runs in fifty pods, the dependency has not one breaker but fifty independent ones: each collects its own statistics and opens on its own.
The threshold is then measured against the traffic of a single instance: a window of twenty calls on a pod that sees one hundredth of the total traffic fills up a hundred times slower, and the breaker reacts late. On the upside, recovery comes out smeared — pods leave Open at different moments, and the load on the recovered dependency ramps up gradually instead of all at once. Either way, you cannot count on the cluster behaving in sync.
Shared state is possible — pybreaker has CircuitRedisStorage — but then Redis sits on the path of every call and becomes a new point of failure. Usually the breaker is left local and the overall picture is watched in the metrics.
That leaves the question of what numbers to put in the config. The threshold should be derived from the normal error level, not from zero: if a service normally serves 2% errors, a 10% threshold is five times above the baseline, and the breaker will stay silent through a real degradation. Look at the error rate distribution over a week and set the threshold above the peak of a normal day, but well below the level at which the dependency stops being useful.
And it belongs on a dependency, not on the service as a whole. One breaker around every call to somebody else's API will open because of trouble on a single endpoint and block the rest. A heavy report and a dozen light reads have different characteristics, which means different breakers.
Fallback and graceful degradation
The first question when something fails is not "what do we return instead" but "can we get the real answer some other way". A standby instance, a second zone, another provider for the same exchange rate — that is not degradation, it is simply a different route to the same data, and if it exists you never get to the fallback.
Degradation starts where there is nowhere to get the real answer. Then the question changes: what do we return instead, so that the user gets at least something and is not deceived in the process.
Four honest answers and one dishonest one
Stale data. A six-hour-old exchange rate, yesterday's catalogue, last hour's search results. Fine when the data changes slowly and the decision based on it is reversible. A staleness limit is mandatory — otherwise one day you will show week-old data — along with a staleness flag that travels upward.
A reduced response. A product page without the "Similar items" block, a feed without personalisation. The best kind of degradation: the user sees less, but everything they see is true.
"Unknown". An explicit "no data" instead of an invented value. It looks worse than a stub, but it is the only answer the calling code can handle correctly.
An explicit failure. "Payment is not possible right now, try again later." The only right answer where there is nothing to invent and nowhere left to degrade.
And the dishonest one: a plausible value. A zero balance, a rate of 1.0, an empty list instead of "we could not compute it". Such an answer is indistinguishable from a real one, passes every check and reaches business logic as a fact. The rule is simple: if you cannot tell from the returned value that it came from a fallback, you must not return it.
A fallback must not mask the problem forever: every time it fires it has to land in a metric, otherwise a system running on stubs looks healthy from the outside. More on that in the monitoring section.
Writing a fallback is easy, it is an alternative branch in a catch. The hard part starts with the question of what exactly to return from it. Often the fallback is attached to a circuit breaker or a retry through fallbackMethod, as in the examples above; without a library you write it by hand.
Python
import logging
import threading
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
import requests
from prometheus_client import Counter
logger = logging.getLogger(__name__)
rate_degraded = Counter("rate_degraded_total", "Rate served from cache")
API = "https://api.exchangerate.host"
MAX_STALENESS = timedelta(hours=6)
@dataclass(frozen=True)
class Rate:
value: float
fresh: bool
as_of: datetime | None = None
@dataclass(frozen=True)
class Cached:
value: float
as_of: datetime
@property
def age(self) -> timedelta:
return datetime.now(timezone.utc) - self.as_of
class RateUnavailable(Exception):
pass
class LastKnownGood:
def __init__(self) -> None:
self._d: dict[str, Cached] = {}
self._lock = threading.Lock()
def put(self, key: str, value: float) -> None:
with self._lock:
self._d[key] = Cached(value, datetime.now(timezone.utc))
def get(self, key: str) -> Cached | None:
with self._lock:
return self._d.get(key)
cache = LastKnownGood()
def get_exchange_rate(currency: str) -> Rate:
key = f"rate:{currency}:USD"
try:
resp = requests.get(f"{API}/latest?base={currency}", timeout=(1, 2))
resp.raise_for_status()
value = resp.json()["rates"]["USD"]
cache.put(key, value) # stash it away for the fallback
return Rate(value=value, fresh=True)
except requests.exceptions.RequestException as e:
logger.warning("exchange rate API failed for %s: %s", currency, e)
rate_degraded.inc()
cached = cache.get(key)
if cached is None or cached.age > MAX_STALENESS:
# Neither fresh data nor acceptably old data. There is no rate - and we won't lie.
raise RateUnavailable(currency) from e
# A stale rate is a valid answer, but the caller must know that it is stale
return Rate(value=cached.value, fresh=False, as_of=cached.as_of)
The cache has a shelf life: six hours is still an exchange rate, thirty is already fiction. If there is no acceptably old data, the function does not return anything plausible — it raises RateUnavailable and lets the caller decide whether to show an error or hide the price block. And the fresh flag travels upward: from a bare number you cannot tell a cached answer from a live one, from a Rate you can.
Java
import org.springframework.web.client.RestClientException;
import java.time.Duration;
@Service
public class RateService {
private static final Duration MAX_STALENESS = Duration.ofHours(6);
public Rate getExchangeRate(String currency) {
try {
ExchangeRates rates = restClient.get()
.uri("/latest?base={cur}", currency)
.retrieve()
.body(ExchangeRates.class);
// the body may come back empty - then the next line would throw an NPE
if (rates == null) {
throw new RestClientException("empty body");
}
double value = rates.rates().get("USD");
cache.put(currency + ":USD", value); // stash it away for the fallback
return Rate.fresh(value);
} catch (RestClientException e) {
log.warn("Exchange rate API failed for {}: {}", currency, e.getMessage());
rateDegraded.increment();
return cache.get(currency + ":USD")
.filter(c -> c.age().compareTo(MAX_STALENESS) < 0)
.map(c -> Rate.stale(c.value(), c.asOf()))
// neither fresh data nor acceptably old data - there is no rate
.orElseThrow(() -> new RateUnavailableException(currency, e));
}
}
}
The RestClient timeout is set on the client rather than in the call chain, and applies to every request made through it at once:
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.web.client.RestClient;
import java.time.Duration;
@Configuration
public class RateClientConfig {
@Bean
public RestClient rateRestClient() {
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setConnectTimeout(Duration.ofSeconds(1));
requestFactory.setReadTimeout(Duration.ofSeconds(2));
return RestClient.builder()
.baseUrl("https://api.exchangerate.host")
.requestFactory(requestFactory)
.build();
}
}
Far from everything can degrade. Analytics, recommendations, reviews, view counters — stale data offends nobody there. But the outcome of a payment is never replaced by a guess under any circumstances: there it is either the real answer or an honest "we don't know", and never a plausible number.
The fallback is the least tested code in the system
It only runs on failure, which means it is almost never exercised in tests, and it fires exactly when everything is on fire.
A fallback must not depend on anything that can go down together with the main path. A cache in Redis is a network dependency too — if Redis died along with the API, your except will raise an exception from inside the exception handler. A local copy of the last known value in process memory is more reliable than an external cache precisely because it has nothing that can fall over.
And the cold start. Right after a deploy the cache is empty, and the branch that fires is not "serve the old value" but "there is no data at all".
Checking this is cheap: toxiproxy in front of the dependency and three scenarios — the dependency is down, the dependency and the cache are down together, the cache is empty.
When there are many degradations at once
One degradation goes unnoticed; three at the same time are a different page altogether. If recommendations, reviews and stock levels all fall away at once, the user gets a 200 OK and an almost empty screen. Formally the service works, in fact it does not.
So the share of fallback answers is worth counting not per dependency but per request in total — and setting a threshold beyond which it is more honest to return an error than to pretend you are working.
Telling that answer apart from a real one is the most awkward of the questions you can ask about a fallback, and the most useful.
Timeouts and deadlines
The service from the first paragraph — the one that started answering in three seconds instead of thirty milliseconds — is dangerous for exactly this reason. A call without a timeout does not fail, it hangs, holding a thread or a connection from the pool. An error is visible immediately; a hung call is invisible until you run out of threads.
An HTTP call has two timeouts, and they are about different things. The connect timeout is short, 0.5–1 second: within that time either there is a TCP connection or the host is not there, and waiting longer buys nothing. The read timeout is longer, 2–5 seconds depending on circumstances — that is the time for the actual work. In the examples above these are timeout=(1, 2) in Python and the setConnectTimeout / setReadTimeout pair in Java.
With a database it gets trickier: there are three timeouts, and mixing them up is expensive. connectionTimeout in HikariCP is how long to wait for a free connection from the pool; it fires when the pool is exhausted and has nothing to do with how long the query takes. Statement.setQueryTimeout or statement_timeout on the Postgres side is how long the query itself runs. socketTimeout in the driver is how long to wait for a network answer if the server died silently. All three have to be set: without the first a thread hangs in the queue for a connection, without the second a heavy query holds a connection to the bitter end, without the third nobody notices a broken connection.
But the most common mistake is not in the numbers — it is that timeouts add up. A waits for B for three seconds, B waits for C for ten, and B keeps working long after A has left and its answer is of no use to anyone. Picking better numbers does not fix this: a chain longer than two links will always find a way to get out of sync.
What has to be passed down is not the timeout but the remaining time. The top level fixes a deadline for the whole operation and tells every call how much time is left. Service B, having received "1.2 s remaining", will not wait ten seconds for C — it sets a timeout no larger than what is left, and if nothing is left it fails immediately without wasting the call.
In gRPC this is built in: the deadline travels in the metadata and propagates along the chain by itself. Over plain HTTP you have to do it by hand:
Python, asyncio + aiohttp
import asyncio
import aiohttp
DEFAULT_BUDGET = 3.0
def budget_seconds(request) -> float:
header = request.headers.get("X-Request-Deadline-Ms")
if header is None:
return DEFAULT_BUDGET
return min(DEFAULT_BUDGET, int(header) / 1000)
async def get_json(session, url: str, budget: asyncio.Timeout) -> dict:
left = budget.when() - asyncio.get_running_loop().time()
headers = {"X-Request-Deadline-Ms": str(int(left * 1000))}
async with session.get(url, headers=headers) as resp:
resp.raise_for_status()
return await resp.json()
async def handle(request, session):
async with asyncio.timeout(budget_seconds(request)) as budget:
user = await get_json(session, USERS, budget)
# parallel calls share the same budget
cart, orders = await asyncio.gather(
get_json(session, CART, budget),
get_json(session, ORDERS, budget),
)
return {"user": user, "cart": cart, "orders": orders}
There is no need to compute what is left and compare timeouts by hand — asyncio.timeout does it for you. Nested blocks combine correctly: if the outer budget is 3 seconds and an inner call brings its own timeout of 10, the outer 3 wins — asyncio tears down the whole task tree, including the ones started through gather. A separate ClientTimeout for aiohttp is not needed either: the cancellation reaches the socket.
Exactly one thing is done by hand, the one thing asyncio cannot know about — passing the deadline over the network. Outward, the remaining time goes out as a header, and budget.when() gives the absolute end time. Inward, budget_seconds() reads somebody else's deadline and takes the minimum with its own: if the caller is only willing to wait a second, waiting three is pointless.
Java, OkHttp
import java.io.InterruptedIOException;
import java.time.Duration;
OkHttpClient client = new OkHttpClient.Builder()
.connectTimeout(Duration.ofSeconds(1))
.readTimeout(Duration.ofSeconds(3))
.writeTimeout(Duration.ofSeconds(3))
// total budget for the whole call, including DNS, redirects and OkHttp's own retries
.callTimeout(Duration.ofSeconds(5))
.build();
Request request = new Request.Builder()
.url("https://api.github.com/repos/user/repo")
.build();
try (Response response = client.newCall(request).execute()) {
if (!response.isSuccessful()) {
// handle the error code
}
String body = response.body().string();
} catch (InterruptedIOException e) {
// both SocketTimeoutException and a callTimeout firing land here
log.warn("GitHub API did not respond in time: {}", e.getMessage());
}
callTimeout matters more than the other three: without it connect, read and write add up, and the real wait turns out longer than the calling code assumes. What you have to catch is InterruptedIOException, not SocketTimeoutException — a callTimeout firing throws exactly that, and a narrow catch will miss it.
There is no equivalent of asyncio.timeout in Java: the deadline has to be tracked yourself and applied to every call separately. The closest thing in spirit is passing the remaining budget as a parameter and subtracting what has been spent, just like in the Python example, only manually.
A timeout stops the waiting, not the work
When a client gives up on a timeout, the server does not learn about it. It keeps executing the request, writing to the database, taking the money — the answer just goes nowhere. A timeout is a decision to stop waiting, not a cancellation of the operation.
Hence the consequence that breaks the most systems: a timeout does not mean the operation did not happen. It means you do not know its outcome. The same payment case as in the breaker section: after a timeout you may only retry idempotent operations — more on that next.
And the converse: if the client is gone, there is usually no point in working on. That is exactly what the deadline is passed down the chain for — so that the service below can check what is left and not start something nobody will wait for.
A timeout is set slightly above the ninety-ninth percentile of a normal day, usually two or three p99s. For a service with a typical 100 ms and a p99 of 300 ms a reasonable value is about a second. Five seconds at the same 100 ms is not headroom, it is a decision to hold a thread five times longer than it takes to conclude that no answer is coming. The average response time is no use here at all — it is the tail of the distribution that decides how many threads you lose during a degradation.
And check the multiplication. A timeout is multiplied by the number of attempts: three retries of three seconds is nine seconds in the worst case, not three. If there is an operation deadline on top, it has to account for the retries, otherwise it is not a deadline but a wish.
Idempotency: operations without a repeated effect
The previous section ended on the fact that after a timeout the outcome of an operation is unknown. There is exactly one case in which retrying it is safe: when the retry adds nothing to the first call.
PUT /users/42 {"name": "Ivan"} is idempotent — repeat it as many times as you like, the name becomes "Ivan" and stays that way. POST /orders is not: every call creates a new order. GET and DELETE are idempotent by the standard as well, but POST is the interesting one, because taking money and sending email live there.
There is one way to make POST idempotent: the client sends an operation identifier and the server remembers the result under it. An Idempotency-Key header, a message ID from a queue, a saga ID — all that matters is that it comes from outside and does not change on a retry. The first request creates the resource, the second one with the same key gets the already created one.
The most reliable place to store this is a unique index, in the same transaction as the effect itself. Then it is the database that does the rejecting: ON CONFLICT DO NOTHING, or a caught uniqueness violation, reads as "already done". Checking "is there such a key already?" with a separate query before the insert does not work, for the same reason as everything else in this section.
Python
from dataclasses import dataclass
import psycopg
INSERT_IF_ABSENT = """
INSERT INTO users (username, email)
VALUES (%(username)s, %(email)s)
ON CONFLICT (username) DO NOTHING
RETURNING id, username, email
"""
FIND_BY_USERNAME = """
SELECT id, username, email FROM users WHERE username = %(username)s
"""
@dataclass(frozen=True)
class User:
id: int
username: str
email: str
def insert_if_absent(conn: psycopg.Connection, username: str, email: str) -> User | None:
with conn.cursor() as cur:
cur.execute(INSERT_IF_ABSENT, {"username": username, "email": email})
row = cur.fetchone()
return User(*row) if row else None
def find_by_username(conn: psycopg.Connection, username: str) -> User | None:
with conn.cursor() as cur:
cur.execute(FIND_BY_USERNAME, {"username": username})
row = cur.fetchone()
return User(*row) if row else None
def create_user(conn: psycopg.Connection, username: str, email: str) -> User:
created = insert_if_absent(conn, username, email)
if created is not None:
return created
# The insert did not go through - so the user is already there
existing = find_by_username(conn, username)
if existing is None:
# The row was deleted between the two queries - rare, but possible
raise RuntimeError(f"user {username!r} vanished between insert and read")
return existing
The work here is done not by the code but by the word UNIQUE in the schema. A separate SELECT and INSERT do not survive the race: a second request fits between them, both see nothing and both insert. On a live Postgres with sixteen concurrent calls this produces duplicates more than half the time. ON CONFLICT DO NOTHING does both in a single statement, and the decision is made by the database on its own index, where there is no room to squeeze in — only, when DO NOTHING fires, RETURNING has nothing to return, so the existing row is read with a second query.
And a caveat: the function is idempotent by username, not by the pair with email — a repeat call with a different address returns the old user without updating them. That is a line in the API contract, not an implementation detail.
Between "not started" and "done" there is a third state
An operation has three states, not two: not started, in progress, finished. A "done already?" flag cannot tell the third one apart, and it breaks exactly where idempotency is needed most.
What do you return if a repeat request with the same key arrives while the first one is still running? Not "already done" — that is untrue. And not executing it a second time either. What is left is "the operation is in progress, ask later": 409 Conflict for a synchronous API, a stored status for an asynchronous one.
So what is stored under the key is a status, not a flag, and it changes according to the outcome of the call, not before it.
Java
import org.springframework.dao.DuplicateKeyException;
@Service
public class PaymentProcessor {
public PaymentResult process(String txId, Order order) {
// 1. Claim the txId. A unique index in the database is the only thing that
// really protects against the race: a local Set does not survive a restart
// and is invisible to the second instance of the service.
try {
attempts.insertStarted(txId, order.id());
} catch (DuplicateKeyException duplicate) {
// This payment has already been started. Return its outcome without redoing it.
return attempts.resultOf(txId);
}
// 2. The external call - with the same txId as the idempotency key at the provider
try {
GatewayReceipt receipt = gateway.charge(txId, order.amount());
attempts.markSuccess(txId, receipt.id());
return PaymentResult.success(receipt.id());
} catch (GatewayDeclined declined) {
// The provider said "no" - that is a final answer
attempts.markFailed(txId, declined.reason());
return PaymentResult.declined(declined.reason());
} catch (GatewayUnavailable unknown) {
// There is no answer. Whether the payment went through we do not know, and we
// have no right to either confirm or cancel it. The row stays STARTED,
// a background reconciliation with the provider will resolve it.
attempts.markUnknown(txId, unknown.getMessage());
return PaymentResult.inProgress();
}
}
}
Rows stuck in STARTED are not a leak, they are a way of storing "I don't know". The reconciliation that resolves them is something you will have to write, otherwise they pile up forever. In Python it is the same table with a status column, only with ON CONFLICT DO NOTHING instead of catching DuplicateKeyException.
Bulkhead (limiting concurrent calls, semaphores)
A timeout limits one call, not the number of them. The service from the first paragraph answered in three seconds instead of thirty milliseconds — with a five-second timeout every call does finish honestly, except that while it lasts the thread is busy. A hundred concurrent requests to a dependency like that means a hundred busy threads, and there is nobody left to serve anything else.
A bulkhead puts a bound on the number of concurrent calls rather than on their duration. The reporting service, the one that can hang for thirty seconds, is given five threads out of fifty — and it will take only those, no matter how many requests arrive. The remaining forty-five keep serving whatever still works.
In practice this is either a separate thread pool for a particular dependency, or a semaphore if the code is asynchronous. A pool also moves the call off the main thread; a semaphore simply does not let more than N tasks run at once.
Against a breaker the roles differ: the breaker decides whether to call the dependency at all, the bulkhead decides how many requests to let through at once. A breaker is useless while the dependency answers but slowly: there are no errors, the circuit is closed, and the threads run out. That is exactly the case a bulkhead covers.
The limit is computed, not guessed: requests per second times the response time. 50 rps at 200 ms means ten requests inside the dependency at any moment, and that is the working limit. Setting a hundred is pointless: the extra ninety will queue up on the other side anyway.
Python, asyncio + Semaphore
import asyncio
from contextlib import asynccontextmanager
import aiohttp
REPORTS = asyncio.Semaphore(10)
MAX_WAIT = 0.5
class ServiceBusy(Exception):
pass
@asynccontextmanager
async def bulkhead(sem: asyncio.Semaphore, max_wait: float, name: str):
try:
async with asyncio.timeout(max_wait):
await sem.acquire()
except TimeoutError:
raise ServiceBusy(name) from None
try:
yield
finally:
sem.release()
async def fetch_report(session: aiohttp.ClientSession, url: str) -> dict:
async with bulkhead(REPORTS, MAX_WAIT, "reports"):
async with session.get(url) as resp:
resp.raise_for_status()
return await resp.json()
The key line here is not the semaphore itself but the timeout around acquiring it. Waiting in the queue is no different from waiting for the answer: the request still stands there, still holds memory, still creeps toward the client's timeout. A semaphore without a bound on the wait gives you exactly half a bulkhead: concurrency stays under the limit, and the number of waiters is bounded by nothing at all.
Java
import java.time.Duration;
import java.util.concurrent.*;
// A separate pool for slow reports: 5 threads, a queue of 10.
// The queue is bounded on purpose: an unbounded one is a deferred OOM.
static final ExecutorService REPORTS = new ThreadPoolExecutor(
5, 5, 0L, TimeUnit.MILLISECONDS,
new ArrayBlockingQueue<>(10));
static CompletableFuture<Object> reportAsync(String id) {
try {
return CompletableFuture.supplyAsync(() -> fetchReport(id), REPORTS);
} catch (RejectedExecutionException busy) {
// The bulkhead is full. Under load this is a normal answer, not an outage.
throw new ServiceBusyException(Duration.ofSeconds(1));
}
}
A pool of 5 threads with a queue of 10 accepts exactly 15 tasks; the sixteenth gets a RejectedExecutionException — synchronously, out of supplyAsync, not deferred inside the CompletableFuture. On fifty concurrent requests that comes out as 15 accepted and 35 rejected in 7 ms. It is the boundedness of the queue that makes a bulkhead a bulkhead: without it excess work is not shed, it accumulates.
In Resilience4j the same effect is available through the Bulkhead module. For example:
import io.github.resilience4j.bulkhead.*;
import java.time.Duration;
import java.util.function.Supplier;
BulkheadConfig config = BulkheadConfig.custom()
.maxConcurrentCalls(10) // the default is 25, not 5
.maxWaitDuration(Duration.ZERO) // no queue: reject straight away
.build();
Bulkhead bulkhead = Bulkhead.of("reportService", config);
Supplier<Object> decorated =
Bulkhead.decorateSupplier(bulkhead, () -> fetchReport("42"));
You most likely have one bulkhead already: the database connection pool is the very same semaphore — maximumPoolSize in HikariCP, limit_per_host in aiohttp. The trouble is not the absence of a limit but the fact that there is only one: a heavy report and an order checkout take connections from the shared pool, and the first one eats it whole. A separate small pool for heavy queries is the cheapest bulkhead you can put in today.
And check the sum: four bulkheads of 50 each with a hundred threads available isolate nothing — any two of them will exhaust the resource completely.
A bulkhead rejection is a normal answer, not an outage. BulkheadFullException and RejectedExecutionException mean "no room right now", not "the dependency is broken": outward that is a 503 with Retry-After, in the logs a WARN, in the metrics a counter of its own. Growing steadily — the limit is too low or the dependency has degraded. Never growing — the bulkhead is not limiting anything, and it is worth checking with a load test.
It is also worth looking at how a bulkhead rejection is accounted for by the breaker. By default Resilience4j treats any exception as a failure, BulkheadFullException included. Limit of 10, forty concurrent requests: ten went through, thirty were turned away by the bulkhead — a 75% failure rate, and the breaker opened. The next wave does not reach the dependency at all, even though it was healthy the whole time: it was switched off by our own concurrency limit. The order of the wrappers is right, the bulkhead does belong inside the breaker — what needs fixing is the accounting:
import io.github.resilience4j.bulkhead.BulkheadFullException;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
// a bulkhead rejection is not a dependency failure, the breaker must not count it
.ignoreExceptions(BulkheadFullException.class)
.build();
Now the breaker stays closed and the next wave is served as usual.
Retrying a request that a bulkhead turned away is almost always pointless too: a rejection means "there is no capacity", not "bad luck", and capacity will not appear within 50 ms — the retry will only eat into the retry budget.
Monitoring, logging and alerts
Every pattern from the previous sections makes failure less visible. A retry hides a one-off error, a breaker hides a dependency that is down, a fallback hides missing data. That is their job, and precisely why the usual metrics stop telling the truth: the error rate drops, latency improves, uptime is green, and half of the answers are assembled from a cache.
What you end up having to measure is what ordinary monitoring does not have: the share of retries in the total number of requests and the cases where the budget ran out; breaker transitions between states and the time spent in Open; the share of answers served from a fallback — per dependency and per request in total; bulkhead rejections and queue length. None of these appears on its own: the counter is placed by hand, in the same code where the fallback is written.
Alert on a share, not on an event. One timeout at night is weather, not an incident — retries exist exactly so that this does not wake the on-call engineer. An incident is when the share of degraded answers stays above the threshold for several minutes in a row, or when a breaker fails to close for longer than the usual recovery time.
It is also worth checking that the alert fires at all. Toxiproxy in front of the dependency, the same scenarios as for the fallback, and watch not only how the code behaves but whether anything lights up on the dashboard. A metric never exercised under failure is no better than no metric.
And monitoring itself must not become a dependency. Sending logs synchronously to a remote server inside request handling means a stalled collector will slow the application down: observability takes down the thing it observes. Keep the logger asynchronous and the buffer local — losing some logs on overflow beats blocking.
Conclusion
None of these patterns removes failures. They change the shape of failure: instead of a hung request, a fast error; instead of a cascade, degradation of a single area; instead of uncertainty, an explicit "unknown" in the answer. Failure does not go away — it becomes something you can plan for in code.
The price is new decisions, every one of which can be made wrong. Retries without a budget multiply the load onto exactly the service that is down. A fallback returning zero instead of "I don't know" lies to your business logic more convincingly than an exception ever could. A breaker with a threshold measured from zero opens on a healthy service, and one with a threshold picked at random stays silent through a real degradation. There is not a single setting here whose default will fit your system.
Which leaves one thing worth checking as soon as everything is written: is the degradation visible in the metrics. No errors, excellent latency, green graphs — and that is exactly what a system answering with stubs looks like. The better your fallbacks are written, the later you will notice it, and the more likely it is that whoever notices will not be you.
Top comments (0)