Fail-closed or fail-open? Decide per side effect, then check what your cache already decided

Fail-closed: when something a decision depends on is unavailable, the answer is no.

Fail-open: the answer is yes.

Fail-bounded: the answer is yes, inside a limit that does not need the missing dependency.

"Fail-closed or fail-open" is the wrong question at the scale of a service. Ask it per dependency and per kind of request. And before you answer, check your cache client: ours was configured to swallow errors, which turns every security decision built on it into fail-open without anyone deciding so.

Why two words are not enough

The third definition has no standard name, and it is the one we use most. OWASP's principle is called fail securely, and it is usually read as "always closed". For a service that sits between customer apps and third-party APIs, always closed means that a cache outage takes every app's weather widget down with it. Always open means that the same outage removes the only brake on a connector that can charge a card.

Both camps are right about their half of the traffic. So the unit of decision cannot be the service.

The table

The proxy behind this table is one we are trialling at GoodBarber. It keeps third-party API keys on the server for widgets that run inside customer apps, and today it serves a few test tenants. Its security specification settles the question in one decision, written before the code: the degraded mode depends on the side effect. A connector is registered either as read-only or as side-effecting, next to its host and its allowed methods, and every dependency gets a row. Everything below is conditional on that flag being right. The proxy applies the column it is given. It does not infer the class from the traffic, so a connector wrongly registered as read-only gets the lenient column.

When this is unavailable	Read-only connector	Side-effecting connector
The keys that verify token signatures	refused, 503	refused, 503
The rate limiter's shared store	allowed, inside a per-process ceiling	refused, 503
The kill switch	cannot be unavailable on its own: it lives in the primary database and is read on every request	same

The first row has no nuance. A signature you cannot check is not a valid signature, whatever the request does. The status code matters as much as the refusal: 503, never 401. A 401 tells the client its token is bad, and a well-behaved client responds by asking for a new one, which turns an outage of the key endpoint into a stampede on the token endpoint.

The third row is a placement decision. A kill switch that lives in a cache disappears exactly when the infrastructure is having a bad day, which is when you reach for it. Ours is a column in the database the request already needs.

The option in the middle

The second row did not start that way. In the first version, read-only traffic simply passed when the limiter's store was down. An adversarial review, run before the first commit of that code landed, flagged it, and the reason fits in one sentence. "Read-only" describes the effect on the third party, not the cost to us or to the customer's API quota. Fail-open on reads is still an unmetered pipe.

The fix is a few lines, and it is the whole idea of fail-bounded:

try:
    ok = hit(tenant) and hit(tenant, connector) and hit(tenant, host)   # shared store
except StoreUnavailable:
    if side_effecting:
        return FAIL_CLOSED                       # 503: no brake, no payment
    return ALLOW if local_window_allows(tenant, connector) else THROTTLED
return ALLOW if ok else THROTTLED

local_window_allows is a sliding one-second window kept in the worker's memory. It is cruder than the shared limiter and it needs nothing that can be down. It is also per process, so the real ceiling is that number multiplied by the number of workers. Reads keep flowing, and "how much" still has an answer.

The setting that had already decided

This is what I would check first in any codebase. Our Redis cache is configured with django-redis's IGNORE_EXCEPTIONS, which its documentation describes as memcached-like behaviour: when the server is unreachable, a read returns None and a write does nothing. For a cache, that is the right call. A cache that is down should look like a cache that is empty.

A rate limiter built on that client inherits the setting. incr does not raise. It returns None. A naive comparison then either crashes or, written a little differently, treats "no answer" as "under the limit". Nobody chose fail-open. A convenience flag, set once for caching, chose it for every security decision stacked on the same client.

A shared client's error policy is a security decision for everything built on top of it. So the limiter treats silence as a failure, explicitly:

current = cache.incr(key)
if current is None:              # the client swallowed a backend error
    raise StoreUnavailable       # silence is a failure, not "under the limit"

One of the six tests of that module exists only for this case, and its name says so: none_from_ignore_exceptions_is_treated_as_unavailable.

Here is the whole chain during an outage. Redis stops answering. incr raises nothing and returns None. The limiter calls that unavailable, and the table decides: 503 for a connector that can charge a card, a local ceiling for one that reads the weather.