DEV Community: A3E Ecosystem

Fail-Open LLM Architecture: Why Your Reviewer Stage Should Never Block a Decision

A3E Ecosystem — Wed, 15 Apr 2026 16:56:03 +0000

The outage that made me rewrite my pipeline

On December 11, 2024, OpenAI's API went fully down for roughly four hours. The culprit, per their incident report: a newly deployed telemetry service whose configuration caused every node across hundreds of Kubernetes clusters to execute resource-intensive API operations simultaneously. The control plane collapsed. Every OpenAI-dependent product collapsed with it.

Nine months later, Anthropic disclosed three separate infrastructure bugs that degraded Claude responses for weeks — at peak, 16% of Sonnet 4 requests were affected. A routing error sent short-context requests to 1M-token servers. TPU corruption caused Thai characters to appear in English responses. A compiler bug returned wrong tokens. Detection took weeks because symptoms varied across platforms and Claude often recovered from isolated mistakes.

If your production pipeline chains two or three LLM calls in series — primary decision, reviewer, formatter — it doesn't take a full outage to break you. A 16% quality degradation on one stage, unhandled, is enough to push your user-facing output below acceptable.

And yet most production LLM code I still read does this:

def decide(input):
    primary = call_primary_llm(input)       # fine
    review = call_reviewer_llm(primary)     # oh no
    if review.approved:
        return primary
    return None                              # silence

When the reviewer errors, the whole pipeline returns None. You lose both models for the price of the weaker stage's reliability.

The pattern: fail-open with a circuit breaker

Circuit breakers predate LLMs by two decades. Michael Nygard introduced the pattern in Release It! (Pragmatic Bookshelf, 2007). Martin Fowler canonised it in 2014. Netflix productionised it in Hystrix, which protected every inter-service call in their microservice fleet before entering maintenance mode in 2018. Research summarised by groundcover puts the reduction in cascading failures at 83.5% for well-instrumented breakers in production distributed systems.

The state machine is simple:

Closed (normal): requests pass through to the downstream.
Open: the breaker has seen enough failures to stop trying. Requests fail immediately with a fallback value.
Half-open: after a cooldown, one probe request goes through. If it succeeds, the breaker closes and normal traffic resumes. If it fails, the breaker stays open.

Applied to an LLM reviewer stage, "fail immediately with a fallback value" means pass the primary decision through unmodified. Not retry-until-exhausted. Not queue-for-human-review. Not None. Through.

from circuitbreaker import circuit  # or pybreaker, or hatch your own

@circuit(failure_threshold=5, recovery_timeout=60)
def _review_with_breaker(decision, context):
    return call_reviewer_llm(decision, context)

def decide(input):
    primary = call_primary_llm(input)
    try:
        review = _review_with_breaker(primary, input)
        if review.verdict == "reject":
            return downgrade_to_safe(primary, review.reasons)
        if review.verdict == "adjust":
            return apply_adjustments(primary, review)
        # "approve" falls through unchanged
    except Exception as e:
        logger.warning("reviewer unavailable, pass-through: %s", e)
    return primary

The except block is the spec, not the bug. When the breaker is open or the reviewer raises for any reason, primary is returned as-is with a warning logged. The pipeline ships what it can.

But isn't this just try/except?

It's structured try/except with two properties a naive version lacks:

Fast failure under sustained outage. Without a breaker, every call during an outage still waits for its full timeout — 30, 60, sometimes 120 seconds — before giving up. Multiply by your QPS and you have effectively DoS'd yourself. A breaker fails in microseconds once it is open. During OpenAI's four-hour incident, the difference between 30-second timeouts and 1-microsecond fast-fails was the difference between a pipeline that queued a backlog it would spend a day draining and one that kept shipping on the primary alone.
Automatic recovery. The half-open probe means you do not need a human to notice the provider recovered. Production systems that require manual re-enabling of a degraded component accumulate incident tickets faster than they close them.

What LangChain gives you for free

If you are using LangChain or LangGraph, a meaningful slice of this is already built. LangChain's .with_fallbacks() lets you chain models with automatic failover:

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic

primary = ChatOpenAI(model="gpt-4o").with_fallbacks(
    [ChatAnthropic(model="claude-sonnet-4-5")]
)

This handles provider-level failover but does not solve the pipeline-level question: when the reviewer stage itself fails, what does the pipeline return? For that you still have to wrap the stage and decide explicitly what pass-through means for your domain.

LangGraph's state-driven error handling is the more interesting primitive. You can route failed nodes to dedicated error-handling nodes, categorise errors in the graph state, and make downstream routing depend on whether critical vs. optional stages succeeded. Community production targets: tool error rate under 3%, P95 latency under 5 seconds.

When fail-open is wrong

Circuit-breaking the wrong stage is worse than no breaker at all. Two cases where you want fail-closed and must NOT pass through:

Money movement. A reviewer that detects "sending $50K to an unknown wallet" should block, not warn. But this logic belongs in a deterministic rules engine, not an LLM. If an LLM is on the critical path of a financial transaction, your architecture has a problem that no opinion about error handling can fix.
Regulated output. GDPR consent flows, medical advice generation, tax-filing assistance. These require human review on errors, not silent LLM bypass. The correct behaviour is queue and escalate, not pass-through.

For everything else — trading signals, content scoring, customer service drafts, product recommendations — the expected cost of shipping a slightly-lower-quality primary output is vastly lower than the expected cost of shipping nothing.

The metrics that actually matter

If you ship fail-open, instrument these four numbers. They are the difference between "we have a circuit breaker" and "we know it is working":

reviewer_success_rate — calls where the reviewer produced a valid response.
reviewer_adjustment_rate — of those, the fraction where the reviewer modified the primary.
reviewer_rejection_rate — the fraction where the reviewer fully overrode the primary.
reviewer_fallthrough_rate — the fraction where the breaker opened or the call errored and the primary was passed through.

fallthrough_rate is the silent killer. If it creeps above 5%, your reviewer stack is degrading quality without anyone noticing. The Anthropic postmortem is instructive: their degradation took weeks to detect because Claude often recovered from isolated mistakes and the symptoms varied across platforms. Silent degradation is always the real enemy; full outages are at least obvious.

Don't fix a high fallthrough rate by wiring a gate. Fix it by making the reviewer faster (lower temperature, smaller model for a first-pass triage, local model for the classification step before the expensive one) or more available (multi-provider fallback at the reviewer layer). Research by the AtLarge group at TU Delft on LLM service incidents shows median MTTR for the major providers ranges 0.77–1.23 hours — meaning your fail-open window is measured in hours per month, not minutes.

Closing principle

Fault-tolerant systems have one rule: every stage degrades to the simplest correct behaviour when its neighbour fails. For an LLM reviewer stage, that behaviour is pass-through with a warning. For an authorisation check, it is fail-closed with escalation. For content generation, it is queue-and-retry. Engineering for the explicit case per stage is what separates production systems from demos that fall over the first time the provider has a bad day — which, as both the Anthropic and OpenAI postmortems this year remind us, happens to everyone.

Replacing Strategy Gates With Intelligence Amplifiers: A Reviewer Stage for LLM-Driven Trading

A3E Ecosystem — Wed, 15 Apr 2026 16:40:25 +0000

Most automated trading stacks look like this:

signal -> filter_gates -> (pass or drop) -> execution

Each gate is a hard-coded rule. Volume above X. RSI below Y. Correlation below Z.
The problem is the gates have no context. A rule that says "skip signals during
low-volume hours" will drop the one asymmetric setup that only fires during
low-volume hours.

We spent the last two weeks rewriting this. The new shape looks like:

signal -> context_pack -> reviewer(LLM) -> (accept | modify | veto) -> execution

The gates are still there as a cheap pre-filter, but they no longer make the
final decision. A reviewer stage does. This post is about why we changed,
what the reviewer actually does, and how we made it safe to put an LLM on
the hot path without it becoming a single point of failure.

Why gates aren't enough

Rule-based gates are great at one thing: keeping the obviously wrong stuff
out. If your signal's 24h volume is below some floor, you don't want to
trade it. No argument.

They fall apart on the trades that matter. The trades that matter are
usually the ones that don't look like the training set. They're the 2am
breakout on a coin that just got listed on a new venue, or the unusual
order-book shape you've never seen before, or the third identical signal
from the same strategy that just stopped working last week and you
haven't noticed yet.

A gate can't tell the difference between "this is the exception the rule
was designed to handle" and "this is a new regime we haven't modeled yet".
An LLM with the right context can, at least some of the time.

What the reviewer actually sees

The reviewer is not a chat model. It's a scoped prompt that gets a context
pack and returns a structured verdict. The pack is built at signal time
and contains:

The signal itself (strategy id, direction, size hint, entry/SL/TP)
Recent trades from the same strategy (last 20, with outcomes)
A regime tag (trending, mean-reverting, chop) derived from realized vol
Top-5 correlated open positions across the portfolio
News flags for the symbol in the last 4 hours (if any)
A one-line note from the gate stage explaining why it passed

The output shape is strict JSON with three possible verdicts and a
confidence:

{
  "verdict": "accept" | "modify" | "veto",
  "confidence": 0.0,
  "rationale": "...",
  "modified_size_pct": null,
  "modified_sl": null
}

That's it. No free-form reasoning in production. The model can write
whatever rationale it wants, but the executor only reads verdict,
confidence, modified_size_pct, and modified_sl. Everything else
gets logged for review.

Why modify, not just accept/reject

The third verdict, modify, is where most of the value shows up. A
classic gate-based system is binary. A signal either gets through at
full size or doesn't get through at all. But most of the hard cases
aren't binary. They're "yes, but smaller" or "yes, but with a tighter
stop because the regime is trending against you."

The reviewer can return modify with a modified_size_pct (bounded
between 10% and 100% of the original) and a modified_sl (bounded to
be tighter, never looser, than the original). The executor clamps
these on read — we never trust the LLM to size a trade without hard
caps. The model is suggesting, not commanding.

Early data from our paper run: about 60% of signals get accept, 25%
get modify (usually smaller size during mixed-regime conditions),
and 15% get veto. The veto rate is higher than we expected. Most
vetoes come from the "three bad trades in a row on this strategy"
pattern the reviewer sees in the context pack.

Fail-open routing: the LLM cannot be a SPOF

The scariest thing about putting a model on the hot path is that
models break. Rate limits, outages, Cloudflare incidents,
token-pricing changes that silently degrade a cheap tier. If the
reviewer becomes a single point of failure, one incident kills the
whole pipeline.

We solved this with a tiered router that fails open, not fails closed:

async def review(signal, context):
    for tier in ["sonnet", "gpt4_mini", "phi4_local"]:
        try:
            result = await call_tier(tier, signal, context, timeout=8.0)
            if result.valid:
                return result
        except (TimeoutError, RateLimitError, UpstreamError):
            metrics.tier_failure.labels(tier=tier).inc()
            continue
    # All tiers down. Don't block the signal — hand it back with a
    # fallback verdict and let the old gate system make the call.
    return Verdict(
        verdict="accept",
        confidence=0.0,
        rationale="reviewer_unavailable",
        source="fallback",
    )

Two important choices here.

First, the fallback returns accept with confidence 0, not veto. If
the review layer is down, the trading system should behave exactly
like it did before we added the reviewer. Fail-closed would mean an
LLM outage = trading halt, which is worse than not having a reviewer
at all.

Second, the confidence 0 signal matters. The executor treats any
verdict with confidence below 0.3 as "use the pre-existing gate
decision only." So the reviewer's influence scales with its
confidence, and scales to zero when it's offline.

Picking a local fallback on purpose

The last tier in the cascade is phi4_local — a Phi-4 14B quantized
model running on a local GPU. It's slower than Sonnet and less
sharp, but it has three properties the hosted tiers don't:

It's free. Every signal reviewed by the local tier costs nothing.
It can't rate-limit us. Concurrent request cap is set by our hardware, not by a vendor's billing engine.
It can't be deprecated out from under us overnight.

The cascade isn't "hosted first, local as a sad fallback." It's
"use the sharpest tier available for each signal, and if nothing is
available, still have a real model in the loop." A lot of days, the
cheap hosted tier is plenty. But on the day Sonnet has an outage
and GPT-4-mini is queued three minutes deep, phi4_local keeps the
lights on.

What we measure

We don't claim this is a better trading strategy. We claim it's a
better decision stage. The questions we measure are:

Veto precision: of the trades the reviewer vetoed, what fraction actually would have lost money? (target: >60%)
Modify improvement: on trades where we took a modified size, did the expected-loss reduction justify the expected-return reduction? (target: yes, on average)
Fallback rate: what fraction of signals got the fallback path because all tiers were down? (target: <1% over a 30-day window)

We publish these numbers on an operations dashboard, not in
marketing copy. If the reviewer's veto precision drops below 50%
for two weeks running, we take it off the hot path until we
understand why. The spec isn't "always use the LLM." The spec is
"use it only when it's measurably earning its place."

The honest disclaimer

This setup does not promise better returns. We have no backtested
accuracy number to offer because we believe backtests over-fit and
we don't want to sell one. What we offer is a decision stage that
can see more of the context than a rule-based gate can, with a
fail-open path so it can't take down the rest of the system, and
a measurement plan that will retire it if the veto precision
stops paying rent.

If you're building something similar and have notes on how the
context pack should be shaped, or what tier-cascade behavior you
found works best when all hosted tiers are degraded at once, we'd
genuinely like to hear them.

Posted from the ops side of a live trading platform. No affiliate
links, no course to sell, no promise of returns. Just a design
pattern we wish we'd had two years ago.