Originally published at Trida AI.
Most production agents have exactly two states: working, and broken. While every dependency is healthy they answer, act, and write. The moment one fails (a model endpoint timing out, a search tool returning errors, a data source gone stale) the run either dies on an error screen or does something worse. It carries on at full confidence and fills the gap from memory.
Neither state earns trust. The error throws away the work the run had already done, and the user starts over. The confident continuation keeps the shape of a good answer after the evidence under it has gone. There is a third state, and it has to be designed while the system is healthy, because it cannot be improvised during an incident: a degraded mode, where the agent does less, says so plainly, and keeps doing the smaller job well.
Shrink the scope, keep the honesty
The idea predates agents by decades. Google's SRE book describes degraded responses as answers that are less accurate, or carry less data than normal, but are cheaper to produce: search only a small percentage of the candidate set rather than the entire corpus, or answer from a local copy that may not be fully up to date. The service stays up. It promises less.
For an agent, promising less rarely means a shorter answer. It means a narrower job. When the write path is down, the agent goes read-only: it can look, summarise, and prepare, and it stops offering to change anything. When fresh data is unreachable, it answers from the last good copy and puts that copy's date in the answer, where the reader can see it. When the model call itself keeps failing, the run queues the request with its context intact instead of spending the whole budget on retries. Each mode is a smaller promise the system can still keep.
The label carries the weight here. A reduced answer formatted to look like a full one misleads the reader even when every sentence in it is accurate, because the reader prices it as a full answer. So the mode belongs in two places: in the response, in ordinary words, and in the trace, as a recorded fact. Which dependency failed, when the mode started, what the agent has stopped doing. If the trace cannot answer those three, the mode exists in the prompt and nowhere else.
How the mode gets set: dependency health signals feed one controller, which sets the run's mode and writes the mode and its reason to the trace.
The fallback can be the outage
Restraint matters as much as the modes themselves. Amazon's engineers make that case in Avoiding fallback in distributed systems: fallback paths are hard to test, they can fail on their own, and they can place unpredictable load on the parts of the system already struggling. Their example is blunt. When the caches all failed around the same time, web servers fell back to querying the database directly; the database locked up under that load, and a partial problem (shipping speeds missing from product pages) became a full outage that also halted fulfilment centres.
Agent fallbacks have the same shape. Fail over to a second model and every run in the fleet arrives at that model at once, carrying longer contexts than it usually sees. Fall back from a failing search tool to the model's memory and the failure goes quiet: answers keep flowing, now ungrounded, and nobody gets paged. A fallback that changes what an answer is made of is a mode change. Label it like one.
The same article points at the escape hatch worth copying: convert fallback into failover by running both paths continuously and treating the second as an equally valid source, so it stays exercised and sized for the day it has to carry everything.
Run the quiet mode before you need it
Google's chapter on cascading failures holds the warning that applies to every rarely used path: "the code path you never use is the code path that (often) doesn't work." A mode entered twice a year gets entered with stale assumptions, a label nobody proofread, and a queue nobody monitors.
So enter it on purpose. Send a small slice of ordinary traffic through read-only mode every week and read what comes back. Put the switch in the eval set: hand the agent a tool that returns errors and check whether it narrows its offer, labels the answer, and stops attempting writes. Then check recovery, the half teams forget. When the dependency comes back, the run should return to full mode on its own, held work should be re-checked before it is written, and anything that went stale while it waited should go to a person instead of going through.
One run, while search was down: the mode change, the labelled answer, and the held work, including the update that went stale and was routed to a person. Example run, invented for illustration.
Where degrading is the wrong answer
Some actions have no honest reduced form. A payment that cannot be confirmed should not soften into a probable payment. A deletion, a send, a filing: when the safe subset is empty, the right degraded mode is a stop, with the context preserved and a person told why. Stopping is a mode too, as long as it is chosen and explained rather than stumbled into.
Modes also cost. Each one is another behaviour to test, another trace shape to read at 2 a.m., another answer format for users to learn. A system with one dependency and forgiving users may need a single extra mode, or none. What no system needs is the unplanned middle, where the agent has quietly degraded and only the model knows.
Decide the modes while the system is healthy. Write down, for each dependency, what the agent may still do when it fails, what it must stop doing, and what it tells the reader. That table is short. Skip it, and the first incident writes it for you, in production, at the worst available moment.
Work with us
If your agents fail whole when one dependency blinks, or keep answering long after they should have stopped trusting their inputs, the fix is a designed degraded mode, not a larger retry budget. Bring the dependency list through Trida AI's intake and we will map what each failure should cost you.
Top comments (0)