AI Will Be Wrong Sometimes. What Then?

#ai #java #llm #sre

Four ways this system goes wrong, and the code that catches each one

Part 11 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.

The agent asks to refund order ORD-999. There is no ORD-999. There never was.

Nothing in this codebase checks for hallucinations. The request dies anyway, in the same line of code that stops a customer reading someone else's order — a lookup that takes the authenticated session and finds nothing. An invented id and a stranger's id are the same thing to a query with a WHERE clause on the customer.

That's the pattern for every failure mode in this post. None of the mitigations ask the model to be right.

The retriever picks the wrong page

Search always returns something. Ask "who won the game last night?" and the ranker dutifully surfaces the rate-limits article, because both contain the word "the".

The honesty has to live in the score, not the ranking. Below a threshold, the best match is treated as noise and the agent declines:

Optional<KnowledgeArticle> best = retrieved.stream()
        .filter(scored -> scored.score() >= MIN_ANSWER_SCORE)
        .map(ScoredArticle::article)
        .findFirst();

"I don't have an answer for that" is a real answer. A confidently wrong one costs more than a shrug.

Somebody else's API is down

The carrier's tracking API is the one dependency I don't control, so it will be unreachable at some point. When it is, the agent must not throw, and must not improvise.

public <T> T callOrFallback(Supplier<T> call, Supplier<T> fallback) {
    if (isOpen()) {
        return fallback.get();
    }
    try {
        T result = call.get();
        consecutiveFailures = 0;
        return result;
    } catch (RuntimeException failure) {
        recordFailure();
        return fallback.get();
    }
}

Two failures open the breaker, a cooldown closes it, and the customer gets a sentence: "I can't reach the carrier right now, so I can't confirm where this order is. Nothing has changed about the delivery itself."

The detail I'd have got wrong a few years ago is keeping that separate from the denial message. "Carrier unavailable" and "not your order" are different answers, and collapsing them teaches customers that unavailable sometimes means not yours. One test asserts exactly that.

The model provider is down

Same shape, different dependency. FallbackIntentClassifier catches the provider failure and hands the message to the deterministic keyword classifier.

Degrading like this is only safe because of what sits underneath. The fallback understands fewer messages, and everything it can't understand is refused rather than guessed at. A fallback that guessed would be worse than an outage.

flowchart LR
    M["Customer message"] --> C{"Classify"}
    C -->|"provider down"| FB["Deterministic fallback"]
    C --> U{"Understood?"}
    FB --> U
    U -->|"no"| R1["Refused"]
    U -->|"yes"| L{"Scoped lookup"}
    L -->|"invented or<br/>not yours"| R2["Refused"]
    L -->|"found"| K{"Carrier call"}
    K -->|"unreachable"| F["Fallback sentence"]
    K -->|"ok"| A["Answer or proposal"]
    classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f
    classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f
    classDef bad fill:#f5ecec,stroke:#c4a29e,color:#5a4442
    classDef good fill:#ecf2ed,stroke:#93b39d,color:#3d5344
    class M,FB,F step
    class C,U,L,K decision
    class R1,R2 bad
    class A good

Four moves, over and over

Read the mitigations together and there are only four:

Validate at the boundary. An invented id fails the same lookup a stranger's id does
Threshold the confidence. A weak retrieval score becomes a declined answer
Contain the dependency. A dead API becomes a fallback sentence, never an exception
Fail closed. Anything unclassified, untiered, or unverified is refused

None of these are AI ideas. They're the circuit breakers in any microservice mesh, the range checks on any industrial sensor, the "sanity failed, hold position" branch in any control loop. The novelty in an AI system is only which component is unreliable — not what you do about it.

Every row is in docs/failure-modes.md with the test that proves it. A row with no test is a row with no mitigation, and I'd rather the table say "planned" than imply coverage I don't have.

What would change my mind

The four rows here are the failures I could think of. That's exactly the wrong sample.

The Planned half of that table is more honest about where this is fragile: approval fatigue, double execution after a retry, and a knowledge base that drifts while every test stays green. The first is a human problem, the second is an idempotency problem, and the third is invisible by construction — nothing turns red when answers quietly get worse.

Which failure mode in your system is real, known, and still described as "we should handle that at some point"?

DEV Community