Your AI Feature Should Not Break Your Endpoint

Welcome back. This is the third article about my Purchase Decision API, after error handling and validation. This week the API got an AI feature, and this piece is about the design decision that mattered most.

Everyone is adding LLMs to their APIs right now. Almost nobody talks about what happens when the LLM is down.

I just built a purchase decision API. You tell it what you want to buy, and it tells you BUY, WAIT, or SKIP, with a score out of 100 and a savings plan. On top of that sits an AI-written explanation in friendly, human language.

The most important design decision I made was this: the AI cannot touch the verdict.

The obvious design, and why I did not use it

The tempting architecture is simple. Send the purchase details to an LLM and let it decide. One API call, no scoring logic to write, and the response already comes back in natural language.

I did not do it, for three reasons.

First, LLMs are not deterministic. The same purchase could get different verdicts on different days, and a decision API that changes its mind for no reason is not a decision API.

Second, it couples my uptime to someone else's. If the model provider has an outage, my endpoint has an outage.

Third, and this one settled it: a hallucinated verdict about someone's money is worse than no verdict at all.

A floor and a ceiling

So I split the response into two layers:

CEILING   LLM explanation          (nice to have, can fail)
FLOOR     score, verdict, plan     (pure Java, never fails)

The floor is everything the endpoint guarantees. My scoring engine is deterministic: it takes disposable income, price ratio, purchase type, and usage frequency, and produces a 0 to 100 score. Plain Java, no AI anywhere in that path. Same inputs, same score, every time.

The ceiling is everything that makes the response nicer but is allowed to fail. The AI explanation lives here, and only here.

Tell the model the answer

By the time the LLM is called, the verdict already exists. The model is not asked to decide anything. It is told the answer and asked only to explain it:

String prompt =
    "A user wants to buy '" + itemName + "' for ₹" + price + ". " +
    "Their monthly disposable income is ₹" + disposableIncome + ". " +
    "The affordability verdict is " + verdict + " with a score of " + score + "/100. " +
    "Explain this verdict in 2-3 friendly, non-judgmental sentences. " +
    "Do not repeat the numbers back mechanically — give practical, warm advice.";

That last line matters more than it looks. Without it, the model just recites your inputs back as a sentence, and you have spent an API call turning numbers into slightly longer numbers.

Notice what this structure buys you. Even if the model hallucinates, the worst it can produce is a badly worded explanation of a correct verdict. The blast radius of a bad AI response is prose, not money.

The fallback

The layers meet a second time at failure handling. The LLM call is wrapped so that any failure, whether a rate limit, a network error, or a malformed response, degrades to a template built from the same numbers:

try {
    return generateExplanation(prompt);
} catch (Exception e) {
    e.printStackTrace();   // more on this line in a second
    return buildFallbackExplanation(itemName, price, verdict, score, disposableIncome);
}

Here are the two outputs side by side, from the same request.

The AI version:

"It looks like investing in those Sony headphones might not be the best choice right now, considering your monthly income and expenditures..."

The fallback version:

"Based on your finances, buying Sony headphones for ₹30000 scored 45/100, giving a verdict of WAIT. This is measured against your monthly disposable income of ₹27000."

Worse prose. Identical information. The user loses warmth, never the answer. That is what graceful degradation actually means: the experience gets worse, the contract does not.

And yes, catch (Exception e) is normally a code smell. Here it is the point. This catch sits at the boundary of the optional layer, and there is no exception the AI layer can throw that should reach the user.

The bug that made me respect logging

My first version of this failed in a way I want to be honest about. Every response came back with the mechanical explanation, and I had no idea why. No errors, no crashes, nothing in the response to suggest a problem.

The fallback was working too well. It was hiding its own trigger.

Once I added logging, the cause turned out to be one missing character. I had written "Bearer" + apiKey. No space after Bearer. The auth header was malformed, OpenAI rejected every single call, and the fallback fired every time, exactly as designed.

That is the uncomfortable property of graceful degradation: a fallback that logs nothing is undebuggable. The system looks fine from the outside while quietly running in degraded mode forever. The rule I took away is short: log the failure, degrade anyway. Users should never see the AI layer fail, but you always should.

Honest notes

A few things this design does not solve.

The latency is still there when the AI is up. A synchronous LLM call adds real time to every response. For a low-traffic project that trade-off is acceptable; a high-traffic API would want the explanation generated asynchronously or cached, and that is not built here yet.

The broad catch is only right at this one boundary. Inside the scoring engine I still want real exceptions with real handling, the kind I wrote about in the error handling article. Catching everything is a deliberate choice for the optional layer, not a general style.

The fallback text is genuinely worse. Users notice the difference between warm advice and a template. An open question I have not answered yet: should the response admit degraded mode explicitly, with a field the client can read, or stay quiet about it?

I am still learning this area. This design is simply what survived contact with my own bugs.

Recap

The endpoint worked before the AI feature existed, and it has to keep working when the AI fails. So the response is split in two: a deterministic floor in plain Java that always returns the score, the verdict, and the plan, and an optional AI ceiling that explains the verdict when the model is reachable and falls back to a template when it is not. The model is told the answer instead of being asked for one, so a hallucination can only ever ruin the wording. And the fallback logs every failure, because a silent fallback once hid a one-character bug from me for days.

Let the AI reduce what your user has to read. Never let it decide what your system guarantees.

If you were building this, which would you choose: a response that openly admits degraded mode with an explicit field, or a fallback the user never knows about? I would like to hear arguments for either side.

P.S. If this was useful, subscribe. I write one piece like this every week.