As a backend developer, I have built hundreds of endpoints, so the typical endpoint flow is deeply ingrained in how I think about web applications....
For further actions, you may consider blocking this person and/or reporting abuse
This is exactly why I stopped thinking of AI calls as simple API wrappers. The model is only one part. The flow around it decides if the result is actually usable.
I really like this breakdown of the challenges we still have with even a simple AI endpoint.
I think I'm going to send this article to my stakeholders. 😄 I'm working on a system that has to be strictly deterministic because it deals with legal instruments. And yet, every now and then, someone (usually from the non-technical side) suggests, "Maybe we could add some AI somewhere?"—without a clear idea of where or why, just because it sounds cool.
The problem is that in our domain, "creative" output isn't a feature. It would be quite unfortunate if AI accidentally turned the victim into the criminal. 😅 This article does a great job of explaining why these things deserve a lot more thought than simply adding an LLM to the architecture.
Thanks, Sylwia! ♥️
Yep, AI has to be everywhere now. Let’s put it into everything! It may cost more and make the system more fragile, but at least we can proudly say that we have AI in our code. 😂
It’s like hearing the same discussions at my workplace.
Hahaha, exactly! 😂 In one of my previous projects, we didn't have a single AI feature, not because we couldn't add one, but because it simply wasn't needed and would have made the system worse. And yet, somehow we were still proudly marketed as "AI-powered." 😄
AI-powered? Take my money right now! 😂
The prompt-from-client vulnerability you hit early is structurally the same trust-boundary violation as SQL injection — user-controlled data flowing into an interpreter without sanitization. For SQL, parameterized queries give you a clean fix; for LLMs there's no equivalent abstraction yet, because system prompt, user content, and tool outputs all collapse into one token stream the model can't reliably compartmentalize. That's what makes the post-processing validation step load-bearing in a way most teams underestimate — it's not just schema enforcement, it's the only layer that can catch prompt-driven semantic drift after the model has already run and the cost is already spent.
Yes, that’s a good analogy. With LLMs, however, post-processing validation alone is not always enough. We should validate the intent before generation as well, both to keep the request within the intended use case and to reduce security risks.
So we validate the input because of intent and security concerns, and we validate the output because the model can still follow the wrong instruction or drift semantically even when the request itself looks valid.
Compared with SQL, the protection is less clean and requires multiple defensive layers.
Agree on the layering — and there's a sharper reason for it: once you add tool calls or RAG, untrusted content enters mid-context through tool output, not the user's opening message, so a pre-generation intent check on that message misses it entirely. I've seen a support bot pull a policy doc and execute instructions embedded in it, no attacker ever touched the request path. The output-side check has its own ceiling too: you're using a probabilistic judge to grade a probabilistic generator, so it converges on "plausible" rather than "correct" the way a parameterized query is definitionally correct.
That’s a good security concern to point out: untrusted content entering through tool or RAG output. Thanks for catching that!
Very informative post! 💯 I'm using AI and developing APIs at work, but I haven't worked with AI APIs yet. They seem more challenging because of the randomness of AI responses. However, thanks to your post, I think I can do it.
Thanks, I’m glad you found it informative! ❤️
Yes, AI APIs are definitely more challenging because of their probabilistic responses. And if we need trustworthy output, we have to change our mindset a little when designing these endpoints.
This is a critical shift that I think many developers are still underestimating. I've been building AgentShare — an MCP server for Solana DeFi data — and the same principle applies: the API is no longer a data retrieval endpoint, it's a reasoning endpoint. You're not asking for a number; you're asking for a judgement (SAFE/CAUTION/AVOID).
The billing model is also evolving — x402 and token-based pricing reflect this change. It's not just about 'how many requests', but 'how much reasoning is needed'. That changes everything from caching to rate limiting to security.
Exactly! I really like the way you summarized it: number vs. judgment and how many requests vs. how much reasoning. I think those two comparisons capture the shift really well. Thanks for sharing your perspective!
This maps pretty well to how I think about AI test automation. You don't assert exact output — you assert behavioral properties: was the right tool called, did the output stay within the allowed range, did it respect the business rules. Same pattern, different domain.
Yep, putting AI behind the code really changes how we should think about both writing and testing it.
Did you also discover this the hard way, like I did? 😄 At first, I treated it almost like deterministic code. Then green tests started failing randomly and my AI endpoint was suddenly being used for completely unrelated topics. That was when I started realizing how differently these systems need to be designed and tested.
Yeah, we all learned it the hard way. Kinda miss the old days — test cases used to look so different before AI. But there's no going back now. Just gotta keep moving forward. 😅
AI still feels new to me, but when I read your reply, it sounds like you’re talking about the good old days. 😅
And actually, maybe you’re right. The AI hype has only been with us for about three years, but everything has changed so quickly that we can already say, “Back in the good old days…” 🤔😂
One thing I'd add to the retry section: a retry that changes nothing just re-samples the same distribution. My version of your .HandleResult branch injects the failure into the next attempt - here's what you returned, here's the check it failed. And for a step that won't converge, I swap the model and rerun only that step. Keeping every step retryable in isolation means one bad generation never costs the whole run.
That’s actually a great point. The code in the article is only a simplified example, but this is an important insight that not everyone will realize immediately. I’m making a note and will update the article to include it. Thanks for mentioning it!
The distinction between validating the request and validating the model’s response is useful. Checking the response structure is relatively straightforward, but a response can be perfectly valid JSON and still be logically wrong. How do you handle that case in practice? Do you retry, reject uncertain results, or send them for human review? That seems like the point where an AI endpoint differs most from a conventional one.
Great question, and I don’t think there is one universal answer, it really depends on the use case and the consequences of a wrong result.
If the failure is recoverable, I would usually retry and include the validation error in the next prompt, explaining exactly why the previous response was rejected. Otherwise, the model may simply repeat the same mistake.
After a predefined number of attempts, the next step depends on the domain. The endpoint can reject the result, fall back to a deterministic alternative, or send it for human review. For high-risk decisions, human review may be necessary. For lower-risk cases, rejecting the result can be enough, but the user should receive clear guidance about what failed and what they can change without exposing internal validation rules or sensitive implementation details.
I'm nowhere near building AI endpoints — I'm a freelancer in Indonesia doing
cheap web work for small businesses, mostly using hosted models to fill content
pages. But the Ollama line caught me because I tried going local once, on a
budget PC, and the output was bad enough that I stopped.
Reading your retry section, I think I understand why now. I was treating the
bad output as a model problem. You're describing it as a flow problem —
validation after generation, then repair or fall back. The local model probably
needed more of that scaffolding, not less, which is the opposite of what I
assumed when I picked it to save money.
So the honest question from outside your world: is a small local model actually
cheaper once you count the validation layers it needs, or does the cost just
move from tokens to code you have to maintain?
Don’t give up on smaller models. they’re definitely usable. In many cases, it really is a flow problem rather than only a model problem. The backend needs to orchestrate the process and send smaller, specialized prompts instead of expecting one model call to handle everything.
I explored exactly that in this article: dev.to/gramli/how-to-use-gemma-4-e.... I ran Ollama with Gemma 4 E2B on a PC from 2015, and with the right orchestration, it worked surprisingly well.
As for the cost, I think it depends on the use case. Some of the cost may move from tokens to engineering and maintenance, but it doesn’t necessarily require a huge amount of extra code. I would start small and add only the validation, retry, or fallback steps that the endpoint actually needs. Just like with traditional APIs, not every endpoint needs the most complex possible flow.
The retry section is where I'd push back on the framing. You treat retry as "consume more tokens, hope for better" — but the failure mode that actually bites is that a naive retry on a schema failure often reproduces the exact same wrong output, because the thing that made it fail (ambiguous instruction, a field the model can't infer from context) is still there. Retrying identical input against a probabilistic system isn't a coin flip toward correct; it's a coin flip weighted by the same prompt. If the temperature is low it can be nearly deterministic in its wrongness.
What changes the odds is feeding the validation error back in — "you returned X, the schema requires Y, the address field was empty" — so the second call has strictly more information than the first. That's repair, not retry, and it's worth separating them explicitly in your flow diagram because they have completely different success curves and cost profiles. A blind retry is the one that quietly burns tokens for nothing.
One more thing on the ID-mapping trick: it's a real cost win, but watch the failure it introduces — the model can hallucinate IDs that aren't in your input set. So that output still needs validation against the source list, otherwise you've traded "wrong object" for "confidently references record 47 that doesn't exist."
Thanks for the great insights! You're absolutely right about the retry vs. repair distinction. You're actually the third person to point it out, which tells me it's an important concept that isn't obvious at first glance. I'll definitely update the article to make that distinction explicit.
And yes, the ID mapping trick also needs validation against the original input set. Otherwise, you just replace one failure mode with another.
This is exactly why I enjoy publishing technical articles, the discussion often adds details that are hard to capture from a single perspective. Thanks for taking the time to write such a thoughtful comment!
The point about idempotency breaking down is the one people underestimate. A traditional retry is safe because the output is deterministic, but retrying a probabilistic endpoint gives you a different answer and a second bill, so the whole retry-on-failure pattern needs rethinking. Have you landed on a good way to make these calls idempotent, like caching on a semantic key rather than an exact-match request hash?
That’s a great question. So far, for simpler AI endpoints, I’ve only implemented the straightforward approach: the exact same request returns the same cached output.
When it comes to semantic keys, I don’t have a good answer yet, as I haven’t built enough AI endpoints to reach that point. A semantic key would require another layer to determine whether two requests should be treated as equivalent and that decision could be approximate and produce incorrect cache hits. But that’s only my current thinking.
It’s definitely an interesting topic!
the 'passing a prompt directly from the client' lesson is the one everyone hits at the same time. we caught it when a user started crafting prompts to make our LLM write content for completely unrelated tasks.
the fix we added beyond sanitization: an intent classifier that runs before the main LLM call. short prompt, cheap model, just yes or no does this request fit the intended use case. rejects are 99% cheaper than full generation.
are you doing prompt classification upstream or relying on structured output schema violations to catch misuse?
Yep, classic. Lot of us learned that lesson the hard way. Fortunately for me, it happened in a side project.
To be honest, I haven’t needed an intent classifier so far because I’ve been using small, specialized prompts. An additional classification call could make the cost roughly 50:50 while also adding more code to maintain. For that reason, output validation has been sufficient for my use cases. But for broader prompts or more general-purpose assistants, validating the intent before generation definitely makes sense. Thanks for mentioning it.
Really enjoyed reading this! AI endpoints are definitely changing the way we think about APIs. It’s not just about sending a request and getting a fixed response anymore — there’s a whole layer of context, validation, and handling uncertainty.
I like how this highlights that building with AI is not only about choosing the right model, but also about creating a reliable system around it. The future of APIs is getting pretty interesting, and there’s a lot for developers to rethink.
Thanks for sharing this Daniel ! ❤️ :D
Thanks for the comment! I’m glad you found it interesting. 😊
Yes, we need to shift our mindset when designing traditional REST endpoints and AI-powered ones and it’s definitely interesting to see how using AI models in our applications changes the way we think about architecture.