Twenty-four articles. One agent. A lot of things I got wrong before I got them right.
This one isn't about code. It's about what I'd tell myself at article 1 if I could go back.
I thought the model was the hard part
It isn't. Spring AI handles the model. You call chatClient.prompt().user(message).call().content() and it works. Five lines.
The hard part is everything around that call. How do you pass context without spending a fortune on tokens? What happens when a tool fails mid-conversation? How do you know if the response is actually correct? How do you ship without putting your API key in git?
Those questions have nothing to do with AI. They're the same questions you'd ask about any external service you're integrating with. I wasted the first few articles thinking I needed to understand the model better. I should have been thinking about the backend.
I should have built the eval dataset first
This is the biggest regret.
I built the eval dataset in article 23. Twenty-two articles after I made my first API call. Before that, every time I changed a prompt or adjusted a tool, I was testing by hand — send a message, read the response, guess whether it was better or worse.
That's not testing. That's hoping.
The eval dataset forces you to define what "correct" means before you build anything. Once you have it, changing the system prompt goes from "seems okay?" to "pass rate held at 87%." The difference is enormous. I'd build five test cases in article 1 if I started again — even dumb ones. Anything is better than nothing.
The tool design mistake I kept making
Early tools returned verbose JSON. Something like:
{
"city": "Mysore",
"weather": "partly cloudy",
"temperature": "24°C",
"humidity": "68%",
"lastUpdated": "2026-09-27T10:30:00"
}
That structure replays in every subsequent message in the conversation. The model doesn't need it. It can read "Mysore: 24°C, partly cloudy, 68% humidity" just as well — and that's 60% fewer tokens in every follow-up turn.
I changed this late. By the time I got to the cost optimisation article, I'd been paying for bloated tool returns for weeks. Small thing, big compounding effect.
The other tool mistake: throwing exceptions. A tool that throws NoRouteException crashes the agent and surfaces a 500 to the user. A tool that returns "No direct route found between Mumbai and Mysore" lets the model acknowledge the gap and keep going. I had to learn this the hard way — an exception in getTicketPrice that took down the whole conversation before I caught it.
Log tokens from day one
I added token logging in article 20. I should have added it in article 2.
Every request logs inputTokens and outputTokens. Without that data, you're guessing where the cost is going. When I finally queried the averages after a few days of traffic, the breakdown was immediately obvious: history was 45% of every request, the system prompt was another 25%. Together they were 70% of input tokens — and both were fixable with no change to output quality.
But I didn't know that until I measured it. If I'd been logging from the start, I would have caught it much earlier and built with more cost awareness from the beginning.
Memory architecture matters more than model choice
Swapping models changes the quality of responses. Getting memory wrong breaks the agent.
A conversation window that's too big makes every request expensive. A window that's too small makes the agent forget context three turns in. Finding the right size — 10 messages in this case — required thinking carefully about actual user behaviour, not just picking a number.
I spent more time tuning the model and its prompts than I spent thinking about memory. In hindsight, that's backwards. RedisChatMemory with a sensible window size was the most impactful architectural decision in the whole project. The model barely matters if the context management is wrong.
What I'd do differently
Build the eval dataset first. Log tokens from the first API call. Use TravelToolsPort as an interface from article 3, not article 23 — testability from the start, not retrofitted later. Think about MCP earlier — not because local @Tool methods are wrong, but because understanding the coupling between the agent and its tools changes how you design tool boundaries.
Everything else I'd keep. The bottom-up approach — one concept per article, each building on the last — was right. Spring AI's abstractions are at the right level. Starting small was right.
The short version
AI backend engineering is backend engineering. The model is the new external service — you call it, handle its failures, measure its cost, and test against it. Everything else is the same: interfaces, observability, evaluation, operational readiness.
Start small. Measure everything. Build testability in from the start. The AI-specific parts follow the same principles as the rest.
Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.
Top comments (0)