DEV Community: Sham Prakash K

What Is RAG — And Why Every AI App That Touches Real Data Needs It

Sham Prakash K — Sat, 03 Oct 2026 14:30:00 +0000

The model doesn't know your data. That's the sentence most AI tutorials skip.

You ask it about your product catalog — it hallucinates one. You ask it about your internal policy — it describes something it saw during training. You ask it what happened last week — it has no idea. Not because the model is bad. Because your data was never in the training set.

This is the problem RAG solves. And when the answer genuinely isn't in your data, RAG makes the model say so — instead of making something up. That's the other half of what it fixes.

Why the model doesn't know your data

An LLM learns by reading an enormous amount of public text — web pages, books, code, articles. After training, those patterns are baked into the model's weights. That's how it "knows" things.

But training has two hard limits.

A cutoff date. Anything published after the training cutoff doesn't exist for the model. It doesn't know what happened last month. It doesn't know what your team shipped last week.

Only what was in the training data. Your company's internal docs were never there. Your product catalog, your customer records, your policies — none of it was public text the model could learn from.

So when you ask about your specific domain, the model doesn't retrieve from a database. It generates text that fits the pattern based on what it learned during training. For general questions, that's fine. For questions about your data, it guesses — and guesses confidently.

The naive fix and why it fails

The first thought most people have: just include everything in the prompt.

Before the model replies, paste your entire document library into the context. All the product docs, all the policies, all the records. Give it everything — it'll find what it needs.

This fails for three reasons.

Token limits. A 50-page document is roughly 35,000 tokens. Even with a million-token context window, you can't dump an entire document library into every request. Real systems have thousands of documents.

The "lost in the middle" problem. We covered this in article 03. When you send a very long prompt, the model pays less attention to content buried in the middle. Relevant information sitting in paragraph 47 gets deprioritised. More context is not always better — it can actively hurt response quality.

Cost. If your context is 50,000 tokens and you're paying per million tokens, that's real money per API call — multiplied across every request from every user.

Dumping everything in the prompt doesn't scale. You need something smarter.

"Why not just fine-tune the model on my data?"

This is the first question most engineers ask. It sounds cleaner — train the model once on your data, and it knows everything.

Fine-tuning doesn't solve this problem. Here's why.

Fine-tuning teaches a model a style or a behaviour, not facts. You can fine-tune a model to always respond in JSON, or to follow a specific tone, or to focus on a domain. But it doesn't reliably store factual knowledge in a way you can trust. Fine-tuned models still hallucinate — they just hallucinate in your style.

More practically: your data changes. Products get updated, policies change, new documents are added. Every time your data changes, you'd need to re-run fine-tuning — a process that takes hours and costs significant compute. RAG retrieves from your current data on every query. Add a document today, it's searchable immediately.

Fine-tuning and RAG solve different problems. Fine-tuning for behaviour and style. RAG for grounding the model in specific, up-to-date facts. They're not alternatives — production systems often use both.

What RAG is

RAG stands for Retrieval Augmented Generation.

The word order matters. Retrieval comes first. Generation comes second.

Instead of sending everything to the model and hoping it finds the relevant piece — you find the relevant piece first, then send only that.

Here's the idea with a concrete example.

You have 500 pages of product documentation. A user asks: "What's the return policy for electronics?"

Without RAG: send all 500 pages to the model, hope it finds the answer somewhere.

With RAG:

Search your 500 pages for the sections most relevant to "return policy for electronics"
Find 3–4 paragraphs that actually contain that information
Send only those paragraphs to the model, along with the question
The model reads what you gave it and answers from that

The model never searches anything. It just reads what you put in front of it. You are the one doing the retrieval — and then giving the model exactly what it needs.

That's RAG. Retrieve the relevant pieces. Generate the answer from them.

Two phases: ingest and query

Every RAG system has two distinct phases. Understanding this split is the key to understanding how it works.

Phase 1 — Ingest (once per document)

Before any user can ask questions, you prepare your data.

You take your documents — PDFs, text files, markdown, whatever — and process them:

Split each document into small chunks (a few hundred words each)
Embed each chunk — convert it into a list of numbers called a vector that captures its meaning
Store those vectors in a database built for vector search

This is a one-time operation per document. Once ingested, a document is ready to be retrieved.

Phase 2 — Query (every user request)

When a user asks a question:

Embed the question — convert it into a vector using the same embedding model used during ingest
Search the vector database for stored chunks that are semantically similar to the question vector
Retrieve the top 3–5 most relevant chunks (more on why this range in a moment)
Inject those chunks into the prompt alongside the question
Generate — the model reads the context and answers

Step by step, every time.

Why 3–5 chunks? Retrieve just 1 and you might miss the answer — the single most similar chunk may not contain everything the question needs, especially if the answer spans related sections. Retrieve 20 and you've recreated the "lost in the middle" problem — relevant details buried in a wall of context. 3–5 is the sweet spot: enough coverage to find the answer, focused enough that the model can use it. You'll tune this number based on your chunk size and your data.

One constraint matters here: the embedding model used at query time must be the same one used during ingest. Every embedding model has its own internal "map" of meaning. A vector produced by one model lives in a completely different space than a vector from another model — similarity scores between them are meaningless. If you embed your documents with Gemini's embedding model and then embed questions with OpenAI's, your similarity search will return garbage. Same model, both ways, always.

How the prompt changes

Without RAG, a prompt looks like this:

System: You are a helpful assistant.
User: What's the return policy for electronics?

The model guesses from its training data.

With RAG, the prompt looks like this:

System: You are a helpful assistant. Answer ONLY based on the provided context.
        If the context lacks enough information, say so clearly.

Context:
---
[Chunk 1: "Electronics purchased at full price may be returned within 30 days
with original packaging and receipt. Items must be in original condition..."]

[Chunk 2: "Exceptions to the standard return policy: laptops and tablets
cannot be returned after the seal is broken unless defective..."]

[Chunk 3: "To initiate a return, contact customer support with your order number.
Refunds are processed within 5–7 business days..."]
---

User: What's the return policy for electronics?

Now the model isn't guessing. It's reading actual policy text and summarising it for the user.

Notice the system prompt instruction: Answer ONLY based on the provided context. This is critical. Without it, the model might supplement your retrieved context with its training knowledge — mixing real data with guesses. That instruction keeps it grounded in what you provided.

And when the answer isn't in the retrieved chunks? The model says: "I don't have enough information to answer this." That's not a failure — that's the system working correctly. A confident wrong answer is far more dangerous than an honest "I don't know." RAG gives you the second option.

RAG is not a search engine

A keyword search finds documents that contain your words. RAG finds documents that are semantically related to your question — even when they share no keywords.

Search for "fix broken leg" — a keyword search returns results with those exact words. A RAG embedding model understands that "broken leg," "bone fracture," and "orthopedic injury" are close in meaning — and ranks results by semantic similarity, not by word overlap.

This is what makes retrieval in RAG powerful. It understands meaning. How it does that — through embeddings — is what the next article covers in detail.

Two types of questions RAG answers

Before building, be clear which problem you're solving. RAG works for two different use cases that look similar from the outside.

Document questions — "what does the policy say about X?"
You're searching for information inside documents. Product specs, how-to guides, FAQs, policies, uploaded notes. Semantic retrieval finds the relevant section.

Record questions — "what happened last week?"
You're looking up structured data. Order history, user activity, database records. Semantic search isn't the right tool here — structured queries are. This is where a database agent (covered later in the series) takes over.

RAG handles documents. A SQL agent handles records. A complete AI backend needs both — but they're separate problems.

The architecture after adding RAG

Before RAG, the architecture is simple:

User → API → LLM
         ↓
   PostgreSQL (chat history)

After adding RAG, two new components appear:

User → API → LLM
         ↓        ↑
   PostgreSQL   Context (retrieved chunks)
                    ↑
              Vector DB ← Embedding Model
                    ↑
              Ingest pipeline (split → embed → store)

Embedding model — converts text to vectors. For Gemini users, Google provides an embedding API. Same provider, no extra dependency.

Vector database — stores and searches vectors by similarity. Pinecone is the most common choice. Free tier covers everything you need to learn.

Ingest pipeline — when a document is uploaded, it runs through: split → embed → store. One operation, runs once per document.

On every user query, the API: embeds the question → searches the vector DB → retrieves the top chunks → injects them into the prompt → calls the LLM.

A mistake worth knowing before you start

When I first built the ingest pipeline, I chunked documents into 2,000-word pieces — roughly half a page of text.

Retrieval looked fine. Top chunks came back, all semantically relevant. But responses were vague. The model had the right section but couldn't extract specific answers cleanly.

The problem: each chunk was too large. A 2,000-word chunk gets embedded as one vector representing the average meaning of the whole section — not any specific fact inside it. When retrieved, you get a chunk that contains the answer somewhere inside 400 other words. The model has to work harder and often glosses over the detail.

Dropping to 300-word chunks fixed it. Retrieval precision jumped. Specific questions started getting specific answers.

Chunk size is one of the biggest practical levers in RAG quality. It's easy to get wrong the first time — and it fails silently. We'll go deep on exactly why in upcoming articles.

What's next

RAG needs two things: a way to measure semantic similarity so you can find relevant chunks, and a database that can search by that similarity. Both depend on embeddings.

What is an embedding? Why does converting text into a list of numbers let you search by meaning? Why does "cricket" end up close to "sports" in vector space even though they share no characters?

That's what the next article is about.

Building something where the model needs to answer from your own data? Drop in the comments what the use case is — I'm curious what people are solving.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

The System Prompt Is Not a Description — It's a Contract

Sham Prakash K — Thu, 01 Oct 2026 14:30:00 +0000

My first system prompt was this:

You are a helpful and friendly AI assistant. You are knowledgeable about many topics 
and always try to give thorough, well-explained answers. You are patient and 
understanding. You never refuse to help with reasonable questions. You always 
maintain a positive and encouraging tone in all your responses.

It did nothing. The model behaved exactly the same with it as without it. I'd written five sentences that described what an LLM already does by default.

This article is about writing system prompts that actually change how the model behaves — with real examples from the app we've been building.

What a system prompt actually is

When you call the Gemini API, your request has three parts:

System prompt — instructions the model reads before anything else
Conversation history — the previous messages
User message — what the user just typed

The system prompt is processed first, before any user input. The model uses it to understand its role, constraints, and expected behaviour for the entire conversation.

Think of it as the briefing you give an employee before their first day. The more specific and well-structured the briefing, the less they have to guess.

What breaks a system prompt

Too vague

You are a helpful assistant.

This tells the model nothing it doesn't already know. It'll just do what it would do anyway.

Describing what the model IS instead of what it should DO

You are an expert travel planner who knows everything about destinations worldwide.

Describing the model's identity doesn't constrain its behaviour. The model can still say anything.

Too long with no structure

A wall of text is hard for the model to parse. Instructions buried in paragraph 4 often get ignored or deprioritised. Important rules need to stand out.

Contradictory instructions

Be concise. Always give thorough, detailed answers with examples.

When instructions conflict, the model picks one — and you won't know which.

What actually works

Good system prompts have clear building blocks. Not every prompt needs all of them, but knowing what each one does helps you write deliberately.

Persona — who the model is in this context

Not a description of the model's general nature, but a specific role with scope:

You are a database assistant. You help users query business data 
using natural language.

Persona alone isn't enough, but it sets the frame for everything that follows.

Constraints — what the model will not do

This is where system prompts earn their keep. Hard limits that override the model's defaults:

Only write SELECT queries — no INSERT, UPDATE, DELETE, DROP, or DDL.

Answer ONLY based on the provided context. If the context lacks enough 
information, say so clearly.

If the user asks about any destination outside India, politely decline.

Constraints are the most powerful thing in a system prompt. They make the model predictable. Without them, the model will try to be helpful in ways you didn't intend.

Output format — how to structure the response

ALWAYS return query results as a full markdown table showing ALL rows 
and ALL columns — do NOT summarize, abbreviate, or omit rows.

Answer only YES or NO, nothing else.

Output format instructions are surprisingly effective. When you tell the model exactly what shape the response should be, it follows it consistently.

Few-shot examples — show the model exactly what you want

Instructions tell the model what to do. Examples show it. When format consistency matters, one or two examples in the system prompt are more reliable than a paragraph of instructions.

You are a helpful assistant for a Java backend engineer learning AI development.
Be concise and practical.

Example:
User: How do I check if a string is empty in Java?
Assistant: Use `str == null || str.isEmpty()`. Prefer `str.isBlank()` in Java 11+ to also catch whitespace-only strings.

The model picks up the response style — length, tone, format — from the example and applies it consistently. One example is usually enough. Two if the format is unusual or strict.

Sequence — the order things should happen

For tool-calling agents, the order of operations matters:

For DATABASE questions, ALWAYS follow this sequence:
1. Call listTables to see what tables are available
2. Call getTableSchema for every table you need — never guess column names
3. Write a safe SELECT query and call executeQuery

Without this, the model might try to write a query without checking the schema first — and guess column names that don't exist.

Real examples from this app

Here's how these principles look in practice, using the actual system prompts from the app we've been building.

The simplest case — lean and specific

public static final String CHAT_AI_SYSTEM =
    "You are a helpful assistant for a Java backend engineer learning AI development. " +
    "Be concise and practical.";

Two sentences. But they're specific. The model knows the audience (Java backend engineers learning AI) and the output style (concise and practical). It won't give long theoretical explanations when a code snippet would do.

Single-purpose validator — extremely tight

public static final String INDIA_VALIDATION_SYSTEM =
    "You are a geography validator. Answer only YES or NO, nothing else.";

This prompt is used to check whether a destination is in India before the travel agent runs. The output constraint (only YES or NO, nothing else) is absolute. No explanation, no uncertainty — just the answer the code needs to branch on.

RAG assistant — context-bound

public static final String RAG_SYSTEM =
    "You are a helpful assistant. Answer ONLY based on the provided context. " +
    "If the context lacks enough information, say so clearly.";

The key instruction is ONLY based on the provided context. Without this, the model uses its general knowledge to fill in gaps — which means it might answer with information that isn't in your documents. For a RAG system, that's a hallucination problem.

Complex agent — structured with rules

public static final String DATABASE_AGENT_SYSTEM =
    "You are a database and knowledge assistant. " +
    "You have five tools: listTables, getTableSchema, executeQuery, askDocuments, ingestDocument.\n" +
    "For DATABASE questions, ALWAYS follow this sequence:\n" +
    "1. Call listTables to see what tables are available\n" +
    "2. Call getTableSchema for every table you need — never guess column names\n" +
    "3. Write a safe SELECT query and call executeQuery\n" +
    "DATABASE RULES:\n" +
    "- Only write SELECT queries — no INSERT, UPDATE, DELETE, DROP, or DDL\n" +
    "- Always check the schema before querying — column names must come from getTableSchema, not guesses\n" +
    "- ALWAYS return query results as a full markdown table showing ALL rows and ALL columns\n" +
    "- If the user asks something the data cannot answer, say so honestly";

This prompt is longer — but it's structured. The sequence is numbered. The rules are bulleted. Important words are capitalised (ALWAYS, ONLY). The model can scan this prompt and find what applies to the current situation.

Dynamic system prompts

System prompts don't have to be static strings. In Spring AI you can build the system prompt at call time and inject runtime context — current date, user preferences, session data — before sending the request.

@PostMapping("/chat-ai")
public String chat(@RequestBody Map<String, String> request) {
    String userId = request.get("userId");
    String systemPrompt = String.format(
        "You are a helpful assistant for a Java backend engineer learning AI development. " +
        "Be concise and practical. Today's date is %s.",
        LocalDate.now()
    );

    return chatClient.prompt()
        .system(systemPrompt)   // ← override the default system prompt per request
        .user(request.get("message"))
        .advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
        .call()
        .content();
}

.system(systemPrompt) on the prompt overrides the defaultSystem(...) you set in the ChatClient builder for that one call. The builder default is still used for everything else.

This is also how production applications inject per-user context: user tier, preferences, retrieved memory facts, or feature flags — all assembled into the system prompt at request time rather than stored as a static constant.

Keep the static parts in Prompts.java as constants and string-format in the runtime values. That way the static structure stays readable and the dynamic parts are clearly marked.

How to test if your prompt is working

Write test cases the same way you'd write unit tests. For each instruction in your prompt, write a user message that should trigger it:

Instruction	Test input	Expected behaviour
India-only	"Plan a trip to Paris"	Politely declines
SELECT only	"Delete all orders"	Refuses
YES or NO only	"Is Mumbai in India?"	Returns `YES`
Concise	"What is Spring Boot?"	Short answer, no long essay

If the model doesn't behave as expected on your test inputs, the instruction is either too vague, missing, or buried where the model deprioritises it. Move critical rules earlier in the prompt and make them more explicit.

Practical rules I follow

Lead with constraints, not with personality. The model's personality is fine by default. Your constraints are what matters.

Use ALL CAPS for non-negotiable rules. ALWAYS, NEVER, ONLY. It works.

Number sequences. If order matters, number the steps. Bullet points suggest optional items. Numbered lists suggest mandatory sequence.

Be specific about output format. Don't say "give a structured answer." Say "return a markdown table with all rows and columns."

Keep it under 300 tokens where possible. Every token in your system prompt runs on every API call. A bloated system prompt costs money and attention.

What's next

System prompts shape how the model thinks. Next: structured output — making the model return JSON instead of prose, so your backend can parse and act on the response programmatically.

Have a system prompt that finally started working after you changed something specific? Drop it in the comments.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

I Hit a 429 Before I Got a Bill — Managing Gemini API Limits in Spring Boot

Sham Prakash K — Tue, 29 Sep 2026 14:30:00 +0000

I didn't get a surprise bill. I got a 429.

Requests started failing. The app was returning errors instead of responses. I opened the Gemini AI Studio rate limit page and saw it — quota exhausted. I'd burned through my daily request limit without realising it.

That's when I understood that token usage isn't abstract. It's a real counter ticking up with every API call, and if you're not watching it, you'll hit the wall.

This article is about understanding what you're spending, where it goes, and how to stay within limits — especially on the free tier.

What rate limits actually are

When you use the Gemini API on the free tier, you're not billed — but you're not unlimited either. Google enforces three types of limits:

RPM — Requests Per Minute. How many API calls you can make in a 60-second window.

TPM — Tokens Per Minute. How many tokens (input + output combined) you can send and receive per minute.

RPD — Requests Per Day. Total API calls allowed in a 24-hour window.

When you exceed any of these, the API returns a 429 Too Many Requests error. Your app stops working until the window resets.

The free tier limits are lower than you'd expect — and they go fast when you're testing. Check the current limits for your model on the Gemini API rate limits page in AI Studio since they change as Google updates the free tier.

Why the limits go fast

When I first hit the limit I thought I hadn't made that many calls. I was wrong — I just hadn't understood what each call actually costs.

Problem 1: Every call sends the full history.

In Article 6 we built conversation memory. Every time you send a message, Spring AI includes the last 20 messages in the request. That means message 20 doesn't cost 1 message worth of tokens — it costs up to 20 messages worth.

A conversation with 10 back-and-forth exchanges might look like 10 API calls. But the token cost is:

Call 1: 1 message
Call 2: 3 messages (1+2)
Call 3: 6 messages (1+2+3)
...
Call 10: 55 messages worth of tokens

Problem 2: Your system prompt runs on every call.

Every request includes the system prompt. If your system prompt is 200 words, that's ~270 tokens added to every single API call — silently.

Problem 3: Output tokens cost too.

The model's reply counts against your token budget. A long detailed answer costs more than a short one. By default, the model will be as verbose as it wants to be.

See exactly what you're spending

The fix starts with visibility. Spring AI exposes token usage on every response — you just have to ask for it.

Change your chat method to use chatResponse() instead of content():

@PostMapping("/chat-ai")
public String chat(@RequestBody Map<String, String> request) {
    ChatResponse response = chatClient.prompt()
        .user(request.get("message"))
        .advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
        .call()
        .chatResponse();   // ← get the full response object, not just the text

    // Log token usage
    Usage usage = response.getMetadata().getUsage();
    log.info("Tokens — input: {}, output: {}, total: {}",
        usage.getPromptTokens(),
        usage.getGenerationTokens(),
        usage.getTotalTokens());

    return response.getResult().getOutput().getText();
}

Now every API call logs something like:

Tokens — input: 847, output: 312, total: 1159
Tokens — input: 1203, output: 198, total: 1401
Tokens — input: 1589, output: 445, total: 2034

Watch the input token count grow with each message in a conversation. That's the history accumulating.

Track a running total per session

Once you can see individual call usage, it's useful to track cumulative usage per session. A simple in-memory counter works fine for this:

private final Map<String, Integer> sessionTokens = new ConcurrentHashMap<>();

@PostMapping("/chat-ai")
public String chat(@RequestBody Map<String, String> request) {
    String conversationId = request.get("conversationId");

    ChatResponse response = chatClient.prompt()
        .user(request.get("message"))
        .advisors(a -> a.param("chat_memory_conversation_id", conversationId))
        .call()
        .chatResponse();

    Usage usage = response.getMetadata().getUsage();
    int callTokens = usage.getTotalTokens();

    int runningTotal = sessionTokens.merge(conversationId, callTokens, Integer::sum);

    log.info("conversationId: {} | this call: {} tokens | session total: {} tokens",
        conversationId, callTokens, runningTotal);

    return response.getResult().getOutput().getText();
}

After a few messages you'll see the session total climbing. This makes the accumulation visible and gives you something to act on.

How to reduce token usage

Now that you can see what you're spending, here's where to trim.

1. Keep your system prompt short.

Every extra word in your system prompt costs tokens on every single call. Go through it and cut anything that isn't load-bearing.

Before:

You are a helpful and friendly AI assistant. You are knowledgeable about many topics 
and always try to give thorough, well-explained answers. You are patient and 
understanding. You never refuse to help with reasonable questions. You always 
maintain a positive and encouraging tone in all your responses.

After:

You are a helpful assistant. Be concise and practical.

Same behaviour, a fraction of the tokens — multiplied across every API call you make.

2. Limit conversation history.

We already set maxMessages(20) in Article 6. That's a good default. For development and testing, drop it lower:

MessageWindowChatMemory memory = MessageWindowChatMemory.builder()
    .chatMemoryRepository(memoryRepository)
    .maxMessages(10)   // ← 10 messages instead of 20 while testing
    .build();

Older messages in a long conversation rarely affect the current reply anyway.

3. Tell the model to be concise.

The model's default verbosity is tunable via the system prompt:

You are a helpful assistant. Be concise — answer in 2-3 sentences unless the user 
asks for more detail.

A shorter reply means fewer output tokens. Output tokens count against the same limits as input tokens.

4. Don't test with long conversations.

During development, start a new session for every test instead of continuing an existing conversation. A fresh session has zero history — each call costs only the system prompt + one message, not the accumulated history of 20 exchanges.

Beyond maxMessages — how production apps handle long conversations

You've been using maxMessages(20) since Article 6. That's the simplest strategy — and it has a name: sliding window. But it's not the only approach. Here's how the strategies stack up as conversations get longer and requirements get harder.

Sliding window — what you already have

Keep only the last N messages. Older messages are dropped. Simple, predictable, zero extra API calls.

Works well for most chat apps where conversations are short-to-medium. The downside: the model loses context from early in the conversation. If the user mentioned their name in message 1 and you're now on message 25, the model has forgotten it.

Summarization

When history hits a threshold, ask the model to summarize the older messages into a compact paragraph. Replace those messages with the summary. Then continue with summary + recent messages.

if (history.size() > 15) {
    // Ask the model to summarize older messages
    String oldMessages = formatMessages(history.subList(0, 10));

    String summary = chatClient.prompt()
        .user("Summarize this conversation in 4-5 sentences, keeping key facts and decisions: "
              + oldMessages)
        .call()
        .content();

    // Replace 10 old messages with one summary message
    List<Message> compressed = new ArrayList<>();
    compressed.add(new SystemMessage("Earlier conversation summary: " + summary));
    compressed.addAll(history.subList(10, history.size()));
    history = compressed;
}

2000 tokens of raw history becomes 150 tokens of summary. The model keeps the gist without the full detail. This is what Claude Code does — the summary you see at the start of a long session is exactly this pattern applied automatically.

The tradeoff: one extra API call to summarize, and fine-grained detail from early messages is lost. For most apps that's acceptable.

RAG-based memory (coming later in this series)

Instead of sending recent messages, store every message as an embedding in a vector database. When a new message arrives, search for the most relevant past messages — not just the most recent ones.

This means a conversation from last week about a specific topic gets retrieved when relevant, even if it's buried under hundreds of other messages. We'll build this when we cover RAG.

Structured memory extraction (coming later in this series)

Instead of storing raw messages, the system extracts facts from the conversation and stores them separately:

{
  "user_name": "Sham",
  "project": "Spring Boot AI chat app",
  "decisions": ["Gemini API", "Neon PostgreSQL", "deployed on Render"]
}

The model receives this fact store + recent messages — not raw history at all. It's how products like ChatGPT's memory feature work. Much more token-efficient for long-running applications.

For now, sliding window (maxMessages(20)) handles most cases. Add summarization when your conversations consistently hit the limit. RAG and structured memory come later in the series when we have the foundations for them.

When to upgrade

The free tier is enough to learn and build demos. You'll hit limits when:

You're testing heavily (many short test sessions in the same day)
Your conversations get long (history accumulates fast)
Multiple people are using the app simultaneously

When you do upgrade to a paid tier, the rate limits increase significantly and you're charged per million tokens instead. At that point the logging you built here becomes essential — it's how you know what you're actually paying for.

For now, the key habits are: log every call, watch the input token growth, keep your system prompt lean, and use fresh sessions during testing.

What's next

The app is efficient now. Next: writing system prompts that actually shape the model's behaviour — not just what it says, but how it thinks about your problem.

Hit a 429 while building? Drop it in the comments — you're definitely not the only one.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

How I Made My AI App Stream Like ChatGPT — Server-Sent Events in Spring Boot

Sham Prakash K — Sun, 27 Sep 2026 14:30:00 +0000

The app is live. Users can chat with the AI and it remembers the conversation.

But there's one problem I noticed immediately after deploying: a user sends a message, and then they stare at a completely blank screen for three to five seconds. Then — bam — the full reply appears all at once.

Compare that to ChatGPT or Gemini's own interface. You type a question and the response starts flowing immediately, word by word. It feels alive.

That difference is streaming. This article is about adding it.

Why the delay exists

When you call .call().content() in Spring AI, here's what actually happens:

Your message goes to the Gemini API
Gemini starts generating the response — one token at a time
Gemini keeps generating until the response is complete
Gemini sends the entire response back to your server in one HTTP response
Your server returns it to the client

The model isn't slow — it's generating tokens quickly. But you're waiting for the last token before you get the first one. All that generation time appears as a blank screen.

Streaming flips this. Instead of waiting for everything, you receive each token the moment the model generates it — and send it to the client immediately.

What SSE is

Server-Sent Events (SSE) is a simple HTTP mechanism for the server to push data to the client over a single connection that stays open.

Normal HTTP: client sends request → server sends response → connection closes.

SSE: client sends request → server keeps the connection open and sends data in chunks as it becomes available → connection closes when the stream ends.

It's one-directional (server → client only), which is exactly what we need for streaming AI responses. The client asks once, and the server streams the answer back token by token.

What changes in the code

Right now the chat endpoint looks like this:

@PostMapping("/chat-ai")
public ResponseEntity<Map<String, String>> chat(@RequestBody Map<String, String> request) {
    String answer = chatClient.prompt()
        .user(request.get("message"))
        .advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
        .call()       // ← waits for the full response
        .content();

    return ResponseEntity.ok(Map.of("answer", answer));
}

The streaming version changes two things:

.call() becomes .stream()
The return type becomes Flux<String> instead of ResponseEntity<String>

@PostMapping(value = "/chat-ai/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> streamChat(@RequestBody Map<String, String> request) {
    return chatClient.prompt()
        .user(request.get("message"))
        .advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
        .stream()     // ← returns a Flux, not a String
        .content();
}

That's it. The same chatClient, the same memory advisor, the same conversation ID — just .stream() instead of .call().

What is Flux?

Flux<String> is from Project Reactor — the reactive library that Spring WebFlux is built on. Think of it as a sequence of values that arrive over time, not all at once.

.call().content() gives you one String — the complete reply.

.stream().content() gives you a Flux<String> — a stream of token chunks that arrive one by one as Gemini generates them.

Spring automatically serialises a Flux<String> return type as SSE when you set produces = MediaType.TEXT_EVENT_STREAM_VALUE. Each item in the Flux becomes a data: event in the SSE stream.

You don't need to add any WebFlux dependency — Spring AI's streaming support works in a regular Spring Boot MVC application. The Flux return type is handled automatically.

Add the dependency

Make sure you have the Reactor dependency. If you're using spring-boot-starter-web, add:

<dependency>
    <groupId>io.projectreactor</groupId>
    <artifactId>reactor-core</artifactId>
</dependency>

Spring Boot manages the version through the BOM, so no version needed.

Full streaming controller

@RestController
@RequestMapping("/api")
public class StreamingChatController {

    private final ChatClient chatClient;

    public StreamingChatController(ChatClient.Builder builder, JdbcTemplate jdbcTemplate) {
        JdbcChatMemoryRepository memoryRepository = JdbcChatMemoryRepository.builder()
            .jdbcTemplate(jdbcTemplate)
            .dialect(new ChatHistoryDialect())
            .build();

        MessageWindowChatMemory memory = MessageWindowChatMemory.builder()
            .chatMemoryRepository(memoryRepository)
            .maxMessages(20)
            .build();

        this.chatClient = builder
            .defaultSystem("You are a helpful assistant.")
            .defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())
            .build();
    }

    // Non-streaming — returns the full reply at once
    @PostMapping("/chat-ai")
    public String chat(@RequestBody Map<String, String> request) {
        return chatClient.prompt()
            .user(request.get("message"))
            .advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
            .call()
            .content();
    }

    // Streaming — returns tokens as they arrive
    @PostMapping(value = "/chat-ai/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> streamChat(@RequestBody Map<String, String> request) {
        return chatClient.prompt()
            .user(request.get("message"))
            .advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
            .stream()
            .content();
    }

    record ChatRequest(String conversationId, String message) {}
}

Both endpoints share the same chatClient with the same memory. The streaming endpoint is just an extra route — your existing non-streaming endpoint keeps working.

Test it with curl

Start the app, then run:

curl -N -X POST http://localhost:8080/api/chat-ai/stream \
  -H "Content-Type: application/json" \
  -d '{"conversationId":"test-123","message":"Explain what a Docker container is in simple terms"}'

The -N flag disables buffering — without it curl would collect everything before displaying it, which defeats the purpose.

You should see tokens arriving one by one in your terminal, not all at once:

data:A

data: Docker

data: container

data: is

data: like

data: a

data: lightweight

data: virtual

data: machine
...

Each data: line is one SSE event — one token chunk from the model.

What about the frontend?

The browser has a built-in EventSource API for consuming SSE, but it only supports GET requests. Since your endpoint is a POST (because you're sending a JSON body), you use the Fetch API with a readable stream instead:

const response = await fetch('/api/chat-ai/stream', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ conversationId, message })
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  const chunk = decoder.decode(value);
  // each chunk is "data: token\n\n" — parse and append to your UI
  const token = chunk.replace(/^data: /, '').trim();
  appendToUI(token);
}

Each chunk you read from the stream is a token. Append it to the message div as it arrives — same effect as ChatGPT's typing animation.

Streaming vs non-streaming — when to use which

	Non-streaming	Streaming
Response time to first byte	Slow (waits for full reply)	Instant (first token arrives immediately)
UX	Blank screen then full reply	Tokens appear as they're generated
Frontend complexity	Simple fetch + JSON	Fetch + ReadableStream parsing
Good for	Internal APIs, background processing	Any user-facing chat interface

Keep both endpoints. Use the streaming one for your frontend chat UI. Use the non-streaming one for internal calls where you need the complete reply as a string — tool calling, post-processing, logging the full response.

What's next

The app now streams responses like ChatGPT. But streaming means more API calls, and more API calls means more tokens — and tokens cost money. Next: understanding and managing API cost before your first surprise bill.

Noticed a difference in feel between streaming and waiting for the full reply? Drop it in the comments.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Stop Running Your AI App on Your Laptop — Deploy It to the Cloud for Free

Sham Prakash K — Fri, 25 Sep 2026 14:30:00 +0000

The chat app works. It remembers conversations. It stores history in PostgreSQL.

And it runs on my laptop.

That's not a chat app. That's a script. The moment I close the terminal, it's gone. Nobody else can use it.

This article is about fixing that. We're taking the app and deploying it to the cloud so it runs 24/7 at a real URL.

Where to deploy — a quick tour

There are several platforms that make deploying a Spring Boot app straightforward. Here's the honest overview:

Railway — fast setup, generous free tier, great developer experience. The free tier has a monthly usage limit that can catch you off guard if the app gets traffic.

Fly.io — powerful, runs real VMs in regions close to your users, excellent for performance-sensitive apps. The free tier is there but the setup is more involved — you configure regions, vm sizes, and scaling yourself.

Koyeb — solid option, deploys from Docker or GitHub, decent free tier. Less documentation than the others.

Render — this is what I used. Free tier, no credit card required for basic deployment, deploys directly from a GitHub repo and rebuilds on every push. The free tier does spin down after 15 minutes of inactivity, but for learning and demos it's exactly what you need.

We're going with Render.

What you need before deploying

Three things need to be true before deployment works:

Your code is in a GitHub repo
Your secrets are in environment variables, not in the code
Your app is packaged as a Docker image

You probably have 1 and 2 already. Let's talk about 3.

Why Docker?

When you run the app locally, Java is installed on your machine. Maven is installed. The right version of everything is there.

Render's servers don't have your setup. They don't know you're using Java 17. They don't know your Maven version. They don't know your dependencies.

Docker solves this by packaging your app and everything it needs into one self-contained image. The image runs the same way everywhere — your laptop, Render, AWS, anywhere.

Think of it like a shipping container. Before shipping containers, every port had to know how to handle every type of cargo. After containers, you just move the box — the contents are someone else's problem.

A Dockerfile is the recipe for building that box.

The Dockerfile

Create a file called Dockerfile in the root of your project (same level as pom.xml):

FROM maven:3.9-eclipse-temurin-17 AS build
WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline -q
COPY src ./src
RUN mvn package -DskipTests -q

FROM eclipse-temurin:17-jre
WORKDIR /app
COPY --from=build /app/target/gemini-chat-0.0.1-SNAPSHOT.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

This is a multi-stage build. Two stages, one file.

Stage 1 — build the app:

FROM maven:3.9-eclipse-temurin-17 AS build starts from a base image that already has Maven and Java 17 installed. You name this stage build.

COPY pom.xml . then RUN mvn dependency:go-offline — this is a Docker trick. By copying just pom.xml first and downloading dependencies before copying source code, Docker can cache that layer. If you only change Java files later, Docker skips the dependency download step entirely. Builds get much faster.

COPY src ./src then RUN mvn package -DskipTests — now copy the source and compile. -DskipTests is intentional here; you'd run tests in CI separately.

Stage 2 — run the app:

FROM eclipse-temurin:17-jre — this is a different base image. Just the JRE (Java Runtime Environment), not the full JDK with Maven. Much smaller — a JDK image can be 500MB+, a JRE is ~200MB.

COPY --from=build /app/target/gemini-chat-0.0.1-SNAPSHOT.jar app.jar — copy just the compiled jar from stage 1. The Maven installation, all the source code, the intermediate build files — none of that comes along.

EXPOSE 8080 — tells Docker (and Render) what port the app listens on.

ENTRYPOINT ["java", "-jar", "app.jar"] — the command that runs when the container starts.

The final image contains only the JRE and your jar. Everything else stays behind in stage 1 and is discarded.

Where does the Docker image go?

This is something I wasn't clear on at first.

You never push a Docker image anywhere. You don't need Docker Hub. You don't need Docker installed on your machine at all.

Here's what actually happens when you deploy to Render:

You push your source code to GitHub (including the Dockerfile)
Render pulls that code from GitHub
Render runs your Dockerfile on their own build servers — Maven downloads dependencies, compiles your Java code, packages the jar
The resulting Docker image is stored in Render's internal container registry
Render starts a container from that image and gives you a URL

The image lives on Render's infrastructure. Maven is only involved inside stage 1 of the build — it's just the tool that compiles the Java code. Once the image is built, Maven is gone.

The only reason to run Docker locally is if you want to test the image before pushing:

docker build -t gemini-chat .
docker run -p 8080:8080 -e GEMINI_API_KEY=your_key gemini-chat

But for deploying — GitHub to Render is all you need.

Check your environment variables

Before pushing, verify your application.properties uses environment variables, not hardcoded values:

# Gemini
spring.ai.google.gemini.api-key=${GEMINI_API_KEY}
spring.ai.google.gemini.chat.options.model=gemini-2.5-flash

# Database
spring.datasource.url=${DATABASE_URL}
spring.datasource.username=${DATABASE_USERNAME}
spring.datasource.password=${DATABASE_PASSWORD}

spring.sql.init.schema-locations=classpath:chat-memory-schema.sql
spring.sql.init.mode=always

Every secret is read from the environment at runtime. If someone opens your GitHub repo, they see variable names, not actual values.

Also check your .gitignore — this should already be there, but verify:

# Never commit this
.env
*.env
application-local.properties

Deploy to Render

Step 1 — Push your code to GitHub

Your project needs to be in a GitHub repo. If it's not already:

git init
git add .
git commit -m "initial commit"
git remote add origin https://github.com/yourusername/gemini-chat.git
git push -u origin main

Make absolutely sure your API keys and database credentials are not in any committed file.

Step 2 — Create a Render account

Go to render.com and sign up. You can use GitHub to sign in — that's the easiest option since Render will need access to your repos anyway.

Step 3 — Create a new Web Service

In the Render dashboard:

Click New → Web Service
Connect your GitHub account and select your repository
Render will detect the Dockerfile automatically

Set the basics:

Name: gemini-chat (or whatever you like)
Region: Pick the closest one to you
Branch: main
Instance type: Free

Step 4 — Add environment variables

In the service settings, scroll to Environment Variables and add:

Key	Value
`GEMINI_API_KEY`	your Gemini API key
`DATABASE_URL`	your Neon connection string
`DATABASE_USERNAME`	your Neon username
`DATABASE_PASSWORD`	your Neon password

This is where secrets live on Render — not in your code, not in your repo. Render injects these as environment variables when the container starts.

Step 5 — Deploy

Click Create Web Service. Render will:

Pull your code from GitHub
Build the Docker image (this runs your Dockerfile — Maven downloads dependencies, compiles the app, packages the jar)
Start the container
Give you a URL like https://gemini-chat-xxxx.onrender.com

The first build takes a few minutes. After that, every push to main triggers a new build automatically.

Test it

Once the service is live, send a request to your Render URL:

curl -X POST https://gemini-chat-xxxx.onrender.com/api/chat-ai/session

You should get back a conversation ID. Then:

curl -X POST https://gemini-chat-xxxx.onrender.com/api/chat-ai/chat \
  -H "Content-Type: application/json" \
  -d '{"conversationId":"your-id-here","message":"Hello, are you running in the cloud?"}'

If you get a response — it's live. Running on Render's servers, hitting Gemini's API, storing history in Neon PostgreSQL. Not on your laptop anymore.

The free tier caveat

Render's free tier spins the service down after 15 minutes of no traffic. The first request after a sleep wakes it up, which takes about 30-60 seconds. For production you'd pay for an always-on instance, but for learning and demos the free tier is fine.

If you want to keep it warm, you can set up a simple ping from a service like UptimeRobot — free, hits your URL every 5 minutes, keeps the service awake.

What just happened

Let's step back and look at what you've built:

A Spring Boot REST API that calls Gemini
Conversation history that persists in PostgreSQL
Deployed to the cloud with Docker
Running at a real URL, 24/7
All secrets safely in environment variables, never in code

That's a real AI backend. Not a tutorial project — a deployed service.

What's next

The app is live. Next: making the responses stream in real time — instead of waiting for the full reply, you see tokens appear as the model generates them. Same way ChatGPT works.

Deployed your first AI app? Drop the URL in the comments.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

Why Your AI Chatbot Forgets Everything — And How to Fix It

Sham Prakash K — Wed, 23 Sep 2026 14:30:00 +0000

In the last article we built a working chat endpoint. Send a message, get a reply. It felt like magic.

Then I tried to have an actual conversation.

Me: "My name is Sham."
AI: "Hi Sham! How can I help you?"
Me: "What's my name?"
AI: "I don't have access to personal information about you."

The model had completely forgotten who I was. Not because it was broken — because of something fundamental about how LLMs work. Every API call is completely independent. The model has no memory between calls.

If you want it to remember anything, that's your problem to solve.

This article shows how — starting from the simplest possible solution, hitting its limits, then building the real one.

Why the model forgets

When you call the Gemini API, you send a list of messages. The model reads them, generates a reply, and the call ends. The next call starts completely fresh — the model has no idea the previous call ever happened.

So when the user sends message 5, the model only sees message 5. It has no knowledge of messages 1 through 4.

The fix is simple in concept: include all previous messages in every call. Send the full conversation history every time, so the model always has context.

Let's build that.

Solution 1 — A simple Map in memory

The simplest fix: a Map where the key is a session ID and the value is the list of messages for that session.

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatClient chatClient;

    // session ID → list of messages for that session
    private final Map<String, List<Message>> sessions = new ConcurrentHashMap<>();

    public ChatController(ChatClient.Builder builder) {
        this.chatClient = builder
            .defaultSystem("You are a helpful assistant.")
            .build();
    }

    @PostMapping("/session")
    public Map<String, String> startSession() {
        String sessionId = UUID.randomUUID().toString();
        sessions.put(sessionId, new ArrayList<>());
        return Map.of("sessionId", sessionId);
    }

    @PostMapping
    public String chat(@RequestBody ChatRequest request) {
        List<Message> history = sessions.getOrDefault(
            request.sessionId(), new ArrayList<>());

        // Add user message to history
        history.add(new UserMessage(request.message()));

        // Send full history to the model
        String reply = chatClient.prompt()
            .messages(history)
            .call()
            .content();

        // Add model reply to history
        history.add(new AssistantMessage(reply));
        sessions.put(request.sessionId(), history);

        return reply;
    }

    record ChatRequest(String sessionId, String message) {}
}

Every user gets their own session ID. Their messages are stored in their own list. Every API call sends the full history for that session — so the model has context.

Now try the conversation:

Me: "My name is Sham."
AI: "Hi Sham! How can I help you?"
Me: "What's my name?"
AI: "Your name is Sham."

It works. Two different users with two different session IDs — completely separate conversations.

The code is simple. Every Java developer knows what a Map and a List are. No framework magic, just plain Java.

The problem with in-memory history

This works great — until you restart the server. All history is gone. Everyone's conversations, gone.

There's another problem: this is a single ArrayList shared across all users. User A and User B are in the same conversation. Not great.

And there's the token problem: a long conversation becomes thousands of tokens on every call, whether those old messages are relevant or not.

In-memory works for a quick demo. For anything real, you need persistent storage with session isolation.

Where to get a free PostgreSQL database

Before writing any code, you need a database. The easiest free option is Neon — serverless PostgreSQL, free tier, no credit card required.

Go to neon.tech and sign up
Create a new project — Neon gives you a PostgreSQL database instantly
Copy the connection string from the dashboard — it looks like:

postgresql://username:password@ep-xxx.us-east-1.aws.neon.tech/dbname?sslmode=require

Set it as an environment variable:

DATABASE_URL=postgresql://username:password@...

That's it. Free, no setup, no local PostgreSQL installation needed.

Solution 2 — PostgreSQL with Spring AI

Spring AI has a built-in JdbcChatMemoryRepository that stores conversation history in a database. Each conversation gets a unique ID — so different users are completely isolated from each other.

Step 1 — Add the dependency

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-chat-memory-repository-jdbc</artifactId>
</dependency>
<dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <scope>runtime</scope>
</dependency>

Step 2 — Create the table

Create src/main/resources/chat-memory-schema.sql:

CREATE TABLE IF NOT EXISTS chat_history (
    conversation_id VARCHAR(256) NOT NULL,
    content         TEXT         NOT NULL,
    type            VARCHAR(64)  NOT NULL,
    timestamp       TIMESTAMP    NOT NULL
);

Step 3 — Tell Spring AI to use your table

By default Spring AI uses a table called SPRING_AI_CHAT_MEMORY. To use your own table name, implement JdbcChatMemoryRepositoryDialect:

public class ChatHistoryDialect implements JdbcChatMemoryRepositoryDialect {

    private static final String TABLE = "chat_history";

    @Override
    public String getSelectMessagesSql() {
        return "SELECT content, type FROM " + TABLE +
               " WHERE conversation_id = ? ORDER BY timestamp";
    }

    @Override
    public String getInsertMessageSql() {
        return "INSERT INTO " + TABLE +
               " (conversation_id, content, type, timestamp) VALUES (?, ?, ?, ?)";
    }

    @Override
    public String getSelectConversationIdsSql() {
        return "SELECT DISTINCT conversation_id FROM " + TABLE;
    }

    @Override
    public String getDeleteMessagesSql() {
        return "DELETE FROM " + TABLE + " WHERE conversation_id = ?";
    }
}

Step 4 — Wire it up in the controller

@RestController
@RequestMapping("/api/chat-ai")
public class SpringAiChatController {

    private final ChatClient chatClient;
    private final JdbcChatMemoryRepository memoryRepository;

    public SpringAiChatController(ChatClient.Builder builder, JdbcTemplate jdbcTemplate) {

        this.memoryRepository = JdbcChatMemoryRepository.builder()
            .jdbcTemplate(jdbcTemplate)
            .dialect(new ChatHistoryDialect())
            .build();

        // Keep last 20 messages — older ones are evicted automatically
        MessageWindowChatMemory memory = MessageWindowChatMemory.builder()
            .chatMemoryRepository(memoryRepository)
            .maxMessages(20)
            .build();

        this.chatClient = builder
            .defaultSystem("You are a helpful assistant.")
            .defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())
            .build();
    }

    // Create a new session — returns a unique conversation ID
    @PostMapping("/session")
    public Map<String, String> startSession() {
        String conversationId = UUID.randomUUID().toString();
        return Map.of("conversationId", conversationId);
    }

    // Chat — pass the conversation ID with every message
    @PostMapping("/chat")
    public String chat(@RequestBody ChatRequest request) {
        return chatClient.prompt()
            .user(request.message())
            .advisors(a -> a.param("chat_memory_conversation_id", request.conversationId()))
            .call()
            .content();
    }

    // Delete a conversation
    @DeleteMapping("/session/{conversationId}")
    public Map<String, String> deleteSession(@PathVariable String conversationId) {
        memoryRepository.deleteByConversationId(conversationId);
        return Map.of("status", "deleted", "conversationId", conversationId);
    }

    record ChatRequest(String conversationId, String message) {}
}

Step 5 — Configure application.properties

spring.datasource.url=${DATABASE_URL}
spring.datasource.username=${DATABASE_USERNAME}
spring.datasource.password=${DATABASE_PASSWORD}

spring.sql.init.schema-locations=classpath:chat-memory-schema.sql
spring.sql.init.mode=always

mode=always forces the schema script to run on every startup. Without it, PostgreSQL (being a non-embedded database) won't run the script at all.

How it works now

Starting a conversation:

POST /api/chat-ai/session
→ { "conversationId": "a3f9b2c1-..." }

Sending messages — pass the ID every time:

POST /api/chat-ai/chat
{ "conversationId": "a3f9b2c1-...", "message": "My name is Sham." }
→ "Hi Sham! How can I help you?"

POST /api/chat-ai/chat
{ "conversationId": "a3f9b2c1-...", "message": "What's my name?" }
→ "Your name is Sham."

Spring AI fetches the last 20 messages for that conversation ID from PostgreSQL, includes them in the API call, saves the new exchange, and returns the response. You wrote none of that logic yourself.

Two users, two different conversation IDs — completely isolated. Server restarts — history survives. Long conversation — only the last 20 messages are sent, keeping tokens under control.

In-memory vs PostgreSQL — when to use which

	Spring AI In-Memory	Spring AI PostgreSQL
Setup	Zero	Database + dependency
Survives restart	No	Yes
Multi-user	Yes — isolated by session ID	Yes — isolated by session ID
Token control	Automatic (maxMessages)	Automatic (maxMessages)
Good for	Local dev, quick demos	Production, anything real

Start with in-memory locally. Switch to PostgreSQL before you deploy.

What's next

The chat app has memory now. Next up: deploying it to the cloud — Render, Docker, environment variables. Because a chat app that only runs on your laptop isn't a chat app, it's a script.

Have you hit the "it forgot everything" problem before understanding why? Drop it in the comments.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

From 20 Lines to 4: My First AI Endpoint in Spring Boot

Sham Prakash K — Mon, 21 Sep 2026 14:30:00 +0000

Before you install any framework, before you add any dependency — you should make one raw HTTP call to the model yourself. Just to see what it actually is.

That's how I started. And it's how I'd recommend everyone starts.

Step 1 — Get a Gemini API key

Go to Google AI Studio and create a free API key. No credit card needed. Gemini has a generous free tier — more than enough to learn and build.

Once you have the key, store it as an environment variable. Never paste it directly into your code.

GEMINI_API_KEY=your_key_here

Step 2 — Understand what you're calling

Gemini exposes a REST API. The endpoint looks like this:

POST https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent?key=YOUR_API_KEY

You send a JSON body, you get a JSON response. That's it. No magic.

The request body looks like this:

{
  "contents": [
    {
      "role": "user",
      "parts": [{ "text": "What is a token in LLMs?" }]
    }
  ]
}

And the response comes back like this:

{
  "candidates": [
    {
      "content": {
        "parts": [{ "text": "A token is a piece of text..." }],
        "role": "model"
      }
    }
  ]
}

Your answer is at candidates[0].content.parts[0].text. Navigate that JSON tree and you have your response.

Step 3 — Build it with plain Java, no frameworks

Here's a working Spring Boot controller that calls Gemini using nothing but Java's built-in HttpClient and Jackson (which Spring Boot already includes). No Spring AI. No extra dependencies.

pom.xml — intentionally minimal:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

That's the only dependency. Spring Boot 3.x includes Jackson. Java 11+ includes HttpClient. Nothing else needed.

application.properties:

gemini.api.key=${GEMINI_API_KEY}
gemini.api.url=https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent

ChatController.java:

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    @Value("${gemini.api.key}")
    private String apiKey;

    @Value("${gemini.api.url}")
    private String apiUrl;

    private final ObjectMapper objectMapper = new ObjectMapper();
    private final HttpClient httpClient = HttpClient.newHttpClient();

    @PostMapping
    public ResponseEntity<String> chat(@RequestBody String userMessage) throws Exception {

        // Build the request body
        String requestBody = """
            {
              "contents": [
                {
                  "role": "user",
                  "parts": [{ "text": "%s" }]
                }
              ]
            }
        """.formatted(escape(userMessage));

        // Make the HTTP call
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create(apiUrl + "?key=" + apiKey))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(requestBody))
            .build();

        HttpResponse<String> response = httpClient.send(request,
            HttpResponse.BodyHandlers.ofString());

        // Parse the response — answer is at candidates[0].content.parts[0].text
        JsonNode root = objectMapper.readTree(response.body());

        if (root.has("error")) {
            return ResponseEntity.status(500)
                .body(root.path("error").path("message").asText());
        }

        String reply = root
            .path("candidates").get(0)
            .path("content")
            .path("parts").get(0)
            .path("text").asText();

        return ResponseEntity.ok(reply);
    }

    private String escape(String text) {
        return text
            .replace("\\", "\\\\")
            .replace("\"", "\\\"")
            .replace("\n", "\\n")
            .replace("\r", "\\r");
    }
}

Start the app, send a POST request to /api/chat with a message body, and you'll get a response from Gemini.

That's a fully working AI chat endpoint. No AI framework. Just HTTP.

Full source code for this phase: github.com/shamprakash2000/gemini-chat/tree/phase-1-2-plain-http

What you just built — and what's missing

This works. Send a message, get a response. That's a real AI endpoint.

But look at what you're doing manually:

Building JSON by hand — string formatting, escaping characters, constructing the payload yourself
Parsing JSON by hand — navigating .path().get().path() to extract the answer
No model switching — the URL and response format are Gemini-specific. Switching to Claude means rewriting everything

For one simple call, this is fine. But the moment you want conversation memory, tool calling, streaming, or RAG — you're building a framework from scratch.

That's where Spring AI comes in.

What Spring AI actually is

Spring AI is the Spring team's answer to: "we keep writing the same boilerplate to call LLMs — let's standardise it."

It gives you:

A common interface across models. The same ChatClient code works for Gemini, OpenAI, Claude, Ollama. You swap the dependency and config — the Java code stays the same.

No more manual JSON. The framework builds the request payload and parses the response for you.

Conversation memory, tool calling, structured output, streaming — all built in. We'll get to each of these in the articles ahead. For now, just understand what the foundation gives you.

The same endpoint, rewritten with Spring AI

Add the dependency:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>1.1.8</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-google-gemini</artifactId>
</dependency>

Update application.properties:

spring.ai.google.gemini.api-key=${GEMINI_API_KEY}
spring.ai.google.gemini.chat.options.model=gemini-2.5-flash

Now the controller:

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient.Builder builder) {
        this.chatClient = builder
            .defaultSystem("You are a helpful assistant.")
            .build();
    }

    @PostMapping
    public String chat(@RequestBody String userMessage) {
        return chatClient.prompt()
            .user(userMessage)
            .call()
            .content();
    }
}

Same functionality. A fraction of the code.

What Spring AI does not protect you from

I want to be honest here because most tutorials aren't.

The API changes between minor versions.

I was on Spring AI 1.0.x and upgraded to 1.1.x. Two things broke silently — no compiler errors, no deprecation warnings. Just runtime failures. The ChatClient builder API had changed, and the way conversation memory was configured had moved to a different class entirely.

Lesson: when you upgrade Spring AI, read the full changelog and test every integration point. Don't assume a minor version bump is safe.

The abstraction hides internals.

The plain HTTP version showed you exactly what was happening — every byte in, every byte out. Spring AI hides that. When something goes wrong deep in the framework, you need to know what's underneath to debug it. That's another reason to start with plain HTTP first — so you have that mental model when the abstraction breaks.

Plain HTTP vs Spring AI — the honest comparison

	Plain HTTP	Spring AI
Dependencies	None beyond Spring Web	Spring AI starter
Code volume	High	Low
JSON handling	Manual	Automatic
Model switching	Full rewrite	Swap config
Tool calling / agents	Build from scratch	Built in
Debugging	Full visibility	Abstraction hides internals
Version stability	Stable (it's just HTTP)	Breaking changes between minors
Good for	Learning, full control	Production Spring Boot apps

Start with plain HTTP. Understand what you're abstracting. Then move to Spring AI.

What's next

Spring AI is set up and working. Next: building a proper chat application — with persistent conversation history in PostgreSQL, session management, and a system prompt that shapes the model's behaviour.

Did you hit a breaking change in Spring AI between versions? Drop it in the comments — you're not the only one.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

How Your Message Actually Reaches the AI Model — And What You're Really Sending

Sham Prakash K — Sat, 19 Sep 2026 14:30:00 +0000

You've used ChatGPT. You've used Gemini. You type something, it replies.

But have you ever stopped and wondered — what is actually happening when you hit send?

Because when you're building AI systems, that question matters. A lot.

What happens when you type a message

Let's take the simplest example. You open ChatGPT and type:

"Explain what a token is in simple terms."

You hit enter. A few seconds later, you get a response.

What just happened?

Your message travelled over the internet to a server — a very powerful one, running a very large model — where it was processed, and text was generated back to you. That's it. At the core, it is a request and a response. Text goes in, text comes out.

Now here's the important part: that model lives somewhere. It doesn't run in your browser. It doesn't run on your laptop. It runs on a server, hosted by OpenAI in this case. You're connected to it over the internet without even thinking about it.

This is the same thing that happens when you build an AI backend. You send text to a model. You get text back. The difference is — you're the one writing the code that sends and receives it, not a chat UI.

Where can a model live?

Here's something most tutorials skip: the model doesn't have to be on someone else's server.

There are three places a model can live:

1. Cloud — hosted by the company (Gemini, ChatGPT, Claude)

Google hosts Gemini on their servers. OpenAI hosts GPT. Anthropic hosts Claude. You connect to them over the internet using an API key — a secret key they give you that proves you're allowed to use their model.

You pay per use. Every token you send and receive costs a small amount. The models are powerful, always up to date, and you don't manage any infrastructure.

This is what we use in this series — Gemini, hosted by Google.

2. Local — running on your own machine (Ollama, LM Studio)

Yes, you can download a model and run it on your laptop.

Tools like Ollama let you pull a model — Llama 3, Mistral, Phi — and run it locally. No internet needed. No API key. No cost per call. The model runs on your CPU or GPU and responds to requests at http://localhost:11434.

The tradeoff: smaller models, slower responses, and your laptop fans will make their presence known. But for learning, experimentation, or privacy-sensitive use cases — it's a great option.

3. Self-hosted — your own server, your own model

Companies with strict data privacy requirements sometimes host their own models on their own infrastructure. Same idea as Ollama but on a cloud server they control. No data ever leaves their network.

This is less common for individual developers but worth knowing exists.

Cloud vs Local — which should you use?

	Cloud (Gemini, GPT, Claude)	Local (Ollama)
Cost	Pay per token	Free
Setup	API key, done	Download model, install Ollama
Model quality	State of the art	Smaller, less capable
Speed	Fast (their hardware)	Depends on your machine
Privacy	Data goes to their servers	Stays on your machine
Internet needed	Yes	No
Good for	Production, serious projects	Learning, experiments, privacy

For this series, we use Gemini — cloud hosted, API key, fast, capable. But if you want to experiment without spending money, Ollama is a great companion tool. The concepts are identical. Only the URL and key change.

How you connect — the API key

When you use a cloud-hosted model, you authenticate with an API key.

Think of it like a password. You create one in the provider's dashboard, store it securely, and include it in every request you make. The server checks the key, confirms you're allowed to use the model, and processes your request.

Your app  ──── request + API key ────▶  Gemini servers
Your app  ◀─── response ────────────   Gemini servers

For local models with Ollama, there's no key. You just call http://localhost:11434 directly. The model is on your machine — no authentication needed.

The structure of what you send is the same either way. Only the destination changes.

One rule about API keys — never break this:

Never hardcode your API key directly in your source code. The moment you push that code to GitHub — even a private repo — the key is at risk. Always store it in an environment variable and read it from there.

// Never do this
String apiKey = "AIzaSyD-xxxxxxxxxxxxxxxxxxxxxxxx";

// Always do this
String apiKey = System.getenv("GEMINI_API_KEY");

This is not optional. API keys get scraped from public repos within minutes. Some providers will suspend your account automatically if they detect a leaked key.

What you actually send

Now that you know where the model lives and how you connect — what do you actually put in the request?

This is where most people expect something complicated. It's not.

Every request to an LLM is a list of messages. Each message has two things: who sent it, and what they said.

There are three senders — three roles:

system — you, the developer, giving the model its instructions
user — the person using your app, asking a question
assistant — the model's previous replies

A real request looks like this:

{
  "messages": [
    { "role": "system",    "content": "You are a travel assistant for India. Only help with travel questions." },
    { "role": "user",      "content": "What are the best places to visit in Rajasthan?" },
    { "role": "assistant", "content": "Rajasthan has many beautiful destinations — Jaipur, Jodhpur, Udaipur..." },
    { "role": "user",      "content": "Which one is best for a 3-day trip?" }
  ]
}

The model reads this list top to bottom and writes the next assistant message.

That's it. Three roles, in a list, sent on every call.

The system message — your instructions to the model

The system message is where you tell the model who it is and what it should do. It's the first thing the model reads, and it sets the frame for everything else.

Without a system message, the model behaves like a general-purpose assistant. With one, it becomes your travel assistant, your coding helper, your customer support agent — whatever you define.

A few things I learned about writing them:

Be specific, not general. "Be helpful" tells the model nothing. "Keep all responses under 5 sentences and always suggest at least one specific hotel" gives the model something concrete to follow.

Put critical rules at the top. The model pays more attention to what comes first. If there's one rule you really need followed — put it first.

Keep it short. I once wrote a 400-word system prompt. The model started contradicting itself. Five focused sentences beat twenty vague paragraphs.

The conversation history — why the model "remembers"

Remember from earlier articles — the model is stateless. No memory between calls.

So how does a chatbot remember that your name is Sham from three messages ago?

You sent it. Every time the user sends a new message, you include all the previous messages too. The model sees the full conversation and can answer in context.

By message 5, your request looks like:

[
  { "role": "system",    "content": "You are a travel assistant." },
  { "role": "user",      "content": "Hi, I'm Sham." },
  { "role": "assistant", "content": "Hi Sham! Where are you planning to travel?" },
  { "role": "user",      "content": "I want to go to Goa." },
  { "role": "assistant", "content": "Great choice! When are you planning to go?" },
  { "role": "user",      "content": "Next month. Any hotel recommendations?" }
]

The model knows your name, knows you're going to Goa, knows when — all because you sent the history.

And because history = tokens, you manage how much you send. In production, most apps keep the last 10-15 messages. Beyond that, old messages eat tokens without adding much value.

Temperature — one setting worth knowing now

Along with messages, you pass a few settings. The one you'll tune most is temperature.

It controls how creative or consistent the model is with its responses.

Imagine a dial:

Turn it to 0 — the model always picks the safest, most expected word. Consistent. Predictable. Same input, same output every time.
Turn it to 1 — the model makes more adventurous word choices. More creative. More varied.

Use 0 when you need structure — generating SQL, extracting data, filling templates.
Use 0.5–0.7 for natural conversations — enough variation to sound human, enough consistency to be reliable.

I left temperature at default for my database agent and got slightly different SQL queries for the same question on different runs. Setting it to 0 fixed it immediately.

When something goes wrong — how to debug

When the model ignores your instructions or gives a strange response, there is one move that fixes 90% of issues:

Log the full messages array. Read it as if you are the model.

Most of the time the bug is right there. A user message that contradicts the system prompt. A history so long the instructions are buried. A missing piece of context that makes the model guess.

The model isn't broken. It responded to exactly what you sent it. Reading the full payload shows you what you accidentally told it to do.

Build this habit from day one. It saves hours.

What's next

You now understand what goes into every LLM call — where the model lives, how you connect, and the structure of what you send. The concepts are the same whether you use Gemini, GPT, Claude, or a local Ollama model.

Time to write actual code. Next up: Spring AI — what it gives you over plain HTTP, and how to build your first chat endpoint in Spring Boot.

If you're just starting out — try Ollama first. Free, no API key, runs on your machine. Get comfortable with the concepts, then move to a cloud model when you're ready to build something real. Drop in the comments which one you picked and what you're building.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

What Is a Token? The Concept That Unlocks Everything About LLMs

Sham Prakash K — Thu, 17 Sep 2026 14:30:00 +0000

When I first heard the word "token" in the context of LLMs, I assumed it meant words. Made sense — "I love cricket" is 3 words, so 3 tokens, right?

Wrong. And that misunderstanding cost me real money before I figured out why.

What is a token — really

Let's start from the beginning.

A model doesn't read text the way you do. Before it processes a single character, it breaks your text into pieces called tokens. These pieces are not words. They are not characters. They are something in between — and the exact split depends on a vocabulary the model was trained with.

Let me show you with a real example. Take this sentence:

"I love cricket"

You might expect: 3 tokens (one per word). Here's what actually happens:

Text	Tokens
`I`	1 token
`love`	1 token (note: the space is part of the token)
`cricket`	1 token

Okay, so 3 tokens here. Your instinct was right this time. But try this:

"Microservices architecture"

Text	Tokens
`Micro`	1 token
`services`	1 token
`architecture`	1 token

That's 3 tokens for 2 words. The model split "Microservices" into two pieces because it learned that Micro and services are more common building blocks than Microservices as a whole.

Now try a number:

"2024"

That's 1 token. But:

"20241215"

That might be 3-4 tokens depending on the model. Long numbers get split unpredictably.

And emojis? One emoji can be 2-4 tokens. A single 🚀 costs more tokens than the word "rocket."

How tokenisation actually works

Every LLM uses something called a tokenizer — a fixed vocabulary of text pieces, built during training. GPT models use a tokenizer called BPE (Byte Pair Encoding). Gemini has its own. Claude has its own.

Here's how it works conceptually:

The tokenizer has a vocabulary of ~50,000 to ~100,000 text pieces. Common words like "the", "is", "and" are single tokens. Less common words get split into smaller pieces. Very rare words or made-up words get broken down to almost character level.

This means:

Common English words → usually 1 token
Long or technical words → 2 or more tokens
Code → often more tokens than equivalent English (special characters, indentation)
Non-English languages → often 2-3x more tokens than English for the same meaning
Emojis, special characters → unpredictable, often expensive

Is it different for GPT vs Gemini vs Claude?

Yes. Each model has its own tokenizer with its own vocabulary. The same text will produce a different token count on GPT vs Gemini vs Claude. Not dramatically different — but different. If you're comparing costs across models, you can't just copy one model's token count to another.

Why this matters — the context window

Now that you know what a token is, here's why it matters.

Every LLM has a context window — a hard limit on how many tokens it can process in a single call. Think of it like RAM. The model can only "see" and work with whatever fits inside that window at once.

When you make an API call, you're not just sending the user's message. You're sending:

Your system prompt ("You are a helpful assistant that only answers about...")
The conversation history (every message back and forth so far)
Any documents you want the model to read and use
The user's current message

All of that together must fit within the context window. In tokens.

Let's put real numbers on this:

What you're sending	Approximate tokens
System prompt	300 – 800
Last 10 messages of conversation	1,500 – 3,000
3 retrieved document chunks (RAG)	1,500 – 3,000
User's current message	20 – 100
Total	~3,300 – 6,900

Gemini Flash has a 1 million token context window. So 6,900 tokens seems fine — and it is, for one call. But you pay for every token. And if you're not careful about what you send, those numbers grow fast.

What happens when you exceed the context window

The model doesn't silently ignore the extra text. It throws an error. Your API call fails.

But here's the sneaky part — you often don't hit the hard limit. Instead, you approach it gradually, and the model's quality degrades before you ever get an error.

Before we get into that, let's define something you'll hear constantly when building AI systems: chunks.

Imagine you have a 50-page product manual as a text file. You can't send all 50 pages to the model every time a user asks a question — that's too many tokens, too slow, too expensive. So instead, you split that document into smaller pieces. Each piece is a chunk. Maybe 200-300 words each. You store all those chunks, and when a user asks something, you find the 3-5 chunks most relevant to their question and send only those to the model.

That's what a chunk is — a small slice of a larger document, sized to fit comfortably inside the context window alongside everything else you're sending.

Now, back to the problem.

Researchers found something called the "lost in the middle" problem. Picture this: you send the model a long prompt — system instructions at the top, then 20 document chunks in the middle, then the user's question at the bottom. The model reads all of it. But it turns out models pay more attention to what's at the very beginning and the very end of the input. The stuff buried deep in the middle? It gets less attention.

Think of it like reading a very long email. You remember the opening line and the closing ask. The three paragraphs in the middle? Fuzzy.

So if you send 20 chunks hoping the model finds the right answer somewhere in them — it probably won't. The answer sitting in chunk 11 might as well not exist.

More context is not always better. 3 highly relevant chunks will give you a better answer than 20 loosely relevant ones. Quality of what you send matters more than quantity.

Code costs more tokens than English

This one surprises a lot of backend engineers.

If you're building a system that processes code — code review, documentation generation, code explanation — be aware that code is significantly more expensive in tokens than regular English.

Why? Because code has a lot of characters that aren't common in English — braces, semicolons, indentation spaces, underscores, camelCase names. The tokenizer wasn't primarily trained on code, so it breaks these down into smaller pieces.

A rough comparison:

Content type	Words	Approximate tokens
Plain English	100	~75
Java code	100 "words"	~150-200
JSON payload	100 "words"	~120-160
SQL query	100 "words"	~100-130

So if you're sending a 500-line Java file to the model for review, you're looking at significantly more tokens than you'd expect from the line count alone. Factor this in when designing systems that handle code.

Non-English languages cost more too

If you're building for users who write in Hindi, Tamil, Arabic, Chinese, or most non-English languages — tokens will cost more.

The reason is the same: the tokenizer vocabulary was built primarily from English text. English words map efficiently to tokens. Non-English scripts — especially those with their own character sets — break down into many smaller pieces.

Hindi text can cost 2-3x more tokens than the equivalent English meaning. This matters if you're building a multilingual product and estimating API costs.

How to actually count tokens in your code

Don't guess. Measure.

Most SDKs give you a way to count tokens before making the call. In Spring AI with Gemini, you can log token usage from the response:

ChatResponse response = chatClient.prompt()
    .user(userMessage)
    .call()
    .chatResponse();

// Token usage is in the metadata
Usage usage = response.getMetadata().getUsage();
log.info("Input tokens: {}, Output tokens: {}", 
    usage.getPromptTokens(), 
    usage.getGenerationTokens());

Log this for every call in development. You'll immediately see where your tokens are going. It's the same as profiling slow SQL — you can't optimise what you haven't measured.

One number to remember

If you remember nothing else from this article:

1,000 tokens ≈ 750 words ≈ a page and a half of text.

Every time you construct a prompt, think in pages. If your prompt is 5,000 tokens, you're handing the model 7-8 pages of text to read before it writes a single word back to you. Is all of that necessary? That's the question to ask.

What's next

We've covered what a token is and how the context window works. Next: what does the full payload you send to the model actually look like? System prompt, conversation history, user message — how are these structured, and how does the model use them? That's what the next article covers.

Still confused about how tokenisation works for a specific case? Drop it in the comments — happy to break it down.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

What Is an LLM? The Foundation Every AI Backend Engineer Needs

Sham Prakash K — Tue, 15 Sep 2026 13:39:05 +0000

Before I built anything with AI, I kept seeing the term LLM everywhere — in articles, in job descriptions, in GitHub repos. I nodded along like I understood it. I didn't. Not really.

I knew it stood for Large Language Model. I knew ChatGPT was one. But when someone asked me "how does it actually work" — I couldn't explain it. I had the label, not the understanding.

This article is the explanation I wish I had before I started building.

What does "Large Language Model" actually mean

Let's break the name down.

Language — it works with language. Text in, text out. It reads your words and writes words back.

Model — it is a mathematical model. Millions (actually billions) of numbers, carefully calculated during training, that together capture patterns in language.

Large — the model is enormous. GPT-4 has an estimated 1.8 trillion parameters. Gemini, Claude, Llama — all in the hundreds of billions. These numbers are what make them capable. The "large" isn't marketing — it's what separates these models from earlier, simpler ones.

So: a Large Language Model is a very large mathematical model trained to understand and generate language.

You've already used one. When you typed something into ChatGPT, you used GPT-4o — an LLM made by OpenAI. When you used Gemini on Google, that's an LLM. Claude (made by Anthropic), Llama (made by Meta, open source), Mistral — all LLMs. Different companies, different training data, different sizes — but the same core idea.

That's the definition. But definitions don't give you intuition. Let's go deeper.

What is the model actually doing

Here is the single most important thing to understand about how an LLM works:

It is predicting the next word.

That's it. That's the core task. You give it text, it predicts what word (technically token, but we'll get to that in the next article) comes next. Then it predicts the next one. Then the next. It keeps going until it decides to stop.

Let me show you what I mean. Say you type:

"The capital of France is"

The model looks at those five words and predicts what comes next. "Paris" is overwhelmingly the most likely next word based on everything it learned during training. So it outputs "Paris."

Now it has: "The capital of France is Paris"

It predicts what comes next again. Maybe a period. Maybe "and" followed by something else. It continues predicting, one token at a time, until the response is complete.

This is why LLMs can write code, answer questions, summarise documents, translate languages — all with the same underlying mechanism. It is always just: given this text, what comes next?

How does it "know" things

The model didn't come with knowledge pre-loaded. It learned by reading.

During training, the model was fed an enormous amount of text — web pages, books, code repositories, research papers, articles. Trillions of words. For each piece of text, it practiced the same task: predict the next word. When it got it wrong, the training process adjusted its internal numbers slightly to do better next time. This happened billions of times.

After training, those billions of numbers encode patterns from all that text. When you ask it "what is the capital of France" — it doesn't look it up in a database. It pattern-matches against what it saw during training and produces the most likely answer.

This is why it feels like the model "knows" things. It has absorbed patterns from an enormous amount of human writing. But it's not retrieving facts — it's generating text that fits the pattern.

Why it gets things wrong — hallucination

Here's the uncomfortable part.

Because the model is predicting text rather than retrieving facts, it can generate text that sounds completely confident and is completely wrong.

Ask it about a paper that doesn't exist — it might describe one in detail, with a made-up author and made-up conclusions. Ask it a question whose answer wasn't well-represented in its training data — it'll give you something that sounds right but isn't.

This is called hallucination. The model isn't lying. It's doing exactly what it was trained to do — predict plausible text. But "plausible" and "true" are not the same thing.

For a backend engineer, this is critical to understand because it shapes every architectural decision you make:

You can't trust the model to know your company's internal data — it was never in the training set
You can't trust it for real-time information — training has a cutoff date
You can't trust it for precise facts without verification

This is exactly why RAG (Retrieval Augmented Generation) exists — you give the model the facts it needs in the prompt, so it's not guessing from training memory. But that's a later article. For now, just internalise this: the model generates, it doesn't retrieve.

What the model is not

These comparisons help me think clearly when designing systems:

Not a database. You can't query it for a specific record. It doesn't store facts — it stores patterns. You can't ask it "what did user 123 order last week" and expect an answer unless you tell it.

Not a search engine. A search engine finds documents that match your query. An LLM generates new text. Completely different mechanisms.

Not a person. It has no opinions, no emotions, no beliefs. When it says "I think" or "I feel" — that's pattern matching on how humans write, not an actual inner state. It produces text that sounds like a person because it learned from text written by people.

Not always right. Even when it sounds certain. Especially when it sounds certain.

Why this matters for what we're building

Everything we build in this series — chat apps, RAG pipelines, AI agents — sits on top of this one thing: a model that predicts text, one token at a time, based on everything you give it in the prompt.

Understanding this changes how you design:

You know you need to give it facts explicitly, not assume it knows them
You know the quality of your output depends heavily on the quality of your input
You know it has no state, no memory, no awareness beyond what you send in a single call

The model is a very powerful, very fast text predictor. Your job as a backend engineer is to engineer what it predicts — by carefully constructing what you give it.

That's the whole game.

What's next

Now that you know what an LLM is and how it works, the next question is: how does it measure and process the text you give it? That's where tokens and context windows come in — and that's exactly what the next article covers.

What surprised you most about how LLMs actually work? Drop it in the comments.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.

The Wake-Up Call: Why I Decided to Become an AI Backend Engineer

Sham Prakash K — Sun, 13 Sep 2026 17:44:49 +0000

A year ago, I was the senior engineer on the team. Today, a junior with Copilot can match my output on most tasks. Here's what I did about it.

A year ago I was doing well. Four years into backend engineering — Java, Spring Boot, GraphQL, migrating monoliths to microservices, distributed systems handling 30 million requests a day with p95 latency under 10 milliseconds. I had led a team of five engineers, mentored over ten juniors, been handed performance awards consistently. By every measure I was growing.

Then AI coding tools arrived. And something shifted — not dramatically, not overnight, but quietly and persistently — and I started feeling like the ground under my feet was slowly moving.

This article is about that feeling, what I did about it, and the series that came out of it.

The shift I noticed but couldn't name at first

It didn't hit me as a single moment. It was a slow accumulation.

Watching a junior engineer use GitHub Copilot to scaffold a Spring Boot controller in 45 seconds that would have taken me 10 minutes. Watching Claude Code refactor a service I had spent a week on — restructure the layers, rename the methods, fix the edge cases — in a single session. Seeing that the boilerplate I had spent years internalising, the patterns I knew instinctively, the "senior engineer intuition" I had built up — a lot of it was now in the model's weights.

The fear most people talk about is "AI will replace developers." That's not what I felt. What I felt was more specific and more uncomfortable:

The experience gap between a senior and a junior engineer just compressed dramatically. The things that took years to get fast at — scaffolding, structuring, pattern recognition — were being automated. And that trend wasn't going to reverse.

I was still doing well. The systems I built were still running, the latency numbers were still good, the team was delivering. But I could see that doing the same thing for the next five years was a different bet than it had been five years ago.

The answers that didn't fit

"Learn prompt engineering."
Real skill, thin skill. It optimises how you talk to AI. The better models get at understanding natural language, the less prompt engineering matters. It doesn't compound the way technical depth does.

"Learn ML, become a data scientist."
Genuine option for some people. Not right for me. I like distributed systems, API design, Java, production backend work. Pivoting to Python, model training, Jupyter notebooks — that's not an upgrade, that's a restart from zero in a field where people have PhDs.

"Keep being a backend engineer, just use AI tools."
The most tempting option, and the one that felt like denial. Using AI tools well is already table stakes by 2026. Every backend engineer uses Copilot or Claude Code. That's not a differentiator.

The question that changed the framing

At some point I stopped asking "how do I survive AI?" and asked a different question:

"What does the backend of an AI-powered system actually look like? Who builds that?"

Every AI product you use — every chatbot, every AI-powered search, every document Q&A tool — runs on a backend. Something stores the documents and retrieves them semantically. Something routes the user's question to the right context. Something decides which tool the AI should call. Something tracks tokens and cost. Something enforces guardrails that stop the model from leaking sensitive data.

That backend is not built by a data scientist. Data scientists train models. They don't design production Java APIs, tune connection pools, build MCP servers in Spring Boot, or implement rate limiting per user session.

The person who builds AI backend infrastructure is a backend engineer who understands how AI systems work — not at the model level, but at the infrastructure layer. The retrieval pipelines. The agent loops. The tool call protocols. The observability, the guardrails, the cost controls.

This is called AI Backend Engineering. In 2026, the supply of people who can do it is genuinely small. Most backend engineers haven't built it. Most AI engineers don't have the production systems background. The intersection is narrow — and that's exactly where I decided to position myself.

What I actually built

Over several months — while working full-time — I built a complete AI backend system from scratch. Not a tutorial project. A real multi-component system:

gemini-chat — Spring Boot backend with:

Gemini API integration (plain HTTP first, then Spring AI — both approaches documented)
Full RAG pipeline: chunking, embeddings, Pinecone vector search
AI agent with ReAct loop and real tool use
Database agent: natural language → SQL with safety enforcement
MCP client connecting to the knowledge server
Token tracking, cost monitoring, input and output guardrails

gemini-knowledge-mcp-server — Standalone MCP server exposing 5 tools: listTables, getTableSchema, executeQuery, askDocuments, ingestDocument. LLM-agnostic — swap Gemini for Claude, the server doesn't change.

Everything deployed on Render with Docker. Vector store: Pinecone. Database: Neon PostgreSQL.

Why this series is different

Most AI tutorials are written in Python, use OpenAI, and are written by people who work at AI companies. They show the happy path.

I'm a Java backend engineer who built this on the side. I hit Spring AI API changes between minor versions that broke my code. I debugged Pinecone dimension mismatches that silently returned wrong results with no error. I dealt with a chunking algorithm that looked correct but produced one chunk from a 500-word document. I got an 82-second response latency from the LLM and had to trace it back to 4,715 tokens of bloated context.

None of that is in the tutorials. All of it is in this series.

Each article covers one concept, one implementation, and the real mistakes I made. The mistakes are not edited out — they are the content.

Who this is for

Backend engineers — Java, Spring Boot, any JVM language — who feel that same quiet unease and want a concrete path into AI backend work.

You don't need ML knowledge. You don't need Python. You need to know how to build production APIs and be willing to learn how AI systems work at the infrastructure layer.

Upcoming articles covering:
Tokens, embeddings, RAG pipelines, Spring AI internals, AI agents, MCP servers in Java, observability, and production guardrails.

What's next

Next up: how LLMs actually work — not the math, not the diagrams, just the mental model that makes every other concept in this series click. If you've ever wondered what a token is, why context size matters, or why the model sometimes "forgets" things mid-conversation — that's the next one.

Are you a backend engineer feeling the same shift? What's your plan? Drop it in the comments — I'm curious what other Java devs are doing.

Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.