DEV Community: Ayush Shrivastava

Model Context Protocol with Spring AI, Building MCP Clients and Servers in Java

Ayush Shrivastava — Sat, 19 Sep 2026 07:37:01 +0000

Model Context Protocol with Spring AI: Building MCP Clients and Servers in Java

In the previous article, we explored how to build AI agents with Spring AI using:

LLMs
 ↓
RAG
 ↓
Tool Calling
 ↓
Memory
 ↓
Agent Workflows

Tool calling gives an AI application the ability to interact with external capabilities.

But another problem appears as AI systems become larger.

Imagine you have:

Customer Service Agent
        ↓
Order APIs
Payment APIs
CRM APIs
Knowledge Base
Email Service

And another application has:

Sales Agent
        ↓
CRM
Calendar
Email
Customer Database

And another has:

Developer Agent
        ↓
Git Repository
Issue Tracker
CI/CD
Documentation

If every AI application implements every integration differently, the architecture quickly becomes difficult to maintain.

This is where Model Context Protocol (MCP) becomes interesting.

MCP provides a standardized way for AI applications to interact with external tools and resources. Spring AI provides support for both building MCP servers and consuming MCP servers from Spring Boot applications.

In this article, we'll build a mental model for MCP and explore how Java developers can use it with Spring AI.

What Is MCP?

MCP stands for:

Model Context Protocol

At a high level, MCP standardizes how an AI application communicates with external capabilities such as:

Tools
Resources
Prompts

Instead of every AI application inventing its own integration mechanism:

AI Application
 ↓
Custom Tool Integration
 ↓
CRM

we can have:

AI Application
 ↓
MCP Client
 ↓
MCP Protocol
 ↓
MCP Server
 ↓
CRM

The MCP server exposes capabilities through a standardized interface.

The AI application doesn't need to understand every internal implementation detail of the external system.

Why MCP Exists

Suppose you build an AI assistant that needs access to:

GitHub
Slack
PostgreSQL
Google Calendar
Internal APIs
File Systems

Without a standard protocol, your application might contain:

GitHub Integration
Slack Integration
PostgreSQL Integration
Calendar Integration
Internal API Integration

Each integration may have its own:

Authentication
Tool Schema
Request Format
Response Format
Connection Management
Error Handling

Now imagine another AI application needs the same capabilities.

You may end up rebuilding many of the same integrations.

MCP addresses this by creating a common protocol for AI applications and external servers.

Conceptually:

                    AI Application
                         │
                    MCP Client
                         │
                MCP Protocol
                         │
       ┌─────────────────┼─────────────────┐
       ↓                 ↓                 ↓
  MCP Server         MCP Server         MCP Server
       ↓                 ↓                 ↓
     CRM              GitHub            Database

This is one of the main ideas behind MCP.

MCP Is Not an LLM

This distinction is important.

MCP is not:

An AI model

It is a protocol for connecting AI applications with capabilities.

Think of the stack like this:

LLM
 ↓
AI Application
 ↓
MCP Client
 ↓
MCP Protocol
 ↓
MCP Server
 ↓
Tools / Resources
 ↓
External System

The LLM performs reasoning.

The MCP layer provides standardized communication.

The external system performs the actual operation.

MCP Client vs MCP Server

MCP introduces two important roles.

MCP Client

The MCP client lives inside the AI application.

Its responsibility is to connect to MCP servers and interact with the capabilities they expose.

For example:

Spring Boot AI Application
        ↓
MCP Client
        ↓
Weather MCP Server

The client can discover and use the server's available capabilities.

MCP Server

The MCP server exposes capabilities.

For example:

Weather MCP Server

Tools:
getWeather()
getForecast()

Resources:
weather://cities

Prompts:
weather-analysis

The server is responsible for implementing those capabilities.

Spring AI provides Boot starters and APIs for both sides of this architecture.

The Basic MCP Architecture

A simplified architecture looks like:

                    User
                     ↓
                 Spring Boot
                     ↓
                  ChatClient
                     ↓
                  MCP Client
                     ↓
                MCP Protocol
                     ↓
                MCP Server
                     ↓
                   Tool
                     ↓
                External API

For example:

User:
What's the weather in Paris?

The AI application can discover a weather tool exposed by an MCP server.

The flow becomes:

User
 ↓
LLM
 ↓
MCP Tool
 ↓
Weather MCP Server
 ↓
Weather API
 ↓
Tool Result
 ↓
LLM
 ↓
Final Answer

MCP and Traditional Tool Calling

At this point, you might ask:

"Isn't this just tool calling?"

There is an important distinction.

Traditional Spring AI tool calling can expose application methods directly:

@Tool
public String getWeather(String city) {
    return weatherService.getWeather(city);
}

Your application owns the tool.

With MCP:

AI Application
      ↓
MCP Client
      ↓
Remote MCP Server
      ↓
Tool

The tool can live outside the application.

This creates a cleaner separation between:

AI Application

and:

Capability Provider

Spring AI integrates MCP tools into its tool-calling architecture, allowing applications to consume tools exposed by MCP servers.

MCP Tools

One of the most important MCP capabilities is the tool.

A tool represents an action that an AI application can invoke.

For example:

getWeather()
createTicket()
searchCustomers()
getOrder()
sendEmail()

A weather server might expose:

getTemperature(city)

A CRM server might expose:

findCustomer(email)
createLead(customer)
updateLead(leadId)

A developer server might expose:

searchRepository(query)
getBuildStatus()
createIssue(title)

The MCP client can discover these tools and make them available to the AI application.

MCP Resources

MCP is not limited to actions.

It can also expose resources.

A resource represents information that an MCP client can access.

For example:

customer://123
order://ORD-10291
file://README.md
database://schema

Think of the distinction as:

Tool
=
Do something

Resource
=
Access something

For example:

Tool:
createTicket()

Resource:
customer://123

A server can expose both.

MCP Prompts

MCP also supports prompts.

A server can provide reusable prompt templates for specific tasks.

For example:

Prompt:
analyze-customer

Input:
customerId

Or:

Prompt:
summarize-order

Input:
orderId

This allows prompt templates to become part of the server-provided capabilities rather than being hardcoded independently in every client.

Spring AI's MCP support includes annotations for tools, resources, and prompts.

Building an MCP Server with Spring AI

Let's build a simple MCP server.

Imagine a weather service.

Our application already has:

@Service
public class WeatherService {

    public String getTemperature(String city) {
        return "22°C";
    }
}

We can expose this capability through an MCP tool.

With Spring AI's annotation-based MCP support:

@Service
public class WeatherTools {

    @McpTool(description = "Get the current temperature for a city")
    public String getTemperature(
            @McpToolParam(
                description = "City name",
                required = true
            )
            String city) {

        return weatherService.getTemperature(city);
    }
}

The MCP annotation model allows Spring services to expose capabilities as MCP operations.

Creating the MCP Server

For a Spring Boot application, Spring AI provides MCP server starters.

For example:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-mcp-server-webmvc</artifactId>
</dependency>

You can configure the server to use Streamable HTTP:

spring.ai.mcp.server.protocol=STREAMABLE

Spring AI 2.x supports MCP server transports including Streamable HTTP, stateless Streamable HTTP, SSE, and STDIO. Streamable HTTP is the current recommended HTTP transport in Spring AI 2.x, while SSE is deprecated for this use case.

What Happens Inside the MCP Server?

Conceptually:

Spring Boot
     ↓
MCP Server
     ↓
Tool Registry
     ↓
@McpTool
     ↓
WeatherService
     ↓
Weather API

The server exposes the tool through the MCP protocol.

The client doesn't need to know how the weather service works internally.

It only needs to understand:

Tool Name
Description
Input Schema

Building an MCP Client

Now let's create the other side.

Suppose our AI application needs to consume the weather MCP server.

Add the MCP client starter:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-mcp-client</artifactId>
</dependency>

Then configure the MCP server connection.

For example, using Streamable HTTP:

spring:
  ai:
    mcp:
      client:
        streamable-http:
          connections:
            weather-server:
              url: http://localhost:8080

Spring AI can connect to the configured MCP server and discover its tools.

Connecting MCP Tools to ChatClient

Once the MCP client discovers the server's tools, those tools can be integrated into Spring AI's tool-calling architecture.

Conceptually:

@Bean
CommandLineRunner demo(
        ChatClient chatClient,
        ToolCallbackProvider mcpTools) {

    return args -> {

        String response = chatClient
                .prompt("What's the weather in Paris?")
                .tools(mcpTools)
                .call()
                .content();

        System.out.println(response);
    };
}

This is a powerful abstraction.

The application doesn't need to manually implement every weather function.

The MCP server provides the capability.

The MCP client discovers it.

Spring AI makes the discovered tools available to the model.

The flow becomes:

User
 ↓
ChatClient
 ↓
LLM
 ↓
MCP Tool
 ↓
MCP Client
 ↓
MCP Server
 ↓
Weather API
 ↓
Tool Result
 ↓
LLM
 ↓
Answer

Spring AI's current MCP documentation demonstrates this client pattern using ToolCallbackProvider.

MCP Tool Discovery

One of the interesting capabilities of MCP is tool discovery.

Instead of hardcoding:

Tool A
Tool B
Tool C

the client can connect to an MCP server and discover what capabilities it provides.

For example:

MCP Server
 ↓
tools/list
 ↓
getWeather()
getForecast()
searchAlerts()

The AI application can then make these tools available to the model.

This creates a more modular architecture.

Multiple MCP Servers

Now imagine our AI assistant needs multiple capabilities.

We could have:

AI Application
      │
      ├── MCP Client
      │
      ├── Weather Server
      │
      ├── CRM Server
      │
      ├── GitHub Server
      │
      └── Internal API Server

The architecture becomes:

                         AI Agent
                            │
                        MCP Client
                            │
             ┌──────────────┼──────────────┐
             ↓              ↓              ↓
        Weather MCP      CRM MCP       GitHub MCP
             ↓              ↓              ↓
        Weather API       CRM API      GitHub API

The AI application can consume tools from multiple MCP servers.

This is one reason MCP becomes useful as an AI system grows.

MCP + Spring AI Agents

Now connect this to the previous article.

We previously had:

Agent
 ↓
Tools
 ↓
APIs

With MCP, we can move the tools outside the application boundary:

Agent
 ↓
MCP Client
 ↓
MCP Servers
 ├── CRM
 ├── Payments
 ├── Search
 └── Internal APIs

The resulting architecture becomes:

                         User
                           ↓
                       Spring Boot
                           ↓
                        ChatClient
                           ↓
                         Agent
                           ↓
                       MCP Client
                           ↓
             ┌─────────────┼─────────────┐
             ↓             ↓             ↓
          CRM MCP      Payment MCP    Search MCP
             ↓             ↓             ↓
           CRM API     Payment API   Search API

This creates a modular tool ecosystem.

MCP vs Direct Tool Calling

Let's compare the two approaches.

Direct Spring AI Tool

ChatClient
    ↓
@Tool
    ↓
Service
    ↓
Database

Everything lives inside the application.

MCP Tool

ChatClient
    ↓
MCP Client
    ↓
MCP Server
    ↓
Service
    ↓
Database

The capability provider can be separated from the AI application.

This can be useful when:

Multiple AI applications need the same capability
Tools need independent deployment
Teams own different integrations
External systems need standardized AI access
You want a reusable tool ecosystem

MCP Server as a Capability Layer

One useful architectural pattern is:

Business System
       ↓
MCP Server
       ↓
AI Applications

For example:

CRM
 ↓
CRM MCP Server
 ↓
 ├── Sales Agent
 ├── Support Agent
 └── Internal Assistant

Instead of implementing CRM integration separately in every AI application, the MCP server becomes the standardized capability layer.

MCP + RAG

MCP doesn't replace RAG.

They solve different problems.

RAG:

Retrieve relevant knowledge

MCP:

Connect AI applications to external capabilities

You can combine them:

                       Agent
                         ↓
             ┌───────────┼───────────┐
             ↓           ↓           ↓
            RAG        MCP Tools    Memory
             ↓           ↓           ↓
        Vector DB    External APIs  Database

For example:

User:
Can I refund order ORD-10291?

The agent could:

1. MCP → Get order information
2. RAG → Retrieve refund policy
3. Agent → Compare the two
4. Return answer

This gives the model both:

Live Data
+
Business Knowledge

MCP + Memory

Memory can also coexist with MCP.

For example:

User:
Use my preferred delivery address.

Agent:
Which address?

User:
The one I used last time.

The application may use:

Memory
 ↓
Previous Address

while MCP provides:

Order Service
 ↓
Update Delivery Address

The complete flow becomes:

Agent
 ├── Memory
 ├── RAG
 └── MCP
       ├── Orders
       ├── Payments
       └── CRM

This is becoming a much more complete agent architecture.

MCP Transports

MCP supports multiple ways for clients and servers to communicate.

Common options include:

STDIO
SSE
Streamable HTTP
Stateless Streamable HTTP

For local process-based integrations:

AI Application
 ↓
STDIO
 ↓
MCP Server Process

For network-based applications:

AI Application
 ↓
HTTP
 ↓
MCP Server

In Spring AI 2.x, Streamable HTTP is the current HTTP-oriented approach, while SSE has been deprecated in favor of Streamable HTTP.

STDIO vs HTTP

A simple way to think about it:

STDIO

Application
 ↓
Local MCP Process

Useful for local integrations and process-based communication.

Streamable HTTP

Application
 ↓
Network
 ↓
MCP Server

Useful when the MCP server runs as an independent service.

Stateless Streamable HTTP

Client
 ↓
Request
 ↓
Server
 ↓
Response

This can be useful for stateless, cloud-native service architectures.

The right transport depends on deployment and communication requirements.

MCP Security

This is extremely important.

An MCP server may expose powerful capabilities:

readCustomer()
createInvoice()
refundPayment()
deleteUser()

Simply exposing those tools does not make them safe.

Spring AI's MCP server starters do not automatically provide authentication or authorization for network-accessible MCP endpoints. The documentation specifically warns that HTTP-based MCP endpoints need a security boundary before being exposed beyond localhost.

A production architecture should look like:

Client
 ↓
Authentication
 ↓
Authorization
 ↓
MCP Server
 ↓
Tool
 ↓
Business Logic

Not:

Internet
 ↓
MCP Server
 ↓
Dangerous Tool

MCP Tool Authorization

Imagine an MCP server exposes:

getCustomer()
updateCustomer()
deleteCustomer()

Different users should have different capabilities.

For example:

READ
 ↓
getCustomer()

WRITE
 ↓
updateCustomer()

DESTRUCTIVE
 ↓
deleteCustomer()

Your security layer should determine whether the caller is allowed to invoke each capability.

The model should never be considered the authorization layer.

The application must enforce it.

MCP and Multi-Tenant Systems

MCP becomes particularly interesting in SaaS environments.

Suppose:

Tenant A
 ↓
CRM MCP Server

and:

Tenant B
 ↓
CRM MCP Server

The MCP layer must preserve tenant context.

A request might carry:

tenantId
userId
roles
permissions

The server can then enforce:

Authentication
 ↓
Tenant Resolution
 ↓
Authorization
 ↓
Tool Execution
 ↓
Tenant-Scoped Data

This is especially important for tools such as:

searchCustomers()
getInvoices()
searchDocuments()
createTicket()

A model must never be able to use a tool to cross tenant boundaries.

MCP Error Handling

External tools can fail.

For example:

Agent
 ↓
MCP Tool
 ↓
CRM API
 ↓
Timeout

Your application needs controlled failure behavior.

For example:

Tool Failure
 ↓
Capture Error
 ↓
Return Structured Result
 ↓
Agent
 ↓
Retry / Alternative Tool / Final Response

The agent might decide:

CRM unavailable.

Try cached customer information.

Or:

Unable to retrieve the customer's order.
Please try again later.

The important part is that failures should be observable and controlled.

MCP Observability

When MCP is added to an agent architecture, your observability requirements increase.

You may need to track:

MCP Server
MCP Client
Tool Name
Tool Arguments
Request ID
Latency
Status
Errors
Retries
Model Calls
Token Usage

A useful trace could look like:

User Request
    ↓
LLM Call
    ↓
MCP Tool Discovery
    ↓
Tool Call
    ↓
CRM API
    ↓
Tool Result
    ↓
LLM Call
    ↓
Final Response

Without tracing, debugging multi-server agent systems can become difficult.

MCP Doesn't Replace Your Business Logic

This is another important principle.

Suppose you have:

public RefundResult refundPayment(
        String orderId,
        BigDecimal amount) {
    ...
}

You shouldn't move all business logic into an MCP handler.

Instead:

MCP Tool
 ↓
Application Service
 ↓
Business Rules
 ↓
Repository
 ↓
Database

For example:

@McpTool(description = "Refund an eligible order")
public RefundResult refundOrder(String orderId) {

    return refundService.refund(orderId);
}

The MCP layer becomes an interface.

Your existing business service remains responsible for the actual business rules.

This keeps the architecture clean.

MCP as an Integration Boundary

One of the strongest ways to think about MCP is as an integration boundary.

Instead of:

AI
 ↓
Everything

use:

AI
 ↓
MCP
 ↓
Controlled Capabilities

The MCP layer becomes a contract between AI applications and external systems.

For example:

AI Application
      ↓
MCP
      ↓
CRM

or:

AI Application
      ↓
MCP
      ↓
Payment System

or:

AI Application
      ↓
MCP
      ↓
Internal Developer Platform

A Complete Spring AI + MCP Architecture

Now combine everything from this series:

                              User
                                ↓
                         Spring Boot API
                                ↓
                           ChatClient
                                ↓
                              Agent
                                ↓
             ┌──────────────────┼──────────────────┐
             ↓                  ↓                  ↓
           Memory              RAG              MCP Client
             ↓                  ↓                  ↓
         PostgreSQL          pgvector        ┌─────┼─────┐
                                             ↓     ↓     ↓
                                           CRM  GitHub  Search
                                           MCP    MCP     MCP
                                             ↓     ↓     ↓
                                           APIs  APIs   APIs

Around the system:

Authentication
Authorization
Tenant Isolation
Observability
Audit Logging
Rate Limiting
Guardrails
Human Approval

This is a strong foundation for production-oriented AI applications.

When Should You Use MCP?

MCP becomes particularly useful when you have:

Multiple AI applications
        ↓
Shared tools
        ↓
Shared integrations

For example:

Sales Agent
Support Agent
Developer Agent
Internal Assistant

all need access to:

CRM
GitHub
Internal APIs
Documentation

Instead of implementing each integration separately:

Agent A → CRM Integration
Agent B → CRM Integration
Agent C → CRM Integration

you can create:

CRM MCP Server

and allow multiple AI applications to consume it.

When You Don't Need MCP

MCP isn't automatically required for every AI application.

If your application has:

One Agent
 ↓
One Tool
 ↓
One Internal Service

direct Spring AI tool calling may be simpler.

For example:

ChatClient
 ↓
@Tool
 ↓
OrderService

Introducing an MCP server could add unnecessary infrastructure.

A useful rule is:

Use MCP when standardization, reuse, separation, or interoperability provides real value.

Don't introduce another protocol simply because it is popular.

Direct Tools vs MCP

A simple comparison:

Approach	Best suited for
Spring AI `@Tool`	Local application capabilities
MCP	Shared/external capabilities
RAG	Knowledge retrieval
Memory	Conversation context
Agent	Dynamic decision-making

They are not mutually exclusive.

A production system may use all of them:

Agent
 ├── Local Spring AI Tools
 ├── MCP Tools
 ├── RAG
 └── Memory

The Bigger Picture

Our AI architecture has evolved throughout this series.

We started with:

LLM
 ↓
Response

Then:

LLM
 ↓
RAG
 ↓
Knowledge

Then:

LLM
 ↓
Tools
 ↓
Actions

Then:

LLM
 ↓
Tools
 ↓
Memory
 ↓
Agent

And now:

Agent
 ↓
MCP
 ↓
External Capabilities

The architecture is becoming increasingly modular.

A Mental Model for MCP

Remember it this way:

LLM
=
Reason

RAG
=
Retrieve Knowledge

Memory
=
Remember Context

Tool Calling
=
Invoke Capabilities

MCP
=
Standardize Capability Access

Spring Boot
=
Business Application

Agent
=
Coordinate Decisions

Together:

LLM
+
RAG
+
Memory
+
Tools
+
MCP
+
Business Logic
=
Production AI Application

Final Takeaways

MCP gives AI applications a standardized way to interact with external tools and resources.

The key ideas are:

MCP Client connects an AI application to MCP servers.
MCP Server exposes tools, resources, and prompts.
Tools allow actions to be performed.
Resources provide access to information.
Prompts can provide reusable prompt templates.
Spring AI supports MCP clients and servers through Boot starters and annotations.
MCP tools integrate with Spring AI's existing tool-calling architecture.
Streamable HTTP is the current HTTP-oriented transport in Spring AI 2.x.
MCP does not replace your business logic.
Authentication and authorization must be enforced before exposing network-accessible MCP servers.
Multi-tenant applications must preserve tenant isolation across MCP calls.
MCP is particularly useful when capabilities need to be shared across multiple AI applications.
For simple local integrations, direct Spring AI tools may be sufficient.

The architecture can now look like:

                         User
                           ↓
                        Agent
                           ↓
        ┌──────────────────┼──────────────────┐
        ↓                  ↓                  ↓
      Memory              RAG             MCP Client
        ↓                  ↓                  ↓
    Conversation       Vector DB       MCP Servers
                                             ↓
                              ┌──────────────┼──────────────┐
                              ↓              ↓              ↓
                             CRM          GitHub         Internal APIs

The important shift is this:

Before:

AI Application
 ↓
Custom Integrations
 ↓
External Systems

With MCP:

AI Application
 ↓
MCP Client
 ↓
Standardized Protocol
 ↓
MCP Servers
 ↓
External Capabilities

MCP doesn't make your AI application automatically intelligent.

It gives your AI application a standardized way to connect to capabilities.

And when you combine MCP with Spring AI's:

ChatClient
+
Tool Calling
+
RAG
+
Memory
+
Agents

you get a powerful foundation for building modular AI applications in Java.

What's Next?

We've now connected our AI agent to external capabilities.

But another challenge appears:

One Agent
      ↓
Multiple MCP Servers
      ↓
Multiple Tools
      ↓
Multiple Decisions

How do we control which tools an agent can access?

How do we handle permissions?

How do we observe agent behavior?

How do we evaluate whether an agent is making the right decisions?

And how do we build reliable AI workflows instead of simply hoping the model does the right thing?

That takes us into the next stage of AI engineering:

Building Production-Ready AI Agents with Spring AI — Guardrails, Evaluation, Observability, and Human-in-the-Loop Workflows.

Building AI Agents with Spring AI — Tool Calling, Memory, and Autonomous Workflows

Ayush Shrivastava — Mon, 07 Sep 2026 08:41:41 +0000

Building AI Agents with Spring AI — Tool Calling, Memory, and Autonomous Workflows

Large Language Models are excellent at generating text.

But generation alone isn't enough to build truly useful AI applications.

Imagine asking an AI assistant:

What's the status of my order?

A normal LLM can explain how order tracking works.

But it cannot magically access your order database.

Or suppose you ask:

Cancel my order #ORD-10291.

The model can tell you how to cancel an order.

But it cannot actually cancel anything unless your application gives it the ability to perform that action.

This is where tool calling and AI agents come in.

Instead of simply generating an answer, an AI application can:

Understand the request
        ↓
Decide what action is required
        ↓
Select a tool
        ↓
Execute the tool
        ↓
Observe the result
        ↓
Continue reasoning
        ↓
Generate the final response

In this article, we'll explore how to build this architecture using Spring AI.

What Is an AI Agent?

An AI agent is an application where an LLM can decide what actions need to be performed and use available tools to accomplish a goal.

A traditional LLM application looks like:

User
 ↓
Prompt
 ↓
LLM
 ↓
Response

An agent-based application looks more like:

User
 ↓
LLM
 ↓
Decision
 ↓
Tool
 ↓
Result
 ↓
LLM
 ↓
Decision
 ↓
Another Tool
 ↓
Result
 ↓
Final Answer

The important difference is:

The LLM is no longer limited to generating text.

It can interact with the application through controlled capabilities.

Tool Calling

Tool calling allows an LLM to request the execution of a function exposed by your application.

For example, imagine our application provides:

getOrderStatus()
cancelOrder()
getCustomer()
createSupportTicket()

The user asks:

Where is my order?

The model might determine that it needs:

getOrderStatus()

The application executes the function and returns:

Order #10291
Status: Shipped
Expected delivery: September 10

The LLM can then generate:

Your order has been shipped and is expected to arrive on September 10.

The LLM didn't directly access the database.

Instead:

LLM
 ↓
Tool Request
 ↓
Application
 ↓
Database
 ↓
Tool Result
 ↓
LLM
 ↓
Answer

This distinction is extremely important for enterprise applications.

Why Tool Calling Matters

Without tools:

LLM
 ↓
Text

With tools:

LLM
 ↓
Tools
 ├── Database
 ├── REST APIs
 ├── Search
 ├── Payment systems
 ├── CRM
 ├── Internal services
 └── Business workflows

This turns the LLM from a text-generation component into an interface for interacting with your application.

For example, an AI sales assistant could have:

getCustomer()
getCustomerOrders()
createLead()
updateLead()
sendEmail()
scheduleMeeting()

A support agent could have:

searchKnowledgeBase()
getCustomerAccount()
getOrder()
createTicket()
updateTicket()

An internal developer assistant could have:

searchDocumentation()
searchGitRepository()
getBuildStatus()
createIssue()

The possibilities are much broader than simple question answering.

RAG vs Tool Calling

At this point, it is useful to distinguish RAG from tool calling.

RAG is primarily about retrieving information.

Tool calling is about performing actions or retrieving live data through application capabilities.

For example:

RAG
 ↓
Retrieve company documentation
 ↓
Answer question

Tool calling:

LLM
 ↓
Call order API
 ↓
Get live order status
 ↓
Answer

They can also be combined.

For example:

User
 ↓
AI Agent
 ├── RAG → Search company policies
 │
 ├── Tool → Get customer account
 │
 └── Tool → Check order status
          ↓
       LLM
          ↓
       Answer

This combination is extremely powerful.

Spring AI and Tool Calling

Spring AI provides abstractions that make it easier to expose application capabilities to language models.

A simplified architecture looks like:

Spring Boot
     │
     ├── ChatModel
     │
     ├── Tools
     │
     ├── Advisors
     │
     ├── Chat Memory
     │
     └── Vector Store

The application controls which tools are available.

The model decides whether a tool is needed.

This separation is important.

The model should not have unrestricted access to your application.

Instead, the application exposes specific capabilities.

Creating a Tool

Imagine we have an order service.

@Service
public class OrderService {

    public OrderStatus getOrderStatus(String orderId) {
        // Fetch order from database
        return orderRepository.findStatus(orderId);
    }
}

We can expose a controlled method as an AI tool.

Conceptually:

@Tool(
    description = "Get the current status of an order"
)
public OrderStatus getOrderStatus(String orderId) {

    return orderService.getOrderStatus(orderId);
}

The description is important.

The model uses the tool description to understand:

What does this tool do?
When should I use it?
What parameters does it require?

For example:

Tool:

getOrderStatus

Description:
Returns the current shipping and delivery status
for a customer order.

Input:
orderId

The model can then determine whether this tool is appropriate.

Tool Schema

A tool can be thought of as:

Tool Name
     +
Description
     +
Input Schema
     +
Execution Logic

For example:

getOrderStatus(
    orderId: String
)

The model might produce a tool request conceptually like:

{
  "name": "getOrderStatus",
  "arguments": {
    "orderId": "ORD-10291"
  }
}

The application receives the request and executes the corresponding Java method.

The Tool Calling Loop

A typical interaction looks like this:

User
 │
 │ "What's the status of ORD-10291?"
 ↓
LLM
 │
 │ Tool Request
 ↓
getOrderStatus("ORD-10291")
 │
 ↓
Order Service
 │
 ↓
Database
 │
 ↓
Tool Result
 │
 ↓
LLM
 │
 ↓
Final Answer

Notice something important.

The LLM doesn't execute Java code itself.

The application remains responsible for execution.

The model only requests the action.

Tool Calling with ChatClient

Spring AI's ChatClient provides a convenient API for interacting with chat models.

Conceptually:

ChatClient chatClient;

String response = chatClient.prompt()
        .user("What's the status of order ORD-10291?")
        .tools(orderTools)
        .call()
        .content();

The exact APIs may vary depending on the Spring AI version you're using, but the architecture remains the same:

ChatClient
   ↓
ChatModel
   ↓
Tool Selection
   ↓
Tool Execution
   ↓
Tool Result
   ↓
Final Response

Multiple Tools

A real agent usually has more than one tool.

For example:

CustomerAgentTools

├── getCustomer()
├── getCustomerOrders()
├── getOrderStatus()
├── createSupportTicket()
└── updateCustomer()

Now consider this question:

My order is late. Please check the status
and create a support ticket if necessary.

The model might determine:

1. getOrderStatus()
2. Analyze result
3. createSupportTicket()
4. Return final response

The application executes each requested operation.

This is where the concept of an agent starts becoming much more interesting.

Agents and Decision Making

A simple agent loop can be represented as:

              ┌───────────────┐
              │     User      │
              └───────┬───────┘
                      ↓
                ┌───────────┐
                │    LLM    │
                └─────┬─────┘
                      ↓
                Need a Tool?
                /          \
              No            Yes
              ↓              ↓
          Final Answer    Tool Call
                             ↓
                        Tool Execution
                             ↓
                         Tool Result
                             ↓
                            LLM
                             ↓
                      Need Another Tool?

The model can repeatedly interact with tools until it has enough information to produce the final response.

Tool Calling Is Not Full Autonomy

This distinction is important.

People often hear:

AI Agent

and immediately think:

Give AI access to everything
        ↓
Let AI do whatever it wants

That is not how production systems should be designed.

A production agent should operate inside clear boundaries.

For example:

Allowed Tools
     ↓
Authorization
     ↓
Validation
     ↓
Execution
     ↓
Audit

The application remains in control.

The model should not be trusted with unrestricted capabilities.

Tool Security

Imagine we expose:

@Tool
public void deleteCustomer(String customerId) {
    ...
}

This is potentially dangerous.

An LLM should not automatically receive unrestricted permission to perform destructive operations.

Instead, sensitive tools should have additional controls.

For example:

User
 ↓
Authentication
 ↓
Authorization
 ↓
Agent
 ↓
Tool Request
 ↓
Permission Check
 ↓
Confirmation
 ↓
Execution

For destructive operations, you may require explicit user confirmation.

Example:

AI:
I found customer account C-19291.

Deleting this account is irreversible.
Do you want me to continue?

User:
Yes.

AI:
Executing deletion...

The AI should assist with the decision process, not bypass your security model.

Tool Permissions

A useful production architecture is to classify tools.

READ

getCustomer()
getOrder()
searchDocuments()
getInvoice()

Then:

WRITE

createTicket()
updateCustomer()
createLead()

And:

DESTRUCTIVE

deleteCustomer()
cancelSubscription()
refundPayment()

Different permission levels can then be applied.

For example:

READ
→ Automatically allowed

WRITE
→ Role-based authorization

DESTRUCTIVE
→ Authorization + confirmation

This makes agent behavior much safer.

AI Agent Memory

Tool calling solves one problem.

But another problem appears quickly:

What does the agent remember?

Consider this conversation:

User:
My order is late.

AI:
What's your order number?

User:
ORD-10291.

AI:
Let me check it.

Now the next message is:

Can you create a support ticket for it?

The AI needs to understand that:

"it"

refers to:

ORD-10291

This requires conversational context.

That's where chat memory becomes important.

Chat Memory

A simple conversation can be represented as:

User:
My order is late.

Assistant:
What's your order number?

User:
ORD-10291.

Assistant:
Let me check that order.

The application maintains the conversation history.

Conceptually:

Conversation ID
       ↓
Chat Memory
       ↓
Previous Messages
       ↓
Current Prompt
       ↓
LLM

Spring AI provides abstractions for managing chat memory.

Short-Term vs Long-Term Memory

It is useful to distinguish two concepts.

Short-Term Memory

Conversation context.

User:
My order is late.

User:
It's order 10291.

User:
Can you check it?

The system remembers the current conversation.

Long-Term Memory

Persistent information about the user.

For example:

Customer:
Ayush

Preferences:
Preferred language = English
Preferred notification = Email

Long-term memory usually requires persistence in a database or another storage system.

A production architecture might look like:

Conversation
     ↓
Chat Memory Store
     ↓
PostgreSQL / Redis

The exact storage mechanism depends on the application.

Agents + Memory

Now we can combine:

User
 ↓
Agent
 ↓
Memory
 ↓
LLM
 ↓
Tools
 ↓
Tool Results
 ↓
Memory
 ↓
LLM
 ↓
Answer

This enables more natural multi-turn interactions.

Advisors

Another important Spring AI concept is the Advisor.

Advisors can intercept and influence the interaction between the application and the model.

They can be used for concerns such as:

Conversation memory
RAG
Logging
Security
Prompt modification
Context injection
Observability

Conceptually:

User
 ↓
ChatClient
 ↓
Advisor
 ↓
ChatModel
 ↓
Advisor
 ↓
Response

This allows cross-cutting AI behavior to be separated from business logic.

Combining RAG + Tools + Memory

Now things become much more powerful.

Imagine an enterprise support agent.

It has:

RAG
 ↓
Company documentation

Tools:

getCustomer()
getOrder()
createTicket()

Memory:

Conversation history

The architecture becomes:

                    User
                     ↓
                 AI Agent
                     ↓
             ┌───────┼────────┐
             ↓       ↓        ↓
            RAG    Tools    Memory
             ↓       ↓        ↓
        Knowledge   APIs   Conversation
             │       │        │
             └───────┼────────┘
                     ↓
                    LLM
                     ↓
                  Response

This is much closer to a production AI application.

Example: Customer Support Agent

Consider the request:

My payment failed for order ORD-19291.
Can you check what happened and tell me
what I should do?

The agent could perform:

1. getOrder("ORD-19291")
2. getPaymentStatus("ORD-19291")
3. searchKnowledgeBase("payment failure")
4. Generate explanation

The final answer could be:

Your payment attempt failed because the transaction
was declined by the payment provider.

According to the payment policy, you can retry the
payment using another payment method.

Would you like me to create a support ticket?

The model combined:

Live application data
+
Knowledge base
+
Conversation context

This is significantly more useful than a standalone chatbot.

Multi-Step Agent Workflow

Agents can also perform multi-step workflows.

For example:

User:
Find my overdue invoices and send reminders.

The agent could reason through:

getCustomer()
      ↓
getInvoices()
      ↓
Filter overdue invoices
      ↓
sendReminder()
      ↓
Return summary

The workflow becomes:

Goal
 ↓
Plan
 ↓
Tool
 ↓
Observe
 ↓
Next Decision
 ↓
Tool
 ↓
Observe
 ↓
Final Result

This pattern is often called an agent loop.

Agent Loop

A simplified conceptual implementation looks like:

while (!completed) {

    AgentDecision decision =
            llm.decide(context);

    if (decision.requiresTool()) {

        ToolResult result =
                toolExecutor.execute(
                        decision.toolCall()
                );

        context.add(result);

    } else {

        return decision.finalAnswer();
    }
}

In real applications, frameworks handle much of this interaction.

But understanding the underlying loop is important.

Don't Build Everything as an Agent

An important engineering lesson:

Not every AI feature needs an agent.

If your workflow is deterministic:

Validate request
 ↓
Call API
 ↓
Save result
 ↓
Return response

you probably don't need an autonomous agent.

A normal service workflow may be better.

Agents become more useful when:

The next step depends on the current result.

For example:

Check order
 ↓
If delayed
 ↓
Check refund policy
 ↓
If eligible
 ↓
Ask for confirmation
 ↓
Create refund request

The dynamic decision-making is where agents become valuable.

Deterministic Workflow vs Agent

Traditional Workflow

A → B → C → D

Everything is predetermined.

Agent Workflow

A
 ↓
LLM decides
 ├── B
 ├── C
 └── D
      ↓
   Observe result
      ↓
   Decide again

Agents provide flexibility.

Traditional workflows provide predictability.

Production systems often use both.

Agent Architecture for Enterprise Java

A practical Spring Boot architecture might look like:

                    ┌───────────────┐
                    │   Frontend    │
                    └───────┬───────┘
                            ↓
                    ┌───────────────┐
                    │ Spring Boot   │
                    │     API       │
                    └───────┬───────┘
                            ↓
                     ┌────────────┐
                     │ ChatClient │
                     └─────┬──────┘
                           ↓
                    ┌──────────────┐
                    │    Agent     │
                    └──────┬───────┘
                           ↓
              ┌────────────┼────────────┐
              ↓            ↓            ↓
           Memory         RAG         Tools
              ↓            ↓            ↓
          PostgreSQL    pgvector      APIs
                                         ↓
                                    Microservices

This architecture fits naturally into existing Spring Boot applications.

Observability

Agent systems can become difficult to debug.

Imagine an agent performs:

Tool 1
Tool 2
Tool 3
Tool 4

and the final response is incorrect.

You need to know:

What did the model decide?
Which tools were selected?
What arguments were sent?
How long did each tool take?
What did each tool return?
How many model calls happened?
How many tokens were consumed?

Therefore, observability is critical.

Track:

LLM latency
Tool latency
Retrieval latency
Token usage
Tool calls
Tool failures
Model responses
Agent iterations
Errors

Preventing Infinite Agent Loops

An agent can potentially continue calling tools indefinitely.

For example:

LLM
 ↓
Tool
 ↓
LLM
 ↓
Tool
 ↓
LLM
 ↓
Tool
 ↓
...

Production systems should enforce limits.

For example:

Maximum iterations = 10
Maximum tool calls = 20
Maximum execution time = 30 seconds

You should also define clear failure behavior.

Agent limit reached
        ↓
Stop execution
        ↓
Return safe response
        ↓
Log failure

Tool Validation

Never blindly trust model-generated tool arguments.

Suppose the model requests:

{
  "orderId": "ORD-999999999"
}

Your application should still validate:

Does the order exist?
Does the user own the order?
Is the user authorized?
Is the order accessible to this tenant?

The architecture should be:

LLM
 ↓
Tool Request
 ↓
Schema Validation
 ↓
Authorization
 ↓
Business Validation
 ↓
Tool Execution

The LLM is not your security boundary.

Your application is.

Multi-Tenant AI Agents

This becomes especially important in SaaS applications.

Imagine:

Tenant A
 ├── Customers
 ├── Orders
 └── Documents

Tenant B
 ├── Customers
 ├── Orders
 └── Documents

An AI agent must never retrieve Tenant B's information while processing a Tenant A request.

Every tool and retrieval operation should carry tenant context.

For example:

tenant_id
user_id
roles
permissions

Then:

User
 ↓
Authentication
 ↓
Tenant Context
 ↓
Agent
 ↓
Tool
 ↓
Authorization
 ↓
Tenant-scoped Data

The same principle applies to RAG.

Vector Search
 +
tenant_id filter

should ensure that retrieved documents belong to the correct tenant.

AI Agent Guardrails

Production agents should have explicit guardrails.

Examples:

Input validation
Output validation
Tool authorization
Rate limiting
Token limits
Iteration limits
PII protection
Audit logging
Human approval

For high-risk actions:

Agent
 ↓
Tool Request
 ↓
Risk Evaluation
 ↓
Human Approval
 ↓
Execution

This creates a human-in-the-loop architecture.

Human-in-the-Loop

Not every decision should be fully automated.

For example:

Refund amount < $50
    ↓
Automatic

Refund amount > $50
    ↓
Human approval

Or:

Create support ticket
    ↓
Automatic

Delete account
    ↓
Confirmation required

This gives us a practical balance:

AI Automation
+
Business Rules
+
Human Oversight

RAG + Tool Calling + Memory

At this point, we can combine everything we've discussed.

                         User
                          ↓
                     Spring Boot
                          ↓
                      ChatClient
                          ↓
                       AI Agent
                          ↓
              ┌───────────┼───────────┐
              ↓           ↓           ↓
            Memory       RAG         Tools
              ↓           ↓           ↓
          PostgreSQL   pgvector    REST APIs
                                      ↓
                               Business Services
                                      ↓
                                   Database

This is a strong foundation for enterprise AI applications.

Example Enterprise Agent

Imagine a sales assistant.

The user asks:

Show me the latest opportunities for Acme
and tell me which ones are likely to close this month.

The agent could:

1. getCustomer("Acme")
2. getOpportunities("Acme")
3. retrieve sales documentation
4. analyze opportunity information
5. generate summary

Now the user says:

Create a follow-up task for the highest priority opportunity.

The agent can:

1. Identify opportunity
2. createFollowUpTask()
3. Return task details

This is where AI starts becoming an application interface rather than simply a chatbot.

A Useful Mental Model

Think about the responsibilities this way:

LLM
=
Reasoning + Language

RAG
=
Knowledge Retrieval

Tools
=
Actions + Live Data

Memory
=
Conversation Context

Spring Boot
=
Application + Security + Business Logic

Together:

LLM
 +
RAG
 +
Tools
 +
Memory
 +
Business Logic
 =
AI Application

What Spring AI Gives Java Developers

Spring AI provides abstractions that allow Java developers to work with AI capabilities using familiar Spring patterns.

Important building blocks include:

ChatClient
ChatModel
EmbeddingModel
VectorStore
Document
Advisors
Chat Memory
Tools

This means an enterprise Java team can integrate AI into an existing Spring Boot architecture instead of creating an entirely separate AI stack.

For example:

Existing Spring Boot Application
              ↓
        Spring AI Layer
              ↓
      Model + RAG + Tools
              ↓
     Existing Microservices

This makes AI integration much more practical for Java teams.

Production Architecture

A more complete production system might eventually look like:

                         ┌───────────────┐
                         │     User      │
                         └───────┬───────┘
                                 ↓
                         API Gateway
                                 ↓
                         Authentication
                                 ↓
                         Spring Boot API
                                 ↓
                            AI Agent
                                 ↓
          ┌──────────────────────┼──────────────────────┐
          ↓                      ↓                      ↓
       Memory                  RAG                    Tools
          ↓                      ↓                      ↓
     PostgreSQL              pgvector              Microservices
                                                        ↓
                                                Business Database
                                 ↓
                           LLM Provider
                                 ↓
                              Response

And around the entire system:

Security
Observability
Rate Limiting
Audit Logging
Guardrails
Evaluation

These are not optional concerns in serious enterprise deployments.

Agent vs Chatbot

It is useful to understand the difference.

Chatbot

User
 ↓
LLM
 ↓
Answer

RAG Chatbot

User
 ↓
Retrieve Knowledge
 ↓
LLM
 ↓
Answer

Tool-Enabled Assistant

User
 ↓
LLM
 ↓
Tool
 ↓
Result
 ↓
Answer

AI Agent

User
 ↓
Agent
 ↓
Reason
 ↓
Tool
 ↓
Observe
 ↓
Reason
 ↓
Tool
 ↓
Observe
 ↓
Final Answer

The complexity increases at every stage.

The Agentic AI Stack

We can now think about the evolution of an AI application:

Level 1
LLM
 ↓
Text Generation

Level 2
LLM + RAG
 ↓
Knowledge Retrieval

Level 3
LLM + Tools
 ↓
Actions

Level 4
LLM + Tools + Memory
 ↓
Contextual Assistant

Level 5
LLM + RAG + Tools + Memory
 ↓
Agent

Level 6
Multiple Agents + Workflows
 ↓
Agentic System

This progression is useful when deciding how much complexity your application actually needs.

Final Takeaway

The evolution from traditional AI applications to agentic applications can be summarized as:

LLM
 ↓
Generate Text

RAG
 ↓
Retrieve Knowledge

Tool Calling
 ↓
Take Actions

Memory
 ↓
Remember Context

Agents
 ↓
Make Decisions

Workflows
 ↓
Coordinate Multiple Steps

Spring AI provides Java developers with abstractions for building many of these capabilities inside the Spring ecosystem.

The most important engineering principle is:

Let the model decide, but let your application control.

The LLM can decide which tool may be useful.

Your application should decide whether that tool is actually allowed to execute.

That separation gives us a much safer architecture for enterprise AI.

What's Next?

We've now covered three major capabilities:

LLM
 ↓
Generate

RAG
 ↓
Retrieve

Tools
 ↓
Act

But there is another challenge.

What happens when a system has:

Multiple agents
        ↓
Multiple tools
        ↓
Multiple services
        ↓
Multiple AI models

How do these agents communicate?

How do we standardize tool discovery?

How can an AI agent securely interact with external tools and services?

This leads us toward another important concept in modern AI engineering:

Model Context Protocol — MCP.

In the next article, we'll explore:

Building MCP Clients and Tool-Based AI Applications with Spring AI.

Key Takeaways

Tool calling allows LLMs to interact with application capabilities.
The LLM requests tools; your application executes them.
Tools can expose APIs, databases, business operations, and services.
Chat memory provides conversational context.
RAG provides external knowledge.
Agents can combine RAG, memory, and tools.
Not every workflow requires an agent.
Deterministic workflows are often better for predictable business processes.
Tool permissions and authorization are critical.
Destructive operations should require stronger controls.
Multi-tenant applications must enforce tenant isolation at the tool and retrieval layers.
Agent loops should have iteration, timeout, and tool-call limits.
Observability is essential for debugging agent behavior.
Human-in-the-loop approval is useful for high-risk actions.
Spring AI provides abstractions that make these patterns accessible to Java and Spring Boot developers.

The future of enterprise AI isn't just:

LLM → Answer

It's increasingly:

LLM
 ↓
Reason
 ↓
Retrieve
 ↓
Call Tools
 ↓
Observe
 ↓
Act
 ↓
Remember
 ↓
Complete the Goal

And that's where AI agents with Spring AI become truly interesting.

Building RAG Applications with Spring AI: Connect LLMs to Your Own Data

Ayush Shrivastava — Mon, 31 Aug 2026 13:07:35 +0000

Large Language Models are powerful.

But there is one fundamental limitation:

An LLM doesn't automatically know your application's private data.

Your company policies, product documentation, internal knowledge base, customer records, PDFs, technical documentation, or database content are not necessarily part of the model's training data.

This is where RAG — Retrieval-Augmented Generation comes in.

Instead of asking an LLM to answer directly, we first retrieve relevant information from our own data and provide that information as context to the model.

In this article, we'll build the foundation of a RAG application using Spring AI.

What is RAG?

RAG stands for:

Retrieval-Augmented Generation

The idea is simple:

User Question
      ↓
Retrieve Relevant Information
      ↓
Add Retrieved Context to Prompt
      ↓
LLM
      ↓
Generated Answer

For example, imagine we have a company's internal documentation.

A user asks:

What is our refund policy for annual subscriptions?

Instead of expecting the LLM to magically know the answer, our application:

Searches the company's documents.
Finds the relevant refund-policy content.
Adds that content to the prompt.
Sends the prompt to the LLM.
Generates an answer grounded in the retrieved information.

This is the core idea behind RAG.

Why Do We Need RAG?

A normal LLM application looks like this:

User
  ↓
Application
  ↓
LLM
  ↓
Answer

The problem?

The LLM only has access to the information available to it.

With RAG, the architecture becomes:

                  ┌──────────────┐
                  │  Documents   │
                  └──────┬───────┘
                         ↓
                    Chunking
                         ↓
                    Embeddings
                         ↓
                  Vector Database
                         ↑
                         │
User → Query → Similarity Search
                         ↓
                  Relevant Context
                         ↓
                        LLM
                         ↓
                      Answer

Now the model can work with information from our own knowledge base.

The RAG Pipeline

A production RAG pipeline typically contains these stages:

Documents
    ↓
Document Loading
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector Database
    ↓
Similarity Search
    ↓
Relevant Context
    ↓
Prompt
    ↓
LLM
    ↓
Answer

Let's understand each step.

1. Document Loading

First, we need to get our data into the application.

The source could be:

PDF files
Markdown files
HTML pages
Word documents
Database records
APIs
Knowledge bases
Internal documentation

Spring AI provides abstractions for working with documents.

A document can be represented using Spring AI's Document abstraction.

Conceptually:

Document document = new Document(
    "Spring Boot is a framework for building Java applications..."
);

We now have content that can be processed by our RAG pipeline.

2. Chunking

We shouldn't usually store an entire document as a single vector.

Imagine a 100-page PDF.

A user asks:

How do I configure authentication?

We don't want to retrieve the entire PDF.

Instead, we split the document into smaller pieces called chunks.

For example:

Document
   ↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4
...
Chunk 100

Each chunk represents a smaller piece of knowledge.

A simple example:

Chunk 1:
Introduction to Spring Security

Chunk 2:
Configuring authentication

Chunk 3:
Creating users

Chunk 4:
JWT authentication

Chunk 5:
Role-based authorization

Now when the user asks about JWT authentication, we can retrieve the relevant chunk instead of the entire document.

3. Embeddings

This is where things get interesting.

A computer doesn't understand the semantic meaning of text in the same way humans do.

We need a way to represent text numerically.

That's where embeddings come in.

An embedding model converts text into a vector.

For example:

"How do I configure JWT authentication?"

might become something conceptually like:

[0.12, -0.42, 0.87, 0.31, ...]

The actual vector contains many dimensions.

The important part is:

Semantically similar text produces vectors that are relatively close together in vector space.

For example:

"How can I configure JWT?"
        ↓
[0.12, 0.81, 0.42, ...]

"JWT authentication configuration"
        ↓
[0.15, 0.78, 0.45, ...]

These vectors should have high similarity.

4. Vector Database

Now we need somewhere to store these embeddings.

That's where a vector database comes in.

Popular choices include:

PostgreSQL + pgvector
Pinecone
Weaviate
Qdrant
Milvus
Elasticsearch
Redis

For a Spring Boot application, PostgreSQL with pgvector is an especially interesting option because you can keep your relational data and vector data within the same ecosystem.

Conceptually:

Document Chunk
      ↓
Embedding Model
      ↓
Vector
      ↓
Vector Database

Our database might contain something conceptually like:

ID | Content                    | Embedding
---|----------------------------|----------------
1  | JWT configuration...       | [0.12,...]
2  | OAuth2 configuration...    | [0.42,...]
3  | Database configuration...  | [0.71,...]

5. Similarity Search

Now suppose the user asks:

How do I configure JWT authentication in Spring Boot?

We first generate an embedding for the question.

User Query
    ↓
Embedding Model
    ↓
Query Vector

Then we search the vector database for similar vectors.

Query Vector
     ↓
Vector Database
     ↓
Similarity Search
     ↓
Top K Relevant Chunks

For example:

Result 1 → JWT configuration
Result 2 → Spring Security authentication
Result 3 → SecurityFilterChain configuration

These results become our context.

6. Building the Prompt

Now we combine:

User question
Retrieved context

For example:

Context:

Spring Security can be configured using SecurityFilterChain.
JWT authentication can be implemented using a custom
authentication filter...

Question:

How do I configure JWT authentication in Spring Boot?

The LLM receives this information and generates the answer.

This is the Augmented part of Retrieval-Augmented Generation.

7. Sending Context to the LLM

The final flow looks like:

User Question
      ↓
Embedding
      ↓
Vector Search
      ↓
Relevant Documents
      ↓
Prompt + Context
      ↓
LLM
      ↓
Answer

The LLM isn't searching the database itself.

Our application retrieves the information first and provides it to the model.

Building RAG with Spring AI

Now let's look at how Spring AI simplifies this architecture.

A typical Spring AI RAG application contains:

Spring Boot
     │
     ├── Document Reader
     │
     ├── Text Splitter
     │
     ├── Embedding Model
     │
     ├── Vector Store
     │
     └── Chat Model

The exact model and vector store can be swapped without rewriting the entire application.

That's one of the strengths of Spring AI's abstraction-based approach.

Project Setup

Let's create a Spring Boot project.

We'll need Spring AI dependencies for:

Chat model
Embeddings
Vector store

For example, with Maven:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-bedrock-converse</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
</dependency>

You should use versions compatible with the Spring AI version used by your project.

Configuring the Model

If we're using AWS Bedrock, our application needs AWS credentials and model configuration.

Conceptually:

spring:
  ai:
    bedrock:
      aws:
        region: us-east-1

The exact configuration depends on the Spring AI version and Bedrock model you're using.

For production environments, don't hard-code AWS credentials inside application.yml.

Use:

Environment Variables
        ↓
IAM Roles
        ↓
AWS Credentials Provider Chain

Whenever possible, prefer IAM roles over static credentials.

Creating a Vector Store

For PostgreSQL + pgvector, the architecture looks like:

Spring Boot
     ↓
Spring AI VectorStore
     ↓
PostgreSQL
     ↓
pgvector

The vector store becomes the bridge between our application and the vector database.

Conceptually:

VectorStore vectorStore;

We can then add documents:

vectorStore.add(documents);

And later search:

List<Document> results =
        vectorStore.similaritySearch(
                SearchRequest.builder()
                        .query(question)
                        .topK(5)
                        .build()
        );

The important abstraction here is:

VectorStore

Our application doesn't need to manually implement vector similarity calculations.

Loading Documents

Let's say we have a PDF containing product documentation.

Spring AI provides document readers that can load different document formats.

Conceptually:

var documents = reader.get();

We then split the documents into chunks.

For example:

TokenTextSplitter splitter = new TokenTextSplitter();

List<Document> chunks =
        splitter.apply(documents);

The exact splitter and configuration should be chosen based on your document structure and model context window.

Creating Embeddings

The embedding model converts every chunk into a vector.

Conceptually:

Document Chunk
      ↓
Embedding Model
      ↓
Vector

Spring AI handles this through the embedding/vector-store integration.

Then we can store the documents:

vectorStore.add(chunks);

Our ingestion pipeline is now:

PDF
 ↓
Documents
 ↓
Chunks
 ↓
Embeddings
 ↓
Vector Store

Retrieval

Now let's handle the user's question.

Suppose the user asks:

How does authentication work in our application?

We search the vector store:

List<Document> results =
        vectorStore.similaritySearch(
                SearchRequest.builder()
                        .query(question)
                        .topK(5)
                        .build()
        );

We now have the most relevant pieces of information.

For example:

Result 1
JWT authentication uses...

Result 2
The authentication filter...

Result 3
Security configuration...

Building the RAG Prompt

We can now construct a prompt using the retrieved documents.

For example:

String context = results.stream()
        .map(Document::getText)
        .collect(Collectors.joining("\n\n"));

String prompt = """
        Answer the question using only the provided context.

        Context:
        %s

        Question:
        %s
        """.formatted(context, question);

Then send it to the chat model.

Conceptually:

ChatResponse response =
        chatModel.call(
                new Prompt(prompt)
        );

And we return:

response.getResult()
        .getOutput()
        .getText();

That's a basic RAG implementation.

The Complete Flow

Putting everything together:

                 INGESTION PIPELINE

              ┌───────────────┐
              │   Documents   │
              └───────┬───────┘
                      ↓
                Text Splitting
                      ↓
                 Embeddings
                      ↓
               Vector Store
                      │
                      │
                      │
                      ▼
                 RETRIEVAL
                      ▲
                      │
               User Question
                      ↓
                  Embedding
                      ↓
               Similarity Search
                      ↓
                Top K Chunks
                      ↓
                   Context
                      ↓
                    Prompt
                      ↓
                     LLM
                      ↓
                   Answer

This is the basic architecture behind many modern knowledge-based AI applications.

Why Chunking Matters

One of the most underestimated parts of RAG is chunking.

Bad chunking can produce bad retrieval.

Imagine this document:

Spring Security

Authentication allows the application to verify
the identity of a user.

Authorization determines whether the authenticated
user has permission to access a resource.

If we split this badly:

Chunk 1:
Authentication allows...

Chunk 2:
the identity of a user. Authorization determines...

Chunk 3:
whether the authenticated user...

we may destroy important semantic relationships.

A better strategy is to preserve meaningful boundaries where possible.

Depending on the data, you may experiment with:

Chunk Size
Overlap
Sentence boundaries
Paragraph boundaries
Markdown headings
Semantic sections

There is no universal chunk size that works for every RAG application.

Why Top-K Matters

When performing similarity search, we usually retrieve the top K results.

For example:

.topK(5)

means:

Return the 5 most relevant chunks.

But bigger isn't always better.

If we retrieve too little:

K = 1

we may miss important context.

If we retrieve too much:

K = 50

we may introduce irrelevant information and increase token usage.

A common approach is to start with a small value and evaluate retrieval quality.

For example:

K = 3
K = 5
K = 10

Then measure which configuration works best for your dataset.

RAG vs Fine-Tuning

A common question is:

Why not fine-tune the model instead?

RAG and fine-tuning solve different problems.

RAG

Best when:

Knowledge changes frequently
You have private documents
You need citations or traceability
You need to update knowledge without retraining
You want to retrieve specific information

Fine-Tuning

Useful when you want to change:

Model behavior
Output style
Domain-specific patterns
Task performance

A useful mental model is:

RAG
→ Give the model the right information.

Fine-tuning
→ Change how the model behaves.

In many production systems, they can also be used together.

RAG Is More Than Vector Search

A basic RAG pipeline is only the beginning.

Production RAG systems often introduce additional stages:

Query
 ↓
Query Transformation
 ↓
Hybrid Retrieval
 ↓
Metadata Filtering
 ↓
Vector Search
 ↓
Reranking
 ↓
Context Compression
 ↓
Prompt Construction
 ↓
LLM

You might eventually introduce techniques such as:

Hybrid search
Metadata filtering
Reranking
HyDE
Query rewriting
Self-RAG
Corrective RAG
Graph RAG
Agentic RAG

This is where RAG becomes an engineering discipline rather than simply "put documents into a vector database."

Production Considerations

If you're building RAG for production, don't stop at:

PDF → Vector DB → LLM

You also need to think about:

1. Document ingestion

How frequently are documents updated?

New document
     ↓
Process
     ↓
Chunk
     ↓
Embed
     ↓
Update Vector Store

2. Metadata

Store useful metadata alongside chunks:

document_id
source
page_number
tenant_id
created_at
updated_at
document_type

This becomes extremely useful for filtering.

For example:

tenant_id = "company-123"

can ensure that users only retrieve documents belonging to their tenant.

3. Access Control

This is critical.

A RAG system must not retrieve documents that the current user isn't authorized to access.

Your retrieval layer should respect application permissions.

User
 ↓
Authentication
 ↓
Authorization
 ↓
Metadata Filters
 ↓
Retrieval
 ↓
LLM

Never assume that because the LLM can't "see" a document directly, the document is secure.

4. Observability

Track things such as:

Retrieval latency
Embedding latency
LLM latency
Token usage
Retrieved chunks
Similarity scores
Failure rate
Answer quality

Without observability, debugging RAG becomes extremely difficult.

A Simple Mental Model

If you're new to RAG, remember this:

Embedding Model
    =
    "Convert meaning into numbers"

Vector Database
    =
    "Store and search those meanings"

Retriever
    =
    "Find relevant information"

LLM
    =
    "Use that information to generate an answer"

Together:

Retrieve → Augment → Generate

That's RAG.

Where Spring AI Fits

Spring AI gives Java developers abstractions around many of these building blocks.

Instead of manually wiring every AI provider and vector database integration, we can work with abstractions such as:

ChatModel
EmbeddingModel
VectorStore
Document
DocumentReader

This allows us to focus more on the application architecture rather than provider-specific implementation details.

And that's particularly useful when building enterprise Java applications where we may want to change:

AWS Bedrock
     ↓
Another Model Provider

without completely rewriting our application.

Final Architecture

A production-oriented Spring AI RAG application can eventually look like this:

                    ┌───────────────────┐
                    │    User / App     │
                    └─────────┬─────────┘
                              ↓
                       Spring Boot API
                              ↓
                       Query Processing
                              ↓
                       Retrieval Layer
                              ↓
                    ┌───────────────────┐
                    │   Vector Store    │
                    │   PostgreSQL      │
                    │    + pgvector     │
                    └─────────┬─────────┘
                              ↓
                     Relevant Context
                              ↓
                      Prompt Assembly
                              ↓
                    ┌───────────────────┐
                    │    Spring AI     │
                    └─────────┬─────────┘
                              ↓
                    ┌───────────────────┐
                    │   AWS Bedrock    │
                    └─────────┬─────────┘
                              ↓
                           Answer

And separately:

Documents
    ↓
Document Reader
    ↓
Chunking
    ↓
Embedding Model
    ↓
PostgreSQL + pgvector

What's Next?

We started this series by exploring how Spring AI can connect Java applications with modern AI models.

With AWS Bedrock, we can access powerful foundation models without managing the underlying model infrastructure.

Now, with RAG, we can take another major step:

We can connect those models to our own data.

The journey now looks like:

LLM
 ↓
Spring AI
 ↓
AWS Bedrock
 ↓
RAG
 ↓
Vector Database
 ↓
Our Own Data

But there is another important capability missing.

What if we don't just want the model to answer questions?

What if we want the model to take actions?

For example:

User
 ↓
AI Agent
 ↓
Decide what to do
 ↓
Call a Tool
 ↓
Execute Action
 ↓
Return Result

That's where tool calling and AI agents come in.

Next up: Building AI Agents with Spring AI — Tool Calling, Memory, and Autonomous Workflows.

Key Takeaways

RAG connects LLMs to external knowledge.
Documents are split into smaller chunks.
Embeddings convert text into vectors.
Vector databases store and retrieve those vectors.
Similarity search finds relevant information.
Retrieved context is added to the LLM prompt.
Spring AI provides abstractions for building these pipelines.
PostgreSQL + pgvector is a practical option for Java applications.
Production RAG requires security, metadata filtering, observability, and evaluation.
RAG is not just vector search — retrieval quality ultimately determines answer quality.

If you're building AI applications with Java and Spring Boot, RAG is one of the most important patterns to understand.

AWS Bedrock with Spring AI: Moving from Local Models to the Cloud

Ayush Shrivastava — Thu, 27 Aug 2026 06:55:35 +0000

In the previous parts of this series, we explored how to build AI-powered applications with Spring AI and run models locally. Local inference is great for experimentation, development, privacy-focused use cases, and understanding how LLM applications work.

But production AI systems often need more.

They need access to powerful foundation models, managed infrastructure, scalability, reliability, and the ability to switch between models without managing GPU servers yourself.

This is where AWS Bedrock comes in.

In this article, we'll explore how to integrate AWS Bedrock with Spring AI and move from local AI inference to a managed cloud-based architecture.

What Is AWS Bedrock?

AWS Bedrock is a fully managed AWS service that provides access to foundation models from multiple AI providers through AWS infrastructure.

Instead of downloading and running a model locally, your application sends requests to AWS Bedrock.

The architecture changes from this:

Spring Boot Application
        ↓
    Spring AI
        ↓
Local Model Runtime
(Ollama / Local LLM)

To this:

Spring Boot Application
        ↓
    Spring AI
        ↓
 AWS Bedrock API
        ↓
 Foundation Model

Your application no longer needs to manage the underlying AI infrastructure.

AWS handles the model hosting, scaling, availability, and infrastructure required to run inference.

Why Move from Local Models to AWS Bedrock?

Local models are incredibly useful, but they come with limitations.

For example:

You may need powerful hardware.
Large models require significant memory and GPU resources.
Scaling inference for multiple users becomes difficult.
You are responsible for hosting and managing the infrastructure.
Production reliability becomes your responsibility.

AWS Bedrock solves many of these infrastructure challenges.

With Bedrock, you can focus more on your application instead of managing model servers.

Your architecture becomes more like this:

Client
   ↓
Spring Boot API
   ↓
Spring AI
   ↓
AWS Bedrock
   ↓
Foundation Model

This makes it easier to build cloud-native AI applications.

How Spring AI Fits Into the Architecture

Spring AI provides abstractions for working with different AI providers.

Your application interacts with high-level APIs such as ChatClient instead of manually writing HTTP requests for every model provider.

For example:

String response = chatClient.prompt()
        .user("Explain AWS Bedrock in simple terms")
        .call()
        .content();

The application code can remain relatively similar even when the underlying AI provider changes.

For example, you might start with:

Spring AI → Ollama → Local Model

And later move to:

Spring AI → AWS Bedrock → Cloud Model

This abstraction is one of the major advantages of using Spring AI.

Your business logic should ideally depend on AI capabilities rather than being tightly coupled to a specific provider.

Setting Up AWS Bedrock

Before your Spring Boot application can communicate with AWS Bedrock, you need access to AWS and the required permissions.

At a high level, the process looks like this:

AWS Account
    ↓
Enable Model Access
    ↓
Configure IAM Permissions
    ↓
Configure AWS Region
    ↓
Spring Boot Application
    ↓
Spring AI + AWS Bedrock

The exact models available can vary depending on your AWS region and account configuration.

You also need appropriate AWS credentials and permissions for invoking models.

For local development, credentials can be provided through the AWS credential provider chain, environment variables, profiles, or other supported AWS authentication mechanisms.

A typical environment configuration might look like this:

AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_REGION=your-region

However, in production, avoid hardcoding credentials in your application configuration.

Prefer IAM roles and managed identity mechanisms whenever possible.

Adding the Spring AI Bedrock Dependency

Spring AI provides integrations for AWS Bedrock models.

The dependency you use depends on the specific Spring AI version and Bedrock model integration you want to work with.

For example, your Maven configuration may include a Spring AI AWS Bedrock starter:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-bedrock-converse</artifactId>
</dependency>

Spring AI's dependency management should also be configured using the appropriate BOM for your version.

For example:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>YOUR_SPRING_AI_VERSION</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

The important idea is that Spring Boot auto-configuration can create the required AI components once the correct dependencies and AWS configuration are available.

Configuring AWS Bedrock

Your application configuration tells Spring AI how to connect to AWS Bedrock.

A simplified configuration might look something like this:

spring:
  ai:
    bedrock:
      aws:
        region: us-east-1

Depending on the Spring AI version and model integration, additional configuration may be required for the selected model, credentials, generation settings, or Bedrock Converse API.

For example, you may configure values such as:

AWS region
Model ID
Temperature
Maximum tokens
Top P
Credential provider settings

Conceptually:

spring:
  ai:
    model:
      chat: bedrock-converse

    bedrock:
      aws:
        region: us-east-1

The exact property names can vary between Spring AI releases, so always check the documentation for the version you are using.

Building a Chat API with Spring AI and Bedrock

Once the integration is configured, we can expose a simple API.

First, inject ChatClient.Builder:

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    @GetMapping
    public String chat(@RequestParam String message) {

        return chatClient.prompt()
                .user(message)
                .call()
                .content();
    }
}

Now a request such as:

GET /api/chat?message=Explain Spring AI

follows this flow:

HTTP Request
     ↓
ChatController
     ↓
ChatClient
     ↓
Spring AI
     ↓
AWS Bedrock
     ↓
Foundation Model
     ↓
Generated Response
     ↓
Spring Boot API

The controller does not need to know the low-level details of the Bedrock API.

Spring AI handles the integration layer.

The Big Architecture Change

The most important change is not just replacing one dependency with another.

The architecture itself changes.

Local Inference

User
 ↓
Spring Boot
 ↓
Spring AI
 ↓
Ollama
 ↓
Local LLM

Here, you are responsible for the machine running the model.

You need to think about:

CPU or GPU resources
RAM
Model downloads
Model startup time
Concurrent requests
Scaling
Monitoring the inference server

Cloud Inference

User
 ↓
Load Balancer
 ↓
Spring Boot Application
 ↓
Spring AI
 ↓
AWS Bedrock
 ↓
Foundation Model

Now the model infrastructure is managed separately from your application.

Your Spring Boot application becomes a consumer of AI infrastructure rather than the host of the model itself.

This separation can significantly simplify production architecture.

Choosing Between Different Foundation Models

One major benefit of Bedrock is access to multiple foundation models.

Different models can be better suited for different workloads.

For example:

Chat Application
        ↓
   General Model

Document Analysis
        ↓
 Long Context Model

Structured Extraction
        ↓
 Model with Strong
 Structured Output

Complex Reasoning
        ↓
 Higher Capability Model

This means model selection becomes an architectural decision.

You should consider factors such as:

Response quality
Latency
Cost
Context window
Tool calling capabilities
Structured output support
Regional availability

The best model is not always the largest or most expensive one.

A production system may even route different requests to different models.

Managing AI Configuration

One common mistake is placing model-specific configuration throughout the application.

Instead of this:

if (provider.equals("bedrock")) {
    // Bedrock logic
}

if (provider.equals("ollama")) {
    // Ollama logic
}

Try to keep provider-specific infrastructure separate from your application logic.

For example:

Application Layer
       ↓
Spring AI Abstraction
       ↓
Provider Configuration
       ↓
AWS Bedrock / Ollama / Other Models

This makes it easier to change providers later.

Your application should ideally focus on:

What should the AI do?

Instead of:

How does this specific provider's HTTP API work?

Authentication and Security

When working with AWS Bedrock, authentication becomes part of your architecture.

Your application needs permission to invoke models.

For production environments, the preferred approach is generally to use AWS IAM roles instead of embedding access keys inside application properties.

For example:

Spring Boot on AWS
       ↓
IAM Role
       ↓
Temporary Credentials
       ↓
AWS Bedrock

This is safer than:

application.yml
       ↓
Hardcoded AWS Keys

You should also think about:

Least-privilege IAM permissions
Credential rotation
Environment separation
Request logging
Sensitive prompt data
PII protection

Moving AI to the cloud also means thinking carefully about what data leaves your application.

Cost Becomes Part of the Architecture

With local models, the main cost is often infrastructure.

You pay for:

GPU
RAM
Compute
Storage
Servers

With managed inference, cost is often related to model usage.

Conceptually:

Application Request
        ↓
Input Tokens
        +
Output Tokens
        ↓
Inference Cost

This means AI applications should treat token usage as an engineering concern.

You may need:

Token monitoring
Request limits
Model selection policies
Budget controls
Prompt optimization
Caching

For example, sending unnecessary context to a model can increase both latency and cost.

A good AI architecture should therefore optimize the entire request pipeline.

Adding Observability

Production AI systems need more than application logs.

You may want to monitor:

Request
   ↓
Prompt Size
   ↓
Model
   ↓
Latency
   ↓
Token Usage
   ↓
Cost
   ↓
Response

This helps answer important questions:

Which requests are slow?
Which model is being used?
How many tokens are being consumed?
Which prompts are expensive?
Are failures coming from the application or the AI provider?

As your AI application grows, observability becomes essential.

Local Development vs Production

You don't necessarily need to abandon local models completely.

A practical architecture could use different providers for different environments.

Development
Spring Boot
     ↓
   Ollama
     ↓
 Local Model

Production
Spring Boot
     ↓
Spring AI
     ↓
AWS Bedrock
     ↓
Cloud Model

This gives developers a fast and inexpensive local development workflow while allowing production to use managed cloud infrastructure.

The important part is keeping your application architecture flexible enough to support both.

What's Next?

AWS Bedrock gives us access to powerful foundation models without managing the underlying AI infrastructure ourselves.

But moving AI into production introduces new challenges.

How do we manage conversation history?

How do we provide the model with our own documents?

How do we build applications that can retrieve relevant information before generating a response?

In Part 7, we'll take the next step and explore RAG with Spring AI.

We'll look at how documents move through the RAG pipeline:

Documents
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector Database
    ↓
Similarity Search
    ↓
Relevant Context
    ↓
LLM
    ↓
Answer

This is where AI applications start becoming truly connected to your own data.

Next up: Building RAG Applications with Spring AI.

Run AI Models Locally with Docker Model Runner and Spring AI

Ayush Shrivastava — Sun, 23 Aug 2026 08:22:51 +0000

Generative AI development doesn't always require a cloud-hosted model.

If you're building Java applications with Spring AI, you can run AI models locally and connect them to your Spring Boot application without depending on external model APIs.

One interesting option is Docker Model Runner.

In this article, we'll explore how Docker Model Runner works, why it is useful for Java developers, and how to connect it with Spring AI.

What is Docker Model Runner?

Docker Model Runner allows you to run AI models locally using Docker.

Instead of sending every prompt to a cloud provider such as OpenAI or AWS Bedrock, you can run supported models on your own machine.

The architecture looks like this:

Spring Boot Application
        |
        v
     Spring AI
        |
        v
   OpenAI-compatible API
        |
        v
 Docker Model Runner
        |
        v
     Local LLM

Your Java application interacts with the model through an API, while Docker handles running the model locally.

This gives developers a convenient way to experiment with LLM applications without immediately provisioning cloud infrastructure.

Why Run Models Locally?

There are several reasons you may want to run an LLM locally.

1. Development without API costs

During development, you may send hundreds or thousands of prompts.

Running a model locally can eliminate API charges during experimentation.

2. Privacy

Your prompts and application data can remain on your machine instead of being sent to an external AI provider.

This can be particularly useful when experimenting with sensitive or proprietary data.

3. Offline experimentation

Once the model is available locally, you don't need an internet connection for every inference request.

4. Faster experimentation

Developers can experiment with prompts, tool calling, RAG pipelines, and application logic without repeatedly configuring cloud credentials.

5. Same Spring AI programming model

One of the biggest advantages is that your application can continue using Spring AI abstractions.

Your business logic doesn't need to be tightly coupled to a specific model provider.

Docker Model Runner + Spring AI

Spring AI provides abstractions such as:

ChatModel
ChatClient

This means your application can interact with an LLM without having to directly implement provider-specific HTTP calls.

The important idea is:

Application
     |
     v
 ChatClient
     |
     v
 ChatModel
     |
     v
 Model API
     |
     v
Local Model

If your local model exposes an OpenAI-compatible API, Spring AI can communicate with it using the appropriate OpenAI configuration.

Step 1: Install Docker

First, make sure Docker Desktop is installed and running on your machine.

Docker Model Runner is available through Docker's model functionality, depending on your Docker Desktop version and configuration.

You can verify that Docker is available with:

docker --version

Then make sure Docker Desktop is running.

Step 2: Enable Docker Model Runner

Docker Desktop provides the model-running infrastructure required to run supported AI models locally.

Once enabled, you can work with models directly through Docker.

The exact commands and model availability can change as Docker's model ecosystem evolves, so check the current Docker documentation for the model you want to use.

The important concept for our Spring AI application is that Docker Model Runner exposes an API endpoint that our application can communicate with.

Step 3: Create a Spring Boot Project

Create a Spring Boot application with Spring AI.

For Maven, add the Spring AI OpenAI starter:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>

The reason we're using the OpenAI starter is not because we're calling OpenAI's cloud service.

We're using the OpenAI-compatible API supported by the local model runtime.

This is an important concept:

An OpenAI-compatible API does not necessarily mean you're using OpenAI's infrastructure.

It simply means the API follows a compatible request/response format.

Step 4: Configure Spring AI

In application.properties, configure the OpenAI base URL to point to your local Docker Model Runner endpoint.

For example:

spring.ai.openai.base-url=http://localhost:<model-runner-port>
spring.ai.openai.api-key=dummy
spring.ai.openai.chat.options.model=<your-local-model>

The exact endpoint and model name depend on your Docker Model Runner setup.

The API key may not actually be required by the local runtime, but the Spring AI OpenAI client expects the configuration property, so a placeholder value can be used when appropriate.

Step 5: Create a ChatClient

Now we can create a ChatClient.

@RestController
@RequestMapping("/ai")
public class AIController {

    private final ChatClient chatClient;

    public AIController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    @GetMapping("/chat")
    public String chat(@RequestParam String message) {

        return chatClient
                .prompt()
                .user(message)
                .call()
                .content();
    }
}

Now your Spring Boot application has a simple endpoint:

GET /ai/chat?message=Explain dependency injection in Spring

The flow becomes:

HTTP Request
     |
     v
Spring Boot
     |
     v
ChatClient
     |
     v
Spring AI
     |
     v
Docker Model Runner
     |
     v
Local LLM
     |
     v
Generated Response

ChatClient Hides Provider Complexity

This is where Spring AI becomes useful.

Your controller doesn't need to know whether the model is:

OpenAI
Azure OpenAI
AWS Bedrock
Ollama
Docker Model Runner
another compatible provider

The application interacts with the Spring AI abstraction.

For example:

chatClient
        .prompt()
        .user("Explain Spring Boot dependency injection")
        .call()
        .content();

The application focuses on what it wants from the model, rather than implementing the underlying model communication itself.

Local LLM vs Cloud LLM

Running a model locally doesn't mean local models are always better.

There are trade-offs.

Local Models	Cloud Models
Data stays locally	Data sent to provider
No per-request API cost	Usually usage-based pricing
Requires local compute	Provider handles infrastructure
Potentially slower	Often faster
Limited by local hardware	Access to larger models
Useful for development	Useful for production workloads

For example, a developer laptop may be perfectly capable of running a smaller model.

But running a large frontier model locally may require significantly more memory and compute.

So the right question isn't:

"Local or cloud?"

It is:

"Which model deployment strategy fits this workload?"

Where Docker Model Runner Fits

Docker Model Runner is particularly interesting for developers who already use Docker as part of their development workflow.

You can think of it as bringing model execution closer to the rest of your local development environment.

Instead of:

Spring Boot
     |
     +----> OpenAI
     |
     +----> AWS

you can have:

Docker Development Environment
        |
        +---- Spring Boot
        |
        +---- Database
        |
        +---- Redis
        |
        +---- AI Model

This can make local AI application development much easier to reproduce.

A More Realistic Architecture

Once you move beyond a simple chatbot, the architecture becomes more interesting.

For example:

             Spring Boot
                  |
             Spring AI
                  |
        +---------+---------+
        |                   |
    ChatClient          Embeddings
        |                   |
        v                   v
Docker Model Runner     Vector Store
        |
        v
     Local LLM

This architecture can support applications involving:

RAG
document processing
semantic search
AI assistants
summarization
classification
question answering
tool calling

And because the application uses Spring AI abstractions, you can change parts of the architecture later.

Local Development Is More Than Cost Saving

One of the biggest benefits of local models is actually developer experimentation.

Imagine you're building a RAG application.

You need to test:

Documents
    ↓
Chunking
    ↓
Embeddings
    ↓
Vector Store
    ↓
Retrieval
    ↓
Prompt
    ↓
LLM

During development, you may repeatedly modify prompts and retrieval strategies.

Having a local model can make that experimentation cheaper and easier.

Later, when you're ready for production, you can evaluate whether a managed model provider makes more sense.

What About Production?

This is where architecture decisions become important.

A local model running on a developer laptop is obviously different from a production AI infrastructure setup.

For production, you need to think about:

GPU availability
model size
concurrency
latency
autoscaling
observability
security
model updates
failure recovery
cost

For some workloads, a managed cloud model will be the better option.

For others, self-hosted inference can make sense because of privacy, compliance, cost, or latency requirements.

The important thing is that your application architecture should avoid unnecessary coupling to one model provider.

Spring AI Makes This Easier

This is one of the reasons I like the Spring AI abstraction.

Your business code can work at a higher level:

ChatClient
    .prompt()
    .user(prompt)
    .call()
    .content();

Instead of manually implementing:

HTTP request
    ↓
Authentication
    ↓
JSON serialization
    ↓
Provider API
    ↓
Response parsing

Spring AI handles much of that integration layer.

That allows Java developers to focus on building the actual AI application.

Final Thoughts

Docker Model Runner gives Java developers another option for experimenting with generative AI locally.

Combined with Spring AI, the architecture becomes relatively straightforward:

Spring Boot
     ↓
Spring AI
     ↓
ChatClient
     ↓
OpenAI-Compatible API
     ↓
Docker Model Runner
     ↓
Local LLM

The bigger lesson isn't simply how to run one particular model.

It's understanding model portability.

Your application shouldn't necessarily care whether the underlying model is running locally or in the cloud.

Spring AI gives you abstractions that help separate your application logic from the model provider.

And that's an important foundation for building production-ready AI applications.

What's Next?

In Part 6, we'll move from local inference to the cloud and explore AWS Bedrock with Spring AI.

We'll look at how Spring Boot applications can interact with foundation models available through AWS and how the architecture changes when AI inference moves from your local machine to a managed cloud platform.

AI Governance Needs Proof, Not Just Logs

Ayush Shrivastava — Fri, 21 Aug 2026 07:41:19 +0000

AI Governance Needs Proof, Not Just Logs

AI systems are becoming more autonomous.

Large language models are no longer limited to generating text. They can call tools, access enterprise systems, retrieve sensitive information, execute workflows, communicate with other agents, and make decisions that can directly affect business operations.

As AI becomes more capable, organizations need more than visibility into what these systems are doing.

They need proof.

Logging Is Not the Same as Governance

Traditional application logging answers a basic question:

What happened?

An application writes an event to a log, and engineers can later inspect it.

But AI systems introduce a more difficult governance challenge.

What if the system being monitored can influence the records used to monitor it?

What if an administrator, compromised service, or malicious actor can modify or delete those records?

At that point, having logs does not necessarily mean having trustworthy evidence.

This is why AI governance cannot stop at collecting logs.

It needs to establish:

Integrity
Accountability
Traceability
Authorization
Auditability

Building a Tamper-Evident Audit Trail

A tamper-evident audit trail is designed so that unauthorized changes to historical records can be detected.

For AI infrastructure, this can include events such as:

LLM requests and responses
Tool calls and tool results
MCP interactions
Agent-to-agent communication
Authorization decisions
Policy enforcement events
Identity and access information
Model and workflow execution
Security and governance decisions

Instead of treating these events as ordinary application logs, they become part of a governance record designed to provide reliable evidence of what happened.

Cryptographic signatures and hash chaining can help establish relationships between records and make unauthorized modifications detectable.

The objective isn't simply to prevent every possible change.

The objective is to make silent manipulation of history difficult to hide.

Your Infrastructure. Your Keys. Your Audit Trail.

This principle is at the heart of DVARA:

Your infra. Your keys. Your audit trail.

AI governance should not require organizations to give up control of their infrastructure, credentials, or sensitive governance data.

Your AI systems may use multiple models, tools, MCP servers, and agents.

But the organization operating those systems should still have control over the environment in which governance data is generated and stored.

This becomes especially important for enterprises operating in regulated or security-sensitive environments.

Organizations need to know:

Who made the request?
Which model processed it?
Which tool was called?
What authorization decision was made?
What action did the agent take?
What happened afterward?
Can we prove that the record hasn't been silently rewritten?

AI Agents Make This Even More Important

AI agents introduce another layer of complexity.

A traditional application may follow a relatively predictable sequence of operations.

An autonomous agent can:

Receive a request
Reason about the task
Select a tool
Execute an action
Observe the result
Decide what to do next
Call another tool
Communicate with another agent
Complete the workflow

One user request could therefore produce dozens of internal events.

When multiple agents interact with each other, the audit trail becomes even more important.

You need to understand not only the final outcome, but also the chain of actions that produced it.

Without reliable records, investigating an unexpected AI action can become extremely difficult.

With a strong audit trail, organizations can reconstruct what happened and establish accountability across the workflow.

From Observability to Accountability

Observability helps engineers understand systems.

Governance goes one step further.

It asks whether the organization can demonstrate that the system behaved according to defined policies and whether the evidence itself can be trusted.

That distinction matters.

Capability	What it answers
Metrics	What is happening?
Logs	What events occurred?
Traces	How did the request flow?
Audit trails	What happened historically?
Tamper-evident audit trails	Can we detect whether the historical record was altered?

For AI systems operating in production, that distinction is becoming increasingly important.

The Question Auditors Will Eventually Ask

Imagine an auditor asks:

"What did your AI system do on this particular request?"

You provide the logs.

Then they ask:

"How do you know those logs weren't modified?"

That second question changes everything.

"Because we have logs" is no longer a sufficient answer.

Modern AI governance needs evidence that can withstand scrutiny.

It needs a chain of accountability from the original request through:

User Request
     ↓
Authorization
     ↓
LLM / Agent
     ↓
Tool / MCP Call
     ↓
Policy Decision
     ↓
Action
     ↓
Audit Record

This creates a stronger foundation for trustworthy AI operations.

Governance Needs Proof

The future of AI isn't just about making models more capable.

It's about making AI systems:

Observable
Controllable
Auditable
Secure
Accountable

As organizations deploy LLMs, MCP infrastructure, and autonomous agents at scale, trustworthy audit trails will become an essential part of the architecture.

Because ultimately, governance isn't about having more logs.

It's about being able to answer three simple questions:

What happened?

Who or what caused it?

Can you prove the record is trustworthy?

That's the difference between simply monitoring AI and building AI systems that organizations can confidently govern.

The DVARA Principle

Your infra. Your keys. Your audit trail.

Your AI infrastructure should remain under your control.

Your governance data should remain within your security boundary.

And your audit trail should provide evidence you can trust.

AI governance needs more than visibility.

It needs proof.

Learn More

Explore DVARA:

https://dvarahq.com/

Topics

AI Governance AI Engineering Artificial Intelligence LLMOps MCP AI Agents AI Security Audit Trail Data Governance DevSecOps Responsible AI Enterprise AI

Run Local LLMs with Ollama and Spring AI

Ayush Shrivastava — Thu, 20 Aug 2026 06:14:52 +0000

In the previous parts, we connected Spring AI with cloud-based AI models.

But there is one important question:

What if you don't want to send your data to an external AI provider?

What if you want to:

Run an LLM on your own machine
Develop AI applications without API costs
Work without an internet connection
Keep sensitive company data private
Experiment with different open-source models
Build AI features locally before moving them to production

This is where Ollama becomes very useful.

In this article, we will learn how to run a local LLM using Ollama and connect it with Spring AI.

We will build a simple real-world AI Customer Support Assistant using Java, Spring Boot, Spring AI, and Ollama.

What We Are Building

Our application will look like this:

                  User
                    |
                    | HTTP Request
                    v
          +---------------------+
          |   Spring Boot API   |
          +---------------------+
                    |
                    v
             +-------------+
             |  Spring AI  |
             |  ChatClient |
             +-------------+
                    |
                    v
               +--------+
               | Ollama |
               +--------+
                    |
                    v
              Local LLM
             (Llama/Qwen)
                    |
                    v
              AI Response
                    |
                    v
                  User

The important part is that the LLM is running locally.

There is no need to send every prompt to OpenAI, Anthropic, or another cloud provider.

1. What Is Ollama?

Ollama makes it easy to run open-source LLMs locally.

Instead of calling a remote API like:

Spring Boot
     |
     v
OpenAI API
     |
     v
Cloud LLM

we can run:

Spring Boot
     |
     v
Spring AI
     |
     v
Ollama
     |
     v
Local LLM

Ollama can run models such as:

Llama
Qwen
Gemma
Mistral
DeepSeek
and many other compatible models

The exact models available change over time, so always check the Ollama model library before choosing one.

2. Why Run an LLM Locally?

Imagine you are building an internal HR application.

Employees may send questions such as:

What is our maternity leave policy?

or:

What is the process for requesting annual leave?

You may not want internal company information leaving your infrastructure.

A local LLM can help:

Employee
   |
   v
Spring Boot
   |
   v
RAG / Business Logic
   |
   v
Ollama
   |
   v
Local LLM

This can provide a useful privacy boundary.

However, remember:

Running an LLM locally does not automatically make your application secure.

You still need proper authentication, authorization, logging, data protection, network security, and prompt/data controls.

3. Install Ollama

First, install Ollama on your operating system.

After installation, verify it:

ollama --version

If the command works, Ollama is installed.

Now we need an LLM.

For example:

ollama pull llama3.2

Then run it:

ollama run llama3.2

You can now talk to the model directly from your terminal.

For example:

>>> Explain Java interfaces in simple English.

The model will generate a response locally.

4. How Ollama Works

At a high level, the architecture is:

             Your Application
                    |
                    v
              Ollama API
                    |
                    v
             Model Runtime
                    |
                    v
              Local Model
                    |
                    v
                Response

Ollama exposes an API that applications can communicate with.

Spring AI can communicate with this API for us.

That means we don't have to manually build HTTP requests to the Ollama API.

5. Create a Spring Boot Project

Let's create a Spring Boot application.

You can use Spring Initializr or your preferred IDE.

Basic project:

local-ai-demo
│
├── src
│   └── main
│       ├── java
│       │   └── com.example.localai
│       │       └── LocalAiApplication.java
│       │
│       └── resources
│           └── application.yml
│
└── pom.xml

We need:

Spring Web
Spring AI Ollama

6. Maven Dependency

Add the Spring AI Ollama starter to your pom.xml.

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

You should use the Spring AI version compatible with your Spring Boot version.

For production projects, avoid randomly mixing Spring Boot and Spring AI versions.

7. Configure Ollama

Now configure the Ollama model.

spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      chat:
        options:
          model: llama3.2

The important part is:

localhost:11434

This is the default Ollama API endpoint.

Your architecture now becomes:

Spring Boot
     |
     | HTTP
     v
localhost:11434
     |
     v
Ollama
     |
     v
llama3.2

8. Create the ChatClient

Spring AI provides ChatClient, which gives us a clean API for interacting with chat models.

Create a configuration class:

package com.example.localai.config;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class AiConfig {

    @Bean
    ChatClient chatClient(ChatClient.Builder builder) {
        return builder.build();
    }
}

That's it.

Spring AI will use the configured Ollama chat model.

9. Create Our First AI Endpoint

Now let's create a simple controller.

package com.example.localai.controller;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/api/ai")
public class AiController {

    private final ChatClient chatClient;

    public AiController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    @GetMapping("/ask")
    public String ask(@RequestParam String question) {

        return chatClient
                .prompt()
                .user(question)
                .call()
                .content();
    }
}

Start your Spring Boot application.

Then call:

GET /api/ai/ask?question=Explain Spring Boot in simple English

The request flows like this:

HTTP Request
     |
     v
AiController
     |
     v
ChatClient
     |
     v
Spring AI
     |
     v
Ollama
     |
     v
Local LLM
     |
     v
AI Response

And the response comes back to the client.

10. The Real-World Example

Let's make our application more useful.

Imagine we are building an AI Customer Support Assistant.

A customer sends:

My payment was deducted but my subscription is still inactive.
What should I do?

Instead of simply passing the question to the model, we can provide a system instruction.

@GetMapping("/support")
public String support(@RequestParam String question) {

    return chatClient
            .prompt()
            .system("""
                    You are a helpful customer support assistant.

                    Answer in simple English.
                    Do not invent company policies.
                    If you do not know something, clearly say that
                    you do not have enough information.
                    """)
            .user(question)
            .call()
            .content();
}

Now the model has some context about its role.

11. System Prompt vs User Prompt

Spring AI allows us to separate instructions from user input.

For example:

return chatClient
        .prompt()
        .system("""
                You are an AI customer support assistant.
                Keep answers short and easy to understand.
                Never make up information.
                """)
        .user(question)
        .call()
        .content();

Think about it like this:

System Prompt
     |
     | "Who are you?"
     | "How should you behave?"
     |
     v
   LLM
     ^
     |
     | "What does the user want?"
     |
User Prompt

This separation becomes extremely useful when building production AI applications.

12. Create a Service Layer

Putting everything inside the controller is not a good architecture.

Instead:

Controller
    |
    v
Service
    |
    v
ChatClient
    |
    v
Ollama

Create:

package com.example.localai.service;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.stereotype.Service;

@Service
public class AiService {

    private final ChatClient chatClient;

    public AiService(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    public String ask(String question) {

        return chatClient
                .prompt()
                .user(question)
                .call()
                .content();
    }
}

Then the controller becomes:

package com.example.localai.controller;

import com.example.localai.service.AiService;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/api/ai")
public class AiController {

    private final AiService aiService;

    public AiController(AiService aiService) {
        this.aiService = aiService;
    }

    @GetMapping("/ask")
    public String ask(@RequestParam String question) {
        return aiService.ask(question);
    }
}

This structure is much easier to extend later.

13. Building a Better Support Assistant

Let's make the prompt more useful.

public String support(String question) {

    return chatClient
            .prompt()
            .system("""
                    You are a professional customer support assistant.

                    Rules:
                    1. Use simple English.
                    2. Be polite and helpful.
                    3. Give step-by-step instructions when possible.
                    4. Never invent policies.
                    5. If information is missing, ask for clarification.
                    """)
            .user(question)
            .call()
            .content();
}

Now a question like:

I cannot reset my password.

could produce something like:

I'm sorry you're having trouble resetting your password.

Please try these steps:

1. Open the login page.
2. Click "Forgot Password".
3. Enter your registered email.
4. Check your email for the reset link.

If you still cannot reset your password, contact support.

This is already a useful AI feature.

14. Add Conversation Memory

A basic AI call is stateless.

For example:

User:
My order is late.

AI:
Please provide your order number.

User:
It is 12345.

AI:
What order are you referring to?

The model doesn't automatically know the previous conversation unless we provide that context.

In a real application, we need conversation memory.

The architecture becomes:

User
 |
 v
Spring Boot
 |
 +------> Conversation Store
 |              |
 |              v
 |          Previous Messages
 |
 v
Spring AI
 |
 v
Ollama
 |
 v
LLM

Depending on your application, the conversation history can be stored in databases such as PostgreSQL or Redis.

For example:

conversation_id
        |
        v
+----------------------+
| User message         |
| AI response          |
| User message         |
| AI response          |
+----------------------+

Then the relevant history can be included when making the next model call.

This is an important step from a simple AI demo toward a production AI application.

15. Local LLM + RAG

This is where local models become especially interesting.

Suppose your company has:

Employee Handbook
Product Documentation
Customer FAQs
Internal Policies
Technical Documentation

We can build a RAG system:

              Documents
                  |
                  v
             Text Parser
                  |
                  v
               Chunking
                  |
                  v
              Embeddings
                  |
                  v
            Vector Database
                  |
                  |
User Question ---> Retrieval
                  |
                  v
             Relevant Data
                  |
                  v
             Spring AI
                  |
                  v
               Ollama
                  |
                  v
             Local LLM
                  |
                  v
               Answer

Now the LLM doesn't need to know your company information beforehand.

We retrieve the relevant information and provide it as context.

16. Example: Internal HR Assistant

Imagine an employee asks:

How many days of annual leave can I take?

The system can search the company's HR documents.

Suppose the vector database retrieves:

Employees receive 24 days of annual leave per calendar year.

Spring AI can then construct a prompt:

Answer the question using only the following context.

Context:
Employees receive 24 days of annual leave per calendar year.

Question:
How many days of annual leave can I take?

The local LLM generates:

According to the company policy, employees receive
24 days of annual leave per calendar year.

The complete architecture becomes:

                    Employee
                       |
                       v
                Spring Boot API
                       |
                       v
                Spring AI RAG
                  /        \
                 /          \
                v            v
        Vector Database    Ollama
                |             |
                v             v
          Relevant Data     Local LLM
                 \            /
                  \          /
                   v        v
                    Answer

This is much closer to a real enterprise AI architecture.

17. Why This Architecture Is Powerful

Imagine an organization has sensitive documents.

With a cloud-only architecture:

Application
     |
     v
Cloud AI API
     |
     v
External Model

With a local architecture:

Application
     |
     v
Internal Infrastructure
     |
     +------> Vector DB
     |
     +------> Ollama
                |
                v
             Local LLM

This can be attractive for:

Internal knowledge assistants
Developer tools
Private document analysis
Customer support prototypes
Offline applications
Sensitive enterprise workloads

But again, local inference is not a complete security strategy by itself.

18. Streaming AI Responses

For chat applications, waiting for the entire response can feel slow.

A better experience is:

AI is typing...

Hello
Hello, how
Hello, how can
Hello, how can I
Hello, how can I help?

Spring AI supports streaming responses through Flux.

For example:

@GetMapping(value = "/stream", produces = "text/event-stream")
public Flux<String> stream(@RequestParam String question) {

    return chatClient
            .prompt()
            .user(question)
            .stream()
            .content();
}

The client can receive pieces of the response as they are generated.

This is useful for:

AI chat applications
Coding assistants
Customer support
AI search
Writing assistants

19. Model Selection Matters

Not every local model is good for every task.

For example:

Small model
    |
    +-- Faster
    +-- Less memory
    +-- Lower hardware requirements

Large model
    |
    +-- Better reasoning potential
    +-- More memory
    +-- Higher latency

Your choice depends on:

RAM
GPU/VRAM
CPU
Model size
Context length
Response speed
Task complexity
Accuracy requirements

For a simple local experiment, start with a relatively small model.

Then benchmark larger models if your hardware allows it.

20. Ollama vs Cloud LLMs

A simple comparison:

Feature	Local Ollama	Cloud LLM
Internet required	Usually no	Yes
API cost	No per-token cloud fee	Usually usage-based
Data leaves machine	Can stay local	Sent to provider
Hardware required	Yes	Mostly no
Scaling	Your responsibility	Provider handles infrastructure
Model choice	Open/local models	Provider-specific models
Setup	More infrastructure	Usually easier
Latency	Depends on hardware	Depends on network/provider

There is no universal winner.

A practical architecture may even use both.

21. Hybrid AI Architecture

For example:

                 AI Gateway
                     |
            +--------+--------+
            |                 |
            v                 v
        Ollama            Cloud LLM
            |                 |
            v                 v
       Local Model       External Model

You could use:

Sensitive requests
        |
        v
      Ollama

and:

Complex reasoning
        |
        v
    Cloud Model

The routing decision can be implemented inside your application.

This gives you more flexibility.

22. Production Considerations

Running Ollama on your laptop is great for development.

Production is different.

You need to think about:

Infrastructure

Load Balancer
      |
      v
Spring Boot Instances
      |
      v
AI Service
      |
      v
GPU-enabled inference servers

Observability

Track:

Request latency
Model latency
Token usage where available
Error rate
Timeout rate
Model failures
Concurrent requests

Security

Protect:

Ollama endpoints
Internal APIs
User prompts
Retrieved documents
Conversation history

Do not expose an unauthenticated Ollama service directly to the public internet.

23. Handling Errors

AI services can fail.

Your application should not assume every model call succeeds.

For example:

public String ask(String question) {

    try {
        return chatClient
                .prompt()
                .user(question)
                .call()
                .content();

    } catch (Exception ex) {
        throw new RuntimeException(
                "AI service is currently unavailable", ex);
    }
}

In a production application, use a proper exception hierarchy and global exception handling rather than exposing raw exceptions.

You may also add:

Timeout
Retry
Circuit Breaker
Fallback
Rate Limiting
Observability

24. A Better Production Architecture

A more realistic architecture could look like this:

                    Client
                      |
                      v
               API Gateway
                      |
                      v
              Spring Boot API
                      |
          +-----------+-----------+
          |                       |
          v                       v
     Conversation             RAG Service
       Service                    |
          |                       v
          |                 Vector Database
          |                       |
          +-----------+-----------+
                      |
                      v
                 AI Service
                      |
             +--------+--------+
             |                 |
             v                 v
          Ollama           Cloud LLM
             |                 |
             v                 v
        Local Model      External Model

This architecture allows you to evolve from a simple local experiment into a production-grade AI platform.

25. Complete Minimal Example

Here is the complete service:

@Service
public class AiService {

    private final ChatClient chatClient;

    public AiService(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    public String ask(String question) {

        return chatClient
                .prompt()
                .system("""
                        You are a helpful AI assistant.
                        Answer in simple English.
                        Do not invent facts.
                        """)
                .user(question)
                .call()
                .content();
    }
}

Controller:

@RestController
@RequestMapping("/api/ai")
public class AiController {

    private final AiService aiService;

    public AiController(AiService aiService) {
        this.aiService = aiService;
    }

    @GetMapping("/ask")
    public String ask(@RequestParam String question) {
        return aiService.ask(question);
    }
}

Configuration:

spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      chat:
        options:
          model: llama3.2

Architecture:

HTTP Client
    |
    v
AiController
    |
    v
AiService
    |
    v
ChatClient
    |
    v
Spring AI
    |
    v
Ollama
    |
    v
Local LLM

That's enough to build your first local AI backend.

26. Test It

Start Ollama:

ollama run llama3.2

Then start Spring Boot:

./mvnw spring-boot:run

On Windows:

mvnw.cmd spring-boot:run

Then call:

GET http://localhost:8080/api/ai/ask?question=Explain dependency injection in Spring Boot

You should receive an answer generated by your local model.

27. What We Learned

In this article, we built a local AI backend using:

Java
   +
Spring Boot
   +
Spring AI
   +
Ollama
   +
Local LLM

We learned:

What Ollama is
How to install and run a local model
How Spring AI connects to Ollama
How to use ChatClient
How to create an AI REST API
How to use system and user prompts
How streaming works
How local LLMs can be used with RAG
How to think about production architecture
When local and cloud models can be combined

Final Thoughts

Running an LLM locally changes the way we think about AI applications.

You don't always need to start with an expensive cloud API.

You can start on your own laptop:

Ollama
   |
Local LLM
   |
Spring AI
   |
Spring Boot
   |
REST API

Then gradually evolve it:

Local LLM
    ↓
RAG
    ↓
Vector Database
    ↓
Conversation Memory
    ↓
Tool Calling
    ↓
AI Agents
    ↓
Observability
    ↓
Production AI Platform

And this is where Spring AI becomes interesting for Java developers.

You can use the Spring ecosystem you already know and add modern AI capabilities without completely changing how you build backend applications.

In the next part, we can go one step further and build a RAG application with Spring AI, embeddings, PostgreSQL + pgvector, and a local Ollama model.

That is where our simple chatbot starts becoming a real AI application.

Build Your First Spring AI Application with OpenAI Using Spring Boot

Ayush Shrivastava — Mon, 17 Aug 2026 19:44:38 +0000

Build Your First Spring AI Application with OpenAI Using Spring Boot

If you have been following this Spring AI series, you already understand the two most important abstractions: ChatClient and ChatModel.

In the previous article, we learned that:

Your Java Code
      |
      v
 ChatClient
      |
      v
 ChatModel
      |
      v
 AI Provider

ChatClient gives us a developer-friendly API, while ChatModel handles the underlying model integration.

Now it is time to build something real.

In this article, we will build our first Spring AI application using OpenAI.

We will start from project setup and configuration, connect Spring Boot to OpenAI, create a REST API, send prompts to an AI model, and understand what happens behind the scenes.

By the end, you will have a working AI-powered Spring Boot API.

What We Are Building

Our application will expose a simple REST endpoint:

GET /api/chat?message=Explain dependency injection

The flow will look like this:

Client
   |
   v
Spring Boot REST API
   |
   v
ChatClient
   |
   v
ChatModel
   |
   v
OpenAI
   |
   v
AI Response

The goal is intentionally simple.

We want to understand the complete flow before adding more advanced concepts such as RAG, tools, memory, structured output, and AI agents.

Prerequisites

Before starting, you should have:

Java installed
Maven installed
A Spring Boot project
An OpenAI API key
Basic knowledge of Spring Boot
Basic understanding of REST APIs

You should also be comfortable with dependency injection and creating REST controllers.

If you have followed the previous articles in this series, most of this should already be familiar.

Step 1: Create a Spring Boot Project

The easiest way to create the project is through Spring Initializr.

Choose:

Project: Maven

Language: Java

Spring Boot: Your compatible Spring Boot version

Packaging: Jar

Java: 17 or later

For dependencies, we need Spring Web and the Spring AI OpenAI starter.

Your project will eventually look something like:

spring-ai-openai-demo
│
├── src
│   ├── main
│   │   ├── java
│   │   │   └── com.example.demo
│   │   │       └── DemoApplication.java
│   │   │
│   │   └── resources
│   │       └── application.properties
│
└── pom.xml

Step 2: Add the Spring AI OpenAI Dependency

Add the Spring AI OpenAI model starter to your Maven configuration.

For example:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>

You will also need Spring Web:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

Your project now has the components required to build a simple AI REST API.

Conceptually:

Spring Boot
     |
     +-- Spring Web
     |
     +-- Spring AI
             |
             +-- OpenAI integration

Step 3: Configure Your OpenAI API Key

Spring AI needs credentials to communicate with OpenAI.

You can configure the API key using an environment variable.

For example:

export OPENAI_API_KEY=your-api-key

On Windows PowerShell:

$env:OPENAI_API_KEY="your-api-key"

Then reference it from your Spring configuration:

spring.ai.openai.api-key=${OPENAI_API_KEY}

The important part is that your API key should not be hard-coded inside your Java source code.

Avoid doing this:

String apiKey = "sk-xxxxxxxx";

Instead, keep secrets outside your source code.

A better approach is:

Environment Variable
        |
        v
Spring Configuration
        |
        v
Spring AI
        |
        v
OpenAI

This becomes even more important when deploying your application to production.

Step 4: Configure the Chat Model

Spring AI needs to know which OpenAI chat model your application should use.

You can configure the model through your application properties.

For example:

spring.ai.openai.chat.options.model=your-model

The exact model you choose depends on the models available to your OpenAI account and the requirements of your application.

The important concept is that your application does not need to manually construct OpenAI HTTP requests.

Spring AI handles that integration.

Step 5: Create the ChatController

Now let's create our first AI-powered controller.

@RestController
@RequestMapping("/api")
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    @GetMapping("/chat")
    public String chat(@RequestParam("message") String message) {

        return chatClient
                .prompt(message)
                .call()
                .content();
    }
}

Let's understand this carefully.

Understanding ChatClient Injection

The constructor receives:

ChatClient.Builder chatClientBuilder

Spring AI provides this builder through Spring Boot auto-configuration when the appropriate model integration is configured.

We then create our ChatClient:

this.chatClient = chatClientBuilder.build();

Now the controller has a ready-to-use AI client.

The architecture looks like:

Spring Boot
     |
     v
ChatClient.Builder
     |
     | build()
     v
ChatClient

This is the same concept we explored in Part 2 of this series.

Step 6: Send Your First Prompt

Now let's call the API.

Start your Spring Boot application.

Then send:

GET /api/chat?message=What is dependency injection in Spring?

The controller receives:

What is dependency injection in Spring?

and passes it to:

chatClient
        .prompt(message)
        .call()
        .content();

The flow is:

HTTP Request
     |
     v
ChatController
     |
     v
ChatClient
     |
     v
ChatModel
     |
     v
OpenAI
     |
     v
AI Response

The generated response is returned to the client.

That's it.

You have now built a Spring Boot application that can communicate with an AI model.

Breaking Down the ChatClient API

Let's look at this code again:

chatClient
        .prompt(message)
        .call()
        .content();

There are three important operations here.

prompt()

.prompt(message)

This defines the prompt that you want to send to the model.

For example:

.prompt("Explain Java interfaces")

or:

.prompt("Write a SQL query to find duplicate users")

or:

.prompt(message)

where message comes from an HTTP request.

call()

.call()

This executes the model interaction.

You can think about it as:

Build Prompt
     |
     v
Call Model

content()

.content()

This extracts the generated text from the response.

So the entire chain:

chatClient
        .prompt(message)
        .call()
        .content();

can be mentally understood as:

Create Prompt
     |
     v
Call AI Model
     |
     v
Extract Text

This fluent style is one of the reasons ChatClient is convenient for application developers.

Step 7: Add a Service Layer

Although putting the AI call directly inside a controller works for a small demonstration, it is not how I would structure a production application.

Instead, let's introduce a service.

Our architecture becomes:

Client
  |
  v
Controller
  |
  v
Service
  |
  v
ChatClient
  |
  v
OpenAI

Create:

@Service
public class ChatService {

    private final ChatClient chatClient;

    public ChatService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public String generateResponse(String message) {

        return chatClient
                .prompt(message)
                .call()
                .content();
    }
}

Then our controller becomes:

@RestController
@RequestMapping("/api")
public class ChatController {

    private final ChatService chatService;

    public ChatController(ChatService chatService) {
        this.chatService = chatService;
    }

    @GetMapping("/chat")
    public String chat(@RequestParam String message) {

        return chatService.generateResponse(message);
    }
}

This separation is much cleaner.

The controller handles HTTP.

The service handles AI interaction.

Why Separate the AI Logic?

Imagine that six months from now your application has:

ChatController
EmailController
SupportController
DocumentController

If every controller directly interacts with the AI model, your code can quickly become difficult to maintain.

Instead:

Controllers
     |
     v
AI Services
     |
     v
ChatClient
     |
     v
ChatModel

This keeps your application organized.

It also makes it easier to add features later.

Step 8: Add System Instructions

So far, we have only sent a user prompt.

But real AI applications usually need more control.

For example, imagine we are building a customer support assistant.

We don't want the model to behave like a generic chatbot.

We want to tell it:

You are a customer support assistant.
Answer clearly.
Keep responses concise.
Do not invent information.

We can do that with a system message.

For example:

return chatClient
        .prompt()
        .system("""
                You are a helpful customer support assistant.
                Answer clearly and concisely.
                Do not invent information.
                """)
        .user(message)
        .call()
        .content();

Now we have two different types of instructions:

System Message
      +
User Message
      |
      v
    Model

This distinction will become extremely important later in the series.

System Message vs User Message

Think about the two messages like this.

System message

Defines the behavior of the assistant.

You are a Java programming assistant.
Always provide production-quality examples.

User message

Contains the actual request.

Explain dependency injection.

Together:

System
  |
  | "You are a Java assistant"
  |
  v
User
  |
  | "Explain dependency injection"
  |
  v
AI Model

The PDF structure for this series introduces system and user messages as dedicated upcoming topics, so we will explore them in much more detail later.

Step 9: Build a Better AI Endpoint

Let's make our API slightly more realistic.

Instead of simply passing the user's message directly to the model, we can create a dedicated service method:

@Service
public class ChatService {

    private final ChatClient chatClient;

    public ChatService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public String ask(String question) {

        return chatClient
                .prompt()
                .system("""
                        You are a helpful Java and Spring Boot assistant.
                        Explain technical concepts clearly.
                        Use examples when appropriate.
                        """)
                .user(question)
                .call()
                .content();
    }
}

Then:

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatService chatService;

    public ChatController(ChatService chatService) {
        this.chatService = chatService;
    }

    @GetMapping
    public String chat(@RequestParam String message) {

        return chatService.ask(message);
    }
}

Now our API becomes:

GET /api/chat?message=What is Spring Boot?

and the service controls how the AI behaves.

What Happens Behind the Scenes?

This simple line:

chatClient
        .prompt()
        .user(message)
        .call()
        .content();

hides several operations.

Conceptually:

                    Spring Boot
                         |
                         v
                    ChatClient
                         |
                         v
                 Build Chat Request
                         |
                         v
                    ChatModel
                         |
                         v
                  OpenAI Integration
                         |
                         v
                    OpenAI API
                         |
                         v
                    AI Response
                         |
                         v
                    ChatClient
                         |
                         v
                       String

This abstraction is one of the major benefits of Spring AI.

You focus on your application.

Spring AI handles the model integration.

What If OpenAI Changes Its API?

This is another reason abstractions are useful.

Without Spring AI, you might build:

Spring Boot
     |
     v
Custom HTTP Client
     |
     v
OpenAI API

Your application would now contain provider-specific request and response handling.

With Spring AI:

Spring Boot
     |
     v
ChatClient
     |
     v
ChatModel
     |
     v
Provider

Your application code stays focused on the AI interaction rather than provider-specific implementation details.

This is the abstraction we discussed in the previous article.

A Better Mental Model

If you're a Spring Boot developer, think about Spring AI in layers.

Application Layer
       |
       v
   ChatClient
       |
       v
    ChatModel
       |
       v
   AI Provider
       |
       v
    AI Model

Each layer has a responsibility.

Your application

Business logic.

ChatClient

Developer-friendly AI interaction.

ChatModel

Model/provider abstraction.

AI Provider

OpenAI, Ollama, AWS Bedrock, and other supported providers.

AI Model

The actual language model generating the response.

Handling API Keys Correctly

One of the biggest mistakes beginners make is committing API keys to Git.

Never do this:

spring.ai.openai.api-key=sk-your-secret-key

inside a repository that will be shared publicly.

Instead:

spring.ai.openai.api-key=${OPENAI_API_KEY}

and configure the environment variable separately.

For local development:

OPENAI_API_KEY

For production, use your deployment platform's secret management mechanism.

The principle is simple:

Source Code
     X
     |
     | No secrets
     |
Environment / Secret Store
     |
     v
Spring Boot

Common Mistakes

Mistake 1: Putting the API Key in Git

Never commit secrets.

If your key is exposed, rotate it immediately.

Mistake 2: Calling the AI Model Directly From Every Controller

Avoid this structure:

Controller 1 -> AI
Controller 2 -> AI
Controller 3 -> AI
Controller 4 -> AI

Prefer:

Controllers
     |
     v
Services
     |
     v
ChatClient

Mistake 3: Treating ChatClient as the Model

Remember:

ChatClient != AI Model

ChatClient is the application-facing abstraction.

The underlying model integration is handled through ChatModel.

Mistake 4: Sending Every Prompt Without Instructions

A production AI application usually needs some control over model behavior.

Instead of:

.prompt(message)

you will often evolve toward:

.prompt()
.system("...")
.user(message)

Later, we will see how prompt templates, advisors, memory, RAG, and tools make this even more powerful.

Mistake 5: Starting With Complex AI Architecture

You don't need this on day one:

RAG
 +
Vector Database
 +
Tools
 +
MCP
 +
Memory
 +
Agents
 +
Multiple Models

Start with:

Spring Boot
     |
     v
ChatClient
     |
     v
OpenAI

Then add complexity when your application actually needs it.

From Simple Chat to Real AI Backend

Our application currently looks simple:

Client
   |
   v
REST API
   |
   v
ChatClient
   |
   v
OpenAI

But this architecture can grow.

For example:

                    Spring Boot
                         |
                         v
                    ChatClient
                         |
          +--------------+--------------+
          |              |              |
          v              v              v
        Memory          RAG           Tools
          |              |              |
          v              v              v
       History      Vector Store    Backend APIs

This is where Spring AI becomes much more interesting.

A simple chatbot can eventually become a complete AI backend.

The Complete Example

Here is a simple production-style starting point.

ChatService

@Service
public class ChatService {

    private final ChatClient chatClient;

    public ChatService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public String ask(String question) {

        return chatClient
                .prompt()
                .system("""
                        You are a helpful Java and Spring Boot assistant.
                        Explain concepts clearly and provide examples when useful.
                        """)
                .user(question)
                .call()
                .content();
    }
}

ChatController

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatService chatService;

    public ChatController(ChatService chatService) {
        this.chatService = chatService;
    }

    @GetMapping
    public String chat(@RequestParam String message) {

        return chatService.ask(message);
    }
}

Configuration

spring.ai.openai.api-key=${OPENAI_API_KEY}

spring.ai.openai.chat.options.model=your-model

The result is a clean architecture:

HTTP Client
    |
    v
ChatController
    |
    v
ChatService
    |
    v
ChatClient
    |
    v
ChatModel
    |
    v
OpenAI

What You Learned

In this article, we built our first Spring AI application using OpenAI.

We learned how to:

Create a Spring Boot project
Add Spring AI's OpenAI integration
Configure an OpenAI API key
Configure a chat model
Inject ChatClient.Builder
Build a ChatClient
Send prompts to an AI model
Create an AI-powered REST endpoint
Separate controller and AI service logic
Use system and user messages
Keep API keys outside source code

Most importantly, you now understand the complete request flow:

Client
   |
   v
Spring Boot
   |
   v
Controller
   |
   v
Service
   |
   v
ChatClient
   |
   v
ChatModel
   |
   v
OpenAI

This is the foundation for everything we will build later.

What's Next?

A hosted AI model is useful, but what if you want to run an LLM locally?

Maybe you don't want to send your data to an external provider.

Maybe you want to experiment without paying for API usage.

Maybe you're building an application where local inference is important.

That's where Ollama comes in.

In the next article, we will explore:

Part 4: Run Local LLMs with Ollama and Spring AI

We will install Ollama, run a local model, connect it to Spring AI, and see how the same ChatClient application can work with a local LLM.

That is one of the most interesting parts of Spring AI:

Same Application
       |
       +------> OpenAI
       |
       +------> Ollama
       |
       +------> Other Providers

The application stays focused on the AI interaction while Spring AI handles the underlying model integration.

Frequently Asked Questions

What is Spring AI?

Spring AI is an abstraction layer that makes it easier for Spring applications to integrate with AI models and AI-related capabilities.

Can I use OpenAI with Spring Boot?

Yes. Spring AI provides an OpenAI integration that allows Spring Boot applications to communicate with OpenAI models.

Where should I store my OpenAI API key?

Do not hard-code it in your source code. Use environment variables or a proper secret-management solution.

Do I need ChatModel directly?

For most application-level use cases, you can work primarily with ChatClient. ChatModel remains important because it represents the underlying model integration.

Can I change the AI provider later?

One of the goals of Spring AI's abstraction model is to reduce application coupling to provider-specific implementation details.

What should I learn after this?

The next step in this series is running a local LLM with Ollama and Spring AI.

Spring AI Series

Part 1: Spring AI Tutorial: How Java Developers Can Build Generative AI Applications with Spring Boot

Part 2: ChatModel vs ChatClient in Spring AI

Part 3: Build Your First Spring AI Application with OpenAI

Part 4: Run Local LLMs with Ollama and Spring AI

Part 5: Run AI Models Locally with Docker Model Runner

Part 6: Using AWS Bedrock with Spring AI

Part 7: Working with Multiple Chat Models in Spring AI

Part 8: Understanding Message Roles in LLMs

Part 9: System Messages and User Messages in Spring AI

Part 10: Configuring Default Behavior in ChatClient

The series will continue into prompt templates, advisors, structured output, tokens, embeddings, chat memory, RAG, vector stores, tool calling, MCP, evaluation, observability, and AI agents.

ChatModel vs ChatClient in Spring AI: Understanding the Core Abstractions

Ayush Shrivastava — Wed, 12 Aug 2026 11:53:42 +0000

In the first article of this Spring AI series, we learned what Spring AI is and built a simple Generative AI application using Spring Boot.

Now it is time to understand two of the most important concepts in Spring AI:

ChatModel
ChatClient

If you are coming from Java and Spring Boot, understanding the difference between these two will make the rest of Spring AI much easier to learn.

The simple relationship is:

Your Java Code
      |
      v
  ChatClient
      |
      v
   ChatModel
      |
      v
 AI Provider

The ChatClient provides the developer-friendly API, while the ChatModel represents the underlying AI model integration.

Let's understand this with a real example.

What Is ChatModel in Spring AI?

ChatModel is a lower-level abstraction for communicating with an AI model.

It provides the contract that Spring AI uses to interact with different AI providers.

For example:

OpenAI
Azure OpenAI
Google
Mistral

Spring AI can provide provider-specific implementations such as:

OpenAiChatModel
GeminiChatModel
MistralChatModel

The purpose of ChatModel is to hide provider-specific communication details behind a common abstraction.

Think about it like this:

ChatModel
    |
    +-- OpenAI
    |
    +-- Gemini
    |
    +-- Mistral
    |
    +-- Other providers

Your application can work with the abstraction instead of directly building HTTP requests for every provider.

What Is ChatClient?

ChatClient is a higher-level abstraction built on top of ChatModel.

It provides a fluent API that makes it easier to work with AI models.

For example:

String response = chatClient
        .prompt("Explain Spring Boot dependency injection")
        .call()
        .content();

Instead of manually handling the lower-level model interaction, you work with a much simpler API.

The PDF describes ChatClient as handling prompt construction, chat history, model invocation, and extracting response content.

ChatModel vs ChatClient

Here is the easiest way to remember the difference:

ChatModel	ChatClient
Lower-level abstraction	Higher-level abstraction
Represents AI model integration	Developer-friendly interface
Handles provider communication	Builds and manages prompts
More technical	Fluent and easier to use
Used underneath ChatClient	Uses ChatModel internally

A simple analogy from the course material is:

ChatModel  = Engine

ChatClient = Steering wheel + Dashboard

The engine performs the actual work.

The steering wheel and dashboard give you an easier way to control it.

How ChatClient and ChatModel Work Together

Suppose you write:

chatClient
        .prompt("What is Spring Boot?")
        .call()
        .content();

What actually happens?

The flow is approximately:

Your Java Code
      |
      v
ChatClient
      |
      v
Prompt
      |
      v
ChatModel
      |
      v
AI Provider API
      |
      v
AI Response
      |
      v
ChatClient
      |
      v
String Response

Spring Boot auto-configuration creates the appropriate ChatModel based on your configured provider.

Spring AI then provides an auto-configured ChatClient.Builder that is wired to the model.

When you call the fluent ChatClient API, the request eventually delegates to the underlying ChatModel.

Let's Build a Real Example

Let's create a simple Spring Boot API:

GET /api/chat?message=Explain dependency injection

The API should send the message to an AI model and return the generated response.

Our architecture will be:

Client
  |
  | GET /api/chat
  v
Spring Boot Controller
  |
  v
ChatClient
  |
  v
ChatModel
  |
  v
AI Provider

Step 1: Create the ChatController

Create:

@RestController
@RequestMapping("/api")
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    @GetMapping("/chat")
    public String chat(@RequestParam("message") String message) {

        return chatClient
                .prompt(message)
                .call()
                .content();
    }
}

The important part is:

ChatClient.Builder

Spring Boot provides this builder through auto-configuration when the appropriate Spring AI model dependency is present.

Then we create our client:

this.chatClient = chatClientBuilder.build();

Now we can use it anywhere inside our service or controller.

Step 2: Send a Prompt

Let's send:

GET /api/chat?message=What is dependency injection in Spring?

The code:

chatClient
        .prompt(message)
        .call()
        .content();

can be understood as:

prompt()
    |
    | Build the request
    v
call()
    |
    | Execute the model call
    v
content()
    |
    | Extract generated text
    v
String

This is one of the biggest advantages of ChatClient.

The API is simple enough that a Java developer can understand what is happening without dealing directly with provider-specific request objects.

Why Not Use ChatModel Directly?

You might now ask:

If ChatModel communicates with the AI provider, why do we need ChatClient?

You can work at the ChatModel level when you need lower-level control.

But most application developers usually want to express something like:

chatClient
        .prompt("Explain Java records")
        .call()
        .content();

rather than manually constructing model requests.

The ChatClient abstraction provides:

Fluent APIs
Prompt construction
Message handling
Model invocation
Response extraction
Synchronous and streaming programming models

These are some of the capabilities highlighted in the Spring AI course material.

A Spring Boot Developer's Mental Model

If you already know Spring Data, you can think about abstractions.

For example:

Spring Data
    |
    v
Repository Abstraction
    |
    v
Database Implementation

Similarly:

Spring AI
    |
    v
ChatClient
    |
    v
ChatModel
    |
    v
AI Provider

The goal is to avoid coupling your application code unnecessarily to the implementation details.

What Happens During Spring Boot Auto-Configuration?

One of the useful features of Spring AI is Spring Boot auto-configuration.

When you add the appropriate Spring AI dependency and configuration, Spring Boot can create the necessary model bean.

Conceptually:

Spring Boot Application Starts
            |
            v
Spring AI Configuration
            |
            v
Create ChatModel Bean
            |
            v
Create ChatClient.Builder
            |
            v
Your Controller

You don't have to manually create every object.

You simply inject:

ChatClient.Builder

and build the client:

this.chatClient = chatClientBuilder.build();

This follows the familiar Spring dependency injection model.

ChatClient Is More Than a Simple Wrapper

It might initially look like ChatClient is just a shortcut for calling the model.

But it becomes much more useful as your application grows.

For example, Spring AI allows you to configure things such as:

System instructions
Advisors
Tools
Options
Prompt templates
Chat memory
Streaming

The PDF later introduces these capabilities as part of the Spring AI journey.

This means your code can evolve from:

chatClient
        .prompt(message)
        .call()
        .content();

to something more sophisticated:

chatClient
        .prompt()
        .system("You are a helpful support assistant.")
        .user(message)
        .call()
        .content();

And eventually:

ChatClient
   |
   +-- System Instructions
   |
   +-- Memory
   |
   +-- Advisors
   |
   +-- Tools
   |
   +-- RAG
   |
   +-- Model

This is why learning ChatClient early is important.

ChatClient Builder vs ChatClient

There are two concepts you will frequently see:

ChatClient.Builder

and:

ChatClient

They are not the same thing.

The builder is used to configure and create a ChatClient.

For example:

public ChatController(ChatClient.Builder chatClientBuilder) {

    this.chatClient = chatClientBuilder.build();
}

After calling:

build()

you have:

ChatClient

You can then use:

chatClient.prompt(...)

Think about it as:

ChatClient.Builder
        |
        | build()
        v
   ChatClient
        |
        | prompt()
        v
   ChatModel

Using ChatClient With a Local Model

One of the advantages of Spring AI is that you are not restricted to a hosted AI provider.

The PDF introduces Ollama as a way to run LLMs locally and demonstrates configuring Spring AI to use it.

For example, you can run:

ollama run llama3.2:1b

Then configure:

spring.ai.model.chat=ollama
spring.ai.ollama.chat.model=llama3.2:1b

With the Ollama starter:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

Your controller can remain almost identical:

@RestController
@RequestMapping("/api")
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    @GetMapping("/chat")
    public String chat(@RequestParam String message) {
        return chatClient
                .prompt(message)
                .call()
                .content();
    }
}

This is a powerful concept.

Your application code can remain focused on the AI interaction rather than provider-specific implementation details.

What If We Need Multiple Models?

This is where things become more interesting.

Imagine you have a production application with different requirements.

You might want:

Simple Questions
        |
        v
Fast / Cheap Model

Complex Reasoning
        |
        v
Powerful Model

Local Development
        |
        v
Ollama

Why would you want multiple models?

The PDF identifies several real-world reasons:

Task-Based Model Selection

Use different models for different workloads.

For example:

Simple FAQ
   -> Lightweight model

Complex analysis
   -> More powerful model

Fallback Strategy

If one model is unavailable:

Primary Model
      |
      X
      |
      v
Fallback Model

A/B Testing

You can compare models based on:

Accuracy
Latency
Cost
User experience

User Preferences

Some applications may allow users to select a model.

Specialized Models

You could use one model for coding and another for creative content.

These scenarios are explicitly discussed in the PDF.

Multiple ChatClients

Spring AI's default configuration provides a single ChatClient.Builder.

That is enough for simple applications.

But if your application needs multiple models, you may need to configure multiple clients yourself.

The course material demonstrates disabling the default ChatClient builder auto-configuration:

spring.ai.chat.client.enabled=false

and creating multiple ChatClient instances manually.

For example:

@Configuration
public class ChatClientConfig {

    @Bean
    public ChatClient openAiChatClient(OpenAiChatModel chatModel) {
        return ChatClient.create(chatModel);
    }

    @Bean
    public ChatClient ollamaChatClient(OllamaChatModel chatModel) {
        return ChatClient
                .builder(chatModel)
                .build();
    }
}

Now you can have:

OpenAI ChatClient
       |
       v
OpenAI Model


Ollama ChatClient
       |
       v
Ollama Model

This gives your application more flexibility.

A Real-World Architecture

Imagine you're building an AI-powered customer support platform.

You could have:

                 Customer
                    |
                    v
              Spring Boot API
                    |
                    v
                ChatClient
                    |
          +---------+---------+
          |                   |
          v                   v
     Fast Model         Powerful Model
          |                   |
          v                   v
     Simple FAQs       Complex Requests

Later, we might add:

                 ChatClient
                     |
       +-------------+-------------+
       |             |             |
       v             v             v
    Memory          RAG          Tools
       |             |             |
       v             v             v
   History       Company Docs   Backend APIs

This is the direction in which a simple Spring AI application grows into a real AI backend.

ChatModel or ChatClient: Which One Should You Use?

For most application-level Spring AI development:

Prefer ChatClient

because it provides the higher-level API and integrates naturally with prompts, advisors, memory, tools, and other application concerns.

You should understand ChatModel because it is the underlying abstraction that connects your application to the actual model provider.

The mental model is:

ChatClient
    |
    | High-level developer API
    v
ChatModel
    |
    | Provider integration
    v
AI Model

Common Mistakes

Mistake 1: Thinking ChatClient Is the AI Model

It is not.

ChatClient != LLM

ChatClient is the interface your application uses to interact with the model.

Mistake 2: Thinking ChatModel and ChatClient Are Competing Alternatives

They serve different abstraction levels.

ChatClient
    |
    v
ChatModel

ChatClient uses ChatModel internally.

Mistake 3: Hard-Coding Provider Logic Everywhere

Avoid building your application around provider-specific HTTP calls when Spring AI already provides abstractions.

Keep your application logic separated from the AI provider whenever possible.

Mistake 4: Creating Multiple Clients Without a Reason

Multiple models can be useful, but they also introduce additional configuration and operational complexity.

Use them when you actually need:

Fallbacks
Different workloads
A/B testing
Cost optimization
Specialized models

Final Takeaway

The most important thing to remember from this article is:

ChatClient
    |
    | Developer-friendly API
    v
ChatModel
    |
    | Model/provider integration
    v
AI Provider

ChatModel is the lower-level abstraction responsible for communicating with AI providers.

ChatClient provides the higher-level fluent API that makes AI interactions easier to build and maintain.

For a typical Spring Boot application, you will spend much of your application-level development working with:

ChatClient

while Spring AI handles the underlying model integration.

In the next article, we will take this knowledge and build a complete Spring AI application with OpenAI, starting from project setup and configuration to creating a production-style REST endpoint.

Frequently Asked Questions

What is ChatModel in Spring AI?

ChatModel is the lower-level abstraction used to communicate with an AI model provider.

What is ChatClient?

ChatClient is a higher-level Spring AI API that makes it easier to build prompts, invoke models, manage responses, and integrate additional AI application features.

Is ChatClient better than ChatModel?

They serve different purposes. ChatClient is generally more convenient for application development, while ChatModel provides the lower-level model abstraction.

Can ChatClient work with Ollama?

Yes. The course material demonstrates using ChatClient with an Ollama-backed model.

Can Spring AI use multiple AI models?

Yes. Spring AI can be configured with multiple ChatClient instances for different models and use cases.

Spring AI Series

Part 1: Spring AI Tutorial: How Java Developers Can Build Generative AI Applications with Spring Boot

Part 2: ChatModel vs ChatClient in Spring AI

Part 3: Build Your First Spring AI Application with OpenAI

Part 4: Run Local LLMs with Ollama and Spring AI

Part 5: Run AI Models Locally with Docker Model Runner

Part 6: Using AWS Bedrock with Spring AI

Part 7: Working with Multiple Chat Models in Spring AI

Part 8: Understanding Message Roles in LLMs

Part 9: System Messages and User Messages in Spring AI

Part 10: Configuring Default Behavior in ChatClient

More articles will cover prompt templates, advisors, structured output, tokens, embeddings, chat memory, RAG, vector stores, tool calling, MCP, evaluation, observability, and AI agents.

Spring AI Tutorial: How Java Developers Can Build Generative AI Applications with Spring Boot

Ayush Shrivastava — Tue, 11 Aug 2026 10:43:51 +0000

If you are a Java developer working with Spring Boot, you already know how to build REST APIs, microservices, database-driven applications, authentication systems, and production backends.

But AI development introduces a new set of questions:

How do I connect an LLM to my Spring Boot application?
How do I send prompts from Java?
How do I work with OpenAI or local models?
How do I give an AI application access to my application's data?
How can an LLM call my Java methods?
How do I build RAG applications with Spring Boot?
How do I build AI agents instead of simple chatbots?

This is where Spring AI becomes interesting.

Spring AI brings AI application development into the Spring ecosystem and provides abstractions for working with AI models, embeddings, vector stores, memory, tools, and other AI application patterns.

According to the Spring AI roadmap in this series, we will move from a simple AI application to concepts such as RAG, tool calling, MCP, evaluation, observability, and multimodal applications.

In this article, we will start from the beginning.

What Is Spring AI?

Spring AI is a framework for integrating AI capabilities into Spring applications.

The idea is simple:

Instead of learning a completely different programming model for AI, Spring developers can use familiar Spring concepts and abstractions to communicate with AI models.

Spring AI provides integrations with multiple AI providers and supports capabilities such as chat models, embeddings, vector stores, memory, tool calling, MCP, and AI observability.

You can think about the architecture like this:

Spring Boot Application
        |
        v
     Spring AI
        |
        +------------------+
        |                  |
        v                  v
   Chat Model          Embedding Model
        |                  |
        v                  v
     LLM API          Vector Store

The important part is that your business application remains a Spring Boot application.

You are simply adding an AI layer to it.

Why Should Java Developers Learn Spring AI?

Imagine that you are building an employee management application.

Your application already has:

Java
Spring Boot
Spring Security
PostgreSQL
REST APIs
Docker
AWS

Now your product team asks:

"Can we add an AI assistant that answers employee questions?"

For example:

User:
How many paid leaves can I take every year?

AI Assistant:
According to the company policy, employees are eligible
for 18 days of paid leave annually.

A traditional backend developer might think:

Frontend
   |
   v
Spring Boot
   |
   v
Database

But an AI-powered application may look more like:

Frontend
   |
   v
Spring Boot
   |
   v
Spring AI
   |
   +------> LLM
   |
   +------> Vector Database
   |
   +------> Internal APIs
   |
   +------> Business Logic

This is the shift from building traditional backend systems to building AI-powered backend systems.

Spring AI provides the abstractions needed to connect these pieces.

Spring AI Is Not an AI Model

This distinction is important.

Spring AI is not itself an LLM.

It is an application framework that helps your Spring application communicate with AI models.

For example:

Your Spring Boot Application
            |
            v
        Spring AI
            |
       +----+----+
       |         |
       v         v
    OpenAI     Ollama

The model is responsible for generating the response.

Spring AI provides the developer-friendly interface for communicating with that model.

This separation is useful because your business logic does not have to be tightly coupled to a single AI provider.

What Can You Build With Spring AI?

Spring AI can be used for much more than a basic chatbot.

Some examples include:

1. AI Customer Support

A customer asks:

Where is my order?

Your application can combine AI with backend APIs to retrieve order information and generate a natural response.

2. Document Question Answering

Suppose your company has:

employee-policy.pdf
leave-policy.pdf
insurance-policy.pdf

Instead of manually searching these documents, users can ask:

How many annual leaves do I get?

A RAG pipeline can retrieve the relevant document content and provide it to the LLM. The PDF's roadmap specifically covers chunking, embeddings, vector stores, and document retrieval for this purpose.

3. AI-Powered Recommendations

You can use embeddings to understand user intent and build semantic recommendations.

4. AI Data Processing

Spring AI can also be used for:

Text generation
Content moderation
Transcription
Summarization
Classification

These are among the use cases highlighted in the course material.

5. AI Agents

Eventually, your application can move beyond simply generating text.

An AI system can:

Understand request
      |
      v
Retrieve information
      |
      v
Call Java method
      |
      v
Call external API
      |
      v
Make another decision
      |
      v
Return final response

This is where concepts such as tool calling and agentic AI become important.

The First Spring AI Application

Let's build something simple.

Our first application will accept a message through a REST API and send that message to an AI model.

The flow is:

HTTP Request
     |
     v
Spring Boot Controller
     |
     v
ChatClient
     |
     v
ChatModel
     |
     v
AI Provider
     |
     v
Response

This is intentionally simple.

The goal is to understand the basic Spring AI programming model before introducing RAG, memory, tools, or agents.

Step 1: Create a Spring Boot Project

Create a Spring Boot project with:

Java
Spring Boot
Spring Web
Spring AI

The PDF uses the Spring Web MVC starter and the Spring AI OpenAI model starter for the initial example.

Your Maven dependencies can look like:

<dependencies>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webmvc</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-openai</artifactId>
    </dependency>

</dependencies>

The exact Spring AI dependency/version should be aligned with the Spring AI version you choose for your project.

Step 2: Configure the API Key

If you are using a hosted model provider, you need credentials to communicate with it.

Do not hard-code API keys inside your Java source code.

Instead, use an environment variable.

For example:

spring.ai.openai.api-key=${OPENAI_API_KEY}

Then configure:

OPENAI_API_KEY=your-api-key

The course material also follows this approach by resolving the API key from an environment variable.

This is much better than:

String apiKey = "sk-xxxxxxxx";

Never commit secrets into Git.

Step 3: Create the Chat Controller

Now let's create our first Spring AI endpoint.

@RestController
@RequestMapping("/api")
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    @GetMapping("/chat")
    public String chat(@RequestParam("message") String message) {

        return chatClient
                .prompt(message)
                .call()
                .content();
    }
}

This is the core of our first application.

The PDF uses the same basic approach: inject ChatClient.Builder, build a ChatClient, provide a user message, call the model, and extract the response content.

Step 4: Run the Application

Start your Spring Boot application.

Now make a request:

GET /api/chat?message=What is Spring Boot?

The request reaches:

ChatController
      |
      v
ChatClient
      |
      v
AI Model
      |
      v
Generated Response

You might receive something like:

Spring Boot is a framework built on top of Spring
that simplifies the development of Java applications...

You have now created your first Spring AI application.

Understanding ChatClient

The most important class in this example is:

ChatClient

ChatClient provides a higher-level API for interacting with an AI model.

It gives us a fluent programming style:

chatClient
        .prompt(message)
        .call()
        .content();

This is much easier to work with than manually constructing provider-specific HTTP requests.

The course material describes ChatClient as the higher-level developer-friendly abstraction that handles prompt construction, message handling, model invocation, and extracting content from the response.

ChatModel vs ChatClient

This is one of the first concepts you should understand when learning Spring AI.

There are two important abstractions:

ChatClient
    |
    v
ChatModel
    |
    v
AI Provider

What Is ChatModel?

ChatModel represents the lower-level interface for communicating with an AI model.

It handles things such as:

Communication with AI providers
Model-specific configuration
API interaction
AI request and response handling

The PDF describes implementations such as:

OpenAiChatModel
GeminiChatModel
MistralChatModel

depending on the provider being used.

What Is ChatClient?

ChatClient sits at a higher level.

Instead of worrying about the underlying provider API, you can write:

chatClient
        .prompt("Explain dependency injection")
        .call()
        .content();

The relationship can be visualized as:

Your Java Code
      |
      v
   ChatClient
      |
      v
   ChatModel
      |
      v
AI Provider API

A useful mental model is:

ChatModel  = Engine

ChatClient = Steering wheel + Dashboard

The engine does the actual work.

The dashboard gives you a convenient interface for controlling it.

The course material describes this same relationship between ChatClient and ChatModel.

Why Is This Abstraction Useful?

Imagine your application starts with one AI provider.

Spring Boot
    |
    v
Spring AI
    |
    v
Provider A

Later, you decide that some workloads should use another model.

You don't want your entire application to become:

OpenAIHttpClient
OpenAIRequest
OpenAIResponse
OpenAIAuthentication
OpenAIModelConfiguration

everywhere.

Instead, Spring AI provides abstractions that allow you to work at the application level.

This becomes particularly valuable when you have multiple models.

For example:

Simple Question
      |
      v
Lightweight Model

Complex Reasoning
      |
      v
Powerful Model

Local Development
      |
      v
Ollama

The course material identifies task-based model selection, fallback strategies, A/B testing, user preferences, and specialized models as reasons to work with multiple chat models.

Spring AI Is Not Limited to OpenAI

One of the important benefits of Spring AI is that it is designed around multiple providers.

The course material covers:

OpenAI
Ollama
Docker Model Runner
AWS Bedrock

as part of the introductory journey.

For example, Ollama allows developers to run models locally.

The course demonstrates:

ollama run llama3.2:1b

and then configures Spring AI to use the local model.

This can be useful during development when you want to experiment with local models instead of depending entirely on a hosted API.

What About AWS Bedrock?

If you are already building applications on AWS, another option covered in the course is Amazon Bedrock.

Bedrock is a managed AWS service for building generative AI applications and provides access to foundation models from multiple vendors.

Your architecture can therefore look like:

Spring Boot
     |
     v
Spring AI
     |
     v
AWS Bedrock
     |
     v
Foundation Model

This is particularly interesting for enterprise applications that already operate inside AWS environments.

The Bigger Spring AI Journey

Our first example is intentionally tiny.

In a production application, simply sending a prompt to an LLM is rarely enough.

Consider a customer support application.

A basic version might look like:

User
 |
 v
ChatClient
 |
 v
LLM
 |
 v
Answer

But a real production system may need:

                    +----------------+
                    |  Chat Memory   |
                    +-------+--------+
                            |
                            v
User --> ChatClient --> Advisor --> LLM
                            |
             +--------------+--------------+
             |                             |
             v                             v
        Vector Store                  Java Tools
             |                             |
             v                             v
        Company Docs                 Backend APIs

And then we can introduce:

RAG
Embeddings
Vector Databases
Tool Calling
MCP
Evaluators
Observability
Memory
Streaming
Structured Output

These concepts form the larger Spring AI journey outlined in the PDF.

From Chatbots to AI Applications

There is an important progression to understand.

A simple LLM application looks like:

LLM

It can answer questions based primarily on what it learned during training.

Then we add external knowledge:

LLM + RAG

Now it can retrieve information from our documents or knowledge base.

Then we add tools:

LLM + RAG + Tools

Now it can retrieve information and interact with external systems.

Finally, we can move toward agentic systems:

Agentic AI

where the application can reason through tasks, use tools, retrieve information, and execute multiple steps.

The PDF presents this progression from LLM to RAG, tools, and agentic AI as the broader generative AI journey.

This is where Spring AI becomes particularly interesting for backend developers.

A Real-World Example

Let's take an e-commerce application.

A user asks:

Where is my order and when will it arrive?

A normal LLM cannot automatically know the user's latest order.

Your backend might have:

PostgreSQL
    |
    +-- orders
    +-- customers
    +-- payments
    +-- shipments

The AI application needs access to that real-time information.

A future Spring AI application could look like:

User
 |
 v
Spring Boot API
 |
 v
Spring AI
 |
 +----> LLM
 |
 +----> Order Tool
 |          |
 |          v
 |       Database
 |
 +----> Shipping Tool
            |
            v
        Shipping API

Now the AI is not simply generating text.

It is interacting with your backend.

This is the foundation for tool calling, which the course introduces as a way to allow an AI application to access current data and perform actions through Java code.

What We Will Build Throughout This Series

This article is only the starting point.

The complete series will gradually move from basic Spring AI concepts to production-oriented AI application development.

The learning path will look approximately like this:

Spring AI Fundamentals
        |
        v
ChatClient and ChatModel
        |
        v
Prompts and Message Roles
        |
        v
Prompt Templates
        |
        v
Chat Options
        |
        v
Structured Output
        |
        v
Generative AI Fundamentals
        |
        v
Tokens and Embeddings
        |
        v
Chat Memory
        |
        v
RAG
        |
        v
Vector Databases
        |
        v
Tool Calling
        |
        v
MCP
        |
        v
AI Evaluation
        |
        v
Observability
        |
        v
AI Agents
        |
        v
Production AI Applications

The source material specifically includes prompt templates, streaming, ChatOptions, advisors, prompt stuffing, structured output, memory, RAG, tool calling, MCP, evaluators, observability, voice, and image generation.

What You Should Know Before Starting

This series is designed primarily for developers who already understand Java and Spring Boot.

You should be comfortable with:

Java
Spring Boot
REST APIs
Basic Docker
Basic Postman

You do not need to be an AI researcher.

You do not need to understand neural networks before writing your first Spring AI application.

We will learn the AI concepts gradually and connect them back to backend development.

The source material also lists Java, Spring Boot, Docker, and Postman familiarity as prerequisites.

Final Thoughts

If you are already a Java and Spring Boot developer, learning AI application development does not mean throwing away everything you already know.

In fact, your backend experience is extremely useful.

You already understand:

APIs
Databases
Authentication
Security
Microservices
Caching
Cloud
Distributed Systems
Testing
Observability

Now you are adding another layer:

AI Models
Prompts
Embeddings
RAG
Tools
Memory
Agents

Spring AI helps bring these AI capabilities into the Spring ecosystem using familiar application-development patterns.

And that is the goal of this series:

Go from Java Developer to AI Engineer by building real AI-powered backend applications with Spring AI.

In the next article, we will go one level deeper into ChatModel vs ChatClient in Spring AI, understand how the two abstractions work together, and build practical examples using different AI providers.

Frequently Asked Questions

What is Spring AI?

Spring AI is a framework for integrating AI models and AI application capabilities into Spring applications.

Is Spring AI only for OpenAI?

No. The course material covers multiple providers and approaches, including OpenAI, Ollama, Docker Model Runner, and AWS Bedrock.

Do I need Python to learn Spring AI?

No. This series focuses on Java and Spring Boot.

Do I need to train my own AI model?

No. Spring AI is primarily about integrating existing AI models into applications.

Can Spring AI work with my existing backend?

Yes. That is one of the main reasons it is useful for Spring developers. AI capabilities can be added alongside existing APIs, databases, services, and business logic.

What will I learn after the basics?

The series will progressively cover prompts, memory, embeddings, RAG, vector stores, tool calling, MCP, evaluation, observability, and more.

Series Navigation

Part 1: Spring AI Tutorial: How Java Developers Can Build Generative AI Applications with Spring Boot

Part 2: ChatModel vs ChatClient in Spring AI

Part 3: Build Your First Spring AI Application with OpenAI

Part 4: Run Local LLMs with Ollama and Spring AI

Part 5: Run AI Models Locally with Docker Model Runner

Part 6: Using AWS Bedrock with Spring AI

Part 7: Understanding Multiple Chat Models in Spring AI

Part 8: Understanding AI Message Roles

Part 9: System Messages and User Messages in Spring AI

Part 10: Default Configuration in Spring AI ChatClient

The series will then move into prompt engineering, structured output, LLM fundamentals, tokens, embeddings, memory, RAG, vector databases, tool calling, MCP, evaluation, observability, and AI agents.

Java Spring Boot Logging: Log Levels, Logback, JSON Logs & Production Best Practices

Ayush Shrivastava — Sat, 08 Aug 2026 06:12:32 +0000

Production-Grade Logging in Spring Boot: A Complete Guide to Logging Levels, Files, JSON Logs, Correlation IDs, and Best Practices

Logging is one of the most important parts of a production backend system.

When everything works, you may not think much about logs.

But when production starts returning 500 errors at 2 AM, a customer reports that an API is failing, or a payment request behaves unexpectedly, logs become one of your most important debugging tools.

Poor logging makes production debugging painful.

Good logging helps you answer:

What happened?
When did it happen?
Which user triggered it?
Which request caused it?
Which service handled it?
How long did it take?
What failed?
Why did it fail?
What should we investigate next?

In this article, we will build a production-grade logging strategy for a Java Spring Boot application.

1. What Does Production-Grade Logging Mean?

Production-grade logging is not simply:

log.info("User created");

Production logging should be:

Structured
Searchable
Consistent
Secure
Configurable
Environment-aware
Correlated across requests
Useful during debugging
Suitable for monitoring and alerting

A good logging architecture might look like this:

Spring Boot Application
        |
        v
     Logback
        |
        +---- application.log
        |
        +---- error.log
        |
        +---- audit.log
        |
        +---- access.log
        |
        v
Log Aggregation
        |
        +---- ELK
        +---- Grafana Loki
        +---- CloudWatch
        +---- Datadog
        +---- Splunk

The goal is not to log everything.

The goal is to log the right information at the right level.

2. Understanding Log Levels

Spring Boot uses SLF4J as the logging abstraction and commonly uses Logback as the underlying logging implementation.

The most common log levels are:

TRACE
DEBUG
INFO
WARN
ERROR

The order represents increasing severity.

TRACE

TRACE is the most detailed logging level.

Example:

log.trace("Entering calculateInvoice() with customerId={}", customerId);

Use TRACE for very detailed diagnostic information.

Usually:

Production: OFF
Development: Sometimes ON
Debugging: Useful

Avoid keeping TRACE enabled globally in production because it can generate huge amounts of logs.

3. DEBUG

DEBUG is useful for developers.

Example:

log.debug("Fetching customer with customerId={}", customerId);

Another example:

log.debug("Payment request received for orderId={}", orderId);

DEBUG logs are useful when troubleshooting a specific feature.

A common production strategy is:

INFO  -> Default
DEBUG -> Temporarily enabled when troubleshooting

4. INFO

INFO should contain important application events.

For example:

log.info("User successfully created. userId={}", userId);

Or:

log.info("Order successfully created. orderId={}, customerId={}",
        orderId,
        customerId);

Good INFO logs might include:

Application started
User registered
Order created
Payment completed
File uploaded
Scheduled job completed
External integration connected

But don't log every line of your application at INFO.

5. WARN

WARN indicates something unexpected or potentially problematic.

Example:

log.warn("Login attempt failed. email={}", email);

Another example:

log.warn("Payment provider response time is high. durationMs={}",
        durationMs);

WARN means:

"The application is still functioning, but someone should pay attention."

Examples:

Retry occurred
External API is slow
Deprecated API was called
Configuration is missing but has a fallback
Login failed repeatedly
Database connection pool is close to its limit

6. ERROR

ERROR represents a failure that needs investigation.

Example:

log.error("Failed to create order. orderId={}", orderId, exception);

Notice that the exception is passed separately:

log.error("Failed to create order", exception);

Instead of:

log.error("Failed to create order " + exception.getMessage());

The first approach preserves the stack trace.

7. Never Log Sensitive Information

One of the most important production logging rules is:

Logs should never become a source of sensitive data leakage.

Never log:

Passwords
Access tokens
Refresh tokens
API keys
Credit card numbers
CVV
Session IDs
Private keys
Authorization headers

Bad:

log.info("Login request: username={}, password={}",
        username,
        password);

Good:

log.info("Login attempt received. username={}", username);

Even better, depending on your privacy requirements, avoid logging email addresses or other personal identifiers unless there is a clear operational reason.

8. Use Parameterized Logging

Avoid string concatenation.

Don't do this:

log.info("User created: " + userId);

Prefer:

log.info("User created. userId={}", userId);

For multiple values:

log.info(
    "Order created. orderId={}, customerId={}, amount={}",
    orderId,
    customerId,
    amount
);

Parameterized logging is cleaner and avoids unnecessary string construction.

9. Create a Centralized Logging Configuration

Spring Boot makes it easy to configure Logback.

You can create:

src/main/resources/logback-spring.xml

A basic configuration:

<?xml version="1.0" encoding="UTF-8"?>

<configuration>

    <property name="LOG_DIR" value="./logs"/>

    <appender name="CONSOLE"
              class="ch.qos.logback.core.ConsoleAppender">

        <encoder>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss.SSS}
                [%thread]
                %-5level
                %logger{36}
                -
                %msg%n
            </pattern>
        </encoder>

    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
    </root>

</configuration>

Now the application produces readable logs such as:

2026-08-08 10:30:25.123 [http-nio-8080-exec-1] INFO
c.example.user.UserService -
User created. userId=123

10. Different Log Files for Different Levels

For a production application, you may want separate files.

For example:

logs/
├── application.log
├── error.log
├── audit.log
└── access.log

This makes troubleshooting easier.

For example:

application.log

contains general application events.

error.log

contains ERROR events.

audit.log

contains important security/business events.

access.log

contains HTTP request information.

11. Creating an ERROR Log File

Example:

<appender name="ERROR_FILE"
          class="ch.qos.logback.core.rolling.RollingFileAppender">

    <file>${LOG_DIR}/error.log</file>

    <rollingPolicy
        class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

        <fileNamePattern>
            ${LOG_DIR}/archive/error.%d{yyyy-MM-dd}.%i.log.gz
        </fileNamePattern>

        <maxFileSize>100MB</maxFileSize>
        <maxHistory>30</maxHistory>
        <totalSizeCap>5GB</totalSizeCap>

    </rollingPolicy>

    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
        <level>ERROR</level>
    </filter>

    <encoder>
        <pattern>
            %d{yyyy-MM-dd HH:mm:ss.SSS}
            [%thread]
            %-5level
            %logger{36}
            -
            %msg%n
        </pattern>
    </encoder>

</appender>

Now ERROR logs can be stored separately.

12. Why Log Rotation Matters

Imagine your application generates:

application.log

and you never rotate it.

After several months:

application.log = 150 GB

Your server's disk eventually becomes full.

This can cause much bigger problems.

For example:

Application
    |
    v
Disk full
    |
    +---- Logging fails
    +---- Database operations may fail
    +---- Temporary files cannot be created
    +---- Application becomes unstable

That's why production applications need:

Maximum file size
Maximum history
Total storage limit
Compression
Time-based rotation

Example:

<maxFileSize>100MB</maxFileSize>
<maxHistory>30</maxHistory>
<totalSizeCap>5GB</totalSizeCap>

13. Application Logs vs Audit Logs

Not every important event is an application error.

Consider:

Admin disabled user 123

This isn't an ERROR.

It is an audit event.

Create a separate audit logger.

private static final Logger auditLogger =
        LoggerFactory.getLogger("AUDIT");

Then:

auditLogger.info(
    "User disabled. adminId={}, targetUserId={}",
    adminId,
    targetUserId
);

This gives you a separate audit stream.

14. Audit Logging Is Extremely Important

For systems involving multiple users, administrators, payments, permissions, or sensitive operations, audit logging becomes extremely valuable.

Examples:

USER_CREATED
USER_DISABLED
USER_ENABLED
PASSWORD_CHANGED
ROLE_CHANGED
LOGIN_SUCCESS
LOGIN_FAILED
API_KEY_CREATED
API_KEY_REVOKED
DATA_EXPORTED
PAYMENT_COMPLETED

Instead of:

log.info("Something happened");

Use structured information:

auditLogger.info(
    "AUDIT event=ROLE_CHANGED actorId={} targetUserId={} oldRole={} newRole={}",
    actorId,
    targetUserId,
    oldRole,
    newRole
);

Now the event can easily be searched.

15. Logging HTTP Requests

For backend systems, request logging is extremely useful.

You want to know:

HTTP Method
URL
Status Code
Execution Time
Request ID
User ID

Example:

GET /api/users/123
status=200
duration=45ms
requestId=9f7a2

A servlet filter is one approach.

@Component
public class RequestLoggingFilter extends OncePerRequestFilter {

    private static final Logger log =
            LoggerFactory.getLogger(RequestLoggingFilter.class);

    @Override
    protected void doFilterInternal(
            HttpServletRequest request,
            HttpServletResponse response,
            FilterChain filterChain)
            throws ServletException, IOException {

        long start = System.currentTimeMillis();

        try {
            filterChain.doFilter(request, response);
        } finally {

            long duration =
                    System.currentTimeMillis() - start;

            log.info(
                "HTTP request method={} uri={} status={} durationMs={}",
                request.getMethod(),
                request.getRequestURI(),
                response.getStatus(),
                duration
            );
        }
    }
}

This gives you a basic access log.

16. Correlation IDs

This is one of the most useful concepts in distributed systems.

Imagine a request:

Frontend
   |
   v
API Gateway
   |
   v
User Service
   |
   v
Payment Service
   |
   v
Notification Service

One request could generate dozens of logs.

How do you identify which logs belong to the same request?

Use a:

Correlation ID

Example:

requestId=7f83ab29

Then every service logs:

requestId=7f83ab29

Now you can search the entire system using that ID.

17. Implementing Correlation ID with MDC

SLF4J provides MDC:

MDC.put("requestId", requestId);

Example:

@Component
public class CorrelationIdFilter extends OncePerRequestFilter {

    private static final String REQUEST_ID = "requestId";

    @Override
    protected void doFilterInternal(
            HttpServletRequest request,
            HttpServletResponse response,
            FilterChain filterChain)
            throws ServletException, IOException {

        String requestId = request.getHeader(REQUEST_ID);

        if (requestId == null || requestId.isBlank()) {
            requestId = UUID.randomUUID().toString();
        }

        MDC.put(REQUEST_ID, requestId);

        response.setHeader(REQUEST_ID, requestId);

        try {
            filterChain.doFilter(request, response);
        } finally {
            MDC.remove(REQUEST_ID);
        }
    }
}

Now add it to Logback:

<pattern>
    %d{yyyy-MM-dd HH:mm:ss.SSS}
    [%thread]
    %-5level
    [%X{requestId}]
    %logger{36}
    -
    %msg%n
</pattern>

The resulting log becomes:

2026-08-08 12:20:10.120
[http-nio-8080-exec-2]
INFO
[7f83ab29]
UserService -
Fetching user userId=123

This is much easier to debug.

18. Structured Logging

Traditional logs look like:

User created successfully userId=123

Structured logs can look like JSON:

{
  "timestamp": "2026-08-08T12:20:10.120Z",
  "level": "INFO",
  "service": "user-service",
  "requestId": "7f83ab29",
  "userId": "123",
  "event": "USER_CREATED"
}

This is much easier for log aggregation systems to process.

For production environments, structured JSON logging is often preferable.

19. Why JSON Logs Are Better for Production

Suppose you use:

ELK
Grafana Loki
Datadog
AWS CloudWatch
Splunk

You can query structured fields.

For example:

level = ERROR
service = payment-service
environment = production

Or:

requestId = 7f83ab29

Or:

durationMs > 1000

This becomes much more powerful than searching plain text.

20. Logging Exceptions Correctly

Bad:

try {
    paymentService.process(payment);
} catch (Exception e) {
    log.error("Payment failed: " + e.getMessage());
}

This loses the stack trace.

Better:

try {
    paymentService.process(payment);
} catch (Exception e) {
    log.error(
        "Payment processing failed. paymentId={}",
        paymentId,
        e
    );
}

Now you get:

ERROR Payment processing failed. paymentId=123

java.lang.IllegalStateException: Payment provider timeout
    at PaymentService.process(...)
    at PaymentController.create(...)

The stack trace is extremely important for debugging.

21. Don't Log the Same Exception Multiple Times

A common mistake is:

Repository
   ↓
Service
   ↓
Controller

Every layer catches and logs the same exception.

You might get:

ERROR Database failure
ERROR Service failure
ERROR Controller failure

Three logs for one problem.

Prefer centralized exception handling when possible.

For Spring Boot:

@RestControllerAdvice
public class GlobalExceptionHandler {

    private static final Logger log =
            LoggerFactory.getLogger(GlobalExceptionHandler.class);

    @ExceptionHandler(Exception.class)
    public ResponseEntity<?> handleException(
            Exception exception) {

        log.error(
            "Unhandled application exception",
            exception
        );

        return ResponseEntity
                .internalServerError()
                .body("Something went wrong");
    }
}

Now unexpected exceptions can be logged centrally.

22. Logging Business Events

Not every useful log is technical.

Business events can be extremely valuable.

Example:

log.info(
    "Order completed. orderId={}, customerId={}, amount={}, currency={}",
    orderId,
    customerId,
    amount,
    currency
);

This can help answer questions such as:

How many orders were completed?
Which payment failed?
How long does checkout take?
Which customer experienced the problem?

Logging should help both developers and operations teams.

23. Logging External API Calls

Suppose your application calls:

Stripe
Salesforce
OpenAI
AWS
Google Maps
Email provider
SMS provider

You should log useful metadata.

Example:

long start = System.currentTimeMillis();

try {

    PaymentResponse response =
            paymentClient.createPayment(request);

    long duration =
            System.currentTimeMillis() - start;

    log.info(
        "Payment provider call completed. provider={} status={} durationMs={}",
        "stripe",
        response.status(),
        duration
    );

} catch (Exception e) {

    long duration =
            System.currentTimeMillis() - start;

    log.error(
        "Payment provider call failed. provider={} durationMs={}",
        "stripe",
        duration,
        e
    );
}

But never log:

Authorization header
API key
Access token
Full card details
Sensitive request payload

24. Logging Database Operations

Don't log every SQL query in production unless you have a specific reason.

For example, enabling:

spring.jpa.show-sql=true

in production can create huge amounts of output.

For development:

spring.jpa.show-sql=true

may be useful.

For production:

spring.jpa.show-sql=false

Instead, monitor slow queries through proper database monitoring and profiling tools.

25. Different Logging Configuration Per Environment

Your logging configuration should change based on the environment.

Development:

DEBUG
Readable console logs
More diagnostic information

Production:

INFO
JSON logs
Error tracking
Structured fields
Log rotation
Centralized log collection

Example:

spring:
  profiles:
    active: dev

You can maintain:

application-dev.yml
application-prod.yml

And configure logging accordingly.

26. Production Logging Architecture

A practical production architecture might look like:

                  Spring Boot
                      |
                      v
                   Logback
                      |
        +-------------+-------------+
        |             |             |
        v             v             v
   Application     Error        Audit
      Logs          Logs         Logs
        |             |             |
        +-------------+-------------+
                      |
                      v
                Log Collector
                      |
        +-------------+-------------+
        |             |             |
        v             v             v
   CloudWatch       Loki          ELK
                      |
                      v
                  Dashboard
                      |
                      v
                   Alerts

This is much better than simply SSHing into a server and running:

tail -f application.log

every time something breaks.

27. Docker and Kubernetes Logging

If your application runs inside Docker or Kubernetes, writing logs only to local files may not be the best strategy.

A common approach is:

Application
     |
     v
stdout / stderr
     |
     v
Docker / Kubernetes
     |
     v
Log Collector
     |
     v
Centralized Logging Platform

For example:

Spring Boot
    ↓
stdout
    ↓
Docker
    ↓
Fluent Bit
    ↓
Elasticsearch
    ↓
Kibana

This allows logs to remain available even when containers are recreated.

28. Logging in Kubernetes

In Kubernetes, pods are disposable.

That means this:

Pod A
  |
  +--- application.log

is not necessarily a reliable long-term logging strategy.

Instead:

Pod
 |
 v
stdout
 |
 v
Container Runtime
 |
 v
Log Collector
 |
 v
Centralized Storage

This is generally more suitable for cloud-native applications.

29. Log Levels Should Be Intentional

A useful rule:

TRACE → Extremely detailed diagnostics

DEBUG → Developer troubleshooting

INFO  → Important application events

WARN  → Unexpected but recoverable situation

ERROR → Failure requiring investigation

Don't do this:

log.error("User logged in successfully");

Use:

log.info("User login successful. userId={}", userId);

Log levels should communicate severity.

30. Don't Log Everything

More logs do not automatically mean better observability.

Bad:

log.info("Starting method");
log.info("Entering service");
log.info("Repository called");
log.info("Repository returned");
log.info("Service completed");
log.info("Controller completed");

This creates noise.

Instead:

log.info(
    "User profile updated. userId={} durationMs={}",
    userId,
    durationMs
);

Log meaningful events.

31. Logging Performance

Logging can affect application performance.

Especially dangerous:

log.debug(
    "Huge object: {}",
    objectWithThousandsOfFields
);

When DEBUG isn't enabled, parameterized logging helps avoid unnecessary string concatenation, but object serialization or expensive argument computation can still cost time.

Avoid:

log.debug("Response: {}", expensiveMethod());

if the computation itself is expensive.

You can guard expensive operations:

if (log.isDebugEnabled()) {
    log.debug("Detailed response: {}", expensiveMethod());
}

Use this only when the computation is genuinely expensive.

32. Async Logging

High-throughput applications can benefit from asynchronous logging.

Instead of:

Application
    |
    v
Write log
    |
    v
Disk

you can use:

Application
    |
    v
Async Queue
    |
    v
Logger
    |
    v
Disk / Collector

This reduces the amount of time application threads spend waiting on logging operations.

However, asynchronous logging should be configured carefully to avoid losing logs during abrupt shutdowns and to prevent queue overflow.

33. Log Retention

Production logs should have a retention policy.

For example:

Application logs → 30 days
Audit logs       → 90 days
Security logs    → 180 days

The actual retention period should be based on:

Compliance
Security requirements
Business requirements
Storage cost
Incident investigation needs

Don't keep everything forever.

34. A Practical Logback Configuration

A simplified production configuration could look like:

<configuration>

    <property name="LOG_DIR" value="./logs"/>

    <appender name="CONSOLE"
              class="ch.qos.logback.core.ConsoleAppender">

        <encoder>
            <pattern>
                %d{yyyy-MM-dd'T'HH:mm:ss.SSS}
                %-5level
                [%X{requestId}]
                %logger{36}
                -
                %msg%n
            </pattern>
        </encoder>

    </appender>

    <appender name="APPLICATION_FILE"
              class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>${LOG_DIR}/application.log</file>

        <rollingPolicy
            class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

            <fileNamePattern>
                ${LOG_DIR}/archive/application.%d{yyyy-MM-dd}.%i.log.gz
            </fileNamePattern>

            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>5GB</totalSizeCap>

        </rollingPolicy>

        <encoder>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss.SSS}
                %-5level
                [%X{requestId}]
                %logger{36}
                -
                %msg%n
            </pattern>
        </encoder>

    </appender>

    <appender name="ERROR_FILE"
              class="ch.qos.logback.core.rolling.RollingFileAppender">

        <file>${LOG_DIR}/error.log</file>

        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>

        <rollingPolicy
            class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

            <fileNamePattern>
                ${LOG_DIR}/archive/error.%d{yyyy-MM-dd}.%i.log.gz
            </fileNamePattern>

            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>2GB</totalSizeCap>

        </rollingPolicy>

        <encoder>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss.SSS}
                %-5level
                [%X{requestId}]
                %logger{36}
                -
                %msg%n
            </pattern>
        </encoder>

    </appender>

    <root level="INFO">

        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="APPLICATION_FILE"/>
        <appender-ref ref="ERROR_FILE"/>

    </root>

</configuration>

This gives you:

logs/
├── application.log
├── error.log
└── archive/
    ├── application.2026-08-08.0.log.gz
    ├── error.2026-08-08.0.log.gz
    └── ...

35. Real-World Example

Imagine a customer reports:

"My payment failed."

Without structured logging, you might search:

payment failed

and find thousands of results.

With production-grade logging, you can search:

requestId=7f83ab29

Then you might see:

INFO  requestId=7f83ab29
Payment request received. orderId=ORD-123

INFO  requestId=7f83ab29
Calling payment provider. provider=stripe

WARN  requestId=7f83ab29
Payment provider response slow. durationMs=4200

ERROR requestId=7f83ab29
Payment provider call failed. orderId=ORD-123

Now you know exactly what happened.

That's the real value of production logging.

36. Logging Best Practices Checklist

Before deploying a Spring Boot application, check:

Log levels

[ ] TRACE is disabled in production
[ ] DEBUG is used intentionally
[ ] INFO contains meaningful events
[ ] WARN represents recoverable problems
[ ] ERROR represents actual failures

Security

[ ] Passwords are never logged
[ ] Tokens are never logged
[ ] API keys are never logged
[ ] Sensitive headers are never logged
[ ] Sensitive payloads are not logged

Reliability

[ ] Log rotation is configured
[ ] Maximum file size is configured
[ ] Retention policy exists
[ ] Disk usage is monitored

Observability

[ ] Request ID exists
[ ] Correlation ID exists
[ ] Important business events are logged
[ ] External API failures are logged
[ ] Slow operations can be identified

Production

[ ] Structured logging is available
[ ] Logs can be centralized
[ ] Alerts can be created
[ ] Logs are searchable
[ ] Audit events are separated when required

37. What I Consider a Good Production Logging Strategy

For a modern Spring Boot backend, my preferred baseline would be:

Spring Boot
     |
     +--- SLF4J
     |
     +--- Logback
     |
     +--- Structured JSON
     |
     +--- Request ID / Correlation ID
     |
     +--- INFO as default
     |
     +--- DEBUG for troubleshooting
     |
     +--- ERROR for failures
     |
     +--- Audit logging for important actions
     |
     +--- Log rotation / retention
     |
     +--- Centralized log aggregation
     |
     +--- Monitoring + Alerting

For cloud deployments, I would generally prefer sending structured logs to a centralized platform rather than depending exclusively on local log files.

38. Final Thoughts

Logging is not something you add at the end of development.

It is part of backend architecture.

A production-ready application should make it easy to understand:

What happened?
When?
Where?
Who triggered it?
Which request?
Which service?
How long?
What failed?
Why?

The goal isn't to create millions of log lines.

The goal is to create useful signals.

A good production logging system gives developers confidence when everything is working and, more importantly, gives them the information they need when something goes wrong.

If your application is running in production, ask yourself:

"If this API fails at 3 AM, can I understand exactly what happened from the logs?"

If the answer is no, your logging strategy probably needs another iteration.

Caching with Redis and Spring Data Redis in Spring Boot

Ayush Shrivastava — Wed, 29 Jul 2026 14:40:29 +0000

Caching with Redis and Spring Data Redis in Spring Boot

Caching is one of the most effective ways to improve the performance and scalability of modern applications. Instead of querying the database for the same data repeatedly, you can temporarily store frequently accessed data in memory and serve it almost instantly.

One of the most popular technologies for implementing caching is Redis, an in-memory data store known for its speed, flexibility, and reliability.

In this article, you'll learn:

What caching is
Why Redis is an excellent caching solution
How Spring Data Redis works
How to configure Redis in Spring Boot
How to use @Cacheable, @CachePut, and @CacheEvict
Best practices for production applications

Let's get started.

What is Caching?

Caching is the process of storing copies of frequently accessed data in a fast storage layer (cache), so future requests can be served without querying the database.

Without caching:

Client
   │
   ▼
Spring Boot
   │
   ▼
Database

Every request reaches the database.

With caching:

Client
   │
   ▼
Spring Boot
   │
   ▼
Redis Cache
   │
 Cache Hit?
  │     │
 Yes    No
 │       │
 ▼       ▼
Return  Database
          │
          ▼
      Store in Redis

The first request loads data from the database and stores it in Redis.

Subsequent requests are served directly from Redis.

This dramatically reduces latency and database load.

Why Use Redis for Caching?

Redis is one of the fastest key-value stores available because it stores data entirely in memory.

Some major advantages include:

Extremely low latency
High throughput
Rich data structures
Optional persistence
Replication support
Redis Cluster support
Automatic expiration (TTL)
Pub/Sub messaging
Transactions

Because of these features, Redis is widely used in high-performance systems.

What is Spring Data Redis?

Spring Data Redis is part of the Spring Data ecosystem.

It provides:

Easy Redis integration
Repository support
RedisTemplate API
Spring Cache integration
Serialization support
Object mapping
Connection management

Instead of writing Redis commands manually, Spring Boot handles most of the complexity for you.

Project Dependencies

Add the following dependencies.

<dependencies>

    <!-- Spring Data Redis -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-redis</artifactId>
    </dependency>

    <!-- Spring Cache -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-cache</artifactId>
    </dependency>

    <!-- Jedis Client -->
    <dependency>
        <groupId>redis.clients</groupId>
        <artifactId>jedis</artifactId>
    </dependency>

</dependencies>

Note: Newer Spring Boot versions use Lettuce as the default Redis client. Unless you specifically need Jedis, Lettuce is generally recommended for production.

Running Redis

Using Docker:

docker run -d \
  --name redis \
  -p 6379:6379 \
  redis:latest

Verify Redis is running:

docker ps

Configure Redis

Create a configuration class.

@Configuration
public class RedisConfig {

    @Bean
    public RedisConnectionFactory redisConnectionFactory() {
        return new JedisConnectionFactory();
    }

    @Bean
    public RedisTemplate<String, Object> redisTemplate() {

        RedisTemplate<String, Object> template = new RedisTemplate<>();

        template.setConnectionFactory(redisConnectionFactory());

        template.setKeySerializer(new StringRedisSerializer());

        template.setValueSerializer(
                new GenericJackson2JsonRedisSerializer());

        return template;
    }
}

Configure application.properties

spring.data.redis.host=localhost
spring.data.redis.port=6379

If authentication is enabled:

spring.data.redis.password=your-password

Enable Caching

Simply add @EnableCaching to your Spring Boot application.

@SpringBootApplication
@EnableCaching
public class RedisCacheApplication {

    public static void main(String[] args) {
        SpringApplication.run(RedisCacheApplication.class, args);
    }

}

Now Spring Boot can automatically manage your cache.

Create the Product Entity

@RedisHash("Product")
public class Product {

    @Id
    private String id;

    private String name;

    private double price;

    // getters and setters

}

Create the Repository

public interface ProductRepository
        extends CrudRepository<Product, String> {
}

Nothing else is required.

Spring Data generates the implementation automatically.

Using Cache Annotations

Spring provides three important annotations:

Annotation	Purpose
`@Cacheable`	Reads from cache before executing the method
`@CachePut`	Updates both database and cache
`@CacheEvict`	Removes data from cache

Product Service

@Service
public class ProductService {

    @Autowired
    private ProductRepository repository;

    @Cacheable(value = "products", key = "#id")
    public Product getProductById(String id) {

        System.out.println("Fetching from database...");

        return repository.findById(id).orElse(null);
    }

    @CachePut(value = "products", key = "#product.id")
    public Product updateProduct(Product product) {

        return repository.save(product);
    }

    @CacheEvict(value = "products", key = "#id")
    public void deleteProduct(String id) {

        repository.deleteById(id);
    }

}

Understanding @Cacheable

When the following method is called:

productService.getProductById("1");

Spring checks Redis.

If the key exists

Redis
   │
Found
   │
Return cached object

The database is never accessed.

If the key does not exist

Database
   │
Load object
   │
Store in Redis
   │
Return object

Future requests will use Redis.

Understanding @CachePut

@CachePut(value = "products", key = "#product.id")

Unlike @Cacheable, this annotation always executes the method.

Workflow:

Update Database

↓

Update Redis

↓

Return Updated Object

This ensures your cache remains synchronized.

Understanding @CacheEvict

@CacheEvict(value = "products", key="#id")

When deleting a product:

Delete Database Record

↓

Remove Redis Cache

↓

Done

Without cache eviction, users could still receive stale data.

Testing the Cache

@Component
public class CacheTestRunner implements CommandLineRunner {

    @Autowired
    private ProductService productService;

    @Override
    public void run(String... args) {

        Product product = new Product();

        product.setId("1");
        product.setName("Laptop");
        product.setPrice(999.99);

        productService.updateProduct(product);

        System.out.println(productService.getProductById("1"));

        System.out.println(productService.getProductById("1"));

        product.setName("Gaming Laptop");

        productService.updateProduct(product);

        System.out.println(productService.getProductById("1"));

        productService.deleteProduct("1");

        System.out.println(productService.getProductById("1"));
    }
}

Expected Output

Fetching from database...

Product{id='1', name='Laptop'}

Product{id='1', name='Laptop'}

Product{id='1', name='Gaming Laptop'}

Fetching from database...

null

Notice that "Fetching from database..." appears only when the cache misses.

Cache Lifecycle

Client Request
      │
      ▼
Check Redis
      │
 ┌────┴────┐
 │         │
Hit       Miss
 │         │
 ▼         ▼
Return   Query DB
            │
            ▼
      Save in Redis
            │
            ▼
      Return Response

Production Best Practices

1. Use TTL (Time-To-Live)

Avoid keeping cached data forever.

Example:

spring.cache.redis.time-to-live=10m

2. Cache Only Frequently Accessed Data

Good candidates include:

Product catalog
User profiles
Categories
Settings
Configuration
Country lists

Avoid caching highly volatile data unless necessary.

3. Use Meaningful Cache Names

Instead of:

value="cache1"

Prefer:

value="products"

value="users"

4. Serialize JSON

Using:

GenericJackson2JsonRedisSerializer

makes cached objects easier to inspect and maintain.

5. Monitor Cache Performance

Keep an eye on:

Cache hit rate
Cache miss rate
Memory usage
Eviction count
Response time

Monitoring helps optimize your caching strategy.

Common Cache Annotations

@Cacheable

Reads from cache.

@CachePut

Updates cache.

@CacheEvict

Deletes cache.

@Caching

Combines multiple cache operations.

@CacheConfig

Defines shared cache configuration.

Advantages of Redis Caching

Faster response times
Reduced database load
Better scalability
Improved user experience
Lower infrastructure costs
High availability
Distributed caching support

When Should You Use Redis?

Redis is an excellent choice when your application needs:

Frequently accessed data
High read traffic
Low latency
Distributed caching
Session storage
API response caching
Rate limiting
Leaderboards
Real-time analytics

Conclusion

Caching is one of the simplest yet most impactful optimizations you can make in a Spring Boot application. By integrating Redis with Spring Data Redis, you can significantly reduce database load, improve response times, and build applications that scale efficiently under heavy traffic.

Spring Boot's caching abstraction makes implementation straightforward with annotations like @Cacheable, @CachePut, and @CacheEvict, allowing you to focus on business logic rather than cache management.

Whether you're building a REST API, microservices architecture, or a high-traffic web application, Redis caching is a proven strategy to boost performance and deliver a better user experience.

If you're working with Spring Boot in production, learning Redis is a skill that will pay dividends across many real-world applications.

Happy Coding!