Generative AI development doesn't always require a cloud-hosted model.
If you're building Java applications with Spring AI, you can run AI models locally and connect them to your Spring Boot application without depending on external model APIs.
One interesting option is Docker Model Runner.
In this article, we'll explore how Docker Model Runner works, why it is useful for Java developers, and how to connect it with Spring AI.
What is Docker Model Runner?
Docker Model Runner allows you to run AI models locally using Docker.
Instead of sending every prompt to a cloud provider such as OpenAI or AWS Bedrock, you can run supported models on your own machine.
The architecture looks like this:
Spring Boot Application
|
v
Spring AI
|
v
OpenAI-compatible API
|
v
Docker Model Runner
|
v
Local LLM
Your Java application interacts with the model through an API, while Docker handles running the model locally.
This gives developers a convenient way to experiment with LLM applications without immediately provisioning cloud infrastructure.
Why Run Models Locally?
There are several reasons you may want to run an LLM locally.
1. Development without API costs
During development, you may send hundreds or thousands of prompts.
Running a model locally can eliminate API charges during experimentation.
2. Privacy
Your prompts and application data can remain on your machine instead of being sent to an external AI provider.
This can be particularly useful when experimenting with sensitive or proprietary data.
3. Offline experimentation
Once the model is available locally, you don't need an internet connection for every inference request.
4. Faster experimentation
Developers can experiment with prompts, tool calling, RAG pipelines, and application logic without repeatedly configuring cloud credentials.
5. Same Spring AI programming model
One of the biggest advantages is that your application can continue using Spring AI abstractions.
Your business logic doesn't need to be tightly coupled to a specific model provider.
Docker Model Runner + Spring AI
Spring AI provides abstractions such as:
ChatModel
ChatClient
This means your application can interact with an LLM without having to directly implement provider-specific HTTP calls.
The important idea is:
Application
|
v
ChatClient
|
v
ChatModel
|
v
Model API
|
v
Local Model
If your local model exposes an OpenAI-compatible API, Spring AI can communicate with it using the appropriate OpenAI configuration.
Step 1: Install Docker
First, make sure Docker Desktop is installed and running on your machine.
Docker Model Runner is available through Docker's model functionality, depending on your Docker Desktop version and configuration.
You can verify that Docker is available with:
docker --version
Then make sure Docker Desktop is running.
Step 2: Enable Docker Model Runner
Docker Desktop provides the model-running infrastructure required to run supported AI models locally.
Once enabled, you can work with models directly through Docker.
The exact commands and model availability can change as Docker's model ecosystem evolves, so check the current Docker documentation for the model you want to use.
The important concept for our Spring AI application is that Docker Model Runner exposes an API endpoint that our application can communicate with.
Step 3: Create a Spring Boot Project
Create a Spring Boot application with Spring AI.
For Maven, add the Spring AI OpenAI starter:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
The reason we're using the OpenAI starter is not because we're calling OpenAI's cloud service.
We're using the OpenAI-compatible API supported by the local model runtime.
This is an important concept:
An OpenAI-compatible API does not necessarily mean you're using OpenAI's infrastructure.
It simply means the API follows a compatible request/response format.
Step 4: Configure Spring AI
In application.properties, configure the OpenAI base URL to point to your local Docker Model Runner endpoint.
For example:
spring.ai.openai.base-url=http://localhost:<model-runner-port>
spring.ai.openai.api-key=dummy
spring.ai.openai.chat.options.model=<your-local-model>
The exact endpoint and model name depend on your Docker Model Runner setup.
The API key may not actually be required by the local runtime, but the Spring AI OpenAI client expects the configuration property, so a placeholder value can be used when appropriate.
Step 5: Create a ChatClient
Now we can create a ChatClient.
@RestController
@RequestMapping("/ai")
public class AIController {
private final ChatClient chatClient;
public AIController(ChatClient.Builder builder) {
this.chatClient = builder.build();
}
@GetMapping("/chat")
public String chat(@RequestParam String message) {
return chatClient
.prompt()
.user(message)
.call()
.content();
}
}
Now your Spring Boot application has a simple endpoint:
GET /ai/chat?message=Explain dependency injection in Spring
The flow becomes:
HTTP Request
|
v
Spring Boot
|
v
ChatClient
|
v
Spring AI
|
v
Docker Model Runner
|
v
Local LLM
|
v
Generated Response
ChatClient Hides Provider Complexity
This is where Spring AI becomes useful.
Your controller doesn't need to know whether the model is:
- OpenAI
- Azure OpenAI
- AWS Bedrock
- Ollama
- Docker Model Runner
- another compatible provider
The application interacts with the Spring AI abstraction.
For example:
chatClient
.prompt()
.user("Explain Spring Boot dependency injection")
.call()
.content();
The application focuses on what it wants from the model, rather than implementing the underlying model communication itself.
Local LLM vs Cloud LLM
Running a model locally doesn't mean local models are always better.
There are trade-offs.
| Local Models | Cloud Models |
|---|---|
| Data stays locally | Data sent to provider |
| No per-request API cost | Usually usage-based pricing |
| Requires local compute | Provider handles infrastructure |
| Potentially slower | Often faster |
| Limited by local hardware | Access to larger models |
| Useful for development | Useful for production workloads |
For example, a developer laptop may be perfectly capable of running a smaller model.
But running a large frontier model locally may require significantly more memory and compute.
So the right question isn't:
"Local or cloud?"
It is:
"Which model deployment strategy fits this workload?"
Where Docker Model Runner Fits
Docker Model Runner is particularly interesting for developers who already use Docker as part of their development workflow.
You can think of it as bringing model execution closer to the rest of your local development environment.
Instead of:
Spring Boot
|
+----> OpenAI
|
+----> AWS
you can have:
Docker Development Environment
|
+---- Spring Boot
|
+---- Database
|
+---- Redis
|
+---- AI Model
This can make local AI application development much easier to reproduce.
A More Realistic Architecture
Once you move beyond a simple chatbot, the architecture becomes more interesting.
For example:
Spring Boot
|
Spring AI
|
+---------+---------+
| |
ChatClient Embeddings
| |
v v
Docker Model Runner Vector Store
|
v
Local LLM
This architecture can support applications involving:
- RAG
- document processing
- semantic search
- AI assistants
- summarization
- classification
- question answering
- tool calling
And because the application uses Spring AI abstractions, you can change parts of the architecture later.
Local Development Is More Than Cost Saving
One of the biggest benefits of local models is actually developer experimentation.
Imagine you're building a RAG application.
You need to test:
Documents
↓
Chunking
↓
Embeddings
↓
Vector Store
↓
Retrieval
↓
Prompt
↓
LLM
During development, you may repeatedly modify prompts and retrieval strategies.
Having a local model can make that experimentation cheaper and easier.
Later, when you're ready for production, you can evaluate whether a managed model provider makes more sense.
What About Production?
This is where architecture decisions become important.
A local model running on a developer laptop is obviously different from a production AI infrastructure setup.
For production, you need to think about:
- GPU availability
- model size
- concurrency
- latency
- autoscaling
- observability
- security
- model updates
- failure recovery
- cost
For some workloads, a managed cloud model will be the better option.
For others, self-hosted inference can make sense because of privacy, compliance, cost, or latency requirements.
The important thing is that your application architecture should avoid unnecessary coupling to one model provider.
Spring AI Makes This Easier
This is one of the reasons I like the Spring AI abstraction.
Your business code can work at a higher level:
ChatClient
.prompt()
.user(prompt)
.call()
.content();
Instead of manually implementing:
HTTP request
↓
Authentication
↓
JSON serialization
↓
Provider API
↓
Response parsing
Spring AI handles much of that integration layer.
That allows Java developers to focus on building the actual AI application.
Final Thoughts
Docker Model Runner gives Java developers another option for experimenting with generative AI locally.
Combined with Spring AI, the architecture becomes relatively straightforward:
Spring Boot
↓
Spring AI
↓
ChatClient
↓
OpenAI-Compatible API
↓
Docker Model Runner
↓
Local LLM
The bigger lesson isn't simply how to run one particular model.
It's understanding model portability.
Your application shouldn't necessarily care whether the underlying model is running locally or in the cloud.
Spring AI gives you abstractions that help separate your application logic from the model provider.
And that's an important foundation for building production-ready AI applications.
What's Next?
In Part 6, we'll move from local inference to the cloud and explore AWS Bedrock with Spring AI.
We'll look at how Spring Boot applications can interact with foundation models available through AWS and how the architecture changes when AI inference moves from your local machine to a managed cloud platform.
Top comments (0)