A few days ago, I posted a tutorial in the Spring AI Dev community on how to use Anthropic Claude’s models with Spring AI and received a curious comment from a user:
Why does it make it sound like you can only use Amazon or Anthropic?
In my reply, I explained that, for most companies, trusting proprietary solutions makes sense. Usually, you do not have time to build your own solution, especially if AI is not the focus of your business. On the other hand, you have the money and such strong demand that you can negotiate better prices. Sounds fair, right?
However, I still could not get that question out of my head. Weren’t there any more options? And why should I spend money just to experiment with a tutorial?
Well, this article is a response to that question!
Fortunately, running models locally offers a practical way to bypass these costs and take full control of your data. Enter Ollama.
It does not mean you can just get rid of proprietary AI. Your business will likely integrate with a proprietary AI provider such as OpenAI, Anthropic, or AWS Bedrock.
But in fact, it does not make sense to spend money if you are still learning AI, and Ollama is an excellent option to fill that gap.
What is Ollama?
Ollama is an AI server that you can install in your computer and then run different AI models locally. Then you can connect harnesses and agents such as Open Code or Open Claw, or build your AI solution with a framework like Spring AI. Everything runs on your hardware, and no data leaves your computer.
With proprietary AI providers like OpenAI and Anthropic, you need to register for a subscription or buy credits in advance so you can connect to their API’s. And in many cases, if you want to use a more advanced model, you have to buy extra credits or upgrade to a premium subscription.
Also, proprietary AI providers host their models in their dedicated infrastructure. Their models do not leave their servers, so you need to transmit your data for processing.
With Ollama, you do not need to spend money to use an AI, and your prompts never leave your computer; everything runs locally. Thus, besides saving money, Ollama is a solution to be considered in production if your company or organization has severe privacy requirements and no data could be sent to third-party services.
Understanding the Trade-Off
Obviously, running AI models locally does not come for free. Because you are running these models locally, your choices are limited by your hardware. Cutting-edge models like deepseek-r1:671b have more than 600 billion parameters and require 400 GB of VRAM to run, which is not feasible for most people.
But don’t worry. Smaller models with 8 billion parameters are also available and run in a common graphics card, providing decent results.
Keeping these hardware boundaries in mind, let’s look at how to check your system specs and select the right model for your setup.
So, now that we understand what Ollama is and its tradeoffs, let’s jump to the practical part of this article. In the next section, we will learn how to install Ollama and run local models.
How to Install Ollama and Run Local Models
In this example, I installed Ollama on in Ubuntu 24.04.3 LTS instance running on a WSL2 system. The official documentation provides additional instructions for other operating systems such as Windows and macOS. The Linux installation does not rely on dependency managers like apt, so these instructions are also portable to other distributions.
1. Installing Ollama
Ollama installation is quite simple. As long as you have the curl command installed, all you have to do is download and run the installation script. First, make sure that curl is installed. Run the command:
curl -V
And observe the result. You should receive a positive message listing the version, libraries, and protocols. If you do not have curl installed, just run the following command to install it:
sudo apt install curl -y
After curl is installed, run the command below:
curl -fsSL https://ollama.com/install.sh | sh
This command will download the installation script and install it in your system. Then you can confirm the installation with the following command:
ollama help
This command will print instructions for using the Ollama runner, including a list of available commands for administering and running models.
If the script executed correctly, it must have configured Ollama as a Linux service, and you can administer it with the systemctl command. For example, you can check the status of the ollama service with the command:
systemctl status ollama
And then the systemctl command will print whether the service is active, along with additional information such as the process ID, memory usage, and CPU usage.
If you received the usage instructions from the ollama help command and verified that the service is active with systemctl, this means the installation was successful. We can proceed to run local models.
Now that the background service is up and running, we can verify everything works by communicating directly with its local API.
2. Running Local Models
If you installed Ollama correctly, you can now run commands in your terminal and call the HTTP API Ollama provides on port 11434 by default. You can test the API by running the command:
curl http://localhost:11434
And receive the message “Ollama is running”.
But that does not mean you can generate text from prompts. For example, try to run the following command:
curl http://localhost:11434/api/generate -d '{'
"model": "gemma3",
"prompt": "Why is the sky blue?"
}'
And you’ll get a 404 error: “model gemma3 not found”
And that’s because Ollama is just a vessel, like the Infinity Gauntlet. The true power of the Infinity Gauntlet comes from the Infinity Stones that you attach to the gauntlet. Ollama is the same thing; its true power comes from the multiple models that you can download and install.
There are two ways you can verify the models installed on your Ollama instance. The first one is the list command. In your terminal, type:
ollama list
The command will print a table with the models available for use in your local installation. If you run the command immediately after installing Ollama, you will only see the table headers. That means you have no model locally available yet.
Another way to check the available models is via the /tags/endpoint. Run the command:
curl http://localhost:11434/api/tags
And you will receive a JSON object with an empty “models” array. Confirming that no model is available yet.
Now that we have Ollama installed and running, let’s learn how to install a model locally in our environment.
3. Choosing a Model
Ollama provides an extensive library of models available at https://ollama.com/library.
When visiting the Ollama library, you’ll notice many models available in different sizes and capabilities, and choosing the best one can be a hard task. Model benchmarks are way beyond the scope of this humble article, but here are some tips.
First, think about the application you are building and what capabilities you need. For example, if your application needs to integrate with external API’s, then you need tooling. If you need to interpret images, then you need a multi-modal model, and so on.
As said before, you are limited by your hardware. If you are not sure, you can run the following command to discover how much memory you have, and therefore, the largest model your hardware supports:
nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free,driver_version --format=csv
In my case, I got the following output:
name, memory.total [MiB], memory.used [MiB], memory.free [MiB], driver_version
NVIDIA GeForce RTX 5070 Laptop GPU, 8151 MiB, 3052 MiB, 4840 MiB, 610.88
Which means I have a hard limit of 8GB of VRAM, which is the maximum size of a model I can run locally.
For this example, I’m choosing a multimodal model - capable of processing image and text - which fits entirely in my VRAM. Given these requirements, gemma3 seems a reasonable choice. Let’s see how to pull and run the model locally.
For this example, I’m choosing a multimodal model - capable of processing image and text - which fits entirely in my VRAM. Given these requirements, gemma3 seems a reasonable choice. Let’s see how to pull and run the model locally.
4. Pulling a Model from the Registry
Fortunately, the analogy to Infinite Stones ends here, and you don’t have to throw someone you love off a cliff to get a model. Instead, all you have to do is run a simple command:
ollama pull gemma3
The pull command will download and install the model in your local environment. If Ollama downloaded the model correctly, then you will see the model listed if you run the Ollama list command or if you access the tags endpoint through the HTTP API. Run the following commands and check that the gemma3 model is now available in your local environment
ollama list
Or
curl http://localhost:11434/api/tags
And they should return the gemma3 model.
5. Running a Model Locally
After downloading the model, you can test it in an interactive session in the terminal with the command:
ollama run gemma3
With this command, you can chat with the model right from the terminal, or even use it in a command pipeline. Also, you can now submit prompts via the Web API at either the /api/chat or /api/generate endpoint. Run the command:
curl http://localhost:11434/api/generate -d '{
"model": "gemma3",
"stream": false,
"prompt": "Why is the sky blue?"
}'
And have a nice conversation about our atmosphere.
6. Integrating with Spring AI (Optional)
Finally, if you are a Spring AI developer like me, connect your Spring AI application to your local Ollama instance. Configure the base URL in your application.yml file:
spring:
ai:
ollama:
base-url: http://localhost:11434
Conclusion
Congratulations! Now you can run AI models in your local environment and save on API credits or subscriptions.
Now it is up to you to explore the possibilities. You can test different models and see which one provides the best answers for your tasks, or connect your preferred harness to use local models. Also, you can use these local models to develop your own AI solutions with Spring AI, and this is the possibility that excites me the most.
References
The discussion that motivated the article: https://www.reddit.com/r/SpringAIDev/comments/1vyt41n/comment/p5zfdzm/
Ollama documentation: https://docs.ollama.com/
Ollama library: https://docs.ollama.com/library
Top comments (1)
Original publication at: rodolfomendes.ai/how-to-run-local-...