DEV Community

Cover image for Blossoming Intelligence: How to Run Spring AI Locally with Ollama
Elattar Saad
Elattar Saad

Posted on

20

Blossoming Intelligence: How to Run Spring AI Locally with Ollama

Nobody can dispute that AI is here to stay. Among many of its benefits, developers are using its capability to boost their productivity.
It is also planned to become accessible for a fee as a SaaS or any other service once it has gained the necessary trust from enterprises.
Still, We can run pre-trained models locally and incorporate them into our current app.

In this short article, we'll look at how easy it is to create a chat bot backend powered by Spring and Olama using the llama 3 model.

TechStack

This project is built using:

  • Java 21.
  • Spring boot 3.2.5 with WebFlux.
  • Spring AI 3.2.5.
  • Ollama 0.1.36.

Ollama Setup

To install Ollama locally, you simply need to head to https://ollama.com/download and install it using the proper executable to your OS.

You check is installed by running the following command:



ollama --version


Enter fullscreen mode Exit fullscreen mode

You can directly pull a model from Ollama Models) and run it using the ollama cli, in my case I used the llama3 model:



ollama pull llama3 # Should take a while.
ollama run llama3


Enter fullscreen mode Exit fullscreen mode

Let's test it out with a simple prompt:

Image description

To exist, use the command:



/bye


Enter fullscreen mode Exit fullscreen mode

Talking Spring

The Spring will have the following properties:



spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.model=llama3


Enter fullscreen mode Exit fullscreen mode

Then is our chat package, will have a chat config bean to handle:



package io.daasrattale.generalknowledge.chat;

import org.springframework.ai.chat.ChatClient;
import org.springframework.ai.ollama.OllamaChatClient;
import org.springframework.ai.ollama.api.OllamaApi;
import org.springframework.ai.ollama.api.OllamaOptions;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ChatConfig {


    @Bean
    public ChatClient chatClient() {
        return new OllamaChatClient(new OllamaApi())
                .withDefaultOptions(OllamaOptions.create()
                        .withTemperature(0.9f));
    }
}


Enter fullscreen mode Exit fullscreen mode

FYI, Model temperature is a parameter that controls how random a language model's output is.
A temperature is set to 0.9 to make the model more random and willing to take more risks on the answers.

The last step is to create a simple Chat rest controller:



package io.daasrattale.generalknowledge.chat;

import org.springframework.ai.ollama.OllamaChatClient;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Mono;

@RestController
@RequestMapping("/v1/chat")
public class ChatController {


    private final OllamaChatClient chatClient;

    public ChatController(OllamaChatClient chatClient) {
        this.chatClient = chatClient;
    }

    @GetMapping
    public Mono<ResponseEntity<String>> generate(@RequestParam(defaultValue = "Tell me to add a proper prompt in a funny way") String prompt) {
        return Mono.just(
                ResponseEntity.ok(chatClient.call(prompt))
        );
    }


}


Enter fullscreen mode Exit fullscreen mode

Let's try and call a GET /v1/chat with an empty prompt:

Image description

What about a simple general knowledge question:

Image description

Of course, let's ask for some code:

Image description

The api url provided is not OK, still the rest of the code works.

Finally

Using models locally with such ease and simplicity can be considered as a true added value, still, the used models must be heavily inspected.

You can find the source code on this Github Repository make sure to star it if you find it useful :))

Resources

https://spring.io/projects/spring-ai

https://docs.spring.io/spring-ai/reference/api/clients/ollama-chat.html

Heroku

Build apps, not infrastructure.

Dealing with servers, hardware, and infrastructure can take up your valuable time. Discover the benefits of Heroku, the PaaS of choice for developers since 2007.

Visit Site

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay