Silver_dev

Posted on Oct 19

Spring AI: First Steps

#ai #tutorial #spring

Preparation

Ensure that Ollama is running and the model is downloaded:

ollama pull qwen2.5:3b

spring.ai.ollama.chat.options.model=qwen2.5:3b
spring.ai.ollama.base-url=http://localhost:11434

# Optional: temperature (0.0 - deterministic, 1.0 - creative)
spring.ai.ollama.chat.temperature=0.7

You can use any other model — the name must exactly match the output of ollama list.

1. AI Hello World

Create a controller:

package com.example.demo;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class AiController {

    private final ChatClient chatClient;

    public AiController(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    @GetMapping("/ai/hello")
    public String aiHello() {
        // Formulate the prompt
        String prompt = "Say 'Hello, World!' and add one short, interesting fact about the Java language.";

        // Call the model and get the response
        return chatClient.prompt()
                .user(prompt)
                .call()
                .content(); // .content() extracts just the response string from the ChatResponse object
    }
}

Open http://localhost:8080/ai/hello in your browser.

Notes:

The first request may take 2–5 seconds — the model is loading into memory.
Error 404 Model not found — check the exact model name in ollama list.
.call().content() is a synchronous call. For a chat with a typing effect, use .stream() (see section 5).

2. Structured Output (JSON → Java Record)

Spring AI can return not just plain text, but a ready-made Java object. Create a record:

package com.example.demo;

public record RoutingIntent(
    String targetService,
    String action,
    String userId,
    int priority
) {}

Add an endpoint to AiController.java:

@GetMapping("/ai/route")
public RoutingIntent routeRequest(@RequestParam String text) {
    String systemPrompt = """
        You are a smart API gateway. Analyze the user's request 
        and extract data for routing. 
        If a field is unknown, leave it empty (null or 0).
        """;

    return chatClient.prompt()
            .system(systemPrompt)
            .user(text)
            .call()
            .entity(RoutingIntent.class);
}

Example: http://localhost:8080/ai/route?text=Disable notifications for user user-999

Response: A ready-made JSON, serialized by Spring. No manual parsing required.

3. Function Calling (LLM calls Java code)

Give the model "tools" — Java methods annotated with @Tool. If the user asks, "What is the status of the payment service?", the LLM won't hallucinate an answer. Instead, it tells Spring AI: "Hey, call the getServiceStatus("payment") method, and I will use its result."

Create the SystemTools.java class. The @Tool annotation makes the method visible to the LLM.

package com.example.demo;

import org.springframework.stereotype.Service;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;

@Service
public class SystemTools {

    @Tool(description = "Checks the current status and latency (in ms) of the specified microservice")
    public String getServiceStatus(@ToolParam(description = "Name of the service, e.g., 'payment', 'auth', 'database'") String serviceName) {
        // A real call to Prometheus, ClickHouse, or your proxy logger could go here
        System.out.println(">>> LLM called Java method for service: " + serviceName);

        return switch (serviceName.toLowerCase()) {
            case "payment" -> "Status: OK, average latency 45ms.";
            case "auth" -> "Status: DEGRADED, latency 300ms, experiencing timeouts.";
            default -> "Status: Unknown service or no data available.";
        };
    }
}

Update AiController.java by adding a dependency on SystemTools:

package com.example.demo;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class AiController {

    private final ChatClient chatClient;

    // Inject our tools
    public AiController(ChatClient.Builder chatClientBuilder, SystemTools systemTools) {
        this.chatClient = chatClientBuilder
                .defaultTools(systemTools) // <--- this is the key line
                .build();
    }

    @GetMapping("/ai/tool")
    public String askSystem(@RequestParam String question) {
        return chatClient.prompt()
                .user(question)
                .call()
                .content();
    }
}

Request: http://localhost:8080/ai/tool?question=What is the current status of the payment and auth services?

The LLM will decide to call getServiceStatus("payment"), Spring AI will execute the method, and the model will formulate the final response based on the result.

What happens "under the hood":

The LLM sees the question and realizes it doesn't know the answer.
It sees your @Tool and generates a special request: "Call getServiceStatus with the argument payment".
Spring AI intercepts this and calls your actual Java method.
In the console, you will see: >>> LLM called Java method for service: payment
The LLM receives the result ("Status: OK...") and formulates the final human-readable response.

4. Prompt Templates (Dynamic Prompts)

You need to formulate prompts dynamically, substituting data from the context (user name, request parameters, database results). Hardcoding prompts is a bad practice. Spring AI allows you to create prompt templates with variables, similar to Thymeleaf or Mustache.

Add a new endpoint to AiController.java:

package com.example.demo;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.prompt.PromptTemplate;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.util.Map;

@RestController
public class AiController {

    private final ChatClient chatClient;
    private final SystemTools systemTools;

    public AiController(ChatClient.Builder chatClientBuilder, SystemTools systemTools) {
        this.chatClient = chatClientBuilder
                .defaultTools(systemTools)
                .build();
    }

    // ... your previous /ai/hello, /ai/route, /ai/tool methods ...

    @GetMapping("/ai/template")
    public String generateWithTemplate(
            @RequestParam String userName,
            @RequestParam String serviceName,
            @RequestParam String action
    ) {
        // Create a prompt template with variables
        String templateText = """
            You are a system administrator. User {userName} wants to perform the action '{action}' 
            on the {serviceName} service. 
            Provide a brief instruction on how to do this safely.
            """;

        // Populate the variables
        PromptTemplate promptTemplate = new PromptTemplate(templateText);
        String finalPrompt = promptTemplate.render(Map.of(
                "userName", userName,
                "serviceName", serviceName,
                "action", action
        ));

        // Send the finalized prompt to the model
        return chatClient.prompt()
                .user(finalPrompt)
                .call()
                .content();
    }
}

Example: http://localhost:8080/ai/template?userName=John&serviceName=payment&action=restart

Response:

John, to safely restart the payment service, follow these steps:
1. Ensure there are no active transactions...
2. Execute the restart command...
3. Check the service logs...

Templates can be stored in separate files (.txt, .prompt) and versioned separately from the codebase.

5. Streaming (Real-time Response Streaming)

If your proxy communicates with the user in real time (e.g., a chatbot or voice assistant), waiting 2–3 seconds for a full response is bad UX. Streaming allows delivering the response token by token as soon as they are generated. For chat interfaces, use .stream() and Flux<String>. Spring Boot automatically converts this to Server-Sent Events (SSE).

Add the dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

Add resources/static/stream-test.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>AI Stream Test</title>
    <style>
        body { font-family: Arial, sans-serif; max-width: 800px; margin: 50px auto; padding: 20px; }
        #question { width: 70%; padding: 10px; font-size: 16px; }
        button { padding: 10px 20px; font-size: 16px; cursor: pointer; }
        #response { margin-top: 20px; padding: 15px; border: 1px solid #ccc; border-radius: 8px; min-height: 150px; background: #f9f9f9; white-space: pre-wrap; font-size: 16px; line-height: 1.6; }
        .cursor { display: inline-block; width: 2px; height: 18px; background: black; animation: blink 0.7s infinite; vertical-align: text-bottom; }
        @keyframes blink { 0%, 100% { opacity: 1; } 50% { opacity: 0; } }
    </style>
</head>
<body>
    <h2>🤖 AI Streaming Chat</h2>
    <div>
        <input type="text" id="question" placeholder="Enter a question..." value="Tell me a short fact about Java">
        <button onclick="startStream()">Send</button>
    </div>
    <div id="response"></div>

    <script>
        function startStream() {
            const question = document.getElementById('question').value;
            const responseDiv = document.getElementById('response');

            // Clear previous response
            responseDiv.innerHTML = '<span class="cursor"></span>';

            // Use fetch + ReadableStream (more reliable than EventSource)
            fetch('/ai/stream?question=' + encodeURIComponent(question))
                .then(response => {
                    const reader = response.body.getReader();
                    const decoder = new TextDecoder('utf-8');

                    function read() {
                        reader.read().then(({ done, value }) => {
                            if (done) {
                                // Remove cursor when the stream ends
                                const cursor = responseDiv.querySelector('.cursor');
                                if (cursor) cursor.remove();
                                return;
                            }

                            // Decode bytes to text
                            const chunk = decoder.decode(value, { stream: true });

                            // Parse SSE: each "data:..." line
                            const lines = chunk.split('\n');
                            let text = '';
                            for (const line of lines) {
                                if (line.startsWith('data:')) {
                                    // Remove "data:" prefix and append to text
                                    text += line.substring(5);
                                }
                            }

                            // Insert text before the cursor
                            const cursor = responseDiv.querySelector('.cursor');
                            if (cursor) {
                                cursor.insertAdjacentText('beforebegin', text);
                            } else {
                                responseDiv.textContent += text;
                            }

                            // Read the next chunk
                            read();
                        });
                    }

                    read();
                })
                .catch(error => {
                    responseDiv.textContent = 'Error: ' + error.message;
                });
        }
    </script>
</body>
</html>

Add the streaming endpoint:

import org.springframework.http.MediaType;
import reactor.core.publisher.Flux;

// ... inside AiController ...

@GetMapping(value = "/ai/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE + ";charset=UTF-8")
public Flux<String> streamResponse(@RequestParam String question) {
    return chatClient.prompt()
            .user(question)
            .stream()
            .content();
}

Open in your browser: http://localhost:8080/stream-test.html

The response will appear gradually, word by word, just like in ChatGPT. This works via SSE (Server-Sent Events) — the browser receives a continuous data stream and renders it in real time.

DEV Community