The app is live. Users can chat with the AI and it remembers the conversation.
But there's one problem I noticed immediately after deploying: a user sends a message, and then they stare at a completely blank screen for three to five seconds. Then — bam — the full reply appears all at once.
Compare that to ChatGPT or Gemini's own interface. You type a question and the response starts flowing immediately, word by word. It feels alive.
That difference is streaming. This article is about adding it.
Why the delay exists
When you call .call().content() in Spring AI, here's what actually happens:
- Your message goes to the Gemini API
- Gemini starts generating the response — one token at a time
- Gemini keeps generating until the response is complete
- Gemini sends the entire response back to your server in one HTTP response
- Your server returns it to the client
The model isn't slow — it's generating tokens quickly. But you're waiting for the last token before you get the first one. All that generation time appears as a blank screen.
Streaming flips this. Instead of waiting for everything, you receive each token the moment the model generates it — and send it to the client immediately.
What SSE is
Server-Sent Events (SSE) is a simple HTTP mechanism for the server to push data to the client over a single connection that stays open.
Normal HTTP: client sends request → server sends response → connection closes.
SSE: client sends request → server keeps the connection open and sends data in chunks as it becomes available → connection closes when the stream ends.
It's one-directional (server → client only), which is exactly what we need for streaming AI responses. The client asks once, and the server streams the answer back token by token.
What changes in the code
Right now the chat endpoint looks like this:
@PostMapping("/chat-ai")
public ResponseEntity<Map<String, String>> chat(@RequestBody Map<String, String> request) {
String answer = chatClient.prompt()
.user(request.get("message"))
.advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
.call() // ← waits for the full response
.content();
return ResponseEntity.ok(Map.of("answer", answer));
}
The streaming version changes two things:
-
.call()becomes.stream() - The return type becomes
Flux<String>instead ofResponseEntity<String>
@PostMapping(value = "/chat-ai/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> streamChat(@RequestBody Map<String, String> request) {
return chatClient.prompt()
.user(request.get("message"))
.advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
.stream() // ← returns a Flux, not a String
.content();
}
That's it. The same chatClient, the same memory advisor, the same conversation ID — just .stream() instead of .call().
What is Flux?
Flux<String> is from Project Reactor — the reactive library that Spring WebFlux is built on. Think of it as a sequence of values that arrive over time, not all at once.
.call().content() gives you one String — the complete reply.
.stream().content() gives you a Flux<String> — a stream of token chunks that arrive one by one as Gemini generates them.
Spring automatically serialises a Flux<String> return type as SSE when you set produces = MediaType.TEXT_EVENT_STREAM_VALUE. Each item in the Flux becomes a data: event in the SSE stream.
You don't need to add any WebFlux dependency — Spring AI's streaming support works in a regular Spring Boot MVC application. The Flux return type is handled automatically.
Add the dependency
Make sure you have the Reactor dependency. If you're using spring-boot-starter-web, add:
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
</dependency>
Spring Boot manages the version through the BOM, so no version needed.
Full streaming controller
@RestController
@RequestMapping("/api")
public class StreamingChatController {
private final ChatClient chatClient;
public StreamingChatController(ChatClient.Builder builder, JdbcTemplate jdbcTemplate) {
JdbcChatMemoryRepository memoryRepository = JdbcChatMemoryRepository.builder()
.jdbcTemplate(jdbcTemplate)
.dialect(new ChatHistoryDialect())
.build();
MessageWindowChatMemory memory = MessageWindowChatMemory.builder()
.chatMemoryRepository(memoryRepository)
.maxMessages(20)
.build();
this.chatClient = builder
.defaultSystem("You are a helpful assistant.")
.defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())
.build();
}
// Non-streaming — returns the full reply at once
@PostMapping("/chat-ai")
public String chat(@RequestBody Map<String, String> request) {
return chatClient.prompt()
.user(request.get("message"))
.advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
.call()
.content();
}
// Streaming — returns tokens as they arrive
@PostMapping(value = "/chat-ai/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> streamChat(@RequestBody Map<String, String> request) {
return chatClient.prompt()
.user(request.get("message"))
.advisors(a -> a.param("chat_memory_conversation_id", request.get("conversationId")))
.stream()
.content();
}
record ChatRequest(String conversationId, String message) {}
}
Both endpoints share the same chatClient with the same memory. The streaming endpoint is just an extra route — your existing non-streaming endpoint keeps working.
Test it with curl
Start the app, then run:
curl -N -X POST http://localhost:8080/api/chat-ai/stream \
-H "Content-Type: application/json" \
-d '{"conversationId":"test-123","message":"Explain what a Docker container is in simple terms"}'
The -N flag disables buffering — without it curl would collect everything before displaying it, which defeats the purpose.
You should see tokens arriving one by one in your terminal, not all at once:
data:A
data: Docker
data: container
data: is
data: like
data: a
data: lightweight
data: virtual
data: machine
...
Each data: line is one SSE event — one token chunk from the model.
What about the frontend?
The browser has a built-in EventSource API for consuming SSE, but it only supports GET requests. Since your endpoint is a POST (because you're sending a JSON body), you use the Fetch API with a readable stream instead:
const response = await fetch('/api/chat-ai/stream', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ conversationId, message })
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
// each chunk is "data: token\n\n" — parse and append to your UI
const token = chunk.replace(/^data: /, '').trim();
appendToUI(token);
}
Each chunk you read from the stream is a token. Append it to the message div as it arrives — same effect as ChatGPT's typing animation.
Streaming vs non-streaming — when to use which
| Non-streaming | Streaming | |
|---|---|---|
| Response time to first byte | Slow (waits for full reply) | Instant (first token arrives immediately) |
| UX | Blank screen then full reply | Tokens appear as they're generated |
| Frontend complexity | Simple fetch + JSON | Fetch + ReadableStream parsing |
| Good for | Internal APIs, background processing | Any user-facing chat interface |
Keep both endpoints. Use the streaming one for your frontend chat UI. Use the non-streaming one for internal calls where you need the complete reply as a string — tool calling, post-processing, logging the full response.
What's next
The app now streams responses like ChatGPT. But streaming means more API calls, and more API calls means more tokens — and tokens cost money. Next: understanding and managing API cost before your first surprise bill.
Noticed a difference in feel between streaming and waiting for the full reply? Drop it in the comments.
Sham Prakash K — Backend Engineer, 4+ years in Java, Spring Boot, and distributed systems. Building AI backend infrastructure. Writing about what I actually learned, mistakes included.
Top comments (0)