How to Stream LLM Responses in Spring AI (Server-Sent Events)

Avaneesh Yadav — Tue, 07 Jul 2026 13:40:00 +0000

When you call an LLM the normal way, the user stares at a spinner while the model generates the entire answer, then the whole block appears at once. For anything longer than a sentence, that feels slow — even when the total time is identical to a streamed response. Streaming flips the experience: tokens appear as they're generated, so the user starts reading immediately. It's the single cheapest UX win in any LLM feature, and with Spring AI it's essentially one method call.

This guide covers the whole path: streaming from the ChatClient, exposing it as Server-Sent Events (SSE) in both WebFlux and classic Spring MVC, consuming it on the frontend, and the production details that trip people up.

Why stream at all?

Perceived latency, mostly. A model that takes 6 seconds to produce a 300-word answer feels broken if the user waits 6 seconds for a wall of text — but feels fast if words start flowing at 400ms. Streaming also lets you:

Show progress and allow the user to cancel a bad generation early.
Start downstream processing (rendering markdown, highlighting code) incrementally.
Reduce time-to-first-token as your primary latency metric, which is what users actually feel.

[!NOTE]
Streaming changes when tokens arrive, not the total cost or token count. You're billed the same — you're just improving the experience.

The one line that streams

Spring AI's ChatClient exposes streaming through .stream() instead of .call(). Where .call().content() returns a String, .stream().content() returns a reactive Flux<String> that emits chunks as the model produces them.

@Service
public class ChatService {

    private final ChatClient chatClient;

    public ChatService(ChatClient.Builder builder) {
        this.chatClient = builder.build();
    }

    public Flux<String> stream(String message) {
        return chatClient.prompt()
                .user(message)
                .stream()
                .content();
    }
}

That's the entire model-side change. Everything else is about getting that Flux to the browser.

Expose it as SSE with WebFlux

If your app uses Spring WebFlux, a streaming endpoint is trivial — return the Flux with a text/event-stream content type and Spring handles the SSE framing:

@RestController
public class ChatController {

    private final ChatService chatService;

    public ChatController(ChatService chatService) {
        this.chatService = chatService;
    }

    @GetMapping(value = "/api/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> stream(@RequestParam String message) {
        return chatService.stream(message);
    }
}

Each emitted chunk is sent to the client as its own data: event the moment the model produces it. If you want explicit SSE events (with ids, event names, or heartbeats), map to ServerSentEvent:

@GetMapping(value = "/api/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent<String>> stream(@RequestParam String message) {
    return chatService.stream(message)
            .map(chunk -> ServerSentEvent.builder(chunk).build());
}

No WebFlux? Use SseEmitter in Spring MVC

Plenty of enterprise apps are still on the classic servlet stack. You don't need to migrate to WebFlux — Spring MVC's SseEmitter streams fine. You subscribe to the Flux and push each chunk onto the emitter, completing (or erroring) it when the stream ends:

@GetMapping("/api/chat/stream")
public SseEmitter stream(@RequestParam String message) {
    SseEmitter emitter = new SseEmitter(Duration.ofMinutes(2).toMillis());

    chatService.stream(message).subscribe(
            chunk -> {
                try {
                    emitter.send(SseEmitter.event().data(chunk));
                } catch (IOException e) {
                    emitter.completeWithError(e);
                }
            },
            emitter::completeWithError,   // onError
            emitter::complete             // onComplete
    );

    return emitter;
}

[!TIP]
Give the SseEmitter an explicit timeout. The default can hang a servlet thread indefinitely if the client disappears mid-stream.

Consume the stream on the frontend

For a GET endpoint, the browser's built-in EventSource is the simplest consumer. Concatenate the chunks as they arrive:

function streamChat(message, onToken, onDone) {
  const url = `/api/chat/stream?message=${encodeURIComponent(message)}`;
  const source = new EventSource(url);

  source.onmessage = (event) => onToken(event.data);

  source.onerror = () => {
    // SSE closes with an error event when the server finishes the stream
    source.close();
    onDone();
  };
}

// Usage
let answer = "";
streamChat("Explain vector databases in one paragraph.",
  (token) => { answer += token; render(answer); },
  () => console.log("done"),
);

If you need POST (to send a large body or headers), EventSource won't work — use fetch with a streaming reader instead:

const res = await fetch("/api/chat/stream", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ message }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let answer = "";
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  answer += decoder.decode(value, { stream: true });
  render(answer); // you'll get raw SSE `data:` frames here — strip the prefixes
}

The flow, end to end

sequenceDiagram
    participant B as Browser
    participant C as Controller (SSE)
    participant S as ChatClient
    participant L as LLM Provider
    B->>C: GET /api/chat/stream?message=...
    C->>S: prompt().user(msg).stream().content()
    S->>L: streaming request
    loop each token
        L-->>S: token
        S-->>C: Flux emits chunk
        C-->>B: data: chunk
    end
    L-->>S: [stream end]
    C-->>B: connection closes

Production details that actually bite

Streaming works in a demo on the first try. These are the things that break it in production:

Proxy buffering. Nginx and many API gateways buffer responses by default, which defeats streaming — the client gets everything at once. Disable it for the stream route (proxy_buffering off; in Nginx) and send X-Accel-Buffering: no.
Compression. Response compression can also buffer the stream. Exclude text/event-stream from gzip.
Cancellation. When the user navigates away, the browser closes the connection. In WebFlux this cancels the Flux automatically (and stops the upstream LLM call); with SseEmitter, wire an onCompletion/onTimeout callback to dispose the subscription so you're not billed for tokens nobody will read.
Errors mid-stream. Once the first byte is sent you can't change the HTTP status. Emit a final SSE event with an error payload (e.g. event: error) so the client can distinguish "done" from "failed."
Resilience. LLM calls are flaky network calls. Apply timeouts and a fallback path — see AI system resilience for the patterns.
CORS. If your frontend is on a different origin, SSE still needs the usual CORS headers on the streaming route.

FAQ

Does streaming reduce token cost?
No — you're billed for the same input/output tokens. Streaming only changes when they arrive, improving perceived latency.

Do I have to switch to WebFlux to stream?
No. SseEmitter on classic Spring MVC streams a Flux fine. WebFlux is cleaner if you're already reactive, but it's not required.

Why does the client get everything at once instead of token-by-token?
Almost always a buffering proxy (Nginx/gateway) or response compression. Disable buffering and exclude text/event-stream from gzip.

Can I stream structured output (JSON) too?
You can stream the raw text, but partial JSON isn't parseable until it's complete. For structured responses, prefer .call().entity(...) unless you specifically want to render tokens as they arrive.

How do I stop the LLM call when the user cancels?
In WebFlux, closing the connection cancels the Flux and the upstream request automatically. With SseEmitter, dispose the reactive subscription in an onCompletion/onTimeout handler.

Streaming is the highest-leverage UX change you can make to an LLM feature, and Spring AI reduces it to swapping .call() for .stream(). Get the endpoint right, disable proxy buffering, handle cancellation, and your app will feel dramatically faster for zero extra cost.

New to Spring AI? Start with the Spring AI overview, then see tool calling and the enterprise Spring AI guide. Planning capacity and cost for a streaming feature? Estimate prompt size with the free LLM Token Counter.

Building a production AI feature and want a second pair of eyes on the architecture? Let's talk.

DEV Community: Avaneesh Yadav