DEV Community

Sujan Lamichhane
Sujan Lamichhane

Posted on

Adding Voice to a Java AI Assistant — Whisper, TTS, and the Voice Conversation Loop

#java #ai #opensource #springboot

How we gave Jarvis the ability to hear and speak — Phase 5 of the Jarvis AI Platform

Where We Left Off

After Phase 4, Jarvis could answer questions using real tools.

You: What is the weather in Kathmandu?
Jarvis: [calls WeatherTool] It is 22°C and sunny.

You: What is 2847 × 391?
Jarvis: [calls CalculatorTool] 1,113,177
Enter fullscreen mode Exit fullscreen mode

But every interaction required typing.

Phase 5 changed that.

The Goal 

BEFORE Phase 5:
You type  → Jarvis types back

AFTER Phase 5:
You speak → Whisper transcribes → AI responds → TTS speaks back
Enter fullscreen mode Exit fullscreen mode

Simple to describe.

Surprisingly nuanced to build correctly.

The First Surprise — Ollama Does Not Support Whisper

The original plan was to run Whisper locally via Ollama.

ollama pull whisper

## Error:
pull model manifest: file does not exist
Enter fullscreen mode Exit fullscreen mode

Ollama is excellent for language models.

It does not support audio transcription models.

This forced a rethink.

The Solution — Two Modes

We designed WhisperTranscriptionService to support two backends.

Mode 1 — Groq API (Cloud)

Groq provides Whisper large-v3-turbo through an OpenAI-compatible API.

The free tier offers 6,000 requests/day with no credit card required.

Set GROQ_API_KEY in .env

↓

Works immediately
Enter fullscreen mode Exit fullscreen mode

Mode 2 — Local whisper.cpp

For users who want completely local transcription:

git clone https://github.com/ggerganov/whisper.cpp

cd whisper.cpp

make

bash ./models/download-ggml-model.sh base.en

./server -m models/ggml-base.en.bin --port 8178
Enter fullscreen mode Exit fullscreen mode

Both implementations expose the same OpenAI-compatible multipart API.

Switching between them is a single configuration flag.

Architecture — The Key Decision

The most important architectural decision of Phase 5 was this:

Voice is only a wrapper around the existing chat pipeline.

AiOrchestrator does not change.

❌ WRONG

Voice Pipeline
   ↓
Different AI Pipeline
   ↓
Different Memory
   ↓
Different Tools


✅ CORRECT

Audio
   ↓
Whisper
   ↓
Text
   ↓
AiOrchestrator.chat()
   ↓
Existing Memory
Existing RAG
Existing Tools
   ↓
Text
   ↓
Text-to-Speech
Enter fullscreen mode Exit fullscreen mode

Everything built in Phases 1–4 continues working automatically.

WhisperTranscriptionService 

@Service
public class WhisperTranscriptionService {

    private final WebClient webClient;
    private final String apiKey;
    private final String model;
    private final boolean isLocalMode;

    public Mono<String> transcribe(byte[] audioBytes) {

        if (audioBytes == null || audioBytes.length == 0) {
            return Mono.error(VoiceException.emptyAudio());
        }

        if (!isConfigured()) {
            return Mono.error(new VoiceException(
                    "WHISPER_NOT_CONFIGURED",
                    "Set GROQ_API_KEY in .env or start whisper.cpp server",
                    HttpStatus.SERVICE_UNAVAILABLE));
        }

        return Mono.fromCallable(() ->
                        callWhisperApi(audioBytes, null))
                .subscribeOn(Schedulers.boundedElastic());
    }
}
Enter fullscreen mode Exit fullscreen mode

Two design choices are worth highlighting.

Schedulers.boundedElastic()

Calling Groq or whisper.cpp is blocking I/O.

Running it on the WebFlux event loop would block every request.

boundedElastic() keeps the reactive event loop free.

isLocalMode

Local whisper.cpp requires no API key.

One boolean changes the backend without changing any business logic.

Text-to-Speech — The Cross-Platform Challenge

Instead of adding another dependency, we chose native OS speech engines.

Why?

  • Zero additional libraries
  • No API keys
  • Works immediately
  • Good enough quality
  • Fully offline

Platform support:

Windows → PowerShell + System.Speech.Synthesis

macOS   → say

Linux   → espeak / text2wave
Enter fullscreen mode Exit fullscreen mode

The service detects the platform once at startup.

private static final boolean IS_WINDOWS = OS.contains("win");
private static final boolean IS_MAC = OS.contains("mac");
private static final boolean IS_LINUX =
        OS.contains("nux") || OS.contains("nix");
Enter fullscreen mode Exit fullscreen mode

Voice configuration is entirely environment-driven.

## Windows
JARVIS_VOICE_NAME=Microsoft Zira Desktop

## macOS
JARVIS_VOICE_NAME=Samantha

## Linux
JARVIS_VOICE_NAME=en+f3

## Speed
JARVIS_VOICE_SPEED=1.2
Enter fullscreen mode Exit fullscreen mode

DST Awareness — A Surprisingly Tricky Bug

A code review caught this subtle issue.

// ❌ Wrong
TimeZone.getTimeZone(zoneId)
        .getDisplayName(false,
                TimeZone.LONG,
                Locale.ENGLISH);
Enter fullscreen mode Exit fullscreen mode

That always reports Standard Time.

The correct implementation derives the current DST state.

boolean isDst =
        TimeZone.getTimeZone(zoneId)
                .inDaylightTime(Date.from(now.toInstant()));

TimeZone.getTimeZone(zoneId)
        .getDisplayName(
                isDst,
                TimeZone.LONG,
                Locale.ENGLISH);
Enter fullscreen mode Exit fullscreen mode

Without this fix, users in DST regions would see incorrect timezone names for half the year.

The Sentence Buffering Problem

LLMs stream tokens.

"The"

"weather"

"in"

"London"

"is"

"22"

"°"

"C"

"and"

"sunny"

"."
Enter fullscreen mode Exit fullscreen mode

Reading individual tokens aloud sounds terrible.

The solution was sentence buffering.

private void startTtsPipeline(Flux<String> tokenStream) {

    StringBuilder buffer = new StringBuilder();

    tokenStream
            .flatMap(token -> {

                buffer.append(token);

                boolean isSentenceEnd =
                        isSentenceBoundary(token);

                boolean isBufferFull =
                        buffer.toString()
                              .split("\\s+").length
                              >= MAX_BUFFER_TOKENS;

                if (isSentenceEnd || isBufferFull) {

                    String sentence =
                            buffer.toString().trim();

                    buffer.setLength(0);

                    if (!sentence.isBlank()) {
                        return Flux.just(sentence);
                    }
                }

                return Flux.<String>empty();
            })

            .concatMap(textToSpeechService::speakAndPlay)

            .subscribeOn(Schedulers.boundedElastic())

            .subscribe();
}
Enter fullscreen mode Exit fullscreen mode

Three implementation details matter.

concatMap()

Sentences must play sequentially.

Using flatMap() would overlap multiple audio streams.

MAX_BUFFER_TOKENS

Some AI responses contain no punctuation.

After 50 words we flush automatically.

Background execution

Speech generation happens on boundedElastic().

The browser continues receiving streamed tokens immediately.

The Two-Pipeline Architecture

The first implementation blocked streaming.

❌ Wrong

Token
 ↓
TTS
 ↓
Next Token
Enter fullscreen mode Exit fullscreen mode

Terrible user experience.

The final architecture separates streaming from speech.

               Token Stream
                    │
      ┌─────────────┴─────────────┐
      │                           │
      ▼                           ▼

Browser SSE               Sentence Buffer

Immediate                  Background

      │                           │
      ▼                           ▼

Real-time UI             Text-to-Speech
Enter fullscreen mode Exit fullscreen mode

Implementation:

public Flux<VoiceChatEvent> voiceChat(...) {

    Flux<String> tokenStream =
            orchestrator.chat(request);

    // Background TTS
    startTtsPipeline(tokenStream);

    // Immediate SSE
    return sessionEvent.concatWith(
            tokenStream.map(VoiceChatEvent::token));
}
Enter fullscreen mode Exit fullscreen mode

The browser updates instantly.

Speech begins as soon as the first sentence is complete.

Neither blocks the other.

VoiceChatEvent

The SSE stream emits strongly typed events.

public record VoiceChatEvent(
        EventType type,
        String data
) {

    public enum EventType {
        SESSION,
        TOKEN,
        DONE
    }

    public static VoiceChatEvent session(UUID id) {
        return new VoiceChatEvent(
                EventType.SESSION,
                id.toString());
    }

    public static VoiceChatEvent token(String text) {
        return new VoiceChatEvent(
                EventType.TOKEN,
                text);
    }
}
Enter fullscreen mode Exit fullscreen mode

The initial SESSION event solves a practical problem.

If the server creates a brand-new conversation, the frontend immediately receives the generated session ID for future requests.

REST API

Five endpoints power the voice system.

POST /api/v1/voice/transcribe
POST /api/v1/voice/speak
POST /api/v1/voice/speak/bytes
POST /api/v1/voice/chat
GET  /api/v1/voice/status
Enter fullscreen mode Exit fullscreen mode

Two speech endpoints exist for different use cases.

/speak

  • Plays audio directly on the server
  • Ideal for CLI usage

/speak/bytes

  • Returns WAV bytes
  • Intended for browsers and desktop clients

What We Learned

Ollama doesn't support audio models

The original plan was simply wrong.

Community feedback caught this before implementation.

Blocking work must be isolated

Every Whisper request is blocking.

Every TTS process is blocking.

Everything runs on boundedElastic().

festival --tts cannot generate files

It only plays audio.

Linux audio generation requires:

text2wave -o output.wav
Enter fullscreen mode Exit fullscreen mode

or Festival's Scheme interface.

Process cleanup matters 

if (!process.waitFor(
        TIMEOUT_SECONDS,
        TimeUnit.SECONDS)) {

    process.destroyForcibly();

    log.warn("TTS generation process timed out");
}
Enter fullscreen mode Exit fullscreen mode

Ignoring waitFor() leaves orphaned child processes.

DST is genuinely difficult

Timezone names depend on the actual instant, not simply the timezone itself.

Voice Status

Before enabling voice, clients can verify availability.

curl http://localhost:8080/api/v1/voice/status \
     -H "Authorization: Bearer $TOKEN"
Enter fullscreen mode Exit fullscreen mode 
{
  "success": true,
  "data": {
    "transcriptionAvailable": true,
    "ttsAvailable": true,
    "voiceReady": true,
    "transcriptionMode": "groq-cloud",
    "ttsEngine": "system-macos"
  }
}
Enter fullscreen mode Exit fullscreen mode

transcriptionMode

  • groq-cloud
  • local-whisper

ttsEngine

  • system-windows
  • system-macos
  • system-linux

A Complete Voice Conversation 

User speaks

"What is the weather in Kathmandu?"

        │
        ▼

Whisper
(Groq / whisper.cpp)

        │
        ▼

"What is the weather in Kathmandu?"

        │
        ▼

AiOrchestrator.chat()

    ├── Session History
    ├── Long-Term Memory
    ├── RAG Context
    └── Tool Calling

        │
        ▼

WeatherTool("Kathmandu")

        │
        ▼

"The weather in Kathmandu is
22°C and clear."

        │
        ├──────────────► Browser (SSE)
        │
        └──────────────► Text-to-Speech
Enter fullscreen mode Exit fullscreen mode

Nothing in the AI pipeline changes.

Voice simply wraps the architecture built during Phases 1–4.

What's Next

Phase 6 introduced the Agent System, allowing Jarvis to plan and execute multi-step tasks autonomously.

Phase 7 brings a complete web interface over everything we've built.

The backend is now complete.

Phases 1–6 are merged, tested, and production-ready.

Jarvis can now hear.

Jarvis can now speak.

Contributing

Jarvis is open source under the Apache 2.0 License.

Current contributor-friendly issues include:

#69  CLI voice commands (voice, listen, speak)

New  Voice integration tests
Enter fullscreen mode Exit fullscreen mode

GitHub:

github.com/sujankim/jarvis-ai-platform
Enter fullscreen mode Exit fullscreen mode

Jarvis AI Platform Series

  • Part 1 — Building a Local-First AI Assistant with Spring Boot 4
  • Part 2 — Building Long-Term Memory with pgvector
  • Part 3 — Implementing Semantic Memory Retrieval
  • Part 4 — Building a Tool Engine with Spring AI
  • Part 5 — Adding Voice with Whisper and Text-to-Speech (this article)
  • Part 6 — Building an AI Agent System with the ReAct Pattern (coming next)

Your AI. Your Data. Your Machine.

Top comments (0)