DEV Community

Alain Airom (Ayrom)
Alain Airom (Ayrom)

Posted on

Building an End-to-End Multilingual Speech and LLM Pipeline with IBM Granite Speech 5.0-2B and Bob

#granite #speechtotext #bob #opensource

Testing the latest granite 🪨 speech 5 capabilities on Apple Silicon.

Introduction

As speech-to-text workflows transition toward more deeply integrated architectures, developers need models that don't just transcribe audio mechanically, but understand context, intent, and translation across multiple languages natively. Enter IBM Granite Speech 5.0-2B (ibm-granite/granite-speech-5.0-2b)—a high-performance, open-weights model leveraging a two-pass design (a CTC speech encoder coupled directly with a Granite-4.0-1B LLM backbone). Released under the permissive Apache 2.0 licence, it delivers state-of-the-art multilingual Automatic Speech Recognition (ASR) and Speech Translation (AST) across English, French, German, Spanish, Portuguese, Italian, and Japanese.

In this guide, we will walk through how to construct a complete, production-ready end-to-end demonstration application. We'll use Bob—our local developer workspace tool—to orchestrate the project structure, and connect Granite Speech's powerful transcription and translation features directly to downstream LLM backends (such as Ollama, llama.cpp, or OpenAI) via a unified Gradio web UI.

IBM Granite Speech 5

Excerpt from Hugging Face model card

Granite Speech 5.0 2B is a compact and efficient speech-language model, specifically designed for multilingual automatic speech recognition (ASR) and bidirectional automatic speech translation (AST) for English, French, German, Spanish, Portuguese and Japanese.

The model was trained on 390,000 hours of audio from public corpora for ASR and AST as well as synthetic datasets tailored to support Japanese ASR, keyword-biased ASR and speech translation. Granite Speech 5.0 2B was trained by modality aligning an intermediate checkpoint of granite-4.0-1b-base to speech on publicly available open source corpora containing audio inputs and text targets. Compared to its predecessor granite-speech-4.1-2b, this model has the same parameter count and provides additional capabilities and improvements:

  • Higher transcription accuracy for English short-form ASR and increased robustness to noise, reverberation and distant microphone speech
  • Supports input language id prompting for multilingual ASR
  • Improved long-form and conversational speech translation
  • Increased keyword list size (up to 2000) for multilingual keyword-biased ASR and AST

The mode card page proposes several sample test codes and implementations runnable as is.

  • Sample code using Transformers; 
  # Usage with transformers
  # pip install transformers torchaudio soundfile

  import torch
  import torchaudio
  from huggingface_hub import hf_hub_download
  from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

  device = "cuda" if torch.cuda.is_available() else "cpu"

  model_name = "ibm-granite/granite-speech-5.0-2b"
  processor = AutoProcessor.from_pretrained(model_name)
  tokenizer = processor.tokenizer
  model = AutoModelForSpeechSeq2Seq.from_pretrained(
      model_name, device_map=device, torch_dtype=torch.bfloat16
  )

  # Load audio
  audio_path = hf_hub_download(repo_id=model_name, filename="multilingual_sample.wav")
  wav, sr = torchaudio.load(audio_path, normalize=True)
  assert wav.shape[0] == 1 and sr == 16000  # mono, 16kHz

  # Create text prompt
  user_prompt = "<|audio|>transcribe the speech with proper punctuation and capitalization."
  chat = [
      {"role": "user", "content": user_prompt},
  ]
  prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)

  # Run the processor + model
  model_inputs = processor(prompt, wav, device=device, return_tensors="pt").to(device)
  model_outputs = model.generate(
      **model_inputs, max_new_tokens=200, do_sample=False, num_beams=1
  )

  # Transformers includes the input IDs in the response
  num_input_tokens = model_inputs["input_ids"].shape[-1]
  new_tokens = model_outputs[0, num_input_tokens:].unsqueeze(0)
  output_text = tokenizer.batch_decode(
      new_tokens, add_special_tokens=False, skip_special_tokens=True
  )
  print(f"STT output = {output_text[0]}")
Enter fullscreen mode Exit fullscreen mode

  • Sample code using vLLM;

    # usage vllm
# pip install vllm

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
from vllm.assets.audio import AudioAsset

model_id = "ibm-granite/granite-speech-5.0-2b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

def get_prompt(question: str, has_audio: bool):
    """Build the input prompt to send to vLLM."""
    if has_audio:
        question = f"<|audio|>{question}"
    chat = [
        {
            "role": "user",
            "content": question
        }
    ]
    return tokenizer.apply_chat_template(chat, tokenize=False)

model = LLM(
    model=model_id,
    max_model_len=2048, # This may be needed for lower resource devices.
    limit_mm_per_prompt={"audio": 1},
)

question = "can you transcribe the speech into a written format?"
prompt_with_audio = get_prompt(
    question=question,
    has_audio=True,
)
audio = AudioAsset("mary_had_lamb").audio_and_sample_rate

inputs = {
    "prompt": prompt_with_audio,
    "multi_modal_data": {
        "audio": audio,
    }
}

outputs = model.generate(
    inputs,
    sampling_params=SamplingParams(
        temperature=0.0,
        max_tokens=64,
    ),
)
print(f"Audio Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")

Sample Application Features and Functions

The application implements the following features;

  • Speech Transcription: upload a recorded audio file or speak directly into your microphone. The system converts speech to accurate written text using IBM's Granite Speech 5.0-2B model — trained specifically for high-quality transcription in real-world conditions.

  • Multilingual Translation: spoken audio in English, French, German, Spanish, Portuguese, or Japanese is automatically translated into any other supported language. Ideal for international teams, multilingual content production, and cross-border customer support.

The example above is a sample translation from English to French. Other languages are available too, such as German for another test;

  Der Bezirksarzt näherte sich mit der Gleichgültigkeit eines Mannes, der es gewohnt ist, die Hälfte seiner Zeit unter den Toten zu verbringen. Er hob dann das über das Gesicht gelegte Blatt und öffnete nur die Lippen.
Enter fullscreen mode Exit fullscreen mode
  • AI-Assisted Analysis: once a transcript is produced, it is forwarded to an AI language model with your custom instruction — "summarise this meeting," "extract all action items," "identify the customer's main concern." The AI response appears alongside the transcript.

  • One-Click Full Pipeline: the Full Pipeline tab combines transcription and AI analysis into a single button click. Provide audio and an instruction; the system handles everything end-to-end and saves a structured result file automatically.

  • Live Configuration: switch AI backends, change models, or adjust processing settings from within the interface — no restart required. Configurations take effect immediately, enabling rapid A/B comparison of different AI backends.

  • Hardware Intelligence: the system automatically detects the capabilities of the host machine — GPU type, available memory, CPU cores — and self-optimises its processing parameters accordingly, ensuring the best performance on every device.

Architecture

The entire pipeline is structured to ensure modularity, separating hardware capability detection, audio processing, configuration management, and multi-backend LLM interfacing.

Implementation

Project Layout & Core Setup

Project's file and organization:

granite-speech5-test/
├── app.py                  # Gradio demo application (entry point)
├── requirements.txt        # Python dependencies
├── .env.example            # Environment variable template
├── config/
│   └── config.yaml         # Application configuration
├── src/
│   ├── speech_processor.py # Granite Speech 5.0-2B wrapper
│   ├── hardware_detector.py# Hardware capability detection
│   ├── llm_client.py       # Multi-backend LLM abstraction
│   └── config_manager.py   # Configuration management
├── tests/
│   ├── test_speech_processor.py
│   ├── test_hardware_detector.py
│   ├── test_llm_client.py
│   └── test_config_manager.py
├── scripts/
│   ├── setup.sh            # One-time environment setup
│   ├── start.sh            # Launch app in background
│   ├── stop.sh             # Graceful shutdown
│   └── cleanup.sh          # Remove artefacts / venv
├── input/                  # Put audio files here
├── output/                 # JSON results written here
└── Docs/
    ├── Quickstart.md
    └── Architecture.md
Enter fullscreen mode Exit fullscreen mode

Hardware Detection & Optimization Advisor

Before loading models, the application queries host resources to assign the optimal compute device (CUDA GPU, Apple Silicon MPS, or CPU) and precision dtype:

"""
Hardware capability detection and optimisation advisor.

Interrogates the host machine for CPU cores, RAM, GPU VRAM and Apple Silicon
presence, then returns a structured HardwareProfile and suggested model-loading
parameters.

No external heavy dependencies – falls back gracefully when optional packages
(torch, psutil) are unavailable so that the module can be imported during the
install phase.
"""

from __future__ import annotations

import logging
import os
import platform
import subprocess
import sys
from dataclasses import dataclass, field
from typing import Optional

...

# Optional torch import at module level so tests can patch it cleanly.
try:
    import torch  # type: ignore
except ImportError:
    torch = None  # type: ignore

logger = logging.getLogger(__name__)
Enter fullscreen mode Exit fullscreen mode

Core Speech Processing Wrapper

The core speech module (GraniteSpeechProcessor) handles raw audio files, normalizes inputs to 16 kHz mono waveforms, and executes inference using the proper prompt templates specified by the model card:

"""
IBM Granite Speech 5.0-2B — Core speech processing module.

Model reference: https://huggingface.co/ibm-granite/granite-speech-5.0-2b
Architecture: two-pass design (CTC speech encoder → Granite-4.0-1B LLM backbone)
Supported tasks: ASR (transcription) and AST (translation) in 6 languages.

Usage constraints (Apache 2.0 licence, no additional restrictions):
- Audio must be 16 kHz mono; longer recordings should be split into ≤2-minute segments.
- Prompt tokens use the <|audio|> placeholder which the processor replaces with audio embeddings.

Performance guide
-----------------
On CPU (M-series or x86) a 2-second clip takes ~60-120 s with float32 because the
Granite-4.0-1B LLM backbone must run greedy-decode token-by-token.  To reach
real-time speed, one of the following is needed:

  1. CUDA GPU (any card ≥ 4 GB VRAM) → bfloat16 → 1-5× real-time
  2. Apple Silicon with MPS           → float16  → ~5× real-time
  3. INT-8 dynamic quantisation       → ~2× CPU speedup (built-in, no GPU)
  4. Reduce max_new_tokens            → linear speedup (safe for short clips)
  5. ONNX runtime (WebGPU / CoreML)   → best CPU/edge throughput

All optimisation flags are controlled via config/config.yaml [speech.perf].
"""

from __future__ import annotations

import logging
import time
from contextlib import contextmanager
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional

import numpy as np
import torch

from src.speech_processor import GraniteSpeechProcessor, PerfConfig
from src.hardware_detector import detect_hardware

hw = detect_hardware()

# Initialise processor with performance profile
processor = GraniteSpeechProcessor(
    device=hw.recommended_device,
    perf=PerfConfig(int8_quantize=False, profile=True)
)
...
# Transcribe audio file
result = processor.transcribe("input/meeting.wav", language="en", verbatim=False)
print("Transcript:", result.text)
print(f"Real-Time Factor (RTFx): {result.realtime_factor:.1f}×")

# Translate speech (e.g., French audio to English text)
translation = processor.translate("input/french_speech.wav", source_language="fr", target_language="en")
print("Translation:", translation.text)
Enter fullscreen mode Exit fullscreen mode

Downstream Processing with Multi-Backend LLMs

Once a transcript is extracted, you can pipe it directly into local or cloud LLM backends using a uniform interface:

"""
LLM backend abstraction layer.

Provides a unified LLMClient interface over multiple local and cloud backends:

  Local
  -----
  • Ollama   – http://localhost:11434 (default)
  • LM Studio – http://localhost:1234  (OpenAI-compatible endpoint)
  • llama.cpp server – http://localhost:9931  (project rule: llama.cpp port)

  Cloud
  -----
  • OpenAI   – https://api.openai.com/v1
  • Anthropic – Claude via Messages API
  • IBM watsonx.ai – https://us-south.ml.cloud.ibm.com (optional)

All backends expose the same .chat(messages) → str interface so the
application layer never needs to know which backend is active.

Usage
-----
>>> from src.llm_client import build_llm_client
>>> llm = build_llm_client(backend="ollama", model="granite3.3:8b")
>>> llm.chat([{"role": "user", "content": "Summarise this transcript: …"}])
"""

from __future__ import annotations

import json
import logging
import time
import urllib.request  # imported at module level for testability
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Any, Optional

logger = logging.getLogger(__name__)

...

# Build a local Ollama client instance pointing to Granite or Llama models
llm = build_llm_client(backend="ollama", model="granite4.2:latest")

response = llm.chat([
    {
        "role": "user", 
        "content": f"Transcript:\n{result.text}\n\nInstruction: Summarise the key action items concisely."
    }
])

print("LLM Reply:", response.text)
print(f"Latency: {response.latency_s:.2f}s | Tokens used: {response.input_tokens} → {response.output_tokens}")
Enter fullscreen mode Exit fullscreen mode

End-to-End Application Process

The user interface combines transcription, translation, and LLM interaction tabs into a unified local control center (app.py).

"""
IBM Granite Speech 5.0-2B — End-to-End Demo Application (Gradio UI).

Tabs
────
1. Transcribe   – upload or record audio → Granite Speech → text output
2. Translate    – speech translation between supported language pairs
3. LLM Interact – send a transcript to the configured LLM backend
4. Config       – inspect and switch settings at runtime
5. Hardware     – display host hardware profile

Run
───
    python app.py
    # or via scripts/start.sh
"""

from __future__ import annotations

import json
import logging
import os
import time
import urllib.request
import urllib.error
from datetime import datetime
from pathlib import Path
from typing import Optional, Tuple

import gradio as gr

from src.config_manager import get_config, reload_config
from src.hardware_detector import detect_hardware
from src.llm_client import build_llm_client
from src.speech_processor import (
    GraniteSpeechProcessor,
    PerfConfig,
    SpeechResult,
    SUPPORTED_LANGUAGES,
)

logger = logging.getLogger(__name__)

...

# ── Entry point ───────────────────────────────────────────────────────────────

def main() -> None:
    cfg = get_config()

    # Ensure output directory exists
    Path(cfg.app.output_dir).mkdir(parents=True, exist_ok=True)

    # Warm up hardware detection
    logger.info("Detecting hardware …")
    hw = _get_hardware_profile()
    logger.info("\n%s", hw.summary_str())

    demo = build_ui()

    # Build SSL kwargs — when cert/key are present the app serves HTTPS,
    # which is required for the browser microphone API to work on localhost.
    ssl_kwargs: dict = {}
    root = Path(__file__).parent
    certfile = cfg.app.ssl_certfile or str(root / "certs" / "cert.pem")
    keyfile = cfg.app.ssl_keyfile or str(root / "certs" / "key.pem")
    if Path(certfile).exists() and Path(keyfile).exists():
        ssl_kwargs = {
            "ssl_certfile": certfile,
            "ssl_keyfile": keyfile,
            # ssl_verify=False tells Gradio's internal httpx health-check to
            # skip cert validation for its own loopback call — required when
            # using a self-signed certificate, otherwise Gradio crashes on
            # startup with CERTIFICATE_VERIFY_FAILED before the browser even
            # connects.
            "ssl_verify": False,
        }
        scheme = "https"
        logger.info("TLS enabled — serving over HTTPS (microphone will work in browser)")
    else:
        scheme = "http"
        logger.warning(
            "No TLS cert found at %s — microphone requires HTTPS.\n"
            "  Run: openssl req -x509 -newkey rsa:2048 -nodes "
            "-keyout certs/key.pem -out certs/cert.pem -days 3650 "
            '-subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"',
            certfile,
        )

    logger.info("App URL: %s://localhost:%d", scheme, cfg.app.server_port)
    demo.launch(
        server_port=cfg.app.server_port,
        share=cfg.app.share,
        server_name="0.0.0.0",
        **ssl_kwargs,
    )


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Launch the app via your terminal and use the full pipeline.

Conclusion

By combining IBM Granite Speech 5.0-2B with a modular application architecture, developers gain an either a cloud based or an air-gappable environment, highly accurate pipeline capable of bridging audio input with advanced reasoning engines[cite: 5]. Whether you're targeting low-latency GPU edge nodes, running offline speech translation on Apple Silicon, or hooking local models into Ollama, this reference pattern gives you total data sovereignty and performance observability out of the box.

Thanks for reading 🎙️

Links

Top comments (0)