Part 5 (bonus): Implementing Multi-LLM Routing in Go for performance
Introduction
Throughout this series, we have treated the LLM orchestration layer as an intelligent optimization problem. In my previous posts, the design of a mathematical utility model to dynamically match incoming prompts against a 4-tier pool of LLM “T-Shirt sizes” was showcased:
Where Q represents the expected quality score of a model given a task’s computed complexity, C represents normalized cost units, and the weights are bounded by β=1−α to represent a strict financial-versus-performance optimization tradeoff.
We (Bob and I) successfully implemented this routing layer in Python using FastAPI, leveraging the LiteLLM gateway to unify local Ollama nodes (granite4.1:3b, llama3.2, gemma3:4b, and mistral-small), and we integrated a live telemetry loop to update acceptable quality thresholds over a rolling execution window.
But as production workloads scale out, the performance characteristics of the orchestrator itself become critical. As traffic densities climb into hundreds of multi-task prompts per second, managing Python’s runtime environment, navigating single-threaded async event loop saturation under heavy CPU classification tasks, and dealing with significant baseline container footprints add unwanted operational friction.
In this fifth and final episode, let’s take the ultimate leap: compiling down to metal. We will walk through how the entire multi-LLM router/scheduler into a zero-dependency, hyper-performant engine written in pure Go (routing-v5) and how it was ported and translated. This post dissects and shows how replacing high-level abstractions with raw Go concurrency patterns, native memory mutexes, ring-buffer telemetry primitives, and strict circuit breaker state machines allows us to cut operational latency to the microsecond level while dramatically simplifying our production deployment topology.
Why Go? The Performance Trade-off of the AI Orchestrator
When evaluating an LLM gateway or router, engineers frequently assume that downstream model inference latency (the time an LLM spends inside Ollama generating tokens on CPUs and/or GPUs) entirely dwarfs the orchestrator’s overhead. In a trivial sequential chat setup, that is true. However, when a compound developer prompt is split into multiple sub-tasks that require simultaneous token classification, dynamic threshold lookups, concurrency synchronization, feedback auditing, and real-time streaming updates over WebSockets, the orchestrator’s language runtime becomes the primary bottleneck for system throughput.
Let’s look at the concrete operational paradigm shift achieved by moving from our Python FastAPI stack (v4) to our compiled pure Go engine (v5):
| Operational Metric | Python Stack (v4) | Go Engine (v5) |
| ----------------------------- | --------------------------------------------------------- | ------------------------------------------------------------ |
| **Runtime Dependencies** | Heavy `venv` (FastAPI, Uvicorn, LiteLLM, Pydantic, HTTPX) | **Zero** (Single self-contained native machine binary) |
| **Baseline Memory Footprint** | ~75MB – 120MB idle baseline | **~12MB idle baseline** |
| **Concurrency Architecture** | `asyncio` loop (Single-threaded multiplexing) | **Native Goroutines** (True multi-threaded OS thread multiplexing) |
| **State Thread-Safety** | Asynchronous application primitives | **Native Mutexes (`sync.RWMutex`) & Memory Channels** |
| **CPU-Bound Classification** | Blocks or requires complex process pools | Fully non-blocking across available CPU cores |
By adopting Go, the application eliminates interpreter latency. We replace layers of dependency wrappers with an optimized execution tree where token counting, regex-based task parsing, and mathematical utility optimizations execute in microseconds, leaving the system’s entire resource pool free to manage I/O throughput and concurrent client connections.
End-to-End System & Core Data Flow Blueprint
To understand the low-overhead architecture of the Go engine, we must trace how a compound prompt is ingested, processed, executed, and streamed. The system treats incoming text payloads as an optimization graph to be aggressively executed in parallel.
┌──────────────────────────────────────────────────────────────┐
│ main.go │
│ Echo HTTP Server · /api/chat · WebSocket /ws/telemetry │
└──────────────┬───────────────────────────────────────────────┘
│
orchestrator.Handle()
│
┌──────────────────┼─────────────────────┬─────────────────────┐
│ │ │ │
▼ ▼ ▼ ▼
classifier. router. llmclient. feedback.
SplitAndClassify() Dispatch() CallModel() Store.Record()
│ │ │ │
│ ├─────────────────────┤ │
│ │ │ │
▼ ▼ ▼ ▼
[Profiles] failover. telemetry. [Append-Only]
Manager.Resolve() Engine.Record() JSONL Audit Logs
End-to-End Data Flow Matrix
-
Ingress Payload Consumption: A user client posts a structured text payload to the
/api/chatEcho handler. This payload is handed directly toorchestrator.Handle(ctx, prompt). -
Tokenization and Dissection: The string is analyzed by
classifier.SplitAndClassify(). It matches text boundaries against explicit logical conjunction markers (and also,additionally,plus) via regular expressions, segmenting the prompt into discrete task clauses. Each clause is evaluated for complexity based on signal keywords, string length, and structural depth penalties, returning an array of TaskProfile structs. -
Dynamic Utility Selection: The
[]TaskProfilearray passesinto router.Dispatch(). For each profile, the router fetches the latest running parameters from the thread-safetelemetry.Engine, cross-references the task type against theStaticTaskThresholds, and evaluates the model capability matrix. It then computes the highest utility option using the model spec's quality-curve functions: U=β⋅Q(complexity)−α⋅CostPer1k -
Circuit-Breaker Interception: Before confirming the target tier, the candidate model is vetted by
failover.Manager.Resolve(). If the model's circuit breaker state is OPEN due to repeated local timeout or connection failures at the Ollama layer, the request is intercepted and systematically rerouted to the next available provider in its configured FailoverChain. - Structured Concurrent Execution: The orchestrator instantiates a dedicated goroutine wrapper for each sub-task in the dispatch map. These routines run concurrently, multiplexing network calls over native OS threads.
-
Network Delivery & Telemetry Ingestion: Each worker goroutine triggers
llmclient.CallModel(), which executes anHTTP POSTto the local Ollama/v1/chat/completionsOpenAI-compatible endpoint. Upon response completion, the worker records the transaction duration and token weight, passing this metadata intotelemetry.Engine.Record()to recalculate the Exponential Moving Average (EMA) quality scores and latency percentiles (p50 and p95). -
Asynchronous Streaming & Audit Logging: Concurrently, the orchestrator appends the finalized routing signatures to the thread-safe, append-only feedback.Store (writing row lines to
output/feedback.jsonl). It also emits progress notifications over a centralized channel (eventCh), allowing the broadcastLoop to instantly push JSON frame payloads to all connected WebSocket visualization dashboards. -
Aggregation Assembly: Once all concurrent routines join
(sync.WaitGroup.Wait()), the system aggregates individual sections within merge(), compiling the outputs into unified Markdown pages returned to the caller inside an enrichedHTTP JSONstructure.
Component Deep-Dives & Production-Grade Code
Let’s dive into the core packages of the pure Go implementation (routing-v5) to analyze the code patterns that enforce thread safety, real-time metrics tracking, and low-latency execution.
Thread-Safe Live Telemetry Engine (telemetry.go)
In our Go implementation, we avoid relying on external analytical engines or global locks that create contention bottlenecks. Instead, telemetry.go maintains an explicit map of isolated fixed-size ring buffers, protected by a fine-grained sync.Mutex per model-and-task combination.
When a sub-task finishes, its execution signature updates an Exponential Moving Average (EMA) of quality scores and updates running latency tables to calculate precise p50 and p95 metrics. If percentiles slip past our targetLatencyMs, the engine increases the cost-penalty coefficient (α), causing the router to favor lighter, faster models until system pressure subsides.
// internal/telemetry/telemetry.go
package telemetry
import (
"math"
"os"
"sort"
"strconv"
"sync"
)
var (
windowSize = envInt("TELEMETRY_WINDOW", 50)
emaAlpha = envFloat("EMA_ALPHA", 0.15)
baseThreshold = 0.72
minThreshold = 0.40
maxThreshold = 0.95
baseCostWeight = 0.40
minCostWeight = 0.10
maxCostWeight = 0.70
targetLatencyMs = 3000.0
)
type Record struct {
LatencyMs int
QualityEval float64
Success bool
}
type ringBuffer struct {
data []Record
head int
size int
maxSz int
}
func newRingBuffer(max int) *ringBuffer {
return &ringBuffer{data: make([]Record, max), maxSz: max}
}
func (r *ringBuffer) append(rec Record) {
r.data[r.head] = rec
r.head = (r.head + 1) % r.maxSz
if r.size < r.maxSz {
r.size++
}
}
func (r *ringBuffer) slice() []Record {
out := make([]Record, r.size)
for i := 0; i < r.size; i++ {
idx := (r.head - r.size + i + r.maxSz) % r.maxSz
out[i] = r.data[idx]
}
return out
}
type Engine struct {
mu sync.Mutex
buffers map[[2]string]*ringBuffer
}
func NewEngine() *Engine {
return &Engine{buffers: make(map[[2]string]*ringBuffer)}
}
func (e *Engine) Record(modelLabel, taskType string, rec Record) map[string]float64 {
e.mu.Lock()
defer e.mu.Unlock()
key := [2]string{modelLabel, taskType}
if e.buffers[key] == nil {
e.buffers[key] = newRingBuffer(windowSize)
}
e.buffers[key].append(rec)
return e.tune(key)
}
func (e *Engine) tune(key [2]string) map[string]float64 {
buf := e.buffers[key].slice()
n := len(buf)
if n == 0 {
return map[string]float64{"quality_threshold": baseThreshold, "cost_weight": baseCostWeight}
}
qEMA := buf[0].QualityEval
for _, r := range buf[1:] {
qEMA = emaAlpha*r.QualityEval + (1-emaAlpha)*qEMA
}
lats := make([]int, n)
successes := 0
for i, r := range buf {
lats[i] = r.LatencyMs
if r.Success {
successes++
}
}
sort.Ints(lats)
p95Idx := int(float64(n) * 0.95)
if p95Idx >= n {
p95Idx = n - 1
}
p95 := float64(lats[p95Idx])
newThr := baseThreshold + 0.5*(qEMA-baseThreshold)
newThr = math.Max(minThreshold, math.Min(maxThreshold, newThr))
latPressure := p95 / targetLatencyMs
costW := baseCostWeight * math.Exp(latPressure-1.0)
costW = math.Max(minCostWeight, math.Min(maxCostWeight, costW))
return map[string]float64{
"quality_threshold": math.Round(newThr*1000) / 1000,
"cost_weight": math.Round(costW*1000) / 1000,
}
}
func envInt(k string, d int) int {
if v := os.Getenv(k); v != "" {
if n, err := strconv.Atoi(v); err == nil { return n }
}
return d
}
func envFloat(k string, d float64) float64 {
if v := os.Getenv(k); v != "" {
if f, err := strconv.ParseFloat(v, 64); err == nil { return f }
}
return d
}
Circuit Breaker State Machine (failover.go)
Resilience at low latency requires keeping tabs on infrastructure node health without adding network overhead. The Go engine wraps every model endpoint inside an isolated state machine tracking three critical operational states: CLOSED (healthy processing), OPEN (failing back directly to alleviate latency bottlenecks), and HALF_OPEN (controlled probe windows testing for backend recovery).
┌──────────────────────────────────┐
│ │
▼ │
┌──────────┐ Failure Threshold ┌────────┐
──►│ CLOSED ├──────────────────────►│ OPEN │
└────▲─────┘ Exceeded └───┬────┘
│ │
│ │ Cooldown
│ Probes Successful │ Window
│ (probeCount >= halfOpenProbes) │ Elapsed
│ │
┌────┴───────┐ │
│ HALF_OPEN │◄────────────────────────┘
└────┬───────┘
│
│ Probe Failure
└─────────────────────────────────►
// internal/failover/failover.go
package failover
import (
"sync"
"time"
"github.com/routing-v5/internal/registry"
)
type CircuitState string
const (
Closed CircuitState = "CLOSED"
Open CircuitState = "OPEN"
HalfOpen CircuitState = "HALF_OPEN"
)
var (
failureThreshold = 3
cooldownSeconds = 30
halfOpenProbes = 2
healthEMAAlpha = 0.20
)
type circuitBreakerState struct {
mu sync.RWMutex
modelLabel string
state CircuitState
consecutiveFails int
lastStateChange time.Time
probeCount int
healthEMA float64
}
func newCircuitBreaker(label string) *circuitBreakerState {
return &circuitBreakerState{
modelLabel: label,
state: Closed,
healthEMA: 1.0,
}
}
func (cb *circuitBreakerState) IsCallable() bool {
cb.mu.Lock()
defer cb.mu.Unlock()
if cb.state == Open {
if time.Since(cb.lastStateChange) > time.Duration(cooldownSeconds)*time.Second {
cb.state = HalfOpen
cb.probeCount = 0
cb.lastStateChange = time.Now()
return true
}
return false
}
return true
}
func (cb *circuitBreakerState) RecordFailure() {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.consecutiveFails++
cb.healthEMA = (1-healthEMAAlpha)*cb.healthEMA + healthEMAAlpha*0.0
if (cb.state == Closed && cb.consecutiveFails >= failureThreshold) || cb.state == HalfOpen {
cb.state = Open
cb.lastStateChange = time.Now()
}
}
func (cb *circuitBreakerState) RecordSuccess() {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.consecutiveFails = 0
cb.healthEMA = (1-healthEMAAlpha)*cb.healthEMA + healthEMAAlpha*1.0
if cb.state == HalfOpen {
cb.probeCount++
if cb.probeCount >= halfOpenProbes {
cb.state = Closed
cb.lastStateChange = time.Now()
}
}
}
func (cb *circuitBreakerState) HealthScore() float64 {
cb.mu.RLock()
defer cb.mu.RUnlock()
return cb.healthEMA
}
type Manager struct {
mu sync.Mutex
circuits map[string]*circuitBreakerState
events []FailoverEvent
}
func NewManager() *Manager {
m := &Manager{circuits: make(map[string]*circuitBreakerState)}
for _, mod := range registry.Models {
m.circuits[mod.Label] = newCircuitBreaker(mod.Label)
}
return m
}
func (m *Manager) RecordFailure(label string) { m.circuits[label].RecordFailure() }
func (m *Manager) RecordSuccess(label string) { m.circuits[label].RecordSuccess() }
func (m *Manager) Resolve(primary *registry.ModelSpec, taskType string) (*registry.ModelSpec, *FailoverEvent) {
if m.circuits[primary.Label].IsCallable() {
return primary, nil
}
// Navigate fallback mapping
for _, entry := range primary.FailoverChain {
if m.circuits[entry.Label].IsCallable() {
fb := registry.ByLabel(entry.Label)
ev := &FailoverEvent{
PrimaryLabel: primary.Label,
FallbackLabel: entry.Label,
Reason: entry.Reason,
Timestamp: time.Now(),
}
m.mu.Lock()
m.events = append(m.events, *ev)
m.mu.Unlock()
return fb, ev
}
}
// Last-resort backup: pick the node with the highest health EMA
best := registry.Models[0]
for _, mod := range registry.Models[1:] {
if m.circuits[mod.Label].HealthScore() > m.circuits[best.Label].HealthScore() {
best = mod
}
}
ev := &FailoverEvent{
PrimaryLabel: primary.Label,
FallbackLabel: best.Label,
Reason: "all_chains_exhausted",
Timestamp: time.Now(),
}
m.mu.Lock()
m.events = append(m.events, *ev)
m.mu.Unlock()
return best, ev
}
The Orchestration Core & WebSocket Broadcasting Topology
To keep analytics and real-time dashboard events from blocking the primary API request loop, the Go implementation isolates client updates within a non-blocking buffered memory channel.
Instead of forcing the front-end to poll the database, the orchestrator broadcasts atomic execution snapshots over eventCh. The server's event loop fans these messages out to active browser clients, dropping slow readers instantly to protect core processing paths.
[Worker Goroutines] [Central Routing backplane] [Client Channels]
(Sub-Task Workers)
┌───────────────────┐
│ Sub-task Start ├───────┐
└───────────────────┘ │
▼
┌───────────────────┐ ┌─────────┐ ┌──────────────────┐
│ Sub-task Complete ├───►│ eventCh │───────────────►│ Client Message │
└───────────────────┘ │ (Buffer │ │ Ring-buffer loop │
│ Slots) │ └────────┬─────────┘
┌───────────────────┐ ▲ │
│ Routing Plans ├───────┘ ▼
└───────────────────┘ ┌───────────────┐
│ Browser Web │
│ UI Dashboards │
└───────────────┘
// internal/orchestrator/orchestrator.go
package orchestrator
import (
"context"
"sync"
"time"
"github.com/routing-v5/internal/classifier"
"github.com/routing-v5/internal/failover"
"github.com/routing-v5/internal/feedback"
"github.com/routing-v5/internal/llmclient"
"github.com/routing-v5/internal/router"
"github.com/routing-v5/internal/telemetry"
)
type Orchestrator struct {
eng *telemetry.Engine
fm *failover.Manager
store *feedback.Store
eventCh chan map[string]interface{}
}
func NewOrchestrator(e *telemetry.Engine, f *failover.Manager, s *feedback.Store, ch chan map[string]interface{}) *Orchestrator {
return &Orchestrator{eng: e, fm: f, store: s, eventCh: ch}
}
func (o *Orchestrator) Handle(ctx context.Context, prompt string) (*OrchestratorResult, error) {
tIndex := time.Now()
profiles := classifier.SplitAndClassify(prompt)
routing := router.Dispatch(o.eng, o.fm, profiles)
o.emit(map[string]interface{}{
"type": "routing_plan",
"instructions": len(routing),
})
var wg sync.WaitGroup
subResults := make([]SubTaskResult, len(routing))
resMu := sync.Mutex{}
for key, dispatchRes := range routing {
wg.Add(1)
go func(k string, dr router.DispatchResult) {
defer wg.Done()
o.emit(map[string]interface{}{"type": "subtask_start", "task_type": k})
t0 := time.Now()
cr := llmclient.CallModel(ctx, o.fm, dr.Selected.Label, dr.Profile.RawText, dr.Selected.MaxTokens)
lat := int(time.Since(t0).Milliseconds())
success := cr.Tokens > 0
o.eng.Record(dr.Selected.Label, string(dr.Profile.TaskType), telemetry.Record{
LatencyMs: lat,
QualityEval: dr.Selected.QualityAt(dr.Profile.Complexity),
Success: success,
})
cost := dr.Selected.CostPer1k * float64(cr.Tokens) / 1000.0
o.store.Record(feedback.RoutingRecord{
Timestamp: time.Now().UTC().Format(time.RFC3339),
TaskType: k,
Complexity: dr.Profile.Complexity,
Tier: dr.Selected.Tier,
ModelUsed: dr.Selected.Label,
LatencyMs: lat,
CostUnits: cost,
QualityEval: dr.Selected.QualityAt(dr.Profile.Complexity),
FailoverUsed: dr.FailoverEvent != nil,
})
resMu.Lock()
subResults = append(subResults, SubTaskResult{
Profile: dr.Profile,
Model: dr.Selected,
Response: cr.Text,
LatencyMs: lat,
OutTokens: cr.Tokens,
UtilityScore: dr.Utility,
OllamaModel: dr.Selected.Label,
CostUnits: cost,
})
resMu.Unlock()
o.emit(map[string]interface{}{
"type": "subtask_complete",
"task_type": k,
"model": dr.Selected.Label,
"latency_ms": lat,
})
}(key, dispatchRes)
}
wg.Wait()
return &OrchestratorResult{
Prompt: prompt,
SubResults: subResults,
TotalMs: int(time.Since(tIndex).Milliseconds()),
}, nil
}
func (o *Orchestrator) emit(p map[string]interface{}) {
if o.eventCh != nil {
select {
case o.eventCh <- p:
default: // Non-blocking: skip slow dashboard consumers
}
}
}
To bridge this asynchronous pipeline with client browsers, main.go sets up a dedicated broadcasting loop. By isolating connection maps behind a mutex and pairing them with unblocked send frames, we ensure the core engine stays safe and responsive under load.
// snippet from main.go
var (
wsClients = make(map[*websocket.Conn]chan string)
wsClientMu sync.Mutex
eventCh = make(chan map[string]interface{}, 500)
)
func broadcastLoop() {
for payload := range eventCh {
data, err := json.Marshal(payload)
if err != nil {
continue
}
msg := string(data)
wsClientMu.Lock()
dead := []*websocket.Conn{}
for conn, ch := range wsClients {
select {
case ch <- msg:
default:
dead = append(dead, conn)
}
}
for _, conn := range dead {
close(wsClients[conn])
delete(wsClients, conn)
}
wsClientMu.Unlock()
}
}
Conclusion of these Series & Key Architectural Takeaways
By porting our multi-LLM routing pipeline from Python to compiled Go, we demonstrate that optimizing AI infrastructure goes beyond fine-tuning models — the efficiency of the orchestration layer itself plays a massive role.
- Abstractions Have a Cost: Dropping external gateway frameworks in favor of native HTTP communication cuts orchestration overhead down to the microsecond level.
- True System Concurrency: While not the main purpose, replacing Python’s single-threaded event loops with Go’s lightweight goroutines ensures that intensive text parsing and utility logic don’t block concurrent I/O streams.
- Operational Independence: Compiling down to a single binary removes package dependency maintenance and simplifies deployment down to a simple executable drop.
This concludes our 5-part journey into building a production-grade multi-LLM router! By prioritizing live metrics tracking, smart task division, circuit breaker failovers, and low-latency code design, we can build workflows that run at peak efficiency — keeping costs minimal and performance high. The aim is to gain a practical, foundational understanding of how an intelligent SDLC engine manages multi-LLM orchestration and parallel routing. While this demonstration abstracts away proprietary or internal mechanics of a production agent, it illustrates the exact architectural paradigm required to industrialize software development: splitting compound engineering tasks, evaluating complexity curves, and dynamically balancing cost against quality thresholds.
- Dynamic Task Deconstruction: Compound engineering prompts (e.g., “Write a Go helper function and document its edge cases”) are split into distinct execution branches rather than being sent as a single block to an expensive model.
- The Utility Trade-Off: Decisions are mathematically routed using a utility function balancing expected quality (Q) against unit cost ( C ): U=β⋅Q−α⋅C.
- T-Shirt Sized Tiering: Tasks are mapped to a model pool matching the required capabilities:
Tier 1 (Prose/Formatting): Lightweight, highly efficient models (e.g., IBM Granite).
Tier 2/3 (Structured Logic): Medium-tier generalists (e.g., LLaMA / Gemma) for standard logic.
Tier 4 (Complex Synthesis/Security): Heavyweight models (e.g., Mistral Small/Large) reserved for architecture reviews or security-critical implementations.
This structural framework effectively conveys that while the implementation details are a global approximation, the pattern itself is the standard for building predictable, cost-efficient, and secure AI-driven engineering tools.
🎯 That’s a wrap 💯 for these series and thanks for reading!
Links
- Code repository for this post: https://github.com/aairom/multillm-taskrouting/tree/main/v5-go-implementation
- Code repository for the whole series: https://github.com/aairom/multillm-taskrouting
- IBM Bob: https://bob.ibm.com/
Top comments (0)