LLMRix Model Router — an open-source multi-model routing and orchestration framework for Java.

Stop Hardcoding Model Names in Business Code: LLMRix Router Adds a Multi-Model Routing Layer for Java AI Apps

When AI applications move from demo to production, the trouble usually isn't with prompts—it's with the model calls themselves. Which model should handle this request? What happens when the primary model is rate-limited? How do you control costs? Can you switch providers mid-stream when a streaming response drops? And when running multiple instances, how do you keep quota and health state consistent?

LLMRix Model Router packages all these concerns into an open-source routing runtime for Java.

Over the past couple of years, the barrier to integrating AI models has dropped significantly.

With a single SDK and a few lines of code, an app can call OpenAI, DeepSeek, or any other compatible interface. But once you enter production, a different set of problems emerges: upstream rate limits, timeouts, regional outages, and models that differ widely in capabilities, pricing, context windows, and response speed. When business code is directly coupled to a specific provider, swapping models means rewriting interfaces, exception handling, monitoring, and configuration all at once.

Simple round-robin or reverse proxies can't make these decisions either. They don't know whether a request needs tool calling, which models to exclude for image inputs, or whether a candidate has already exceeded its cost budget or is in a cooldown period.

LLMRix Model Router sits at this boundary. It's a multi-model routing and orchestration framework for Java that extracts provider differences, model selection, failover, quotas, costs, and observability out of business code.

Positioned between AI applications and model services, it handles runtime decision-making and request forwarding. A chat SDK answers how to call; a Router also decides who to call, when to retry, and how to reconcile state.

1. Architecture Overview: Separating Decisions, Execution, and Infrastructure

The project's architectural tradeoff is clear: routing strategies can change, but the correctness of request execution must not. To achieve this, LLMRix divides the system into layers with well-defined boundaries:

1. Access Layer: Provides an embedded Java API, the Orion remote client, and an OpenAI-compatible HTTP/SSE interface under Spring Boot.
2. Unified Contract Layer: Defines model interfaces (Chat, Embedding, Rerank, Audio, Image, Video), request/response objects, and exception types in llmrix-model-open.
3. Routing Core Layer: llmrix-model-router-core handles model targets, capability matching, strategy selection, execution budgets, timeouts, retries, health, and lifecycle events.
4. Integration Layer: llmrix-model-router-integrations provides adapters for OpenAI, DeepSeek, OpenRouter, Ollama, Redis, Bucket4j, ONNX, Shadow, evaluation, and Fugu.
5. State & Observability Layer: State can live in local memory/Caffeine or in Redis; events are exposed through Listeners, and Spring integration connects them to Micrometer, Observation, and Actuator.
6. Infrastructure Layer: TLS, WAF, external load balancing, secret management, Redis HA, and container orchestration are left to the deployment environment.

Vector source: llmrix-router-architecture.svg. Original panoramic diagram: GitHub architecture SVG.

The Router Core in the middle sits at the junction of decision and execution. It doesn't bind to any single vendor, nor does it rewrite business requests into provider-private objects—instead, it manages candidates through a unified ModelClient and ModelTarget. When adding a new provider, changes stay concentrated in the Provider SPI and Transport adapters, without spreading to routing strategies or business code.

1.1 Production Modules and Responsibilities

Maven Artifact	Responsibility	Typical Usage
llmrix-model-open	Shared model contracts, common exceptions, auth SPI, OpenAI-compatible transport & adapters	Reuse unified types when building clients or custom integrations
llmrix-model-router-core	Router Builder, model targets, strategies, executor, state SPI, quota, health & events	Embed Router in plain Java applications
llmrix-model-router-integrations	Built-in providers, Redis, Bucket4j, ONNX, Shadow, evaluation & Fugu	Use official integrations or advanced routing capabilities
llmrix-model-router-spring-starter	Auto-configuration, YAML properties, HTTP/SSE, auth, Actuator & Micrometer	Build Spring Boot routing services
llmrix-model-orion	Framework-neutral remote Java client	Call a standalone Router from Java services
llmrix-model-orion-spring-starter	Orion auto-configuration & Micrometer adapter	Inject remote model clients in Spring Boot business services

These module boundaries let teams pull in only what they need. Remote business services can depend solely on Orion and the shared model contracts, without dragging in Redis, ONNX, or the entire Router Runtime.

1.2 Request Decision Sequence

The sequence diagram below follows the source code's execution order, covering capability matching, quota control, and failover. It highlights three distinct paths: successful return, retry before the first visible output, and no-replay after streaming has begun.

Vector source: llmrix-router-request-decision-sequence.svg.

"Whether a model is suitable for this request" and "whether this call will succeed" are handled by two separate components: candidate snapshots and strategies handle the former, while the executor, state store, and exception classification handle the latter.

2. Feature Matrix: Model Selection and Runtime Governance

Capability Domain	Specific Features	Problems Solved
Model abstraction	Provider-neutral ModelClient, multimodal request/response, unified exceptions	Business code no longer binds to vendor SDKs
Capability matching	operations, features, input-modalities, traits	Prevents sending tool, image, or audio requests to unsupported models
Routing strategies	priority, round-robin, weighted random, least-busy, latency-aware, cost-aware, balanced, cache-aware	Trade off stability, cost, latency, and cache hits according to business goals
Dynamic decisions	semantic scoring, contextual bandit, customizable RoutingStrategy	Continuously improve model selection using request semantics or feedback data
Reliability	per-attempt timeout, total budget, retry predicates, failure thresholds, target cooldown, candidate pool continuation	Handles rate limits, timeouts, transient failures, and partial outages
Streaming safety	first-token timeout, stream idle timeout, pre-first-chunk switching, cancellation propagation, tool request non-replay	Prevents duplicate output and replay of side-effecting tool calls
Cost governance	input/output/cache/inference token pricing, per-request maxCostUsd, route-level RPM/TPM	Makes cost constraints part of the decision, not a post-hoc statistic
Quota & concurrency	target-level limits, route-level limits, auth quota partitions, local Caffeine, Redis atomic leases	Controls resource usage per model, per route, and per tenant
Multimodal	Chat, Responses core subset, Embeddings, Rerank, Audio, Image, Video	Unified handling of text, image, audio, file, and video workflows
Evaluation & orchestration	Online Shadow, offline Evaluation, Fugu Worker/Thinker/Verifier, ONNX policies	Compare models and organize multi-round collaboration without affecting main traffic
Observability	Router/Fugu Listener, Micrometer, Observation, Actuator, request ID	Answers "who was selected, how long it took, why it retried, and how much it cost"
Extensibility	ModelProvider, ProviderAuthenticator, ModelPricingResolver, RouterStateStore	Integrate enterprise proxies, signed auth, internal pricing catalogs, and custom state systems

2.1 Why Model Capabilities Are Declared in Four Categories

LLMRix doesn't reduce "model capability" to a single boolean. Instead, configuration declares four separate dimensions:

• operations: what the model can do, e.g. chat, embeddings, rerank, video-generation;
• features: what protocol features it supports, e.g. streaming, tools, structured-output, prompt-cache;
• input-modalities: what inputs it accepts, e.g. vision, video, audio, file;
• traits: what task types it excels at, e.g. code, reasoning, long-context.

The four declarations are independent. They're validated at startup and used to filter candidates per request. Compared to maintaining a single "universal model list," this approach is easier to audit and reduces capability mismatches in production.

3. From "Calling a Model" to "Calling a Route"

LLMRix lets applications depend on stable route names rather than upstream model names.

For example, business code simply requests general, code, or reasoning. Whether the request ultimately goes to OpenAI, DeepSeek, OpenRouter, or a local Ollama instance is decided by the router based on capability, health, latency, cost, and strategy.

Model adjustments therefore live in routing configuration, not in business code. When adding providers, replacing models, or changing selection strategies, upstream applications typically don't need to be rewritten.

LLMRix offers three integration modes:

• As an embedded Java SDK, dropped directly into an existing application process;
• As a Spring Boot Starter, auto-assembled through configuration;
• As a standalone OpenAI-compatible service, providing a unified HTTP interface for other languages and existing tools.

When a Java service needs to call a remote Router, it can use the project's lightweight client, Orion. Provider keys stay on the Router side only—clients only see route names and the unified protocol.

4. How a Request Is Processed

LLMRix doesn't just work off a flat list of models—it processes requests through a well-bounded execution pipeline.

Upon receiving a request, the Router first checks the operation type, tool calling, structured output, input modality, context length, and routing constraints passed by the caller. Models that can't satisfy the conditions are eliminated before execution begins. A chat with images won't be sent to a text-only model; when a request requires reasoning or code traits, candidates that don't declare those traits won't enter the selection phase.

The remaining candidates are then passed to a routing strategy for ordering. Built-in strategies include priority, round-robin, weighted random, least-busy, latency-aware, cost-aware, balanced scoring, and prompt-cache affinity. Semantic routing and contextual multi-armed bandit implementations are also provided. Teams can implement RoutingStrategy to incorporate data residency, tenant tier, compliance tags, or internal model scores into selection rules.

Before invoking a model, the Router acquires quota and concurrency leases and checks the per-request cost budget. On success, it settles costs based on actual token usage, releases leases, and publishes lifecycle events. On failure, the Router determines—based on exception type—whether to retry, put the target into cooldown, and move on to the next candidate in the pool.

Thus, "who to pick" and "how to call" are two separate responsibilities: strategies handle ordering, while the executor handles timeouts, retries, quotas, health, and resource cleanup. Selection algorithms can be swapped; execution rules remain consistent.

5. Streaming: No Replay After the First Chunk

Failover for ordinary requests is relatively straightforward. Streaming responses are different.

Suppose Model A has already output half a sentence to the user when the connection drops. If the system switches to Model B and replays from the beginning, the user sees duplicate content, and tool calls may execute twice.

LLMRix's rule is: candidates can be switched before the first data chunk becomes visible to the caller; after output begins, the request is never replayed. The Router manages first-token timeout and stream idle timeout separately, and propagates cancellation signals to any still-running upstream requests.

This rule directly affects whether callers see consistent output. It encodes the streaming response boundary into the executor, rather than leaving it to business code to handle on its own.

6. From Chat to Multimodal Model Interfaces

As of version 1.0.2, LLMRix supports Chat, a core subset of the Responses API, Embeddings, Rerank, audio, image, and video operations. The Spring Boot Starter provides corresponding OpenAI-style endpoints. Both Chat and Responses endpoints support JSON and SSE streaming.

Built-in integrations include OpenAI, DeepSeek, OpenRouter, and Ollama. Not every provider supports the same set of operations, and the Router reads each target's operations, features, input-modalities, and traits—it never treats "OpenAI-compatible interfaces" as having identical capabilities.

Explicit capability declaration has two practical benefits: configuration errors surface at startup, and routing strategies can filter before requests are sent upstream.

6.1 Built-in Provider Capabilities

Integration	Key Operations Implemented	Suitable For
OpenAI	Chat, Embeddings, Audio, Images, Videos	Using OpenAI's native multimodal interface
DeepSeek	Chat	General conversation, code, and reasoning routes
OpenRouter	Chat, Embeddings, Rerank	Accessing multiple models and free models through a single upstream
Ollama	Chat, Embeddings	Local development, offline environments, and private models
Custom Provider	Determined by the ModelProvider implementation	Enterprise model platforms, internal proxies, proprietary protocols

"Integration support" doesn't mean all models under that provider support the same operations. Final capabilities depend on the specific model, provider account, and configuration declarations. For example, a particular OpenRouter model might support text chat but not accept image inputs. The Router only filters by target declarations—it doesn't invent capabilities a model doesn't have.

6.2 OpenAI-Compatible Endpoints

Endpoint	Purpose
POST /v1/chat/completions	Synchronous or SSE streaming chat
POST /v1/responses	Core subset of Responses API, with JSON and SSE
POST /v1/embeddings	Text or token-array embeddings
POST /v1/rerank	Query/document reranking
POST /v1/audio/transcriptions	Audio transcription
POST /v1/audio/translations	Audio translation
POST /v1/audio/speech	Text-to-speech
POST /v1/images/generations	Image generation
POST /v1/images/edits	Multipart image editing
POST /v1/videos	Create a video generation task
GET /v1/videos/{video_id}	Check video task status
GET /v1/videos/{video_id}/content	Download completed video
DELETE /v1/videos/{video_id}	Delete a video task
POST /v1/videos/{video_id}/remix	Create a remix task from an existing video
GET /v1/models	List available route identifiers

7. Local State for Single Instances, Shared State for Multiple

For development and small deployments, you can use in-memory state with no extra infrastructure. Local quota partitions are managed by Caffeine with capacity and idle expiry, preventing unbounded growth from dynamic tenant keys.

When the Router scales to multiple instances, you can switch to Redis. Health state, concurrency leases, RPM, TPM, and multi-armed bandit state can all be shared across instances, with updates performed via Redis atomic operations. Redis mode uses a fail-closed strategy: when configuration is wrong or the store is unavailable, the system won't silently fall back to JVM-local state, avoiding the "it looks rate-limited but each machine is counting separately" problem.

Local mode is suitable for trials and small services; Redis mode handles shared constraints for horizontal scaling.

7.1 Recommended Production Deployment Topology

Vector source: llmrix-router-production-deployment.svg.

This production deployment diagram extends the earlier module boundaries with external components from the runtime environment:

1. Caller Layer: Java services use Orion; other languages and agents use OpenAI-compatible HTTP/SSE. Callers only pass route names and business constraints—they don't hold provider keys.
2. Enterprise Edge Layer: An API Gateway handles TLS, WAF, external identity authentication, public rate limiting, request body limits, and instance load balancing. This layer is not part of LLMRix.
3. Router Cluster Layer: Each replica contains Spring Starter/HTTP access, Router Core decision-execution, and Integrations provider adapters—corresponding to the access, core, and integration layers in the architecture overview. Replicas use identical routing configuration but run in independent processes.
4. Shared State Layer: Redis HA stores cross-replica RPM/TPM, concurrency leases, health cooldown, and bandit state. Redis mode is fail-closed—it won't silently fall back to local rate limiting when shared state is unavailable.
5. Model Access Layer: The Router accesses OpenAI, DeepSeek, and OpenRouter through controlled egress, and Ollama or custom providers over private networks. Upstream timeouts, quotas, and account limits still apply.
6. Platform Services Layer: A Secret Manager injects API keys and provider credentials into the Router; the Router's metrics, traces, and logs flow to the enterprise observability platform. LLMRix emits events and metrics—it doesn't deploy these platforms.

The main path is "caller → enterprise gateway → Router replica → Provider Adapter → model service," with responses returning along the same connection. Green links represent Router↔Redis state reads and writes, orange links represent secret injection, and purple dashed links represent asynchronous telemetry. None of these three link types enter the model response body.

Smaller systems can skip the gateway and Redis, embedding the Router directly in business services or running a single instance. When the Router scales horizontally, shared Redis state should be used, and edge security and public traffic governance should be handed to a gateway. The Router's API key authentication handles the client-to-router boundary; provider keys stay only in the Router deployment environment or a secret management system.

8. Observability Starts with the Request

When model routing lacks explainability, production troubleshooting is slow. At minimum, you need to know which model was selected, why a retry happened, which candidate went into cooldown, how long the first token took, and where costs were spent.

LLMRix defines lifecycle events at the core layer: request started, route selected, attempt started/ended, first token, usage recorded, target cooldown, and request completed. With Spring integration connecting to Micrometer, Observation, and Actuator, you get metrics for request volume, latency, attempt count, first-token latency, candidate availability, in-flight requests, and token usage.

The project doesn't deploy Prometheus, Grafana, or an OpenTelemetry Collector, but it provides a stable observability boundary. Java teams with existing monitoring infrastructure can plug in directly without maintaining a separate console.

9. Making Model Selection Feedback-Driven

Beyond standard routing, LLMRix provides several capabilities for evaluation and feedback.

Semantic routing scores candidates based on request content. Contextual bandits combine selection counts with reward feedback to balance between using the current best model and exploring alternatives.

Online Shadow sends side-effect-free requests to shadow models at a sample rate without affecting the main request result. The offline evaluation component aggregates quality, latency, failure, and cost across multiple models on a sample set.

Fugu orchestration supports roles like Worker, Thinker, and Verifier iterating within bounded turns, token budgets, and cost budgets. Generation, reflection, and verification all have stop conditions, retry, and fallback mechanisms. ONNX policies can be loaded at runtime, but training and policy rollout remain the responsibility of offline systems.

These capabilities aren't the starting point for every team. They show that LLMRix goes beyond static load balancing—it reserves interfaces for evaluable, learnable model decisions.

10. Quick Start

The project requires Java 17 or later (Java 21 recommended). Core artifacts are published to Maven Central under the MIT License.

Without Spring, just include Core and Integrations:

<dependency>
  <groupId>com.llmrix.model</groupId>
  <artifactId>llmrix-model-router-core</artifactId>
  <version>1.0.2</version>
</dependency>
<dependency>
  <groupId>com.llmrix.model</groupId>
  <artifactId>llmrix-model-router-integrations</artifactId>
  <version>1.0.2</version>
</dependency>

Then register models in a route pool:

try (LlmRouter router = LlmRouter.builder()
        .integration("openai", provider -> provider
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .model("gpt-4.1-mini", model -> model
                .operations(ModelOperation.CHAT)
                .features(ModelFeature.TOOLS)))
        .integration("deepseek", provider -> provider
            .apiKey(System.getenv("DEEPSEEK_API_KEY"))
            .model("deepseek-chat", model -> model
                .operations(ModelOperation.CHAT)
                .features(ModelFeature.TOOLS)
                .traits(ModelTrait.CODE)))
        .route("general", route -> route
            .strategy("balanced")
            .quota(600L, 100_000L)
            .models("openai/gpt-4.1-mini", "deepseek/deepseek-chat"))
        .build()) {
    ChatResponse response = router.chat("Review this Java code");
}

Teams using Spring Boot can use the Starter and YAML configuration instead. Non-Java callers can enable the OpenAI-compatible HTTP API and point their existing SDK's base_url at the Router. The model field in requests takes a route name, not a vendor's real model ID.

11. Configuration and Usage Examples

11.1 Example 1: Multiple Business Routes in Spring Boot

In a Spring Boot service, routes can be entirely managed by YAML. The example below splits general chat, code analysis, vector search, and reranking into four routes, with shared RPM/TPM quotas for general chat:

llmrix:
  model:
    router:
      enabled: true
      default-route: general
      routes:
        general:
          strategy: balanced
          quota:
            requests-per-minute: 600
            tokens-per-minute: 100000
          models:
            - integration: openai
              model: gpt-4.1-mini
            - integration: deepseek
              model: deepseek-chat
        code:
          strategy: cost-aware
          models:
            - integration: deepseek
              model: deepseek-chat
            - integration: openrouter
              model: cohere/north-mini-code:free
        embeddings:
          strategy: latency-aware
          models:
            - integration: openai
              model: text-embedding-3-small
            - integration: openrouter
              model: nvidia/nemotron-3-embed-1b:free
        rerank:
          strategy: priority
          models:
            - integration: openrouter
              model: nvidia/llama-nemotron-rerank-vl-1b-v2:free
      execution:
        timeout: 30s
        max-retries: 1
      http:
        enabled: true
        auth:
          mode: api-key
          bootstrap-key: ${LLMRIX_MODEL_ROUTER_API_KEY}
      integrations:
        openai:
          provider: openai
          api-key: ${OPENAI_API_KEY}
          models:
            - name: gpt-4.1-mini
              operations: [chat]
              features: [streaming, tools]
            - name: text-embedding-3-small
              operations: [embeddings]
        deepseek:
          provider: deepseek
          api-key: ${DEEPSEEK_API_KEY}
          models:
            - name: deepseek-chat
              operations: [chat]
              features: [streaming, tools]
              traits: [code]
        openrouter:
          provider: openrouter
          api-key: ${OPENROUTER_API_KEY}
          models:
            - name: cohere/north-mini-code:free
              operations: [chat]
              traits: [code]
            - name: nvidia/nemotron-3-embed-1b:free
              operations: [embeddings]
            - name: nvidia/llama-nemotron-rerank-vl-1b-v2:free
              operations: [rerank]

The models list in a route is the full candidate pool. The system selects a target within the pool and continues trying on failure—no need to maintain a separate, error-prone fallbacks list. A provider's base-url can point to official endpoints, an enterprise proxy, or a private gateway.

11.2 Example 2: Expressing Business Constraints with Routing Hints

Route names express stable business intent; RoutingHints express per-request temporary constraints. For example, this code review request requires code-capable models while capping cost at 5 cents:

ChatRequest request = ChatRequest.builder()
    .userMessage("Find the race condition in this Java code")
    .routingHints(RoutingHints.builder()
        .require(ModelTrait.CODE)
        .maxCostUsd(0.05)
        .maxLatency(Duration.ofSeconds(8))
        .attribute("tenant", "acme")
        .build())
    .build();

ChatResponse response = router.chatRoute("code").chat(request);

Hints can require specific operations, tools, or input modalities; they can allow or deny specific targets; they can set max cost, max latency, and pass tenant or business tags. Authenticated requests can also use RoutingHints.AUTH_QUOTA_KEY to assign independent quota partitions to different callers.

11.3 Example 3: Integrating Existing Apps via the OpenAI-Compatible HTTP API

Once the Spring Starter's HTTP endpoints are enabled, Python, Node.js, Go, or any existing OpenAI SDK doesn't need to know about the Router's internals. Just point base_url at the Router and set the model field to a route name:

export BASE_URL=http://127.0.0.1:8080
export API_KEY=your-llmrix-http-key

curl --no-buffer "$BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "code",
    "messages": [{"role": "user", "content": "Review this pull request"}],
    "stream": true,
    "temperature": 0
  }'

Beyond Chat Completions, the HTTP layer provides a core subset of Responses, Embeddings, Rerank, audio transcription & translation, text-to-speech, image generation & editing, video tasks, and the Models catalog. Controller-level errors use an OpenAI-style structure with error.message, error.type, error.code, and error.param. Auth requests rejected before reaching the controller may return a more compact error body. Callers should branch on HTTP status and error type, not match against full error text.

11.4 Example 4: Calling a Remote Router with the Orion Client

When model keys need to be kept centrally, or business services need to scale independently from the Router, use Orion:

OrionModelClient client = OrionModelClient.builder()
    .baseUrl("https://router.example.com/v1")
    .apiKey(System.getenv("LLMRIX_API_KEY"))
    .defaultModel("general")
    .defaultEmbeddingModel("embeddings")
    .defaultRerankModel("rerank")
    .build();

ChatResponse answer = client.chat(ChatRequest.user("Summarize this incident"));
EmbeddingResponse vector = client.embed(EmbeddingRequest.text("Text to index"));
RerankResponse ranked = client.rerank(new RerankRequest(
    "refund policy", List.of("Support contact", "Refunds within 30 days")));

Orion supports per-request request IDs and controlled custom headers, as well as async CompletionStage, Flow.Publisher<ChatChunk> streaming, and typed models for audio, image, and video. Remote errors and upstream HTTP status are mapped to client exceptions for unified handling on the business side.

11.5 Example 5: Multimodal Requests and Capability Filtering

Multimodal requests don't require callers to manually pick a provider. The Router excludes unsupported candidates based on each target's declared input modalities:

ChatRequest multimodal = ChatRequest.builder()
    .message(Message.user(
        new TextPart("Compare this diagram with the attached report."),
        new ImagePart("https://example.com/diagram.png", "high"),
        new FilePart("https://example.com/report.pdf", "report.pdf")))
    .build();

ChatResponse analysis = client.chatModel("multimodal").chat(multimodal);

The corresponding model configuration needs to declare input-modalities: [vision, file]; audio and video inputs work the same way. This turns "model support status" into verifiable configuration, rather than relying on implicit conventions—and reduces capability errors that only surface after a request reaches upstream.

11.6 Example 6: Plugging Route Explanations, Metrics, and Cost into Ops Systems

When troubleshooting a request, you can first inspect route candidates and exclusion reasons, then get runtime metrics through Listeners or Micrometer:

RouteExplanation explanation = router.chatRoute("code")
    .explain(ChatRequest.user("Find the race condition"));

System.out.println("selected=" + explanation.selectedTarget());
System.out.println("eligible=" + explanation.eligibleTargets());
System.out.println("excluded=" + explanation.excludedTargets());

Common metrics in Spring Boot include llm.router.requests, llm.router.attempts, llm.router.request.duration, llm.router.first.token, llm.router.cooldowns, candidate availability, and in-flight request count. Usage events separately record input, output, cache-read, cache-write, and reasoning tokens, which can be used to aggregate costs by route, provider, and tenant.

11.7 Example 7: Progressive Quality Upgrades with Shadow and Fugu

You don't have to switch all traffic at once when replacing a model. Online Shadow asynchronously calls shadow models at a sample rate, while the main request is still served by the current production route. The evaluation component records response quality, latency, failures, and cost, and teams can use the online comparison data to decide whether to adjust strategy.

When multi-round generation, thinking, and verification are needed, you can compose multiple ChatModels into a Fugu orchestration:

• Worker generates the initial answer;
• Thinker provides improvement suggestions;
• Verifier checks whether acceptance criteria are met;
• maxTurns, token budget, cost budget, and timeout together bound the worst case;
• On candidate failure, retries, cooldown, and fallback to backup models follow the strategy.

These constraints draw resource boundaries for multi-round AI workflows and avoid writing entire orchestration logic as a hard-to-observe block of business code.

12. Typical Deployment Scenarios

Scenario	Recommended Route Design	Key Capabilities
Intelligent customer service	Separate routes for general, reasoning, translation	Stability-first, tenant quotas, failover, usage tracking
Code assistant	Two-tiered code-fast and code-deep	Code traits, cost budgeting, tool calling, long context
Enterprise RAG	Series of embeddings, rerank, answer	Vectorization, reranking, structured output, traceability
Multimodal moderation	vision, audio, video split by operation	Input modality filtering, file safety, binary responses
Model migration assessment	Main route + Online Shadow	Sampling, isolated timeouts, quality scoring, A/B reports
Hybrid cloud & on-prem models	Public models + Ollama/private Provider in candidate pool	Data boundaries, custom auth, health & latency strategies
Multi-agent workflows	Fugu or custom Router organizing roles	Multi-round budgets, stop conditions, verification, fallback & event streams

13. Recommended Rollout Order

1. Start with one non-critical Chat route, integrate two upstream models, and unify request/response/exception types.
2. Then fill in each target's operations, features, modalities, traits, context window, pricing, and concurrency limits.
3. Add request IDs, Micrometer/Observation, and dashboards for first-token latency, retries, cooldown, usage, and cost.
4. After collecting comparison data with Shadow, gradually enable cost-aware, semantic, or contextual bandit strategies. Stick with simple strategies until you have data.

13.1 Production Checklist

• Provider keys come from environment variables or a dedicated Secret Manager—never in Git, YAML templates, or logs;
• Public-facing services run behind TLS, WAF, gateway authentication, and external rate limiting;
• Multi-instance deployments use Redis for shared quota, concurrency leases, and health state, with Redis HA configured;
• Each model's actual capabilities, context window, pricing, RPM, TPM, and max concurrency are explicitly declared;
• Routes are named by business intent—provider names or specific versions are never exposed to business callers;
• Total request timeout, first-token timeout, and stream idle timeout are all set based on real-world link latency;
• Retry counts are bounded, distinguishing retryable errors from non-retryable ones like parameter, permission, or content-safety issues;
• Prompt observability is off by default; when needed, limit it by route, length, and configure a redactor;
• Shadow only applies to side-effect-free requests; tool-calling requests are excluded by default;
• Alerts are set up for 429s, 503s, first-token latency, cooldown count, candidate availability, and cost anomalies;
• Before upgrading, read CHANGELOG.md and run HTTP/SSE and multimodal regression tests in staging.

14. Fit, Boundaries, and Current State

LLMRix is a better fit than maintaining multiple vendor SDKs directly when:

• Java or Spring Boot is the primary tech stack;
• Two or more model providers are already integrated;
• Model selection needs to account for capability, cost, latency, or tenant policy;
• There are production requirements around rate limiting, failover, streaming correctness, and observability;
• You want the freedom to choose between an in-process SDK and a standalone routing service.

LLMRix also has clear boundaries. It is not a model hosting platform, a training platform, a secret management system, or a full API gateway. TLS, WAF, external load balancing, Redis HA, secret custody, and container orchestration remain the responsibility of the corresponding infrastructure. When full gateway capabilities are needed, deploy it behind Nginx, APISIX, Higress, or a cloud load balancer.

The project is still young. The repository was created in July 2026, the community is small, and there's no admin console, official container image, or Kubernetes Helm Chart yet. The number of built-in providers is also limited. Complex enterprise environments typically need to integrate internal proxies, authentication, and pricing systems through the SPI.

That said, the repository contains more than just concept docs. The current codebase has 6 releasable artifacts and 43 test classes. On September 3, 2026, running a full mvn test on the main branch with Java 21 yielded 199 tests run, 0 failures, 0 errors, 0 skipped. Tests cover core routing, quota & health state, protocol mapping, SSE, multimodal controllers, Spring auto-configuration, server startup, and the Orion client. These results indicate basic engineering completeness but don't substitute for production case studies or performance benchmarks.

15. Closing Thoughts

When an AI application is just getting started, calling a single model directly is usually the fastest approach. As model count grows, traffic increases, or the business becomes more critical, model selection needs to become an independent infrastructure capability.

LLMRix Router provides a stable model boundary. Upstream applications express what they need; the Router decides which models can handle it, which one to pick, how to continue on failure, and records the cost.

If your codebase already has scattered vendor SDKs, retry loops, and model names, start by validating with one non-critical route. After unifying the interface, add health, cost, and quota policies; once Shadow and evaluation have accumulated data, refine the selection approach.

Project: github.com/llmrix-inc/llmrix-router

Further reading: Server Deployment · Orion Client · HTTP API · Maven Central

License: MIT

Current version: 1.0.2

Requirements: Java 17+ (Java 21 recommended); Spring Boot 3.x optional

---