🚀 Building a Production-Ready Multi-LLM Router with LangChain & FastAPI
How I designed CVForbes to automatically switch between Groq, OpenRouter, Ollama, and Gemini without changing a single service.
Client
│
▼
FastAPI Endpoint
│
▼
Dependency Injection
│
▼
LLMRouter Service
┌────────────┬──────────────┐
│ │ │
▼ ▼ ▼
GroqProvider OpenRouter OllamaProvider
│ │ │
└────────────┴──────────────┘
│
GeminiProvider
│
▼
LangChain Models
│
▼
Structured Output
Introduction
When building CVForbes, I quickly realized that depending on a single LLM provider wasn't a great long-term strategy.
What if Groq goes down?
What if another provider becomes cheaper or faster?
What if I want to test a new model without touching every service in my application?
Instead of tightly coupling my business logic to one provider, I designed a provider-agnostic routing layer that sits between my application and the LLMs.
Now, every AI feature—resume tailoring, parsing, cover letter generation, and future modules—talks to a single router. The router decides which provider should handle the request, performs automatic failover when necessary, and keeps the rest of the application completely unaware of what's happening behind the scenes.
This article explains the architecture behind that system and the design decisions that made it scalable.
The Problem
A lot of AI projects start like this:
llm = ChatGroq(...)
response = llm.invoke(prompt)
It works.
Until it doesn't.
Once your application grows, changing providers means editing multiple services, updating imports, and introducing provider-specific logic across the codebase.
I wanted to avoid that entirely.
The Solution
Instead of allowing services to communicate directly with an LLM provider, every request goes through a single router.
Client
│
▼
FastAPI Endpoint
│
▼
Dependency Injection
│
▼
LLMRouter Service
┌────────────┬──────────────┐
│ │ │
▼ ▼ ▼
GroqProvider OpenRouter OllamaProvider
│ │ │
└────────────┴──────────────┘
│
GeminiProvider
│
▼
LangChain Models
│
▼
Structured Output
Every service simply asks for "an LLM."
The router decides which one.
Why a Router?
The router has one job:
- Choose the best available provider.
- Handle failover.
- Return a LangChain model.
That's it.
Everything else stays where it belongs.
This keeps the application clean and makes providers interchangeable.
Provider Abstraction
Each provider follows the same interface.
class BaseLLMProvider:
def get_llm(self):
...
Whether it's Groq, Gemini, Ollama, or OpenRouter doesn't matter.
The router simply calls:
provider.get_llm()
Adding a new provider later becomes incredibly straightforward.
Request Lifecycle
Every AI request follows the same journey.
Incoming Request
│
▼
Need LLM?
│
▼
Router receives prompt
│
▼
Is Groq healthy?
/ \
Yes No
│ │
▼ ▼
Use Groq Try OpenRouter
│
▼
Healthy?
/ \
Yes No
│ │
▼ ▼
Use OpenRouter Try Ollama
│
▼
Healthy?
│
▼
Use Ollama
│
▼
Last fallback
Gemini
Notice something missing?
Nowhere in the application do we reference Groq or Gemini directly.
That's intentional.
Automatic Failover
One of my biggest goals was resilience.
Instead of immediately failing when a provider becomes unavailable, the router simply tries the next one.
Groq
│
❌
▼
OpenRouter
│
❌
▼
Ollama
│
✅
▼
Return Response
The user doesn't know a provider failed.
And honestly, they shouldn't have to.
Health Tracking
Trying the same failed provider on every request wastes time.
Instead, the router keeps lightweight health information.
Groq ✅
OpenRouter ✅
Ollama ❌
Gemini ✅
If a provider repeatedly fails, it's temporarily skipped until it becomes healthy again.
This reduces unnecessary delays during outages.
Dependency Injection
Another design choice was using FastAPI's dependency injection.
Instead of creating providers inside every endpoint, a shared router instance is injected wherever it's needed.
Endpoint
│
Depends()
│
LLM Router
│
Providers
This keeps endpoints focused on business logic rather than infrastructure.
Why This Architecture?
This approach gave CVForbes several advantages:
- ✅ Provider-agnostic business logic
- ✅ Automatic failover
- ✅ Easy provider replacement
- ✅ Clean separation of responsibilities
- ✅ Simple scalability
- ✅ Easier testing and maintenance
Perhaps my favorite part is that adding another provider requires almost no changes to the rest of the application.
The architecture grows without becoming more complicated.
Lessons Learned
Building AI applications isn't just about choosing the best model.
It's about designing systems that continue working when models, providers, or APIs inevitably change.
A small investment in abstraction early on saved me from coupling my entire application to a single vendor.
Looking back, that decision has probably been one of the most valuable architectural choices in CVForbes.
What's Next?
The router already supports multiple providers with automatic failover, but there's plenty of room for future improvements:
- Latency-aware routing
- Cost-aware provider selection
- Circuit breakers
- Metrics & monitoring
- Dynamic provider configuration
- Task-specific routing (e.g., different models for parsing vs. generation)
The exciting part is that the architecture already supports these ideas without requiring a redesign.
Final Thoughts
When people think about AI architecture, they often focus on which model to use.
I think the better question is:
How easily can your application switch models tomorrow?
Designing around abstractions instead of vendors made CVForbes significantly more maintainable, resilient, and scalable. Providers may change, APIs may evolve, and new models will continue to emerge—but the rest of the application won't need to know.
That's the kind of architecture I aim for: one that's built around software engineering principles rather than a single AI provider.
⭐ If you found this helpful...
The complete implementation, including the router, providers, health management, and FastAPI integration, is available in the accompanying GitHub repository.
Feel free to explore it, suggest improvements, or adapt the architecture for your own AI applications.
Top comments (4)
The router abstracting Groq, OpenRouter, Ollama, and Gemini behind one interface is clean, but the structured-output box at the end is where I usually get burned across providers. Each one honors JSON schema or function calling a bit differently, so a prompt that yields valid structured output on Gemini can drift on Groq. How are you validating the schema after failover so a provider switch doesn't silently change the output shape?
Great question @kartik-nvjk 🙌. The router is intentionally provider-agnostic; schema consistency is enforced above it using LangChain's structured output with a shared Pydantic schema. A provider switch is only considered successful if the response validates against that schema, otherwise it's treated as a failed generation, not a successful failover.
the context window variance across providers is the silent failure mode worth flagging. Groq's Llama 3.1 70B handles 128K tokens; a local Ollama model often caps at 4K or 8K. a failover from Groq to Ollama with a 12K token prompt technically succeeds, the Pydantic schema validates, but you've served a response built from heavily truncated input. the downstream feature doesn't error, it just produces noticeably worse output with no signal as to why.
the fix is probably token counting before routing and filtering candidates down to providers with sufficient context for that specific request. do you track prompt length per request today, or is context window enforcement something you're relying on the provider to surface?
Good point. Context-window mismatch is definitely one of the failure modes I considered, but it's only one of several reasons the router exists. The goal isn't just failover—it's provider orchestration across multiple constraints such as context size, RPM, TPM, RPD, daily quotas, latency, provider availability, and API failures. My current router already tracks provider capabilities and usage metrics, and the next step is adding pre-routing token estimation so requests are only considered for providers that can accommodate the required context. If a prompt exceeds a provider's effective context window, that provider will simply be excluded from the candidate pool rather than relied upon to truncate or fail. So the long-term routing decision is based on both capacity (context) and operational constraints, not tokens alone.