🚀 Building a Production-Ready Multi-LLM Router with LangChain & FastAPI
How I designed CVForbes to automatically switch between Groq, OpenRouter, Ollama, and Gemini without changing a single service.
Client
│
▼
FastAPI Endpoint
│
▼
Dependency Injection
│
▼
LLMRouter Service
┌────────────┬──────────────┐
│ │ │
▼ ▼ ▼
GroqProvider OpenRouter OllamaProvider
│ │ │
└────────────┴──────────────┘
│
GeminiProvider
│
▼
LangChain Models
│
▼
Structured Output
Introduction
When building CVForbes, I quickly realized that depending on a single LLM provider wasn't a great long-term strategy.
What if Groq goes down?
What if another provider becomes cheaper or faster?
What if I want to test a new model without touching every service in my application?
Instead of tightly coupling my business logic to one provider, I designed a provider-agnostic routing layer that sits between my application and the LLMs.
Now, every AI feature—resume tailoring, parsing, cover letter generation, and future modules—talks to a single router. The router decides which provider should handle the request, performs automatic failover when necessary, and keeps the rest of the application completely unaware of what's happening behind the scenes.
This article explains the architecture behind that system and the design decisions that made it scalable.
The Problem
A lot of AI projects start like this:
llm = ChatGroq(...)
response = llm.invoke(prompt)
It works.
Until it doesn't.
Once your application grows, changing providers means editing multiple services, updating imports, and introducing provider-specific logic across the codebase.
I wanted to avoid that entirely.
The Solution
Instead of allowing services to communicate directly with an LLM provider, every request goes through a single router.
Client
│
▼
FastAPI Endpoint
│
▼
Dependency Injection
│
▼
LLMRouter Service
┌────────────┬──────────────┐
│ │ │
▼ ▼ ▼
GroqProvider OpenRouter OllamaProvider
│ │ │
└────────────┴──────────────┘
│
GeminiProvider
│
▼
LangChain Models
│
▼
Structured Output
Every service simply asks for "an LLM."
The router decides which one.
Why a Router?
The router has one job:
- Choose the best available provider.
- Handle failover.
- Return a LangChain model.
That's it.
Everything else stays where it belongs.
This keeps the application clean and makes providers interchangeable.
Provider Abstraction
Each provider follows the same interface.
class BaseLLMProvider:
def get_llm(self):
...
Whether it's Groq, Gemini, Ollama, or OpenRouter doesn't matter.
The router simply calls:
provider.get_llm()
Adding a new provider later becomes incredibly straightforward.
Request Lifecycle
Every AI request follows the same journey.
Incoming Request
│
▼
Need LLM?
│
▼
Router receives prompt
│
▼
Is Groq healthy?
/ \
Yes No
│ │
▼ ▼
Use Groq Try OpenRouter
│
▼
Healthy?
/ \
Yes No
│ │
▼ ▼
Use OpenRouter Try Ollama
│
▼
Healthy?
│
▼
Use Ollama
│
▼
Last fallback
Gemini
Notice something missing?
Nowhere in the application do we reference Groq or Gemini directly.
That's intentional.
Automatic Failover
One of my biggest goals was resilience.
Instead of immediately failing when a provider becomes unavailable, the router simply tries the next one.
Groq
│
❌
▼
OpenRouter
│
❌
▼
Ollama
│
✅
▼
Return Response
The user doesn't know a provider failed.
And honestly, they shouldn't have to.
Health Tracking
Trying the same failed provider on every request wastes time.
Instead, the router keeps lightweight health information.
Groq ✅
OpenRouter ✅
Ollama ❌
Gemini ✅
If a provider repeatedly fails, it's temporarily skipped until it becomes healthy again.
This reduces unnecessary delays during outages.
Dependency Injection
Another design choice was using FastAPI's dependency injection.
Instead of creating providers inside every endpoint, a shared router instance is injected wherever it's needed.
Endpoint
│
Depends()
│
LLM Router
│
Providers
This keeps endpoints focused on business logic rather than infrastructure.
Why This Architecture?
This approach gave CVForbes several advantages:
- ✅ Provider-agnostic business logic
- ✅ Automatic failover
- ✅ Easy provider replacement
- ✅ Clean separation of responsibilities
- ✅ Simple scalability
- ✅ Easier testing and maintenance
Perhaps my favorite part is that adding another provider requires almost no changes to the rest of the application.
The architecture grows without becoming more complicated.
Lessons Learned
Building AI applications isn't just about choosing the best model.
It's about designing systems that continue working when models, providers, or APIs inevitably change.
A small investment in abstraction early on saved me from coupling my entire application to a single vendor.
Looking back, that decision has probably been one of the most valuable architectural choices in CVForbes.
What's Next?
The router already supports multiple providers with automatic failover, but there's plenty of room for future improvements:
- Latency-aware routing
- Cost-aware provider selection
- Circuit breakers
- Metrics & monitoring
- Dynamic provider configuration
- Task-specific routing (e.g., different models for parsing vs. generation)
The exciting part is that the architecture already supports these ideas without requiring a redesign.
Final Thoughts
When people think about AI architecture, they often focus on which model to use.
I think the better question is:
How easily can your application switch models tomorrow?
Designing around abstractions instead of vendors made CVForbes significantly more maintainable, resilient, and scalable. Providers may change, APIs may evolve, and new models will continue to emerge—but the rest of the application won't need to know.
That's the kind of architecture I aim for: one that's built around software engineering principles rather than a single AI provider.
⭐ If you found this helpful...
The complete implementation, including the router, providers, health management, and FastAPI integration, is available in the accompanying GitHub repository.
Feel free to explore it, suggest improvements, or adapt the architecture for your own AI applications.
Top comments (0)