I recently built an internal AI gateway that routes user requests to specialized LLM microservices based on intent. Splitting models by responsibility keeps latency predictable, isolates failures, and lets us optimize each workload independently. In this tutorial, I will walk through building a minimal version with an API gateway, a router service, and two downstream workers, all running on Oxlo.ai.
What you'll need
- Python 3.10 or newer
pip install openai fastapi uvicorn httpx- An Oxlo.ai API key from https://portal.oxlo.ai
- A free Oxlo.ai account. The platform uses request-based pricing, so passing large contexts between services does not inflate costs the way token-based billing would. See https://oxlo.ai/pricing for details.
Step 1: Build the intent router
The router is the first stop for every request. It classifies the user message as either code or general so the gateway knows where to send it. I use Qwen 3 32B because its agentic reasoning is sharp for low-latency classification.
The router uses this system prompt:
SYSTEM_PROMPT = """You are an intent classifier. Read the user message and respond with exactly one word: 'code' if the user wants help with programming, debugging, or technical implementation, or 'general' for everything else. Do not explain your reasoning."""Save the following as router_service.py:
from openai import OpenAI
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are an intent classifier. Read the user message and respond with exactly one word: 'code' if the user wants help with programming, debugging, or technical implementation, or 'general' for everything else. Do not explain your reasoning."""
class RouteRequest(BaseModel):
message: str
@app.post("/route")
async def route_request(req: RouteRequest):
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": req.message},
],
)
intent = response.choices[0].message.content.strip().lower()
return {"intent": intent}
Step 2: Build the code specialist service
The code service handles anything technical. I point it at DeepSeek V3.2 because Oxlo.ai serves it with no cold starts, which matters when this microservice scales from zero to many replicas.
from openai import OpenAI
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
CODE_SYSTEM_PROMPT = """You are a senior software engineer. Provide concise, production-ready code with brief explanations. Include comments only when logic is non-obvious."""
class CodeRequest(BaseModel):
message: str
@app.post("/generate")
async def generate_code(req: CodeRequest):
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[
{"role": "system", "content": CODE_SYSTEM_PROMPT},
{"role": "user", "content": req.message},
],
)
return {"response": response.choices[0].message.content}
Step 3: Build the general chat service
For everything else, I run a generalist service backed by Llama 3.3 70B. Isolating it means a traffic spike on the chat endpoint never starves the code pipeline.
from openai import OpenAI
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
CHAT_SYSTEM_PROMPT = """You are a helpful assistant. Answer clearly and accurately. If you do not know something, say so."""
class ChatRequest(BaseModel):
message: str
@app.post("/chat")
async def chat(req: ChatRequest):
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": CHAT_SYSTEM_PROMPT},
{"role": "user", "content": req.message},
],
)
return {"response": response.choices[0].message.content}
Step 4: Build the API gateway
The gateway receives external traffic, calls the router, then forwards the request to the correct downstream service. I keep it thin and stateless so it never becomes a bottleneck.
import httpx
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
ROUTER_URL = "http://localhost:8001/route"
CODE_URL = "http://localhost:8002/generate"
CHAT_URL = "http://localhost:8003/chat"
class GatewayRequest(BaseModel):
message: str
@app.post("/ask")
async def ask(req: GatewayRequest):
async with httpx.AsyncClient(timeout=60.0) as client:
route_resp = await client.post(ROUTER_URL, json={"message": req.message})
intent = route_resp.json()["intent"]
if intent == "code":
service_resp = await client.post(CODE_URL, json={"message": req.message})
else:
service_resp = await client.post(CHAT_URL, json={"message": req.message})
return {
"intent": intent,
"response": service_resp.json()["response"]
}
Run it
Open four terminal windows and start each service on its own port.
# Terminal 1: Router
uvicorn router_service:app --port 8001
# Terminal 2: Code specialist
uvicorn code_service:app --port 8002
# Terminal 3: General chat
uvicorn chat_service:app --port 8003
# Terminal 4: Gateway
uvicorn gateway:app --port 8000
Test the pipeline with curl. The gateway should route a coding question to DeepSeek V3.2.
curl -X POST http://localhost:8000/ask \
-H "Content-Type: application/json" \
-d '{"message": "Write a Python function that validates an email address with regex."}'
Example output:
{
"intent": "code",
"response": "Here is a concise Python function using the `re` module...\n\n
```python\nimport re\n\ndef validate_email(email):\n pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$'\n return re.match(pattern, email) is not None\n```
"
}
And a general question:
curl -X POST http://localhost:8000/ask \
-H "Content-Type: application/json" \
-d '{"message": "What are the tradeoffs between REST and gRPC?"}'
Example output:
{
"intent": "general",
"response": "REST and gRPC serve similar purposes but differ in several key areas. REST uses HTTP/1.1 or HTTP/2 with JSON, which makes it human-readable and easy to debug. gRPC uses HTTP/2 and Protocol Buffers by default, which yields smaller payloads and stronger contracts but adds tooling complexity."
}
Wrap-up
From here, add a Redis cache in front of the gateway to avoid redundant LLM calls for repeated questions. Then containerize each service and replace the localhost URLs with internal DNS so the mesh can scale horizontally.
Because Oxlo.ai uses flat per-request pricing, adding replicas or increasing context windows does not change your unit economics. That makes it straightforward to grow the architecture without surprise bills.
Top comments (0)