We're building a lightweight, multi-model LLM gateway in FastAPI that routes requests to the best Oxlo.ai model for the task. If you need to ship internal AI tools without managing infrastructure, this gives you a working backend in under an hour.
What you'll need
Grab Python 3.10+, an Oxlo.ai API key from https://portal.oxlo.ai, and install the dependencies.
pip install openai fastapi uvicorn pydantic python-dotenvStep 1: Scaffold the project
I start with a single main.py file and a .env file to keep secrets out of code.
# .env
OXLO_API_KEY=YOUR_OXLO_API_KEY
Step 2: Initialize the Oxlo.ai client and model registry
I configure the OpenAI-compatible client to point at Oxlo.ai and define which models the gateway can use. Oxlo.ai uses flat per-request pricing, so routing to a larger model for complex tasks does not blow up our bill the way token-based pricing would. See https://oxlo.ai/pricing for details.
import os
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
MODEL_REGISTRY = {
"general": "llama-3.3-70b",
"coding": "deepseek-v3.2",
"reasoning": "deepseek-r1-671b",
"vision": "kimi-k2.6",
"multilingual": "qwen-3-32b"
}
Step 3: Define request schemas and the router system prompt
The gateway uses a small router call to classify the request. Here is the system prompt that guides that classifier.
from pydantic import BaseModel
from typing import List, Optional, Dict, Any
class ChatMessage(BaseModel):
role: str
content: str
class ChatRequest(BaseModel):
messages: List[ChatMessage]
tools: Optional[List[Dict[str, Any]]] = None
stream: Optional[bool] = False
SYSTEM_PROMPT = """You are a model router. Analyze the user's request and respond with exactly one word from this list: general, coding, reasoning, vision, multilingual.
Rules:
- Pick 'coding' for programming, debugging, or script generation.
- Pick 'reasoning' for math, logic puzzles, or step-by-step analysis.
- Pick 'vision' if the user references images or visual content.
- Pick 'multilingual' for non-English translation or generation.
- Otherwise pick 'general'."""
Step 4: Build the model router
I send the last user message to a fast Oxlo.ai model with the router prompt. Oxlo.ai has no cold starts on popular models, so this classification call returns immediately.
def route_model(messages: List[ChatMessage]) -> str:
user_message = next(
(m.content for m in reversed(messages) if m.role == "user"),
""
)
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
max_tokens=10,
)
choice = response.choices[0].message.content.strip().lower()
return MODEL_REGISTRY.get(choice, MODEL_REGISTRY["general"])
Step 5: Create the chat completions endpoint
Now I wire up FastAPI. The endpoint picks a model, forwards the full conversation to Oxlo.ai, and returns the result. Because Oxlo.ai is fully OpenAI SDK compatible, the drop-in replacement requires no client-side changes beyond the base URL.
from fastapi import FastAPI
app = FastAPI(title="Oxlo.ai Gateway")
@app.post("/v1/chat/completions")
def chat_completions(request: ChatRequest):
model = route_model(request.messages)
response = client.chat.completions.create(
model=model,
messages=[m.model_dump() for m in request.messages],
tools=request.tools,
stream=request.stream,
)
if request.stream:
return response
return {
"model": model,
"choices": [
{
"index": 0,
"message": {
"role": response.choices[0].message.role,
"content": response.choices[0].message.content,
},
"finish_reason": response.choices[0].finish_reason,
}
],
}
Step 6: Add tool use support
To make the gateway useful for agents, I add a calculator tool and a loop that handles tool calls. Oxlo.ai models like Llama 3.3 70B and DeepSeek V3.2 support function calling out of the box.
import json
TOOLS = [
{
"type": "function",
"function": {
"name": "calculator",
"description": "Evaluate a math expression safely.",
"parameters": {
"type": "object",
"properties": {
"expression": {"type": "string"}
},
"required": ["expression"]
}
}
}
]
def call_tool(name: str, arguments: str) -> str:
args = json.loads(arguments)
if name == "calculator":
try:
result = eval(args["expression"], {"__builtins__": {}}, {})
return str(result)
except Exception as e:
return f"Error: {e}"
return "Unknown tool"
@app.post("/v1/chat/completions")
def chat_completions(request: ChatRequest):
model = route_model(request.messages)
messages = [m.model_dump() for m in request.messages]
response = client.chat.completions.create(
model=model,
messages=messages,
tools=TOOLS,
)
msg = response.choices[0].message
if msg.tool_calls:
messages.append({
"role": msg.role,
"content": msg.content or "",
"tool_calls": [tc.model_dump() for tc in msg.tool_calls]
})
for tc in msg.tool_calls:
result = call_tool(tc.function.name, tc.function.arguments)
messages.append({
"role": "tool",
"tool_call_id": tc.id,
"content": result,
})
final = client.chat.completions.create(
model=model,
messages=messages,
tools=TOOLS,
)
return {
"model": model,
"choices": [
{
"index": 0,
"message": {
"role": final.choices[0].message.role,
"content": final.choices[0].message.content,
},
"finish_reason": final.choices[0].finish_reason,
}
],
}
return {
"model": model,
"choices": [
{
"index": 0,
"message": {
"role": msg.role,
"content": msg.content,
},
"finish_reason": response.choices[0].finish_reason,
}
],
}
Step 7: Run the server
I use Uvicorn to start the gateway locally.
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
Run it
Start the service, then send a request. The router will pick a model and return the response.
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": "Write a Python function that computes the factorial of n."}
]
}'
Example output:
{
"model": "deepseek-v3.2",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Here is a Python function to compute factorial using recursion:\n\n
```python\ndef factorial(n):\n if n < 0:\n raise ValueError(\"n must be non-negative\")\n if n == 0:\n return 1\n return n * factorial(n - 1)\n```
"
},
"finish_reason": "stop"
}
]
}
Wrap-up
This gateway gives you a clean abstraction over Oxlo.ai's model catalog. Next, add streaming by returning a FastAPI StreamingResponse around the Oxlo.ai stream, or persist request logs to SQLite so you can tune the router prompt later. Oxlo.ai's flat per-request pricing makes it cheap to experiment with both the router and the worker calls without tracking tokens.
Top comments (0)