Most AI tutorials focus on a single model type, but real applications mix LLMs, vision models, embeddings, and code models. In this guide we will build a hybrid research assistant that routes tasks to the right Oxlo.ai model for the job. You will see exactly where a general-purpose LLM shines and where specialized models do the work better.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK:
pip install openai - NumPy for vector math:
pip install numpy - A sample image file named
diagram.pngin your working directory
Step 1: Set up the Oxlo.ai client
We start with a single OpenAI-compatible client pointed at Oxlo.ai. This one client will hit every endpoint we use, from chat to embeddings.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
Step 2: Add embedding retrieval with BGE-Large
Embeddings turn text into vectors so we can search by meaning rather than keywords. This is not an LLM. It is a retrieval model, and Oxlo.ai hosts BGE-Large for this exact use case.
import numpy as np
def get_embedding(text):
resp = client.embeddings.create(
model="bge-large",
input=text
)
return resp.data[0].embedding
KB = [
"LLMs predict the next token in a sequence using transformer attention.",
"Vision models encode image patches to classify objects or caption scenes.",
"Embedding models map discrete tokens to dense vectors for semantic search.",
"Code models are trained on AST-aware data to autocomplete and debug programs."
]
kb_embeddings = [get_embedding(doc) for doc in KB]
def retrieve(query, top_k=2):
q = np.array(get_embedding(query))
scores = [np.dot(q, np.array(e)) for e in kb_embeddings]
idx = np.argsort(scores)[-top_k:][::-1]
return [KB[i] for i in idx]
Step 3: Analyze images with Gemma 3 27B
Vision models read pixels, not tokens. When a user references an image, we send base64 bytes to a vision-capable chat model. Oxlo.ai offers Gemma 3 27B for this.
import base64
def analyze_image(path):
with open(path, "rb") as f:
b64 = base64.b64encode(f.read()).decode("utf-8")
resp = client.chat.completions.create(
model="gemma-3-27b",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in one sentence."},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}
]
}]
)
return resp.choices[0].message.content
Step 4: Generate code with DeepSeek V3.2
General LLMs can write code, but specialized code models often produce tighter syntax and better logic. We will use DeepSeek V3.2 on Oxlo.ai as our coding specialist.
def generate_code(task):
resp = client.chat.completions.create(
model="deepseek-v3.2",
messages=[
{"role": "system", "content": "You are an expert Python programmer. Return only the code."},
{"role": "user", "content": task}
]
)
return resp.choices[0].message.content
Step 5: Build the LLM orchestrator
The orchestrator is a general LLM, Llama 3.3 70B, but we do not ask it to do everything. We ask it to choose tools and synthesize answers. Here is the system prompt.
SYSTEM_PROMPT = """You are a hybrid research assistant. You have three tools:
1. retrieve(query) - semantic search over a tech knowledge base.
2. analyze_image(path) - describe an image.
3. generate_code(task) - write Python code.
Given a user question, decide which tools to invoke, then synthesize a concise final answer from their outputs. If the user asks for code, delegate to generate_code. If they upload an image, delegate to analyze_image. For conceptual questions, use retrieve."""
Now we implement the planner that decides which tools to call.
import json
def plan(user_msg):
resp = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Plan tools for this request. Reply with JSON keys: retrieve (bool), image_path (string or null), code_task (string or null). Request: {user_msg}"}
],
response_format={"type": "json_object"}
)
return json.loads(resp.choices[0].message.content)
Step 6: Wire everything into the agent
This class gathers the tool outputs and asks the orchestrator to produce the final response.
class HybridAgent:
def run(self, user_msg):
plan_data = plan(user_msg)
parts = []
if plan_data.get("retrieve"):
parts.append("Retrieved:\n" + "\n".join(retrieve(user_msg)))
if plan_data.get("image_path"):
parts.append("Vision:\n" + analyze_image(plan_data["image_path"]))
if plan_data.get("code_task"):
parts.append("Code:\n" + generate_code(plan_data["code_task"]))
context = "\n\n".join(parts) or "No tools needed."
resp = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_msg},
{"role": "user", "content": f"Tool outputs:\n{context}"}
]
)
return resp.choices[0].message.content
Run it
Call the agent with a request that needs both conceptual retrieval and code generation.
if __name__ == "__main__":
agent = HybridAgent()
print(agent.run("Explain how LLMs differ from embedding models and write a cosine similarity function."))
Example output:
LLMs generate text token by token, while embedding models produce dense vectors for search.
Here is the cosine similarity function:
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
Wrap-up and next steps
You now have a working multi-model agent that delegates to the right architecture for each task. Try adding audio transcription with Whisper via Oxlo.ai, or swap in Qwen 3 Coder 30B for larger code generation jobs. If you are running this in production, Oxlo.ai request-based pricing keeps long context windows cheap because you pay per request, not per token. See https://oxlo.ai/pricing for details.
Top comments (0)