Building a virtual assistant that actually gets things done means moving beyond simple chat completions. You need a pipeline that can parse speech, maintain multi-turn memory, execute tools, and optionally reason over images. Each of these stages sends data to an LLM or an auxiliary model, and when that assistant runs in an agent loop, context windows grow fast. If your inference provider bills by the token, costs become unpredictable the moment you add retrieval memory or agentic tool use. That is why the backend you choose shapes the economics of the product as much as the model choice does, and it is where Oxlo.ai becomes a natural fit.
The Architecture of a Modern Assistant
A production assistant usually combines four services:
- An LLM for reasoning, dialogue, and tool orchestration.
- An embedding model for long-term memory and retrieval.
- A speech pipeline: ASR and TTS for voice interfaces.
- A vision encoder for image inputs.
Oxlo.ai covers all four through a single API and SDK, so you do not need to stitch together multiple providers. The platform offers 45+ open-source and proprietary models across seven categories, with full OpenAI SDK compatibility and no cold starts on popular models.
Core LLM with Tool Use
For the brain of the assistant, you want strong function-calling support and a wide context window. Oxlo.ai offers several candidates:
- Llama 3.3 70B: general-purpose flagship, reliable for intent parsing and multi-turn dialogue.
- Qwen 3 32B: multilingual reasoning and agent workflows.
- Kimi K2.6: 131K context, advanced reasoning, agentic coding, and vision.
- DeepSeek V3.2: strong coding and reasoning, available on the free tier.
- GLM 5 or Minimax M2.5: optimized for long-horizon agentic tasks and tool use.
All of these support streaming, JSON mode, and multi-turn conversations, so you can build deterministic tool loops without extra parsing logic.
Drop-In SDK Example
Because Oxlo.ai is fully OpenAI SDK compatible, you can port existing assistant logic by changing only the base_url and API key. Below is a minimal tool-use loop using Python:
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"}
},
"required": ["location"]
}
}
}
]
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "What's the weather in Berlin?"}],
tools=tools,
stream=False
)
print(response.choices[0].message)
If the model returns a tool call, your executor runs the function, appends the result to the message list, and sends a follow-up request. In an agent loop, prompt length increases every turn. On token-based providers, that linear growth directly inflates your bill. Oxlo.ai charges a flat rate per request, so a four-turn agent loop with a 10K context costs the same per step as a single-turn greeting.
Long-Term Memory with Embeddings
To avoid sending the entire conversation history on every request, store facts in a vector database and retrieve only the relevant chunks. Oxlo.ai provides embedding endpoints via models such as BGE-Large and E5-Large.
embedding = client.embeddings.create(
model="bge-large",
input="User prefers Celsius for weather updates."
)
# Store embedding.vector in Postgres/pgvector, Milvus, or similar.
At inference time, retrieve top-k chunks and inject them into the system prompt. Because Oxlo.ai bills per request, you can refresh embeddings or run retrieval augmentations without watching token meters spin.
Voice and Vision Inputs
A complete assistant should hear and see. Oxlo.ai exposes audio and vision endpoints under the same API key and base URL.
Speech-to-text with Whisper:
with open("audio.wav", "rb") as f:
transcript = client.audio.transcriptions.create(
model="whisper-large-v3",
file=f
)
print(transcript.text)
Text-to-speech with Kokoro:
speech = client.audio.speech.create(
model="kokoro-82m",
voice="af",
input="The weather in Berlin is 22 degrees Celsius."
)
with open("output.wav", "wb") as f:
f.write(speech.content)
For vision, pass an image URL or base64 payload to a vision-capable model such as Kimi K2.6 or Gemma 3 27B using the standard chat completions format. This lets your assistant process screenshots, documents, or camera frames without a separate image API.
Why Request Pricing Wins for Assistants
Agentic assistants are inherently long-context workloads. Each tool result, each retrieved memory chunk, and each prior assistant message adds tokens. Competitors that bill per token, such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, scale costs with every byte of context. Oxlo.ai uses request-based pricing: one flat cost per API call regardless of prompt length. For multi-turn agents or retrieval-heavy pipelines, that difference can be 10-100x in favor of Oxlo.ai. You can explore exact plan details at https://oxlo.ai/pricing.
Deployment Notes
Run the orchestrator as a lightweight Python service. Keep conversation state in Redis, vector memory in pgvector or Milvus, and route all model traffic through https://api.oxlo.ai/v1. Because there are no cold starts on popular models, the assistant responds immediately even after idle periods. If you need dedicated throughput, the Enterprise tier offers custom quotas and dedicated GPUs.
Summary
A virtual assistant is only as reliable as its inference layer. You need tool support, streaming, vision, audio, and embeddings, all under one roof. Oxlo.ai provides that with an OpenAI-compatible stack and a pricing model that stays flat as your context grows. If you are prototyping a voice-enabled agent or shipping a retrieval-heavy assistant, Oxlo.ai keeps costs predictable while giving you access to 45+ models across every modality.
Top comments (0)