Multimodal reasoning systems combine vision, language, and tool use to solve tasks that require understanding both visual input and complex context. A typical pipeline ingests an image, extracts structured observations through a vision model, reasons over those observations with a large language model, and optionally invokes external tools to verify or act on the conclusion. Building this on traditional token-based infrastructure introduces unpredictable costs, especially when high-resolution images expand prompt length by tens of thousands of tokens. Oxlo.ai removes that uncertainty with request-based pricing, so you can iterate on long-context multimodal agents without watching token meters. This guide walks through a concrete implementation using Oxlo.ai's OpenAI-compatible API, vision models, and reasoning models.
Architecture Overview
A robust multimodal reasoning stack has three layers. The perception layer processes images or video into text or structured embeddings. The cognition layer performs chain-of-thought reasoning, consistency checking, and planning. The action layer executes function calls, database lookups, or code interpreters. Separating these concerns lets you upgrade individual components without rewiring the entire system.
For this guide, the perception layer uses a vision-language model to describe image contents. The cognition layer routes those descriptions to a reasoning model capable of complex analysis. The action layer exposes a set of Python functions via OpenAI-style function calling. All three layers run against Oxlo.ai's unified endpoint at https://api.oxlo.ai/v1, which removes the need to manage multiple provider contracts or SDKs.
Model Selection
Oxlo.ai hosts several models that fit distinct roles in this stack.
For vision perception, Gemma 3 27B and Kimi VL A3B accept image inputs and generate detailed captions or structured JSON. Kimi VL A3B is particularly useful when you need fine-grained spatial reasoning or OCR over screenshots.
For reasoning, DeepSeek R1 671B MoE excels at deep, multi-step logic and complex coding. Kimi K2.6 offers advanced reasoning with a 131K context window and native vision support, so it can optionally handle both perception and cognition in a single call for simpler workflows. Qwen 3 32B provides strong multilingual reasoning and agentic tool use, while Llama 3.3 70B serves as a reliable general-purpose fallback.
You can mix and match these models in the same application because Oxlo.ai exposes them through one OpenAI-compatible schema.
SDK Setup and Authentication
Because Oxlo.ai is fully OpenAI SDK compatible, you only need to swap the base URL and API key.
from openai import OpenAI
import base64
import json
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
Store your key in an environment variable rather than committing it. The client supports streaming, JSON mode, and function calling out of the box, so the rest of your agent code stays identical to any OpenAI-targeted implementation.
Step 1: Vision Encoding
Convert your image to a base64 data URL and send it to a vision model. The following example uses Gemma 3 27B to extract a structured description.
def encode_image(path):
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
b64 = encode_image("diagram.png")
response = client.chat.completions.create(
model="gemma-3-27b-it",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in structured JSON. Include fields: objects, text_content, layout_type."},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}
]
}
],
response_format={"type": "json_object"}
)
vision_output = json.loads(response.choices[0].message.content)
The model returns a JSON object that downstream stages can consume predictably. If your images are large or numerous, note that Oxlo.ai's request-based pricing means the cost does not scale with image resolution or token count. You pay one flat rate per request, which simplifies budgeting for high-fidelity perception pipelines. See https://oxlo.ai/pricing for current plan details.
Step 2: Structured Reasoning
Feed the vision output into a reasoning model to draw conclusions or plan next steps. Here we use DeepSeek R1 671B MoE with a system prompt that enforces analytical rigor.
reasoning_prompt = f"""
You are an analytical engine. You will receive structured image observations.
Verify consistency, identify anomalies, and propose hypotheses.
Respond in JSON with fields: consistency_score, anomalies, hypothesis, recommended_action.
Observations: {json.dumps(vision_output)}
"""
response = client.chat.completions.create(
model="deepseek-r1-671b",
messages=[
{"role": "system", "content": "You are a meticulous reasoning assistant. Think step by step."},
{"role": "user", "content": reasoning_prompt}
],
response_format={"type": "json_object"}
)
reasoning_result = json.loads(response.choices[0].message.content)
DeepSeek R1 exposes its chain-of-thought reasoning in the response, which you can log for debugging without parsing it into the final JSON. If you prefer a model that handles vision and reasoning in one call, Kimi K2.6 accepts image URLs directly and can perform the prior two steps in a single request.
Step 3: Tool Use and Action
Once the reasoning layer decides on an action, invoke external tools through OpenAI-style function calling. Define your tools as JSON schemas and let the model choose which to call.
tools = [
{
"type": "function",
"function": {
"name": "query_database",
"description": "Look up part numbers from an engineering database",
"parameters": {
"type": "object",
"properties": {
"part_name": {"type": "string"}
},
"required": ["part_name"]
}
}
}
]
action_response = client.chat.completions.create(
model="qwen3-32b",
messages=[
{"role": "system", "content": "You may use tools to verify hypotheses. Call only one function at a time."},
{"role": "user", "content": f"Hypothesis: {reasoning_result['hypothesis']}. Verify using available tools."}
],
tools=tools,
tool_choice="auto"
)
message = action_response.choices[0].message
if message.tool_calls:
tool_call = message.tool_calls[0]
print(f"Tool called: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")
Qwen 3 32B is well suited for agentic tool use, especially when the workflow requires multilingual reasoning or rapid function calling. Because Oxlo.ai carries no cold starts on popular models, the tool invocation returns immediately, which keeps agent loops responsive.
Step 4: Orchestration and State Management
In production, you need a state machine that tracks which stage of the pipeline is active and handles failures gracefully. A minimal implementation stores context in a list of messages and appends tool results before re-invoking the model.
messages = [
{"role": "system", "content": "You are a multimodal reasoning agent. Process the image, reason, and act."},
{"role": "user", "content": [
{"type": "text", "text": "Analyze this engineering schematic and check inventory."},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}
]}
]
# First pass: vision + reasoning in one pass with Kimi K2.6
response = client.chat.completions.create(
model="kimi-k2-6",
messages=messages,
tools=tools
)
messages.append(response.choices[0].message)
# If tool call requested, execute locally and append result
if messages[-1].tool_calls:
args = json.loads(messages[-1].tool_calls[0].function.arguments)
db_result = query_database(args["part_name"]) # local implementation
messages.append({
"role": "tool",
"tool_call_id": messages[-1].tool_calls[0].id,
"content": json.dumps(db_result)
})
# Second pass: synthesize final answer
final = client.chat.completions.create(
model="kimi-k2-6",
messages=messages
)
print(final.choices[0].message.content)
Kimi K2.6 supports vision, advanced reasoning, and function calling within its 131K context window, so you can collapse multiple turns into a single long-context session. On a token-based provider, that context length would generate significant input charges on every turn. Oxlo.ai's flat per-request pricing makes it feasible to keep the full multimodal conversation history in context without cost spikes.
Cost Control with Long Multimodal Contexts
Multimodal agents often balloon in token count. A single 1024x1024 image can represent thousands of tokens, and agentic loops with tool results can accumulate tens of thousands of input tokens per turn. On token-based platforms, this creates a direct tradeoff between fidelity and cost.
Oxlo.ai inverts that tradeoff. Because you pay per request rather than per token, you can send high-resolution images, maintain extensive system prompts, and preserve full conversation history without incremental charges. For agentic workloads that iterate over visual inputs, this pricing structure is often significantly cheaper than scaling by token volume. Plans start with a free tier offering 60 requests per day across 16+ models, which is enough to prototype a full multimodal pipeline before committing to a paid tier. Visit https://oxlo.ai/pricing to compare plans.
Full Working Example
Here is a complete, self-contained script that ties the stages together. It reads an image, describes it, reasons about the contents, and conditionally calls a tool.
from openai import OpenAI
import base64
import json
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
def encode_image(path):
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
def query_database(part_name):
# Placeholder for your real database layer
return {"part_name": part_name, "stock": 42, "location": "Aisle-7"}
b64 = encode_image("schematic.png")
messages = [
{"role": "system", "content": "You are a multimodal reasoning agent. Analyze images carefully, then reason and act."},
{"role": "user", "content": [
{"type": "text", "text": "Describe this schematic, check for anomalies, and look up any part numbers in inventory."},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}
]}
]
# Use Kimi K2.6 for unified vision + reasoning + tool use
response = client.chat.completions.create(
model="kimi-k2-6",
messages=messages,
tools=[
{
"type": "function",
"function": {
"name": "query_database",
"description": "Look up part numbers from an engineering database",
"parameters": {
"type": "object",
"properties": {
"part_name": {"type": "string"}
},
"required": ["part_name"]
}
}
}
],
tool_choice="auto"
)
messages.append(response.choices[0].message)
if messages[-1].tool_calls:
call = messages[-1].tool_calls[0]
args = json.loads(call.function.arguments)
result = query_database(args["part_name"])
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": json.dumps(result)
})
final = client.chat.completions.create(
model="kimi-k2-6",
messages=messages
)
print(final.choices[0].message.content)
else:
print(messages[-1].content)
This script runs unchanged against Oxlo.ai because the platform exposes chat completions, function calling, and vision inputs through the standard OpenAI schema.
Conclusion
Building a multimodal reasoning system does not require stitching together disparate APIs or accepting unpredictable token costs. By composing vision models like Gemma 3 27B and Kimi VL A3B with reasoning models like DeepSeek R1 671B MoE and Kimi K2.6, you can construct a layered agent that perceives, analyzes, and acts. Oxlo.ai hosts all of these models behind one OpenAI-compatible endpoint, with request-based pricing that keeps long-context and multimodal workloads affordable. If you are prototyping an agent that processes images, maintains state, and calls tools, the free tier at Oxlo.ai gives you 60 requests per day to validate the architecture before scaling. Start at https://oxlo.ai/pricing and point your existing OpenAI client to https://api.oxlo.ai/v1.
Top comments (0)