We're building a lightweight multimodal evaluation harness that routes image, audio, and video frames through different Oxlo.ai models in a single script. It helps developers and ML engineers quickly see which model handles a given modality best without managing multiple provider SDKs. Everything runs against Oxlo.ai's flat per-request pricing through OpenAI-compatible endpoints.
What you'll need
- Python 3.10 or newer
pip install openai opencv-python- An Oxlo.ai API key from https://portal.oxlo.ai
- A sample JPEG image, a WAV or MP3 audio clip, and a short MP4 video for testing
1. Set up the Oxlo.ai client and system prompt
I start every multimodal run with a strict system prompt so the models return structured, comparable output. This keeps evaluation consistent across vision and text.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
SYSTEM_PROMPT = """You are a precise multimodal analyst. Describe what you see or hear in three bullet points. Do not speculate beyond the input. Keep each bullet under fifteen words."""2. Compare vision models on a static image
I encode a local image to base64 and send identical prompts to Gemma 3 27B and Kimi K2.6. Both requests use the same chat.completions endpoint, so the only variable is the model.
import base64
def encode_image(path):
with open(path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
def compare_vision(image_path, question):
b64 = encode_image(image_path)
image_url = f"data:image/jpeg;base64,{b64}"
models = ["gemma-3-27b", "kimi-k2.6"]
results = {}
for model in models:
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": [
{"type": "text", "text": question},
{"type": "image_url", "image_url": {"url": image_url}}
]},
],
)
results[model] = response.choices[0].message.content
print(f"\n=== {model} ===\n{results[model]}")
return results3. Transcribe audio with Whisper
For audio, I use the same OpenAI client to call Oxlo.ai's Whisper Large v3 endpoint. The transcription becomes the text input for later reasoning steps.
def transcribe_audio(audio_path):
with open(audio_path, "rb") as audio_file:
transcript = client.audio.transcriptions.create(
model="whisper-large-v3",
file=audio_file,
)
print(f"Transcription: {transcript.text}")
return transcript.text4. Extract a video frame and analyze it
Video does not need a separate endpoint. I grab the middle frame with OpenCV, save it to disk, and push it through Qwen 3 32B to get a concise description.
import cv2
def describe_video_frame(video_path):
cap = cv2.VideoCapture(video_path)
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
cap.set(cv2.CAP_PROP_POS_FRAMES, total_frames // 2)
ret, frame = cap.read()
if not ret:
raise ValueError("Could not read frame")
cv2.imwrite("mid_frame.jpg", frame)
cap.release()
b64 = encode_image("mid_frame.jpg")
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": [
{"type": "text", "text": "Describe the action in this video frame."},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
]},
],
)
return response.choices[0].message.content5. Synthesize a ranked comparison
Finally, I feed all raw outputs into both Llama 3.3 70B and DeepSeek V3.2 to see which evaluator gives the sharper critique. This turns isolated model outputs into an actionable comparison.
def evaluate_with_model(model, image_results, transcript, video_description):
payload = f"""
Image analysis from Gemma 3 27B:
{image_results.get('gemma-3-27b', '')}
Image analysis from Kimi K2.6:
{image_results.get('kimi-k2.6', '')}
Audio transcription:
{transcript}
Video frame description from Qwen 3 32B:
{video_description}
Task: Rank the vision descriptions by accuracy and conciseness. Summarize whether the audio transcription captured the main speaker intent. Keep under 100 words.
"""
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "You are a critical evaluator. Be brief."},
{"role": "user", "content": payload},
],
)
return response.choices[0].message.content
def synthesize_comparison(image_results, transcript, video_description):
for model in ["llama-3.3-70b", "deepseek-v3.2"]:
result = evaluate_with_model(model, image_results, transcript, video_description)
print(f"\n=== {model} evaluation ===\n{result}")Run it
Wire the functions together and run the full pipeline against your own media files.
if __name__ == "__main__":
image_results = compare_vision("photo.jpg", "List the visible objects.")
transcript = transcribe_audio("voice_note.wav")
video_desc = describe_video_frame("clip.mp4")
synthesize_comparison(image_results, transcript, video_desc)Example output:
=== gemma-3-27b ===
- Red bicycle leaning against wall
- Concrete floor with oil stains
- Single window, partially open
=== kimi-k2.6 ===
- Bicycle with red frame
- Garage floor, stained concrete
- Window half open, daylight visible
Transcription: "Hey, can you pick up the dry cleaning after the meeting?"
=== qwen-3-32b ===
- Person reaching for door handle
- Office hallway background
- Blurred motion on left side
=== llama-3.3-70b evaluation ===
Kimi K2.6 provided the most accurate vision summary. Gemma 3 27B was close but less specific. The transcription captured the speaker intent.
=== deepseek-v3.2 evaluation ===
Gemma 3 27B wins on conciseness. Kimi K2.6 wins on detail. Audio is accurate. Video frame correctly infers motion.Wrap-up
Feed the harness a directory of images and use the results to build a cost projection against your current token-based provider. You can also swap in Oxlo.ai's image generation endpoints to create synthetic test data for the vision pipeline. Because Oxlo.ai charges one flat rate per request, adding extra images or long transcripts to a single call does not inflate the bill, which makes large-scale evaluation predictable.
Top comments (0)