Unlocking Video Analysis with LLMs

Video generates more data per minute than any other modality, yet most of it sits unanalyzed. Large language models with vision capabilities have changed this. By sampling key frames, transcribing audio, and feeding both into a multimodal LLM, developers can extract structured insights from hours of footage without building custom computer vision pipelines. The catch is cost. A single analysis job can easily involve tens of high-resolution images and lengthy transcripts, which balloons token counts on traditional inference providers. Oxlo.ai solves this with request-based pricing: one flat cost per API call regardless of how many frames or tokens you pack inside.

The Architecture of LLM-Powered Video Analysis

A production video analysis pipeline usually has three stages. First, you extract representative frames from the video, either at fixed intervals or via scene-change detection. Second, you extract audio and transcribe it into text. Third, you feed the frames and transcript into a multimodal LLM and ask for structured analysis.

Each stage benefits from long context. A 10-minute video might yield 60 to 120 sample frames plus a 1,500-word transcript. Feeding all of that into a single prompt lets the model understand temporal continuity, repeating objects, and narrative flow. Splitting the job into tiny chunks forces you to reassemble context later, which adds latency and complexity.

Why Long Context Changes the Game

Context length is the hidden bottleneck in video analysis. If your provider caps context at 8K or 32K tokens, you must aggressively downsample frames or truncate transcripts. That sacrifices accuracy.

Oxlo.ai hosts models that remove this ceiling. DeepSeek V4 Flash supports a 1 million token context window, and Kimi K2.6 offers 131K tokens with advanced reasoning and vision. That means you can pass dozens of 512x512 frames and a full Whisper transcript in a single request without hitting a limit. The model sees the entire video narrative at once, and you avoid the engineering overhead of chunking and merging.

Code Walkthrough: Analyzing a Video with Oxlo.ai

Below is a minimal Python pipeline. It extracts one frame per second, transcribes audio through Oxlo.ai's audio/transcriptions endpoint, and sends everything to a vision-capable chat model. The SDK is fully OpenAI compatible, so you only need to change the base URL and API key.

import base64, os, subprocess
from openai import OpenAI

# Oxlo.ai client: drop-in replacement for OpenAI
client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

VISION_MODEL = "gemma-3-27b-it"   # vision-capable model
AUDIO_MODEL = "whisper-large-v3" # audio transcription

video_path = "interview.mp4"
frame_dir = "frames"
os.makedirs(frame_dir, exist_ok=True)

# 1. Extract one frame per second with ffmpeg
subprocess.run([
    "ffmpeg", "-i", video_path,
    "-vf", "fps=1,scale=512:-1",
    f"{frame_dir}/frame_%04d.jpg"
], check=True)

# 2. Collect frames as base64 data URLs
frames = []
for fname in sorted(os.listdir(frame_dir))[:60]:  # cap at 60 frames for demo
    with open(os.path.join(frame_dir, fname), "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
        frames.append({
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{b64}"}
        })

# 3. Transcribe audio via Oxlo.ai Whisper
with open(video_path, "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model=AUDIO_MODEL,
        file=audio_file,
        response_format="text"
    )

# 4. Multimodal analysis
messages = [
    {
        "role": "system",
        "content": "You are a video analysis engine. Return JSON with keys: summary, key_moments, sentiment."
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": f"Audio transcript:\n{transcript}\n\nAnalyze the video frames and transcript."},
            *frames
        ]
    }
]

response = client.chat.completions.create(
    model=VISION_MODEL,
    messages=messages,
    response_format={"type": "json_object"}
)

print(response.choices[0].message.content)

This script sends up to 60 frames plus a transcript in one request. Because Oxlo.ai charges per request, not per image or per token, the cost is the same whether you send 5 frames or 60.

Keeping Costs Predictable with Request-Based Pricing

Token-based billing punishes video workloads. A 512x512 image can consume hundreds or thousands of vision tokens, and a long transcript adds thousands more. If you analyze hundreds of videos per day, costs scale linearly with content length and resolution.

Oxlo.ai uses request-based pricing. Every API call carries one flat fee, no matter how many frames, tokens, or seconds of audio you include. For long-context video analysis, this can be 10 to 100 times cheaper than token-based alternatives. You can upgrade frame rates, raise resolution, or add full transcripts without watching a meter run. See the exact tiers on the Oxlo.ai pricing page.

During development, the free tier includes 60 requests per day and access to 16+ models, which is enough to prototype a full pipeline before committing to a paid plan.

Model Selection on Oxlo.ai

Oxlo.ai offers 45+ models across seven categories, all accessible through the same OpenAI-compatible endpoint.

For vision, use Gemma 3 27B or Kimi VL A3B. Both accept image inputs through the chat/completions endpoint.

For reasoning and synthesis, route the combined transcript and frame analysis through Kimi K2.6, DeepSeek V4 Flash, or Llama 3.3 70B. DeepSeek V4 Flash is especially useful when you need near state-of-the-art reasoning across a 1 million token context window.

For audio, Whisper Large v3, Whisper Turbo, and Whisper Medium handle transcription and diarization. You can call them with the same client instance, just switching the endpoint to audio/transcriptions.

There are no cold starts on popular models, so latency stays consistent even when you alternate between transcription and vision calls.

Production Tips for Reliable Pipelines

Sampling strategy matters more than model choice. A frame every second is usually overkill for slow-moving content; scene-change detection or one frame every 5 seconds cuts token volume dramatically. Downscale images to 512 pixels on the longest edge before base64 encoding. You keep visual fidelity while reducing payload size and latency.

Use JSON mode or function calling to force structured output. If you need multi-turn verification, store the first analysis in conversation history and ask a follow-up question. Oxlo.ai supports multi-turn conversations and streaming, so you can build agentic review loops where one model extracts events and another validates them.

If you run agentic workloads that chain transcription, vision, and tool use into a single logical job, request-based pricing keeps the cost flat across the entire chain. You are not penalized for long system prompts or extensive tool definitions.

Video analysis with LLMs does not require a bespoke infrastructure stack. With standard tools like ffmpeg, the OpenAI SDK, and Oxlo.ai's inference platform, you can build a production pipeline in an afternoon. The combination of long-context vision models, Whisper transcription, and flat per-request pricing removes the traditional cost barrier, making it practical to analyze video at scale.