We are going to build a command-line video analyzer that samples frames from an MP4 file, feeds them to a vision-capable LLM, and emits a structured JSON report describing events, objects, and actions. This kind of tool is useful for content moderation, automated scene logging, or turning raw footage into searchable documentation. Because Oxlo.ai charges a flat rate per request regardless of input length, sending a dozen frames in one shot costs the same as a single text prompt, which makes long-context vision workloads far more predictable than token-based billing.
What you'll need
- Python 3.10 or newer
pip install openai opencv-python- An Oxlo.ai API key from https://portal.oxlo.ai
- A sample video file named
sample.mp4in your working directory
Step 1: Extract frames from the video
I use OpenCV to read the video and sample one frame every few seconds. To keep the payload manageable, I resize and JPEG encode each frame to base64 before sending anything to the API.
import cv2
import base64
def extract_frames(video_path, interval_sec=5, max_frames=12):
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
frame_interval = int(fps * interval_sec)
frames = []
count = 0
success, frame = cap.read()
while success and len(frames) < max_frames:
if count % frame_interval == 0:
resized = cv2.resize(frame, (512, 384))
_, buf = cv2.imencode(".jpg", resized, [int(cv2.IMWRITE_JPEG_QUALITY), 75])
b64 = base64.b64encode(buf).decode("utf-8")
frames.append(b64)
success, frame = cap.read()
count += 1
cap.release()
return frames
frames = extract_frames("sample.mp4")
print(f"Extracted {len(frames)} frames")Step 2: Write the system prompt
The system prompt is the agent's instruction set. I treat it as a strict contract so the model returns predictable, parseable output every time.
SYSTEM_PROMPT = """You are a video analysis engine. You receive a sequence of JPEG frames sampled from a single video.
Your job is to return a structured JSON report with exactly these top-level keys:
- summary: one sentence describing the overall video
- objects: list of distinct physical objects visible across frames
- actions: list of activities or events that appear to occur
- setting: inferred location or environment
- notable_details: any text, logos, safety concerns, or anomalies
Rules:
1. Output valid JSON only. No markdown fences, no commentary.
2. Infer motion and temporal context from the frame sequence.
3. If you are uncertain about a detail, omit it."""Step 3: Build the multimodal message
The OpenAI SDK accepts an array of content blocks for multimodal input. I prepend a short text instruction, then append each base64 frame as a data URL with low detail to reduce payload size.
def build_message(frames):
content = [{"type": "text", "text": f"Analyze these {len(frames)} frames from the video and return the structured report."}]
for b64 in frames:
content.append({
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{b64}", "detail": "low"}
})
return content
user_message = build_message(frames)Step 4: Call Oxlo.ai
Now I send the entire batch to Oxlo.ai. I use kimi-k2.6 because it handles vision and offers a 131K context window, which easily covers a long sequence of frames in a single request. Because Oxlo.ai uses flat per-request pricing, detailed breakdowns are available on the pricing page.
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"}
)
report = json.loads(response.choices[0].message.content)
print(json.dumps(report, indent=2))Step 5: Wrap it in a CLI
I package the pipeline into a single script with argument parsing so I can reuse it across folders of footage without editing paths.
import argparse
import cv2
import base64
import json
from openai import OpenAI
SYSTEM_PROMPT = """You are a video analysis engine. You receive a sequence of JPEG frames sampled from a single video.
Your job is to return a structured JSON report with exactly these top-level keys:
- summary: one sentence describing the overall video
- objects: list of distinct physical objects visible across frames
- actions: list of activities or events that appear to occur
- setting: inferred location or environment
- notable_details: any text, logos, safety concerns, or anomalies
Rules:
1. Output valid JSON only. No markdown fences, no commentary.
2. Infer motion and temporal context from the frame sequence.
3. If you are uncertain about a detail, omit it."""
def extract_frames(video_path, interval_sec, max_frames):
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
frame_interval = int(fps * interval_sec)
frames = []
count = 0
success, frame = cap.read()
while success and len(frames) < max_frames:
if count % frame_interval == 0:
resized = cv2.resize(frame, (512, 384))
_, buf = cv2.imencode(".jpg", resized, [int(cv2.IMWRITE_JPEG_QUALITY), 75])
frames.append(base64.b64encode(buf).decode("utf-8"))
success, frame = cap.read()
count += 1
cap.release()
return frames
def analyze(video_path, interval_sec, max_frames):
frames = extract_frames(video_path, interval_sec, max_frames)
content = [{"type": "text", "text": f"Analyze these {len(frames)} frames and return the structured report."}]
for b64 in frames:
content.append({
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{b64}", "detail": "low"}
})
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": content},
],
response_format={"type": "json_object"}
)
return json.loads(response.choices[0].message.content)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Analyze a video with an LLM")
parser.add_argument("video", help="Path to .mp4 file")
parser.add_argument("--interval", type=int, default=5, help="Seconds between sampled frames")
parser.add_argument("--max-frames", type=int, default=12, help="Maximum frames to send")
args = parser.parse_args()
result = analyze(args.video, args.interval, args.max_frames)
print(json.dumps(result, indent=2))Run it
Save the script as video_analyzer.py, place a sample video in the same directory, and run the following command.
python video_analyzer.py sample.mp4 --interval 5 --max-frames 10Example output:
{
"summary": "A person walks through an office space, stops at a whiteboard, and writes notes while a laptop sits open on a nearby desk.",
"objects": [
"whiteboard",
"laptop",
"office chair",
"backpack"
],
"actions": [
"walking",
"writing on whiteboard",
"sitting down"
],
"setting": "modern office",
"notable_details": "Whiteboard contains the text 'Q3 Roadmap'. No safety hazards detected."
}Next steps
Two directions to take this next. First, add an embedding step. Pass the generated JSON summary to Oxlo.ai's embeddings endpoint using bge-large or e5-large, then store the vectors in a database so you can search across a library of videos by semantic meaning. Second, build a batch processor. Use the Free tier's 60 requests per day or a paid plan to recursively analyze a directory of footage overnight, writing one JSON report per file.
Top comments (0)