How to extract keyframes from a video for GPT and Claude vision

Kanto Labs — Fri, 30 Oct 2026 23:28:40 +0000

GPT, Claude and Gemini models are good at reading images, but most API endpoints do not take a video file. If you want a model to describe a product demo, tag a stock clip, check an ad, or summarise a screen recording, you first have to turn the video into a handful of well-chosen frames with timestamps. This guide shows how to do that in about 20 lines of Python, without installing ffmpeg or OpenCV, and then send the frames to an OpenAI model or Claude.

Disclosure: we (Kanto Labs) built the Apify actor used below. Everything here also works with your own ffmpeg script - the "how many frames, which frames" advice applies either way.

Why not just send every frame?

A 10-minute video at 30 fps has 18,000 frames. Vision models bill per image (or per image tile), so sending them all is slow and expensive, and most of them are near-duplicates. What you want is:

Scene-change keyframes for edited videos (ads, trailers, tutorials with cuts): one frame per shot tells the story.
Evenly spaced frames for one-take footage (talking heads, webinars, CCTV): one frame every N seconds.
No black frames. Fades and intros produce black or single-colour frames that waste a slot and confuse the model.
Timestamps on every frame, so the model can say when something happens and you can jump to it.

8-20 frames at around 1024 px wide is enough for most "what happens in this video" questions.

Step 1 - Extract the keyframes

Video Frame Extractor is a pay-per-frame API on Apify ($0.002 per frame; failed downloads and skipped black frames are free). It takes direct video file links (MP4, MOV, WebM, MKV... from S3, R2, a CDN, Dropbox ?dl=1), and returns one row per video with a public image URL and a timestamp for every frame. It does not download from YouTube or TikTok pages.

Install the client and set your token (free Apify account -> Settings -> API & Integrations):

pip install "apify-client>=3" openai anthropic
export APIFY_TOKEN=...      # Windows PowerShell: $env:APIFY_TOKEN="..."

import os
from apify_client import ApifyClient

apify = ApifyClient(os.environ["APIFY_TOKEN"])

run = apify.actor("kantolabs/video-frame-extractor").call(run_input={
    "videoUrls": ["https://media.w3.org/2010/05/sintel/trailer.mp4"],
    "mode": "auto",          # first frame + strongest scene changes, topped up with even spacing
    "frameCount": 12,        # frames per video
    "maxWidth": 1024,        # plenty for vision models, keeps image tokens down
})

video = next(apify.dataset(run.default_dataset_id).iterate_items())
print(video["durationSeconds"], "s,", video["frameCount"], "frames")
for f in video["frames"]:
    print(f["timecode"], f["sceneScore"], f["url"])

For the 52-second Sintel trailer this returns 12 frames (00:00:01.500, 00:00:02.175 ... 00:00:41.333) - the black opening frame is stepped past automatically. sceneScore is set when the frame is a detected cut. The run costs 12 x $0.002 = $0.024.

Pick the mode by the kind of video:

Video	Input	Frames you get
Edited (ads, trailers, tutorials with cuts)	`"mode": "scenes"`, `"maxFramesPerVideo": 20`	one frame per real cut
One take (webinar, talking head, screen recording)	`"mode": "interval"`, `"intervalSeconds": 10`	one frame every 10 s
Anything, fixed budget	`"mode": "count"`, `"frameCount": 8`	exactly 8 evenly spaced frames
Need a thumbnail	`"mode": "bestThumbnails"`, `"thumbnailCount": 3`	the 3 best-looking frames, ranked

Cost check for interval: a 10-minute video at one frame every 10 s is 60 frames = $0.12 (raise maxFramesPerVideo, which is 50 by default).

Each row also has a contactSheetUrl: one grid image of all frames labelled with their timecodes. For a quick summary, that single image is often enough and costs one image's worth of tokens.

Step 2a - Send the frames to an OpenAI model

The frame URLs are public HTTPS links, so the model can fetch them directly. Interleave a short text label with each image so the model knows the order and the time. This uses the Responses API:

from openai import OpenAI

content = [{"type": "input_text", "text":
            "These are keyframes of one video, in time order, each labelled with its timestamp. "
            "Describe what happens, scene by scene, and cite the timestamps."}]
for f in video["frames"]:
    content.append({"type": "input_text", "text": f"Frame at {f['timecode']}:"})
    content.append({"type": "input_image", "image_url": f["url"], "detail": "low"})

response = OpenAI().responses.create(      # reads OPENAI_API_KEY
    model="gpt-6-astra",                   # any current vision model works; cheaper ones are fine for this
    input=[{"role": "user", "content": content}],
)
print(response.output_text)

"detail": "low" keeps each frame at a small token cost; switch to "high" when the model needs to read small on-screen text. (If you are still on GPT-4o and Chat Completions, the same idea works with image_url parts - but OpenAI has scheduled the GPT-4o snapshots for shutdown in January 2027, so new code should use a current model.)

Step 2b - Send the frames to Claude

Claude's Messages API takes images by URL as well. The structure is the same: label, image, label, image.

import anthropic

content = []
for f in video["frames"]:
    content.append({"type": "text", "text": f"Frame at {f['timecode']}:"})
    content.append({"type": "image", "source": {"type": "url", "url": f["url"]}})
content.append({"type": "text", "text":
                "These are keyframes of one video in time order. Describe what happens, "
                "scene by scene, and cite the timestamps."})

client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=4000,
    messages=[{"role": "user", "content": content}],
)
if response.stop_reason == "refusal":
    print("The model declined this request.")
else:
    print("".join(b.text for b in response.content if b.type == "text"))

Gemini works too: download each frame's bytes (requests.get(f["url"]).content) and pass them as inline image/jpeg parts, or pass the single contact sheet.

Step 3 - Do it for many videos

videoUrls takes a list, and the actor returns one row per video, so a batch is the same call:

from decimal import Decimal

run = apify.actor("kantolabs/video-frame-extractor").call(run_input={
    "videoUrls": my_list_of_video_urls,
    "mode": "count",
    "frameCount": 8,
}, max_total_charge_usd=Decimal("2.00"))   # hard spending cap for this run

for video in apify.dataset(run.default_dataset_id).iterate_items():
    if video.get("error"):
        print("skipped", video["videoUrl"], "-", video["error"])   # free, with a plain-English reason
        continue
    ...  # send video["frames"] to your model as above

100 videos x 8 frames = 800 frames = $1.60. With max_total_charge_usd the actor stops extracting once the budget is reached instead of overspending. Failed videos come back as rows with an error message and are not charged.

Tips that save tokens and mistakes

Fewer, better frames beat more frames. Start with 8-12. Add frames only when the model misses something.
Keep the timestamps in the prompt. Models will happily quote them back, which makes the answer checkable.
Use scene mode for edited content. Evenly spaced frames on a fast-cut ad land between cuts and miss shots.
Ask for structure. "Return JSON with timestamp, description, onScreenText" turns the answer into data.
Need the audio too? Frames carry no speech. Video to LLM Context returns a keyframe and the Whisper transcript per scene in one text block, which is better for talks and tutorials.
Need exact cut timecodes? Video Scene Detection gives every cut, dissolve and fade as a CSV/EDL shot list.

No-code version

The same pipeline works in n8n, Make or Zapier: run the actor with the Apify node or an HTTP request to https://api.apify.com/v2/acts/kantolabs~video-frame-extractor/run-sync-get-dataset-items, split the frames array into items, and send each url to an OpenAI "Analyze image" step (or an Anthropic node).

Summary

Turn the video into 8-20 timestamped keyframes (scene changes for edited videos, even spacing for one-take footage).
Send them to the model as labelled images by URL, in time order.
Ask for timestamps and structure in the answer.

Store page and full input reference: https://apify.com/kantolabs/video-frame-extractor

DEV Community: Kanto Labs