Liquid AI shipped six open-weight models in the LFM3 family. Every one of them is derived from the same LFM3-1.7B base: a vision model, an omni model, an audio model, a text model, and their quantized variants. The launch coverage was benchmarks, which is the least interesting part of any model release.
So I ignored the benchmarks and asked the only question that matters: what can I build with this that I could not build last month?
The answer I landed on is Clip Detective. Drop in a video, ask a question in plain English, get an answer with timestamps that you can click. It runs entirely on my laptop. No API key, no upload, no per-second billing.
Q: What error is on screen, and at what second does it appear?
A: The error shown on screen is "ERROR 403 Forbidden from billing API" and it
appears at t=04.50s [00:04]
12 frames · 24.9s · 33 tokens · 0 cloud calls
That is a 1.9B parameter model on an M2 Mac mini with 8GB of RAM. The whole stack uses 1.2GB resident and 1.2GB of model weights on disk.
Here is how it works, what broke, and the numbers.
Why video Q&A is the right demo for a 2B vision model
A 2B model is not going to beat a frontier model at describing a photograph. If your demo is "look at this image and tell me what it is", you are competing on a benchmark you will
lose.
Video is different, because video turns a static model into a temporal one, and
temporal answers are not something people expect from a small local model. "What error
appeared, and at what second" is a question with a checkable answer. So is "which frame
shows the button change color". Both are real work, both are worth money, and both are
gated behind cloud APIs today.
There are three reasons video is the right target:
- The value is verifiable. A timestamp is either right or wrong. You can check it by scrubbing. No vibes.
- The constraint is real. Nobody expects a 2B model to watch 30 minutes of footage. Expectations are already calibrated, so a good sampling layer over delivers.
- It is offline by nature. Video files are large and often private. "It never leaves your machine" is not a marketing line here, it is the reason the tool is usable for screen recordings of internal systems.
The core insight: the model does not see video
LFM3-VL-2B takes images. llama.cpp added LFM3 text and vision support in
PR #31252. Video is not in the
multimodal path.
So the interesting engineering is not in the model call. It is in deciding which frames
the model gets to see, and how those frames are labeled so the answer comes back in the
video's own timeline.
video ──ffmpeg scene detect──▶ ≤12 frames @ real timestamps
│
labeled "Frame at 00:07"
▼
LFM3-VL-2B (Q4_K_M + Q8 mmproj)
▼
answer with mm:ss citations → click to seek
Three processes, all on localhost:
┌────────────────────────────────────────────────────────────────────────────┐
│ browser :8787 │
│ ┌──────────────┐ upload raw bytes ┌───────────────────────────────┐ │
│ │ drop zone │ ────────────────────▶ │ server.py │ │
│ │ frame strip │ │ ThreadingHTTPServer │ │
│ │ ask box │ ◀──────────────────── │ no framework, stdlib only │ │
│ └──────────────┘ answer + timestamps └───────────┬───────────────────┘ │
└─────────────────────────────────────────────────────┼──────────────────────┘
│ in-process call
┌───────────▼───────────────────┐
│ sampler.py │
│ ffmpeg scene detect │
│ uniform fallback, dedupe │
│ ≤12 frames @ 896px │
└───────────┬───────────────────┘
│ data/<id>/*.jpg
│ base64 in JSON
┌───────────▼───────────────────┐
│ llama-server :9911 │
│ LFM3-VL-2B Q4_K_M │
│ + mmproj Q8_0, Metal GPU │
│ OpenAI-compatible /v1 │
└───────────────────────────────┘
Sampling, which is where the quality lives
My first version picked frames at even intervals. It was fine for anything with constant
motion and useless for screen recordings, where the interesting thing is a single state
change in the middle of 40 static seconds.
Scene detection solves that, because ffmpeg already has it and it costs one pass:
def scene_frames(path: str, outdir: str, threshold: float):
"""Extract scene-change frames, return [(timestamp, filename)]."""
cmd = [FFMPEG, "-v", "info", "-i", path, "-an",
"-vf", f"select='gt(scene,{threshold})',showinfo,scale={FRAME_WIDTH}:-2",
"-vsync", "vfr", "-q:v", "3", os.path.join(outdir, "scene_%04d.jpg")]
proc = subprocess.run(cmd, capture_output=True, text=True)
stamps = [float(m) for m in re.findall(r"pts_time:([0-9.]+)", proc.stderr)]
files = sorted(f for f in os.listdir(outdir) if f.startswith("scene_"))
return list(zip(stamps, files))
showinfo prints each selected frame's presentation timestamp to stderr. Parse it with a
regex and you get exact frame times, for free, with no second pass and no OpenCV.
The threshold needs a fallback, though, and this bit me. A talking-head clip has almost no
scene changes. At threshold 0.30 I got two frames and a model that had effectively seen
nothing. So the sampler walks the threshold down, then fills the timeline uniformly if it
still has too few:
for thresh in (0.30, 0.18, 0.10):
frames = scene_frames(path, outdir, thresh)
if len(frames) >= 4:
break
if len(frames) < 4:
frames += uniform_frames(path, outdir, dur, MAX_FRAMES, 0)
Then a de-dupe pass, because my first UI showed the same second twice and it looked like a
bug even to me:
def pick(frames, max_n):
"""Keep frames spread across the timeline, always keeping first and last.
Dedupes by whole-second so the strip never shows the same moment twice."""
seen_sec, uniq = set(), []
for t, f in sorted(frames, key=lambda x: x[0]):
if int(t) in seen_sec:
continue
seen_sec.add(int(t))
uniq.append((t, f))
if len(uniq) <= max_n:
return uniq
step = len(uniq) / max_n
return [uniq[int(i * step)] for i in range(max_n)]
Resolution is not a detail, it is the feature
12 frames at 448px was my starting point, because smaller images mean fewer prompt tokens
and a faster answer. Then I tested it on a real screen recording of Hacker News and it read
the top headline as "Git for the US US".
The actual headline was "Git for the US Code".
I raised the frame width to 672. Still wrong: "US US". At 896 it read "Git for the US
Code", exactly, and kept reading it correctly across repeated runs.
MAX_FRAMES = 12 # 12 frames is where answers stop getting better on 8GB
FRAME_WIDTH = 896 # below this, small UI text misreads
The cost is real: about 9 seconds per question at 672px, about 16 at 896px. I paid it
without hesitating. A fast wrong answer is worth nothing, and OCR on UI text is half the
reason to run this on a screen recording in the first place.
This is the general lesson for anyone building vision products on small models: the
token cost of resolution is invisible in a benchmark and decisive in a demo. Benchmarks
resize to 224 and call it a day.
Labeling frames so citations are possible
The model cannot cite what it cannot identify. So every frame goes in preceded by its own
timestamp line, and the system prompt pins the output format:
content = [{
"type": "text",
"text": (f"Video: {meta['filename']}, {mmss(meta['duration'])} long, "
f"{len(frames)} frames sampled.\n\nQuestion: {question}")
}]
for fr in frames:
with open(os.path.join(DATA, meta["id"], fr["f"]), "rb") as fh:
b64 = base64.b64encode(fh.read()).decode()
content.append({"type": "text", "text": f"Frame at {mmss(fr['t'])}:"})
content.append({"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{b64}"}})
SYSTEM = (
"You are Clip Detective. You are given frames sampled from ONE video, each labeled "
"with its timestamp. The user asks a question about the video. Answer only from "
"what is visible in the frames. Be specific and short: state the answer, then cite "
"the frame timestamp in the form [mm:ss]. If the frames do not show it, say what "
"you would need instead of guessing."
)
That last sentence is the highest value line in the codebase. It converts a model that
wants to be helpful into one that is willing to say "not visible". On a clip of Big Buck
Bunny's opening shot, which contains a mossy mound and no characters, it answered that no
characters were visible and that a wider shot would be needed. A model that admits absence
is worth more than one that invents a rabbit.
The reasoning-model trap that cost me two hours
First working version returned this:
{"answer": "", "seconds": 18.1, "tokens": 400, "frames": 14}
Empty answer, 18 seconds, 400 tokens burned. I assumed the chat template was wrong, then
that the image encoding was wrong. It was neither. LFM3-VL-2B is a reasoning model, and it
had spent the entire 400 token budget inside reasoning_content, so content came back
empty.
Two fixes, both cheap:
# run.sh
--reasoning off
# server.py, belt and braces for any reasoning model behind this endpoint
text = (msg.get("content") or "").strip()
if not text:
think = (msg.get("reasoning_content") or "").strip()
text = think.split("\n\n")[-1] if think else ""
max_tokens went from 400 to 700 too. If you take one thing from this post for your own
work: when a local model returns empty content, check the token budget and the thinking
mode before you touch anything else. Both are one-line fixes and both look like model bugs.
Metal shaders, or why the Homebrew build does not work
brew install llama.cpp gives you a build that predates LFM3. You get:
unknown model architecture: lfm3
So I cloned master and built it. The build succeeded and the server still died, this time
inside the Metal compiler:
error: constexpr function never produces a constant expression
inline constexpr bfloat4 ggml_metal_unary_cast<bfloat4, float4>(...)
The cause is a constexpr on two bf16 vector cast specializations that the Metal compiler
refuses to fold. Removing constexpr from both makes it compile, and the model runs on
GPU at 39.8 tokens per second instead of falling back to CPU at single digits.
That fix is now automated, because anyone else building this on a Mac will hit exactly the
same wall:
python3 scripts/fix-metal-bf16.py ~/llama.cpp-src
It is idempotent and it does nothing if upstream has already fixed it, which is the
correct shape for a patch like this. Never ship a manual build step.
Making the timestamp clickable
An answer that says [00:04] in grey text is decoration. An answer where clicking
00:04 seeks the player and resumes playback is a tool. The whole interaction is twenty
lines of vanilla JavaScript, no framework:
function linkTimestamps(s) {
return s.replace(/\b(\d{1,2}):(\d{2})\b/g, (m, mm, ss) =>
`<span class="ts" data-t="${+mm * 60 + +ss}">${m}</span>`);
}
ad.querySelectorAll('.ts').forEach(el => {
el.onclick = () => { vid.currentTime = +el.dataset.t; vid.play(); };
});
The regex is deliberately loose so it catches timestamps the model writes mid-sentence,
not just the ones in brackets. The first version only matched [mm:ss] and half the
citations were dead.
No dependencies, on purpose
server.py is 166 lines of http.server. sampler.py is 103 lines of subprocess.
The UI is one HTML file with no build step. Total runtime dependencies: ffmpeg and
llama.cpp.
Why bother, in 2026, when fastapi exists?
Because the target is a zip that someone else runs on their own machine. Every dependency
is a chance for their Python to be wrong, their pip to be locked down, or their company
laptop to refuse an install. Standard library only means ./setup.sh has to worry about
exactly two external things, ffmpeg and a C++ build, both of which are already required.
The upload path is a good example. The browser posts the file as a raw request body with
the name in a header, and the server streams it to disk in 1MB chunks:
with open(src, "wb") as fh:
remaining = n
while remaining > 0:
chunk = self.rfile.read(min(1 << 20, remaining))
if not chunk:
break
fh.write(chunk)
remaining -= len(chunk)
No multipart parser, no werkzeug. It is twenty lines and it has never failed.
Numbers on real hardware
Apple M2 Mac mini, 8GB unified memory, macOS 15.5.
| Metric | Value |
|---|---|
| Generation | 39.8 tok/s |
| Prompt ingestion | 407 tok/s |
| Answer, scene questions (672px) | ~9s |
| Answer, OCR questions (896px) | ~16s |
| Model resident | ~1.2GB |
| Cloud calls | 0 |
Three test clips, each with ground truth I checked by hand:
| clip | question | answer | verdict |
|---|---|---|---|
| synthetic billing console, 12s | "what error, and when?" | ERROR 403 Forbidden from billing API [00:04] |
correct, the 403 sits at 4.0s |
| live Hacker News recording, 11s | "top headline?" | Git for the US Code [00:00] |
correct at 896px |
| Big Buck Bunny, 10s | "name characters and action" | "mossy mound, burrow, no characters visible" | correct, and admits absence |
I generated the first clip myself with PIL and ffmpeg specifically so I would know the
answer. If you build vision demos, make your own ground truth. Real footage is great for
vibes and useless for verification.
What is still missing
The honest list:
- No audio. LFM3-Audio-2B is a sibling model on the same base. Transcript plus frames is the obvious v2 and it is not built.
- No cross-video reasoning. One clip per question. "Which of these 40 recordings shows the regression" is a different, harder product.
- Long videos degrade. Twelve frames from a 40 minute lecture is a summary, not a search.
- Frame granularity. A 300ms flash between two sampled frames does not exist to the model, and no amount of prompting fixes that.
If you want to build on it
The repo is MIT, the whole thing is about 500 lines, and the seams are obvious:
-
sampler.pydecides what the model sees. Changescene_frames,pick,MAX_FRAMES,FRAME_WIDTH. -
server.pydecides how it is asked. ChangeSYSTEM,ask(). -
run.shdecides which model and quant. Swap in LFM3-Omni or a bigger quant if you have the RAM. -
static/index.htmldecides how it feels. One file, no build.
The features I would most like to see someone else build: a transcript lane for audio, a
two-pass sampler that finds the region of interest at low resolution then re-samples it
high, and an agent loop where the model can request "show me 00:07 again, closer". That
last one is two round trips and would beat any single large prompt.
unzip clip-detective.zip && cd clip-detective
./setup.sh
./run.sh
open http://127.0.0.1:8787
Code & more: https://www.dailybuild.xyz/project/303-clip-detective
Top comments (0)