What I Built
When an AI coding agent or browser-use sandbox fails in CI at 2:00 AM, everyone flies blind. The agent acts on a headless display without visual feedback, the test runner spits out a 400-line stack trace, and the human reviewer gets a single static PNG that doesn't show which action triggered the regression.
If you try to fix this by feeding raw screen recordings into a Vision-Language Model (VLM), you blow your token budget in under a minute. If you hack together a bash script around Xvfb and ffmpeg, a sudden exception sends SIGKILL, corrupts the MP4 moov atom, and leaves you with zero artifacts when you need them most.
Dynamo is a headless, zero-dependency Python visual and video harness for coding agents, Playwright browser automation, and computer-use sandboxes. In a single with VisualHarness(): context manager (or a single CLI command), Dynamo:
-
Manages the headless X11 lifecycle (
Xvfb) with a socket readiness probe so your browser never races the display server. -
Streams non-blocking screen capture (
ffmpeg -f x11grab) with gracefulSIGINTfinalization β even when your agent crashes mid-run. -
Logs timestamped action telemetry (
telemetry.jsonl) and burns visual click-rings and crosshairs directly into MP4 and looping PR GIFs. -
Answers visual questions with model-free arithmetic β cropping before encoding, pixel OCR on
<canvas>elements, visual diffs, and Set-of-Marks element grounding. - Keeps only the frames that matter β scene changes anchored to exact action timestamps β instead of paying for a thousand-odd near-identical frames.
-
Routes surviving keyframes through local open-weight models (
qwen3-vl:4bunder Apache-2.0 andlayafor lightweight typed filtering) so proprietary screens never leave your infrastructure.
Above: A live headless Chromium session recorded inside Dynamo's virtual display (:99), compiled into a two-pass palette GIF with telemetry click rings and action timestamps baked in.
π¬ Video Demo & Visual Showcase
Here is the full product walkthrough demonstrating Dynamo's architecture, crash safety, live autonomous browser execution, and model-free primitives:
πΊ Watch on YouTube: https://youtu.be/zAtmsYpyuMY
Chapters:0:00The Agent Blind Spot β’0:30The Fatalmoov atomCrash β’1:04One Context Manager SDK β’1:37Live Headed Chromium Checkout β’2:11Four Model-Free Primitives β’2:54Measured Hackathon Benchmark β’3:28Local VLM Triage (82.6% Token Cut) β’3:58Production Ready & MIT License.
- GitHub Repository: https://github.com/0dvsh0/dynamo
- Interactive Demo & Token Calculator: https://0dvsh0.github.io/dynamo/
Who Would Actually Pay for This? (The Side Hustle)
Anyone running autonomous coding agents, QA browser fleets, or computer-use sandboxes in production hits two walls immediately: multimodal inference bills and security compliance.
Dynamo follows an open-core infrastructure side-hustle model built for engineering teams who need visual observability that their security team will actually sign off on:
- Who buys it: AI dev-shop agencies, QA automation teams running parallel Playwright workers on Kubernetes/EKS, and startups building computer-use agents that interact with sensitive staging environments, internal admin panels, or checkout flows.
- Why they pay: SaaS session-replay tools charge per-seat or per-minute and require shipping recordings of proprietary code and customer data to a third-party cloud. Dynamo is a self-hosted library with zero data egress and zero GPU requirement for capture.
- How it monetizes: The core Python SDK and CLI are open-source to drive bottom-up developer adoption. The paid side hustle offers custom sandbox integration, VLM token-budget tuning, and fleet-scale deployment packs (isolated display pools running hundreds of parallel headless sessions per node pool, plus audit-grade compliance artifacts) with direct engineering onboarding.
The Problem: Why DIY Agent Observation Fails
When an agent fails, hand-rolled shell scripts and naive time-based sampling fail in two catastrophic ways:
-
Silent Failure & Corrupted Files on Crash: Standard ffmpeg holds the MP4 container index in memory until a clean shutdown. When an unhandled exception or CI job timeout triggers
SIGKILL, the header is never written:
DIY script, agent killed mid-run : moov atom not found (0 bytes, 0 frames β unrecoverable)
Dynamo, agent killed : playable MP4, 4/4 logged actions (SIGINT exit path)
- The 38% Duplicate Token Disaster: Naive time-based sampling captures 1 frame per wall-clock second regardless of screen activity. In our benchmark, 19 of 50 consecutive frames were pixel-identical, burning 56,100 tokens on static screens while waiting on network I/O.
The Solution: One Context Manager
No daemons to babysit, no DISPLAY variables to export manually, and no ffmpeg flags to memorize:
from agent_visual_harness import VisualHarness
# 1. Enter context: Xvfb starts, readiness probe passes, ffmpeg begins recording
with VisualHarness(display=":99", resolution=(1920, 1080), record=True) as harness:
# 2. Run browser, agent process, or Playwright tests
harness.run_on_display(["python", "run_agent.py"])
# 3. Log actions into telemetry.jsonl
harness.log_action("click", x=500, y=300, target="#pay-button")
# 4. Instant screenshot bytes mid-run
screenshot_bytes = harness.take_screenshot()
# 5. Upon exit (even on exception!), review artifacts are exported cleanly:
artifacts = harness.export(
video_path="output/repro.mp4",
gif_path="output/demo.gif",
vlm_keyframes_dir="output/vlm_frames/"
)
print(f"Generated {len(artifacts.keyframes)} frames for VLM context.")
Or run any existing shell-driven agent or test suite directly from the CLI:
# Start a managed Xvfb display + recorder and run any command inside it
$ dynamo record --display :99 --resolution 1920x1080x24 --fps 24 \
-- python run_agent.py
# Compile PR-ready artifacts with click overlays baked in
$ dynamo export ./dynamo-session \
--mp4 output/repro.mp4 --crf 23 \
--gif output/demo.gif --max-colors 128 --dither bayer \
--overlay-clicks
# Produce a token-budgeted keyframe set + manifest.json for your local VLM
$ dynamo keyframes ./dynamo-session --scene 0.05 --max-size 1280 \
--format webp --out output/vlm_frames/
# β 29 keyframes Β· 1,043 frames scanned Β· manifest.json written
Live Run: Autonomous Headed Chromium Checkout
Inside demo/assets/dynamo_repro.mp4, a headless Chromium browser runs inside Dynamo's virtual display (:99), navigating a multi-step e-commerce storefront:
- 02.0s: Enters customer shipping address.
-
06.4s: Types postal code
"97"β Validator Trap: Storefront rejects invalid 2-digit postal code. -
11.6s: Agent inspects page, corrects postal code to
"97209", and clicks Continue. - 17.1s: Enters credit card details.
-
25.1s: Clicks Pay β Issuer Decline 1: Refused with
do_not_honor. -
31.7s: Clicks Pay β Issuer Decline 2: Refused with
do_not_honor. - 38.3s: Clicks Pay β Authorized! Charge captured on attempt 3 ($429.56).
-
42.0s: Order Confirmed (
#NW-9821) with click overlays timestamped into telemetry.
Four Model-Free Arithmetic Primitives
Recording gets an agent a replay. Four more primitives get it an answer: what error the page showed, which pixels moved, where the button is, and which coordinate space those pixels are in.
These four are pure arithmetic β no model involved, which is why they are fast, reproducible, and free:
# 1. Capture a specific region, not the whole screen (crops in ffmpeg before encoding)
$ dynamo capture --region 0,0,480,90 --out banner.png
# 2. Text off pixels β for canvas, VNC, RDP, and BIOS surfaces that give up no DOM text
$ dynamo ocr banner.png --find DO_NOT_HONOR --fuzzy # exits 1 if absent
# 3. Visual regression gate β exits 1 when the UI moved
$ dynamo diff before.png after.png --out diff.png
# 4. Number elements so a model answers "12" instead of guessing "(740, 318)"
$ dynamo som shot.png --boxes boxes.json --labels Pay,Cancel --out som.png --resolve 12
Visual Inspection Gallery
1. Region Snip vs. Full Viewport (20Γ Byte Cut)
Cropping before encoding shrinks a 92 KB full screenshot down to a 4 KB snip. Vision models pricing by image dimensions or bandwidth receive only the error banner:
| Region Snip (4 KB) | Full Viewport (92 KB) |
|---|---|
 |
 |
2. Visual Regression Diff Overlay
Highlights moving UI elements with a red-on-dimmed bounding box overlay. A control comparison with unchanged UI reports zero changed pixels:
3. Set-of-Marks Numbered Element Grounding
Numbers elements directly on the live framebuffer so VLMs answer "12" rather than guessing floating-point coordinates:
4. Pixel OCR on Non-DOM Surfaces
Extracts text directly from <canvas>, VNC, or RDP frames with fuzzy confusable matching (D0_ $\rightarrow$ DO_):
What the Benchmark Actually Shows
We evaluated the exact same checkout workflow captured three ways β plain per-action screenshots, DIY wall-clock ffmpeg, and Dynamo:
| Metric | A Β· Screenshots | B Β· DIY Wall-Clock | C Β· Dynamo |
|---|---|---|---|
| Images to the model | 24 | 51 | 29 (Action-anchored) |
| Token estimate (@1,100/img) | 26,400 | 56,100 | 31,900 |
| Bit-identical repeats | 1 | 19 | 1 (3.6% redundancy) |
| Mean frame-to-frame Ξ | 3.29 | 1.76 | 11.36 (High Information) |
| Total frame payload | 2.4 MB | 9.5 MB | 901 KB (2.6Γ Smaller!) |
| Survives torn-down CI job | β | moov atom not found |
YES (Playable MP4) |
| Artifacts generated | PNGs only | Unindexed raw | MP4 + GIF + WebP + Manifest |
The honest takeaways:
- Dynamo and screenshots cost similar tokens (31,900 vs 26,400) because both sample on actions rather than wall-clock time. Dynamo's win is everything screenshots cannot do: the video, the GIF, click overlays, timing manifest, crash resilience, and a 2.6Γ smaller payload using WebP.
- Time-driven sampling is the real token disaster. The DIY script kept 1 frame/sec and paid 2.1Γ tokens with 19 of 50 frames pixel-identical.
Intelligent VLM Triage: Estimated vs. Counted
For questions arithmetic cannot answer β what did the decline message say? β Dynamo routes frames through a 3-stage funnel:
Routing strategy (vision.py) |
Frames sent | Heuristic estimate | Counted by qwen3-vl:4b |
|---|---|---|---|
| A Β· every keyframe, whole | 29 | 22,707 tokens | 32,132 tokens |
| B Β· cropped to diff bounding box | 29 | 10,947 tokens | 32,040 tokens (99.7%) |
C Β· triaged: skip unchanged β laya filter β crop |
24 | 8,575 tokens | 26,528 tokens (82.6%) |
Key findings:
- Cropping saved almost nothing in model tokens (0.3%). Because local VLM servers resize incoming images to a fixed internal token grid, a full frame and a small crop cost the exact same tokens. The crop cuts wire bytes and points the model at what moved, but is not a token saving here.
-
Skipping frames via a cheap decision model (
laya) did the real work. In Mode C, a lightweight typed decision model (Laya) scored candidate frames and dropped 5 low-signal frames before they ever touched the vision encoder. Not sending the frame cut prompt tokens to 82.6% and reduced wall time from 255s to 83s. -
100% Local & Offline: Powered by Apache-2.0 models (
qwen3-vl:4bandlaya), running entirely on CPU/GPU without cloud API costs or data egress.
Prize Categories
Overall Winner (Side Hustle): Dynamo solves an expensive infrastructure problem for teams building autonomous AI agents, backed by transparent benchmarks, a zero-dependency Python SDK/CLI, model-free arithmetic inspection, and a clear B2B open-core monetization path.
Best Use of GitHub Copilot / GitHub Actions Integration: Built for native GitHub Actions CI workflows (
dynamo record -- pytest tests/e2e$\rightarrow$dynamo export --gif pr/demo.gif --overlay-clicks) so every pull request automatically receives a visual replay artifact.Best Use of Sentry Agent Tracing / Agent Observability Spirit: Delivers full visual + JSONL action telemetry (
telemetry.jsonl), four model-free inspection primitives, and side-by-side estimated vs. actual model token accounting.
Top comments (8)
The finding that cropping saved 0.3% of model tokens is the most useful line here. Most people assume the crop is the win, and you measured that the local VLM resizes to a fixed grid anyway, so skipping the frame is what actually pays.
One question on action-anchored keyframes: which frame do you keep for an action, the one at the click timestamp or the first one after the screen settles? The click frame usually still shows the old state, and something like the do_not_honor banner can land a second or two later behind a spinner. Do you wait for the scene-change score to go quiet after each logged action, or use a fixed offset? On real desktops we have found "wait until the region stops changing, with a timeout" catches far more of these than any fixed delay, but it costs time on screens with a blinking cursor or an animation.
Action Anchors (the "before/at" frame)
When h.log_action("click", x=500, y=300) is called, the frame at that exact
timestamp is pinned as an anchor so you never lose the initiating state.
Scene-Change Candidates (the "after/settle" frames)
ffmpeg's scene filter (e.g. gt(scene, 0.05)) runs across the stream and extracts
transition points where pixels actually move (e.g., when the spinner resolves).
Merging & Deduplication Window
Anchor frames and scene cuts are merged by timestamp. Consecutive frames within
the dedup window with low pixel delta (< threshold) are pruned.
Thanks, pinning the click frame and letting the scene cuts supply the settled state is a clean split. The reviewer gets cause and effect without you guessing a delay.
One edge case worth a test: a spinner or blinking caret can sit right around gt(scene,0.05). Small animations either never cross it, so the "after" frame is missed, or keep crossing it, so you get a burst of near-identical cuts that dedup then has to clean up. Do you pair each anchor with the first cut after it, or keep every cut inside a window? A "first cut followed by N quiet frames" rule might give you exactly one settled frame per action.
Haha yeah, that exact edge case is brutal in practice! Blinking carets and looping spinners are the bane of threshold-based scene detection.
Right now, we donβt strictly pair them 1:1. We throw the anchors and scene cuts onto a single timeline, and our dedup pass compares each candidate against the last kept frame (not just the one right before it). That way, if a spinner just oscillates between two visual states, the repeats get caught and pruned pretty quickly. The trickier side of the coin was tiny spinners (like a 20px icon in a 1080p window) that never even cross scene > 0.05βwhich is why we ended up adding localized diffs between actions.
That said, your idea of βfirst cut followed by N quiet framesβ is honestly brilliant. Itβs basically a visual debounce, and itβd give you a guaranteed, clean βcause
β
β settled effectβ pair per action without needing to tidy up bursts in post.
Iβm actually going to write a test case for a 1Hz blinking cursor to play with that settling window logic. Really appreciate the sharp catch!
Comparing against the last kept frame is a good call. One thing the 1Hz caret test may show: if the spinner flips between two states, A then B then A, each candidate still differs from the last kept frame, so you can keep every flip. Checking a candidate against the last two or three kept frames, or ignoring changes smaller than a few dozen pixels unless they are near the anchor's click point, usually stops that.
Localized diffs between actions for the 20px icon case makes a lot of sense. Curious what the caret test turns up.
Youβre 100% right on the A β B β A ping-pongβcomparing only against the immediate previous kept frame totally falls for that trap if the cycle is slower than our min_gap (0.3s).
Right now, our safety net for that is actually downstream in the triage layer: we set a min_region_px = 400 threshold, so glyph-level changes like a 2Γ18px caret get flagged as noise and don't trigger model calls.
But having the keyframe reducer itself be smart enough not to capture those flips in the first place is way cleaner. A rolling ring buffer of the last 2β3 kept frames or anchoring changes to the click locus would kill the flip loop right at the source.
Definitely stealing the ring-buffer idea for the reducer pass. Really appreciate you thinking through these edge cases with me!
One thing to watch with min_region_px = 400: a checkbox or radio toggle is roughly 16x16 = 256 px, so the triage layer would drop exactly the change that proves a click landed. If the threshold only applies away from the click point, you keep the caret filtered and still see the checkbox flip.
Good luck with the 1 Hz caret test.
π