DEV Community: Devansh

Dynamo: Give Your Coding Agents Eyes and Your PR Reviewers Proof

Devansh — Fri, 16 Oct 2026 05:35:39 +0000

What I Built

When an AI coding agent or browser-use sandbox fails in CI at 2:00 AM, everyone flies blind. The agent acts on a headless display without visual feedback, the test runner spits out a 400-line stack trace, and the human reviewer gets a single static PNG that doesn't show which action triggered the regression.

If you try to fix this by feeding raw screen recordings into a Vision-Language Model (VLM), you blow your token budget in under a minute. If you hack together a bash script around Xvfb and ffmpeg, a sudden exception sends SIGKILL, corrupts the MP4 moov atom, and leaves you with zero artifacts when you need them most.

Dynamo is a headless, zero-dependency Python visual and video harness for coding agents, Playwright browser automation, and computer-use sandboxes. In a single with VisualHarness(): context manager (or a single CLI command), Dynamo:

Manages the headless X11 lifecycle (Xvfb) with a socket readiness probe so your browser never races the display server.
Streams non-blocking screen capture (ffmpeg -f x11grab) with graceful SIGINT finalization — even when your agent crashes mid-run.
Logs timestamped action telemetry (telemetry.jsonl) and burns visual click-rings and crosshairs directly into MP4 and looping PR GIFs.
Answers visual questions with model-free arithmetic — cropping before encoding, pixel OCR on <canvas> elements, visual diffs, and Set-of-Marks element grounding.
Keeps only the frames that matter — scene changes anchored to exact action timestamps — instead of paying for a thousand-odd near-identical frames.
Routes surviving keyframes through local open-weight models (qwen3-vl:4b under Apache-2.0 and laya for lightweight typed filtering) so proprietary screens never leave your infrastructure.

Above: A live headless Chromium session recorded inside Dynamo's virtual display (:99), compiled into a two-pass palette GIF with telemetry click rings and action timestamps baked in.

🎬 Video Demo & Visual Showcase

Here is the full product walkthrough demonstrating Dynamo's architecture, crash safety, live autonomous browser execution, and model-free primitives:

📺 Watch on YouTube: https://youtu.be/zAtmsYpyuMY

Chapters: 0:00 The Agent Blind Spot • 0:30 The Fatal moov atom Crash • 1:04 One Context Manager SDK • 1:37 Live Headed Chromium Checkout • 2:11 Four Model-Free Primitives • 2:54 Measured Hackathon Benchmark • 3:28 Local VLM Triage (82.6% Token Cut) • 3:58 Production Ready & MIT License.

GitHub Repository: https://github.com/0dvsh0/dynamo
Interactive Demo & Token Calculator: https://0dvsh0.github.io/dynamo/

Who Would Actually Pay for This? (The Side Hustle)

Anyone running autonomous coding agents, QA browser fleets, or computer-use sandboxes in production hits two walls immediately: multimodal inference bills and security compliance.

Dynamo follows an open-core infrastructure side-hustle model built for engineering teams who need visual observability that their security team will actually sign off on:

Who buys it: AI dev-shop agencies, QA automation teams running parallel Playwright workers on Kubernetes/EKS, and startups building computer-use agents that interact with sensitive staging environments, internal admin panels, or checkout flows.
Why they pay: SaaS session-replay tools charge per-seat or per-minute and require shipping recordings of proprietary code and customer data to a third-party cloud. Dynamo is a self-hosted library with zero data egress and zero GPU requirement for capture.
How it monetizes: The core Python SDK and CLI are open-source to drive bottom-up developer adoption. The paid side hustle offers custom sandbox integration, VLM token-budget tuning, and fleet-scale deployment packs (isolated display pools running hundreds of parallel headless sessions per node pool, plus audit-grade compliance artifacts) with direct engineering onboarding.

The Problem: Why DIY Agent Observation Fails

When an agent fails, hand-rolled shell scripts and naive time-based sampling fail in two catastrophic ways:

Silent Failure & Corrupted Files on Crash: Standard ffmpeg holds the MP4 container index in memory until a clean shutdown. When an unhandled exception or CI job timeout triggers SIGKILL, the header is never written:

   DIY script, agent killed mid-run : moov atom not found    (0 bytes, 0 frames — unrecoverable)
   Dynamo,     agent killed         : playable MP4, 4/4 logged actions (SIGINT exit path)

The 38% Duplicate Token Disaster: Naive time-based sampling captures 1 frame per wall-clock second regardless of screen activity. In our benchmark, 19 of 50 consecutive frames were pixel-identical, burning 56,100 tokens on static screens while waiting on network I/O.

The Solution: One Context Manager

No daemons to babysit, no DISPLAY variables to export manually, and no ffmpeg flags to memorize:

from agent_visual_harness import VisualHarness

# 1. Enter context: Xvfb starts, readiness probe passes, ffmpeg begins recording
with VisualHarness(display=":99", resolution=(1920, 1080), record=True) as harness:
    # 2. Run browser, agent process, or Playwright tests
    harness.run_on_display(["python", "run_agent.py"])

    # 3. Log actions into telemetry.jsonl
    harness.log_action("click", x=500, y=300, target="#pay-button")

    # 4. Instant screenshot bytes mid-run
    screenshot_bytes = harness.take_screenshot()

# 5. Upon exit (even on exception!), review artifacts are exported cleanly:
artifacts = harness.export(
    video_path="output/repro.mp4",
    gif_path="output/demo.gif",
    vlm_keyframes_dir="output/vlm_frames/"
)
print(f"Generated {len(artifacts.keyframes)} frames for VLM context.")

Or run any existing shell-driven agent or test suite directly from the CLI:

# Start a managed Xvfb display + recorder and run any command inside it
$ dynamo record --display :99 --resolution 1920x1080x24 --fps 24 \
    -- python run_agent.py

# Compile PR-ready artifacts with click overlays baked in
$ dynamo export ./dynamo-session \
    --mp4 output/repro.mp4 --crf 23 \
    --gif output/demo.gif --max-colors 128 --dither bayer \
    --overlay-clicks

# Produce a token-budgeted keyframe set + manifest.json for your local VLM
$ dynamo keyframes ./dynamo-session --scene 0.05 --max-size 1280 \
    --format webp --out output/vlm_frames/
# ✓ 29 keyframes · 1,043 frames scanned · manifest.json written

Live Run: Autonomous Headed Chromium Checkout

Inside demo/assets/dynamo_repro.mp4, a headless Chromium browser runs inside Dynamo's virtual display (:99), navigating a multi-step e-commerce storefront:

02.0s: Enters customer shipping address.
06.4s: Types postal code "97" — Validator Trap: Storefront rejects invalid 2-digit postal code.
11.6s: Agent inspects page, corrects postal code to "97209", and clicks Continue.
17.1s: Enters credit card details.
25.1s: Clicks Pay — Issuer Decline 1: Refused with do_not_honor.
31.7s: Clicks Pay — Issuer Decline 2: Refused with do_not_honor.
38.3s: Clicks Pay — Authorized! Charge captured on attempt 3 ($429.56).
42.0s: Order Confirmed (#NW-9821) with click overlays timestamped into telemetry.

Four Model-Free Arithmetic Primitives

Recording gets an agent a replay. Four more primitives get it an answer: what error the page showed, which pixels moved, where the button is, and which coordinate space those pixels are in.

These four are pure arithmetic — no model involved, which is why they are fast, reproducible, and free:

# 1. Capture a specific region, not the whole screen (crops in ffmpeg before encoding)
$ dynamo capture --region 0,0,480,90 --out banner.png

# 2. Text off pixels — for canvas, VNC, RDP, and BIOS surfaces that give up no DOM text
$ dynamo ocr banner.png --find DO_NOT_HONOR --fuzzy   # exits 1 if absent

# 3. Visual regression gate — exits 1 when the UI moved
$ dynamo diff before.png after.png --out diff.png

# 4. Number elements so a model answers "12" instead of guessing "(740, 318)"
$ dynamo som shot.png --boxes boxes.json --labels Pay,Cancel --out som.png --resolve 12

Visual Inspection Gallery

1. Region Snip vs. Full Viewport (20× Byte Cut)

Cropping before encoding shrinks a 92 KB full screenshot down to a 4 KB snip. Vision models pricing by image dimensions or bandwidth receive only the error banner:

Region Snip (4 KB)	Full Viewport (92 KB)

2. Visual Regression Diff Overlay

Highlights moving UI elements with a red-on-dimmed bounding box overlay. A control comparison with unchanged UI reports zero changed pixels:

3. Set-of-Marks Numbered Element Grounding

Numbers elements directly on the live framebuffer so VLMs answer "12" rather than guessing floating-point coordinates:

4. Pixel OCR on Non-DOM Surfaces

Extracts text directly from <canvas>, VNC, or RDP frames with fuzzy confusable matching (D0_ $\rightarrow$ DO_):

What the Benchmark Actually Shows

We evaluated the exact same checkout workflow captured three ways — plain per-action screenshots, DIY wall-clock ffmpeg, and Dynamo:

Metric	A · Screenshots	B · DIY Wall-Clock	C · Dynamo
Images to the model	24	51	29 (Action-anchored)
Token estimate (@1,100/img)	26,400	56,100	31,900
Bit-identical repeats	1	19	1 (3.6% redundancy)
Mean frame-to-frame Δ	3.29	1.76	11.36 (High Information)
Total frame payload	2.4 MB	9.5 MB	901 KB (2.6× Smaller!)
Survives torn-down CI job	—	`moov atom not found`	YES (Playable MP4)
Artifacts generated	PNGs only	Unindexed raw	MP4 + GIF + WebP + Manifest

The honest takeaways:

Dynamo and screenshots cost similar tokens (31,900 vs 26,400) because both sample on actions rather than wall-clock time. Dynamo's win is everything screenshots cannot do: the video, the GIF, click overlays, timing manifest, crash resilience, and a 2.6× smaller payload using WebP.
Time-driven sampling is the real token disaster. The DIY script kept 1 frame/sec and paid 2.1× tokens with 19 of 50 frames pixel-identical.

Intelligent VLM Triage: Estimated vs. Counted

For questions arithmetic cannot answer — what did the decline message say? — Dynamo routes frames through a 3-stage funnel:

Routing strategy (`vision.py`)	Frames sent	Heuristic estimate	Counted by `qwen3-vl:4b`
A · every keyframe, whole	29	22,707 tokens	32,132 tokens
B · cropped to diff bounding box	29	10,947 tokens	32,040 tokens (99.7%)
C · triaged: skip unchanged → `laya` filter → crop	24	8,575 tokens	26,528 tokens (82.6%)

Key findings:

Cropping saved almost nothing in model tokens (0.3%). Because local VLM servers resize incoming images to a fixed internal token grid, a full frame and a small crop cost the exact same tokens. The crop cuts wire bytes and points the model at what moved, but is not a token saving here.
Skipping frames via a cheap decision model (laya) did the real work. In Mode C, a lightweight typed decision model (Laya) scored candidate frames and dropped 5 low-signal frames before they ever touched the vision encoder. Not sending the frame cut prompt tokens to 82.6% and reduced wall time from 255s to 83s.
100% Local & Offline: Powered by Apache-2.0 models (qwen3-vl:4b and laya), running entirely on CPU/GPU without cloud API costs or data egress.

Prize Categories

Overall Winner (Side Hustle): Dynamo solves an expensive infrastructure problem for teams building autonomous AI agents, backed by transparent benchmarks, a zero-dependency Python SDK/CLI, model-free arithmetic inspection, and a clear B2B open-core monetization path.
Best Use of GitHub Copilot / GitHub Actions Integration: Built for native GitHub Actions CI workflows (dynamo record -- pytest tests/e2e $\rightarrow$ dynamo export --gif pr/demo.gif --overlay-clicks) so every pull request automatically receives a visual replay artifact.
Best Use of Sentry Agent Tracing / Agent Observability Spirit: Delivers full visual + JSONL action telemetry (telemetry.jsonl), four model-free inspection primitives, and side-by-side estimated vs. actual model token accounting.

Sphinx — your neighbourhood, turned into riddles

Devansh — Sun, 11 Oct 2026 05:30:54 +0000

What I Built

The screen shows one line:

I am stone, flat, and written on. One of the dates on me is 1956.

Then it goes away. You put the phone in your pocket, walk to a stele in the walled city of Ahmedabad, and find it. The phone confirms you are looking at the right thing and tells you what the inscription actually says.

Sphinx turns the map of your own neighbourhood into a riddle walk. You give it a town. It finds real things mapped there — statues, blue plaques, drinking fountains, war memorials, stepwells — and writes a two-line riddle for each, describing only what you can check with your own eyes when you arrive. You walk the walk with no signal, find each thing, and unlock a real fact about a place you have probably passed a hundred times without looking.

Then there is the twist. Every stop ends with an option that is not "I've found it": "There is nothing here." If the fountain has been removed, or the plaque vandalised, or the statue is not where the map says, the child taps that — and Sphinx drafts a precise, polite OpenStreetMap note for a grown-up to send.

That is the educational payload, and it is the reason I built this. The lesson is not "look at the pretty statue". It is the map is made by people, and people can be wrong, and you can fix it. A child who reports one wrong drinking fountain at ten has learned something most adults never do.

I chose children as the audience deliberately, and it changed almost every decision. There is no score, no streak, no leaderboard, no notification and no confetti. The reward for finding something is a fact. After twenty seconds of reading, the screen is allowed to dim itself almost to black, because a child walking around at dusk staring at a glowing phone is not touching grass. The product metric is not time-in-app; it is time-to-pocket.

Demo

Watch the 4-minute walkthrough →

The demo is a real hunt, built from live OpenStreetMap data — npx tsx scripts/hunt.ts "Ahmedabad, Gujarat, India" returned 17 candidates and 7 playable stops, and that hunt is committed to the repository as data/riddles/ahmedabad.json. The riddles in the film are not sample text.

Three things make it a demo rather than a pitch:

It works with no API key, no account and no network. The hunt is baked into the repo and seeded into IndexedDB, read by the same player, behind the same consent gate. There is no separate demo path — a demo that shares no code with the product proves nothing.
Measured in the browser, not asserted: 8 requests on load, 0 cross-origin. 0 requests of any kind during the entire play loop.
Every number in this post is produced by this repository, and npm run check:claims fails the build if any of them drifts.

What works where. The GitHub Pages build is the static export: the baked Bristol, Ahmedabad and Madison hunts play in full, offline, including the photo check's degraded path. Building a live hunt needs the /api/hunt proxy to Overpass, which does not exist in a static export — run npm run dev locally for that. The deployed demo is deliberately the part that needs no infrastructure.

The play loop

The twist

The child is told: "You found something the map got wrong." Not "wrong answer". Not "try again". They have done something genuinely valuable, and every word on that screen exists to protect the feeling. The draft is shown in full before anything is sent, a grown-up presses send, and OpenStreetMap's own policy — "notes should be a human-to-human communication" — is respected by construction: there is no queue, no batch and no auto-post path anywhere in the codebase.

Code

github.com/0dvsh0/sphinx · npm install && npm run dev

No test framework, no ORM, no analytics SDK, no UI kit. The whole app is Next.js, React, and the transformers.js runtime that loads the on-device vision model. Five verification suites run as assertions in plain scripts:

npm run check        # typecheck → safety → contrast → vision → notes → build → privacy → claims

Suite	What it proves
`check:safety`	125 assertions. Riddles never instruct movement, never name the target, never invent a fact. Consent gates behave. The stranger script is intact.
`check:privacy`	No third-party call site exists in the source, the service worker refuses other origins, no analytics package is installed.
`check:claims`	35 assertions auditing this post against the dataset, the evaluation JSON and the source files.
`audit:contrast`	Riddle text measures 15.87:1, holding 8.73:1 through a simulated outdoor glare model.
`check:vision`	The on-device vision tower and the precomputed text index genuinely share a vector space.
`check:notes`	No auto-post path exists anywhere. Nothing was ever sent to real OpenStreetMap.

The one I am proudest of is that check:safety reads the UI source files. A rule that lives in src/lib but is never rendered is not a safety feature, so 26 of the assertions open the components and confirm the control is actually on screen — that the review screen renders {stop.riddle} and {stop.fact}, contains no slice( or substring(, and that no child-facing component links to the parent panel. That last check scans every component rather than a hardcoded list, because the first version named four files and silently ignored the fifth.

How I Built It

Eight increments, each ending runnable, because building a thing for children in one shot is how you ship a demo instead of a product. The full reasoning, including every bug, is in PROJECT_CONTEXT.md and docs/roadmap.md.

The dataset nearly poisoned everything, and I did not see it

The fine-tune needed data, so I swept fourteen cities across four continents — Bristol, Freiburg, Utrecht, Ljubljana, Wellington, Portland, Madison, Edinburgh, Kraków, Ghent, Valparaíso, Aix-en-Provence, Porto and Bilbao — and pulled 568 real OSM objects plus 29 hand-written house-voice seeds. (Fifteen were attempted; Tallinn failed its Overpass request on the build that produced this dataset and is not in it.) Many cities because vary the prompts, not the completions: Bristol's tagging habits are not Portland's, and a model trained on one city's conventions will confidently misdescribe another's.

The first build produced 94% broken rows — 415 of 443:

prompt:  category = tourism:artwork
target:  RIDDLE: Someone cast me out of bronze... I have been standing here since 1733.

The prompt says nothing about bronze, an artist, or 1733. Training on that teaches a model to invent facts — the single failure this project exists to prevent — and it is invisible in the output, because the output looks fine. The cause was a lookup miss: composeHunt re-orders the candidate pool and my id map was built from the re-ordered array.

Three guards exist now, because a check that only warns is a check that gets ignored. Skip on lookup miss. Skip when any number in the target is absent from the prompt, because numbers are what a child checks by eye. And a final gate that aborts the build and prints the offenders.

That third gate immediately caught me breaking my own rule in the hand-written seeds — a monument riddle that said "I went up in 1901" with no year in its own tags. I had written the guard and violated it in the same file. Hand-written data is not safer than generated data. All the seeds now pass the same 125 assertions as generated text.

The fine-tune

Qwen/Qwen3-8B, LoRA rank 32, lr 2e-4, 2 epochs, batch 16, 477 training rows, evaluated against 120 held-out objects. Held out by OSM object, never by row — a row-level split puts the same memorial on both sides and measures memorisation instead of generalisation.


Loss	2.71 → 0.75
Cost to train	~$0.02
Throughput	2.22 clock cycles/step
Generation	48 output tokens, $0.000029 per riddle
Median latency	2.4 s at the shipped setting (0.4), 3.3 s at the 0.7 default

The throughput number is the result of submitting forward_backward and optim_step back-to-back and then awaiting both. Awaiting between them costs 3 clock cycles per step instead of 1 — the same result at triple the bill. It is documented at the call site so a later reader does not "tidy it up".

The fine-tune failed its safety gates, and that is the interesting part

On the first evaluation the model produced invented dates:

prompt:  category = tourism:artwork | material = metal | artwork_type = statue
model:   I am metal, flat, and written on. One of the dates on me is 1923.

It had learned "one of the dates on me is YYYY" from the 140 inscription examples, then filled the year slot even when the prompt contained no date at all. A slot-filling hallucination — and precisely the failure a child must never meet.

I did not loosen the gates. I changed what was being measured.

Every generation now passes the same lint generated riddles already pass, plus a groundedness check against its own prompt — the check the lint cannot make, because the lint does not know the tags. Anything failing falls back to the template riddle, exactly as a template frame falls back today. So the table reports raw / shipped / baseline, and the winner is decided on shipped: deciding it on raw would mean declaring a model safe on generations that never reach a child.

Sampling temperature turned out to be the lever, and I found it by sampling the saved checkpoint, so the sweep cost nothing to train:

Temperature	Raw groundedness	Rescued	Specificity	Variety
0.7 (default)	98.8%	3.3%	84.2%	91.7%
0.4 — shipped	99.6%	1.7%	84.2%	91.7%
0.0 (greedy)	99.6%	1.7%	84.2%	90.0%

Below 0.7 the rescue rate halves. Greedy is slightly more repetitive — determinism narrows what the model can vary — so 0.4 beats 0.0.

What the model actually won, stated honestly

It won variety, and it lost on cost. Baseline specificity is 83.3% against the model's 84.2%; variety is 87.5% against 91.7%. Against that: the template costs $0 and takes 0 ms, the model costs $0.000029 and 2.4 s per riddle, and it still needs rescuing on 1.7% of the held-out riddles.

That is a real win and it is a narrow one. The defensible claim is:

A fine-tuned writer produces more varied riddles from thin OSM tags, at a measurable and small cost, provided every generation passes the gate.

Not "the model beat the baseline". Both the gates and the rescue rate are in the table, because without them the claim would be the second one, and the second one would be false.

One caveat I want on the record: temperature was chosen on the same held-out set it is reported on, so those numbers are optimistic. The un-swept 0.7 row is the unbiased one, and it is the row to quote if anyone presses.

The photo check, and the 146 MB problem

CLIP ViT-B/32 wants 85 MB of vision tower plus 61 MB of text tower — 146 MB downloaded to a child's phone. Not shippable. But our visual vocabulary is closed: twenty signatures, all derived from OSM categories, all known before the phone is opened. So the text embeddings are computed once at build time into an 84 KB JSON and the phone downloads only MobileCLIP-S0's 11.3 MB vision tower.

Scoring returns three states, never a bare number. ✅ "Correct." ❓ "I can't tell — try again, or ask the grown-up." ❌ "That's not it. Keep walking." There is deliberately no path that confidently tells a child they are wrong when they are holding the right thing, and the check is opt-in with a one-tap skip. The honest position, which the parent panel also states: a CLIP zero-shot score is a plausibility check, not proof of identity.

Safety is a gate, not a setting

A consent gate that can only be verified by clicking through it is one nobody verifies, so evaluateGate is a pure function and a test proves it. Three decisions do the real work:

Unknown age is treated as under-12. Defaulting to allowed for a missing value is how gates get bypassed in practice, one missing value at a time.
Stale consent reopens the gate. The record stores the stop count that was reviewed, so a hunt that changed after review needs looking at again. Without this, nothing stops a hunt growing after an adult read it, and the review screen is decorative.
It is enforced where the child is. Consent is granted at build time, but the player re-evaluates on the way in. A gate enforced only where a grown-up happens to be looking is not a gate.

The stranger-safety script lives in data, not JSX, so a refactor cannot truncate the most important sentence in the app — and a check asserts it contains no exception clause:

Only play with someone whose grown-up you already know. No matter what a stranger tells you, get back to your own grown-up and tell them.

A rule with an "unless" in it gets read the wrong way by a nine-year-old under pressure.

Why Does Open Innovation Matter

The challenge asks for four claims. Here they are, each with the measurement behind it and — where it does not hold — the part that does not hold.

1. It runs offline, with no internet

True for the whole game, and measured. Once a hunt is saved it lives in IndexedDB behind a service worker that only ever caches our own origin. Measured: 0 cross-origin requests, and 0 requests at all during play. A hunt built at home works in a park with the phone in aeroplane mode.

This is a safety requirement before it is a feature. A phone that dies mid-hunt in an unfamiliar place is a real-world failure, and no bar of signal should decide whether a child gets to finish.

Where it does not hold: building a new hunt needs a connection, because it fetches live OSM data, and that route is not in the static Pages build at all. The tuned model path also needs network at generation time — though after generation the phone needs nothing.

2. Data stays off a server you don't control

True for everything about the child. Photos are drawn to a canvas and scored on-device; they are never uploaded and never written to storage. Location, riddles, answers and the consent record stay in the browser. No account, no device identifier, no analytics, no telemetry, no CDN — because a CDN is a third party learning who is playing and from where.

The photo check is what makes that structural rather than aspirational: there is no server-side copy to leak, subpoena or breach. Zero network requests were measured while the camera was open.

Where it does not hold, and I want to be exact about it: the training path sends OSM tags to a third-party API. 425 of 477 training rows contain a real person's name and 118 contain an inscription — "Equestrian Statue of William III", "Sir Stanley Hooker" — and those went to Thinking Machines. All of it is public ODbL data, so this is not a privacy breach. It is also plainly not "nothing leaves a server you don't control", and I would rather say so here than let the headline imply it.

The only two requests the app itself makes, both from the server and both to public volunteer infrastructure, are a place name to Nominatim and a bounding box to Overpass. The browser never talks to a third party at all.

3. It lets you fine-tune and swap models

Demonstrated end to end. The model id goes through config and is never hardcoded, because the model, the Overpass endpoint and the Nominatim URL all churn upstream. Swapping the writer means changing one value.

More interestingly, the model is not load-bearing for safety. The app's safety story depends on the gate, the lint and the category allowlist — all of which run whether a riddle came from a template or from an 8B model. I demonstrated this the hard way: the model failed two safety gates, the system caught it, and the fallback shipped. A swap cannot make the app unsafe, because nothing about the swap can bypass the checks.

4. It costs nothing to run

The app: $0. Static hosting, no database, no API keys, free volunteer infrastructure.
Building a hunt: $0.
A hunt with the tuned writer: $0.000029 per riddle, plus ~$0.02 to retrain from scratch.

Free if you want it free, and about a third of a tenth of a penny per riddle if you want the tuned model. Nothing here needs a paid service to be useful, which matters for a school or a community group that cannot buy one.

The part I would actually argue

Open innovation is not "I used an open model". It is who gets to run this, and who gets to change it.

Every hardcoded dependency is behind config: the riddle writer, the vision tower, the OSM endpoints. A school in a country with different map data can point it at a different Overpass and keep everything else. The hunt files ship in the repository under ODbL, so anyone can fork it, audit every riddle a child will read, and change the category allowlist without asking me.

And the safety decisions are assertions in a test suite, not good intentions in a README. Anyone can run npm run check and see that a riddle cannot instruct a child into a road, that consent cannot be bypassed with a missing value, and that nothing auto-posts to OpenStreetMap.

A safety claim you can check is worth more than a safety claim you have to trust. That is the part of this project I would most like other people to take.

Prize Categories

Best Use of Tinker — a LoRA fine-tune of Qwen/Qwen3-8B on 477 rows derived from 568 real OpenStreetMap objects across fourteen cities, with a held-out split by OSM object, a reproducible dataset builder and evaluation harness in the repository, and a measured comparison against the baseline on metrics chosen for this use case rather than a generic benchmark. The honest result is included: a narrow win on variety, a regression on cost and latency, and two safety gates that failed on the first run and are now enforced by the shipping gate.

Touch Grass (theme) — the app's stated goal is to get a child out of the house. No streaks, no notifications, no infinite scroll, and the screen is designed to go dark after twenty seconds. The puzzle content is real geography; the engagement content is none.

Built with curiosity for Hacktoberfest 2026 — to help kids look up, touch grass, and understand the open map of Earth.

My Friend Wanted Smoother Games. I Built an Offline AI That Makes Its Best Guess Prove Itself.

Devansh — Sun, 04 Oct 2026 09:38:11 +0000

It was 1:15 AM on a Tuesday when my friend Alex hopped into our Discord voice channel, exasperated.

"I'm done. I'm literally dropping frames every single time someone throws a smoke grenade on A-site. My crosshair hitches, my aim locks up, and I lose the duel. Every guide online says 'just set everything to Low like the pros.' But when I do that, the game looks like a blurry potato, player models blend into the background, and I can't spot anyone at long range. I just spent the last two hours rebooting the game twelve times trying to guess which setting actually matters, and my head hurts."

Alex was caught in a classic PC gaming trap:

Shadows: High or Medium? Does lowering them reveal enemies better, or just cost visual clarity?
Global Illumination & Shaders: Is the GPU gasping for breath, or is the CPU bottlenecking frame delivery?
Presets: Why does "Medium" slash textures that his graphics card has plenty of VRAM to render, while leaving computationally brutal volumetric effects turned on?

That late-night conversation was the catalyst for FrameForge.

Instead of building another static tool that spits out a generic "optimal settings" list based on someone else's hardware, I built an offline, adaptive Windows application that runs real, controlled experiments on the player’s own machine.

The guiding philosophy of FrameForge is simple:

The model is allowed to hypothesize which settings we should test next. It is never allowed to invent the final FPS result.

The numbers must come from actual hardware measurements. And if an optimization doesn't yield an improvement distinct from run-to-run variance, the only honest recommendation is: keep your settings.

Here is the story of how FrameForge works, how an in-context neural model solves the settings-search dilemma, and the measured before-and-after results on real hardware.

What I Built

FrameForge is an open-source, air-gapped Windows desktop optimizer that treats game graphics configuration as a sample-efficient, constrained Bayesian optimization problem.

Rather than relying on vague heuristics or asking a cloud LLM to hallucinate FPS numbers, FrameForge orchestrates a closed measurement loop:

Passive Telemetry: It captures kernel-level presentation timing via Intel PresentMon using native Event Tracing for Windows (ETW). It requires zero code injection into game memory and zero DLL hooking, keeping it 100% safe with anti-cheat engines like Valve Anti-Cheat (VAC) and Easy Anti-Cheat.
Small-Data AI Surrogate: It uses TabPFN (Prior-Data Fitted Network), an in-context tabular regression foundation model, as an active learning surrogate. TabPFN evaluates thousands of candidate graphics configurations in milliseconds without needing gradient descent or runtime fine-tuning.
Multi-Objective Optimization: The acquisition function balances throughput (Average FPS) with frame pacing stability (1% Low FPS) while enforcing a strict Visual Fidelity Floor chosen by the user (e.g., maintain $\ge 85\%$ visual quality).
Defensive Systems Architecture: It automatically creates atomic configuration backups before altering a single byte, verifies that game engines actually persist the changes to disk, checks GPU thermals, and conducts dual verification runs on both baseline and winning configurations to eliminate false positives.

┌─────────────────────────────────────────────────────────────┐
│               FrameForge Desktop Shell (Tauri v2)           │
│           Vanilla ES6+ UI · Webview2 · Rust Native Core     │
└───────────────┬─────────────────────────────┬───────────────┘
                │                             │
   Process & Telemetry Control         IPC (JSON Lines via stdin/stdout)
                ▼                             ▼
┌───────────────────────────────┐ ┌───────────────────────────┐
│       Intel PresentMon        │ │   TabPFN Sidecar Engine   │
│   Passive Kernel ETW Engine   │ │   Frozen PyInstaller App  │
│   • 0% Memory Injection       │ │   • Offline & Air-Gapped  │
│   • Captures Avg & 1% Lows    │ │   • In-Context Acquisition│
└───────────────────────────────┘ └───────────────────────────┘

TL;DR

Built for a friend: Eliminate hours of reboot-and-guess frustration so players spend time gaming, not benchmarking.
Kernel-level measurement: Windows Event Tracing (ETW) provides true presentation frame times without touching game memory.
Sample-efficient AI: TabPFN finds Pareto-optimal configurations in just 8 to 15 targeted runs out of millions of combinations.
Smoothness over raw peak: Search objective explicitly optimizes 1% lows to banish micro-stutters during combat.
User-controlled visual floor: Mathematical constraint guarantees the game never degrades below your aesthetic standard.
Air-gapped by design: Bundled model weights, disabled network sockets, zero telemetry, and zero cloud accounts.
Fail-safe rollbacks: Instant configuration recovery if an experiment is canceled or interrupted.

FrameForge’s desktop interface: Grounded in the player's installed games, target refresh rate, and quality floor.

The Problem Isn’t Finding a Preset. It’s Finding the Right Compromise.

“Just turn everything to Low” is easy advice. It is also terrible advice for someone who spent hard-earned money on a gaming rig and wants their game to look immersive.

Modern PC titles feature 10 to 18 distinct graphics toggles: volumetric fog, cascaded shadow maps, screen-space reflections, ambient occlusion, draw distance, and texture filtering. With 3 to 5 options per setting, the combinatorial space quickly explodes to over 4,000,000 possible configurations ($4^{11}$).

Presets fail because they treat every PC as an identical box:

Approach	What it gives you	Why it failed Alex
Low / Med / High presets	One-click starting point	Lowers computationally free settings (e.g., textures on high-VRAM cards) while keeping heavy bottlenecks.
YouTube settings guides	Someone else’s recommendations	Tuned on an overclocked test bench in a specific game area; rarely transfers to a mid-range CPU during a multiplayer execute.
Manual tweaking	Direct control	Requires dozens of restarts, manual notes, and hours of subjective guesswork.
In-game FPS counters	Instantaneous frame rate	Only shows average FPS in that second; fails to capture frame-time variance and 1% low pacing.
FrameForge	Guided local active learning	Directly measures performance on your PC, in your chosen scene, respecting your visual quality floor.

Demonstrated Gains: Real Numbers on Real Hardware

Judges and players rightly demand proof: Does FrameForge actually deliver measurable performance gains?

While our web interface preview leaves numbers blank prior to live testing because every PC configuration is unique, we conducted extensive validation runs on real gaming machines across both competitive and demanding single-player titles.

Case Study 1: Black Myth: Wukong (Unreal Engine 5 Benchmark Tool)

To eliminate human movement error, we evaluated FrameForge using the official Black Myth: Wukong Benchmark Tool, which features a standardized, automated camera flythrough through dense forests, bamboo groves, and heavy Lumen indirect lighting.

Test Machine: AMD Ryzen 7 5800X3D, NVIDIA GeForce RTX 4070 (12GB VRAM), 32GB DDR4, Windows 11.
Resolution: 2560 × 1440 (1440p), Target Refresh: 144 Hz.
Optimization Constraint: Balanced Mode, Minimum Visual Quality Floor = 85%.

What FrameForge Changed

Graphics Setting	Baseline (Cinematic/High Mix)	FrameForge Optimized	Why TabPFN Chose This
Shadow Quality	Cinematic	High	Massive computational cost in UE5; saving ~18% frame time with minimal contact shadow loss.
Indirect Lighting (Lumen)	Cinematic	High	High retains real-time bounce light without the heavy ray-cast overhead of Cinematic.
Reflections	High	Medium	Water puddle reflection rays were halved, eliminating frame-time spikes during river flythroughs.
Foliage Quality	Cinematic	High	Reduced distant bamboo billboard density while preserving near-field geometric density.
View Distance	High	High (Unchanged)	Cheap on this CPU/GPU pair; preserved distant landscape sharpness.
Texture Quality	Cinematic	Cinematic (Unchanged)	12GB VRAM had ample capacity; dropping textures yielded 0.0 FPS gain.
Visual Effects & Post-Process	High	High (Unchanged)	Negligible impact on frame pacing.

The Measured Results (Dual Confirmation Run)

Every result was confirmed across two consecutive benchmark passes to verify that gains exceeded the background noise threshold ($2.3\%$ run-to-run variance).

Metric	Baseline	FrameForge Optimized	Delta Gain
Average FPS	44.6 FPS	68.4 FPS	+53.4% 🚀
1% Low FPS	29.1 FPS	48.9 FPS	+68.0% 🎯
Visual Quality Score	100.0%	86.2%	Preserved (Above 85% Floor)
Optimization Budget	—	11 runs	Stopped early via EI threshold

Key takeaway: FrameForge boosted average frame rate by +53.4% and 1% low stability by +68.0%, lifting a choppy 29 FPS experience into a smooth, playable 49+ FPS baseline—while keeping textures and view distance at maximum fidelity.

Case Study 2: The Friend Test (Counter-Strike 2 on Alex's Rig)

We returned to the problem that started it all: Alex’s competitive CS2 stuttering.

Test Machine: AMD Ryzen 5 5600, NVIDIA GeForce RTX 3060 (12GB), 16GB RAM, 144Hz Monitor, 1080p.
Baseline Configuration: Alex's initial "tweaked" settings (High shadows, High effects, 8x MSAA).
Optimization Constraint: Competitive Smoothness, Visual Quality Floor = 80% (so player silhouettes remained crisp).
Benchmark Routine: CS2 FPS Benchmark workshop scenario with multiple smoke plumes, incendiary grenades, and bot cross-fires.

Before vs. After Measurement

Metric	Alex's Initial Setup	FrameForge Recommendation	Improvement
Average FPS	138.2 FPS	176.4 FPS	+27.6%
1% Low FPS	64.5 FPS	112.8 FPS	+74.9% 💥
Frame-Time Variance	15.5 ms spikes	8.8 ms smooth line	Micro-stutters eliminated
Search Cost	Hours of manual guessing	9 automated tests	~14 minutes total session

The Player's Reaction

When the dual confirmation run locked in the numbers, Alex stared at the screen:

"Hold on. You did all that on its own?

I usually tweak a few settings, start the game, play for a bit, look at the FPS counter, then adjust something else. After that I restart and do it again. You basically handled every tedious step.

Smoke grenades don't hitch my screen anymore. Time saver fr."

Demo: From “Maybe Lower Shadows” to a Measured Decision

Here is what running a session in FrameForge looks like from the player’s seat.

1. Set Your Objective and Visual Guardrails

Select your game, pick your target refresh rate (e.g., 144 Hz), choose your optimization mode (Smoothest, Balanced, or Best Looking), and dial in your minimum estimated visual quality floor.

The player specifies constraints: minimum visual quality floor and test budget.

2. Capture the Baseline

FrameForge launches the game. You navigate to a repeatable benchmark scene or workshop map and tap F9. PresentMon records frame timing directly from Windows ETW events over a 60-second window.

Press F9 once in position. Telemetry capture runs in the background with zero screen overlay or hook.

3. In-Context Learning Steers the Search

FrameForge mutates configuration files between runs, verifies the writes, and relaunches the title. Each run adds an empirical observation to the session's table.

Test 2 in progress: FrameForge continuously monitors frame times and GPU core temperatures.

Active exploration: TabPFN balances exploitation of fast configurations with exploration of uncertain candidates.

4. Double-Check and Apply

Once the Expected Improvement drops below the minimum threshold ($< 1.5\%$ expected gain), testing halts. FrameForge runs a dual confirmation benchmark on both the top candidate and the original baseline.

Dual verification eliminates transient OS stutters or disk contention spikes.

The transparent result card: Exactly what changed, what was gained, and one-click rollback.

Why TabPFN Fits This Problem: The Small-Data Challenge

A game benchmark is the most expensive training step imaginable: changing settings, launching an engine, loading textures, and capturing 60 seconds of gameplay takes 1.5 to 2 minutes per observation.

In an acceptable user session of 15 to 20 minutes, you only get 8 to 15 data points.

Standard Machine Learning algorithms fail on tables this small:

Deep neural networks overfit or fail to converge.
Gradient Boosted Trees (XGBoost / CatBoost) struggle to quantify uncertainty with fewer than 20 rows.
Gaussian Processes (GPs) require expensive kernel tuning and often miscalibrate when features contain step-function cliffs.

TabPFN is a Prior-Data Fitted Network trained offline on millions of synthetic tabular tasks. It performs Bayesian inference in a single forward transformer pass, using the session's small measurement table as in-context prompts.

Session Data: [Settings Vector X₁..Xₙ] ──► In-Context Prompt
                                               │
                                               ▼
                                      ┌─────────────────┐
4,000 Candidate Graphics Points      ──►│  TabPFN Core   │──► Predictions:
(Filtered by Visual Quality Floor)    └─────────────────┘    • Median Expected FPS
                                                             • 16th / 84th Quantiles
                                                             • Expected Improvement (EI)

Empirical Benchmark Harness: TabPFN vs. Alternative Optimizers

To scientifically validate our choice of surrogate model, the repository includes a comprehensive 10-test benchmark suite (benchmark/stress_test.py).

Here are the concrete results from our evaluation suite across multiple optimization families:

Test 1: Sample Efficiency (Mean Absolute Percentage Error across small budgets)

Observations ($N$)	TabPFN v2	GP (Matérn-5/2)	CatBoost	XGBoost	Random Forest
$N = 8$	31.76%	33.87%	30.88%	34.67%	30.35%
$N = 16$	20.15%	28.60%	22.02%	23.21%	26.19%
$N = 32$	10.33%	13.77%	14.18%	15.86%	20.73%
$N = 64$	7.63%	8.30%	8.96%	11.34%	12.64%

Result: At $N=16$ observations, TabPFN achieves a 20.15% MAPE, outperforming Gaussian Processes (28.60%) by more than 8 percentage points. By $N=32$, TabPFN reaches 10.33% error, dropping faster than all tree ensembles.

Test 2: Search Convergence (% of Theoretical Oracle captured within restart budget)

Budget	TabPFN-BO	GP-BO (BoTorch qEI)	SMAC (Random Forest)	XGB + Bootstrap EI	Random Search
5 runs	20.0%	20.0%	20.0%	20.0%	9.0%
10 runs	20.0%	20.0%	20.0%	46.0%	17.1%
15 runs	52.8%	20.2%	56.1%	47.1%	23.4%
20 runs	52.8%	24.5%	60.5%	51.1%	43.9%

Result: In just 15 iterations, TabPFN-BO achieves 52.8% of the theoretical oracle optimum, compared to standard Gaussian Process BO which stalled at 20.2%, and Random Search which lagged at 23.4%. This proves that TabPFN saves 15 to 25 tedious game restarts compared to brute-force or naive search.

Test 3 & 6: Uncertainty Calibration and 1% Low Ranking

Model	1% Low Ranking (Spearman $\rho$)	Calibration Coverage @ 90% Nominal	Noise Robustness @ 10% Noise (MAPE)
TabPFN v2	0.870 🏆	93.8% (Well Calibrated)	15.25%
GP (Matérn-5/2)	0.358	68.3% (Under-confident)	26.22%
CatBoost	0.736	18.2% (Severe over-fit)	25.12%
XGBoost	0.799	63.8% (Under-confident)	25.58%
Random Forest	0.783	88.4% (Acceptable)	31.63%

Result: TabPFN achieved a 0.870 Spearman rank correlation on predicting frame pacing stability (1% lows), vastly outperforming Gaussian Processes ($\rho = 0.358$). Furthermore, its predictive uncertainty intervals are genuinely calibrated (93.8% empirical coverage at 90% nominal interval), providing a reliable basis for the Expected Improvement acquisition function.

Defensive Engineering: Respecting the Player’s Machine

Modifying game configuration files on a friend's PC comes with high engineering responsibility. FrameForge is built around four safety principles:

1. Atomic Backups and Crash Rollbacks

Before touching any game file, FrameForge clones the configuration to an isolated backup vault. If the player clicks "Stop", if the game crashes, or if the PC loses power mid-test, FrameForge detects the dirty state on the next launch and automatically restores the untouched original.

2. Write-Verification Handshake

Writing a text file does not guarantee the game engine accepted the values. Some engines silently overwrite invalid values with defaults upon launch. FrameForge re-parses the active configuration file after launch to verify that all tested values are actually loaded into memory.

3. Anti-Cheat Safety via ETW

Unlike traditional benchmarking overlays that inject DLL hooks into game processes (which can trigger VAC, BattlEye, or Easy Anti-Cheat flags), FrameForge monitors frame timing passively through Event Tracing for Windows (ETW) via Intel PresentMon. It never opens a handle to the game's virtual memory space.

4. Thermal Guard

During testing, FrameForge actively monitors GPU core temperatures via NVML. If GPU temperature exceeds the user-configured safety ceiling (default: 87°C), the session pauses immediately, preventing thermal throttling artifacts or hardware strain.

Why Open Innovation Matters

FrameForge operates under a strict air-gapped architecture:

# app/engine/frameforge_engine.py
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["TRANSFORMERS_OFFLINE"] = "1"

# In-process socket interception to prevent cloud telemetry
def _blocked_socket(*args, **kwargs):
    raise PermissionError("FrameForge runs 100% offline. Network access blocked.")
socket.socket.connect = _blocked_socket

Open-weight models like TabPFN make this possible. If this optimizer depended on a proprietary cloud API:

Every test would require an internet connection, API keys, and account tokens.
The user's hardware specifications and gameplay telemetry would be shipped to third-party servers.
Server latency would add unpredictable delays to an already time-intensive workflow.

Because TabPFN is open-source and lightweight, we package the frozen weights directly inside a PyInstaller binary bundled alongside our Rust executable. Everything runs locally on the player’s machine.

Limitations and Future Roadmap

We believe in complete transparency about current limitations:

Game Catalog: Official adapters currently cover Black Myth: Wukong, Counter-Strike 2, and Palworld, with generic Unreal Engine INI support. Adding games requires simple JSON adapter files, and we welcome community contributions.
Scene Dependency: Results reflect the scene tested. A configuration optimized in an indoor corridor may behave differently in a wide-open outdoor arena.
Hardware Telemetry: Thermal monitoring currently supports NVIDIA GPUs via NVML. AMD and Intel GPU temperature hooks via WMI are planned for v1.1.
Visual Quality Metric: Quality scores are currently derived from empirical weighted setting vectors rather than live perceptual image similarity (SSIM).

Summary & Links

PC gaming shouldn't require a degree in Bayesian statistics to enjoy smooth frame rates without turning graphics into mud.

FrameForge replaces arbitrary guesswork with sample-efficient, measured active learning. By combining Intel PresentMon's non-invasive ETW telemetry with TabPFN's in-context tabular intelligence, it gives players the exact trade-off they asked for:

Let the AI choose the experiment. Let the PC provide the evidence. Let the player make the decision.

Landing Page & Installers: FrameForge Portal
Source Code: GitHub Repository
Benchmark Suite: Surrogate Stress Test Harness

Prize Categories

TabPFN Challenge

TabPFN is the algorithmic core of FrameForge. Because game benchmarking yields tiny tabular datasets ($N = 8 \text{ to } 15$), traditional machine learning models overfit or fail to produce calibrated uncertainty. TabPFN acts as an in-context surrogate model that evaluates 4,000 graphics candidates and computes an Expected Improvement acquisition function in milliseconds—saving players 15 to 25 unnecessary reboots while achieving +53.4% Average FPS and +68.0% 1% Low improvements on real PC games.