DEV Community: leviyi

Syncing a rhythm game to a YouTube iframe (a metronome on someone else's player)

leviyi — Sun, 06 Sep 2026 08:27:06 +0000

I shipped a little browser game where the backing track is a YouTube embed and the gameplay has to land on the beat. Not "close to the beat". On it, within tens of milliseconds, or the whole thing feels broken.

That sounds simple until you try it. The YouTube IFrame API gives you a video player, not a clock. This post is what broke on the way, and the three small modules that came out of it. The code is all here: yt-beat-sync. No dependencies, take what you need.

Problem 1: whose clock do you trust?

My first version ran game logic on performance.now() and started the YouTube player at the same moment. Two clocks, started together. What could go wrong?

Quite a lot, it turns out:

The player buffers. Your clock keeps running. The beat grid slides half a second off the audio and never comes back.
The user seeks. Your clock doesn't know.
getCurrentTime() is coarse. It updates in chunks and freezes while buffering.

The fix felt backwards: the coarse, freezing, low-resolution player clock has to be the source of truth. Not because it's accurate, but because it's honest. When the video buffers, game time should freeze too, because that's what the player is actually hearing. Any extrapolation you layer on top drifts away from the audio and the beats visibly slide.

So the game reads one clock:

time() {
  if (this.useExternal) {
    const t = this.externalTime && this.externalTime(); // player.getCurrentTime()
    if (t != null) return Math.max(0, t - this.intro);
  }
  if (!this.wallStart) return 0; // not started yet, never return an epoch
  return Date.now() / 1000 - this.wallStart - this.pausedAccum + this.base;
}

The intro offset shifts t=0 to the moment the beat actually drops in the video, so the game logic can think in "seconds since the drop" and ignore however much lead-in the video has.

And when the embed is blocked entirely (some networks and regions do this), the same clock falls back to wall time and the game keeps running. Degraded, but playable.

Problem 2: autoplay policy is a game-design problem

Browsers won't let you play audio without a user gesture. Fair enough. The nasty part is in the details.

An AudioContext starts suspended. You must call resume() inside a gesture handler. Fine. But creating a YouTube player costs a network round-trip: the API script, the iframe, the video. If you start all of that inside the click handler, the player materializes after the gesture window closes, and mobile Safari and Chrome block the sound.

The fix is to preload the API long before you need it:

// on page load, not in the click handler
window.addEventListener('pointerdown', () => YTEngine.preload(), { once: true });

Then the play click only has to do new YT.Player(...) and playVideo(), which fits inside the gesture window.

Android has a secret third state. Granting microphone permission (my game uses the mic) can flip the context to 'interrupted'. Every state === 'suspended' check sails right past it. Your scheduler runs, notes get scheduled, and nothing is heard. Silently. I found this on a live Android test where the beat audio was just gone. Now a wake() covers both states, hooked to onstatechange:

wake() {
  if (this.ctx && this.ctx.state !== 'running') {
    try { const p = this.ctx.resume(); if (p && p.catch) p.catch(() => {}); } catch (e) {}
  }
}

Problem 3: setInterval is not a metronome

setInterval(playClick, 60000 / bpm); // do not do this

Timers jitter by tens of milliseconds and get clamped to once per second in background tabs. For anything rhythm-critical that's unusable.

The standard answer is a lookahead scheduler: a short-interval timer (40ms) that schedules every beat falling inside a 150ms window, with sound placed on the AudioContext clock, which is sample-accurate:

this.loop = setInterval(() => {
  while (next < wallNow() + 0.15) {
    const at = this.ctx.currentTime + Math.max(0, next - wallNow());
    this.kick(at); // scheduled ahead, sample-accurate
    next += interval;
  }
}, 40);

One wrinkle I didn't expect: the audio clock freezes while the context is suspended. If you drive visuals off the same schedule, the whole metronome starves. So visuals run on wall time. A setTimeout aligned to each beat fires the CSS pulse whether or not audio is currently allowed. Sound catches up when the context resumes; the grid never stops.

That separation ended up being the whole architecture: audio on the audio clock, visuals on wall time, game logic on the player's timeline.

The demo

github.com/dengyu123456/yt-beat-sync is a single page: paste a YouTube URL, set the BPM and the intro length, and a ball bounces on the beat. Buffer the video, seek around, background the tab. The grid follows the player, because everything reads the one honest clock.

Three files, no dependencies: yt-engine.js (API preload, singleton player with loadVideoById reuse, a 12s watchdog for dead embeds), beat-clock.js (the two-source clock), metronome.js (the lookahead scheduler and kick/hat synths).

If you're building anything that syncs to embedded media (karaoke, play-along tabs, rhythm games, video-synced quizzes), steal it.

I use this in production at The Rhyme Game, a free browser game where you practice freestyle rhymes over real beats. The same clock drives the bouncing ball and the bar-by-bar word drops there. If you try the demo on something weird (SoundCloud embeds, livestreams, 7/8 time), I'd be curious what breaks.

I open-sourced a deterministic crochet chart engine — no AI hallucinations, plus an MCP server so agents can use it

leviyi — Mon, 10 Aug 2026 15:43:26 +0000

A crochet chart is a grid. Each cell is one stitch, each color is a yarn, and if row 14 says "7 blue, 3 white" then row 14 has exactly ten stitches — not eleven, never "approximately ten."

That property is exactly why AI image generators are useless at making them. Ask a diffusion model for a "crochet chart of a frog" and you get something that looks like a chart: wavy grid lines, cells that change size mid-row, a legend with nine colors and a frog made of eleven. Pretty. Unworkable. You can't crochet from a vibe.

So the engine behind my crochet site is deterministic image processing, and this week I open-sourced it: crochetpatterngen-engine — Python library, CLI, and MCP server. Same input, same chart, same stitch counts, every time.

What it does

Feed it a photo, get back a stitch-ready chart PNG plus exact per-color stitch counts that sum exactly to the grid size:

from PIL import Image
from crochet_chart_engine import generate_chart, estimate_yarn

png_bytes, meta = generate_chart(
    Image.open("dog.jpg"),
    technique="c2c",   # graph | c2c | tapestry | mosaic | filet
    grid_w=60,         # 20..120
    n_colors=8,        # 2..16
)
# meta: {"grid_w", "grid_h", "colors": [{"hex", "count"}],
#        "total_stitches", "technique", "warning"}

yarn = estimate_yarn(meta, weight="worsted")
# per-color yardage, so you know how much yarn to buy before you start

The pipeline is resize → median-cut color quantization → confetti cleanup (isolated single-stitch specks get absorbed into their neighbors, because nobody wants to change yarn for one stitch). Pillow + numpy, no heavy deps.

The fun bug: C2C blankets come out squashed

The most interesting problem wasn't color — it was geometry. Corner-to-corner (C2C) crochet blocks are physically wider than tall, roughly a 0.7:1 height-to-width ratio. A square grid of C2C blocks does not produce a square piece of fabric.

Every naive photo-to-chart converter gets this wrong: your dog photo comes out vertically stretched on the actual blanket, like a funhouse mirror. The engine corrects grid dimensions by the block aspect ratio, so the worked piece keeps the photo's proportions instead of the chart's.

It's a one-line insight and a week of testing. That's most of applied software, honestly.

Why an MCP server

The same engine also ships as an MCP server, so AI agents (Claude Desktop, Cursor, whatever comes next) can call it as a tool:

{
  "mcpServers": {
    "crochet": {
      "command": "uvx",
      "args": ["--from", "crochetpatterngen-engine[mcp]", "crochetpatterngen-mcp"]
    }
  }
}

Three tools: photo_to_chart, render_ascii_design, estimate_yarn_tool. The pitch: an agent that can say "here's your chart, 60×44, 8 colors, you'll need about 210 yards of the green" is doing something generative AI structurally can't — because the answer is computed, not sampled.

I think "deterministic tools behind an agent frontend" is an underrated pattern right now. Everyone is wrapping LLMs around fuzzy tasks; there's a lot of value in wrapping them around precise ones.

The ASCII design format is my favorite part

Charts can also come from text files — one character per stitch:

palette: G=#58A05C, W=#FFFFFF, .=#FAFAF7
title: Frog Face
technique: graph
---
..GGGG..........GGGG....
.GGWWGG........GGWWGG...

I hand-draw designs in a text editor and git-diff my frogs. There's something deeply satisfying about version-controllable pixel art that becomes a physical object.

Links

GitHub: dengyu123456/crochet-engine — MIT, pytest tests/ passes, PRs welcome
PyPI: pip install crochetpatterngen-engine ([mcp] and [rembg] extras)
The hosted version with a free pattern library: crochetpatterngen.com

If you build something with it — graphgans, a knitting variant, a cross-stitch port — I'd genuinely love to hear about it. And if you've ever tried to crochet from an AI-generated "chart," I owe you a beer and an apology on behalf of the industry.