DEV Community

Cover image for Building Beat Clash: An AI Rhythm Game with React, Tone.js, and Multi-Provider LLM Inference
Harish Kotra (he/him)
Harish Kotra (he/him)

Posted on

Building Beat Clash: An AI Rhythm Game with React, Tone.js, and Multi-Provider LLM Inference

#ai #programming #javascript #dailybuild2026

Why this app exists

Most rhythm game prototypes fail at one of two things:

  • timing fidelity (UI animation drifts from audio)
  • content pipeline (lyrics are static or hardcoded)

Beat Clash solves both by combining:

  • transport-locked audio timing with Tone.js
  • dynamic rap + timing generation via LLMs

The result is a fast MVP where each run is new, playable, and debuggable.

Product loop

  1. User enters roast topic + style + difficulty
  2. Backend generates rap JSON (bpm, line timings, emphasis words, hook)
  3. Frontend starts transport at generated BPM
  4. Grid + lyric word highlighting follows current beat
  5. Player (or AI Agent mode) taps each beat
  6. Engine scores Perfect/Good/Miss
  7. Results + replay export

This keeps session length short (<30s) and replay value high.

System architecture

System architecture

Provider abstraction strategy

The backend normalizes generation into a single shape regardless of provider.
That means OpenAI, Featherless, and Ollama all return the same game-ready contract.

Backend design

API shape

  • POST /api/generate-rap returns normalized rap JSON
  • GET /api/models?provider=ollama lists local models

Important implementation detail

If generation fails, backend returns a deterministic fallback rap so users still play.
This is key for demo reliability.

Generation contract (must-have) 

{
  "bpm": 92,
  "structure": [
    {
      "line": "...",
      "timing": { "start_beat": 0, "duration_beats": 4 },
      "emphasis_words": ["..."]
    }
  ],
  "hook": {
    "line": "...",
    "timing": { "start_beat": 16, "duration_beats": 4 },
    "emphasis_words": ["..."]
  }
}
Enter fullscreen mode Exit fullscreen mode

OpenAI-compatible inference snippet 

const client = new OpenAI({ apiKey, baseURL });
const completion = await client.chat.completions.create({
  model,
  response_format: { type: "json_object" },
  messages: [
    { role: "system", content: systemPrompt() },
    { role: "user", content: userPrompt(payload) }
  ]
});
Enter fullscreen mode Exit fullscreen mode

Frontend design

Timing source of truth

Tone.Transport is the master clock.
The UI does not schedule beats with setTimeout; it responds to transport callbacks.

this.transportBeatEvent = Tone.Transport.scheduleRepeat((time) => {
  const beatIndex = beatCount;
  Tone.Draw.schedule(() => onBeat(beatIndex, time), time);
  beatCount += 1;
}, "4n");
Enter fullscreen mode Exit fullscreen mode

Using Tone.Draw.schedule keeps visual updates aligned with audio time.

Input judgement pipeline 

const deltaMs = getNearestBeatDeltaMs(tapTime, beatTimesRef.current);
if (Math.abs(deltaMs) <= 50) return "Perfect";
if (Math.abs(deltaMs) <= 120) return "Good";
return "Miss";
Enter fullscreen mode Exit fullscreen mode

This gives a clear skill curve while still feeling fair.

AI Agent mode (autoplay)

Manual tapping is fun for gameplay but poor for demos and QA.
So Beat Clash includes AI Agent mode:

  • generates auto taps per beat
  • injects light jitter for realistic performance
  • runs through the same scoring path as player input

That means every metric and replay format stays consistent across manual and automated runs.

Engineering choices that mattered

1. Keep contract tiny

Small JSON schema made it easier to validate and recover from malformed generations.

2. Normalize everything at the backend edge

No provider-specific logic in gameplay components.
Frontend receives one shape and stays deterministic.

3. Ship fallback behavior first

Graceful degradation turned API outages into playable sessions.

4. Build for observability

Replay export captures generated rap + taps + judgements.
This helps tuning scoring thresholds and generation quality.

Local development 

npm install
npm install --prefix client
npm install --prefix server
cp .env.example .env
npm run dev
Enter fullscreen mode Exit fullscreen mode

If using Ollama:

ollama serve
Enter fullscreen mode Exit fullscreen mode

Extensions worth building next

  • voice synthesis for generated lines
  • real-time multiplayer battles
  • waveform + beatmap editor UI
  • ranked mode + persistent leaderboard
  • anti-latency calibration flow per device
  • creator mode with custom beat patterns

Final take

Beat Clash demonstrates a practical pattern for AI-native interactive apps:

  • generate structured content with LLMs
  • run deterministic runtime logic from that structure
  • keep user-facing interaction tight with transport-locked timing

It is not just “AI text in a game.” It is AI as authored game content + deterministic systems.

Github Repo: https://github.com/harishkotra/Beat-Clash

Top comments (0)