Technical Deep-Dive: Building an AI-Powered Hand-Drawn Whiteboard Video Generator with Revideo & Rough.js

#ai #webdev #video #dailybuild2026

Visual storytelling is one of the most effective ways to communicate complex technical subjects—whether explaining how a Kubernetes Control Plane coordinates worker nodes, how Retrieval-Augmented Generation (RAG) indexes document embeddings into a vector database, or how Kafka Consumer Groups process partition offsets.

However, creating high-quality whiteboard animation videos traditionally requires hours of manual keyframing in video editors or software like VideoScribe.

In this deep-dive, we will explore the architecture and implementation of AI Whiteboard Video Generator—a full-stack web application that takes natural language prompts and automatically synthesizes hand-drawn Excalidraw-style whiteboard videos alongside executable Revideo motion graphics code.

System Architecture & Data Flow

The application is built around a multi-stage pipeline:

┌─────────────────────────────────────────────────────────────────────────┐
│                              1. USER INPUT                              │
│         Prompt: "Explain how RAG (Retrieval Augmented Generation) works" │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                          2. STRUCTURED LLM AGENT                        │
│   Gemini 2.5 Flash / OpenAI JSON Mode -> Parses concept into scenes     │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                       3. WHITEBOARD PLAN SCHEMA                         │
│   • Scene 1: Document Chunking & Embedding into Vector DB               │
│   • Scene 2: Similarity Search & LLM Prompt Context Augmentation        │
│   • Scene 3: Response Synthesis & Highlight Swipe                      │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                 ┌───────────────────┴───────────────────┐
                 │                                       │
                 ▼                                       ▼
┌───────────────────────────────────┐   ┌───────────────────────────────────┐
│     4A. ROUGH CANVAS PLAYER       │   │    4B. REVIDEO MOTION CODE GEN    │
│  • Rough.js Canvas API            │   │  • Programmatic @revideo/2d code  │
│  • Real-time interpolation        │   │  • Organic Jitter & Spring Ease   │
│  • Multiline text auto-scaling    │   │  • Timeline Voiceover Audio Sync  │
└───────────────────────────────────┘   └───────────────────────────────────┘

1. Structuring the AI Prompt for Visual Precision

One of the biggest challenges in AI-generated visual diagrams is preventing overlapping text, clipped container boxes, and unreadable arrow connectors.

To solve this, the LLM is guided by a strict layout & design system prompt:

export const WHITEBOARD_SYSTEM_PROMPT = `You are an expert instructional illustrator and whiteboard animator.
Your task is to take an explanation request and generate a structured JSON Whiteboard Plan.

STRICT LAYOUT & DESIGN RULES:
1. Break down the concept into 2 to 4 clear sequential scenes.
2. Canvas coordinate space: (0,0) is screen center. x spans -300 to +300, y spans -150 to +150.
3. Box & Container Sizing: Make rectangles, rounded_rectangles, databases, and clouds at least 150-180px wide and 100-130px tall so labels fit without clipping.
4. Multiline Labels: Always break long text or titles into 2-3 short lines using '\\n' (e.g. "Control Plane\\n(API Server)" or "Vector DB\\n[Embeddings]").
5. Non-overlapping Spacing: Keep at least 90-120px distance between centers of adjacent shapes.
6. Arrows & Connectors: Connect from outer edge of source element to outer edge of target element. Do NOT overlap shapes or text labels.
7. Maximum 5-7 objects per scene. Do NOT overcrowd.
`;

2. Real-Time HTML5 Rough.js Canvas Rendering

For immediate in-browser playback, the application utilizes roughjs to render sketch-style vector primitives on an HTML5 <canvas>.

Solving Text Overflow with Auto-Scaling Multiline Typography

In hand-drawn diagrams, box dimensions vary. To ensure text labels never overflow containers or clip arrow bounds, a custom multiline text scaler dynamically calculates the bounding box width and scales down the font size proportionally:

function drawMultilineText(
  ctx: CanvasRenderingContext2D,
  text: string,
  x: number,
  y: number,
  fontSize: number = 16,
  fontFamily: string = 'Poppins, Arial, sans-serif',
  fontWeight: string = '500',
  color: string = '#141413',
  align: CanvasTextAlign = 'center',
  maxWidth?: number
) {
  if (!text) return;
  const lines = text.split('\n');

  let effectiveFontSize = fontSize;
  ctx.font = `${fontWeight} ${effectiveFontSize}px ${fontFamily}`;

  if (maxWidth && maxWidth > 20) {
    let maxLineLen = 0;
    lines.forEach((l) => {
      const w = ctx.measureText(l).width;
      if (w > maxLineLen) maxLineLen = w;
    });
    // Dynamically scale down font if text line exceeds container padding
    if (maxLineLen > maxWidth) {
      effectiveFontSize = Math.max(11, Math.floor(fontSize * (maxWidth / maxLineLen)));
      ctx.font = `${fontWeight} ${effectiveFontSize}px ${fontFamily}`;
    }
  }

  const lineHeight = effectiveFontSize * 1.25;
  const totalHeight = lines.length * lineHeight;

  ctx.fillStyle = color;
  ctx.textAlign = align;
  ctx.textBaseline = 'middle';

  const startY = y - totalHeight / 2 + lineHeight / 2;
  lines.forEach((line, index) => {
    ctx.fillText(line, x, startY + index * lineHeight);
  });
}

3. Revideo Code Generation (`@revideo/2d`)

Beyond browser previewing, developers often want to programmatically render video files for YouTube Shorts, TikTok, or documentation sites.

The revideoGenerator.ts module translates the JSON scene graph into executable @revideo/2d code.

Organic Hand-Drawn Jitter & Spring Overshoot Easing

To ensure Revideo animations don't feel artificially stiff or robotic, we introduced two mathematical physics helpers:

Jitter Variance (jitter): Adds subtle random offsets to shape coordinates, mimicking hand-drawn human inaccuracy.
Spring Overshoot Easing (springEase): Dampened sine-wave spring dynamics that cause elements to slightly overshoot their target size before settling naturally.

/** Adds subtle positional variance to shapes, mimicking hand-drawn Excalidraw primitives */
function jitter(val: number, amount = 2.5): number {
  return val + (Math.random() - 0.5) * amount * 2;
}

/** Spring-based overshoot easing for smooth organic hand-drawn movement */
function springEase(t: number): number {
  if (t <= 0) return 0;
  if (t >= 1) return 1;
  const c4 = (2 * Math.PI) / 3;
  return Math.pow(2, -10 * t) * Math.sin((t * 10 - 0.75) * c4) + 1;
}

Generated Revideo Scene Code Example

Here is an snippet of the generated Revideo TypeScript code:

import { makeScene2D, Rect, Line, Txt, Node, createRef, all, easeOutBack } from '@revideo/2d';

export default makeScene2D(function* (view) {
  const mainContainer = createRef<Node>();
  view.add(<Node ref={mainContainer} />);

  // --- Revideo Audio Track Synchronization Pipeline ---
  // Track 1 [VOICEOVER]: "rag_narration.mp3" (18.0s, Vol: 80%)
  //   • Sync Marker [Indexing Documents into Vector DB]: startAt = 0.0s -> Narration: "First, documents are chunked into embeddings..."

  // --- Scene 1: Indexing Documents into Vector DB ---
  const scene0_Container = createRef<Node>();
  mainContainer().add(<Node ref={scene0_Container} />);

  const ref_doc_box = createRef<Rect>();
  yield* drawBox(
    scene0_Container(),
    ref_doc_box,
    { x: -220, y: 0, width: 150, height: 130, color: '#0284c7', fill: '#e0f2fe', label: 'Docs & PDFs\n(Raw Data)' },
    0.8
  );
});

4. Audio Track Management & Timeline Synchronization

Audio is crucial for instructional videos. The application features an Audio Panel where users can:

Drag and drop local MP3/WAV/M4A voiceover recordings or ambient audio.
Fine-tune volume levels per track.
Automatically bind audio sync markers to scene timeline timestamps (t = 0.0s, t = 6.0s, etc.).

By combining structured Gemini 2.5 Flash prompts, Rough.js vector canvas drawing, and Revideo motion code generation, AI Whiteboard Video Generator bridges the gap between static diagrams and animated instructional media.