ChalkTalk: a whiteboard tutor that draws like a teacher, and lets you raise your hand

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Think about the best explanation you ever got. It probably wasn't a slide deck. It was someone with a marker, drawing the idea while they talked, and stopping when you said "wait, why?".

When you prepare for an ML or system-design interview, you get the opposite: papers, long videos, and blog posts. None of them draw for you, and none of them let you raise your hand.

So I built ChalkTalk. You type a topic ("how attention works") or drop in a PDF, and a tutor teaches it on a whiteboard, out loud. It writes the example sentence word by word, draws the formula and circles the part it's talking about, builds a small table or bar chart, crosses out the wrong idea, and writes the key points at the end. When you're lost, you press ✋ (or H). It finishes its sentence, answers on the side of the board while pointing at the drawing you're confused about, and then goes back to the lesson.

• Three levels: Basic, Intermediate, and Interview (with interview tips and follow-up questions).
• Notes: one short line per step appears next to the board, a summary, not a transcript.
• History: replay any lesson or answer.
• Voice on by default, made inside your browser.

Who would pay? People preparing for ML engineer, data scientist and system-design interviews already pay for courses, mock interviews and tutors. A tutor that teaches any topic or paper at 1 a.m., at your level, and takes questions, is the part they can't get today. The closest thing is Gauth's canvas mode, which is closed and general. ChalkTalk is narrow on purpose (ML and papers), and because the model is open and the voice runs on your device, a lesson costs me close to nothing. That means it can be cheap.

Demo

Try it: https://chalktalk-ohpg.onrender.com, press Try as guest, then Watch the demo, or type any topic. It's on Render's free plan, so the first visit can take a minute to wake up.

Code

Danialmoa / chalktalk

A whiteboard tutor that talks and draws like a teacher. Raise your hand to ask questions.

ChalkTalk

A whiteboard tutor for ML and system-design interview prep. It talks and draws like a teacher at a board (the example sentence, formulas, small tables and bar charts, links, marks and crossed-out mistakes), with a pen dot that moves to what it's drawing. It writes the main points in a Notes panel as it goes, and you can raise your hand to ask a question at any time.

Lessons are generated with Gemma 4 (google/gemma-4-26b-a4b-it) through Backboard, which routes the call to OpenRouter. A small FastAPI server (server/) keeps the Backboard key, handles log in and sign up, and serves the built page.

Voice is Kokoro (onnx-community/Kokoro-82M-v1.0-ONNX), running in the browser. It is on by default. The first visit downloads the model (about 115 MB); until it's ready, lessons play silently.

Demo plays a built-in XGBoost lesson and needs no key. Try as…

View on GitHub

How I Built It

Gemma 4 (26B-A4B, open weights) writes every lesson and every answer, through Backboard, which routes the call to OpenRouter. Kokoro-82M, an open voice model, speaks it inside your browser, in a Web Worker. Excalidraw, the open-source whiteboard, is the board, so the hand-drawn look comes for free. A small FastAPI app on Render keeps the API key, handles accounts, and serves the page.

One rule shaped everything: the model never gives pixels. LLMs are bad at coordinates, and smaller open models are worse. So Gemma writes NDJSON, one step per line: what to say, a 3–8 word gist for the notes, and up to three drawing actions, placed with words like below and right_of. My code does the real layout.

{"say":"A mistake is the real price minus the guess.","gist":"Mistake = price − guess","actions":[{"type":"write","id":"mistake","text":"mistake = price − guess","at":{"region":"main"}},{"type":"mark","target":"mistake.mistake","note":"also called residual"}]}

One step per line also means streaming: step 1 is on the board while Gemma is still writing step 5.

It worked on the first day. Then I watched it like a student, and it took four rounds to make it feel like a teacher.

1. My first board looked like a flowchart, not a teacher

The first version could write, draw boxes and arrows, and highlight. To keep the layout clean, I told Gemma "every item is a box". So every lesson became a chain of boxes: Query → Key → Value → Matching Score → Softmax… It even taught something wrong: Q, K and V are made side by side, not one from the other.

A teacher draws the thing itself. So I gave the board a teacher's moves:

• words: a sentence split into words you can point at
• link: a curved line between two words, thicker for a stronger link (attention weights, finally)
• table, bars, and a list that grows one line at a time
• mark: underline or circle one part in red, with a short note under it
• strike: cross out the wrong idea

And one small trick that made the biggest difference: any target can be id.part. s.cat is one word of a sentence, houses.150k is one table cell, and sum.η is the η inside a formula (I measure the text before it to find where to circle).

The split is the important part: Gemma chooses what to draw, the code decides how it looks and where it goes. That keeps a 26B model reliable. The rest is staging: a new idea starts a new row, a tall board wraps into a second column like a teacher walking across the board, the camera moves to the spot before drawing, a red pen dot follows whatever is being written, and at the end the camera zooms out to the whole board.

2. Getting reliable JSON out of Gemma

The most confusing bug: "it stops after a couple of seconds". The lesson started, played two or three steps, and then sat on "Writing more steps…".

The server logs were fine. The raw stream wasn't. Gemma closes almost every line with a stray ]}, like this: …"note":"far away"}]}]}. JSON.parse fails, and my parser quietly skipped the line. In one run, 4 of the 5 opening steps disappeared, so the lesson ran dry.

The fix is small: read only the first complete JSON object on each line, counting brackets and ignoring anything inside strings. Two more lessons from the same hunt:

• One bad action no longer throws away its whole step and its spoken line.
• My own prompt listed "point at one part of an item…" as a bullet under Actions, and Gemma politely answered with {"type":"point at"}. Small models take your formatting literally.

After that: 10 of 10 steps valid in my test runs.

3. The voice that froze the board, and then couldn't keep up

Kokoro first ran on the page itself, and every sentence froze the board and the buttons. I measured the long tasks: 8.1 s, 6.1 s, 15.4 s. Moving it into a Web Worker fixed the freezing, but not the speed: it took 10.7 s to make 7.3 s of speech, so after a few long lines the voice fell behind and the lesson had gaps.

The fix was two HTTP headers. onnxruntime-web, which runs Kokoro, only uses more than one thread when the page is cross-origin isolated (it needs SharedArrayBuffer):

response.headers["Cross-Origin-Opener-Policy"] = "same-origin"
response.headers["Cross-Origin-Embedder-Policy"] = "credentialless"

Same sentence, 5.3 s, now faster than real time. Safari ignores credentialless and keeps one thread, and if the threaded version ever fails to load, the worker restarts on one thread, so the voice never just dies.

The last gap was my own timing. Each step's drawing was spread over a duration guessed from the text length, and the guess ran about 2 s long, so there was a silence after every line. Now the drawing follows the real length of the voice clip, and the voice is made three steps ahead, starting the moment a step arrives.

	Before	After
Opening steps that survived parsing	1 of 5	5 of 5
Time to make 7.3 s of speech	10.7 s	5.3 s
Silence after each line	~2 s	~0 s
Page freezes while speaking	3–15 s	none
From Start to the first word	(stalled)	~7–10 s, with 5 steps ready

4. The bugs that would have ended my demo

• Invisible arrows. Excalidraw's converter moves an arrow's first point if it isn't at (0, 0). Every arrow that pointed up collapsed into a 1-pixel stub. Now every line is given relative to its first point.
• The step that never ended. On my Mac, Chrome's audio clock stopped while still reporting "running", so the clip's ended event never fired and the lesson froze for good. Each clip now has a backup timer a little past its length.
• Pause moved the board. Redrawing a step after pause or skip counted its own old spot as taken, so the item slid down a row every time.
• Answers that stayed on the board. Gemma sometimes placed an answer next to a lesson item, so "Continue" couldn't clear it. Anything drawn while answering is now forced into the side area.

Raise hand

When you press ✋, the player finishes the current step and pauses. Gemma gets the lesson title, everything said so far, a one-line-per-item summary of the board (id | type | text), and your question. It answers in the same NDJSON, in a side area, and it can mark items on the main board (miss.C, circle, "worst"), so the answer points at the exact thing you didn't get.

Built with an AI agent, tested by measuring

I built this with Claude Code as my pair. I played the student and said what felt wrong ("it looks like fixed components", "it seems stuck"), and the agent didn't guess: it drove the app in headless Chrome and measured long tasks, audio start and end times, and what Gemma actually sent. Every number in this post comes from those measurements. That's also how the two worst bugs were found: the stray brackets and the stuck audio clock.

The backend is small on purpose: FastAPI with email and password accounts (scrypt hashes, a signed session cookie), Postgres on Render, and an /api/llm proxy that picks the model, caps the size, and rate-limits each client, so the Backboard key never reaches the browser.

Why Does Open Innovation Matter?

For a tutor, the voice and the model are the product, and that's exactly where open matters.

• It's what makes it cheap. The voice is made on your device, so there's no per-character bill. A lesson is a minute or two of speech, and with a paid voice API the margin is gone before you've asked a question.
• It's private. The voice never leaves your computer, and history stays in your browser.
• I'm not locked in. Gemma's weights are open. Today it runs through Backboard; tomorrow it can run on my own GPU, or be fine-tuned on how good teachers explain.
• I could read the code. I fixed the invisible arrows by reading Excalidraw's converter, and doubled the voice speed by learning how onnxruntime-web picks threads. With a closed API, both would have been a support ticket.

What's Next

• Real stroke-by-stroke handwriting instead of fade-ins
• Formulas rendered properly (KaTeX) inside the board
• Ask out loud: hold Space to talk, with Whisper in the browser
• History saved per account, not per browser

Prize Categories

• Best Use of Gemma: Gemma 4 writes every lesson and every answer, and decides what the board draws.
• Best Use of Render: the whole app, the FastAPI backend, the built page and Postgres, runs on Render from one render.yaml.
• Best Use of Backboard: one Backboard key streams Gemma 4 for lessons and raise-hand answers, kept on the server behind a rate-limited proxy.