I built a napkin that turns scribbles into real diagrams, entirely in the browser

Napkin reads your rough sketch with a small on-device model and gives you back editable geometry. No account, no upload, no server. Here is how it works, and the three bugs that nearly sank it.

Every diagramming tool I have used starts the same way: you have an idea, you open the tool, and the tool asks you to pick a rectangle. The idea was a shape. The tool wanted you to name it first.

So the premise for Napkin was simple. Draw the thing badly with a mouse. Get the clean version back. Keep editing it as a real object: move it, label it, restyle it, attach arrows to it, export it.

The interesting constraint was the model. Shapes 4.2.1 from Desert Ant Labs is not an image generator. It takes a stroke and returns a shape kind plus an outline of points. That output is geometry, which means the result is editable instead of being a picture of a diagram. Everything good about this app follows from that one property.

---

The stack, and the library I deliberately did not use

• Vite 8 and React 19 with TypeScript 7
• @desert-ant-labs/shapes with @litertjs/core for on-device inference
• Hand written SVG for rendering, not canvas
• localStorage for persistence
• oxlint, one config file, no ceremony

The obvious move was to build on tldraw or Excalidraw. Both are excellent, both already have shapes, arrows, labels and export. I skipped them for a specific reason: their scene models would have hidden the exact work this project is about. If a library already knows what a rectangle is, integrating a shape recogniser becomes a small adapter, and the interesting decisions (how to accept or reject a model result, how connectors stay attached, what the model output maps onto) get made for you.

Rendering to SVG by hand was also a deliberate trade. Canvas would be faster for thousands of items. SVG makes every shape a DOM node you can style with CSS, hit-test with elementFromPoint, and export by serialising. At diagram scale, tens of items, the performance argument does not apply yet, and the debuggability argument does.

---

Architecture

Five steps per stroke:

1. Canvas.tsx collects pointer events into a draft stroke and renders the scene from items[].
2. geom.ts clusters near-duplicate points, then resamples the stroke at roughly 3 px spacing.
3. model.ts hands the points to Shapes and gets back a kind plus an outline. If the model is unsure, you get null.
4. App.tsx turns that into an Item and commits it through a reducer, which is what makes undo and redo trivial.
5. scene.ts autosaves, and export.ts turns the same items into SVG, PNG, Mermaid, or a .napkin file.

Three rules hold the layers apart:

• The model hook returns { status, recognize } and knows nothing about scenes, selection or history. Delete it and the editor still works as an ink canvas.
• Recognition returns geometry, never pixels.
• Every mutation goes through one reducer, so history is an array of previous item arrays rather than an event log.

One Item type for everything

The whole document model is a single type:

export type Item = {
  id: string;
  kind: Kind;               // 12 shapes + text + ink + scribble
  outline: Point[];         // clean geometry, or raw points for ink
  closed: boolean;
  style: Style;
  label?: string;
  attachments?: [string | null, string | null];
  endHead?: boolean;
  startHead?: boolean;
};

A connector is not a separate concept. It is an item with a two point outline and two attachment slots. That decision pays off in applyAttachments(), which is the only code that needs to know connectors exist, and in export, where a connector is just a path that happens to be short.

Loading the model without blocking the editor

The model is under a megabyte, but it is still a download, and the editor should not wait for it:

useEffect(() => {
  let alive = true;
  (async () => {
    try {
      const mod = await import("@desert-ant-labs/shapes");
      const loaded = await mod.Shapes.load({
        onProgress: (f: number) => { if (alive) setProgress(f); },
      });
      if (!alive) { loaded.dispose(); return; }
      instance.current = loaded;
      setStatus("ready");
    } catch (err) {
      console.warn("Shapes model failed to load", err);
      if (alive) setStatus("error");
    }
  })();
  return () => { alive = false; instance.current?.dispose(); instance.current = null; };
}, []);

Two details that matter more than they look:

Dynamic import. The SDK and its WebAssembly never enter the initial bundle, so the app paints and accepts strokes while the model is still arriving.

alive plus dispose(). React 19 StrictMode mounts effects twice in development. Without both, you leak a WASM instance on every mount, and the second instance quietly wins the race to answer. It is the kind of bug that shows up as "recognition works, but only after you navigate away and back once".

The status chip in the corner is the visible contract: Shapes loading, Shapes ready, or Ink only. The last one is not an error state, it is a promise that the app degrades instead of breaking.

The bug that ate a day: resample before you recognise

My first integration looked correct and failed constantly. Slow, careful mouse strokes came back as ink. Quick confident strokes recognised fine.

The cause was point density. Pointer events arrive at whatever rate the browser and the trackpad feel like, so a slow stroke delivers many points a few pixels apart and a fast stroke delivers a few points far apart. The model was trained on strokes sampled at a roughly constant spacing, and it wants {x, y} objects rather than tuples.

The fix is four lines and lives before every model call:

shape = await recognize(resample(clusterPoints(raw, 1.2), 3));

resample(points, step) walks the polyline and rebuilds it at fixed arc length spacing. After that, slow and fast strokes behave the same, and recognition went from unreliable to boring, which is what you want from a model call.

A second, smaller fix came from the same investigation. A straight line resampled down is legitimately two points. My acceptance test demanded three, so short arrows were rejected as ink while long ones passed. The threshold is now per kind:

const CONNECTORS = new Set(["line", "arrow", "curve"]);
const minPts = shape && CONNECTORS.has(shape.kind) ? 2 : 3;

Keeping arrows attached, the hard part

Attachments are stored as ids, never coordinates. Geometry is recomputed on render:

export function applyAttachments(item: Item, byId: Map<string, Item>): Point[] {
  const pts = item.outline.map((p) => ({ ...p }));
  const [a, b] = item.attachments ?? [];
  const last = pts.length - 1;

  if (a) {
    const target = byId.get(a);
    if (target && saneAnchor(target, pts[0], pts[last])) {
      const anchor = clipToBBox(itemBBox(target), pts[0], pts[last]);
      const dx = anchor.x - pts[0].x;
      const dy = anchor.y - pts[0].y;
      for (let i = 0; i <= last; i++) {
        const w = 1 - i / last;
        pts[i] = { x: pts[i].x + dx * w * w, y: pts[i].y + dy * w * w };
      }
    }
  }
  // mirrored for the far end
  return pts;
}

The w * w weighting is what makes this feel like a drawn arrow instead of a rubber band. The endpoint moves all the way to the new anchor, the middle barely moves, and the curve bends rather than translating.

Two guards keep it from misbehaving:

saneAnchor(). If a shape has been dragged more than roughly three times its own diagonal away, the anchor is refused. Otherwise a stray id in a stale scene flings a connector across the canvas on load.

Slab intersection, not per-axis clamping. My first version clamped the x and y coordinates independently, which is correct for the common case and wrong for near horizontal segments: the maths divides by a number close to zero and the endpoint lands somewhere off screen. clipToBBox() now intersects the segment with all four edges of the box and picks the real crossing. It is less code than the clamp and it is correct.

Erase by scribbling

There is no eraser mode required. A stroke that the model classifies as scribble becomes a lasso: any item whose points fall inside the returned hull is deleted, and any connector attached to a deleted item is detached rather than left dangling.

That last part is the unglamorous half of the feature. Deleting a node in a diagram has to repair the edges around it, and doing it in the same commit is what keeps undo honest.

Export is where the item model earns its keep

Four formats, one source of truth, no format specific state:

export function toMermaid(scene: Scene): string {
  const CONNECTORS = new Set(["line", "arrow", "curve"]);
  const nodes = scene.items.filter((i) => !CONNECTORS.has(i.kind) && i.closed);
  // flowchart TD, ids from item ids, edges from attachments
}

Mermaid is a lossy target and the code says so out loud. Connectors become edges. Unlabelled shapes export with a blank label instead of a generated Step, because an invented label in a diagram someone pastes into a README is worse than an empty one. Every export decision like that got made in favour of the person receiving the file, not the person generating it.

SVG export is a straight serialisation of pathFor(item) plus a <text> per label, with padding from the union bounding box. PNG is that SVG drawn into an offscreen canvas at 2x. The .napkin file is the same JSON as autosave, which means reopening a file and reloading a tab exercise identical code.

What I would tell someone building the same thing

Treat the model as a suggestion, not an oracle. Every stroke gets an accept or reject decision with a visible consequence: accept becomes a shape, reject stays ink. Users forgive a wrong guess when nothing is destroyed. They do not forgive a stroke disappearing.

Resample everything. Not just for this model. Any geometry model trained on strokes is sensitive to input density, and pointer events are the worst possible source of it.

Keep the model out of your state. The moment a model result lives inside a reducer action, testing the editor requires a model. Behind a hook that returns null, the whole editor is testable with fixtures.

Make the capability discoverable in the product. My first version quietly supported twelve shapes and communicated four, because the empty state hint mentioned four and nothing else in the UI contradicted it. A tester asked whether the app was "too basic". The model was never the limit, the interface was. There is now a Shapes I can read panel that draws all twelve outlines and says what to draw to get each one, and the hint mentions the erase gesture too. Cheap to build, and it changed the first impression completely.

Where it goes next

The gaps I would close first, roughly in order of value:

Grouping. Select two shapes, group them, move them as one. This is the point where a flat Item[] stops being enough and a real scene graph earns its keep, which is why it is worth doing deliberately rather than early.

Multi-select styling and alignment. Style changes apply to one item today. Alignment and distribution are the first six operations any diagrammer reaches for.

Copy as PNG to clipboard, and duplicate with ⌘ D.

Handwriting. Labels are typed. Desert Ant Labs has models in the same family, and model.ts already isolates model concerns behind one hook, so a second recogniser for handwritten labels is a small, well contained project.

The premise stays the same though. Rough input, clean output, no server in the middle, and everything you draw stays yours.

Code & more: https://www.dailybuild.xyz/project/300-napkin