In CommandCanvas, a sketch can become a structured diagram beside its source. A person can rearrange the result, a collaborator can edit it, and a supported agent host can address it through WebMCP. Each operation refers to objects in the same room, with identities and versions that survive changes in how people interact with them.
I built the shared canvas mutations around one commit path. Pointer input, voice, and tools can propose changes, but the server must establish which member authorized a change and whether it still applies to the current room state. Adding another input should not introduce another definition of what counts as saved.
Objects that people and tools can both address
A note in CommandCanvas has an identity, a title, a position, dimensions, and a version. Its content follows a note schema. A task board has columns and tasks. A sketch has strokes and points. A structured diagram has its own payload and can retain a reference to the sketch it came from.
These objects share spatial behavior while keeping their different kinds of content. Moving a board should preserve its tasks. Resizing a sketch should preserve the drawing. An operation can address a particular object and check its version before changing it. The object model makes those distinctions explicit.
That gives a tool request something concrete to address: the selected sketch, a populated board, or an existing object to transform. It also constrains the result. A model response has to fit a supported object schema before the application can submit it as a canvas change.
The distinction is useful when people and an agent take turns. They can refer to the same object without reconstructing its content from a screenshot or carrying a separate copy into another conversation.
Where a canvas change commits
The browser holds the visible objects, current selection, and room state. Direct input produces canvas commands. WebMCP and optional embedded voice pass through a capability runtime that validates their arguments and current context before invoking the command adapters.
WebMCP exposes capabilities to a supported agent host. Embedded voice uses a separate OpenAI Realtime connection and a narrower set of capabilities. Both reach the application's command adapters; neither gets its own route around room membership or revision checks.
The durable room path looks like this:
flowchart TD
subgraph browser[Browser workspace]
direct[Pointer, touch and hand intents] --> command[Canonical canvas command]
tools[WebMCP and optional voice] --> capability[Capability validation]
capability --> command
end
subgraph server[Server and database]
api[Authenticated room API] --> guards[Membership and revision checks]
guards --> commit[Postgres mutation transaction]
commit --> records[Objects, room revision and receipt]
end
command -->|Command, source and base revision| api
records -->|Authoritative readback| origin[Originating canvas]
records -->|Revision notification| reload[Other clients reload room state]
reload --> peers[Collaborator canvases]
presence[Presence and cursor Broadcast] -.-> origin
presence -.-> peersThe browser proposes a command; the server establishes who may commit it. Solid arrows trace the durable command and synchronization path. Dotted arrows show participant presence and cursor traffic, which do not create mutation receipts. On narrow screens, scroll the diagram horizontally.
The request carries a command ID, its reported input source, and the room revision the client worked against. The HTTP route verifies the bearer token. The room service loads membership, derives the actor, reads the current canvas, and builds a mutation plan. The database operation checks the expected revision again when committing the change.
That second check matters because the room can change between the server reading it and attempting the write. A command prepared against revision 12 cannot commit against revision 13 by assuming its original view still applies. The stale request is rejected.
The shared revision makes the order of committed changes explicit. It also makes conflicts coarser: two people working on different objects can still contend for the same room revision. The application does not automatically merge those edits. That is the cost of using a single room revision as a commit precondition.
The transaction persists the object changes, advances the room revision, and writes the receipt. The server reloads the authoritative state and checks for the expected receipt before returning success. Other clients receive a compact revision notification and reload the room. The room service contains that commit and readback path.
Why drag previews stay out of history
Applying that commit path to every cursor sample would make ordinary movement compete with object edits for room revisions. It would also fill history with intermediate positions that are of little use when someone wants to undo a completed action.
Cursors and movement previews therefore remain ephemeral. Supabase Presence describes connected participants, and Broadcast carries the frequent updates. Stable object changes go through the mutation path and produce the receipt that other clients can verify.
The room can show motion before it has a new committed state. That distinction is necessary for responsive interaction, and it has to survive into the UI: seeing another person's cursor move is not evidence that their edit was saved.
Turning a sketch into another object
Sketch interpretation introduces a longer gap between preparing a change and trying to commit it. A provider can finish its work after the drawing it received has changed.
The transformation captures the selected sketch's ID and version, then rasterizes it to a PNG in the browser. A provider interprets that image and returns structured output. The application validates the payload, checks the source reference, and checks that the requested output kind agrees with the result.
The person can keep drawing while the provider works. Before submitting the new object, the transformation checks the source version again. If the sketch changed, interpretation may have succeeded, but the application refuses to create the diagram from that result. The provider time has already been spent; accepting its answer anyway would attach an outdated interpretation to the current work.
sequenceDiagram
participant C as Canvas
participant P as Interpretation service
participant R as Room API
C->>C: Capture sketch ID and version
C->>P: Rasterized sketch and instructions
P-->>C: Structured result
C->>C: Validate payload and recheck source
alt Source unchanged
C->>R: Create linked diagram at current room revision
R-->>C: Committed object and receipt
else Source changed
C->>C: Refuse the stale transformation
endThe generated diagram becomes a separate canvas object. The original sketch remains available, including when interpretation fails or its source has changed. On narrow screens, scroll the diagram horizontally.
Creating the result still uses the ordinary command path. After the save, the transformation looks for the new diagram and its receipt in the authoritative returned state. A provider response alone is not enough to report that the room contains the diagram.
The source-version check and the room-revision check address different changes. One checks the drawing used for interpretation; the other protects the shared commit. Neither establishes that the model understood the drawing correctly.
That is why the result appears beside the preserved source. Someone can compare them, point out a missing relationship, or try another interpretation without recovering an overwritten sketch. The output remains a structured object that later commands can address. These checks and the separate-object creation are implemented in the sketch transformation orchestrator.
Deciding when a hand is drawing
The commit checks only help after an input has produced the right command. Hand tracking has an earlier decision to make: whether a moving fingertip should create ink at all. A valid room membership and a current revision cannot answer that.
On the default local path, an opt-in camera feeds MediaPipe Hand Landmarker in a browser worker. The interaction layer receives landmarks and decides what action they may produce. The index fingertip supplies the pen position. A separate thumb-to-middle-finger clutch supplies pen-down intent.
That separation lets someone move their hand to the start of another stroke without drawing a line across the intervening space. Closing the clutch engages the pen; opening it lifts the pen. A later engagement begins another stroke with its own identity.
The clutch uses different engagement and release thresholds, along with temporal confirmation. This hysteresis prevents a small amount of motion near one threshold from repeatedly switching the pen on and off. The drawing policy also distinguishes provisional thresholds from calibrated ones.
Those rules are testable, but a state-machine test cannot tell me whether the interaction feels comfortable after ten minutes, or whether it holds up under poor lighting and partial occlusion. Physical-hand usability remains experimental. Pointer, touch, and typed controls keep the workspace usable while that work continues.
The default hand path processes camera frames locally. Asking for sketch interpretation is a separate operation: it sends an image of the selected drawing for interpretation. The distinction matters when explaining what leaves the device.
The agent is not the actor
Once several inputs could change the same room, the receipt needed to describe both the member responsible for a committed change and the path that requested it.
The earlier schema already had a separate source column. Its rule was too restrictive: a WebMCP source could only accompany an agent actor. The receipt still retained the authorizing user's ID and required room membership, but its actor classification and display name presented the action as belonging to a generic agent.
The correction binds durable WebMCP canvas mutations to the authenticated member. A host is classified as human; another room member is classified as participant. The source remains webmcp. The SQL wrapper derives the classification from stored membership:
v_effective_actor_type := case v_member_role
when 'host' then 'human'
when 'participant' then 'participant'
else null
end;
This aligns the receipt's visible attribution with the identity that authorized the request. It also allows the reader to distinguish a tool-originated change from direct interaction.
The compatibility work was less visible. The wrapper accepts request shapes from both releases and canonicalizes them before persistence. The table constraint continues to permit the historical agent-plus-WebMCP shape so older rows remain valid; a stricter validator in the write path enforces the new behavior. The migration adds the broader constraint without scanning historical rows immediately, and a separate migration performs validation. The attribution migration shows both responsibilities.
There are limits to what this receipt says. The source field records an application input path; it does not authenticate a particular external agent host. The current service also normalizes most participant inputs to collaborator, preserving webmcp and system separately. It therefore cannot answer every question about whether a remote participant used touch, voice, or a pointer.
The shared write path gives those receipts a consistent meaning across supported inputs. Each refers to committed work, identifies the affected objects, and can support guarded undo.
Reviewing the work before it leaves the room
The optional participant filmstrip keeps a conversation beside the canvas. Its small peer-to-peer WebRTC connection uses Supabase signaling and remains separate from canvas persistence and embedded voice.
Sending a meeting packet introduces another boundary. A saved canvas object does not authorize an email. The packet has its own workflow: preparation creates a snapshot, and approval binds the content and recipients being reviewed. A tool can stage a send request, but the host must perform the final Send action before the configured server transport may submit it.
That requires an explicit review step after the room has produced useful work. The host approves the packet that will leave the room, with its intended recipients. Actual email delivery remains a separate result to verify.
What the recorded checks establish
I built CommandCanvas intending to submit it to the WebMCP Challenge, but didn't submit it.
The verification ledger records native Chrome WebMCP execution, a real voice-provider call that created a board, a real sketch-interpretation request that created structured output beside its source, and two browser clients converging on a durable mutation and receipt. Those are individual recorded checks, rather than proof that the whole intended experience has passed on physical devices.
The documented gaps include a ChatGPT built-in-browser Site Tools invocation, final physical-hand and microphone ergonomics, real email delivery, and cross-network TURN traversal. WebMCP's shared-page model is described in the official Site Tools documentation; host support and an observed invocation still have to be established separately for CommandCanvas.
The next useful rehearsal is a complete session with two people: draw something, explain it, create structured work from it, let the other person change it, and review the outcome together. I want to see whether revision conflicts interrupt that sequence, where selection becomes unclear, and whether the hand controls earn their place beside the pointer. Those are questions about working in the room that individual commit checks cannot settle.
🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series
Top comments (0)