DEV Community: Makoto Tanji

How I Added WebMCP to a Collaborative Sheet Music Editor

Makoto Tanji — Thu, 05 Nov 2026 14:28:58 +0000

"Rewrite these two bars, then let me hear them."

That is a reasonable request to make while working on a score. For a browser agent, though, the visible page is a difficult interface: notes are SVG shapes, several voices can occupy the same beat, and a small change in duration can break the measure.

I build ScoreTail, a browser-based sheet music editor with real-time collaboration. I recently added WebMCP tools so an agent can read and edit the score the musician already has open.

The interesting work was deciding what an edit means: how to describe music to the agent, validate its changes, and put them into the same document and undo history as a person's edits.

ScoreTail's public score viewer, showing Pachelbel's Canon in D and the playback controls.

The page exposes the operations

WebMCP is a proposed browser API for registering page-local tools with descriptions and input schemas. In ScoreTail, those tools call the application's existing score operations. The user's current selection and loaded document are already available in the tab.

There is no separate model call inside these tool handlers. The user's agent supplies the reasoning; ScoreTail supplies the score representation, validation, editing, and playback.

I split the tool surface by the page and its editing permissions:

Surface	Available tools
Public score viewer	`get_score_info`, `read_measures`, `play`, `stop_playback`
Editor with view access	The viewer tools, plus `get_selection` and `select_measures`
Editor with edit access	All of the above, plus `rewrite_measures`, `transpose`, `undo`, and `redo`

This lets someone explore a public score without signing in. Tools that change notation are only registered when the editor has edit access. Read operations also carry readOnlyHint; that hint describes behavior, while the application's access controls remain responsible for permissions.

The smaller surface is useful for the agent too. A public viewer has no reason to advertise a rewrite operation that it cannot perform.

Give the agent a representation it can read back

MusicXML is ScoreTail's interchange format. It is expressive, but it includes details an agent does not need for a small rewrite, including the sequential cursor movements used to encode simultaneous voices.

For reading and writing notes, I reuse a compact text representation from ScoreTail's built-in AI editing feature. I call it lanes-seq. A line identifies a part, staff, voice, and measure:

P1 s1 v1 m2: D5=1/4 E5=1/4 F5=1/2
P2 s1 v1 m2: G2=1

The first line contains a quarter-note D, a quarter-note E, and a half-note F. The second contains a whole-note G in another part. Durations are fractions of a whole note, and pitches are absolute: C4 means middle C. Chords use parentheses:

P1 s1 v1 m3: (C5 E5 G5)=1/2 r=1/2

read_measures returns this notation with a header describing the score's parts and musical context. The agent can then submit the same format to rewrite_measures:

{
  "start_measure": 2,
  "parts": [1],
  "lanes_seq": "P1 s1 v1 m2: D5=1/4 E5=1/4 F5=1/2"
}

This example assumes a full 4/4 measure and the indicated existing part, staff, and voice. A useful workflow reads the score first rather than guessing them.

The replacement unit is one staff in one measure. Supplying its lines replaces all of that unit's voices; leaving a unit out keeps it unchanged. This matters when an agent is asked to change a melody while preserving an accompaniment on another staff.

The tool description includes the format guide generated by the same function used for the built-in AI feature. Maintaining one guide helps keep the two interfaces consistent as the notation format evolves.

There are also practical bounds: a read call accepts up to 64 measures, and a rewrite accepts up to 32. Measure insertion and deletion require an explicit allow_measure_count_change argument.

Parse, plan, then apply

A JSON schema can establish that lanes_seq is a string. Musical structure needs another layer of validation.

For example, this is valid JSON but an incomplete replacement for a full 4/4 measure:

{
  "start_measure": 2,
  "lanes_seq": "P1 s1 v1 m2: D5=1/4"
}

The rewrite handler uses this pipeline:

tool arguments
  -> parse lanes-seq
  -> build and validate a rewrite plan
  -> start an undo gesture
  -> apply through existing score operations
  -> select the changed range

The planning stage checks things such as note lines outside the requested measure or part scope and durations that do not fill the measure correctly. It also handles notation rules such as tie and slur repair.

If parsing or planning fails, the tool returns an error starting with Nothing was changed, followed by the validation errors. The agent can correct its input and try again without leaving a partial rewrite behind.

After a successful edit, the changed range is selected in the editor. That gives the person a visible place to inspect before playing it back.

Tool calls need their own undo boundaries

ScoreTail already groups ordinary edits into undo gestures using input events such as pointerdown and keydown. A tool invocation arrives without either event.

That creates a subtle problem: consecutive agent edits need separate undo entries even when there was no human input between them.

The writing tools explicitly call beginUndoGesture() before applying a change. The rewrite then passes through the same operation layer as other edits, as one operation. A successful rewrite can be undone in one step.

One regression test specifically checks this sequence:

rewrite measure 2
save the document snapshot
transpose measure 3
undo once
assert that the document matches the saved snapshot

This catches accidental merging of two tool calls into one undo entry. Another test checks that a valid rewrite followed by one undo restores the original score.

In the current implementation, rewrites apply immediately. There is no separate preview-and-approve stage in the WebMCP handler. That makes clear tool descriptions, visible changes, and predictable undo especially valuable. The agent should still obtain whatever approval the user's request requires before making an edit.

One shared document

ScoreTail stores the live score in Yjs. WebMCP changes go through the existing operation layer into that same document, so they follow the existing synchronization path to collaborators.

Human UI ----\
              -> score operations -> Yjs document -> rendering / sync
WebMCP -----/

There is no second AI-owned score that later needs to be merged into the editor.

There is still a limit to what this guarantees. Shared document convergence does not tell us whether a change is a good arrangement, or whether another musician wanted it. Validation protects structural rules; the musician still judges the musical result.

Lifecycle and audio details

The bridge feature-detects document.modelContext and waits for the score data to be ready before dynamically loading the tools. If the API is unavailable, the normal editor continues without registering them.

Registrations use an AbortSignal. The bridge aborts it on cleanup, including when its permission-related dependencies change. Playback callbacks use a ref so registered tools can call the latest playback function without re-registering on every render.

Playback also needs to respect browser audio policies. When the page has never received user activation, the handler returns an error asking for a click. Otherwise an AudioContext.resume() can remain pending and leave the agent waiting for a result. The tool reports that situation instead of pretending playback started.

What is verified, and what remains experimental

The current WebMCP score-tool suite has 15 passing tests. It checks permission-dependent tool sets, scoped reads, successful and rejected rewrites, explicit measure-count changes, undo grouping, transposition, selection, and playback status handling.

These tests call the handlers with a real in-memory score model; playback is mocked. They establish application behavior, rather than universal compatibility with every browser agent. The public demo also loads and enters playback through its normal UI.

WebMCP itself is still experimental. Chrome's setup documentation covers its origin trial and local testing flag. A supporting browser alone also does not guarantee that a particular assistant discovers and calls the tools. I would check the browser and agent combination before planning a demo.

Try a public score

Open Canon in D in ScoreTail with a compatible WebMCP setup. The public viewer exposes reading and playback tools without requiring an account.

A starting request is:

Read the first eight measures and explain the chord progression, then play them.

For editing, copy a score into the full editor with edit access. Read a small range first, make one bounded change, listen, and try undo. WebMCP is currently attached to the full editor and public score viewer; the guest quick-edit workspace does not expose it.

I'm also working toward a library where public-domain sheet music can be opened, played, corrected, and arranged together, with OMR helping bring scanned scores into that workflow. These tools give an agent a way to assist with a specific passage while the musician stays in the same editor.

If you are adding agent tools to a creative app, how are you handling changes to a document that another person may be editing at the same time?

I Built a Real-Time Collaborative Sheet Music Editor — Here's What MusicXML and CRDTs Taught Me

Makoto Tanji — Sat, 15 Aug 2026 10:14:05 +0000

I wanted a "Google Docs for sheet music": open a browser tab, start writing notation, share a URL, and have someone else edit the same score with you in real time. Nothing like that existed. MuseScore, Finale, and Sibelius are all excellent, but they're desktop apps built for people who already know notation software inside out.

So I built ScoreTail, a browser-based score editor, mostly alone, with a lot of AI-assisted development. This post is about the parts that turned out to be genuinely hard: the MusicXML format, real-time collaborative editing over CRDTs, and rendering performance.

The AI-driven development bet

MusicXML is a huge spec — pitch, duration, voices, staves, backup/forward, divisions, ties, slurs, ornaments, tuplets, lyrics, chord symbols, dynamics, repeat structures. Implementing all of it solo, the traditional way, is a multi-year project.

Late last year I made a bet: let AI (Claude Code, ~90% of the code; Antigravity, ~10%) write most of the implementation, while I focused on defining what "correct" means — architecture decisions and test cases — rather than typing every line myself.

The thing that made this workable wasn't the AI, though, it was the test harness: I collected around 250 real-world scores (Bach, Beethoven, Chopin, Brahms, Debussy, plus the LilyPond regression test corpus) and ran them through an import → export → re-import round-trip. Any structural drift between the first and second import is a bug. AI writes code, the round-trip suite catches regressions, AI fixes them, repeat. Humans stayed in the loop for the judgment calls; the AI did the volume work.

MusicXML's backup/forward problem

The single hardest part of MusicXML to get right is <backup>/<forward>. MusicXML is a flat, sequential XML structure, but a measure isn't sequential — it has multiple simultaneous voices, and a piano part has two staves (treble/bass) sharing one measure. To represent that in a flat stream, the format writes voice A's notes, then emits a <backup> element to rewind the internal time cursor, then writes voice B's notes starting from the same beat.

That means any code that reads or writes MusicXML has to maintain its own notion of "where am I on the timeline right now" independent of document order — and get it right across combinations of backup + multi-voice + multi-staff, which is where most real-world parser bugs live.

Keeping every edit "musically valid"

Early on I decided every operation had to leave the score in a musically valid state — correctness over flexibility. Concretely: if a 4/4 measure has beat 4 occupied by a quarter note, you can't just change it to a half note, because that would overflow the measure boundary. For every edit, the editor has to decide: overflow into the next measure, auto-insert a tie, or reject the operation outright.

The harder cases are things like inserting a note when another voice already has something at that beat, notes of different lengths sharing a beat, or replacing a rest with a note that doesn't fill the remaining space cleanly. This constraint exists because the target user isn't a notation expert — someone who doesn't know music theory should still end up with a structurally correct score, not a fragile one.

Real-time collaboration: CRDTs weren't built for this

For multiplayer editing I used Yjs, a CRDT (Conflict-free Replicated Data Type) library — no server-side conflict resolution, edits from multiple clients converge automatically, the same approach Google Docs and Figma use.

Text CRDTs only have to reason about character insertion and deletion in a 1D sequence. A score has two simultaneous temporal dimensions per note — its position (which beat) and its duration (how many beats it occupies) — and multiple users can be editing different voices in the same measure concurrently, or one user can be inserting a note while another is changing the measure's time signature or key signature mid-edit.

My approach: map the MusicXML tree onto Yjs's Y.Map/Y.Array types 1:1. Yjs guarantees structural convergence — every client ends up with the same tree — but it has no idea what a musically valid tree looks like, so structural convergence and musical validity are two separate problems. The safety net is an auto-undo: if an export ever produces an invalid document, the last operation is automatically rolled back rather than left in a broken state.

Rendering with Verovio, and the ID-mapping problem

Rendering runs through Verovio, an open-source C++ engraving engine compiled to WebAssembly. Two problems showed up once scores got non-trivial in size:

Performance. Full-score re-rendering on every edit blocks the main thread for large scores. Fix: move rendering into a Web Worker, and stream rendering page-by-page, prioritizing whichever page the user is currently looking at.

Identity. Verovio has its own internal IDs for rendered SVG elements; Yjs has its own IDs for the underlying data. When a user clicks a note on screen, you need to resolve an SVG element back to the exact Yjs entry it came from. I ended up computing a fingerprint per note — part index, measure number, beat position, voice, pitch — and rebuilding the mapping table on each render pass. Building a notation renderer from scratch was on the table at one point; I shelved it as a multi-year project of its own and leaned on Verovio instead.

Click coordinates → beat position + pitch

Turning a click into "beat X, pitch Y" is deceptively harder than the equivalent problem in a text editor. A whole note is visually wide, a sixteenth note is narrow, dotted notes are wider still, and note spacing varies measure-to-measure based on how packed the measure is — so the mapping has to be derived from Verovio's actual final layout, not from a fixed grid.

For feedback, the editor shows a translucent "ghost" preview of the pending note before you commit the click: purple for a new note, blue for adding to a chord, green for replacing a rest, gray when the position conflicts with something already there.

What I chose not to build

Notation software has effectively unlimited surface area — tablature, drum notation, transposing instruments, automatic part extraction, fine-grained layout control, and on. As a (mostly) solo project, the operative question stayed "what helps the most people right now," and a lot got deliberately cut or deferred. Current honest state: a 5–10 minute piano score performs well; large orchestral scores are still rough.

MusicXML's other trap is that the spec and what software actually emits diverge — MuseScore and Dorico export structurally different XML for the same piece. There's no shortcut around this beyond feeding the parser a large, varied corpus of real files and absorbing the "dialects" one by one.

Try it

ScoreTail is free, browser-based, and works without an account for a quick edit. If you've dealt with MusicXML, CRDT-based collaborative editing, or WASM rendering engines, I'd be curious to hear how you approached the same problems.