Score image processing

A fifth piece, after What I build now that the machine writes the code and Why I am not sad that the machine writes the code. Those were about how I work now. This one is about a single feature, and a detour I did not plan.

I have not much background in machine learning. I know what a neural network is in the way most developers do: I have read about it, used a few. I did train a small neural network, 20 years ago during my studies, for an assignment. So I want to explain how I ended up, this week, with a model I trained myself, which does the same job as the open source one I started from in a file 21 times smaller, and runs about 25 times faster on the same laptop.

Why a photo of sheet music.

Omnitext, my offline editor for everything, recently learned to open sheet music: you can play a score, edit it, and practise with a MIDI keyboard. The obvious next request was the paper on the music stand. Take a photo of it, press the OCR button, get a score you can play. The best open source tool I found for that is oemer, by BreezeWhite, under the MIT licence. It runs a neural network over the page, a few hundred pixels at a time, to find noteheads, stems, rests and clefs, and then a lot of careful code turns those pixels into music. The agents ported all of that to TypeScript so it could run in the browser and in the Android app. One part of oemer was trained on a dataset that only allows non-commercial research, so we replaced it with plain image processing instead.

It worked, and on my phone it took 13.6 minutes for one page. You do not wait 13.6 minutes after pressing a button. You assume it crashed.

Trying to make it faster.

The first ideas did not go far. The phone's GPU lost its device halfway through the page. Smaller number formats saved 10 to 15 percent. Running the network only over the staffs, and not over the margins and the gaps between systems, saved 40 percent, which was nice but still meant eight minutes a page. The agent that had been measuring all of this came back with the one idea that could change the order of magnitude: train a small network to copy the big one. It is called distillation, and it is not new. You use the big network as a teacher, show it thousands of examples, and train a much smaller student to give the same answers.

What made it possible here is that the teacher can label as many examples as you want, for free. MuseScore rendered pages of a couple of dozen scores, most of them public domain, in five different music fonts, oemer's network labelled every pixel, and the student learned from that. To prepare it for real photos, every training example was damaged on purpose: rotated a little, blurred, darkened on one side, sprinkled with noise, compressed as a bad JPEG, with ink showing through from the back of the page. The picture below is a sample of those examples, with the labels the student had to reproduce under each one (red for noteheads, green for stems and rests, blue for clefs and accidentals).

The training itself took 48 minutes on the graphics chip of my 2019 MacBook Pro. Everything is public in scoredit, the library behind Omnitext's score editor, and the model is on Hugging Face.

What came out.

oemer's network is a 38 MB file. The student is 1.8 MB, and it does about 23 times less arithmetic for each piece of the page. On the same laptop, in the browser, it handles a piece of the page in 0.3 seconds where the original took 7. On my test pages it gets 82.5 percent of the pitches right, and 73.7 percent of pitch and duration together, where oemer's own network gets 80.2 and 72.1. On the phone, a full page now takes 33 seconds instead of 13.6 minutes, including downloading the model the first time. Some of that phone gain comes from skipping the margins, not from the student alone, so the fair comparison for the network itself is the 21 times smaller and 23 times less work.

The result is a draft, and the picture shows it honestly: most notes and chords are right, but here it read one clef wrong, added a sharp to the key signature and lost the time signature and the beams. You still have to correct it. You just do not have to wait a quarter of an hour to start.

Better on rough pages, which I did not expect.

A student that only learns to copy its teacher should not be better than the teacher, and on clean pages it is not: a few points above on some, a few below on others. The surprise was on the rough ones. Half of my test pages are clean renders, and the other half are the same pages made to look scanned (turned by a degree, blurred, noisy, saved as a poor JPEG). On the rough version of the piano page, oemer's network got 47 percent of the pitches right. The student got 88 percent. My understanding is that oemer's network was never shown a grey, slightly crooked page while it was trained, and the student saw thousands of them. It learned to give the answer the teacher would have given on a clean copy. I should add that the student was trained with the same kind of damage it was tested on, so the real test is photos from real music stands, and I have not tested that properly yet.

What I would not claim.

I want to be careful here, because the numbers are exciting and the test is small. My test set is 14 pages, half of them simulated scans, and two identical setups can differ by a point or two on it, so "at least as good" is fair and "better" is only fair for those simulated scans. I have not compared it with newer tools like homr or Audiveris, or on a standard benchmark, and I would want both before calling this research. The training pages came from 24 scores: 15 public domain scores from OpenScore, 6 test files from OpenSheetMusicDisplay, and 3 guitar scores that their authors published on musescore.com. The model card lists exactly that.

Who wrote it.

Coding agents did the work, as they do for almost everything I build now. It would be odd to present it otherwise. My part was the decisions. I picked the permissively licensed model and left out the non-commercial one. When the phone turned out to be this slow, I asked for a smaller, faster model rather than living with it. At one point an agent was running 34 Python processes and my laptop stopped responding, so I told it, and it went back to one job at a time. Later I went out for three hours, told it the machine was all its own, and came back to a table comparing model sizes and settings, with their speed and accuracy. The agents wrote the PyTorch version of oemer's network, the renderer, the training, and the measurements. I would guess this used to take a few weeks for someone who knows the field. I do not know the field, and it took a few hours, most of which I spent on something else.

The thread.

In the earlier pieces I wrote that AI changed what is worth starting. This is the same idea in a place I did not expect it. Training a model always felt like a separate job, for other people with other hardware. This week it was one step in making a button useful, done on the laptop I already had.