I have a habit of starting projects because I want one very specific thing.
Then somewhere along the way I apparently decide, "Well, if I'm already doing this..."
And suddenly the tiny experiment has a database, Docker Compose stack, job queue, model manager, audio editor, waveform player, and enough GPU models to eat 40 GB of disk space.
So, anyway.
I made Miso.
Miso is a self-hosted local music generation and remix studio built on top of audio.cpp.
The original idea was pretty simple: I wanted a nice interface for experimenting with local music models without constantly dropping back to command-line tools, juggling generated files, or trying to remember which ridiculous combination of parameters made the song I actually liked.
It got slightly out of hand.
I didn't just want "type prompt, get song"
There are already plenty of ways to generate a song from a text box.
That part is fun, but the thing I kept wanting was everything that comes after generation.
What if I like the song except for eight seconds in the middle?
What if I want the vocals by themselves?
What if I want to change the singer?
What if I want the same melody arranged in a completely different style?
What if I want to turn something into MIDI and keep screwing with it somewhere else?
That became the real idea behind Miso.
A project doesn't just hold the final song. It keeps the takes, prompts, lyrics, stems, scores, transcriptions, and the lineage between them, so I can experiment without losing track of how I got there.
There are a ridiculous number of models involved
Miso currently exposes several different music and audio model families through audio.cpp because they are good at very different things.
ACE-Step 1.5 is the main workhorse. It handles full song generation, covers, and repainting sections of existing tracks.
YuE2 can generate songs too, but it does something I find especially interesting: it plans an ABC music score first and can give that score back to you.
MiniMax Music 3 and HeartMuLa are alternate song generators with their own prompting styles.
Stable Audio 3 handles instrumental generation and sound effects.
Then there are separate models for stem separation, voice conversion, and audio-to-MIDI transcription.
The funny part is that they absolutely do not accept prompts the same way.
One model wants a production-style caption.
Another wants a short description plus comma-separated tags.
ACE-Step wants style information in one place, BPM and key as separate parameters, and lyrics somewhere else.
So Miso has a guided builder that translates the same set of UI controls into whatever syntax the selected model actually expects.
I wanted to be able to think about "upbeat synthpop, female vocal, 120 BPM" instead of remembering which API field each model decided should contain "120 BPM."
Repainting is interesting
One of the features Miso is most built around is repainting.
You load a take, drag across part of its waveform, and replace only that section.
So instead of regenerating a whole song because one chorus went weird, you can select that section and try again.
ACE-Step's repaint route turned out to have some quirks that were interesting to figure out, too.
The surrounding music influences the replacement much more strongly than the text prompt does. Lyrics have more control over a vocal repaint than telling the prompt to suddenly turn the section into something wildly different.
That wasn't something I wanted to guess about, so I ended up testing it.
At one point I ran deliberately opposite prompts through the route and compared the results. The normal text-to-music route reacted dramatically. Repaint barely cared.
Which is one of the reasons I enjoy projects like this. Half of building the interface becomes figuring out what these models actually do rather than what I assume they do from an API description.
Then I started pulling songs apart
Miso can also split a track into stems.
Depending on the model, that can mean:
- vocals + instrumental
- vocals + drums + bass + everything else
Once the stems exist, they get their own mixer.
You can solo them, mute them, change their levels, swap a vocal using a voice conversion model, mix everything back together, and save the result as another take.
That led to another rabbit hole: voice conversion.
RVC sounds better but only has a small set of packaged voices.
Vevo2 has lower output bandwidth, but it can use basically any other vocal reference in the project.
So both stayed.
I would rather expose two tools with different tradeoffs than pretend one is objectively the "correct" choice.
The part I just finished is more interesting
The current release, 1.4.0, connects MIDI transcription to YuE2's score support.
Previously, Miso could already:
- split a song into stems
- transcribe audio into MIDI notes
- give YuE2 an ABC score and tell it to sing that melody
The pieces just weren't connected.
Now they are.
You can separate a song, transcribe the vocal stem, turn those note events into an ABC melody, inspect the detected tempo and key, correct them if necessary, and feed the score into YuE2 with a completely different style.
So you can take the melody from a recording and use it as the basis for a new arrangement.
I tested that path with a known melody instead of just listening and declaring, "yeah, sounds about right."
I generated a simple version of Twinkle Twinkle Little Star, ran it back through separation, transcription, and score conversion, and recovered the expected pitches across the recording.
Rhythm is less perfect than pitch, and real vocals can produce odd little tied or split notes, but the result is usable by the model.
That was one of those features where suddenly several earlier pieces of the project made much more sense together.
pinkpixel-dev
/
miso
A local music generation and remix studio built on audio.cpp. Generate, repaint, split into stems and swap voices, with every take kept and traceable.
Miso
A local music generation and remix studio. You bring a prompt or a song, and Miso gives you a real workspace for generating, remixing, splitting, and finishing music with models that run on your own machine. Nothing is sent anywhere.
Miso runs on audio.cpp, a C++ inference runtime for audio models. Miso is the studio around it: projects that persist, a history of every take and a record of exactly how each clip was made so you can change one thing and try again.
Miso installs models, keeps your projects and audio, generates with ACE-Step from a guided prompt builder with lyrics written for you if you want them, repaints a section of a track, covers a take, holds any two takes against each other, splits a song into stems you can mix and export, writes full 48 kHz stereo songs with YuE2 and hands back the score…
Miso is local-first
The whole thing runs as two Docker containers.
One runs Miso itself.
The other runs audio.cpp and owns the GPU models.
The browser never talks directly to audio.cpp. It talks to Miso, and Miso talks to the inference backend.
That separation became useful almost immediately.
Projects, imports, playback, exports, and editing still work if the model backend is stopped.
It also means the audio.cpp server can live on another machine with the GPU while Miso runs somewhere else.
The app itself uses React, TypeScript, Vite, Node, Hono, and SQLite.
I deliberately kept most ordinary audio editing in the browser rather than turning everything into another server job.
Things like trimming, fading, gain changes, normalization, splitting, WAV conversion, and MP3 export do not need a giant AI model involved.
Sometimes software can just... do the normal thing.
Wild concept, I know.
There's an optional AI assistant too
I did sneak one language model feature in.
Miso can optionally connect to an OpenAI-compatible API or a local llama.cpp server to help write lyrics or expand a music prompt.
But it is completely separate from the music models.
Nothing gets silently rewritten.
It gives you a suggestion, shows it to you, and you decide whether to use it.
That was important to me because I don't want an assistant "improving" a prompt behind my back and then making it impossible to figure out why two generations were different.
If you accept an expanded prompt, Miso keeps both the original and the version that actually ran.
This project taught me a lot about building around AI models
Not really about calling them.
That part is usually the easy bit.
The interesting problems were things like:
- models that require different sample rates
- models that return multiple huge audio files
- models that need several components loaded in sequence to fit in VRAM
- jobs that can run for several minutes
- uploads that need to survive browser tabs closing
- model APIs that technically accept a parameter but quietly ignore it if you put it in the wrong part of the request
- keeping enough generation history to reproduce something useful
- dealing with a backend that can only run one expensive job at a time
- determining which behavior belongs in the UI versus the server versus the inference runtime
And, of course, learning that "the documentation says this option does X" and "the model meaningfully responds to X" are not always the same statement.
That last one is probably applicable to a lot more than music models.
It's still very much a project
Miso is open source, and right now it has mostly been tested on my own Linux machine with an RTX 4090 laptop GPU.
So I am absolutely not going to pretend I have tested every GPU, driver, operating system, or model combination in existence.
But it has grown from "I want a good way to play around with local music models" into something I actually think is pretty awesome.
And that is my favorite kind of project.
The kind where I started building a tool because I wanted to experiment with something...
...and then ended up building the experiment itself.
Top comments (9)
okay yeah this is so cool😭
the whole lineage/take history thing is such a smart choice because AI workflows get messy SO fast. being able to look back and actually understand “wait, which prompt / stem / take got me here?” is huge.
also massive respect for normalizing all those completely different model interfaces into one workflow because that is exactly the kind of engineering problem nobody thinks about until they try to actually build something usable lol
and the “sometimes software can just do the normal thing” philosophy is honestly my favorite part. trimming, fading, exporting, etc. absolutely do not need AI sprinkled on top just because AI is involved somewhere else. it makes the whole project feel way more intentional instead of AI-for-the-sake-of-AI.
I'm looking forward to working with local models, though I have no clue where to start lol!
seriously cool project 💚
Ahh thank you 💖 and YES, the lineage/history thing became way more important than I expected once I started actually using it. AI audio gets messy fast when you’ve got generations, repaints, stems, converted vocals, mixes, etc. and suddenly you’re like “wait... what the hell did this come from?” 😂
And the different model interfaces were definitely one of those “this seems simple until you actually build it” problems. They all want basically the same musical ideas expressed in completely different ways, so making that feel like one coherent workflow was a huge part of the project.
Also I’m 100% with you on the normal-software part. I really didn’t want to shove AI into every single feature just because the app uses AI elsewhere. Sometimes trim should just trim lol.
And honestly local models are way less scary once you start playing with them. I have a feeling you’d have a ridiculous amount of fun with them 😂💖
Hello Salika,
I hope you're doing well.
I have a good business idea that I'd love to discuss with you in more detail.
To give you some background, a friend of mine started this business with a U.S.-based partner three years ago. Since then, he's been paying his partner between $8,000 and $10,000 per month, and the business has been working well.
If you're interested in learning more, I'd be happy to share the details.
Whatsapp: +81 70-9427-3751
Telegram: @ono0319
Best regards,
Ono
A repaint can improve the selected section and still fail at its edit boundaries because the replacement may not share the original phase or ambience. Did you test those joins separately from the section's musical quality?
That’s a really good point. I tested the repaint results mostly for prompt responsiveness and whether the replacement itself behaved the way I expected, but I didn’t separately measure the edit boundaries for phase/ambience continuity.
In practice I’ve mostly been judging the joins by ear so far, especially on shorter repaint regions. That’s definitely something worth testing more deliberately though, because a musically good replacement can still sound wrong if the seam gives it away.
The "documentation says X but the model actually responds to Y" line hits home - ran into the same thing wiring local model backends into CogniRunner, where a parameter was silently ignored unless it was nested exactly where the model's own reference implementation expected it, not where the API schema claimed it belonged. The Docker split keeping playback and editing alive while the inference container is down is the right call too - a lot of local-AI tools tie the UI to the same process serving the model, and the whole thing locks up the moment a generation job backs up.
Yes, exactly. The silent-parameter thing is especially nasty because nothing necessarily fails. You get a valid result back, it just completely ignored the setting you thought you changed, which can waste a ridiculous amount of time.
And that separation between Miso and the inference backend ended up being one of the better architectural decisions in the project. I really wanted the library, playback, editing, exports, etc. to still feel like normal software even if the model server is busy or completely down.
Local model tooling is full of these tiny integration traps that only show up once you stop doing single happy-path calls and try to build an actual usable app around them 😅
The audio.cpp-on-a-separate-machine split is the detail worth stealing - browser talks to Miso, Miso talks to inference, so the app half doesn't care whether the GPU box is even up. I've been doing something close to that splitting compute across two Macs on a LAN, and decoupling the transport like that is what makes the rest (editing, export, playback) keep working when the model server doesn't.
On the repaint quirk - sounds like repaint starts denoising from something closer to the original waveform than a fresh noise draw would, so there's less room left for the prompt to move it. Did ACE-Step's code confirm that, or is it still inferred from the opposite-prompt test?
Sharded the model across two GPUs after wiping 12GB VRAM to avoid OOM; the audio tokenizer’s hallucination trap was the real nightmare. Any other local AI music hacks you swear by?