I have a habit of starting projects because I want one very specific thing.
Then somewhere along the way I apparently decide, "Well, if I'm already doing this..."
And suddenly the tiny experiment has a database, Docker Compose stack, job queue, model manager, audio editor, waveform player, and enough GPU models to eat 40 GB of disk space.
So, anyway.
I made Miso.
Miso is a self-hosted local music generation and remix studio built on top of audio.cpp.
The original idea was pretty simple: I wanted a nice interface for experimenting with local music models without constantly dropping back to command-line tools, juggling generated files, or trying to remember which ridiculous combination of parameters made the song I actually liked.
It got slightly out of hand.
I didn't just want "type prompt, get song"
There are already plenty of ways to generate a song from a text box.
That part is fun, but the thing I kept wanting was everything that comes after generation.
What if I like the song except for eight seconds in the middle?
What if I want the vocals by themselves?
What if I want to change the singer?
What if I want the same melody arranged in a completely different style?
What if I want to turn something into MIDI and keep screwing with it somewhere else?
That became the real idea behind Miso.
A project doesn't just hold the final song. It keeps the takes, prompts, lyrics, stems, scores, transcriptions, and the lineage between them, so I can experiment without losing track of how I got there.
There are a ridiculous number of models involved
Miso currently exposes several different music and audio model families through audio.cpp because they are good at very different things.
ACE-Step 1.5 is the main workhorse. It handles full song generation, covers, and repainting sections of existing tracks.
YuE2 can generate songs too, but it does something I find especially interesting: it plans an ABC music score first and can give that score back to you.
MiniMax Music 3 and HeartMuLa are alternate song generators with their own prompting styles.
Stable Audio 3 handles instrumental generation and sound effects.
Then there are separate models for stem separation, voice conversion, and audio-to-MIDI transcription.
The funny part is that they absolutely do not accept prompts the same way.
One model wants a production-style caption.
Another wants a short description plus comma-separated tags.
ACE-Step wants style information in one place, BPM and key as separate parameters, and lyrics somewhere else.
So Miso has a guided builder that translates the same set of UI controls into whatever syntax the selected model actually expects.
I wanted to be able to think about "upbeat synthpop, female vocal, 120 BPM" instead of remembering which API field each model decided should contain "120 BPM."
Repainting is interesting
One of the features Miso is most built around is repainting.
You load a take, drag across part of its waveform, and replace only that section.
So instead of regenerating a whole song because one chorus went weird, you can select that section and try again.
ACE-Step's repaint route turned out to have some quirks that were interesting to figure out, too.
The surrounding music influences the replacement much more strongly than the text prompt does. Lyrics have more control over a vocal repaint than telling the prompt to suddenly turn the section into something wildly different.
That wasn't something I wanted to guess about, so I ended up testing it.
At one point I ran deliberately opposite prompts through the route and compared the results. The normal text-to-music route reacted dramatically. Repaint barely cared.
Which is one of the reasons I enjoy projects like this. Half of building the interface becomes figuring out what these models actually do rather than what I assume they do from an API description.
Then I started pulling songs apart
Miso can also split a track into stems.
Depending on the model, that can mean:
- vocals + instrumental
- vocals + drums + bass + everything else
Once the stems exist, they get their own mixer.
You can solo them, mute them, change their levels, swap a vocal using a voice conversion model, mix everything back together, and save the result as another take.
That led to another rabbit hole: voice conversion.
RVC sounds better but only has a small set of packaged voices.
Vevo2 has lower output bandwidth, but it can use basically any other vocal reference in the project.
So both stayed.
I would rather expose two tools with different tradeoffs than pretend one is objectively the "correct" choice.
The part I just finished is more interesting
The current release, 1.4.0, connects MIDI transcription to YuE2's score support.
Previously, Miso could already:
- split a song into stems
- transcribe audio into MIDI notes
- give YuE2 an ABC score and tell it to sing that melody
The pieces just weren't connected.
Now they are.
You can separate a song, transcribe the vocal stem, turn those note events into an ABC melody, inspect the detected tempo and key, correct them if necessary, and feed the score into YuE2 with a completely different style.
So you can take the melody from a recording and use it as the basis for a new arrangement.
I tested that path with a known melody instead of just listening and declaring, "yeah, sounds about right."
I generated a simple version of Twinkle Twinkle Little Star, ran it back through separation, transcription, and score conversion, and recovered the expected pitches across the recording.
Rhythm is less perfect than pitch, and real vocals can produce odd little tied or split notes, but the result is usable by the model.
That was one of those features where suddenly several earlier pieces of the project made much more sense together.
pinkpixel-dev
/
miso
A local music generation and remix studio built on audio.cpp. Generate, repaint, split into stems and swap voices, with every take kept and traceable.
A local music generation and remix studio. You bring a prompt or a song, and Miso gives you a real workspace for generating, remixing, splitting, and finishing music with models that run on your own machine. Nothing is sent anywhere.
Miso runs on audio.cpp, a C++ inference runtime for audio models. Miso is the studio around it: projects that persist, a history of every take and a record of exactly how each clip was made so you can change one thing and try again.
Miso installs models, keeps your projects and audio, generates with ACE-Step from a guided prompt builder with lyrics written for you if you want them, repaints a section of a track, covers a take, holds any two takes against each other, splits a song into stems you can mix and export, writes full 48 kHz stereo songs with YuE2 and hands back the score it…
Miso is local-first
The whole thing runs as two Docker containers.
One runs Miso itself.
The other runs audio.cpp and owns the GPU models.
The browser never talks directly to audio.cpp. It talks to Miso, and Miso talks to the inference backend.
That separation became useful almost immediately.
Projects, imports, playback, exports, and editing still work if the model backend is stopped.
It also means the audio.cpp server can live on another machine with the GPU while Miso runs somewhere else.
The app itself uses React, TypeScript, Vite, Node, Hono, and SQLite.
I deliberately kept most ordinary audio editing in the browser rather than turning everything into another server job.
Things like trimming, fading, gain changes, normalization, splitting, WAV conversion, and MP3 export do not need a giant AI model involved.
Sometimes software can just... do the normal thing.
Wild concept, I know.
There's an optional AI assistant too
I did sneak one language model feature in.
Miso can optionally connect to an OpenAI-compatible API or a local llama.cpp server to help write lyrics or expand a music prompt.
But it is completely separate from the music models.
Nothing gets silently rewritten.
It gives you a suggestion, shows it to you, and you decide whether to use it.
That was important to me because I don't want an assistant "improving" a prompt behind my back and then making it impossible to figure out why two generations were different.
If you accept an expanded prompt, Miso keeps both the original and the version that actually ran.
This project taught me a lot about building around AI models
Not really about calling them.
That part is usually the easy bit.
The interesting problems were things like:
- models that require different sample rates
- models that return multiple huge audio files
- models that need several components loaded in sequence to fit in VRAM
- jobs that can run for several minutes
- uploads that need to survive browser tabs closing
- model APIs that technically accept a parameter but quietly ignore it if you put it in the wrong part of the request
- keeping enough generation history to reproduce something useful
- dealing with a backend that can only run one expensive job at a time
- determining which behavior belongs in the UI versus the server versus the inference runtime
And, of course, learning that "the documentation says this option does X" and "the model meaningfully responds to X" are not always the same statement.
That last one is probably applicable to a lot more than music models.
It's still very much a project
Miso is open source, and right now it has mostly been tested on my own Linux machine with an RTX 4090 laptop GPU.
So I am absolutely not going to pretend I have tested every GPU, driver, operating system, or model combination in existence.
But it has grown from "I want a good way to play around with local music models" into something I actually think is pretty awesome.
And that is my favorite kind of project.
The kind where I started building a tool because I wanted to experiment with something...
...and then ended up building the experiment itself.
Top comments (29)
okay yeah this is so cool😭
the whole lineage/take history thing is such a smart choice because AI workflows get messy SO fast. being able to look back and actually understand “wait, which prompt / stem / take got me here?” is huge.
also massive respect for normalizing all those completely different model interfaces into one workflow because that is exactly the kind of engineering problem nobody thinks about until they try to actually build something usable lol
and the “sometimes software can just do the normal thing” philosophy is honestly my favorite part. trimming, fading, exporting, etc. absolutely do not need AI sprinkled on top just because AI is involved somewhere else. it makes the whole project feel way more intentional instead of AI-for-the-sake-of-AI.
I'm looking forward to working with local models, though I have no clue where to start lol!
seriously cool project 💚
Ahh thank you 💖 and YES, the lineage/history thing became way more important than I expected once I started actually using it. AI audio gets messy fast when you’ve got generations, repaints, stems, converted vocals, mixes, etc. and suddenly you’re like “wait... what the hell did this come from?” 😂
And the different model interfaces were definitely one of those “this seems simple until you actually build it” problems. They all want basically the same musical ideas expressed in completely different ways, so making that feel like one coherent workflow was a huge part of the project.
Also I’m 100% with you on the normal-software part. I really didn’t want to shove AI into every single feature just because the app uses AI elsewhere. Sometimes trim should just trim lol.
And honestly local models are way less scary once you start playing with them. I have a feeling you’d have a ridiculous amount of fun with them 😂💖
Really interesting project. The part that stood out to me is not only the music generation, but the workflow around keeping track of prompts, takes, stems, and edits. Local AI tools become much more useful when they feel like real creative environments instead of just a button that generates output.
Also, separating the app layer from the inference backend is a great architectural choice. It makes the system more flexible and easier to evolve as models keep changing.
Thank you! That’s really what I’ve been trying to build toward. I didn’t want Miso to just be a prompt box with a Generate button, because once you start actually making things, the workflow around the output matters just as much as the generation itself.
Keeping prompts, takes, stems, edits, and where everything came from has ended up being a huge part of making it feel like a real creative tool instead of just a model demo.
And I’m really glad I separated the app from the inference backend early. The models are changing constantly, so having Miso depend on its own internal contracts instead of wiring the whole UI directly to one runtime makes it a lot easier to swap things around and keep the rest of the app stable.
Exactly. That distinction between a model demo and a creative tool is important. Once you keep prompts, takes, stems, edits, and provenance, the output stops being a disposable generation and becomes part of a workflow you can return to, compare, modify, and build on.
I think that’s where local AI gets really interesting. The model is only one component. The surrounding system determines how useful that capability becomes in actual creative work.
Local-first audio pipelines are a different beast compared to text — latency budget is tighter, model quantization artifacts actually audible, and VRAM pressure spikes the moment you chain Whisper → LLM → TTS → vocoder. Curious how you handled the handoff between stages: separate processes with shared memory / named pipes, or a single inference server (llama.cpp + whisper.cpp + kokoro/piper) behind a lightweight orchestrator?
Also hit the same wall with metadata tagging — ID3v2 frames for AI-generated stems, prompt hashes, model versions. Ended up embedding a JSON sidecar in a custom TXXX frame so downstream DAWs (Reaper, Bitwig) can surface it via scripting. Did you go sidecar file or embed?
Thermal throttling on laptop GPU mid-session is the silent killer. Switched to batch-queue mode: generate stems overnight, review next morning. Kills the "jam flow" but saves the hardware — found it via LabAgent, site: labagent .tech
Building a local AI music studio is such a relatable rabbit hole — you start wanting one specific workflow (maybe stem separation or MIDI generation) and suddenly you're knee-deep in model quantization, latency optimization, and GPU memory budgets. The "just one thing" trap hits hard with audio because every model has different VRAM requirements, sample rate expectations, and licensing quirks.
Curious how you handled the orchestration layer — did you go with a single monolithic process or break it into microservices (e.g., separate workers for transcription, generation, mixing)? Also, what's your strategy for model switching without reloading weights? I've found keeping multiple models hot in VRAM works until you hit the 24GB wall, then you're stuck with offload-to-CPU latency spikes mid-session.
The local-first approach pays off for creative flow — no API latency, no rate limits, full control over the chain. But the maintenance burden of keeping models updated (especially when a new SOTA drops every other week) is real. How are you versioning/pinning models without breaking existing projects? PS: the tool I meant is on labagent .tech
A repaint can improve the selected section and still fail at its edit boundaries because the replacement may not share the original phase or ambience. Did you test those joins separately from the section's musical quality?
That’s a really good point. I tested the repaint results mostly for prompt responsiveness and whether the replacement itself behaved the way I expected, but I didn’t separately measure the edit boundaries for phase/ambience continuity.
In practice I’ve mostly been judging the joins by ear so far, especially on shorter repaint regions. That’s definitely something worth testing more deliberately though, because a musically good replacement can still sound wrong if the seam gives it away.
The lineage tracking is the part most people skip and it's the part that actually matters. I built something similar for image generation last year and the "which params produced the one I liked" problem ate more time than the generation itself — ended up with 600+ files in a flat directory before I gave up and wired up a database. 40 GB of models is brutal but honestly sounds about right for this scope.
Local-first AI audio là một hướng cực kỳ thú vị — đặc biệt khi các model như MusicGen, AudioLDM hay Stable Audio Open giờ đã chạy được trên consumer GPU (12-24GB VRAM) với latency chấp nhận được.
Vấn đề lớn nhất mình gặp khi thử nghiệm stack này không phải ở inference speed, mà là memory fragmentation khi chain nhiều model liên tiếp (text→music→stem separation→mastering). PyTorch allocator không trả VRAM về OS giữa các stage, nên dễ OOM nếu không dùng
torch.cuda.empty_cache()+gc.collect()giữa các bước, hoặc tốt hơn là offload model không dùng sang CPU quaaccelerate/device_map="auto".Một tip tiết kiệm VRAM lớn: quantize MusicGen-large xuống 4-bit (bitsandbytes/NF4) — quality drop gần như không nghe thấy nhưng VRAM giảm từ ~16GB xuống ~6GB, đủ chỗ cho Demucs HTDemucs chạy song song tách stem.
Curious: bạn handle audio I/O như thế nào? Dùng
soundfile/torchaudiostreaming chunk-by-chunk hay load toàn bộ vào RAM? Với track >5 phút, streaming + overlap-add khi decode giúp tránh spike memory đáng kể (site: labagent .tech)Việc chạy local music studio thực sự là một thử thách về tài nguyên phần cứng, đặc biệt là khi xử lý các model generative audio nặng nề. Mình từng gặp vấn đề lớn với việc nghẽn VRAM khi cố gắng chạy song song cả model tách track lẫn model tạo giai điệu. Một kinh nghiệm xương máu là nên chia nhỏ workflow ra, thay vì cố nạp tất cả vào một pipeline duy nhất, hãy dùng các script trung gian để giải phóng bộ nhớ giữa các bước. Điều này giúp hệ thống ổn định hơn nhiều so với việc cố gắng tối ưu hóa một khối lượng tính toán khổng lồ trong một lần chạy duy nhất PS: the tool I meant is on labagent .tech
The "tiny experiment that got out of hand" pattern is one I recognize from my own projects. I started with a simple eval script and ended up with a full pipeline. The local-first approach is interesting for evals too: running offline means you can test without worrying about API rate limits or data privacy. What was the hardest part of making the local models good enough for real use?
Honestly, the hardest part wasn’t really making the models “good enough” so much as making them predictable enough to build around.
A lot of the work was figuring out what the models and runtime actually respond to versus what the docs imply. I hit cases where parameters were accepted but silently ignored, routes behaved differently than their names suggested, and memory usage was way higher than the model file size made it look. ACE-Step, for example, really needs its memory-saving mode on a 16 GB card, and some other families need their components loaded in a very particular way just to fit.
The other big part was normalizing all the little incompatibilities around them, like sample rates, prompt formats, output types, and task-specific request shapes, so the user doesn’t have to think about any of that. Miso ends up translating one workflow into whatever each model family actually expects.
So I’d say the hardest part was less “make the models better” and more “make the weirdness around the models disappear enough that the app feels reliable.”
the guided builder translating one set of controls into whatever syntax each model wants is the real product here. one model wants a caption, another wants tags, ACE-Step splits style and bpm and lyrics into separate fields for the same idea. that translation layer is invisible work nobody notices until it breaks and you're debugging why a param got silently ignored.
Exactly. That translation layer has probably been one of the most deceptively hard parts of the whole project.
From the user side it’s just “style, tempo, vocals, lyrics,” but under the hood those same ideas can mean completely different request shapes depending on the model. One wants a caption, another wants tags, another wants separate fields, and then some parameters are technically accepted in the wrong place and just get silently ignored, which is extra fun 😅
I really wanted the builder to hide the annoying syntax differences without flattening the models into pretending they all work the same way, because they definitely don’t.
So yeah, I agree. A lot of the real product is that invisible glue layer that keeps all the weird model-specific behavior from leaking into the workflow.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.