DEV Community

jackmeng
jackmeng

Posted on Edited on

How Music LLMs Actually Work: From Tokens to Waves (MusicGen, DiT, and Building Your Own Engine)

Everyone thinks Suno is magic.

It’s not. It’s tokenizers + transformers + diffusion + a lot of audio engineering.

If you’ve ever asked:

• “How does an AI turn a text prompt into a 3-minute song?”

• “Why does MusicGen sound decent but Suno sounds produced?”

• “Can I build a music-gen backend that doesn’t melt my GPU?”

This post is the technical map. No hype. Just the pipeline.

  1. The Big Lie: “It Generates Audio”

Music generation models don’t generate raw waveforms directly.

Raw audio at 32kHz = ~32,000 samples per second.

One minute = ~1.9M timesteps. Attention over that? Good luck.

So the industry uses a 3-stage stack:

Text / Lyrics / Melody
↓
Condition Encoder (T5 / CLAP / LLM)
↓
Music “Brain” (Transformer LM or DiT)
↓
Neural Codec Tokens (EnCodec / RVQ / VQ)
↓
Decoder / Vocoder / VAE
↓
Waveform (wav / mp3)

Two dominant paradigms in 2026:

Paradigm Core idea Examples

Autoregressive LM Predict audio tokens like GPT predicts words MusicGen, ACE-Step, Suno (rumored)

Latent Diffusion / DiT Denoise in compressed latent space Stable Audio, AudioLDM, DiffRhythm

Most production systems are hybrid: LM plans structure, diffusion renders texture.

  1. Audio Tokenization: The Part Nobody Talks About

Before any “music model” exists, you need a codec.

Models like EnCodec compress 32kHz audio into:

  • 50 frames per second • 4 RVQ codebooks per frame

• Discrete token IDs
waveform ──EnCodec Encoder──▶ [frame][codebook_0, codebook_1, codebook_2, codebook_3]
50 tokens/sec × 4 layers

Why this matters:

• Reduces 32k samples/sec → 200 tokens/sec

• Lets you use Transformer attention

  • Keeps acoustic detail better than MIDI

But there’s a tradeoff:

• More codebooks = better audio, harder to predict

  • Fewer codebooks = easier LM, worse fidelity

That’s why newer models (StepAudio, ACE-Step) experiment with single-codebook VQ + diffusion decoder instead of naive RVQ.

  1. MusicGen: The Cleanest Reference Architecture

Meta’s MusicGen is still the best open teaching model.

Pipeline

  1. T5 encodes prompt → text embedding
  2. Transformer LM autoregressively predicts EnCodec tokens
  3. Delay pattern aligns 4 codebooks
  4. EnCodec decoder → 32kHz waveform

Minimal runnable example:

pip install audiocraft

from audiocraft.models import MusicGen
import torchaudio

model = MusicGen.get_pretrained('facebook/musicgen-small')
model.set_generation_params(duration=8, temperature=1.0, cfg_coef=3.0)

prompts = [
"lo-fi hip hop beat, warm keys, vinyl crackle, 85 BPM",
"epic orchestral buildup with taiko drums"
]

wav = model.generate(prompts) # list of [1, channels, samples]

for i, audio in enumerate(wav):
torchaudio.save(f"track_{i}.wav", audio.cpu(), 32000)

Key params:

• duration: 8–30s (longer = quality drop)

• cfg_coef: how tightly it follows text

  • temperature: creativity vs stability

Architecture intuition:

[T5 text emb] ──► Transformer LM ──► EnCodec token stream
│
delay pattern
│
EnCodec decoder
│
waveform

MusicGen proves one thing:

You don’t need end-to-end waveform modeling to get good music.

  1. The Diffusion Path: Stable Audio & DiT

Autoregressive models struggle with:

• long-form structure

  • global coherence • high-fidelity timbre

Diffusion fixes that.

Stable Audio / DiffRhythm style pipeline:

noise z_T
↓
DiT (Transformer denoiser)
conditioned on:

  • T5 text embedding
  • lyric phoneme embeddings
  • timestamp / section markers
  • BPM / chord grid ↓ z_0 (clean latent) ↓ VAE Decoder → waveform

Toy denoising loop:
import torch

def denoise_step(model, x, t, cond):
# model predicts noise; simplified update
pred_noise = model(x, t, cond)
return x - 0.5 * pred_noise

x = torch.randn(1, 64, 1024) # latent noise
for t in reversed(range(50)):
x = denoise_step(net, x, t, text_emb)
wave = vae_decoder(x)

Why DiT wins for music:

• parallel generation (not strict AR)

• better frequency stability

• easier to condition on time-stamped events (“chorus at 0:45”)

  1. How Suno/Udio Probably Work (Engineering Guess)

They don’t publish weights, but the leaks/behaviors tell us:

  1. LLM planner • receives prompt + lyrics

• outputs song structure: intro / verse / chorus / bridge

  • maybe ABC notation or chord CoT
  1. Token LM • generates coarse audio tokens

• controls melody + rhythm

  1. Diffusion renderer
    • refines timbre, reverb, vocal texture

  2. Vocal specialist
    • separate head for singing

  • pitch-constrained decoding (not normal TTS)
  1. Post-processing • loudness matching

• stem separation

  • watermarking (AudioSeal style)

So “Suno” is not one model.

It’s a music operating system.

  1. Controllability: The Real Hard Problem

Text-to-music is easy now.

“Make the snare hit harder in bar 16” is hard.

Useful control signals:

• Chord progression (Roman numeral or ABC)

• Melody chroma

• Reference audio clip

• Section timeline

• BPM / key / instrumentation mask

• Lyric timestamps

Example: MusicGen melody conditioning
model.set_generation_params(duration=10)
wav = model.generate_with_chroma(
descriptions=["ambient piano"],
melody_wav=reference_waveform,
melody_sample_rate=32000
)

JASCO-style systems go further: timed chords + drum grid + melody.

The future isn’t “better prompts”.

It’s structured music DSLs for LLMs.

  1. Building a Music-Gen Backend (Practical Stack)

If you want to ship this in an app:

Inference stack

FastAPI / Node gateway
↓
Task queue (Celery / Temporal)
↓
GPU worker:

  • MusicGen (fast drafts)
  • Stable Audio / ACE-Step (full songs)
  • Demucs (stem split)
  • FFmpeg (mastering) ↓ Object storage (S3/R2) ↓ Client player + waveform UI

VRAM budget

Model VRAM

MusicGen-small 4–6 GB

MusicGen-3.3B 16 GB+

Stable Audio 3 Medium 8–12 GB

ACE-Step XL 12–24 GB

Use:
• torch.compile

• chunked decoding

• KV-cache for AR

• FP16/BF16

• prewarm workers

  1. The 3 Bugs That Will Burn You

  2. Wrong sample rate

MusicGen = 32kHz.

Save as 16kHz → chipmunk audio.
torchaudio.save("out.wav", wav, sample_rate=32000) # NOT 16000

  1. Ignoring CFG scale

Too low → ignores prompt

Too high → robotic, overcompressed

Start: cfg_coef=2.5–3.5

  1. No watermarking

If you deploy publicly:

• add AudioSeal / WavMark

• log prompt + seed + model version

  • store raw output before mastering

Otherwise you’ll have legal + traceability issues fast.

  1. Evaluation: Don’t Trust “Sounds Cool”

Technical metrics:

• FAD (Fréchet Audio Distance)

• KL-PAANNs (semantic similarity)

• Chromagram accuracy (melody follow)

• Section consistency (intro vs chorus contrast)

• CLAP score (text-audio alignment)

Human metrics:

  • Does it loop naturally? • Would you put it in a game / reel / scene?

• Can you edit one instrument without regenerating the whole track?

  1. Where This Is Going (2026–2027)

• Symbolic + acoustic joint modeling

LLM writes chords → DiT renders audio
• Agentic music production

“Extend bridge, add strings, sidechain the kick”
• On-device music models

phone-local, <4GB VRAM
• Interactive DAW copilots

not “generate a song”, but “fix this mix”

  • Copyright-aware training licensed corpora + watermarking by default

Final Take

Music LLMs aren’t “AI composers”.

They’re:

Tokenizers + language models + diffusion decoders + audio DSP pipelines

Once you see the plumbing, the magic turns into engineering — and engineering is where you actually build something.

If you can make this pipeline run reliably, stream results, and let users edit stems — you’re not following the music AI trend. You’re building the studio.

clone + try locally

pip install audiocraft diffusers torchaudio demucs

Then stop prompting like a user.

Start architecting like a builder.

Suggested dev.to meta:

  • Title: How Music LLMs Actually Work: From Tokens to Waves
    • Tags: #ai #machinelearning #llm #tutorial #showdev #audio

  • Cover: dark background, waveform + transformer nodes + “Music = Tokens”

Top comments (0)