YuE2: How Symbolic Planning Brings Transparency and Control to AI Music Generation

Music generation models have made rapid progress, but most treat composition as a black box: you provide a text prompt, and audio emerges from a dense neural process that no one—including the model—can inspect or edit mid-flight. YuE2, introduced in a technical report from the multimodal-art-projection team published in late September 2026, takes a different approach. It generates a readable musical score first, then renders that score into audio. The result is a system that is both more controllable and, according to expert listening evaluations, more musical.

The Core Idea: Write the Score Before Playing the Notes

YuE2's central contribution is what the authors call symbolic planning — a staged generation process that mirrors how a human composer works. Rather than mapping a text prompt directly to audio tokens, the model first writes a structured score in ABC notation, a compact text format that encodes melody, chords, key, meter, tempo, and song structure. Only after this symbolic plan exists does the model proceed to audio synthesis.

This matters for two reasons. First, the symbolic plan acts as a commitment device: the model cannot contradict itself mid-song because the harmonic and melodic structure is already fixed. Second, the plan is human-readable and machine-editable, which opens the door to a class of interactions that audio-only models cannot support.

Architecture: AR–NAR Mixture-of-Transformers

Under the hood, YuE2 uses a 28-layer Mixture-of-Transformers (MoT) architecture that runs two expert types — autoregressive (AR) and non-autoregressive (NAR) — side by side, sharing attention computation but using separate normalization, projections, and feed-forward layers.

Generation is factorized into three representations:

1. Symbolic tokens (s): A sparse sequence encoding melody, chords, key, meter, tempo, and form — the ABC score.
2. Semantic music tokens (c): A dense 25-Hz sequence that bridges the abstract score and the acoustic signal.
3. Continuous acoustic latents (z): A 25-Hz sequence decoded into 48-kHz stereo audio by a variational autoencoder.

The AR component handles the symbolic and semantic sequences causally (left-to-right). The NAR component realizes the acoustic sequence via bidirectional flow matching, which allows it to condition on the full symbolic context when generating audio. The joint probability is:

p(s, c, z | y) = p_AR(s, c | y) × p_NAR(z | y, s, c)

where y is the text prompt (lyrics, style description, or both).

Training Without Aligned Scores: MERT2 and SheetSage2

A practical challenge for this approach is that most audio training data does not come with aligned symbolic annotations. The team addressed this by building two auxiliary models:

• SheetSage2 performs full-song lead-sheet transcription, converting audio recordings into ABC notation. It provides the symbolic supervision target s during training, allowing the model to learn from vast amounts of existing music without requiring hand-annotated scores.
• MERT2 is a multi-view music representation encoder that supplies the semantic tokens c. It achieves state-of-the-art results on 14 of 15 tasks in the MARBLE music understanding benchmark, making it a strong foundation for the semantic layer.

Both models are released alongside YuE2, which means the full pipeline — transcription, semantic encoding, and generation — is available for researchers to build on.

What Symbolic Planning Enables

The ABC score is not just an internal implementation detail; it is an explicit output that users and downstream systems can interact with.

Agentic music editing: Because the score is text, a language model can translate natural-language feedback ("make the chorus more upbeat" or "change the key to D minor") into specific score revisions. YuE2 then re-renders the modified score into audio, preserving the unedited sections. This creates a genuine edit loop rather than a generate-and-discard workflow.

Zero-shot covers: SheetSage2 can transcribe a reference recording into ABC notation. YuE2 can then apply a new style prompt while preserving the melodic structure, producing a stylistically transformed cover without any fine-tuning.

Targeted local edits: Because the score is structured, a user can modify a single measure — changing a chord voicing or a melodic phrase — and re-render only the affected section. Audio-only models have no equivalent mechanism.

Performance: Competitive with Proprietary Systems

On the WildSongBench (WSB) benchmark, YuE2 achieves a SongBench Global Average score of 6.73, the highest among public baselines at the time of the report. With a best-of-8 selection strategy, the score rises to 6.96 — the highest observed mean across all evaluated systems, including proprietary ones.

In expert listening evaluations, participants preferred the symbolic-planning output over non-planned generation from the same checkpoint at a rate of 49.3% vs. 34.6% (with the remainder expressing no preference). When compared directly against Suno v5, the best-of-8 configuration produced nearly balanced preferences — a notable result for an open-weight system competing against a closed commercial product.

Practitioner Implications

For developers building music-adjacent applications, YuE2 offers something that most generative audio models do not: a structured intermediate representation that can be inspected, validated, and modified programmatically. If you need to enforce harmonic constraints, ensure a specific song structure, or integrate user feedback into a generation loop, the ABC score gives you a natural intervention point.

The model weights, code, and both auxiliary models (MERT2 and SheetSage2) are available on GitHub and Hugging Face. The project page at map-yue2.github.io includes audio examples that demonstrate the cover and editing workflows.

Conclusion

YuE2's contribution is not just a better music generator — it is a demonstration that adding an explicit, inspectable intermediate representation to a generative pipeline can improve both quality and controllability simultaneously. The symbolic planning approach trades some generation speed for a system that composers, developers, and AI agents can actually work with, rather than just prompt and hope. As music generation matures from a novelty into a production tool, that kind of structured transparency is likely to matter more, not less.