DEV Community

jackmeng
jackmeng

Posted on

Stop Recommending Songs. Start Understanding Intent.

Why the next generation of music recommendation systems is multimodal, agentic, and no longer “click-prediction”

By Jack Meng · AI Systems & Generative Audio Engineer

  1. The old music recommender is dead

For a decade, music recommendation looked like this:

• Log user plays

• Train a two-tower model

• Retrieve similar songs

• Sort by CTR

• Call it “personalization”

It worked. But it was shallow.

Because users never actually say:

“Recommend me three tracks with a 124 BPM groove, minor-key melancholy, 2010s R&B texture, and a little tape saturation.”

They say:

“Something for late-night driving. Not too sad. A bit nostalgic.”

That sentence has no tag, no genre, no clean label.

And traditional recommender systems choke on it.

The future of music discovery is not a better classifier.

It’s a system that can hear, read, reason, and generate.

  1. The architecture: Retrieve → Understand → Rerank → Explain

A modern music recommendation system is no longer one model.

It’s a pipeline.
User Intent
│
▼
┌──────────────────────────────┐
│ 1. Query Understanding (LLM) │ ← rewrite vague prompts
└──────────────┬───────────────┘
▼
┌────────────────────────────────────────────┐
│ 2. Multi-Modal Retrieval │
│ - Collaborative embeddings (BPR / Two-Tower)
│ - Text embeddings (lyrics, tags, mood)
│ - Audio embeddings (CLAP / VGGish)
│ - Visual embeddings (album art)
│ - Lexical search (BM25)
└──────────────┬─────────────────────────────┘
▼
┌────────────────────────────────────────────┐
│ 3. Fusion & Candidate Pooling (RRF / Router)│
└──────────────┬─────────────────────────────┘
▼
┌────────────────────────────────────────────┐
│ 4. Reranking │
│ - Transformer / LambdaMART
│ - Diversity constraints
│ - Skip-rate penalty
│ - Session-level coherence
└──────────────┬─────────────────────────────┘
▼
┌────────────────────────────────────────────┐
│ 5. Agentic Explainer (LLM) │
│ “Because you liked X, but want calmer now”
└────────────────────────────────────────────┘

This is not theoretical.

This is basically what top RecSys 2026 teams converged on.

  1. Step 1: Don’t trust the raw query

Users are vague. The LLM’s job is not to generate music—it’s to translate feeling into structure.

Example:

“I want rain, neon, and a heartbeat”

The intent parser outputs:
{
"mood": ["melancholic", "cinematic"],
"tempo_range": [80, 110],
"instruments": ["synth", "pads", "soft drums"],
"avoid": ["heavy metal", "hyperpop"],
"context": "late night, focus, urban"
}

Now we have something a retrieval system can use.

  1. Step 2: Multi-modal embeddings are non-negotiable

A song is not a row in a database.

It’s:

• sound

• language

• image

• culture

• memory

So we encode it multiple ways.

Text tower

• Lyrics

• Tags

• Editorial descriptions

• User-generated mood words

Model: Qwen3-Embedding, BGE-M3

Audio tower

• Mel-spectrogram

• Chroma

• Tempo / key / timbre

Model: CLAP, VGGish, fine-tuned HTS-AT

Collaborative tower

• User–item interaction graph

• BPR loss

• Artist / session co-occurrence

Visual tower

• Album cover semantics

• Color palette / aesthetic clustering

Then we store all of them in vector indexes:
FAISS / Milvus
├── text_vectors
├── audio_vectors
├── collab_vectors
└── fused_vectors

  1. Step 3: Hybrid retrieval, not “one vector to rule them all”

Pure vector search lies.

Pure collaborative filtering forgets new music.

Pure metadata search can’t feel vibe.

So we do heterogeneous retrieval:
candidates = set()

candidates |= bm25_search(query_text, k=200)
candidates |= faiss_search(text_vec, index="text", k=200)
candidates |= faiss_search(audio_vec, index="audio", k=200)
candidates |= collab_neighbors(user_id, k=200)
candidates |= artist_expansion(query)

Then merge with Weighted Reciprocal Rank Fusion:
def rrf(lists, k=60):
scores = {}
for lst in lists:
for rank, item in enumerate(lst):
scores[item] = scores.get(item, 0) + 1 / (k + rank)
return sorted(scores.items(), key=lambda x: -x[1])

Simple. Robust. Beats most “fancy” end-to-end models in production.

  1. Step 4: Reranking is where the money is

Retrieval gets you “plausible”.

Reranking gets you “right”.

A real reranker looks at:

• click probability

• skip probability

• completion rate

• genre balance

• artist repetition penalty

• time-of-day fit

• novelty vs familiarity tradeoff

Sketch:
class Reranker(nn.Module):
def forward(self, seq_features, item_features, context):
x = torch.cat([seq_features, item_features, context], dim=-1)
score = self.mlp(x)
return score # P(listen > 30s)

Then apply business rules:
if same_artist_in_last_3:
score *= 0.6

if item_novelty_high and user_explorer:
score *= 1.15

Precision@10 jumps. Fatigue drops.

  1. Step 5: Make the system explain itself

A recommendation without explanation is a black box.

Modern UX needs:

“Because you listened to Lowlight last week, but tonight your prompt sounds calmer—so I pulled ambient R&B with softer attack.”

That’s an LLM reranker + constrained generation, not free-form hallucination.

Rules:

• Only talk about retrieved items

• No fake artist names

• No invented lyrics

• Always ground explanation in metadata

  1. Cold start? Solve it with content, not hacks

New song. Zero plays.

Old systems panic.

New system says:

“I don’t know who listens yet, but acoustically this sits between Bonobo and Tycho, lyrically it’s abstract, visually it’s dusk-toned.”

So we bootstrap with:

• audio embedding

• lyric embedding

• mood taxonomy

• creator-provided brief

Cold start becomes a content understanding problem, not a statistics problem.

  1. Engineering reality: the stack I’d ship

If I were building this today:
Backend API : Go (low latency, concurrency)
Realtime Gateway : WebSocket + gRPC
ML Inference : Python + PyTorch + ONNX
Embeddings : Qwen3 / CLAP / BGE-M3
Vector DB : Milvus or FAISS
Cache : Redis (session vectors)
Orchestration : LangGraph / custom agent loop
Frontend : Vue 3 + Web Audio preview

Target latency:

• p95 retrieval: < 40ms

• rerank: < 25ms

• generation/explanation: < 300ms streaming

That’s production-grade, not demo-grade.

  1. The bigger shift: from recommender to sonic agent

We’re moving from:

“What song should I play next?”

to:

“Build me a 40-minute sonic environment.”

That means the system can:

• generate missing tracks

• remix existing ones

• adapt BPM to heart rate

  • shift mood mid-session • respect brand, scene, and emotion

Recommendation is no longer search.

It’s intent-driven audio synthesis with guardrails.

  1. Lessons I’ve learned shipping AI music systems

  2. Don’t fine-tune first. Architect first.

    Most teams waste months on models before defining the retrieval boundary.

  3. LLMs are planners, not oracles.

    Let the LLM call tools. Don’t let it invent songs.

  4. Diversity > perfection.

    A slightly imperfect but varied playlist beats 10 near-duplicates.

  5. Implicit feedback lies less than you think.

    Skips, replays, and abandon-time are gold.

  6. Generative + retrieval hybrid is the endgame.

    Pure generation hallucinates. Pure retrieval stagnates.

  7. Final thought

The next Spotify-killer won’t be a better playlist algorithm.

It’ll be a system that understands:

• what you say

• what you feel

• what you heard yesterday

• what you didn’t know you wanted tonight

Music recommendation is no longer about predicting the next click.

It’s about modeling human atmosphere.

And that’s a much harder—and much more beautiful—problem.

If you build AI music, generative audio, or recommendation infra:

I write about real-world architecture for multimodal ML, agentic systems, and production-grade generative audio.

No hype. Just pipes, vectors, and signal.

Tags: #MusicAI #RecSys #MultimodalML #LLM #AgenticSystems #PyTorch #VectorSearch #GenerativeAudio

Top comments (0)