"Something melancholic for tonight": Building semantic music search that runs on a Raspberry Pi

MadEnvel — Mon, 20 Jul 2026 16:22:30 +0000

I wanted to type "something melancholic for tonight" and get a playlist from my own FLAC collection — matched by how the music sounds, not by tags. And I wanted it to run entirely locally, on the same Raspberry Pi 4 with 4 GB of RAM that plays the music. No cloud, no external APIs.

This post is about getting there: two failed approaches, a benchmark that told me exactly what my model couldn't do, a tiny 66k-parameter network that fixed it, and the memory tricks needed to make everything fit next to a running audio server.

Some context first. For the last three years I've been building Kalinka (GitHub), an open-source Hi-Fi music streamer: a server that runs on a Raspberry Pi connected to a DAC and plays audio bit-perfect and gapless through ALSA, with desktop/mobile apps acting as remote controls. It grew out of frustration with my receiver's built-in streaming (every 192 kHz track started with an audible stutter — I only noticed after two years, when I heard the same song open cleanly on YouTube), and with the alternatives: Roon wanted a subscription, Volumio put Qobuz behind an extra paywall on top of the Qobuz subscription I already paid for. So I did the reasonable thing and wrote my own.

The C++ audio graph, gapless playback against CDN links that expire mid-queue, and the real-time client sync each deserve their own write-up. Today: search.

The problem with tags

By the time Kalinka's local library became central, I had a collection big enough to hit a familiar wall: lots of music, no way to pick something for right now. Genre, year, album — none of it describes how a track sounds. Text search assumes you already know what you're looking for. I wanted the opposite: describe a mood, get tracks.

Attempt 1: pre-trained taggers (it went badly)

The obvious approach: auto-tag the whole library, then search the tags. The Essentia project ships pre-trained TensorFlow models for exactly this, so I wired up three:

Predicts	Model (backbone → head)	Output
Genre	EffNet-Discogs → Discogs-400	top-N genre labels + scores
Mood	VGGish → MIREX mood	one of 5 mood clusters
Danceability	VGGish → danceability	a single float

It worked… disappointingly. The genre classifier was noisy — a suspicious number of tracks came back as electronic---ambient regardless of what they actually were — and produced usable genres for only about a third of my library. The mood model collapsed the entire collection into three or four clusters. The published numbers explain why: PR-AUC below 0.2. Multi-label music tagging is just a hard problem.

And even ignoring quality, the stack was too heavy. TensorFlow plus its dependency tree is not something you want to install on a Raspberry Pi.

Attempt 2: betting on CLAP

In parallel I was already computing an embedding for every track with CLAP (Contrastive Language-Audio Pretraining). CLAP's core idea is what I actually needed: audio and text map into the same vector space, so a text query becomes an embedding and search becomes nearest-neighbour lookup — no intermediate tags at all.

It quickly became clear this was the real solution and the tagging pipeline had been a temporary crutch.

Making CLAP behave on a Pi took a few iterations:

Dropped PyTorch + laion-clap for ONNX Runtime — far easier to distribute, much lighter at startup.
Swapped the general audio model for an HTSAT-based CLAP trained specifically on music.
Since the model only "hears" ~10 seconds at a time, I sample several fragments per track and average the embeddings.

Measure before you optimize

"Vibe" is too subjective to develop against, so before tuning anything I built a proper benchmark on MTG-Jamendo, where tracks are labelled by humans. The results were clarifying - and a little sobering:

Query category	P@10	Lift over random	MRR	Hit-rate@10
Genre	0.296	+0.212	0.670	0.880
Instrument	0.273	+0.157	0.476	0.933
Mood / theme	0.140	+0.050	0.315	0.733
Overall	0.247	+0.153	0.520	0.855

CLAP is genuinely good at concrete sonic properties. Instrument queries shine — piano hits P@10 = 0.80, meaning eight of the top ten results are relevant. Guitar, strings, violin all do well; among genres, hip-hop and jazz stand out.

Abstract emotional concepts are another story. uplifting, inspiring, love — effectively zero. Worse than picking tracks at random:

	Strong (P@10)	Weak (P@10)
Instruments	piano 0.80, guitar 0.50, acoustic guitar / strings / cello 0.40	synthesizer 0.30, voice 0.10, "electronic production" 0.00
Genres	hip-hop 0.80, jazz / rock / electronic 0.60	pop 0.20, techno / world 0.00
Mood / theme	epic 0.50, film / meditative 0.30	uplifting / inspiring / love 0.00

This turns out to be a well-known property of CLAP. But reproducing it on my own benchmark was oddly satisfying: instead of a vague feeling that mood search was weak, I had a number — and a direction.

Fixing mood: a small model on top of CLAP

My hypothesis: if CLAP embeddings describe the sound well, maybe a small extra model can learn to extract the emotional component from them.

The DEAM dataset fit perfectly — ~1,800 tracks hand-annotated on two axes: valence (how positive the music feels) and arousal (how energetic it is). I computed CLAP embeddings for DEAM and trained a two-layer MLP with just 66k parameters to map an embedding to a (valence, arousal) point.

(Training ran on my Lenovo ThinkBook 13x, which got hot enough to burn my fingers twice. The eventual fix was parking it under the air conditioner. A cooling pad is probably in my future.)

The part where I got a negative R²

My first attempt was a textbook lesson in train/serve skew. I had computed DEAM's embeddings with the general-audio CLAP model — while production had already moved to the music-trained one. On real production embeddings the results were catastrophic: R² = −0.42. The model performed worse than a baseline that assigns every track the same average value.

After finding the mismatch and retraining on the exact embeddings used in production:

Axis	Model	R²	CCC	RMSE
Valence	MLP head	0.594	0.761	0.755
Arousal	MLP head	0.711	0.827	0.683
Valence	ridge baseline	0.480	0.626	0.855
Arousal	ridge baseline	0.631	0.742	0.773

I plug this mood signal into ranking through a confidence gate, so it only influences queries that are actually about emotion — it stays out of the way when you search for piano or hip-hop.

Re-running the benchmark: mood search improved by ~80% (lift over random went from +0.05 to +0.09), and P@10 for uplifting went from 0.00 to 0.30. Genre and instrument search degraded by an amount small enough to ignore. A trade I'll take:

Query category	P@10	Random baseline	vs random
Genre	0.296	0.084	3.5×
Instrument	0.273	0.116	2.4×
Mood / theme	0.140	0.090	1.6×
Overall	0.247	0.094	2.6×

Tag	P@10	Random baseline	vs random
hip-hop	0.80	0.063	12.7×
trance	0.40	0.032	12.5×
jazz	0.60	0.067	9.0×
rock	0.60	0.083	7.2×
piano	0.80	0.277	2.9×

Making it fit in 4 GB

On the Pi, the bottleneck was memory, not compute. Indexing several tracks in a row kept growing RSS until the OOM killer took the process down; forcing Python's garbage collector after every iteration stopped the bleed. Beyond that, two optimizations did the heavy lifting:

Quantized stored embeddings to int8. Vectors shrank 4×, and ~99% of top-10 results stayed identical. Practically free.
Split CLAP in two. The text encoder (needed for every user query) stays resident. The much heavier audio encoder is only needed for indexing — so it unloads automatically once indexing finishes and comes back when new tracks arrive.

Deleting the scaffolding

At this point CLAP embeddings fully covered semantic search, and the little valence/arousal head covered mood — both jobs the Essentia taggers were originally there for, done better and cheaper. Meanwhile the taggers had turned into pure liability: essentia-tensorflow ships ARM wheels only for Python 3.11, which had silently pinned my entire environment to 3.11 right when I needed 3.13.

So the whole auto-tagging pipeline went in the bin — TensorFlow (a multi-hundred-megabyte dependency), the Python version pin, the tags_predicted column, and all the tag-based ranking logic. What replaced genre and danceability detection? Nothing separate — CLAP already does it. What replaced the mood model? The one I trained myself.

What shipped

Kalinka's smart search today is exactly two components: CLAP audio embeddings and a 66k-parameter valence/arousal model, both running locally through ONNX Runtime. No TensorFlow, no external taggers, no cloud.

A nice side effect: a Russian song and a Pink Floyd track can land next to each other in the results if they share a mood. The system ranks by how music sounds, not what it's called. And all of it fits on a Raspberry Pi 4 with 4 GB of RAM.

(For completeness: semantic search is one layer of a larger query system — SQLite FTS for full-text, streaming-service search, and semantic catalog matching, with RapidFuzz-based ranking and smart-search results surfacing only when there's no exact match. One search box, several engines behind it.)

What's next

Recent work went into first-run setup and easier installation, plus a second free streaming source (Jamendo) with semantic search built the same fully-local way. The model still can't search by origin or era — italian 80s won't find my Toto Cutugno records — so there's plenty left to explore.

Kalinka is open source, and I'd genuinely welcome testers and contributors — not just code: trying it on different hardware, bug reports, UI feedback, docs, new plugins. AI-assisted PRs are fine by me; what matters is that the author understands the solution, can explain and maintain it, and the code is tested and fits the architecture.

Kalinka Player GitHub: Kalinka Player
Kalinka Player Client: Kalinka AI App
Site: https://kalinkaplayer.com

DEV Community: MadEnvel