I wanted to type "something melancholic for tonight" and get a playlist from my own FLAC collection — matched by how the music sounds, not by tags. And I wanted it to run entirely locally, on the same Raspberry Pi 4 with 4 GB of RAM that plays the music. No cloud, no external APIs.
This post is about getting there: two failed approaches, a benchmark that told me exactly what my model couldn't do, a tiny 66k-parameter network that fixed it, and the memory tricks needed to make everything fit next to a running audio server.
Some context first. For the last three years I've been building Kalinka (GitHub), an open-source Hi-Fi music streamer: a server that runs on a Raspberry Pi connected to a DAC and plays audio bit-perfect and gapless through ALSA, with desktop/mobile apps acting as remote controls. It grew out of frustration with my receiver's built-in streaming (every 192 kHz track started with an audible stutter — I only noticed after two years, when I heard the same song open cleanly on YouTube), and with the alternatives: Roon wanted a subscription, Volumio put Qobuz behind an extra paywall on top of the Qobuz subscription I already paid for. So I did the reasonable thing and wrote my own.
The C++ audio graph, gapless playback against CDN links that expire mid-queue, and the real-time client sync each deserve their own write-up. Today: search.
The problem with tags
By the time Kalinka's local library became central, I had a collection big enough to hit a familiar wall: lots of music, no way to pick something for right now. Genre, year, album — none of it describes how a track sounds. Text search assumes you already know what you're looking for. I wanted the opposite: describe a mood, get tracks.
Attempt 1: pre-trained taggers (it went badly)
The obvious approach: auto-tag the whole library, then search the tags. The Essentia project ships pre-trained TensorFlow models for exactly this, so I wired up three:
| Predicts | Model (backbone → head) | Output |
|---|---|---|
| Genre | EffNet-Discogs → Discogs-400 | top-N genre labels + scores |
| Mood | VGGish → MIREX mood | one of 5 mood clusters |
| Danceability | VGGish → danceability | a single float |
It worked… disappointingly. The genre classifier was noisy — a suspicious number of tracks came back as electronic---ambient regardless of what they actually were — and produced usable genres for only about a third of my library. The mood model collapsed the entire collection into three or four clusters. The published numbers explain why: PR-AUC below 0.2. Multi-label music tagging is just a hard problem.
And even ignoring quality, the stack was too heavy. TensorFlow plus its dependency tree is not something you want to install on a Raspberry Pi.
Attempt 2: betting on CLAP
In parallel I was already computing an embedding for every track with CLAP (Contrastive Language-Audio Pretraining). CLAP's core idea is what I actually needed: audio and text map into the same vector space, so a text query becomes an embedding and search becomes nearest-neighbour lookup — no intermediate tags at all.
It quickly became clear this was the real solution and the tagging pipeline had been a temporary crutch.
Making CLAP behave on a Pi took a few iterations:
-
Dropped PyTorch +
laion-clapfor ONNX Runtime — far easier to distribute, much lighter at startup. - Swapped the general audio model for an HTSAT-based CLAP trained specifically on music.
- Since the model only "hears" ~10 seconds at a time, I sample several fragments per track and average the embeddings.
Measure before you optimize
"Vibe" is too subjective to develop against, so before tuning anything I built a proper benchmark on MTG-Jamendo, where tracks are labelled by humans. The results were clarifying - and a little sobering:
| Query category | P@10 | Lift over random | MRR | Hit-rate@10 |
|---|---|---|---|---|
| Genre | 0.296 | +0.212 | 0.670 | 0.880 |
| Instrument | 0.273 | +0.157 | 0.476 | 0.933 |
| Mood / theme | 0.140 | +0.050 | 0.315 | 0.733 |
| Overall | 0.247 | +0.153 | 0.520 | 0.855 |
CLAP is genuinely good at concrete sonic properties. Instrument queries shine — piano hits P@10 = 0.80, meaning eight of the top ten results are relevant. Guitar, strings, violin all do well; among genres, hip-hop and jazz stand out.
Abstract emotional concepts are another story. uplifting, inspiring, love — effectively zero. Worse than picking tracks at random:
| Strong (P@10) | Weak (P@10) | |
|---|---|---|
| Instruments | piano 0.80, guitar 0.50, acoustic guitar / strings / cello 0.40 | synthesizer 0.30, voice 0.10, "electronic production" 0.00 |
| Genres | hip-hop 0.80, jazz / rock / electronic 0.60 | pop 0.20, techno / world 0.00 |
| Mood / theme | epic 0.50, film / meditative 0.30 | uplifting / inspiring / love 0.00 |
This turns out to be a well-known property of CLAP. But reproducing it on my own benchmark was oddly satisfying: instead of a vague feeling that mood search was weak, I had a number — and a direction.
Fixing mood: a small model on top of CLAP
My hypothesis: if CLAP embeddings describe the sound well, maybe a small extra model can learn to extract the emotional component from them.
The DEAM dataset fit perfectly — ~1,800 tracks hand-annotated on two axes: valence (how positive the music feels) and arousal (how energetic it is). I computed CLAP embeddings for DEAM and trained a two-layer MLP with just 66k parameters to map an embedding to a (valence, arousal) point.
(Training ran on my Lenovo ThinkBook 13x, which got hot enough to burn my fingers twice. The eventual fix was parking it under the air conditioner. A cooling pad is probably in my future.)
The part where I got a negative R²
My first attempt was a textbook lesson in train/serve skew. I had computed DEAM's embeddings with the general-audio CLAP model — while production had already moved to the music-trained one. On real production embeddings the results were catastrophic: R² = −0.42. The model performed worse than a baseline that assigns every track the same average value.
After finding the mismatch and retraining on the exact embeddings used in production:
| Axis | Model | R² | CCC | RMSE |
|---|---|---|---|---|
| Valence | MLP head | 0.594 | 0.761 | 0.755 |
| Arousal | MLP head | 0.711 | 0.827 | 0.683 |
| Valence | ridge baseline | 0.480 | 0.626 | 0.855 |
| Arousal | ridge baseline | 0.631 | 0.742 | 0.773 |
I plug this mood signal into ranking through a confidence gate, so it only influences queries that are actually about emotion — it stays out of the way when you search for piano or hip-hop.
Re-running the benchmark: mood search improved by ~80% (lift over random went from +0.05 to +0.09), and P@10 for uplifting went from 0.00 to 0.30. Genre and instrument search degraded by an amount small enough to ignore. A trade I'll take:
| Query category | P@10 | Random baseline | vs random |
|---|---|---|---|
| Genre | 0.296 | 0.084 | 3.5× |
| Instrument | 0.273 | 0.116 | 2.4× |
| Mood / theme | 0.140 | 0.090 | 1.6× |
| Overall | 0.247 | 0.094 | 2.6× |
| Tag | P@10 | Random baseline | vs random |
|---|---|---|---|
| hip-hop | 0.80 | 0.063 | 12.7× |
| trance | 0.40 | 0.032 | 12.5× |
| jazz | 0.60 | 0.067 | 9.0× |
| rock | 0.60 | 0.083 | 7.2× |
| piano | 0.80 | 0.277 | 2.9× |
Making it fit in 4 GB
On the Pi, the bottleneck was memory, not compute. Indexing several tracks in a row kept growing RSS until the OOM killer took the process down; forcing Python's garbage collector after every iteration stopped the bleed. Beyond that, two optimizations did the heavy lifting:
-
Quantized stored embeddings to
int8. Vectors shrank 4×, and ~99% of top-10 results stayed identical. Practically free. - Split CLAP in two. The text encoder (needed for every user query) stays resident. The much heavier audio encoder is only needed for indexing — so it unloads automatically once indexing finishes and comes back when new tracks arrive.
Deleting the scaffolding
At this point CLAP embeddings fully covered semantic search, and the little valence/arousal head covered mood — both jobs the Essentia taggers were originally there for, done better and cheaper. Meanwhile the taggers had turned into pure liability: essentia-tensorflow ships ARM wheels only for Python 3.11, which had silently pinned my entire environment to 3.11 right when I needed 3.13.
So the whole auto-tagging pipeline went in the bin — TensorFlow (a multi-hundred-megabyte dependency), the Python version pin, the tags_predicted column, and all the tag-based ranking logic. What replaced genre and danceability detection? Nothing separate — CLAP already does it. What replaced the mood model? The one I trained myself.
What shipped
Kalinka's smart search today is exactly two components: CLAP audio embeddings and a 66k-parameter valence/arousal model, both running locally through ONNX Runtime. No TensorFlow, no external taggers, no cloud.
A nice side effect: a Russian song and a Pink Floyd track can land next to each other in the results if they share a mood. The system ranks by how music sounds, not what it's called. And all of it fits on a Raspberry Pi 4 with 4 GB of RAM.
(For completeness: semantic search is one layer of a larger query system — SQLite FTS for full-text, streaming-service search, and semantic catalog matching, with RapidFuzz-based ranking and smart-search results surfacing only when there's no exact match. One search box, several engines behind it.)
What's next
Recent work went into first-run setup and easier installation, plus a second free streaming source (Jamendo) with semantic search built the same fully-local way. The model still can't search by origin or era — italian 80s won't find my Toto Cutugno records — so there's plenty left to explore.
Kalinka is open source, and I'd genuinely welcome testers and contributors — not just code: trying it on different hardware, bug reports, UI feedback, docs, new plugins. AI-assisted PRs are fine by me; what matters is that the author understands the solution, can explain and maintain it, and the code is tested and fits the architecture.
Kalinka Player GitHub: Kalinka Player
Kalinka Player Client: Kalinka AI App
Top comments (0)