A recommender that has never heard a note of music can be excellent, and one that understands the audio perfectly can be useless. The two approaches fail in opposite directions, and knowing which failure you have is the whole of the design decision.
What collaborative filtering actually uses
Collaborative filtering models the user–item interaction matrix and nothing else. Rows are listeners, columns are tracks, entries are plays or skips or likes. Matrix factorisation learns a low-dimensional latent vector for each user and each track such that their inner product reconstructs the observed interactions; similar tracks end up with similar vectors because similar people played them.
Nothing in that procedure knows what the track sounds like. That is a feature, not a limitation: the latent space encodes co-listening, which captures things audio cannot — that two tracks belong to the same scene, appeared in the same film, are played at the same kind of party, or were released by the same label to the same audience. A cover version in an entirely different genre can sit next to the original because the same people seek out both. No audio model will ever recover that, because the information is not in the signal.
The cold start is structural
The consequence is a hard boundary. A track with zero interactions has no column in the matrix, so it has no latent vector, so it cannot be recommended, so it accumulates no interactions. This is not a tuning problem or a data-sparsity problem that more users fix. It is circular, and it applies to every new release on the day it lands and to the entire long tail permanently.
The scale is worth being concrete about. In a catalogue of tens of millions of tracks, the fraction with enough interaction data for a stable latent vector is small, and it is not a random sample — it is the popular fraction. A pure collaborative system is therefore structurally biased toward what is already popular, and cannot be debiased by reweighting, because the missing vectors do not exist to be reweighted.
What an audio embedding gives you
Content-based recommendation takes the other route: compute a fixed- length vector from the audio itself, and recommend by proximity in that space. A track that was released ten minutes ago has an embedding the moment it is uploaded, because the input is the file.
The cold start is not the only place this helps. Collaborative systems also have a feedback loop: what gets recommended gets played, what gets played accumulates interaction data, and what accumulates data gets recommended more confidently. Left alone the loop concentrates exposure, and the usual mitigations — exploration bonuses, randomised slots — spend impressions to buy data. A content-based candidate generator changes the economics of that, because it can propose a plausible unseen track without first paying for a random impression to discover it.
What such an embedding encodes depends entirely on how it was trained. An embedding from a genre or tag classifier’s penultimate layer encodes whatever distinguishes those tags. A self-supervised or contrastive embedding encodes whatever its pretext task made useful. In practice, for music, the dominant axes are timbral and rhythmic: instrumentation, production texture, tempo, energy. That is a real similarity and it is not the same similarity a listener means. Two tracks with identical instrumentation and tempo can belong to communities that have nothing to do with each other.
A worked nearest-neighbour lookup
The retrieval itself is arithmetic you can check. Embeddings are L2-normalised so that cosine similarity reduces to a dot product, and the neighbours are ranked by that. With four-dimensional vectors, so the numbers stay legible:
Query track Q, already L2-normalised:
q = [ 0.60, 0.40, -0.50, 0.48 ]
||q|| = sqrt(0.36 + 0.16 + 0.25 + 0.2304) = sqrt(1.0004) ~= 1.00
Candidates (also normalised):
A = [ 0.58, 0.45, -0.47, 0.49 ]
B = [ 0.10, 0.90, 0.20, -0.37 ]
C = [ 0.62, 0.33, -0.55, 0.45 ]
cos(Q,A) = 0.60*0.58 + 0.40*0.45 + (-0.50)*(-0.47) + 0.48*0.49
= 0.3480 + 0.1800 + 0.2350 + 0.2352 = 0.998
cos(Q,B) = 0.0600 + 0.3600 - 0.1000 - 0.1776 = 0.142
cos(Q,C) = 0.3720 + 0.1320 + 0.2750 + 0.2160 = 0.995
Ranking: A (0.998), C (0.995), B (0.142).
The vectors above are illustrative, chosen to make the arithmetic checkable by hand; a real embedding is 512 or 1024 dimensions and the same operation. Two things about this result generalise. First, the top of a normalised-embedding ranking is compressed — A and C differ by 0.003, which is well inside the noise of which track you would rather hear, so the ordering within the top ten is close to arbitrary and should be broken by something else (recency, diversity, artist deduplication). Second, the gap to B is enormous, which is the useful part: the embedding is excellent at excluding the irrelevant and weak at ordering the relevant.
At catalogue scale you do not compute this against every track. The neighbours come out of an approximate index; the trade-offs there are general to vector search and covered in HNSW and vector similarity metrics. The normalisation step matters more than it looks; see vector normalization.
The hybrid that actually ships
The approach that resolves the cold start without discarding the cultural signal is due to van den Oord, Dieleman and Schrauwen, whose 2013 NeurIPS paper “Deep content-based music recommendation” trains a network to predict a track’s collaborative latent vector from its audio. The target is the vector that factorisation produced for tracks that have interaction data; the input is the spectrogram. For a new track with no plays, you predict where it would sit in the collaborative space and serve it from there.
This is a better formulation than plain audio similarity because the space you are predicting into is the one that encodes listening behaviour, so the model learns which acoustic properties are behaviourally relevant rather than which are acoustically salient. It also degrades sensibly: as a track accumulates plays, you shift from the predicted vector to the observed one.
It cannot predict what is not predictable from audio. A track whose audience is defined by the artist’s reputation, a scene, or a film placement has a collaborative vector with a component no spectrogram contains, and the model will place it by how it sounds. Expect the residual to be large for exactly the tracks whose appeal is extra-musical, and do not treat that as a bug to be trained away.
Top comments (0)