“Find me more sounds like this one” is a nearest-neighbour query over vectors, and the hard part is not the search. It is that two embedding models will both return plausible results, and only a labelled evaluation on your own library tells you which one is answering the question you meant.
This page describes an evaluation protocol. It deliberately does not report scores for particular models: no such comparison was run here, and a benchmark number carried over from another corpus would not predict your result anyway, because retrieval quality on a sound library depends heavily on the recording conditions in that library. The protocol is the transferable part.
Frames to one vector per clip
Audio models produce a sequence, not a vector. YAMNet’s documented geometry gives one 1024-dimension embedding per 0.96-second patch at 50% overlap, so a 30-second clip yields about 61 of them. Similarity search needs exactly one vector per item, so something must collapse the sequence, and that collapse is a modelling decision rather than a formality.
- Mean pooling describes the clip as a whole. It is the right choice when the clip is homogeneous — a two-second sound effect, a loop, a machine recording.
- Max pooling per dimension keeps the strongest evidence for each learned feature and is better when the thing you are matching on is a brief event inside a longer, mostly irrelevant clip.
- Mean plus standard deviation concatenated doubles the dimension and keeps a description of how much the clip varies, which separates a steady drone from a sequence of unrelated sounds that happens to have the same average.
- Not pooling at all — indexing every frame and aggregating at query time — is the right answer for long, heterogeneous recordings where a match may occupy one second of an hour. It multiplies your index size by the frame count, which is the reason it is the last option considered rather than the first.
Choose the clip length before the model. If your library is one-second UI sounds and you pool over ten seconds, you are averaging nine seconds of silence into every vector.
What the families encode
Three broad families are in common use and they encode different things, which is why they retrieve differently rather than just better or worse.
Supervised tagging embeddings — the penultimate layer of a model trained on AudioSet, such as YAMNet or the PANNs checkpoints published with Kong et al.’s reference code — encode whatever distinguishes the 500-odd training classes. They are strong at “same kind of thing” and indifferent to distinctions the ontology did not make.
Self-supervised embeddings such as OpenL3, whose reference implementation is built on the audio-visual correspondence task from Arandjelović and Zisserman’s “Look, Listen and Learn”, are trained without a label set at all. OpenL3 exposes the choices that matter directly: input representation (linear, mel128 or mel256), content type (music or environmental), and embedding size. Those switches are not cosmetic — the music and environmental variants were trained on different subsets and are genuinely better at different libraries.
Language-aligned embeddings such as CLAP place audio and text in one space, which means you can retrieve by typing a description instead of supplying an example. The LAION-CLAP repository documents a 48 kHz input requirement and checkpoints trained on combinations including LAION-Audio-630k and AudioSet; the projection dimension is set in the checkpoint config and should be read from there rather than assumed. The mechanism is covered separately in CLAP audio embeddings.
Sample rate is a correctness issue, not a quality knob. A model expecting 48 kHz that receives 16 kHz audio resampled up will see a spectrogram with nothing above 8 kHz, which is a large distribution shift that shows up as quietly degraded retrieval rather than an error. Resample to the model’s documented rate and check what your files actually contain, not what their container claims.
Building the labelled pairs
The evaluation set is a list of query clips, each with a set of clips in your library that count as correct matches. Constructing it is most of the work and determines what you are measuring.
A concrete construction for a sound-effects library: take 200 query clips spread across your categories. For each, have two people independently list every library item they would accept as a result, working from the same brief about what “similar” means for your product — and write that brief down, because “similar” can mean same source object, same acoustic character, or same usable function in a mix, and those three produce three different ground truths. Keep only items both annotators listed, and record how often they disagreed. That disagreement rate is the ceiling on any score you subsequently report.
Two adversarial additions make the set far more informative. Include hard negatives: clips that are acoustically close but categorically wrong, such as a door closing against a book dropping. And include near-duplicate queries — the same source sound at a different level, with different room tone, or through a different microphone — where a good general embedding should return the counterpart at rank 1 and a fingerprinting approach would too. If exact reidentification is what you actually need, audio fingerprinting is the appropriate tool and an embedding is the wrong one.
Recall@k, MRR and what each hides
For each query q with relevant set R(q), retrieve a ranked list.
recall@k(q) = |{top k results} ∩ R(q)| / |R(q)|
reciprocal rank(q) = 1 / (rank of first relevant result)
MRR = mean over queries of reciprocal rank
Worked, one query, |R(q)| = 4, k = 10:
ranks of relevant items: 1, 3, 9, 24
recall@10 = 3 / 4 = 0.75
recall@5 = 2 / 4 = 0.50
reciprocal rank = 1/1 = 1.00
The two numbers answer different questions and the worked example shows why you need both. MRR is 1.00 here — the first result was relevant, so a user who wanted one good answer is perfectly served. Recall@5 is 0.50, so a user browsing for options is missing half of what exists. An embedding that puts one obvious match at rank 1 and scatters the rest can look excellent by MRR and be poor for a library browsing interface.
Report both, at the k your interface actually shows, and report the spread across queries rather than only the mean — a mean recall@10 of 0.6 made of half the queries at 1.0 and half at 0.2 is a different product than one where every query is at 0.6, and the mean cannot tell them apart. When you compare two embeddings, run them over the identical pair set with the identical pooling and the identical index settings, changing exactly one thing, or you have measured your pipeline rather than the model.
The index itself introduces its own recall loss, separate from the embedding’s. Approximate nearest neighbour search trades recall for speed by construction; measure against exact search on a sample so you know which of the two is costing you. The parameters are covered in HNSW, and the choice of distance in vector similarity metrics.
The failure that looks like success
The characteristic way audio retrieval goes wrong is that the embedding encodes the recording conditions rather than the content. Microphone response, room reverberation, preamp noise floor, codec artefacts and loudness normalisation all leave a consistent signature, and it is often more consistent than the semantic content you wanted. The result is a system that reliably returns other clips from the same recording session, which looks like it is working until someone queries with material from a different source.
You detect it by holding out an entire recording session or an entire source library and querying across the boundary. If cross-source recall collapses while within-source recall is high, that is the diagnosis. Mitigations are augmentation during any fine-tuning — random gain, reverb, resampling, codec round-trips — and normalising loudness before embedding, which removes the crudest of the channel cues for free.
Top comments (0)