Hello, everyone.
When people hear an animal call or keyboard typing, they can infer something about the source and setting. If we turn those sounds into numerical vectors, does their distance preserve the same kind of meaning?
Today, I will export the CLAP audio encoder to ONNX and compare its environmental sound embeddings across the 50 classes in ESC-50.
What I Tested
CLAP (Contrastive Language-Audio Pretraining) maps audio and text into a shared embedding space. This experiment runs only the audio encoder from laion/clap-htsat-unfused with ONNX Runtime and asks:
- Whether it consistently produces a normalized 512-dimensional embedding from a five-second environmental sound
- Whether a different recording from the same class becomes the nearest neighbor by cosine similarity
- Whether sounds still cluster near the correct one of five coarse categories when exact classes do not match
- How long the pipeline takes on an Apple Silicon CPU
The complete code and reproducible environment are available in the clap-onnx-esc50 lab in kiarina/labs.
Reproducing the Lab
You will need the following tools, an internet connection for the first run, and approximately 2 GB of free space:
The following commands fetch only this lab and run it:
git clone --depth 1 --filter=blob:none --sparse \
https://github.com/kiarina/labs.git
cd labs
git sparse-checkout set .gitignore .mise/tasks Makefile mise.toml \
2026/07/06/clap-onnx-esc50
mise -C 2026/07/06/clap-onnx-esc50 run
On the first run, the task downloads the checkpoint from Hugging Face, generates model.onnx inside the lab, and performs the evaluation. Later runs verify and reuse the ONNX model. The task also downloads ESC-50 metadata and the selected 100 WAV files from a pinned dataset revision.
Model and Dataset
The model is pinned to commit 84bcbbd1d619e407a8216371ddef36e458d95d93 of laion/clap-htsat-unfused. Using PyTorch 2.8.0, Transformers 4.57.3, and opset 18, the lab exports the audio encoder, projection layer, and L2 normalization.
SHA-256: b23099962830b1afa5398efbb6f5321ef8f63f8fcf93f5019837c47118a8a1c5
inputs: input_features [batch, 1, 1001, 64], is_longer [batch, 1]
output: embeddings [batch, 512]
This ONNX graph does not include the text encoder. The experiment therefore compares sounds with sounds instead of performing CLAP zero-shot classification from textual class names.
The input comes from ESC-50, a dataset of 2,000 five-second environmental recordings divided into 50 classes and five coarse categories:
- animals
- natural soundscapes and water
- human non-speech
- interior and domestic
- exterior and urban
The dataset is pinned to revision 33c8ce9eb2cf0b1c2f8bcf322eb349b6be34dbb6. The full ESC-50 dataset is distributed under CC BY-NC, so the noncommercial restriction applies to this experiment, which uses all 50 classes.
A Training-Free 1-NN Comparison
Instead of using all 2,000 recordings, the evaluation selects two clips per class, for 100 clips in total:
- Take the first filename for every class in fold 1 as a 50-clip reference set
- Take the first filename for every class in fold 5 as a 50-clip query set
- Rank the reference clips for each query by cosine similarity
- Check the nearest class, the top five classes, and the nearest coarse category
There is no classifier training and no five-fold cross-validation. This is a fixed 50-way, one-nearest-neighbor evaluation with only one reference recording representing each class.
The preprocessing pipeline is implemented in NumPy based on the checkpoint's preprocessor_config.json.
| Setting | Value |
|---|---|
| sample rate | 48 kHz mono |
| input length | 10 seconds |
| five-second input handling | repeatpad |
| FFT size / hop length | 1,024 / 480 |
| frequency range | 50 Hz–14 kHz |
| mel filter bank | 64-bin HTK |
| embedding dimension | 512 |
| normalization | L2 |
| similarity | cosine similarity |
| execution provider | CPUExecutionProvider |
Each five-second ESC-50 clip is repeated and then padded to ten seconds. The implementation passes its log-mel spectrogram to ONNX and L2-normalizes the output again. For normalized embeddings, a dot product is equivalent to cosine similarity.
Results
One run on a Mac Studio without a warm-up produced:
fine accuracy@1: 45/50 = 90%
fine accuracy@5: 50/50 = 100%
coarse accuracy@1: 47/50 = 94%
100 clips elapsed: 5.458 s
seconds per clip: 0.055 s
For 45 of the 50 queries, the nearest reference was a different recording of the same fine-grained class. The correct class appeared among the top five references for every query. At the coarse-category level, 47 queries matched.
The five fine-grained top-1 errors were:
| actual | nearest | cosine similarity | coarse category |
|---|---|---|---|
| crickets | frog | 0.690 | mismatch |
| footsteps | door_wood_knock | 0.603 | mismatch |
| drinking_sipping | brushing_teeth | 0.492 | match |
| mouse_click | keyboard_typing | 0.616 | match |
| airplane | wind | 0.717 | mismatch |
drinking_sipping → brushing_teeth and mouse_click → keyboard_typing missed the exact class but stayed within the same coarse category. They are also sounds associated with similar actions or settings. The remaining three errors crossed coarse-category boundaries.
The verification environment was:
- machine: Mac Studio (Mac16,9)
- chip: Apple M4 Max
- OS: macOS 26.5.1 (25F80), arm64
- Python: 3.12.11
- NumPy: 2.5.1
- ONNX Runtime: 1.27.0
- execution provider: CPUExecutionProvider
The elapsed time covers FFmpeg decoding, NumPy preprocessing, ONNX inference, and L2 normalization for all 100 files. It excludes downloads and model initialization. This was a single run without a warm-up, so it is a reference measurement rather than a rigorous benchmark.
Interpreting the Results
For these 100 selected clips, the CLAP audio embeddings preserved environmental sound categories remarkably well. Fine-grained top-1 accuracy reached 90%, and top-5 reached 100%, even though each class had only one reference. This suggests that the embedding distance captured information related to what produced the sound rather than merely matching identical acoustics.
The errors are interesting as well. Mouse clicks were close to keyboard typing, while drinking was close to tooth brushing. For audio search, returning semantically related sounds may be useful even when the exact label differs.
However, the 90% result is not directly comparable to a standard ESC-50 classification score. The evaluation uses only a fixed subset of folds 1 and 5 and depends heavily on one reference recording per class. It is neither cross-validation over all 2,000 clips nor a test of generalization to an unknown dataset. I also did not investigate possible overlap between CLAP's training data and ESC-50.
The ESC-50 repository documents potential information leakage from class-dependent preprocessing of the original source recordings. In addition, I have not verified that the NumPy preprocessing implementation is numerically identical to Hugging Face's ClapFeatureExtractor. Both are limitations when interpreting the metrics.
The pipeline took about 55 ms per clip. Because preprocessing represents each clip as ten seconds of audio, the result is fast enough to consider for real-time environmental sound retrieval. Still, this sequential 100-clip run does not compare batching or execution providers.
My Takeaway
After testing speaker embeddings, environmental sound embeddings make it clear that the phrase "turn audio into a vector" can hide very different kinds of retained information. Seeing separate recordings of frogs or typing return to nearby positions through nothing more than a 512-dimensional dot product makes CLAP feel promising as a foundation for audio search.
At the same time, a high similarity does not explain itself. The airplane → wind error scored 0.717; the model may have focused on wind noise in the airplane recording, or on something else entirely. This evaluation cannot tell us why.
Next, I would like to test an audio tagging model and see whether it can label multiple acoustic events within a single recording. This will offer a different perspective from embedding-based similarity search and help reveal how much we can identify about what is happening in an environment.
Top comments (0)