Can YAMNet Detect Unseen Sudden Sounds in Real Time? A 48-Stream Evaluation

Hello, everyone.

There are many situations where we may want to monitor sounds that happen without warning, such as breaking glass or a car horn. Registering every possible sound in advance, however, is not realistic.

Today, I am testing whether YAMNet can detect that a sound stream has changed, without first specifying which sound it should recognize.

The short answer is that the best configuration detected 22 of 48 events, for 45.8% recall. Processing was easily fast enough, but the results did not support the hypothesis that a simple distance over YAMNet embeddings can reliably detect unseen sudden sounds.

What Is YAMNet?

YAMNet is an audio classification model that Google added to TensorFlow Models on November 21, 2019. It was trained on AudioSet and predicts 521 acoustic event classes, including speech, rain, vehicles, and animals.

The network uses MobileNet V1, a convolutional architecture designed to reduce computation on mobile hardware. It reads roughly 0.96 seconds of 16 kHz mono audio at a time and returns:

• scores (521 dimensions): confidence for each class
• embedding (1,024 dimensions): a compact numerical representation of the sound
• spectrogram (64 bands): frequency content over time

This experiment needs the embedding as well as the class scores, so I used the SavedModel version on TensorFlow Hub.

The relevant licenses are:

• TensorFlow Models, including YAMNet: Apache License 2.0
• TensorFlow: Apache License 2.0
• ESC-50: Creative Commons Attribution-NonCommercial 3.0 (CC BY-NC 3.0)

ESC-50 includes a noncommercial restriction. The model and dataset have separate licenses, so both need to be considered for redistribution or product use.

What I Tested

The goal was to determine whether the detectors could notice a new sound mixed into a normal background, without being told the event label.

• Whether a simple frequency-change detector can catch sudden events
• Whether YAMNet scores or embeddings represent the change more clearly
• Whether distance from the previous sound works better than distance from a memory of normal sounds
• Whether the pipeline can process audio faster than real time

The complete code and results are available in the yamnet-streaming-novelty lab in kiarina/labs.

Reproducing the Lab

You will need mise, uv, FFmpeg, and an internet connection for the first download. The model and data use about 120 MB, while the Python environment uses about 1.3 GB.

git clone --depth 1 --filter=blob:none --sparse \
  https://github.com/kiarina/labs.git
cd labs
git sparse-checkout set .gitignore .mise/tasks Makefile mise.toml \
  2026/07/17/yamnet-streaming-novelty
mise -C 2026/07/17/yamnet-streaming-novelty run

The first run downloads the pinned YAMNet model and 208 required WAV files from a fixed ESC-50 revision. The task verifies the SHA-256 of the YAMNet archive and writes the results to output/report.json.

Roles and Data Flow

YAMNet is the only AI model in this pipeline. Small detectors for frequency change and vector distance operate around it.

16 kHz mono audio stream
  ├─ frequency change every 32 ms ─────────> spectral flux
  └─ 0.975 s window sent to YAMNet every 0.48 s
       ├─ 521 class scores
       │    ├─ distance from previous frame -> score delta
       │    └─ distance from normal memory -> score kNN
       └─ 1,024-dimensional embedding
            ├─ distance from previous frame -> embedding delta
            └─ distance from normal memory -> embedding kNN

The delta detectors measure change from the immediately previous sound. The kNN detectors compare each frame with the five nearest examples in a memory of normal sounds. They use cosine distance, which measures the difference in direction between numerical vectors.

I also evaluated temporal fusion, which produces an alert when either score delta or embedding delta fires.

Evaluation Setup

The normal sounds were rain, sea_waves, wind, and clock_tick. I inserted one of crying_baby, door_wood_knock, glass_breaking, siren, car_horn, and fireworks from 2.0 to 3.0 seconds in the same five-second background clip. There were eight examples of each event, for 48 positive streams in total.

Event-to-background level was tested at -10, -5, 0, and +5 dB. At -10 dB, the event is substantially quieter than the background; at +5 dB, it is louder. The mixed streams existed only in memory during evaluation and were not saved as files.

Recordings were separated by purpose:

Purpose	ESC-50 fold	Number of clips
Normal-sound memory	1-3	96
Threshold calibration	4	32
False-alert evaluation	5	32
Sudden-event evaluation	5	48 streams

Each threshold was set high enough to produce no alerts on the normal fold-4 calibration streams. I did not tune it against the final fold-5 evaluation set.

Results

Which Score Ranked Anomalies Best?

I first compared whether positive streams received higher anomaly scores than negative streams without fixing a threshold. AUROC summarizes this ranking: 1.0 is ideal, while a value around 0.5 is close to chance.

Detector	AUROC	-10 dB	-5 dB	0 dB	+5 dB
spectral flux	0.449	0.393	0.315	0.555	0.534
score delta	0.717	0.622	0.672	0.776	0.797
embedding delta	0.734	0.638	0.625	0.815	0.859
score kNN	0.661	0.484	0.698	0.789	0.672
embedding kNN	0.632	0.479	0.581	0.721	0.745

Embedding delta ranked first at 0.734, but it was only 0.018 ahead of score delta. These 80 evaluation streams are not enough to conclude that embeddings are generally better.

The broader pattern is clearer: distance from the previous frame worked better than distance from the normal-sound memory. For short events like these, asking "did the sound suddenly change?" was more useful than asking "is this sound globally unlike the normal set?"

Alerts with Strict Thresholds

Detector	Precision	Recall	F1	False alerts/hour	Median latency
spectral flux	0.000	0.0%	0.000	0.0	—
score delta	0.810	35.4%	0.493	22.5	0.415 s
embedding delta	0.765	27.1%	0.400	22.5	0.415 s
score kNN	1.000	6.2%	0.118	0.0	0.895 s
embedding kNN	0.800	16.7%	0.276	0.0	0.895 s
score delta + embedding delta	0.786	45.8%	0.579	22.5	0.415 s

Precision is the proportion of emitted alerts that were correct. Recall is the proportion of the 48 events that were found. The best temporal-fusion detector found 22 and missed 26.

There was only one false alert in 160 seconds of normal audio. The reported 22.5 false alerts/hour extrapolates that single event to one hour, so it is a highly uncertain estimate rather than a production false-alert rate. Several hours of continuous audio would be needed for a useful measurement.

Recall by event level was:

Event level	Detected	Recall	Median latency
-10 dB	4/12	33.3%	1.855 s
-5 dB	3/12	25.0%	0.415 s
0 dB	7/12	58.3%	0.415 s
+5 dB	8/12	66.7%	0.415 s

By class, the detector found 6/8 glass-breaking events, 5/8 fireworks, and 4/8 car horns. It found only 2/8 crying-baby and 2/8 siren events. Short, sharp changes were relatively easy, while sounds that blended into the background or were weak in the selected one-second segment were harder.

What Did Not Work

With a looser threshold, simple spectral flux reached 29.2% recall but produced 967.5 false alerts/hour. Rain and waves naturally contain many frequency changes, so this score could not isolate unusual events.

Combining spectral flux with embedding kNN also failed to help. Under the strict threshold it matched embedding kNN alone at 16.7% recall. Under the looser threshold it reached 45.8% recall but produced 607.5 false alerts/hour. Adding detectors was not automatically an improvement.

A high novelty score also does not guarantee a correct YAMNet label. Glass was the top label for only three of the eight glass-breaking events. At low event levels, background labels such as Water, Rain, and Vehicle often remained on top. Detecting that something changed and explaining what changed are separate problems.

Processing Speed

I processed 1,040 seconds of audio as sequential windows on a Mac Studio with an Apple M4 Max.

feature extraction elapsed: 6.262 s
real-time factor:           0.0060x

A repeat run while writing this article took 6.921 seconds, for a real-time factor of 0.0067x. The detection counts, AUROC values, thresholds, and latencies matched the original run. Including YAMNet inference and spectral flux, both runs processed audio more than 150 times faster than its duration. Compute throughput was not a problem.

However, YAMNet reads about 0.96 seconds at a time, so the shortest observed detection latency was still 0.415 seconds. Fast inference does not remove the wait introduced by the input window.

The verification environment was:

machine: Mac Studio (Mac16,9)
chip: Apple M4 Max
OS: macOS 26.5.2, arm64
Python: 3.12.10
TensorFlow: 2.21.0
NumPy: 2.3.5
FFmpeg: 8.1.2
random seed: 20260717

A Plain-Language Reading of the Results

The results come down to three points.

1. The pipeline was fast enough.

It processed 1,040 seconds of audio in 6.262-6.921 seconds across two runs. The compute throughput needed for real-time monitoring was available.

1. Detection accuracy was not good enough.

The best method found only 22 of 48 events. Quiet events were especially likely to disappear into the background. This is not ready for a safety-monitoring application.

1. Recent change mattered more than distance from normal.

Comparing the embedding with a memory of normal sounds was weaker than comparing each frame with the previous one. Local change was the more useful signal for these short events.

This experiment is limited to 48 synthetic five-second streams, 160 seconds of negative audio, ten selected ESC-50 categories, and one Mac. Continuous microphone input, several-hour false-alert measurements, different recording devices, and environments with multiple simultaneous events remain outside its scope.

My Takeaway

YAMNet was lightweight and easy to use as a foundation because it exposes both class scores and embeddings. Still, an embedding may contain useful acoustic information without its raw distance being a reliable anomaly score.

In this experiment, recent change worked more directly than storing a large normal-sound memory. A lightweight first stage that proposes sharp events such as glass breaks or fireworks, followed by YAMNet labels or another decision step, looks like a more promising use of this approach.