Estimating the Side of Sudden Sounds in Unreal Engine 5.8: 35/35 Detected, 97.14% Side Accuracy

Hello, everyone.

Even in a noisy place, people can quickly direct their attention toward a sudden sound. Giving a game agent a similar input becomes more interesting when it cannot inspect each sound source directly and must decide “something happened” and “it came from the left” using only the audio that reached its ears.

Today, I mix 40 ms transients into continuous ambient sound in Unreal Engine 5.8, then detect them, estimate their side, and extract candidate clips using only post-HRTF stereo PCM. In a short fixed-listener demo, the system detected all 35 events and classified 34 of 35 sides correctly, for 97.14% accuracy. Detection latency ranged from 20 to 60 ms, with a median of 20 ms.

This result needs an important qualification: it predates the movable listener and comes from a short demo. The five-minute negative test and SI-SDR measurement of extraction quality have not been completed.

Binaural Audio in Unreal Engine

Unreal Engine 5 became generally available in April 2022. Its Audio Mixer goes beyond source playback and provides procedural synthesis, submix routing, DSP, and C++ APIs. This experiment uses Unreal Engine 5.8 from 2026.

Resonance Audio is a spatial-audio SDK released by Google in November 2017. It can apply HRTF, or Head-Related Transfer Function, processing that represents source direction through differences in arrival time, level, and frequency response at the two ears.

I configured BINAURAL_HIGH. Unreal Engine 5.8's Resonance Audio settings describe it as the high-quality mode using third-order Ambisonics. Its output here is two-channel binaural audio intended for headphones.

Licenses

Component	License
Unreal Engine 5.8	Unreal Engine EULA
Public Resonance Audio SDK	Apache License 2.0

Unreal Engine is not open-source software, and its EULA governs use and distribution. The public Resonance Audio source uses Apache License 2.0. When working with the plugin bundled into Unreal Engine or with Engine code, Epic's distribution terms must also be checked.

What I Tested

I mix the ambient and target sources, then capture the Main Output Submix as 48 kHz stereo after it reaches the listener. This PCM is the analyzer's only input. The detector does not receive source-actor positions or playback state directly.

• Whether per-band energy rise and full-band RMS rise can detect a 40 ms burst in ambient sound with low latency
• Whether masked GCC-PHAT on changed time-frequency bins plus ILD voting can determine left versus right
• Whether a soft mask derived from detection time and estimated direction can resynthesize a candidate clip

The complete C++ source, configuration, Automation Tests, and aggregate JSON are available in the unreal-mixed-audio-attention lab in kiarina/labs.

Reproducing the Lab

Install Unreal Engine 5.8, a compatible Xcode version, and mise on macOS. Set UE_ROOT to the Engine root. Because this is a C++ project, close the project in the Editor before building it.

git clone --depth 1 --filter=blob:none --sparse \
  https://github.com/kiarina/labs.git
cd labs
git sparse-checkout set .gitignore .mise/tasks Makefile mise.toml \
  2026/07/25/unreal-mixed-audio-attention

export UE_ROOT=/path/to/UE_5.8
mise -C 2026/07/25/unreal-mixed-audio-attention run
mise -C 2026/07/25/unreal-mixed-audio-attention run test
mise -C 2026/07/25/unreal-mixed-audio-attention run editor

The first mise ... run builds the Editor target and verifies the project structure. test runs Automation Tests for signal generation, side estimation, and the SI-SDR function. editor opens the ordinary demo with a five-second calibration phase.

After starting PIE, use W/S to move forward and backward, A/D to move sideways, and the mouse to rotate the camera and head listener. Touching a cyan ENV sphere or orange BURST sphere toggles that source. Press Shift+F1 to return control to the Editor.

From Mixed Audio to a Decision

The processing path is:

two continuous ambient sources + 40 ms targets from four directions
  -> Resonance Audio / HRTF
  -> 48 kHz stereo Main Output Submix
  -> five-second ring buffer
  -> 1024-point STFT, 480-sample hop
       ├─ onset: eight-band energy rise + full-band RMS rise
       ├─ side: 200–1500 Hz masked GCC-PHAT + 1.5–8 kHz ILD
       └─ extraction: build a soft mask from time and direction
  -> raw/extracted stereo WAV + run JSON + HUD

An STFT divides a short segment of audio into frequency components. At 48 kHz, 1,024 samples span about 21.3 ms, and the 480-sample hop advances by 10 ms.

1. Detecting a Transient

The analyzer divides the combined left/right spectrogram into eight bands and measures how much energy rose from the preceding context. A separate path measures the full-band RMS rise, and the larger value becomes the onset score.

An early version averaged the rise across three bands, which diluted a narrow burst with unchanged bands. Switching to the maximum band rise made the fast path more responsive to short sounds. The threshold is derived from the median and median absolute deviation of the scores collected during calibration.

2. Estimating Left or Right

GCC-PHAT finds the delay between the left and right waveforms through cross-correlation. Here, it uses only onset-changed bins between 200 and 1,500 Hz.

At higher frequencies, head shadow tends to produce a stronger level difference, so ILD, or Interaural Level Difference, from 1.5 to 8 kHz also votes on the result. If GCC-PHAT and ILD do not provide enough agreement, the system returns Unknown instead of forcing a side.

3. Extracting a Candidate Clip

The analyzer takes a one-second interval from 250 ms before the event to 750 ms after it. It uses the pre-event interval as a baseline, then gives more mask weight to bins whose energy rises above that baseline and whose ILD agrees with the estimated side. The same mask is applied to both channels before reconstruction with the inverse STFT.

Applying one mask to both channels avoids inventing a new interaural difference during extraction. The raw and extracted signals are saved as PCM16 stereo WAV files under Saved/MixedAudioAttention/.

Scene and Test Conditions

Two deterministic continuous-noise sources sit on an inner three-meter ring, while four target sources sit on an outer five-meter ring. Distance attenuation is disabled. Moving the listener therefore changes the HRTF direction relative to the head without using distance-dependent loudness as a cue.

Item	Condition
Engine	Unreal Engine 5.8
OS / machine	macOS 26.5.2 / Apple M4 Max
Audio	48 kHz stereo, Resonance Audio BINAURAL_HIGH
Ambient sound	Two deterministic continuous-noise sources
Targets	40 ms chirp / deterministic noise burst
Azimuths	-120 / -60 / +60 / +120°
Requested SNR	+12 / +6 / 0 / -6 dB
Analyzer	1024-point Hann STFT, 480-sample hop

The ordinary demo calibrates for five seconds and then cycles through enabled burst sources every two seconds. The implemented full mode uses a 60-second calibration, a 300-second negative period, and 160 events: two signal types, four azimuths, four SNR levels, and five repetitions.

The block pawn at the center of the screenshot carries the listener at head height. Listener yaw follows the camera. The top HUD shows position, orientation, score, threshold, side prediction, confidence, lag, ILD, and latency. The bottom dock displays two seconds of waveform, four seconds of spectrogram, and the latest event for both ears.

Observed Results

On July 25, 2026, I launched PIE through Unreal MCP and observed the fixed-listener demo from before the block pawn was added.

Metric	Observed value
Matched detections	35 / 35
Correct side	34 / 35 (97.14%)
Latency	20–60 ms, 20 ms median
Unmatched detections	0
Analyzer queue overruns	0

Every target had a corresponding detection, and there were no extra detections during this short run. The side was correct on 34 events. With processing advancing in 10 ms hops, responses arrived at a median of 20 ms and a maximum of 60 ms.

The HUD simultaneously updated the left/right stream waveforms, scrolling spectrograms, selected bins, detection markers, and raw/extracted waveforms for the latest event. The run also produced the expected WAV files and JSON record.

A Plain-Language Reading of the Results

1. The system found short changes and their side quickly from the mixed audio alone.

It did not use the Engine's knowledge of which actor emitted the sound as the detector input. Operating on post-HRTF stereo makes the design easier to adapt to inputs such as recordings or voice chat, where source metadata may not exist.

1. 35/35 is promising, but it is not a false-alarm evaluation.

The zero unmatched detections apply only to a short demo. The planned five-minute negative test has not run, so the provisional requirement of no more than one false alarm in five minutes has not been demonstrated.

1. Producing an extracted WAV is not the same as separating the sound well.

The soft mask generated output files, but there is no post-HRTF target-only reference yet. Without SI-SDR, improvement in extraction quality remains unmeasured.

Reproducibility Findings

Signal processing was only part of the work. Several Unreal-specific issues mattered:

• Adjusting volume alone did not prevent Resonance Audio's external send from becoming silent when PIE lacked focus. The experiment temporarily corrects VR focus and pause-on-focus-loss as well, then restores the previous values in EndPlay.
• The first implementation refilled procedural ambient audio on a timer and became silent after about two seconds. Queueing enough audio for the known experiment duration at startup fixed it.
• With an ordinary TArray<float> and non-aligned settings, Unreal Engine 5.8's FFT factory returned null. The analyzer now uses Audio::FAlignedFloatBuffer and 128-bit alignment.
• The audio callback only copies into a preallocated five-second ring buffer. A dedicated worker performs STFT processing so that analysis does not block the callback.

Limitations

• The aggregate comes from a short fixed-listener demo, not the current movable block pawn.
• The full 60-second calibration and 300-second negative run has not been performed.
• Results have not yet been broken down by SNR, signal type, or azimuth.
• No post-HRTF target-only reference exists, so raw and extracted SI-SDR are unmeasured.
• Generating a soft-masked WAV alone does not establish extraction quality.
• The experiment does not classify sound type or estimate front/back or elevation, occlusion, or reverberation.
• It uses a generalized HRTF and has not been tested with real-ear recordings or individualized HRTFs.
• It was observed on one setup: macOS 26.5.2, Apple M4 Max, and Unreal Engine 5.8.
• It assumes that the project is the only audio feeding the Main Output.

My Takeaway

If I let the code inspect source actors, determining left and right would be trivial. Recovering the signal from the actual mixed audio at the listener, detecting all 35 events, and getting 34 sides right felt much more useful. Walking through the scene and turning the listener also makes the HUD's numbers tangible: the same source changes in the binaural stream and in the analyzer's decision as the head rotates.

The extracted waveform looks convincing on screen, which makes it especially important not to call extraction successful while its quality metric is still blank. As a low-latency “auditory attention” sensor for an NPC or interactive agent, this is already a fun system to build with.