Kempo: Real-Time Audio Time-Stretching, Pitch-Shifting, and Formant Control in Pure Kotlin Multiplatform

Zero-allocation streaming, phase-locked clarity, and multichannel spatial coherence across JVM, Android, iOS, and WebAssembly.

Whether you are building a podcast player with variable playback speed, a music practice app that slows down a complex guitar solo without dropping the tuning, a real-time voice modulator, or a spatial audio engine for a game, you eventually hit the same fundamental challenge: decoupling time from pitch.

In raw digital audio, playback speed and pitch are locked together. Play a buffer twice as fast, and the pitch jumps up a full octave. Slow it down, and everything drags into a muddy baritone.

Breaking that link—while keeping drums punchy, vocals natural, and the stereo image intact—requires a dedicated Digital Signal Processing (DSP) engine.

Meet Kempo (io.github.numq:kempo): an open-source, pure Kotlin Multiplatform audio time-stretching and pitch-shifting library built for studio-grade acoustic transparency and sub-millisecond real-time streaming across JVM, Android, iOS, and WebAssembly (Wasm/JS).

---

What Kempo Brings to Your Audio Stack

Kempo is designed to be drop-in simple for high-level application code while exposing full, granular control for low-level audio threads:

• Independent Time, Pitch, and Formant Control: Speed up or slow down audio (0.25x to 3.0x and beyond), transpose musical pitch in semitones, and shift vocal timbre independently.
• Zero-Allocation Real-Time Streaming: Once configured, the block-by-block processing loop allocates 0 bytes on the heap, eliminating Garbage Collection pauses on high-priority audio threads.
• Vertical Harmonic Phase-Locking: Keeps transients crisp and eliminates the hollow, "underwater" phasing artifacts common in basic time-stretchers.
• Multichannel & 5.1 Surround Coherence: Preserves exact inter-channel phase relationships across stereo and surround tracks so the soundstage never collapses.
• Sample-Accurate Offline Rendering: Built-in pre-roll warm-up, algorithmic latency compensation, and automatic overlap-add tail flushing.
• Built-In PCM-16 Serialization: Convert raw little-endian or big-endian WAV/PCM byte arrays directly into normalized floating-point tracks and back.

---

How Kempo Keeps Audio Crisp (Without the Artifacts)

If you have ever used a naive time-stretching algorithm, you probably recognize three classic audio defects: smeared drum hits, "chipmunk" vocals, and a collapsing stereo field. Kempo solves all three at the engine level.

1. Eliminating Transient Smearing with Vertical Phase-Locking

Standard phase vocoders split audio into hundreds of narrow frequency bins and advance the phase of each bin over time. Because each bin is updated in isolation, the natural harmonic alignment between a fundamental note and its overtones quickly drifts apart. Percussive attacks lose their snap, and instruments sound phasey and diffuse.

Kempo continuously analyzes the smoothed spectral energy contour of each frame to locate prominent spectral peaks. Instead of letting neighboring bins drift, Kempo performs vertical phase-locking—anchoring the phase trajectories of surrounding frequency bins to their dominant harmonic peaks. Harmonics move together as a unified acoustic event, keeping transients tight.

When stretching audio to extreme lengths (beyond 2.0x), Kempo automatically transitions into controlled stochastic phase dispersion, preventing the metallic ringing that usually plagues ultra-slow playback.

2. Fixing the "Chipmunk" Effect with Formant Decoupling

When you pitch a human voice up by +5 semitones, you aren't just raising the pitch of the vocal cords—you are also shrinking the perceived size of the singer's vocal tract. The result is the notorious "chipmunk" effect (or the "monster" effect when pitching down).

In Kempo, pitch and vocal timbre (formants) are decoupled. Kempo extracts the spectral envelope of the signal and allows you to shift or lock formants independently of the musical transposition:

• Natural Pitch Shifting: Enable formant compensation (compensatePitch = true) to keep a singer's vocal tract resonance anchored while changing their melody.
• Creative Voice Design: Shift formants up or down while keeping pitch at 0 st to transform the character and age of a voice without changing the note being sung.

3. Preserving Stereo Width and 5.1 Surround Imaging

Our ears localize sound in space by detecting tiny timing and phase differences between the left and right channels. If an audio processor stretches the Left and Right channels independently, their relative phases drift apart—causing the stereo image to wobble and center-panned vocals to hollow out when downmixed.

Kempo processes multichannel audio coherently. At every frequency band, it identifies the dominant channel and locks the Inter-channel Phase Differences (IPD) of all remaining channels to that reference. Whether you are processing a stereo master or a discrete 6-channel 5.1 surround mix, panning stays locked and center dialogue remains phase-aligned.

---

Getting Started

Kempo is published on Maven Central. Add the dependency to your build.gradle.kts:

kotlin {
    sourceSets {
        commonMain.dependencies {
            implementation("io.github.numq:kempo:1.0.0")
        }
    }
}

Or for a single-platform JVM/Android module:

dependencies {
    implementation("io.github.numq:kempo:1.0.0")
}

1. The One-Liner: Offline Track Processing

If you have raw 16-bit PCM bytes (from a WAV file, network stream, or microphone recording) and want to stretch or transpose them in a background coroutine, PcmConverter and AudioTrack handle the entire pipeline in a few lines:

import io.github.numq.kempo.PcmConverter

// 1. Decode interleaved 16-bit PCM bytes into a normalized [-1.0f, 1.0f] AudioTrack
val track = PcmConverter.fromPcm16(
    bytes = rawWavBytes,
    channels = 2,
    sampleRate = 44100f,
    isLittleEndian = true
)

// 2. Speed up by 25%, pitch up by 3 semitones, keep natural formants
val stretchedTrack = track.stretch(
    timeRatio = 1.25f,
    pitchSemitones = 3.0f,
    formantSemitones = 0.0f
)

// 3. Convert back to 16-bit PCM bytes for playback or file export
val outputBytes = PcmConverter.toPcm16(stretchedTrack, isLittleEndian = true)

2. Zero-Allocation Real-Time Streaming

Real-time audio callbacks operate under strict deadlines. Allocating arrays or objects inside the render loop will eventually trigger the garbage collector and cause audible clicks.

KempoStretch pre-allocates all ring buffers, FFT plans, twiddle factors, and spectral workspaces upfront during presetDefault() or configure(). Inside your audio callback, process() mutates pre-allocated buffers in place:

import io.github.numq.kempo.KempoStretch
import kotlin.math.max

class LiveAudioProcessor(
    channels: Int = 2,
    sampleRate: Float = 48000f
) {
    private val stretch = KempoStretch().apply {
        presetDefault(channels, sampleRate)
    }

    val outBlockSize: Int = stretch.defaultInterval
    private val outBuffer = Array(channels) { FloatArray(outBlockSize) }

    fun updateParameters(pitchSemitones: Float, formantSemitones: Float) {
        stretch.setTransposeSemitones(pitchSemitones)
        stretch.setFormantSemitones(formantSemitones, compensatePitch = true)
    }

    // Safe for high-priority audio threads: 0 B heap allocations
    fun processFrame(
        inputChannels: Array<FloatArray>,
        playbackSpeed: Float
    ): Array<FloatArray> {
        val inSamples = max(1, (outBlockSize * playbackSpeed).toInt())
        stretch.process(
            inputs = inputChannels,
            inputSamples = inSamples,
            outputs = outBuffer,
            outputSamples = outBlockSize
        )
        return outBuffer
    }
}

Need to support interactive scrubbing on a timeline without clicks or phase resets? Use stretch.seek() before resuming playback to pre-seed the internal phase accumulators with the audio frames right before the new cursor position:

val seekPreRoll = minOf(stretch.seekLength(), targetSampleIndex)
stretch.seek(
    inputs = track.channels,
    inputSamples = seekPreRoll,
    playbackRate = 1.0,
    inputOffset = (targetSampleIndex - seekPreRoll).coerceAtLeast(0)
)

3. Sample-Accurate Offline Rendering with exact()

When rendering an audio clip for a DAW timeline or a game asset pipeline, you often need the output buffer to match an exact target sample count—without leading silence from the STFT window latency or cut-off reverb tails at the end.

KempoStretch.exact() automates the entire offline lifecycle: it calculates the required pre-roll (outputSeekLength), warms up the vocoder state (outputSeek), processes the body of the track, and flushes the final synthesis tails (flush):

import io.github.numq.kempo.KempoStretch
import kotlin.math.roundToInt

fun renderExactClip(
    input: Array<FloatArray>,
    sampleRate: Float,
    timeRatio: Float,
    pitchSemitones: Float
): Array<FloatArray> {
    val channels = input.size
    val inSamples = input[0].size
    val outSamples = (inSamples / timeRatio).roundToInt().coerceAtLeast(1)
    val output = Array(channels) { FloatArray(outSamples) }

    val stretch = KempoStretch().apply {
        presetDefault(channels, sampleRate)
        setTransposeSemitones(pitchSemitones)
    }

    stretch.exact(
        inputs = input,
        inputSamples = inSamples,
        outputs = output,
        outputSamples = outSamples
    )
    return output
}

4. Custom Frequency Mapping & Tonality Limits

For experimental sound design, spectral effects, or specialized speech conditioning, Kempo lets you bypass standard semitone math entirely:

• Tonality Limits: When transposing speech or vocals up, multiplying high-frequency sibilance ("s" and "sh" sounds) can make them sound lispy or artificial. Passing a tonalityLimit to setTransposeSemitones(semitones, tonalityLimit) transitions the upper spectrum from multiplicative harmonic scaling to a clean linear shift.
• Custom Transfer Functions: Pass any lambda to setFreqMap to warp normalized frequencies [0.0f, 0.5f] non-linearly:

stretch.setFreqMap { normalizedFreq ->
    // Custom non-linear spectral warping
    (normalizedFreq * 1.15f).coerceIn(0f, 0.5f)
}

---

Engineered for Speed in Pure Kotlin

Writing high-throughput DSP in pure Kotlin without falling back to platform-specific C/C++ libraries requires careful attention to memory layout and CPU cache locality:

• Flat Power-of-Two Ring Buffers (MultiChannelBuffer): Instead of nested object arrays or modulo arithmetic, Kempo backs all channels with a single flat FloatArray. Wrap-around indices are resolved via bitwise masks ((head + offset) and mask), and overlap-add accumulation uses 4x unrolled loops.
• In-Place Mixed-Radix FFT (FastFft & ModifiedRealFFT): Supports radix-2, radix-3, radix-4, and composite window sizes with precomputed twiddle tables. Real signals are evaluated on a half-bin shifted grid (k + 0.5) / N, avoiding DC and Nyquist edge singularities and cutting transform overhead in half.
• Split Computation Mode (split = true): In real-time audio, doing a full STFT analysis and synthesis every N samples can cause periodic CPU spikes. Enabling split = true in configure() or presetCheaper() slices the spectral workload into small incremental steps distributed evenly across every sample frame.
• Automatic Silence Bypass: When input energy drops below the noise floor, Kempo automatically bypasses heavy spectral math while keeping ring buffers synchronized.

In JMH benchmarks on standard desktop JVM hardware (44.1 kHz audio), Kempo processes a real-time streaming block in ~220 μs (mono) and ~484 μs (stereo), and renders offline audio at over 128x real-time speed for mono and ~70x real-time for stereo.

---

Try It Live: The Compose Desktop Studio App

Want to hear how Kempo handles your own audio files before writing a line of code?

The repository includes an interactive Compose Multiplatform Desktop Studio application (:example). Drop any .wav file onto the window to inspect its waveform, scrub through the timeline in real time, and tweak live parameters on the fly:

• Playback Speed (0.25x – 3.00x) with instant presets (0.50x, 0.75x, 1.25x, 1.50x, 2.00x)
• Pitch Shift (-12 st to +12 st) in semitone steps
• Formant Shift (-12 st to +12 st) for real-time vocal timbre shaping

---

Explore the Code

Check out the full source code, architecture docs, and the runnable CLI attack simulation on GitHub.