Audio visualization with Rust

Since the start of this year I have been very determined to learn more about rust. I knew I wanted to do something with audio but had no idea where to start. Thankfully Justin Wernick created Rusty Mic and wrote a blogpost about it. I decided to build my own simple 'microphone visualization' with a different graphics library. If you catch something that could be improved let me know, I'd love to learn!

If you just want to check out the code you can check out the repo I made for this small project:

maniflames / MicViz

A simple real time audio visualisation application build in Rust.

MicViz

A simple real time audio visualisation app build in Rust.
This is just a small and simple thing implemented with (rust) portaudio and three(-rs).

Getting started

Environment setup

You should be able to compile this project for any OS. For now this docs will focus on how to setup the enviroment on MacOS.

To setup the environment run the following commands (not everything may apply to you):

# Clone this repository
git clone https://github.com/maniflames/MicViz.git

# Install Rust with homebrew
brew install rust

# Install portaudio with homebrew
brew install portaudio

# Unpack MacOS SDK headers 
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

Compile and Run

Now that you're all set you can run this:

# Navigate to project
cd MicViz

# Compile and run with cargo
cargo run

Enjoy 😁

View on GitHub

Dependencies

For this project I'll be using two crates: portaudio for working with audio and three. So go ahead and put the following in your Cargo.toml:

[dependencies]
portaudio = "0.7.0"
three = "0.4.0"

Usually that would be all you'd have to do but in this case things are a little different. The portaudio crate just consists of Rust bindings to an actual installation of portaudio so you'll have to install that yourself. You can download and compile portaudio for Windows, Linux and MacOS. However if you happen to dev on a mac it's probably easier to install it through homebrew with pkg-config.

brew install pkg-config
brew install portaudio

Reading audio input

Let's read input from the default mic of your computer first. To accomplish this with portaudio's non-blocking inputstream. I'll try to be clear as possible by just commenting code line by line.

use portaudio;
use std::sync::mpsc::*;

fn main() {

    // Construct a portaudio instance that will connect to a native audio API
    let pa = portaudio::PortAudio::new().expect("Unable to init PortAudio"); 
    // Collect information about the default microphone
    let mic_index = pa.default_input_device().expect("Unable to get default device");
    let mic = pa.device_info(mic_index).expect("unable to get mic info");

    // Set parameters for the stream settings.
    // We pass which mic should be used, how many channels are used,
    // whether all the values of all the channels should be passed in a 
    // single audiobuffer and the latency that should be considered 
    let input_params = portaudio::StreamParameters::<f32>::new( mic_index, 1, true, mic.default_low_input_latency);

    // Settings for an inputstream.
    // Here we pass the stream parameters we set before,
    // the sample rate of the mic and the amount values we want to receive
    let input_settings = portaudio::InputStreamSettings::new(input_params, mic.default_sample_rate, 256);

    // Creating a channel so we can receive audio values asynchronously
    let (sender, receiver) = channel(); 

    // A callback function that should be as short as possible so we send all the info to a different thread
    let callback = move |portaudio::InputStreamCallbackArgs {buffer, .. }| {
        match sender.send(buffer) {
            Ok(_) => portaudio::Continue, 
            Err(_) => portaudio::Complete
        }
    };

    // Creating & starting the input stream with our settings & callback
    let mut stream = pa.open_non_blocking_stream(input_settings, callback).expect("Unable to create stream"); 
    stream.start().expect("Unable to start stream");

    //Printing values every time we receive new ones while the stream is active
    while stream.is_active().unwrap() {
       while let Ok(buffer) = receiver.try_recv() {
            println!("{:?}", buffer); 
       }
    }
}

Running this should look something like this:

Visualizing the audio

If you look closely you'll notice that all you get back from your microphone is a slice containing numbers between -1.0 and 1.0. These numbers are a representation of how sound waves have manipulated a moving part within your microphone over time. There are different ways to visualise this change but to keep it simple let's draw a line for every change that occurs in time. We'll animate the lines by simply removing them and drawing new lines when we receive new changes.

use portaudio;
use std::sync::mpsc::*;
use three;

//struct for storing the application state
#[derive(Debug)]
struct State {
    sound_values: Vec<f32>,
    scene_meshes: Vec<three::Mesh>
}

fn main() {
    // Receiving audio input 
    let pa = portaudio::PortAudio::new().expect("Unable to init PortAudio"); 
    let mic_index = pa.default_input_device().expect("Unable to get default device");
    let mic = pa.device_info(mic_index).expect("unable to get mic info");

    let input_params = portaudio::StreamParameters::<f32>::new(mic_index, 1, true, mic.default_low_input_latency);
    let input_settings = portaudio::InputStreamSettings::new(input_params, mic.default_sample_rate, 256);

    let (sender, receiver) = channel();

    let callback = move |portaudio::InputStreamCallbackArgs {buffer, .. }| {
        match sender.send(buffer) {
            Ok(_) => portaudio::Continue, 
            Err(_) => portaudio::Complete
        }
    };

    let mut stream = pa.open_non_blocking_stream(input_settings, callback).expect("Unable to create stream"); 
    stream.start().expect("Unable to start stream"); 

    // Create a full screen window with a black background
    let mut builder = three::Window::builder("My Mic"); 
    builder.fullscreen(true); 
    let mut win = builder.build(); 
    win.scene.background = three::Background::Color(0x000000);

    // Create a variable that will contain the state off the app
    let mut state = State {
        sound_values: Vec::new(),
        scene_meshes: Vec::new()
    };

    // Create a camera that will be put in the scene on location 0.0, 0.0
    let camera = win.factory.orthographic_camera([0.0, 0.0], 1.0, -1.0 .. 1.0); 

    //Animation loop that will run until you press ESC or exit the program
    while win.update() && !win.input.hit(three::KEY_ESCAPE) {
        // Put new lines in the scene temporarily save them in the state
        update_lines(&mut win, &mut state);
        // Show the lines
        win.render(&camera);
        // Remove all lines from the scene and the state
        remove_lines(&mut win, &mut state);

        //Update state
        while let Ok(buffer) = receiver.try_recv() {
            update_sound_values(&buffer, &mut state); 
       }
    }
}

// Pass new samples into the state by overriding the vector
fn update_sound_values(samples: &[f32], state: &mut State) {
   state.sound_values = samples.to_vec(); 
}

// Put new lines in the scene temporarily save them in the state
fn update_lines(win: &mut three::window::Window, state: &mut State) {
    for (index, y_position) in state.sound_values.iter().enumerate() {

        // calculate the x position of the line by calculating a normalized x position between 0.0 and 1.0 (i / num_samples)
        // With the scale variable the size of the visualization can be changed. 
        let i = index as f32; 
        let num_samples = state.sound_values.len() as f32; 
        let scale = 3.0; 
        let x_position = (i / (num_samples / scale)) - (0.5 * scale);

        // create the geometry for a line with the calculated positions
        // three is a 3D graphics library so we pass the x, y, z values
        let geometry = three::Geometry::with_vertices(vec![
            [x_position, y_position.clone(), 0.0].into(),
            [x_position, -y_position.clone(), 0.0].into()
        ]);

        // create material so for our line, in this case white line material
        let material = three::material::Line {
            color: 0xFFFFFF,
        };

        // create a 3D object from the geometry and the material
        let mesh = win.factory.mesh(geometry, material);

        // Put the line in the scene and store it in the state
        win.scene.add(&mesh); 
        state.scene_meshes.push(mesh); 
    }
}

// Remove all lines from the scene and the state
fn remove_lines(win: &mut three::window::Window, state: &mut State) {
    for mesh in &state.scene_meshes {
        win.scene.remove(&mesh); 
    }

    state.scene_meshes.clear(); 
}

If you run the script it should look something like this:

Enjoy your MicViz, if you have some time on your hands show off your version of it 😄👀

And again, let me know any way to improve presented code if you happen to know a thing or two about Rust!

Comments

[UKJP]

Hey! I'm just starting out on my dive into rust now it's matured enough that there aren't (many) breaking changes in a 0.0.1 build increment, be great to have someone else around the same progress in rust to go over a few projects I have and keep the motivation going, plus some of them have a few commercial applications and the other contribute freely to the owasp foundation of which play (small) part in. If your interested GitHub /sam-aldis/ although I switched to gitea and bit bucket a while ago.. Or samuel.aldis at owasp.org - this is extended to anyone reading these comments as well, especially if your interested in becoming a part of the security community at owasp. Free (details under wraps) distributed vm for anyone who can help :)

[Maniflames]

That's super nice!
Not gonna lie I know nothing about security but there's nothing wrong with learning about it while diving into Rust. I'm still pretty new myself but determined to do some graphics and/or embedded stuff with Rust this summer. Let's chat from time to time here on DEV would love to see your work 😁

[UKJP]

I'm so new to Rust too, I want to get into UI development in it plus with its wasm support I want to build the next generation of security toolkits that runs within any web-browser, platform independent with the majority written in rust (I've always skipped hello worlds and gone way too big to start, my first project in VB6 (before .net 👴🏻) when I was 16 was also my first - and only - Trojan Virus with remote screen shots etc. and strangely although I only published it on a small page and didn't link to it anywhere, sophos picked it up within a month, but its still there.. imgur.com/a/oQDcuEa which just shows some web services (lycos) have quite some sticking power, even for their free hosting! and rust itself is so secure by using life-times, ownership etc, it will litterally change owasp's top 10 because you have to TRY to code insecurely so as long as you've got configuration errors and incorrect permisions down then the old vulnerabilities like race conditions, memory leaks, bufer overflows/underruns, read-after-free etc will be a thing of the past!

anyway It would be good to pick up rust at the same time as someone else to make sure I have my head around everything correctly - its the first language from the ML family I've learnt - and to help each other stay focussed, even if you don't have time to help on my project, I'd love to learn graphic based stuff from you.. will be following!

[Maniflames]

Will be following you too!

Also in case you haven't bumped into it yet Qt has webassembly support! I'm pretty sure there are some crates out there that allow you to use Qt with Rust.