Sending Audio Over Sockets Using Socket.io and Tone.js

Recently, I've been working to create collaborative music chat rooms. I knew to do this I'd need to send audio information over a socket connection. At first, I assumed this would be a long, difficult problem, but when I took a step back I realized that I could send relatively simple information over sockets, as long as I implemented a way to interpret that data when received and trigger the correct audio response.

Of course, the first step was to set up my socket connection. For my back end, I'm using Express. Here's a simplified version of my initial setup:

const express = require('express');
const socketio = require('socket.io');

const app = express();

const server = app.listen(8081);

const io = socketio(server);

io.on('connection', (socket) => {
  // socket event listeners go here
});

I'm using VueJS for my front end, so I took advantage of Vue-Socket.io, which allows for some convenient ways to emit and receive socket messages from any component within your app. Now, if you're not using Vue, don't give up on this guide! Some of your client-side will necessarily look different, but the logic should be pretty similar.

Now, to be able to use Vue-Socket.io, I needed to set up a client-side connection to the server-side socket, and then use that as the connection option for my globally available Vue-Socket.io plugin.

import 'socketio' from 'socket.io-client';
import 'VueSocketIO' from 'vue-socket.io';

// make connection to server socket
export const connection = socket.io(':8081');

// make Vue-Socket.io globally available
Vue.use(VueSocketIO, connection);

Thanks are due to this very clear guide by Joshua Bemenderfer for this setup.

Now, I wanted to create a way for the music jam rooms to be distinct from each other, meaning that only users in the same room would be able to hear each other, rather than everyone being lumped together in one noisy room. This would mean assigning each jam room a unique socket namespace, which could be accessed through that jam room's URL query.

First, I dealt with the situation in which someone makes a new jam room, simply by navigating to the Jam component. I'm using Vue Router, which allows for a variety of navigation-related hooks. In this case, I used beforeRouteEnter, which executes before the navigation is completed, to add a random string as the route's query. I used the aptly named randomstring library to create this random query. My plan was to use this query as the unique namespace for the jam room, meaning that anyone with the link would automatically be connected to the same namespace.

beforeRouteEnter(to, from, next) {
  if (to.fullPath === '/jam') { // only runs if this is a new jam room
    next({
      path: '/jam',
      query: { room: randomstring.generate() },
    });
  } else { // do nothing if following a link to existing room
    next();
  }
}

This gives me a unique URL, e.g. http://localhost:8080/jam?room=WwwCbmpKGaPBoXNAvKx9ln2wfbfYCDZ7. If not using Vue Router, the above could also be accomplished by manually setting window.location.search for certain conditions.

Now that every jam room had its own unique query, the next step was to use that query to connect to a socket namespace. On mount, the unique namespace was accessible with this.$route.query.room. Thanks to Vue-Socket.io, I was able to emit a socket message from any component using this.$socket.

mounted() {
 this.$socket.emit('join', this.$route.query.room);
}

This sends out a 'join' message for this particular room, but is anyone listening? Not at the moment. I needed to add a server-side listener for this message that would subscribe this particular client's socket to this room's namespace:

io.on('connection', (socket) => {
  socket.on('join', (room) => socket.join(room));
});

Now, with distinct jam rooms, the next step was to actually start sending some audio information over those sockets. At this point, I already had a keyboard-controlled piano that made sound on the individual client (you can read about its setup here and here), so it was just a matter of sending and interpreting messages that allowed those same sounds to be reproduced on other clients.

The meat of the piano code is the piano change event handler, which starts a note when a key is toggled on, and stops the same note when that key is toggled off:

piano.on('change', (k) => {
  if (k.state) {
    if (!activeSynth[k.note]) {
      activeSynths[k.note] = new Tone.Synth().toMaster();
    }
    activeSynths[k.note].triggerAttack(note(k.note));
  } else {
    activeSynths[k.note].triggerRelease();
  }

So what if I just sent out start and stop messages to the appropriate room, containing the note in question?

piano.on('change', (k) => {
  if (k.state) {
    if (!activeSynth[k.note]) {
      activeSynths[k.note] = new Tone.Synth().toMaster();
    }
    activeSynths[k.note].triggerAttack(note(k.note));
    this.$socket.emit('startNote', {
      note: k.note,
      room: this.$route.query.room
    });
  } else {
    activeSynths[k.note].triggerRelease();
    this.$socket.emit('stopNote', {
      note: k.note,
      room: this.$route.query.room
    });
  }

Working with sockets is really like coordinating a game of catch. Now that my piano was sending out start and stop messages, I needed my server to catch these messages and then distribute to the other clients in the same room. Because I didn't want the user playing the piano to receive their own sound, I used socket.broadcast to exclude their client.

socket.on('startNote', ({ note, room }) => {
  socket.broadcast.to(room).emit('receiveStart', note);
});

socket.on('stopNote', ({ note, room }) => {
  socket.broadcast.to(room).emit('receiveStop', note);
});

Finally, I bounce back to the client and add some socket listeners there. With Vue-Socket.io, I can do this in the sockets object, with methods named for the messages for which they listen. Upon receiving a signal to start or stop a note, I essentially wanted to reproduce the behavior already in place locally when a user presses a key on their keyboard.

sockets: {
  receiveStart(midi) {
    // if there is not yet a dedicated synth for this note, create one
    if (!this.activeExternalSynths[midi]) {
      const synth = { [midi] : new Tone.Synth().toMaster() };
      // add new synth to activeExternalSynths object
      this.activeExternalSynths = {...this.activeExternalSynths, ...synth };
    }
    // start playing the note
    this.activeExternalSynths[midi].triggerAttack(note(midi));
  },
  receiveStop(midi) {
    // release the currently playing note
    this.activeExternalSynths[midi].triggerRelease();
  },
},

And that's all there is to it! Of course, as you start wanting to create more and more complex sounds, there will be more information you need to send and interpret to share these sounds with other users, but this should be a good starting point. Thanks for reading!