DEV Community: KP Kaiser

Building a Voice Controlled Flappy Bird

KP Kaiser — Tue, 19 Dec 2017 22:07:21 +0000

Flappy Bird is a very simple game, one where a user would tap their cell phone screen, to make a bird "flap", and avoid pipes. It was an unexpectedly successful game, going viral after having been picked up by the YouTube gaming commnunity as an especially frustrating game to play.

In today's post, we'll remake Flappy Bird, but this time, we'll use pitch detection and vocal range to make it a voice controlled singing game. Our bird will fly at the level of the current note we sing.

Because we'll be dealing with musical notes, this post will walk through an introduction into music theory, and then jump into some basic tools for building a musical game.

If you've ever played anything like Rocksmith or Rock Band, we'll be using the same sort of pitch detection and note correlation to give our player feedback that they've hit the right notes.

A (Short) Background of Music Theory

Before we dive in to the code, we should probably first talk about human perception of sound, and how music works in the West (in 2017).

In general, we start from a completely arbitrary center for most of Western music.

Collectively, we've decided that A4 is equal to 440Hz. This means that all the other places for notes are relative to this. With A4 at 440Hz, we then create all the other notes around it.

From here, we have what are called "Octaves". Each time we double or half this frequency (880Hz, 220Hz), we get the same note A, but either one octave higher, or one octave lower. As we go down, we subtract from the second number, so 220Hz is A3, and when we go up an octave, we add to the number. (880Hz is A5).

Each octave is then broken down into twelve parts. Each of these twelve parts are supposed to be equal distance from the next, ending on the octave. We call these "half-steps".

The whole thing loops around all over again, at the next octave.

You'll notice some notes have # at then end of them. These represent "sharps", or in between notes. Most notes have in between notes, with the exception of B and C, and E and F. These notes are unique, as they are both a half step away from one another.

We start our scale on C, and not A. Why is that? It turns out, music has been a multi-century invention, and musical notes were invented the half note scale we're talking about. So we went back and put our new scale on the already existing musical notation.

You can see a great video explaining the reasons for this confusing setup here.

As we get into the higher octaves, the distances between notes in Hertz gets bigger and bigger. For example the distance between A1 and A2 is only 55 Hertz, but the distance between A5 and A6 is 880 Hertz! If we're going to look at distances between notes, we'll need to adapt to the octave we're in to measure distances.

So, what does music theory have to say about all the "gaps" between the actual notes? How do we manage them, and how do we label the "in-between" notes from imperfect singers?

Introducing the Cent, Or A Hundredth of a Half-Step

"Cents" are a unit of measuring in between half steps. Using cents, we can tell how far away a note if from the "perfect pitch", or ideal sound for our note.

A cent is one-one hundredth of a half step. So, using cents, we can tell how far we are from our ideal note, regardless of what octave we're working at. Each cent should evenly spaced, regardless of our octave, because each individual half-step is different, depending on our octave.

A proper implementation of cents should allow us to measure how close or far away our singer is from the proper pitch, regardless of the octave they're singing in.

Converting Between Pitch and Musical Notes

aubio, the pitch detection framework we introduced in the Video Synthesizer post, has a pitch detection method. This method returns the dominant pitch frequency detected by an audio input.

Using this pitch returned in Hertz, we can then use a musical framework (in this case, Music21), to convert our frequency into a musical note.

With this musical note, we can then measure how far away our frequency is from the ideal pitch in cents. We can set up a threshold for staying in key for our software, and provide visual feedback when our player isn't singing the proper note.

The code to convert from a frequency to a note in Music21 looks like this:

import music21

a4 = music21.pitch.Pitch() # create a Pitch object
a4.frequency = 440 # set the pitch in Hertz

print(a4.nameWithOctave) # prints 'A4'

As a bonus, when our frequency is off by a bit, we can get this directly from our new Pitch object in cents like this:

bad_note = music21.pitch.Pitch()
bad_note.frequency = 488
print(bad_note.microtone.cents) # prints -20.74, number of cents from closest note
print(bad_note.nameWithOctave)  # in this case, B4 is closest note

Discovering a User's Vocal Range

Each person has what's called their own "vocal range". A person's vocal range can be between three to at most currently, ten octaves.

In order to have our player able to sing comfortably, we'll need to detect their vocal range, and allow them to work within their own framework of what is possible for a low note, all the way up to a high note.

To do this, we'll need to have them sing the lowest note possible, while checking for a specific cent deviation, so we know that they can truly hold that note that they're trying to sing.

After we've detected the lowest note, we can then grab the highest note they can sing, again using the criteria of a set amount of cent deviation we'll allow.

Setting the Spaces on a Vocal Range

If we want to make our video game musical, it'll need to follow some sort of musical structure. The most basic way we can do that is by using what's called a scale.

In our case, we'll use the most basic scale there is, the C major scale. I say it's easy, because it's just exactly all the notes without any "#"s in our set of notes.

So we can then set our pipe spaces to be in the places where the major notes would all fall. And through this we'll have the basic start of a musical sound.

Music works via a set of intervals, or distances from the base note it's in. We can use this to create automatic music for our game.

Where to Begin on Larger Programs

So, with all this theory in place, how do you even begin?

With bigger, more complicated programs like this, the best thing to do is start from the smallest possible thing.

Forget all the other problems, and start with the one thing that you'll need first, before you can jump into the others.

In my case, it's figuring out a person's vocal range. So, let's do that first. We'll need to take in a person's voice from the microphone, and then we'll need them to sing a low note, hold it, and then a high note, and hold it. With this, we'll want two notes out, so that we can send them into our game, where our users will try and sing in even notes between the two.

Once we've got that, we'll then want a way to tell where a note is on a person's vocal range. We can then map the places on the screen to the notes a person is capable of singing. With this, we should have all we need to turn their voice into a controller.

Writing the Vocal Range Detector

Since the vocal range detector is the first thing we'll need, we should begin with it.

We'll want to open up the microphone, and then use this to pass into aubio's pitch detection routine. Once we've got that pitch in Hertz, we'll then use Music21's note function to convert it into a pitch.

Music21's pitch object then automatically sets the pitch's name to be the closest note, with it's nameWithOctave variable. If the pitch isn't perfect, it'll also have a set microtone.cents, with the amount of cents deviation from the perfect pitch.

We can use the microtone cents in order to detect if our singer isn't quite hitting the proper note, and determine if we should let them use this note in their vocal range.

We'll also need to set a duration for how long they should hold their note. This can make using the program very difficult, so choose wisely. In my case, I ended up with a note_hold of 20 samples in a row of being in the same key with a cent range set fairly loose.

Here's what the code looks like:

import aubio
import numpy as np
import pyaudio

import time
import argparse

import music21 # yes!

parser = argparse.ArgumentParser()
parser.add_argument("-input", required=False, type=int, help="Audio Input Device")
args = parser.parse_args()

if not args.input:
    print("No input device specified. Printing list of input devices now: ")
    p = pyaudio.PyAudio()
    for i in range(p.get_device_count()):
        print("Device number (%i): %s" % (i, p.get_device_info_by_index(i).get('name')))
    print("Run this program with -input 1, or the number of the input you'd like to use.")
    exit()

# PyAudio object.
p = pyaudio.PyAudio()

# Open stream.
stream = p.open(format=pyaudio.paFloat32,
                channels=1, rate=44100, input=True,
                input_device_index=args.input, frames_per_buffer=4096)
time.sleep(1)
# Aubio's pitch detection.
pDetection = aubio.pitch("default", 2048,
    2048//2, 44100)
# Set unit.
pDetection.set_unit("Hz")
pDetection.set_silence(-40)

def get_vocal_range(volume_thresh=0.01, cent_range=20, note_hold=20):

    note_curr = 0 # counter for how many consistent samples while recording
    range_low = "" # name of note we achieved at lowest range
    range_high = "" # name of note achieved at highest


    have_range = False

    previous_note = ""
    current_pitch = music21.pitch.Pitch()

    while not have_range:

        data = stream.read(1024, exception_on_overflow=False)
        samples = np.fromstring(data,
                                dtype=aubio.float_type)
        pitch = pDetection(samples)[0]

        # Compute the energy (volume) of the
        # current frame.
        volume = np.sum(samples**2)/len(samples) * 100

        # You can uncomment the volume below to make sure the threshold matches
        # your microphone's threshold
        #print(volume)

        if pitch and volume > volume_thresh: # adjust with your mic! .0002 if for my telecaster, .001 for my mic
            current_pitch.frequency = pitch
        else:
            continue

        if current_pitch.microtone.cents > cent_range:
            print("Note %s outside of Cent Range with %i" %
                  (current_pitch.nameWithOctave, current_pitch.microtone.cents))
            previous_note = ""
            continue

        current = current_pitch.nameWithOctave


        if current == previous_note:
            note_curr += 1
            if note_curr == note_hold:
                if range_low != "" and range_low != current:
                    range_high = current
                    have_range = True
                    print("got range of high")
                else:
                    range_low = current
                    print("got range of low")
        else:
            note_curr = 0
            note = current
            previous_note = current
            print(current)

    return range_low, range_high

min_note, max_note = get_vocal_range()
print("total range: %s to %s" % (min_note, max_note))

In the above code, there's one more thing I haven't mentioned yet, the volume threshold of your microphone.

In my case, my microphone has audio coming in at around the volume level of .001. You may need to adjust this to fit your microphone, by uncommenting out the print(volume) line, and watching for the change when you begin singing into your microphone.

What we're looking for here is for the volume threshold to be high enough that pitch detection is only tried when we're actually singing, and not when our microphone pics up background noise.

Calculating the Current Note's Position on Our Vocal Range

Now, there are a few different ways we can calculate our user's position on the vocal range.

For one, we could map the entire vocal range to our screen, directly. If our user sings a note, we take that note, and calculate the distance from the lowest note they can sing. We should then have a distance in cents.

Using this distance in cents, we can then divide it by the total interval in cents from the lowest to the highest note the person can sing. We'll then have a number from zero to one, with each note representing a decimal in between.

We can then directly map this number to the screen by multiplying by our screen's height resolution. Each note would then directly map to our screen.

But there's a trade off here, because this number doesn't really mean anything musically. It just represents a space, and in order to make it musical, we'll have to have our hazards represent the places on the screen of the specific notes we want our user to sing.

For now, let's use the current note's position as a number from zero to one:

# code continues from above

def position_on_range(low_note, high_note, volume_thresh=.0001, cent_range=5):
    lowNote = music21.note.Note(low_note)
    highNote = music21.note.Note(high_note)

    vocalInterval = music21.interval.notesToInterval(lowNote, highNote)

    current_pitch = music21.pitch.Pitch()

    while True:

        data = stream.read(1024, exception_on_overflow=False)
        samples = np.fromstring(data,
                                dtype=aubio.float_type)
        pitch = pDetection(samples)[0]

        # Compute the energy (volume) of the
        # current frame.
        volume = np.sum(samples**2)/len(samples)

        if pitch and volume > volume_thresh: # adjust with your mic! .0002 if for my telecaster, .001 for my mic
            current_pitch.frequency = pitch
        else:
            continue

        if current_pitch.microtone.cents > cent_range:
            #print("Outside of Cent Range with %i" % current_pitch.microtone.cents)
            previous_note = ""
            continue

        current = current_pitch.nameWithOctave

        cur_interval = music21.interval.notesToInterval(lowNote, current_pitch)
        print(cur_interval.cents / vocalInterval.cents)

position_on_range(min_note, max_note)

The code to calculate the current pitch's place is mostly the same as detecting our initial vocal range. The only thing new we've added is calculating the intervals using music21's notesToInterval. This gives us back an interval, and one of the units we can use here to measure is cents.

We don't actually put this number anywhere, and instead we print it out to the screen. Later, we'll put this number on a Queue, so that our voice detection can run on it's own thread, and our game can pop all the sung notes into places on the screen for our bird to be.

Separating Our Audio and Game Code

If our audio code is going to be running on it's own thread, we should really separate it out from our game code. We can do this easy enough, by creating a Queue object in the audio file, and then importing the Queue into our main game loop.

From here, we can also add the following code, to make it so our functions don't get run automatically when we're importing our get_vocal_range and position_on_range functions.

This can go at the end of our program, and turns our prototype program into a reusable library for when we want to work with a user's voice as input:

if __name__ == '__main__':
    low_note, high_note = get_vocal_range()
    position_on_range()

Finally, I import Queue, and create a Queue called q in the program. I then make sure to q.put our vocal range intervals instead of printing them.

We can then import the q from our main program, and get each of the values as they come in on their own thread.

Writing the Game Code

Since this post is mostly concerned with explaining how to get voice into our program as a type of input, the I won't focus on the specifics of the game code as much.

I started from the great FlapPyBird clone, and took the image assets.

From here, I then replaced the "flap" function calls with the loop checking if there's anything on the queue. If there is, we then put the bird in the position on the screen that matches the current position in the vocal range.

I haven't yet added a handler for when the player hits a pipe, or made it so the pipes make musical sense. For now, they just spawn randomly.

from voiceController import get_vocal_range, position_on_range, q, stream

from threading import Thread

import pygame
import random
from itertools import cycle

PIPEGAPSIZE = 100
screenWidth, screenHeight = 288, 512
screen = pygame.display.set_mode((screenWidth, screenHeight)) 

clock = pygame.time.Clock()

bird = ('images/redbird-downflap.png',
        'images/redbird-midflap.png',
        'images/redbird-upflap.png')

background = 'images/background-day.png'
pipe = 'images/pipe-green.png'

IMAGES = {}
HITMASKS = {}
IMAGES['background'] = pygame.image.load(background).convert()
IMAGES['player'] = (
    pygame.image.load(bird[0]).convert_alpha(),
    pygame.image.load(bird[1]).convert_alpha(),
    pygame.image.load(bird[2]).convert_alpha(),
)

IMAGES['pipe'] = (
    pygame.transform.flip(
        pygame.image.load(pipe).convert_alpha(), False, True),
    pygame.image.load(pipe)
)

IMAGES['base'] = pygame.image.load('images/base.png').convert_alpha()
BASEY = screenHeight * 0.89

def checkCrash(player, upperPipes, lowerPipes):
    """returns True if player collders with base or pipes."""
    pi = player['index']
    player['w'] = IMAGES['player'][0].get_width()
    player['h'] = IMAGES['player'][0].get_height()

    # if player crashes into ground
    if player['y'] + player['h'] >= BASEY - 1:
        return [True, True]
    else:

        playerRect = pygame.Rect(player['x'], player['y'],
                      player['w'], player['h'])
        pipeW = IMAGES['pipe'][0].get_width()
        pipeH = IMAGES['pipe'][0].get_height()

        for uPipe, lPipe in zip(upperPipes, lowerPipes):
            # upper and lower pipe rects
            uPipeRect = pygame.Rect(uPipe['x'], uPipe['y'], pipeW, pipeH)
            lPipeRect = pygame.Rect(lPipe['x'], lPipe['y'], pipeW, pipeH)

            # player and upper/lower pipe hitmasks
            pHitMask = HITMASKS['player'][pi]
            uHitmask = HITMASKS['pipe'][0]
            lHitmask = HITMASKS['pipe'][1]

            # if bird collided with upipe or lpipe
            uCollide = pixelCollision(playerRect, uPipeRect, pHitMask, uHitmask)
            lCollide = pixelCollision(playerRect, lPipeRect, pHitMask, lHitmask)

            if uCollide or lCollide:
                return [True, False]

    return [False, False]

def pixelCollision(rect1, rect2, hitmask1, hitmask2):
    """Checks if two objects collide and not just their rects"""
    rect = rect1.clip(rect2)

    if rect.width == 0 or rect.height == 0:
        return False

    x1, y1 = rect.x - rect1.x, rect.y - rect1.y
    x2, y2 = rect.x - rect2.x, rect.y - rect2.y

    for x in range(rect.width):
        for y in range(rect.height):
            if hitmask1[x1+x][y1+y] and hitmask2[x2+x][y2+y]:
                return True
    return False

def getHitmask(image):
    """returns a hitmask using an image's alpha."""
    mask = []
    for x in range(image.get_width()):
        mask.append([])
        for y in range(image.get_height()):
            mask[x].append(bool(image.get_at((x,y))[3]))
    return mask

def getRandomPipe():
    """returns a randomly generated pipe"""
    # y of gap between upper and lower pipe
    gapY = random.randrange(0, int(BASEY * 0.6 - PIPEGAPSIZE))
    gapY += int(BASEY * 0.2)
    pipeHeight = IMAGES['pipe'][0].get_height()
    pipeX = screenWidth + 10

    return [
        {'x': pipeX, 'y': gapY - pipeHeight},  # upper pipe
        {'x': pipeX, 'y': gapY + PIPEGAPSIZE}, # lower pipe
    ]

# hismask for pipes
HITMASKS['pipe'] = (
    getHitmask(IMAGES['pipe'][0]),
    getHitmask(IMAGES['pipe'][1]),
)

# hitmask for player
HITMASKS['player'] = (
    getHitmask(IMAGES['player'][0]),
    getHitmask(IMAGES['player'][1]),
    getHitmask(IMAGES['player'][2]),
)

def draw_pygame():
    running = True
    playerIndex = 0
    playerIndexGen = cycle([0, 1, 2, 1])
    # iterator used to change playerIndex after every 5th iteration
    loopIter = 0

    basex = 0
    # amount by which base can maximum shift to left
    baseShift = IMAGES['base'].get_width() - IMAGES['background'].get_width()

    playerX = int(screenWidth * .2)
    playerY = screenHeight // 2

    basex = 0
    baseShift = IMAGES['base'].get_width() - IMAGES['background'].get_width()

    # get 2 new pipes to add to upperPipes lowerPipes list
    newPipe1 = getRandomPipe()
    newPipe2 = getRandomPipe()

    # list of upper pipes
    upperPipes = [
        {'x': screenWidth + 200, 'y': newPipe1[0]['y']},
        {'x': screenWidth + 200 + (screenWidth / 2), 'y': newPipe2[0]['y']},
    ]

    # list of lowerpipe
    lowerPipes = [
        {'x': screenWidth + 200, 'y': newPipe1[1]['y']},
        {'x': screenWidth + 200 + (screenWidth / 2), 'y': newPipe2[1]['y']},
    ]

    pipeVelX = -2


    while running:
        key = pygame.key.get_pressed()
        if key[pygame.K_q]:
            running = False
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                running = False

        screen.fill((0,0,0))

        if not q.empty():
            b = q.get()
            if b > 0 and b < 1:
               playerY = screenHeight - int(screenHeight * b) 
        else:
            playerY = playerY + 2

        crashTest = checkCrash({'x': playerX, 'y': playerY, 'index': playerIndex},
                               upperPipes, lowerPipes)

        for pipe in upperPipes:
            pipeMidPos = pipe['x'] + IMAGES['pipe'][0].get_width() / 2

        # move pipes to left
        for uPipe, lPipe in zip(upperPipes, lowerPipes):
            uPipe['x'] += pipeVelX
            lPipe['x'] += pipeVelX

        # add new pipe when first pipe is about to touch left of screen
        if 0 < upperPipes[0]['x'] < 4:
            newPipe = getRandomPipe()
            upperPipes.append(newPipe[0])
            lowerPipes.append(newPipe[1])

        # remove first pipe if its out of the screen
        if upperPipes[0]['x'] < -IMAGES['pipe'][0].get_width():
            upperPipes.pop(0)
            lowerPipes.pop(0)

        screen.blit(IMAGES['background'], (0,0))

        for uPipe, lPipe in zip(upperPipes, lowerPipes):
            screen.blit(IMAGES['pipe'][0], (uPipe['x'], uPipe['y']))
            screen.blit(IMAGES['pipe'][1], (lPipe['x'], lPipe['y']))

        if (loopIter + 1) % 5 == 0:
            playerIndex = next(playerIndexGen)
        loopIter = (loopIter + 1) % 30
        basex = -((-basex + 4) % baseShift)

        screen.blit(IMAGES['base'], (basex, BASEY))
        screen.blit(IMAGES['player'][playerIndex],
                    (playerX, playerY))

        pygame.display.flip()
        clock.tick(60)

min_note, max_note = get_vocal_range()
t = Thread(target=position_on_range, args=(min_note, max_note))
t.daemon = True
t.start()

draw_pygame()
stream.stop_stream()
stream.close()
pygame.display.quit()

You'll see at the very end of the program, how we use the imported get_vocal_range function to first get our vocal range.

Because we start Pygame in our main program, we'll see an empty blank window while the console spits our our controller.

In the next post, we'll address this, and give our user a visual feedback while they're trying to establish their vocal range.

Once this is done, we then begin our main Flappy Bird loop, tracking the voice locations and updating the bird to match. If the player stops hitting the note, the bird starts to fall.

With this, we have our very first, voice controlled Flappy Bird prototype. It's not nearly ready to be a production game, but it gives us the basic start for an interactive singing game.

What We'll Do In the Next Post

In the next post, we'll further refine our version of Flappy Bird, and turn it into a musical game. We'll help our player out by giving them reference notes to sing to, and give them some visual feedback when we're establishing our vocal range.

We'll make Singy-Bird a more musical game in general, with each of the pipes representing a place that makes musical sense, building a melody.

Finally, we'll bundle our program into a program we can share with other people, using the py2exe extension.

Where to Go From Here

This post is a work in progress, but I wanted to show how a larger program comes together, outside of a smaller, single Python file application.

You can see all the code in progress now at Github. The program should run on your computer, but be mindful that you'll need Music21, Aubio, Pygame, PyAudio, and NumPy in order to make everything work.

If you're interested in more projects like these, please sign up for my newsletter or create an account here on Make Art with Python. You'll get the first three chapters of my new book free when you do, and you'll help me continue making projects like these.

Building a Video Synthesizer in Python

KP Kaiser — Fri, 24 Nov 2017 14:36:26 +0000

Video synthesizers are devices that create visual signal from an audio input. They've got a long history, since at least the 60's.

Lee Harrison started building a video synthesizer called Scanimate that worked from voice input to animate a face in the 1960s.

More recently, Critter and Guitari have built a bunch of insane video synthesizers that react to live music performance. They're meant to enhance and automatically accompany a performance.

Here's one of their video syntheseizers, called the Rhythm Scope, in action:

Today, we'll write a basic video synthesizer in Python, using aubio for Onset detection, and Pygame to display our graphics visually. We'll end up with a program ready to be played out through a projector.

Since I don't have a drum set (yet!), I'll use my guitar to drive onsets and draw on the screen.

Before We Begin

You'll need to have a way to get audio into your program.

For my video, I used a cheap Rocksmith USB cable. But if you've got an audio interface for your computer, feel free to use that instead, hooked up to your instrument of choice.

If you don't have a guitar to use as an input, you can also just use your computer's mic, and either make loud noises, or play drums, like you see in the Critter and Guitari video above. We'll be using Onset detection to draw our circles, so it won't really matter where the sound comes from, as long as it's loud enough to trigger an onset.

(And if you don't have a guitar, why the hell not?! You can get one of these along with this, and be playing in a few days.)

Architecture of A Video Synthesizer in Python

It's not as complicated as it looks!

With visual effects accompanying a performance, we'll want the least amount of latency as possible. We don't want to be waiting for each set of audio frames to come in before we begin drawing to the screen.

Because of this, we use threads in this program, one to take in our audio and detect our onsets, and another to do the drawing to the screen of our circles.

We use the PyAudio and aubio libraries, to first read in audio input from our computer, and then analyze a set of frames for an onset. Once we've detected an onset, we then push a new value on to a queue, for the other thread to read.

In our display thread, we loop infinitely, first checking if there's a new onset detected on the queue, and then creating a new circle if there is. We then shrink and draw our circles to the screen.

With this basic flow, we've got the start of a video synthesizer, ready to react in real time to the audio coming in from our computer.

Writing the Code

The first thing we'll want to do is allow our users to select the proper input device to begin reading audio from. On my personal computer, I have 7 different audio inputs to choose from.

So it's important that after we import our libraries, we open and describe what the available inputs are before proceeding:

import pyaudio
import sys
import numpy as np
import aubio

import pygame
import random

from threading import Thread

import queue
import time

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("-input", required=False, type=int, help="Audio Input Device")
parser.add_argument("-f", action="store_true", help="Run in Fullscreen Mode")
args = parser.parse_args()

if not args.input:
    print("No input device specified. Printing list of input devices now: ")
    p = pyaudio.PyAudio()
    for i in range(p.get_device_count()):
        print("Device number (%i): %s" % (i, p.get_device_info_by_index(i).get('name')))
    print("Run this program with -input 1, or the number of the input you'd like to use.")
    exit()

With this argparse setup, our users get a list of the audio inputs, and are given the option to specify one from the command line. So your first time running it might look like this:

$ python3 video-synthesizer.py 
Device number (0): HDA Intel PCH: ALC1150 Analog (hw:0,0)
Device number (1): HDA Intel PCH: ALC1150 Digital (hw:0,1)
Device number (2): HDA Intel PCH: ALC1150 Alt Analog (hw:0,2)
Device number (3): HDA NVidia: HDMI 0 (hw:1,3)
Device number (4): HDA NVidia: HDMI 1 (hw:1,7)
Device number (5): HDA NVidia: HDMI 2 (hw:1,8)
Device number (6): HDA NVidia: HDMI 3 (hw:1,9)
Device number (7): HD Pro Webcam C920: USB Audio (hw:2,0)
Device number (8): Rocksmith USB 

Run this program with -input 1, or the number of the input you'd like to use.

$ python3 video-synthesizer.py -input 8

With this, we can start up our program, using the Rocksmith USB as our input.

Next, we can initialize Pygame. If we plan on using a projector for our real time performance, we should run Pygame in fullscreen mode. Otherwise, we should run in a windowed mode for debug and development purposes.

You'll see I added a flag to our argparse parser above, if we pass in -f from the command line we'll run fullscreen, otherwise windowed. The code to handle this flag looks like this:

pygame.init()

if args.f:
    # run in fullscreen
    screenWidth, screenHeight = 1024, 768
    screen = pygame.display.set_mode((screenWidth, screenHeight), pygame.FULLSCREEN | pygame.HWSURFACE | pygame.DOUBLEBUF)

else:
    # run in window
    screenWidth, screenHeight = 800, 800
    screen = pygame.display.set_mode((screenWidth, screenHeight))

Next, we define our Circle object, and the set of colors we'll use to draw our circles with, along with a list to store all circles in.

white = (255, 255, 255)
black = (0, 0, 0)

class Circle(object):
    def __init__(self, x, y, color, size):
        self.x = x
        self.y = y
        self.color = color
        self.size = size

    def shrink(self):
        self.size -= 3

colors = [(229, 244, 227), (93, 169, 233), (0, 63, 145), (255, 255, 255), (109, 50, 109)]
circleList = []

You'll notice that the Circle object has one method on itself, where it can shrink. We shrink the circle by 3 pixels on every loop.

Later, we'll pop circles off our main list if they get below one in size.

Next, we can add in our audio setup, initializing PyAudio and our aubio Onset detector:

# initialise pyaudio
p = pyaudio.PyAudio()

clock = pygame.time.Clock()

# open stream

buffer_size = 4096 # needed to change this to get undistorted audio
pyaudio_format = pyaudio.paFloat32
n_channels = 1
samplerate = 44100
stream = p.open(format=pyaudio_format,
                channels=n_channels,
                rate=samplerate,
                input=True,
                input_device_index=args.input,
                frames_per_buffer=buffer_size)

time.sleep(1)

# setup onset detector
tolerance = 0.8
win_s = 4096 # fft size
hop_s = buffer_size // 2 # hop size
onset = aubio.onset("default", win_s, hop_s, samplerate)

q = queue.Queue()

These values are mostly guesses, and I've used aubio's default onset detector. You might want to play around with the window, hop size, and tolerance to get a better fit for your data and your audio.

One thing to note about the PyAudio stream we open up. I had to set my buffer size to be a fairly large 4096 in Linux in order to get good audio quality. If it was a number below that, I'd get a bit of static coming out of my input.

Finally, we create our queue that we'll use to pass messages from the audio thread back to the main, Pygame thread.

Drawing Our Onsets

We use a very basic example to begin drawing from our onsets. We create a main Pygame drawing loop that clears the screen, and then iterates over each of the remaining circles in our list.

We shrink each circle on every loop, and also check for any new onsets in the queue. If there are any, we add it to the list.

def draw_pygame():
    running = True
    while running:
        key = pygame.key.get_pressed()

        if key[pygame.K_q]:
            running = False
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                running = False

        if not q.empty():
            b = q.get()
            newCircle = Circle(random.randint(0, screenWidth), random.randint(0, screenHeight),
                               random.choice(colors), 700)
            circleList.append(newCircle)

        screen.fill(black)
        for place, circle in enumerate(circleList):
            if circle.size < 1:
                circleList.pop(place)
            else:
                pygame.draw.circle(screen, circle.color, (circle.x, circle.y), circle.size)
            circle.shrink()

        pygame.display.flip()
        clock.tick(90)

Creating Our Audio Thread

Our audio and onset detection thread is pretty straightforward once we've got the infrastructure set up for our drawing.

We merely loop over reading the audio buffer, and running it through our onset detector. If there's a new onset detected in the current buffer, we create a new object in the queue, and wait for the main drawing thread to pick it up.

This can happen in it's own thread, regardless of the current state of the drawing part of the program.

def get_onsets():
    while True:
        try:
            buffer_size = 2048 # needed to change this to get undistorted audio
            audiobuffer = stream.read(buffer_size, exception_on_overflow=False)
            signal = np.fromstring(audiobuffer, dtype=np.float32)


            if onset(signal):
                q.put(True)

        except KeyboardInterrupt:
            print("*** Ctrl+C pressed, exiting")
            break


t = Thread(target=get_onsets, args=())
t.daemon = True
t.start()

draw_pygame()
stream.stop_stream()
stream.close()
pygame.display.quit()

With this, our code is ready to go, and creates the effect we can see in the YouTube video.

Next Steps

aubio has pitch detection, and that's the most obvious next thing to program. Making some visual feedback based upon which note is being played on your guitar.

But besides that, we can also make use of the FFT to break down the signal we're getting from the guitar itself, to make a more interactive visual. Onsets are really just the tip of the iceberg.

Critter and Guitari have done a really good job here too, because they've got a new product, called the ETC, that allows for these sorts of things, and even better still, is programmable in Python and Pygame!

Judging from their demo videos, they've really nailed the video synthesizer in this little box.

What can you make? I'd love to see what you build and how you play! Reach out to me on Twitter or Instagram with any of your creations.

As always, the code for this post is available at Github.

If you're interested in more projects like these, please sign up for my newsletter or create an account on Make Art with Python. You'll get the first three chapters of my new book free when you do, and you'll help me continue making projects like these.

DEAL WITH IT in Python with Face Detection

KP Kaiser — Fri, 17 Nov 2017 16:04:27 +0000

DEAL WITH IT is a meme where glasses fly in from off the screen, and on to a user's face. The best instances of this meme do so in a unique way.

Today we'll write an automatic meme generator, using any static image with faces as our input. This code makes a great starting point for a meme API, or building your own animated version using video input.

I got the idea for this post from Erik Taheri, and his in browser Javscript version of this effect.

At the end of the article as a bonus, I've also included a version where the effect can be done in real time from a webcam, using OpenCV.

The Tools of Face Detection and Gif Creation

We'll use Dlib's get_frontal_face_detector, along with the 68 point shape prediction model we used in the Snapchat Lens article.

Our program will take in a command line argument, the input image. It will then use the face detection algorithm in Dlib to see if there are any faces. If there are, it will create an end position for each face, where the glasses should end up.

We'll need to then scale and rotate our glasses to fit each person's face. We'll use the set of points returned from Dlib's 68 point model to find a center of mass for the eyes, and a rotation for the space between them.

After we've found a final position and rotation for the glasses, we can then animate a gif, with the glasses coming in from the top of the screen. We'll draw this using MoviePy and a make_frame function.

Architecture of Automatic Gifs

The architecture of the application is fairly straightforward. We first take in an image, and then convert it to a grayscale NumPy array. Once we've got that, we can then pass our detected faces into our face orientation prediction model.

With the returned orientation of the face, we can then select out the eyes, and scale and rotate our glasses frames to fit the person's face.

We can accumulate a set of faces and their final positions, appending them to a list.

Finally, with this list we can then create a draw routine using MoviePy that then generates our animated gif.

Writing the Code

With our code architecture planned out, the next thing we need to do is build our code step by step.

We'll first import all our tools, and take in an image from the command line:

import dlib
from PIL import Image
import argparse

from imutils import face_utils
import numpy as np

import moviepy.editor as mpy

parser = argparse.ArgumentParser()
parser.add_argument("-image", required=True, help="path to input image")
args = parser.parse_args()

With this in place, we can then resize our images to fit a smaller width, so our gifs don't turn out massive, and import our face detector and shape predictors.

We can also open the glasses and text we'll be pasting on to our image.

At this point, we should also detect if there are even any faces detected in the image. If not, we should exit immediately.

detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor('shape_predictor_68.dat')

# resize to a max_width to keep gif size small
max_width = 500

# open our image, convert to rgba
img = Image.open(args.image).convert('RGBA')

# two images we'll need, glasses and deal with it text
deal = Image.open("deals.png")
text = Image.open('text.png')

if img.size[0] > max_width:
    scaled_height = int(max_width * img.size[1] / img.size[0])
    img.thumbnail((max_width, scaled_height))

img_gray = np.array(img.convert('L')) # need grayscale for dlib face detection

rects = detector(img_gray, 0)

if len(rects) == 0:
    print("No faces found, exiting.")
    exit()

print("%i faces found in source image. processing into gif now." % len(rects))

Great! Now we can loop over each of the faces detected, and build up a list of scaled and rotated glasses, along with their final positions.

faces = []

for rect in rects:
    face = {}
    print(rect.top(), rect.right(), rect.bottom(), rect.left())
    shades_width = rect.right() - rect.left()

    # predictor used to detect orientation in place where current face is
    shape = predictor(img_gray, rect)
    shape = face_utils.shape_to_np(shape)

    # grab the outlines of each eye from the input image
    leftEye = shape[36:42]
    rightEye = shape[42:48]

    # compute the center of mass for each eye
    leftEyeCenter = leftEye.mean(axis=0).astype("int")
    rightEyeCenter = rightEye.mean(axis=0).astype("int")

    # compute the angle between the eye centroids
    dY = leftEyeCenter[1] - rightEyeCenter[1] 
    dX = leftEyeCenter[0] - rightEyeCenter[0]
    angle = np.rad2deg(np.arctan2(dY, dX)) 

    # resize glasses to fit face width
    current_deal = deal.resize((shades_width, int(shades_width * deal.size[1] / deal.size[0])),
                               resample=Image.LANCZOS)
    # rotate and flip to fit eye centers
    current_deal = current_deal.rotate(angle, expand=True)
    current_deal = current_deal.transpose(Image.FLIP_TOP_BOTTOM)

    # add the scaled image to a list, shift the final position to the
    # left of the leftmost eye
    face['glasses_image'] = current_deal
    left_eye_x = leftEye[0,0] - shades_width // 4
    left_eye_y = leftEye[0,1] - shades_width // 6
    face['final_pos'] = (left_eye_x, left_eye_y)
    faces.append(face)

With our final positions in place along with our scaled and rotated glasses, we can then put together our movie. We'll set a duration for the whole gif, along with a time to stop the glasses coming down, so we can put the deal with it text on the screen.

# how long our gif should be
duration = 4

def make_frame(t):
    draw_img = img.convert('RGBA') # returns copy of original image

    if t == 0: # no glasses first image
        return np.asarray(draw_img)

    for face in faces: 
        if t <= duration - 2: # leave 2 seconds for text
            current_x = int(face['final_pos'][0]) # start from proper x
            current_y = int(face['final_pos'][1] * t / (duration - 2)) # move to position w/ 2 secs to spare
            draw_img.paste(face['glasses_image'], (current_x, current_y) , face['glasses_image'])
        else: # draw the text for last 2 seconds
            draw_img.paste(face['glasses_image'], face['final_pos'], face['glasses_image'])
            draw_img.paste(text, (75, draw_img.height // 2 - 32), text)

    return np.asarray(draw_img)

You'll notice I used an image for the text overlay, and not Pillow's built in Text drawing functions. I did this because Pillow doesn't have a built in Stroke function for text. Without the stroke, the text becomes illegible on brighter images.

Finally, we need to create a VideoClip object in MoviePy, and pass in our animation generating frame along with a fps.

animation = mpy.VideoClip(make_frame, duration=duration)
animation.write_gif("deal.gif", fps=4)

With this, we're done!

Animating Memes In Real Time

Now that we've got a base set for generating our gifs, it isn't too difficult to adapt our code to work with the webcam in real time.

Instead of loading our source image from the command line, we can use OpenCV as our source images, and keep track of the animations with a counter. The new code to do this is pretty straightforward, the real meat being:

        # I got lazy, didn't want to bother with transparent pngs in opencv
        # this is probably slower than it should be
        if dealing:
            if current_animation < glasses_on:
                current_y = int(current_animation / glasses_on * left_eye_y)
                img.paste(current_deal, (left_eye_x, current_y), current_deal)
            else:
                img.paste(current_deal, (left_eye_x, left_eye_y), current_deal)
                img.paste(text, (75, img.height // 2 - 32), text)

    if dealing:
        current_animation += 1
        if current_animation > animation_length:
            dealing = False
            current_animation = 0
        else:
            frame = cv2.cvtColor(np.asarray(img), cv2.COLOR_RGB2BGR)
   cv2.imshow("deal generator", frame)
   key = cv2.waitKey(1) & 0xFF
    if key == ord("q"):
        break

    if key == ord("d"):
        dealing = not dealing

This creates a counter, and steps through how many frames have progressed to keep track of time. Using this, we can then animate our glasses to their final position. We see if the user presses the d key, to deal with it, and when they do, we begin our animation.

You can read the rest of the code for this at Github.

Where To Go From Here

We've successfully built the very first part in a program that could be used as an API to generate memes automatically.

By hooking up our program to something like Flask, we could display a web page allowing for users to upload their own images, and get back fully complete memes.

With something like youtube-dl, we could have users paste in YouTube url videos to automatically generate memes.

In case you missed it, the code is available at Github.

If you're interested in more projects like these, please sign up an account on Make Art with Python. You'll get the first three chapters of my new book free when you do, and you'll help me continue making projects like these.

A Universe in One Line of Code with 10 PRINT

KP Kaiser — Wed, 08 Nov 2017 20:03:47 +0000

10 PRINT in Python 3 with Pygame

10 PRINT is a new book which explores the magical power of a single line of code.

That single line of code ran on the Commodore 64, in 1982. It gave us one of the earliest examples of the power of computers to generate entire worlds.

With just a single incantation:

~~~~basic
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
~~~~

Our intrepid user got to watch an entire universe unfold in front of them.

This magic of new possibilities is a great example of what makes programming so exciting. With just one line of code, we can see an entire world, randomly building itself. We see paths and imagine new places with our eyes.

Understanding How 10 PRINT Builds a Universe

10 PRINT was written in the Basic programming language. This means it has a few quirks, looking back at it from our modern languages.

For one, Basic was just a raw terminal you typed your programs into. With each entered, you were writing a specific set of commands to be followed linearly.

First we have the line number, which is used at the end to repeat the line of code infinitely. Most modern programming languages don't have this idea of GOTO built in, because we've tried to abstract away from line numbers, and into more definite modules and functions with names.

Next, we use the CHR$ function, and pass it the number 205.5 + a random value between 0 and 1. All numbers were floating point values on the platform, and rounded down. So this meant our code would generate either the character 205 (\), or 206 (/).

With this, we loop infinitely, generating either a forward slash, or a back slash.

Rewriting 10 PRINT in Python 3 and Pygame

Unfortunately, writing 10 PRINT in Python and Pygame isn't really doable in one line.

The code to generate the lines is easy enough, but setting up our window for drawing takes up the majority of our code.

~~~~python
import pygame

import random

set our screen's width and height

screenWidth, screenHeight = (800, 800)

pygame.init()

pygame.display.set_caption("10 PRINT in Pygame")
screen = pygame.display.set_mode((screenWidth, screenHeight))

running = True

white = (255, 255, 255)
black = (0, 0, 0)

size of square, in pixels

square = 20

def drawScreen():
screen.fill(black)
# Python version of 10 PRINT happens here
for x in range(0, screenWidth, square):
for y in range(0, screenHeight, square):
if random.random() > 0.5:
pygame.draw.line(screen, white, (x, y), (x + square, y + square))
else:
pygame.draw.line(screen, white, (x, y + square), (x + square, y))

while running:
key = pygame.key.get_pressed()
# only draw if user presses spacebar
if key[pygame.K_SPACE]:
drawScreen()
# quit if user presses q
if key[pygame.K_q]:
exit()
for event in pygame.event.get():
if event.type == pygame.QUIT:
running = False
# this updates the display
pygame.display.flip()
~~~~

The real work of drawing 10 PRINT in our program comes from the following lines:

~~~~python
for x in range(0, screenWidth, square):
for y in range(0, screenHeight, square):
if random.random() > 0.5:
pygame.draw.line(screen, white, (x, y), (x + square, y + square))
else:
pygame.draw.line(screen, white, (x, y + square), (x + square, y))
~~~~

We use the size of the square in an x and y coordinate across the screen using range. If the square size is set to 20, x and y will be 0,20,40,60,80... all the way through to our screenHeight and screenWidth.

Next, we use the random.random() function to generate a number between 0 and 1. If that number is greater than 0.5, we draw a diagonal line of our square size, from the bottom to the top right (/).

If the random.random() number is less than 0.5, we make a diagonal line going from the top left of our square to the bottom right ().

With this, we've got our own implementation of the maze effect in Python 3, but none of the scrolling forever.

Coding Challenge: Animating 10 PRINT

If we want to animate our 10 PRINT, we'll need to start with a static set of line points to manipulate, instead of creating new ones every time the spacebar is pressed.

Something like this might work:

~~~~python
for x in range(0, screenWidth, square):
for y in range(0, screenHeight, square):
if random.random() > 0.5:

lines.append([x, y, x + square, y + square])

else:
lines.append([x, y + square, x + square, y])
~~~~

With this, we've got a set of line points we can access and manipulate using our loop.

Try making your own changes to 10 PRINT to make a new sort of animation. If you get stuck, you can check out the code used to make the above gif on Github.

Where to Go From Here

The above code gives you a starting point for manipulating your own interpretation of 10 PRINT. If you get stuck, feel free to look at the Github repository to figure out how to animate it yourself.

Be warned that the code currently writes an image to disk for every step, so you'll want to comment out that line if you don't want to save your images.

Feel free to share your images on Twitter or Instagram with the hashtag #10PRINT. I'd love to see what else you can make.

For other references to 10 PRINT, check out the YouTube channel The Coding Train's interpretation of 10 PRINT done in Processing.js.

Finally, if you enjoyed this post and would like to see more like it, sign up for my newsletter, or create an account at Make Art with Python to get access to the first three chapters of my book free.

Introduction to the Terminal

KP Kaiser — Mon, 06 Nov 2017 17:53:51 +0000

Introduction to the Terminal for New Programmers

Graphical user interfaces are all around us. They've taken over the way we interact with our phones and our computers.

But not every computer has a monitor for interaction. In fact, the majority of computers on the planet don't have a monitor.

Instead, they're accessed using something called "the terminal"". In this way, millions of computers are manipulated and told what to do, without needing a physical monitor attached to them.

What Is the Terminal?

The terminal is a text based way to interact with the files and programs on your computer.

A terminal allows you full access to all the inner workings of your computer, through text based input.

Using a program called ssh from within the terminal, we can change the computer we're working on to be a remote computer anywhere else in the world. The terminal then behaves with the same commands as if we were using our local computer.

The terminal is installed on almost every major operating system now, although today we'll be focusing on the one installed in MacOS / Linux.

Why Should I Bother?

The ability to work effectively with the terminal adds a key skill for the maturing developer. It gives you the ability to work with files in bulk, and to script and control a computer remotely.

If you're interested in server or web application development, having a decent understanding of the command line will give you the ability to deploy and manage your servers remotely. This also gives you an upper hand when it comes time to debug why systems stopped working.

Accessing the Terminal

On MacOS, the terminal can be opened by pressing CMD+SPACE together, and then typing out terminal. The MacOS launcher will open a terminal, which gives you a blank empty window, ready to receive text.

From here, you can start typing commands, and have them execute in a purely text based environment.

Navigating In the Terminal

The first command you'll want to enter from within your terminal is ls. That's the letter 'L', in lower case, followed by 'S', also in lower case.

Execute the ls command by pressing ENTER.

This lists the files in the current directory.

In MacOS, you can type open into the terminal, and you'll get a finder, showing exactly the files in your current directory.

If we use the command cd, we can change into other directories. We type in cd, and then the start of the name of the directory we'd like to switch into. If we press the TAB key, the terminal will try to finish the name of the directory we've started typing.

There are a few special directory names in the terminal. One of these is ., which means the current directory. This is a way of specifying the place where you are currently.

The other special directory name is .. (two periods). It means the directory above me.

Directory names can be chained together, using the / to link them. So if we wanted to go up two directories, we'd type in '../../'. And typing in cd ../../ would bring us up two directories from where we currently are.

With this, you can see that the text based environment of the terminal allows you to move through the files and directories of your host computer.

Creating Directories

Directories can be created with the memorable mkdir command.

To use it, just type out mkdir directoryname. This will create a new directory, inside of the current location.

Just like before, we can use the / and .. separators to specify different places to create our directory. For example, to create a new directory in the one above our current directory, we'd enter mkdir ../testname.

Creating Files

Files can be created in a few different ways. Our programs we execute can create files for us, or we can use some built in utilities.
jw
One of these is called touch. Touch allows us to create new files on the computer, which are completely empty.

The way we use it is just by passing touch a filename. It works just like cd and mkdir, in that you can pass a file path in the current directory or in another.

You can type out your own filename extension for the files you create with touch. So, to create an empty Python file in the current directory, you'd only need to type out touch myProgram.py, and myProgram.py would be created in your current directory.

Editing Text Files

Editing text files is easy enough, using the nano command.

When nano is run, it takes over the screen for the terminal. It then allows you to type directly into the file you've created in plain text. From here, you can also open other text files, and save them.

To open our myProgram.py file we created before, we'd need to type in nano myProgram.py. We could type out just myP, and then press TAB, and the terminal would finish typing the rest of the name of our file for us.

Once we've got our editor open, try typing out the following Python 3 program:

print("hello!")

The way you save files in nano is by pressing CTRL+o. You can exit nano, and go back to the terminal by pressing CTRL+x.

You can then run your new Python program with python3 myProgram.py.

Nano is nice, in that it has a little bar at the bottom of the screen to let you know how to use it. That helper can be useful when you're just beginning your work in the terminal, and don't yet have all the shortcuts memorized.

Executing Files

Executing files is easy enough.

We just type out the name of the program we want to run, and hit enter.

If we want to find out where the programs we're running exist, we use the command which.

Which spits back out the directory where the program is being run from. Find out where your computer's Python3 interpreter is by pressing which python3.

In my case, it's /usr/bin/python3. If I open up myProgram.py, and change it to be the following:

#!/usr/bin/python3
print("hello!")

I now have a program that the terminal can run on it's own. It will execute /usr/bin/python3 myProgram.py, when I send it this filename.

But in order to let the terminal know that it can run this as a program, I'll need to do one more thing, and tell it that this file is now an executable file.

The way we do that is with the chmod command. We type in chmod +x myProgram.py, and the chmod program turns it into an executable file on the system.

From here, we can run our program with a ./myProgram.py.

Piping Output

With our new command, we're sending text to the terminal.

But one of the main features of the terminal is the ability to pass the output of programs into each other.

Let's try passing our program's output to the less program, to see how output passing works:

$ ./myProgram.py | less

The | or pipe operator is made by holding down SHIFT and pressing the \ key on the US keyboard. When we run the previous command, it takes over the screen, and we can see the results of our less command.

To exit less, just press q.

We can also write our program's output to a file. Try using the > to create a new file like this:

$ ./myProgram.py > newFile.txt

You can then see the results of our program are written out to our new text file.

Where to Go From Here

If you're interested in learning software development, or you know somebody who is, I've written a book called Make Art with Python, and it will be available for purchase soon. It teaches the fundamentals of programming through art, instead of bland print statements.

For now, you can sign up as a user, and get access to the first three chapters free, along with a video walk through for each chapter.