Speech to Musical Notation with AssemblyAI [Updated with code, screenshots and video demonstration]

This is a submission for the AssemblyAI Challenge : Sophisticated Speech-to-Text.

What I Built

I built Speech-to-Note, an innovative web application that combines speech recognition and musical note detection. The application allows users to record audio (either speech or singing) and processes it in two ways:

1. Converts spoken words into text using AssemblyAI's Speech-to-Text API
2. Analyzes the audio to detect musical notes, including their pitch, octave, and duration

The application features a modern, responsive UI built with React and TailwindCSS, and a robust backend powered by FastAPI. It's particularly useful for musicians, music teachers, and anyone interested in analyzing the musical properties of their voice or instruments.

Demo

Link to site https://speech.vicentereyes.org/
GitHub:

reyesvicente / speech-to-music-note

Speech to Musical Notes Converter

This application converts spoken words into musical notes using FastAPI, React, and AssemblyAI.

Prerequisites

• Python 3.8+
• Node.js and npm
• AssemblyAI API key

Setup

1. Clone the repository

2. Set up the backend:

   # Install Python dependencies
   pip install -r requirements.txt
   
   # Set up your AssemblyAI API key in .env file
   # Replace 'your_api_key_here' with your actual API key

3. Set up the frontend:

   cd frontend
   npm install

Running the Application

1. Start the backend server:

   uvicorn main:app --reload

2. Start the frontend development server:

   cd frontend
   npm run dev

3. Open your browser and navigate to the URL shown in the frontend terminal output (usually http://localhost:5173)

Usage

1. Click the "Start Recording" button to begin recording audio
2. Speak into your microphone
3. Click "Stop Recording" when finished
4. Click "Process Audio" to send the recording to the server
5. The transcribed text will appear below

Features

• Audio recording using the Web Audio API
• Real-time…

View on GitHub

Speech to note demo

Vidyard video

share.vidyard.com

Landing Page

Audio Processing

Result

Journey

AssemblyAI's Universal-2 Speech-to-Text Model was integrated into the application through their Python SDK. The implementation can be found in the upload_audio endpoint of our FastAPI backend:

1. When a user records audio, it's sent to our backend as a WAV file
2. The audio file is processed in parallel:
   • Sent to AssemblyAI's API for transcription
   • Analyzed locally using librosa for musical note detection
3. The transcribed text and detected musical notes are returned to the frontend

The AssemblyAI integration was straightforward thanks to their well-documented SDK:

transcriber = aai.Transcriber()
transcript = transcriber.transcribe(audio_file_path)
transcribed_text = transcript.text

What makes this implementation sophisticated is the dual-processing approach:

1. Using AssemblyAI's advanced speech recognition for accurate text transcription
2. Complementing it with custom pitch detection algorithms to extract musical information
3. Providing a synchronized playback experience where users can hear the detected notes while seeing the transcribed text

This creates a unique tool that bridges the gap between spoken word and musical notation, making it valuable for various musical applications, from education to composition.

The application qualifies for additional prompts as it implements:

• Real-time audio processing
• Custom pitch detection algorithms
• Interactive audio playback
• Modern, responsive UI with TailwindCSS
• Full-stack implementation with React and FastAPI

The project demonstrates how AssemblyAI's technology can be combined with custom audio processing to create innovative applications that go beyond simple speech-to-text conversion.

Comments

[Jess Lee]

This was a very very cool project.

[Vicente G. Reyes]

Thanks, Jess!