DEV Community

Rounak Sen
Rounak Sen

Posted on

SonicScan - A Music Fingerprinting and Identification App

#rust #music #programming

Want to try this app download it from here

Features

  • 🪙 ABSOLUTELY FREE
  • 👆🏿 Song Fingerprinting
  • 🔍 Song Search
  • 📥 Song Download
  • 🎸 Song Identification
  • 🌐 Multi Device Support (Windows, Linux, Mac, Web, Android, iOS)
  • 🎨 Multi Themed (powered by daisyui)

Note: My implementation of fingerprinting is based on this old blogpost
How_does_Shazam_work_Coding_Geek
from 2015 so it maynot be very good at identifying songs.

Note: Due to spotify not allowing to download songs, and Youtube downloader
(yt-dlp) is not working in deployment server due to youtube's cache policy I have to use jiosaavn api to get the songs, So the song search may not be very accurate.

Tech Stack

How I Build This

  • One Day while watching youtube I found a video by Chigozirim about how shazam works and I was amazed by It. He build a music fingerprinting app using golang and it was amazing. He build it based on the blogpost by Coding_Geek. So inspired by it I decided to build a music fingerprinting app using rust.

  • I started by exploring the audio processing part and then I discovered Symphonia which is a rust crate for audio processing. First I build a function to extract mono audio from a file using it and kept a frequency limited to 41.1 KHz as it is the maximum frequency that the human ear can hear. I also implemented functions to downsample the audio and normalise it. Then I visualized the spectrogram of the audio using image.

  • Then I started to build the fingerprinting part and I used the STFT to extract the frequency domain representation of the audio and used a Hann window function and a hop size of 512 and FFT size of 2048. Then I calculated the STFT for each band using realfft crate.

  • I implemented a function to filter the STFT based on the bands. To limit the number of bands based on which I compute the fingerprint so I used these bands
    (0, 32),
    (32, 64),
    (64, 128),
    (128, 256),
    (256, 512),
    (512, 1024),
    (1024, 2048),
    (2048, 4096),
    (4096, 8192),
    (8192, 20000)
    based on the human ear's frequency response by this blogpost and this blogpost.

  • For the bands I also tried to implement the Mel frequency scale as it is more accurate to human ear's frequency response but it didn't work out as expected and I haven't found any good implementation of it so I used the linear frequency scale with fixed band sizes.

  • Using the filtered STFT I computed the fingerprint using nearest neighbors of 5 peaks for each band and used the kdtree crate and each of the fingerprint I computed the hash using the tuple values of the frequency and time and stored it in a valkey database.

  • Then When I want to find the similar audio files I get the fingerprint of the audio file the functions and then search using the fingerprint as keys from the valkey database and then I sort the values based on the frequency, time difference and anchor count and then I return the top 3 respective songdata based on the filtered values with song_ids.

  • Then I learned Tauri and used my favourite frontend framework SolidJS with TailwindCSS and DaisyUI to build the UI with MotionOne to add animations and Tauri to build the desktop/web/android/ios app.

  • Due to spotify not allowing to download songs, and Youtube downloader (yt-dlp) is not working in deployment server due to youtube's cache policy I have to use jiosaavn api to get the songs, So the song search may not be very accurate.

  • Still the music downloader is working in dev mode as it can use local browser cookies to verify and download the songs.

  • For the Jiosaavn api I used the Jiosaavn Api by Sumit Kolhe and it is under MIT license so it is free for personal use.

  • For the music finder I also discovered vercel rust runtimes and used it to deploy the music finder api functions for free.

  • For the database I used Valkey on Aiven for free.

  • Ultimately I enjoyed building this app and I hope you enjoy using it.

thanks for reading

Top comments (0)