What is Lyrisee?
Lyrisee takes any audio file — MP3, M4A, WAV, or video — and turns it into a real-time kinetic typography experience. Words appear, animate, and exit in sync with the music, styled based on what the lyrics mean, not just how they sound.
Under the hood it's a full AI pipeline:
- Transcription — faster-whisper with word-level timestamps (every word gets an exact start/end time)
- Beat tracking — librosa finds every beat drop
- AI art direction — Gemini reads the full lyrics, picks a visual metaphor per line, decides which words to hit hard, assigns icon symbols where literal (🔥 on "fire", 💸 on "money")
- Rhyme coloring — CMUdict finds true rhyme families; rhyming words share a color
- Typography engine — a Three.js renderer plays it all back, animated to the audio
Step 1: Open Lyrisee
Go to https://acecalisto3-lyrisee.hf.space.
The interface has two panels:
- Left: controls, upload, playback
- Right: the canvas where the kinetic typography renders
Step 2: Load Your Audio
Click Choose File (or drag and drop) and select any audio file:
- MP3, WAV, M4A, FLAC, OGG
- Video files work too (MP4, WebM) — the audio track is extracted
The file stays local — it's only sent to the backend for transcription, never stored.
Step 3: Enable Cloud AI (Recommended)
Toggle the Cloud AI switch on (it's on by default).
Under it you'll see the AI provider dropdown — Gemini is selected by default and is what powers the art direction.
Cloud AI does:
- Word-level transcription via faster-whisper
- Lyric repair (fixes mishear errors using context and rhyme scheme)
- Visual art direction (metaphor per line, word emphasis, icon assignments)
- Rhyme family detection
If Cloud AI is off, Lyrisee falls back to in-browser transcription using a small on-device model — faster but lower quality and no art direction.
Step 4: Click Process
Hit the Process button. The log panel shows live progress:
[upload] your-song.mp3 (8.8 MB) received
[pipeline] provider=gemini
[asr] loading faster-whisper 'tiny.en' (int8 cpu) …
[asr] transcribing (word timestamps) …
[asr] detected language: english (99%)
[asr] 312 words
[beats] 143 beats @ ~92 BPM
[ai] repaired + art-directed -> 318 words, 28 line cues, 14 rhyme families
[done] 318 words · 143 beats
Processing time depends on song length — typically 30-90 seconds for a 3-4 minute song.
Step 5: Play
Once processing completes, hit Play (or press Space).
The canvas comes alive:
- Words appear and exit in sync with audio playback
- Rhyming words glow in matching colors
- Lines with heavy emotional weight animate differently (scale, drift, snap)
- Icon symbols appear on words like "fire", "money", "cage" where the AI decided they hit
Controls
| Control | What it does |
|---|---|
| Space | Play / pause |
| ← → Arrow keys | Seek ±5 seconds |
| F | Fullscreen |
| ✎ Edit lyrics | Open the lyric editor to fix transcription errors |
The Lyric Editor
If the AI mishears a word (happens with heavy slang or unusual pronunciation), click ✎ Edit lyrics.
Each row is one line. Edit the text, nudge the start time if needed, then click Apply & re-render. The canvas updates live with your corrections.
Visual Constructs
The Construct dropdown lets you switch visual styles mid-session:
- Rhyme Scheme — rhyming words animate together, shared color palette
- Embodiment — each line's motion matches its meaning (falling words drop, rising words rise)
- Kinetic Art — pure typographic energy, no semantic logic, maximum visual noise
- Chameleon — the AI picks the best construct per line based on content
FAQ
What audio formats work?
MP3, WAV, FLAC, M4A, OGG, MP4, WebM. If it has audio, Lyrisee can process it.
How accurate is the transcription?
For clear vocals, 90-95%. For heavy reverb, distortion, or layered vocals, lower — use the lyric editor to fix errors.
Does it work for non-English songs?
The current model (tiny.en) is English-optimized. Multi-language support via tiny (no .en) is available by changing the backend model size.
Can I download the result?
Yes — the Download button in the lyric editor saves a corrected lyric_data.json you can reload without re-processing.
Is it free?
The HF Space is free to use. It runs on shared CPU, so processing can take 1-2 minutes for longer tracks. Enterprise/private deployments with GPU available — see the landing page.
Top comments (0)