Standard lyrics-sync apps take a line's timestamp and guess when each word lands. If a line starts at 4.2s and ends at 6.8s across 8 words, each word gets ~0.3s — regardless of whether the singer held one word for a full second and rapped the next seven in a burst.
Lyrisee doesn't guess. Every word gets its own measured start and end time.
How Word-Level Timestamps Work
Lyrisee uses faster-whisper — an optimized implementation of OpenAI's Whisper model — with word_timestamps=True.
Whisper processes audio in 30-second chunks and produces attention weights over the spectrogram. faster-whisper uses those attention weights to find exactly when the model's "attention" peaks for each word token — that peak is the word's center timestamp. Start and end are derived from the attention envelope.
from faster_whisper import WhisperModel
model = WhisperModel("tiny.en", device="cpu", compute_type="int8")
segments, info = model.transcribe(audio_path, word_timestamps=True, vad_filter=True)
words = []
for seg in segments:
for w in seg.words:
words.append({
"text": w.word.strip(),
"start": round(w.start, 3),
"end": round(w.end, 3),
})
The result: a list like:
[
{"text": "dark", "start": 12.480, "end": 12.720},
{"text": "nights", "start": 12.720, "end": 13.200},
{"text": "running", "start": 13.440, "end": 13.800},
{"text": "cold", "start": 13.880, "end": 14.120}
]
Why This Matters for Typography
With interpolated timing, a held note ("niiiights") would show the word for exactly its share of the line duration — even though the singer held it 4× longer than a normal word.
With word-level timing, "niiiights" shows exactly as long as it sounds. The typography breathes with the vocal performance.
AI Lyric Repair
Whisper sometimes mishears words — especially rap (fast delivery, slang, AAVE, deliberate wordplay). Lyrisee sends the raw transcript to Gemini for correction:
System: You are Lyrisee's lyric-repair stage. Fix transcription errors using
the subject matter, rhyme scheme, and surrounding lines. Preserve the artist's
voice: slang, contractions, profanity, proper nouns. Do not rephrase correct words.
Input:
1. dark nights running through my vein
2. pain is all i fill i got nothing to gain
3. [...]
Output:
1. dark nights running through my veins
2. pain is all I feel I got nothing to gain
After repair, corrected tokens are re-aligned to the original word timings using difflib.SequenceMatcher. Sync stays tight even after word corrections.
Beat Tracking
In parallel, librosa analyzes the audio for beat positions:
import librosa
y, sr = librosa.load(audio_path, mono=True)
tempo, frames = librosa.beat.beat_track(y=y, sr=sr)
beats = librosa.frames_to_time(frames, sr=sr).tolist()
# [0.371, 0.742, 1.114, 1.485, ...]
The renderer uses beat timestamps to trigger visual "hit" animations — the canvas reacts to the music, not just the lyrics.
POS Tagging
spaCy tags each word with its part of speech (NOUN, VERB, ADJ, PROPN, etc.). The visual engine uses POS to set default sizing — nouns and verbs render larger, function words smaller — before AI art direction overrides specific words.
The Output Format
Everything feeds into a single lyric_data.json:
{
"words": [
{"text": "dark", "start": 12.48, "end": 12.72, "pos": "ADJ",
"dir": {"emphasis": 2, "register": "heavy", "glow": true, "icon": null}},
{"text": "nights", "start": 12.72, "end": 13.2, "pos": "NOUN",
"rhyme": 3}
],
"beats": [0.371, 0.742, 1.114],
"metaphors": [{"start": 12.48, "metaphor": "fall"}],
"rhyme_families": [["veins","gains","pain","brain"]],
"rhyme_palette": {"0": "#5CE1E6", "1": "#FF2E2E", "3": "#FFD166"}
}
The frontend reads this file and the renderer handles the rest — no backend connection needed during playback.
Try It
https://acecalisto3-lyrisee.hf.space — upload any song, watch it render.
Top comments (0)