DEV Community: Pulkit Goyal

SongBhaav v3: Deduplication, Realtime, and Building a Self-Healing Pipeline

Pulkit Goyal — Sat, 20 Jun 2026 17:33:30 +0000

In Part 1, I covered how SongBhaav's backend broke two days after launch and how the v2 rewrite solved the serverless timeout problem with an async job queue.

That architecture held up well. But a few things still bothered me — some were UX rough edges, one was a ticking time bomb I found out about from an email.

This post covers the three changes that make up v3: synchronous lyrics scraping for faster failure feedback, job deduplication, switching from polling to Realtime, and a self-healing system for a Spotify policy change that was about to break my daily sync pipeline.

Problem 1: "Lyrics Not Found" Took Too Long to Tell You

In v2, the flow was: kick off a background job → wait → eventually find out if lyrics existed at all.

This meant that if a song's lyrics genuinely couldn't be found anywhere — LRCLIB, Genius, none of it — the user would sit through the entire async handoff, watch a loading state, and then get told there was nothing to show. The failure mode was slow in exactly the case where it should have been instant.

The fix in v3 was to move lyrics fetching back into the synchronous part of the request, but only the fetching — not the AI processing.

User → POST /api/start-job
         ├─ Check processed_songs (cache hit? return immediately)
         ├─ Check background_jobs (job already in-flight? return existing job_id)
         └─ New song: run fetchLyricsCascade() synchronously
                ├─ No lyrics found anywhere → return immediately, no job ever created
                └─ Lyrics found → save to DB → create job → push to QStash → return job_id

If lyrics genuinely don't exist, the user gets that answer in the time it takes to query three lyrics providers — a couple of seconds — instead of going through the full async detour first. Only the slow, unpredictable part (Gemini analysis) stays async. The fast, deterministic part (does this lyric exist anywhere) happens upfront.

This also let me split the loading UI into two honest stages instead of one generic spinner — an orange "hunting down the lyrics" stage while scraping runs, and a purple "breaking down the bars" stage once AI processing kicks in. Small thing, but it makes the wait feel like progress rather than a black box.

Problem 2: Two Users, Same Song, Double the Work

This one only shows up at a small scale of concurrent traffic, but it's a real bug. If two people searched the same uncached song within a few seconds of each other, both requests would independently check the cache, both would miss, and both would spin up separate QStash jobs — duplicate lyrics fetches, duplicate Gemini calls, duplicate API spend, for the exact same song.

The fix is a simple in-flight check before creating a new job:

checking background_jobs for this spotify_id:
  - status = 'pending' or 'processing' already exists?
      → return that existing job_id to the new request
  - otherwise:
      → proceed to create a new job

Both users end up subscribed to the same job, watching the same background worker resolve once, and both get the result when it completes. No wasted API calls, no race condition, no double billing on Gemini calls for a song that was already being processed.

Problem 3: Polling Worked, But Realtime Is Just Better Here

In v2, the frontend polled /api/check-job every few seconds to check status. It worked, and I wrote about why polling was a reasonable choice over WebSockets at the time — simpler, no persistent connection overhead.

In v3, I switched to Supabase Realtime instead. A few reasons pushed this:

Polling means either over-fetching (checking too often, wasting requests) or under-fetching (checking too rarely, feeling laggy). Realtime sidesteps that tradeoff entirely — the update arrives the instant the row changes, no guessing on interval timing.
With deduplication now in place, multiple clients can be waiting on the same job. Realtime handles "many clients subscribed to one row" more naturally than each client running its own polling loop against the same endpoint.
The actual implementation overhead turned out to be smaller than I expected — subscribing to a filtered channel on job_id is a few lines, and Supabase handles the connection lifecycle.

So this wasn't a case of "polling was wrong" — it was right for v2's constraints. It just stopped being the better tradeoff once concurrent job-sharing entered the picture.

Problem 4: An Email From Spotify About Something That Hadn't Broken Yet

SongBhaav runs a daily sync via GitHub Actions that pulls my recently played tracks from Spotify, so songs are pre-processed before anyone searches for them. This depends on a Spotify refresh token staying valid indefinitely.

Spotify recently emailed every developer with a policy change: starting July 20, 2026, refresh tokens will expire after six months. Once expired, any attempt to refresh returns an invalid_grant error, and the only fix is sending the user through the sign-in flow again.

This hadn't broken anything yet. But it was going to, on a specific date, with zero warning beyond that email — exactly the kind of failure that's easy to forget about until your daily sync silently stops working months from now and you have no idea why.

So instead of waiting for it to break, I built the recovery flow ahead of time:

Daily sync script runs
  → reads refresh_token from system_credentials table (DB, not env vars)
  → requests new access token from Spotify
  → if invalid_grant:
        → generate one-time secure ticket, insert into admin_sessions (1hr expiry)
        → POST a magic link to a private Discord webhook
        → exit gracefully (no retry, no crash loop)

When I click the Discord link:
  → /api/admin/spotify-login validates the ticket, marks it used, redirects to Spotify OAuth
  → /api/admin/spotify-callback exchanges the auth code, UPSERTs new refresh_token into system_credentials

Next sync run → reads the new token → succeeds

A few deliberate choices here:

Credentials live in the database, not environment variables. This means re-authorizing doesn't require a redeploy — the next scheduled run just picks up the new token from the DB automatically.

The ticket is single-use and time-boxed. Even if the Discord webhook URL ever leaked, an expired or already-used ticket can't be replayed.

The script exits cleanly on invalid_grant instead of retrying. Retrying a dead token wastes calls and clutters logs with the same error repeatedly. Better to fail once, loudly, and wait for the human-in-the-loop fix.

The actual mechanism is simple — a magic link sent over a webhook. What made it worth building was reading the policy email in advance and treating "this will eventually fail" with the same urgency as "this is currently failing."

What Changed, In Summary

Concern	v2	v3
Lyrics-not-found feedback	Async, slow	Synchronous, instant
Duplicate concurrent searches	Each spawns a new job	Deduplicated, shared job
Client notification	Polling `/api/check-job`	Supabase Realtime subscription
Spotify token expiry	Not handled	Self-healing via Discord + OAuth

None of these were emergencies. That's sort of the point — v2 wasn't broken, it just had room to get better, and one looming policy change that hadn't caused damage yet but eventually would have.

If you want to try SongBhaav yourself, drop any song in the search bar: song-bhaav.vercel.app

I Built an App to Finally Understand the Songs I Listen To (And Then Had to Rewrite the Entire Backend)

Pulkit Goyal — Sat, 13 Jun 2026 19:40:19 +0000

I listen to a lot of Punjabi music. Always have. The beats are incredible — but I'd be lying if I said I understood more than 60% of what was actually being said. The usual fix is to Google the lyrics, find a translation, switch tabs, lose the vibe entirely. Or paste the lyrics into ChatGPT every single time. It works, but it's friction.

So I built SongBhaav — a webapp where you search any song and get line-by-line translations, emotional interpretation, cultural context, and what I like to call the "kavi kya kehna chahte hain" treatment. The thing your English teacher used to do with poems in school. Breaking down what the artist actually meant, not just what the words say.

It started as a weekend side project. Then I made it a proper webapp. Then two days after going live, half the backend broke in production.

This post is about how the app works, what broke, and the architectural decisions that came out of fixing it.

What SongBhaav Actually Does

The core flow is simple: search for a song → get a full breakdown.

The breakdown includes:

Line-by-line translation — not just a raw word-for-word translation but one that preserves the intent
Interpretation — what the song is actually about, the emotional arc, the metaphors
Fun facts — context about the artist, the album, cultural references in the lyrics
Emotional themes — the overall sentiment and mood

I use it daily now. Mostly for Punjabi tracks but it works across languages — anything Gemini can process, which is quite a lot.

The tech stack: Next.js (App Router) on the frontend, Supabase (PostgreSQL) as the database, Gemini models for AI processing, and — as we'll get to — Upstash QStash for the async architecture.

How v1 Worked (And Why It Was Naive)

The first version was a straightforward synchronous pipeline:

User searches song → POST /api/process → Fetch lyrics → Gemini AI → Return response

For songs already in the database, it was fast — just a cache hit on Supabase and done. The problem was cache misses. Any song that hadn't been analyzed before required the full pipeline: fetch lyrics from an external source, send them to Gemini, wait for a structured JSON analysis to come back. That whole flow was consistently taking 15 to 25 seconds depending on song complexity and Gemini's response time.

Vercel's Hobby tier enforces a hard 10-second execution limit on serverless functions. So for any new song, users would wait, the browser would spin indefinitely, and they'd eventually get a FUNCTION_INVOCATION_TIMEOUT error. Not a great first impression.

I knew this was a problem during development. I launched anyway, telling myself it "probably wouldn't be that bad in practice."

It was that bad in practice.

What Broke in Production

Problem 1: Cloudflare Killed My Lyrics Scraper

Before integrating official APIs, I was scraping lyrics directly from Genius. It worked fine locally, worked fine in the first couple of days after launch.

Then Genius added Cloudflare protection to their website.

The errors started showing up in my pipeline_logs table — not clean API errors, but this:

SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON

What was happening: my serverless function would make a request to scrape the Genius page, Cloudflare would intercept it, identify it as an automated agent, and serve back an HTML "Access Denied" page instead. My code would then try to parse that HTML as JSON and crash immediately.

The fix was straightforward once I understood the cause — register an official Genius Developer API client, switch to authenticated API calls with Bearer tokens, bypass the anti-bot layer entirely. Since making that switch, lyrics fetching has had zero failures.

But the lesson was more important than the fix: you cannot deploy a web app and consider it done. Third-party services update, add protections, change behavior. The two days between launch and breakage weren't anything I did wrong — the external environment just changed. Production monitoring isn't optional; it's how you find out about this before your users do.

Problem 2: The 10-Second Ceiling

This one I knew was coming and hadn't properly solved. The serverless timeout wasn't an occasional edge case — it was guaranteed for every new song search. There was no optimizing my way out of it within a synchronous request model. The question was: what's the right architecture for a task that inherently takes longer than your platform allows?

The v2 Rewrite: Going Async

The core insight was simple: don't make the user wait for the slow thing. Start the slow thing, tell the user you've started it, then notify them when it's done.

This is a standard pattern in distributed systems — sometimes called a "front desk / back office" model. The front desk takes your request immediately and gives you a ticket. The back office does the actual work. You don't stand at the counter for 25 seconds.

Here's what the new flow looks like:

Three components make this work:

Upstash QStash is the message broker. When a user requests a new song, /api/start-job pushes a message to QStash with the track metadata and immediately returns a job_id to the client — the whole thing resolves in under 50ms. QStash then handles triggering the background worker, with automatic retries built in if anything fails.

One practical note: QStash delivers messages via HTTP POST to your worker endpoint, which means it can't reach localhost in local development. I handled this by detecting the environment in /api/start-job and calling the worker directly in dev, bypassing QStash entirely. Not elegant, but it works cleanly and keeps the local development loop fast.

The polling loop handles the feedback to the user. While the background worker runs, the frontend periodically hits /api/check-job?id=... every few seconds, checking the background_jobs table until the status flips from processing to completed. Once it does, the client fetches the result and swaps the skeleton loader for the actual data. Simple, lightweight, and it avoids the complexity and connection quota costs of maintaining a persistent WebSocket.

The worker runs the actual pipeline: concurrent lyrics fetching via Promise.any across LRCLIB and Genius (whichever resolves first wins), then Gemini processing, then writing the structured result to processed_songs.

Handling Gemini 429s

One edge case I didn't think about until it happened: what if Gemini rate limits mid-job?

Before adding proper error handling, a 429 from Gemini would cause the worker to throw an unhandled error. QStash would mark the delivery as failed, and background_jobs would stay stuck at processing forever. The user would sit on a skeleton loader indefinitely with no feedback or resolution.

The fix: explicitly return a 500 status when Gemini throws a 429 or 503. QStash treats any non-2xx response as a failed delivery and automatically schedules a retry using exponential backoff. QStash's backoff formula is min(86400, e^(2.5*n)) seconds — so roughly 12 seconds after the first failure, ~2.5 minutes after the second, ~30 minutes after the third. Aggressive enough to let Gemini's rate window reset, spaced out enough not to hammer it again immediately.

This means transient rate limit errors self-heal entirely without manual intervention.

Securing the Worker Endpoint

Easy thing to overlook with async architectures: your background worker is just an HTTP endpoint sitting on the public internet. Anyone who finds the URL can POST to it and trigger AI processing, draining your API credits.

QStash solves this with cryptographic signature verification. Every message QStash sends includes an upstash-signature header signed with your secret key. The worker validates this using the QStash SDK before executing anything — invalid or missing signatures get a 401 immediately.

For rate limiting the public-facing endpoints, I used Upstash Redis with the @upstash/ratelimit SDK inside Next.js middleware. Since middleware runs at the edge (CDN level on Vercel), spam requests are blocked before they ever touch the database or trigger any compute.

What I'd Do Differently

Start async from day one. The v1 synchronous architecture was always going to hit the timeout wall for new songs. Building the QStash pipeline upfront isn't significantly more complex than the naive approach — it would have saved the rushed production rewrite.

Set up alerting before going live, not after. I found out about the Cloudflare failure by manually checking pipeline_logs. A simple alert on error rate spikes would have caught it immediately. Logging without alerting is only half the job.

Don't assume external services stay stable. Genius adding Cloudflare wasn't predictable. The right response isn't to avoid third-party dependencies — official APIs are fine — but to monitor them actively and build fallbacks where possible.

If you want to try SongBhaav, drop any song in the search bar: SongBhaav

Feedback welcome — especially if something in the architecture could be done better.