On Monday I checked my Snapchat Spotlight scraper and it looked fine. Green status, run finished in five seconds, exit code 0.
The dataset was empty.
It had been empty since Snapchat pushed a change to their website, and nothing told me. No error, no failed run. If I hadn't opened the output by hand I would have kept "selling" a tool that returned nothing.
Here's what broke, how I found the data again, and the second bug I only noticed because of the first one.
What changed
Public Snapchat profiles (snapchat.com/@username) used to be a classic Next.js page. All the data sat in one script tag:
<script id="__NEXT_DATA__" type="application/json">{ ...props.pageProps... }</script>
My parser grabbed that JSON and read spotlightHighlights and spotlightStoryMetadata. Simple.
This week the profile pages moved to the Next.js App Router. __NEXT_DATA__ is gone. My code raised an error, the loop caught it, logged it, and moved on to the next URL. With one URL in the input, "moving on" meant finishing with zero items and a happy exit code.
That's the real lesson for me: a scraper that swallows parse errors will report success while delivering nothing. Now I check the item count, not the run status.
Where the data went
The App Router streams the page as React Server Components. In the HTML that shows up as a lot of these:
self.__next_f.push([1,"...a chunk of the RSC payload, JSON-escaped..."])
If you join all those chunks you get one long string (about 100 KB for a big profile) that mixes component markup and plain JSON props. The video data is in there as normal JSON objects:
..."playback":{"kind":"spotlight","video":{"id":"W7_EDl...","postingUsername":"mrbeast",
"caption":"Slippery vs Sticky Stairs","videoUrl":"https://...","transcriptUrl":"https://....vtt",
"viewCount":3844229,"likeCount":65752,...}}...
You don't need to parse the whole RSC format. You only need to find a marker and decode one JSON value starting right after it. Python's json.JSONDecoder().raw_decode does exactly that: it reads one value and tells you where it ended, ignoring whatever comes after.
import json, re
FLIGHT = re.compile(r'self\.__next_f\.push\(\[1,"((?:[^"\\]|\\.)*)"\]\)')
def flight(html: str) -> str:
return ''.join(json.loads('"' + m.group(1) + '"') for m in FLIGHT.finditer(html))
def spotlight_videos(html: str) -> list[dict]:
text = flight(html)
marker = '"kind":"spotlight","video":'
videos, seen, pos = [], set(), text.find(marker)
while pos >= 0:
try:
video, _ = json.JSONDecoder().raw_decode(text, pos + len(marker))
except ValueError:
video = None
pos = text.find(marker, pos + len(marker))
if isinstance(video, dict) and video.get('id') not in seen:
seen.add(video.get('id'))
videos.append(video)
return videos
The creator info works the same way with the marker "publicProfile":. Keywords and upload dates are also in the ld+json VideoObject blocks, which you can match to each video by the id at the end of its URL.
One thing I couldn't get back: the old page listed around 15 Spotlights per profile. Now it depends on the creator. MrBeast's page shows 3, Kylie Jenner's 12, DJ Khaled's 36. There's no API call behind the "load more" because there is no "load more". What the page shows is what you get.
The bug hiding behind the bug
Each Spotlight has a transcriptUrl pointing to a WebVTT file that Snapchat generates. My scraper uses it when it exists and only falls back to Whisper when it doesn't. And it had one more rule: if the VTT file is empty, the video has no speech, so mark it no_speech and don't spend money transcribing it.
Once the parser worked again I ran it on MrBeast. Two out of three videos came back as no_speech.
MrBeast videos. No speech. Right.
I downloaded one of the VTT files:
WEBVTT
That's the whole file. The video has a loud audio track (I checked with ffmpeg's volumedetect, mean volume around -17 dB) and Whisper transcribes it without trouble: "Sticky versus slippery stairs. Oh my god. This is so slippery..."
So an empty caption file from Snapchat doesn't mean silence. Sometimes it just means they didn't generate the captions. The fix was to treat "empty" the same as "missing" and let Whisper decide. Whisper is good at saying when there is really nothing to transcribe.
What I'm keeping from this
- Check the number of results, not just the exit code. A run that "succeeds" with 0 items should look like a failure in your monitoring.
- When a site moves to the App Router, the data is usually still in the HTML, just inside the
__next_fstream. Look for a stable key andraw_decodefrom there. - Don't trust a platform's "no captions" as "no speech". Double check with your own transcription before you skip a video.
The scraper is on Apify if you want to try it: Snapchat Spotlight Scraper & Transcript. It runs without a login, and you can test it on the free plan.
Has a site change ever broken one of your scrapers without telling you? I'd like to hear how you noticed.
Top comments (0)