DEV Community: Jefferson Valandro

7 tricks and 3 traps from scraping Twitch, YouTube, TikTok, Kick, Bluesky and Patreon with no login and no browser

Jefferson Valandro — Sat, 10 Oct 2026 18:35:50 +0000

Over the last few weeks I built a family of scrapers for creator data: profiles, followers, social links and business contacts. My rule was no headless browser and no login, plain HTTP only. That makes them cheaper, faster and less fragile. Most of this isn't documented anywhere, so here is what worked, what didn't, and the numbers.

1. Twitch: the public GraphQL does almost everything, except pagination

The Twitch website talks to gql.twitch.tv/gql with a public Client-Id (the web player's). With it you get profiles, followers, live status, VODs, clips and even the channel panels, which is where streamers put their business email.

The trap: paginating with after: returns failed integrity check, because it needs an integrity token generated in the browser. The workaround is to not paginate. Each query accepts first: 100, so I run one query per language and take the 100 most-watched streams of each.

In practice, for VALORANT + Portuguese, 6 out of 10 streamers had a contact email in their panels.

2. Bluesky: search works without login, but only on one host

public.api.bsky.app serves profiles, feeds and followers, but searchPosts returns 403 there. On api.bsky.app the same search works unauthenticated. The cursor for page two, though, returns 403.

The workaround is to walk back in time: the next call uses until = the oldest indexedAt of the previous page. That gets thousands of posts per keyword, no login.

3. YouTube: the About tab is already in the HTML

/@channel/about ships an aboutChannelViewModel inside ytInitialData, with subscribers, views, country, join date, links and the description. It also has a signInForBusinessEmail field. The business email itself sits behind login and captcha, but that field tells you whether the channel has one.

Channel search uses the sp=EgIQAg== filter plus youtubei/v1/search continuations. Two gotchas:

the right continuation token is inside continuationItemRenderer, not the first continuationCommand you find;
in the new layout, the subscriber count comes in a field called videoCountText. Really.

4. TikTok: profiles are easy, video lists are not

The whole profile is in <script id="__UNIVERSAL_DATA_FOR_REHYDRATION__">: followers, likes, bio link and business category. So are the stats of a single video, from its URL. A profile's video list needs a signature (X-Bogus), and I skipped it.

Fun fact: from the cloud, residential proxies got an empty page, while datacenter IPs worked.

5. Kick, Linktree and Patreon

Kick: kick.com/api/v2/channels/<slug> returns the channel and its social links. web.kick.com/api/v1/livestreams lists live streams with a real cursor.
Linktree: everything is in __NEXT_DATA__, including country, plan and page creation date.
Patreon: there is a public JSON:API with search and membership tiers. Patron counts show up in search results even when the creator hides them on their page.

Trap 1: Cloudflare looks at your HTTP library

On Patreon, the same URL from the same cloud IP returned 200 with requests and 403 ("Just a moment...") with httpx. That's the TLS fingerprint. So "it worked in my test" means nothing if the test used a different library.

Trap 2: memory with 2.7 million rows

Another product of mine matches Google Maps places to Brazil's company registry. Searching São Paulo loaded the whole city (2.7M companies) and blew past 4 GB.

The fix was a two-pass filter in PyArrow:

count how many companies in the city contain each searched word;
keep only rows with the same phone, the same postal code, or rare words (name matching gives up on words with more than 400 occurrences anyway).

Before shipping, I diffed old against new on 210 places: zero differences. Memory went from 3.9 GB (crashing) to 350 MB.

Trap 3: marketplace search rewards whoever got there first

I publish these on the Apify Store, priced per result. Plenty of people open the pages, about 15% open the input form, and half of those run it. But in the store's own search, new products barely show up, because ranking weighs users and reviews. Outside posts brought the visitors.

The first scraper outside my Brazilian-company niche to get a paying customer was the Twitch one, on the day it went live.

If you want to try them, they're on Apify (the free plan includes $5 of monthly credit):

Twitch: https://apify.com/jeffev/twitch-scraper
YouTube channels with email: https://apify.com/jeffev/youtube-channel-scraper
TikTok profiles: https://apify.com/jeffev/tiktok-profile-scraper
Kick: https://apify.com/jeffev/kick-scraper
Linktree and link-in-bio pages: https://apify.com/jeffev/link-in-bio-scraper
Patreon: https://apify.com/jeffev/patreon-creator-scraper
Bluesky: https://apify.com/jeffev/bluesky-scraper

I turned Brazil's 73M-establishment company registry into second-fast lookups, with no server and no database

Jefferson Valandro — Sat, 03 Oct 2026 00:36:17 +0000

Brazil's tax authority (Receita Federal) publishes its entire company registry (CNPJ) as open data every month: every company, its address, phone, e-mail, industry code (CNAE), size, tax regime and partners. It's a goldmine for B2B sales, and it's painful to use: 37 zip files, ~8 GB compressed, ~27 GB of semicolon-separated text, and several traps along the way.

I wanted answers like "restaurants opened in the last 90 days in Campinas, with a phone number" or "which company owns acme.com.br?" in seconds, without paying for a server or a database. Here's the setup and, more usefully, everything that went wrong.

The architecture

Receita zips ─► DuckDB (on disk) ─► Parquet ─► private Hugging Face dataset
                                                        │
                 Apify Actors (DuckDB / pyarrow) ◄──────┘

DuckDB joins establishments, companies, the Simples/MEI table and lookup tables.
The result becomes Parquet laid out four ways, one per query shape: per state sorted by industry code (lead search), per CNPJ prefix (bulk lookup), per city (matching Google Maps places) and per e-mail domain.
The files live in a private Hugging Face dataset: free, no credit card, and it serves HTTP range requests.
Each search runs as an Apify Actor that reads only the row groups it needs. Because files are sorted, per-row-group min/max statistics tell you exactly where to look.

The traps

1. The files say ISO-8859-1. They aren't. Some bytes only make sense in Windows-1252 and DuckDB's latin-1 reader rejects the file. I transcode during extraction: chunk.decode("cp1252", errors="replace").

2. ignore_errors=true silently dropped 74% of the companies. Names like BAR ""DO ZE"" LTDA use doubled quotes as escapes. With default settings DuckDB treated those lines as malformed and ignore_errors skipped them without a word: my first build had 8M active establishments instead of 28M. The fix is escape='"'; the real lesson is always compare parsed rows with the file's line count. The converter now refuses to publish when they differ.

3. 32 GB of RAM wasn't enough. Joining 28M establishments with 70M companies and sorting got the process killed. A disk-backed DuckDB database, memory_limit='6GB', a temp directory and one state at a time fixed it. For the 73M-row lookup table, a single global sort hadn't finished after 20 minutes; partitioning by CNPJ prefix in one pass and sorting each ~700k-row piece took 15 minutes total.

4. DuckDB over HTTP was slow for point lookups. Every query re-validated the file and followed Hugging Face's redirect: ~2 s per CNPJ. Switching to pyarrow + HfFileSystem changed that: read each file's footer once, group the requested keys by the row group they fall in (min/max stats), fetch each row group once, 64 in parallel. Shrinking row groups from 20k to 2k rows cut the bytes downloaded for 2,000 scattered keys from ~3.2 GB to ~0.5 GB. Net result: 209 s → 53 s for 2,000 lookups, at 1/6 of the cost.

5. The registered e-mail is often the accountant's. One accounting firm appears as the contact e-mail of 438 companies. To find a company by domain, the chosen name has to resemble the domain (corsi.com.br → CORSI CONTABILIDADE); names that only share the city ("... CAMPINAS") don't count.

6. Matching Google Maps places to the registry. Phone and ZIP + street number produce candidates; name similarity and industry compatibility decide (a dentist must not match the real-estate agency in the same building). On 180 real places in Campinas, 60% matched, almost all with high confidence.

Privacy (LGPD)

It's public data, but: individual micro-entrepreneurs (MEI) are excluded from lead searches by default, partners come without their personal tax ID, and the personal ID that the registry appends to sole-proprietor names is stripped.

Keeping it fresh

A Windows scheduled task checks for a new month every night. When one appears it downloads, converts, compares row counts with the previous month, uploads to Hugging Face, verifies every file actually arrived (I once had an upload "succeed" without a whole folder), and runs a test against each Actor. Any failure becomes a desktop notification instead of a silently broken product.

Try it

The four searches are public on Apify, pay per result:

Brazilian company leads by industry and city, including companies opened in the last N days
Bulk CNPJ lookup with registration status, alphanumeric CNPJ supported
Find the CNPJ of Google Maps places
CNPJ by website domain or e-mail

If you've worked with Brazilian open data and hit another trap, I'd love to hear about it in the comments.