DEV Community: Yuhe He

The 429 Playbook: Handling Rate Limits Before Your IP Gets Banned

Yuhe He — Sun, 11 Oct 2026 01:02:53 +0000

The 429 Playbook: Handling Rate Limits Before Your IP Gets Banned

Every scraping tutorial shows the happy path. The page that matters is the one you get after an hour of collecting: 429 Too Many Requests.

Here's the playbook I run, ordered by what each response should cost you.

1. Read the headers first, back off second

Most servers tell you what to do in the response:

Retry-After: the contract. If it says 30, sleep 30. Not 5, not "I'll retry in a loop for a second".
X-RateLimit-Remaining / X-RateLimit-Reset: your budget and its refill time. When Remaining hits 2, you throttle before the 429, not after.
No headers (common on free-tier APIs): treat every 429 as a warning shot. The next escalation is usually a temporary IP ban, and those are invisible — you get a 403 with a "blocked" page, not a rate-limit message.

2. The three-stage ladder

1st 429  → exponential backoff: 2^n * base (cap at 60s), 2 retries
2nd 429  → drop concurrency to 1, switch to a paced schedule (e.g. 1 req / 5s)
3rd 429  → stop. Persist the cursor, exit non-zero, let the scheduler retry later

The third stage is the one people skip. A tight retry loop that never gives up is how a free-tier API learns your IP address very well, very permanently. My collection jobs are resumable (cursor in a state file), so "stop and come back in an hour" is cheap — the pipeline picks up exactly where it died.

3. The failure mode nobody shows you

A 429 is not the only rate signal. Watch for:

Silent partial responses: 200 OK with an empty list, where yesterday's identical call returned 50 items. Treat "200 + empty" as a soft 429 and re-verify against a known-good channel/endpoint before trusting the gap.
Latency creep: responses slowing from 200 ms to 2 s is a rate limiter warming up. Throttle proactively.
Quality cliffs: the payload structure changes (null fields, truncated text) — some services degrade responses before they refuse them.

4. Budget math before you write code

Free tiers are rate-limited per IP, per key, per day. Do the arithmetic up front: 45 requests/min free tier × 60 min = 2,700 requests. At 200 items per page, a 20,000-message collection is 100 pages — fits easily on one channel, dies instantly at 30 channels. The playbook only works if your request budget covers the job; otherwise you're choosing between key rotation, sampling, or a paid tier today, not at 2 a.m. mid-collection.

The short version: 429s are a contract, not an obstacle. Read it, back off on schedule, persist your cursor, and know when stopping is the cheapest move in the pipeline.

Merging Two Messy CSVs: The Key Problems Excel Formulas Hide From You

Yuhe He — Sun, 11 Oct 2026 00:26:32 +0000

Merging Two Messy CSVs: The Key Problems Excel Formulas Hide From You

You have two spreadsheets. One is your export, one is the client's. Someone says "just merge them on the ID column." That sentence has caused more quiet data disasters than any schema migration, because it assumes three things that are never true: the IDs match, the columns mean the same thing, and the rows are unique.

I reconcile exported data files for a living — chat logs, server logs, marketplace exports — and the merge is where the bodies are buried. Here are the failure modes, in the order they will find you.

1. The Key Is Not a Key

"Same ID column" usually means: one file has id, the other has customer_id, order.customer, and a vlookup_key someone added by hand. Open both, count distinct values, and compare sets before you merge:

a_ids = set(df_a["id"])
b_ids = set(df_b["customer_id"])
print(len(a_ids), len(b_ids), len(a_ids & b_ids))
print("orphans A:", len(a_ids - b_ids), "orphans B:", len(b_ids - a_ids))

If the intersection is 91%, not 100%, your merge is a research project, not a formula. Excel's VLOOKUP returns #N/A and moves on; a pandas merge silently picks a join type for you. You must count the misses, not spot-check them.

2. Invisible Keys

The intersection is computed on strings. "A102" and "A102 " — trailing space from a CRM export — are different strings. "42" and "42.0" — one file went through a spreadsheet app that cast numbers — are different strings. The fix is boring and mandatory: trim, cast, and casefold both sides before joining, then re-count. Ninety percent of "impossible" 4% miss rates are whitespace.

3. Uniqueness Is a Claim, Not a Fact

Run df["id"].is_unique on both sides. If the right side has duplicates, your 1,000-row left table becomes 1,400 rows after the merge, and every downstream sum is inflated by 40%. In Excel you see it as weird double-counting you can never explain. In code, the assertion is one line, and it fails loudly.

4. The Merge Type Is a Business Decision

Inner drops unmatched rows. Left keeps your rows and nulls theirs. Outer keeps everything and doubles your review work. There is no default — there is only what the deliverable promises. A reconciliation report is outer-merge, annotate. A clean master table is inner-merge, log the drops. Pick before you write the line, and record the count of rows that fell out the bottom of every join you do.

5. Dates Are Four Formats in a Trench Coat

2026-1-5, 01/05/2026, 5 Jan 26, and a serial number 46027 are the same date in four costumes, and the file with the serial numbers does not tell you it is wearing one. Parse with explicit formats; never trust auto-detection to pick month-first. 03/04/2026 is unambiguous to no one.

The Workflow That Survives

Load both files raw. Normalize (trim, cast, casefold keys; explicit date parse). Assert uniqueness. Merge with an explicit how and an indicator. Count before, after, matched, dropped — all four, in the log, every run.

Total: one screen of pandas, zero #N/As, and a number you can hand the client when the totals do not match: "14 rows in their file had no match on our side; here they are." That sentence — not the merge itself — is what you are being paid for.

IP Geolocation APIs Compared: What the Free Tier Really Gives You

Yuhe He — Sun, 11 Oct 2026 00:03:18 +0000

IP Geolocation APIs Compared: What the Free Tier Really Gives You

You have an IP address. You want a city. The obvious move is to grab a free geolocation API key and call it a day. The bill arrives later, in a currency you did not price in: quota cliffs, stale databases, and the moment your "free" city lookup starts costing you real money.

I ran the same list of 1,271 IPs through the free tiers of the usual suspects. Here is what each route actually gives you before it starts charging.

The Five Routes

ipapi.co — 1,000 calls/day free. Zero setup, no key needed for the first tier.
ipinfo.io — 50 calls/day free. Fast, accurate, and the free tier is a demo, not a tool.
ip-api.com — 45 requests/minute free, no key. Self-host option for the whole database.
MaxMind GeoLite2 — free database file, license key required, updated weekly. Local lookups, unlimited.
BigDataCloud — free tier with 10,000 requests/day, no key, non-commercial clause.

What the Free Tiers Hide

The quota cliff is not a ramp. ipinfo's 50/day means any batch over fifty needs a card. ipapi's 1,000/day looks generous until you realize a single day of log enrichment for a mid-size site is 50,000 lookups. The cliff arrives at the exact moment your project stops being a toy.

ip-api's rate limit is the friendly one. 45/minute, no key, no card. For a solo builder enriching a CSV of a few thousand IPs, you just sleep 1.4 seconds between batches and never pay. The catch: free tier is non-commercial, and the cloud endpoint lags the paid one on database freshness.

MaxMind is the only free tier with no per-call ceiling — because it is not an API at all. You download the GeoLite2 file, load it locally, and every lookup is free and instant, forever. The real cost is the license-key registration dance (account required), the weekly update cycle (a location that moved yesterday is stale until you re-download), and the accuracy floor on mobile ISPs: a cell-tower IP will place you in the ISP's regional hub, not the user's street. No free tier fixes that; it is a property of the data, not the vendor.

BigDataCloud trades money for clauses. 10,000/day free is the most generous quota in the group, but the license is non-commercial. For a side project measuring where readers come from, fine. The moment the project earns a dollar, you are re-reading contracts.

The Decision Table

Situation	Route	Why
One-off CSV, <1,000 IPs	ip-api.com	No key, rate limit is the only rule
Recurring enrichment, thousands/day	MaxMind GeoLite2 local	No per-call cost, weekly freshness is enough
Need a city on one request	ipapi.co	No-key tier, one call, done
Compliance-sensitive, no card on file	ip-api.com + sleep	Key-free by design
Product with revenue	Paid tier of your pick	Free tiers' non-commercial clauses will bite

What I Would Not Do Again

Do not build your pipeline against the API shape of the free tier and assume you can scale into the paid one. The databases differ between free and paid at several vendors — same endpoint, different accuracy, and you will debug "why did the city change" for a week before you notice the vendor swapped the underlying file behind the same URL. Pin the database, version your expectations, cache aggressively: an IP-to-city map you already fetched is the only geolocation lookup that is both free and instant.

For OSINT-style work — enriching exported chat logs, server logs, forum scrapes — I settled on MaxMind local for bulk and ip-api for the trickle, and the whole stack costs nothing. The day you need city-level accuracy for ad targeting, none of these free tiers are your product anymore, and that is fine: they were never priced to be.

What free geolocation route has burned you with a cliff? I am curious which quota developers actually hit first.

requests vs httpx for Scraping: When the Faster Client Stops Saving You Money

Yuhe He — Sat, 10 Oct 2026 17:00:32 +0000

requests vs httpx for Scraping: When the Faster Client Stops Saving You Money

requests is the default Python HTTP client; httpx is the modern one that added async, HTTP/2, and a compatibility layer. The pitch is "httpx is strictly newer, migrate." The scraping reality is more boring and more useful: the client is rarely your bottleneck — and when it is, the savings have a ceiling worth pricing before you touch code.

What each one actually buys you

requests: the ecosystem default. Every scraping snippet, every library example, every Stack Overflow answer assumes it. Session objects give you keep-alive and cookie persistence, which covers 90% of polite collection. It is synchronous. That's not a bug in the library; it's a design contract: one request, one wait, one result.

httpx: requests-shaped API plus three real features —

async I/O: hundreds of in-flight requests from one process.
HTTP/2: multiplexed connections to servers that support it.
Proxy/transport hooks that are cleaner for pools and rotations.

The migration line is one import. The trap is thinking that line buys you speed.

Where the money actually goes

Model a collection run: N pages, per-page server latency L, network RTT, politeness delay P (what robots.txt and your conscience demand), and rate-limit ceiling R for the target.

If P and R dominate — polite collection against one site — concurrency is dead weight. The target says "one request per 3 seconds"; httpx with 200 async slots will happily queue 199 of them. requests with a sleep does the same job at the same throughput, with less code. The async upgrade costs you: event-loop debugging, async-infection through your whole pipeline, harder tracing. The savings: zero, because the bottleneck was never your client.
If you hit many independent sites — search engines, APIs, DNS-adjacent lookups, per-site spot checks — async pays. 200 targets × 500 ms: sequential = 100 seconds; async = under a second of wall time per batch. This is where httpx earns its migration.
HTTP/2 is situational. It helps when a target serves many small resources over one connection and the server supports it. Against a plain nginx with HTTP/1.1 enabled, you get parity with a version negotiation handshake.

The cost of the async switch, honestly

Everything up and down the stack becomes async. BeautifulSoup parsing stays sync (fine), but your pipeline, scheduler, CLI — if you use asyncio, your functions become coroutines. For a script you maintain alone, that's a taste; for a team, it's a training cost.
Debugging changes shape. Tracebacks through an event loop are a genre of their own. requests failures read like books; async failures read like crime scenes.
Ecosystem drag. Older tutorials and helper libraries assume requests; you'll occasionally be the compatibility layer.

Decision table

Collection shape	Right client	Why
One site, polite pacing	requests	bottleneck is P and R, not the client
Hundreds of independent targets	httpx async	wall-time wins are real
HTTP/2 APIs you poll continuously	httpx	multiplexing cuts connection churn
Team codebase, mixed maintainers	requests	debugging and onboarding cost less
Drop-in modernization of existing code	httpx (sync mode)	free compatibility layer, no async tax

The clients are both free. The real ledger is wall-clock time you save against code complexity you add — do the arithmetic on your actual bottleneck first. For most polite, single-site scrapes, the answer is: you're already at the speed limit; the new car changes nothing.

The collector and dedupe pipeline behind my public-data runs is here; the free public-source field guide is here. Related: Scrapy vs BeautifulSoup, where each starts costing you.

Scrapy vs BeautifulSoup for Public-Data Collection: Where Each One Starts Costing You

Yuhe He — Sat, 10 Oct 2026 16:25:32 +0000

Scrapy vs BeautifulSoup for Public-Data Collection: Where Each One Starts Costing You

Every scraping tutorial starts with pip install scrapy or pip install beautifulsoup4 like they're interchangeable. They're not even the same kind of thing, and picking wrong costs you a rewrite at exactly the worst moment — when the pilot works and the real job is 100× the size. Here's the cost map I wish I'd had.

The category error first

BeautifulSoup is a parser. It takes HTML you already fetched and lets you query it. It does not fetch, does not schedule, does not retry, does not respect robots.txt. Fetching is requests (or httpx), a separate decision.
Scrapy is a crawling framework. Fetch scheduling, concurrency, retries, pipelines, export, robots.txt — an opinionated machine you configure and feed.

Asking "Scrapy vs BeautifulSoup" is a bit like asking "freight train vs clipboard." Both appear in logistics.

Where BeautifulSoup (+requests) costs you

The pilot is unbeatable: ten lines, running in two minutes, zero ceremony.

The bill arrives with scale:

You rebuild the framework, badly. Concurrency, retries with backoff, politeness delays, dedupe, incremental saves — every scraper past 100 pages grows these organically, ad hoc, untested. That pile of one-off logic is the real cost, and it's paid in late nights.
Memory model is naive. Parsing whole trees in one process is fine per page; at millions of pages, your hand-rolled loop has no backpressure story.
No pipeline story. Cleaning, validating, and exporting (CSV/JSON/DB) is all your code, all your bugs.

Where it stays the right answer: one page, known structure, low volume, throwaway analysis. The clipboard is correct for notes.

Where Scrapy costs you

Ceremony tax up front. Items, spiders, middlewares, settings — a "hello scrape" is a project scaffold, not a snippet. For a 50-page job, you've paid a framework for a clipboard task.
Async mindshift. Callbacks/Deferreds, errbacks, the shell's quirks: debugging Scrapy is debugging a framework plus your code. Newcomers lose days here.
JavaScript pages are a toll booth. Raw Scrapy speaks HTTP; JS-rendered content needs scrapy-playwright or a browser hop — extra infra, extra memory, extra "why is it slow."

Where it pays off: many pages, many sites, sustained collection with export pipelines. That's exactly when the train earns its track.

Decision rule (the one I use)

< ~200 pages, one site, analysis once → requests + BeautifulSoup. Ship the clipboard.
Sustained, multi-page, scheduled collection with cleaning/export → Scrapy, and budget a day for the framework, not an hour.
JS-heavy sites at scale → plan for the browser layer's bill before choosing either. The scraping framework isn't your main cost there; headless browsers are.

The libraries are free. The rewrite is the price — pick by the shape and volume of the collection, not by the size of the first page.

I publish the collection notes from my own runs here; the collector + dedupe pipeline behind them is here, and the free public-source field guide is here. Related: scraping public Telegram channels without an API key.

Telethon vs Tweepy: What Each Python Client Library Actually Costs You to Collect Public Data

Yuhe He — Sat, 10 Oct 2026 16:01:46 +0000

Telethon vs Tweepy: What Each Python Client Library Actually Costs You to Collect Public Data

Two libraries, both free to pip install, both promise "just read the public feed." Then the bill arrives in a currency that isn't dollars: credentials, rate limits, and account risk. I ran collection pipelines against both ecosystems, and the cost shapes are genuinely different. Worth mapping before you pick one.

Telethon (Telegram MTProto): free code, paid identity

Telethon is the Python standard for talking to Telegram's raw MTProto API. The library is MIT-licensed and costs nothing. The bill:

You bring your own credentials. Every Telethon script needs an api_id + api_hash issued to your Telegram account at the developer portal. No key, no connection. The free library quietly makes your personal account the product's dependency.
The first run is a login. A new session string means a phone-code prompt (on the app, not SMS). Headless servers and this interact badly; plan for a manual step the first time.
Flood limits are per-account, not per-script. Telegram's server counts your account's request velocity. Cross a line and you get FloodWaitError with a delay attached — minutes to hours. Repeat offenders get the account rate-limited harder, and abusive collection gets it banned. The library gives you the API; you wear the consequences.

The public-channel read itself is genuinely cheap: joining a public channel and paging through history is a couple dozen requests if you set a sane messages.GetHistory limit. One channel's full public history, politely fetched, is a few minutes at conservative pacing. The cost is all amortized: registration once, pacing forever.

Realistic ceiling: one well-paced Telethon session is fine for dozens of channels. Hundreds of channels polled continuously is a fleet problem — multiple accounts, session pools, per-account budgets. The library won't tell you that; the FloodWaitError will.

Tweepy (X/Twitter): free code, metered platform

Tweepy wraps X's API. The library is free and excellent. The bill is the platform:

The free API tier is a write-only tier. Current free-tier app access is roughly: you can post, you can do a small number of reads per month — on the order of hundreds of read calls per month, not per day. Collecting a public timeline on the free tier is not "slow," it's arithmetically impossible for anything beyond spot checks.
Reads are a subscription now. Real timeline/stream reads moved behind monthly plans that start in the hundreds of dollars. The pip install tweepy moment hides a "contact sales" moment.
OAuth is a maze. Bearer tokens, app-only vs user-context, per-endpoint auth rules. Most first-week Tweepy errors are auth errors wearing rate-limit costumes (401 that looks like a 429's cousin).

What the free tier is good for: the spot check. Verify a public account exists, pull its last few statuses occasionally, sign a bot that posts. For collection, the honest free path is scraping-adjacent (public web previews) with all the fragility that implies.

Side-by-side, honestly

Cost axis	Telethon	Tweepy on free tier
Library	free (MIT)	free (MIT)
Credentials	your account's api_id/hash	app bearer token
First-run friction	phone-code login	developer portal + apps
Read budget	generous, per-account pacing	tiny (hundreds/month)
Over-budget penalty	FloodWait → ban risk	hard 429, subscription prompt
Public-history depth	good (page through)	near zero on free
Continuous polling	realistic at dozens of channels	unrealistic

Pick by collection shape

Deep history, many channels, patient polling → Telethon. Your costs are real (one account's risk, careful pacing) and the ceiling is high. Budget a few seconds between channel requests and treat FloodWaitError as a scheduling signal, not an error.
Occasional spot checks, one account, low volume → Tweepy free tier is exactly sized for you. Buy the subscription only when the read volume mathematically demands it.
Public pages only, zero credentials → skip both. Telegram's t.me/s/ preview page gives you the last ~20 public messages with no key at all; for X, public web surfaces get you the visible slice. No account, no meter, no ban risk — and correspondingly shallow history.

The libraries are free. The meter underneath isn't — it's just denominated in accounts, delays, and subscriptions, so it doesn't show up in the pip install receipt. Check the meter before you build the pipeline, not after it's live.

I keep working notes and reusable pieces in the open: the polling collector behind my own runs is here, and the public-OSINT field guide (the free tier of it is genuinely free) is here. If you're choosing a Telegram route, this comparison of free options covers the non-library paths.

Telegram API Free Options, Ranked: What Each Route Actually Costs You

Yuhe He — Sat, 10 Oct 2026 14:57:48 +0000

Every tutorial for reading Telegram channel history starts the same way: "create a bot at @botfather and get your API key." Then you discover the channel you actually want to monitor belongs to a stranger, and the bot API quietly stops being an API at all.

Here is the map of what "free" means for each route, as of today, from someone who ships a poller in production.

Route 1: Official Bot API — free, with a doorbell problem

api.telegram.org costs nothing. The catch is social, not financial:

Your bot must be a member (usually admin) of the channel to see its messages.
For channels you do not own, that is a cold-outreach funnel: email channel owners, ask for admin rights, get ignored 90% of the time.
getUpdates long-polling is free; webhooks need a public HTTPS endpoint (free tier hosts suffice).

Free money-wise. Expensive in business development.

Route 2: MTProto (Telethon / Pyrogram) — free, with a phone problem

Full client API: history, members, media. Costs:

An api_id/api_hash from my.telegram.com — requires a phone-verified Telegram account.
New accounts get aggressive flood-wait limits. Scraping 300 channels from a fresh account is a good way to get the account limited within a day.
Old accounts used as scrapers get banned at a nonzero rate. That is the real cost line.

Route 3: The public web preview — free, with a ceiling

https://t.me/s/<channel> renders the last ~20 messages as plain HTML. ?before=/?after= page backwards and forwards from the current tip. No key, no phone, no bot.

The honest limits:

No deep backfill. Discovery-time messages are gone; the preview window is what it is.
It is a UI, not a contract. The HTML can change shape; parse defensively, alert when your column count drops.
Be polite. One request per channel per interval, back off on 429.

For monitoring — watch many channels from day zero, get every new message — it is the only route that scales to hundreds of channels without renting a phone farm.

What "free" should not mean

Paying $50/mo to a reseller for bytes that are public. People arrive there after Route 1's doorbell problem breaks their spirit, not after measuring the preview window's actual ceiling.

I run Route 3 in production. The dedup math (forward-trees make naive counting lie by 30%), the six-column schema, and the polling scheduler are written up in the Telegram OSINT Sample Brief (pay what you want, $1+). The full pipeline config ships in the Telegram & Web OSINT Bundle ($5).

What is your flood-wait war story? Comments open.

Telegram Bot vs Telegram Scraper: Which One You Actually Need (No API Key Required)

Yuhe He — Sat, 10 Oct 2026 14:29:24 +0000

A "Telegram bot" and a "Telegram scraper" solve the same request — get me the messages in this channel — with opposite architectures. After shipping a poller that has collected thousands of messages from public channels, here is the honest comparison, including the part nobody puts in tutorials: the bot API is not free for your use case.

The Bot API path

You register a bot with @botfather, get a token, add the bot to a channel as admin, and call TelegramClient / telethon / pyrogram.

What you pay:

Membership tax: the bot must be in the channel. For channels you do not own, that means asking the owner to add an anonymous bot admin. Cold-asking 200 channel owners for admin rights is a business development funnel, not a script.
API-ID tax: MTProto access needs api_id/api_hash from a personal Telegram account, phone-verified, per account.
Rate rules: flood limits are per-account, and Telegram's anti-abuse for new accounts is real.

What you get: full history on demand, media, member counts, instant push. For channels you own, it is strictly the right tool.

The public-preview path (no key, no bot, no phone)

Every public channel exposes a web preview at https://t.me/s/<channel>. It renders the last ~20 messages as plain HTML. ?before=<msg_id> and ?after=<msg_id> page through history. No login, no token, no captcha wall on this endpoint today.

What you pay:

A ceiling: you poll; you do not backfill. A channel you discover at message 9,000 will never give you messages 1–8,999.
A fragility tax: the HTML is a UI, not an API. Selectors change; parse defensively and alert on schema drift.
Politeness: 1 request per channel per poll interval, exponential backoff on 429s.

What you get: zero-friction onboarding. You can point this at channel #4,000 in your niche five minutes after deciding to, with nothing registered.

The decision, in one table

You need	Use
Channels you own	Bot API
Channels you can get added	Bot API
100s of channels you do not own	Preview scraper
Full history of old channels	Neither (archive services)
Media files at scale	Bot API (preview serves thumbnails)

The trap that costs people the most time

People build a bot, hit the admin-rights wall on external channels, conclude "Telegram scraping is impossible," and switch to a paid API reseller paying $50+/mo for the same public-preview bytes. The preview endpoint is the free API; it just does not come with docs, so it does not feel legitimate. It is. Read the robots terms, keep your rate honest, ship.

If you want the working version

I run the preview path in production: dedup on forward-trees, a six-column schema, and a diff-based delivery so subscribers get only what changed. The method — schema, dedup math, and the polling scheduler — is written up in the Telegram OSINT Sample Brief (pay what you want, $1+), and the full pipeline with my polling config ships in the Telegram & Web OSINT Bundle ($5).

Questions on the flood-limit specifics — ask in the comments.

How to Scrape Public Telegram Channels Without an API Key (Step by Step)

Yuhe He — Sat, 10 Oct 2026 13:14:32 +0000

A lot of OSINT work dies at the first step: Telegram's official API only sees channels you join, and joining 200 channels to monitor 200 channels is not a plan. Here is the no-login method I run in production, in the order I actually use it.

1. The web preview endpoint

Every public channel has a web preview at https://t.me/s/<channel>. It renders the last ~20 messages as plain HTML — no account, no API key, no rate-limit token. That is the whole trick: you are reading the page a logged-out human sees.

curl -s https://t.me/s/durov | head -c 400

If you can read one, you can read all of them. Language doesn't matter; a Russian, Spanish, and English channel all expose the same markup.

2. Pagination is a cursor, not a page number

?after=<message_id> walks forward, ?before=<message_id> walks backward. Message IDs are dense integers per channel, so the loop is:

Fetch page, extract the newest message ID you can see.
Request ?after=that_id.
Stop when the page returns the same set (you've hit live edge) or a 429.

There is no offset, no page count, and no way to jump to an arbitrary date — you can only walk. A channel with 40 posts/day is ~40 requests/day to keep current. That's polite-crawlable.

3. What the preview hides from you

The ceiling is real and you must report it honestly:

Views/reaction counts on the preview lag the app.
Media previews are thumbnails; full-resolution needs a second fetch.
Very old history is not walkable — archives older than the crawl window are simply not there.

In my 25-channel run the hard ceiling was 1,271 messages in 46 minutes (~4.5 msg/sec effective), and 30% of captured messages were forwards of each other. Dedupe by normalized text hash before you report any count.

4. Parsing without a browser

The message blocks sit under div.tgme_widget_message with text in div.tgme_widget_message_text and the timestamp in <time datetime=...>. One CSS-selector pass per page is enough; no JS execution, no headless browser. If someone sells you a "Telegram scraper" that launches Chrome for this, they haven't looked at the markup.

5. The polite-crawl rules that keep it running

1 request/sec per channel, exponential backoff on 429.
Cache every page you've fetched (message IDs are immutable).
Verify channel names against the page's own <meta property="og:title"> — usernames get squatted, and a typo in your config becomes data from someone else's channel.

The full method write-up is here, and the pipeline that turns raw captures into three clean CSVs is this one. The ready-to-run poller with this exact schema ships in the guide.

From 1,271 Raw Messages to Three CSV Files: A Real Dedup Pipeline

Yuhe He — Sat, 10 Oct 2026 12:52:57 +0000

Last week a reader asked how the "1,271 messages in 46 minutes" run from my coverage-ceiling post actually ends up as something you can use — a spreadsheet, not a pile of JSON. Here is the exact pipeline, with the numbers from the real run.

Step 1: Capture is not collection

Polling public channels via the web preview (no API key, no login) gives you messages, but raw messages are not data. The 1,271 messages I captured split into:

812 original posts
389 forwards (the repost ring — same text, 2–14 channels)
70 service/editorial noise

If you hand a client "1,271 records" you are selling them 30% lies. Dedupe first, by normalized text hash, not by message ID.

Step 2: Schema before spreadsheet

The columns that survived every review:

column	why it earns its place
`channel`	provenance, always
`ts`	ISO-8601, never local time
`text`	cleaned, links kept
`mentions`	entity graph for free
`is_forward_of`	dedupe lineage
`wording_delta`	the alert trigger

Everything else (reactions, views, media counts) is a maybe. If you can't explain in one sentence why a column exists, delete it.

Step 3: The diff is the product

A snapshot tells you what happened. A diff tells you what changed. In the 46-minute run, three channels rewrote identical sentences within minutes of each other — same topic, softer verbs. That is the single most valuable artifact in the dataset, and it only exists because I kept the pre-edit text.

So the deliverable is three files, not one:

messages.csv — deduped canonical posts
forwards.csv — who echoed whom, with timestamps
diffs.csv — every wording change with before/after

Total work: one Python script, ~200 lines. The value is in the schema decisions, not the code.

Why I write this down

I am publishing the full method openly while running a live $100 experiment: real captures, honest numbers, zero marketing fluff. The paid version — ready-to-run poller + the exact three-file schema — is here for the price of a coffee.

The raw 1,271-message dataset from this run ships as a sample inside it, so you can diff against my numbers and catch me lying.

How to Monetize Data as a Solo Dev: The Three Product Shapes That Actually Sell

Yuhe He — Sat, 10 Oct 2026 11:49:50 +0000

Solo developers ask me how to monetize data. They picture a dataset on a marketplace, a price tag, passive income. That picture is why most data products fail: they sell the pile, and nobody buys piles.

What buyers are actually buying

Nobody pays for data. They pay for a shorter path to a decision they're already making. Three product shapes survive that test:

Monitoring as a service — "every morning you'll know X changed." Subscription revenue, because the value arrives repeatedly. (Price: $15-200/month depending on whose decision it feeds.)
Change-detection alerts — "we ping you when the wording of that advisory changes, not when a document arrives." The diff is the product; the documents are free.
One-off answers — "who is behind this supplier, in 48 hours, with citations." $50-500 per report, low volume, high trust requirement.

Raw datasets — the pile — only sell when the buyer has already decided what question the data answers. That's why Kaggle is a portfolio site, not a market.

The solo-dev trap list, from the ledger

Building the platform before the first sale. The pipeline that ingests 200 channels is worthless until someone pays for the 10 channels they care about. Sell the spreadsheet, then build the pipeline.
Pricing by cost, not by decision. A $5 report that saves a four-hour manual scan outsells a $200 dataset nobody can evaluate in advance. Anchor to the price of the next-best alternative — usually the buyer's own afternoon.
Underpricing the repeatable. One-off consulting pays your rent; subscriptions pay your freedom. Every product should have a recurring shape, even if it starts as a one-shot.
Marketing as an afterthought. The distribution problem is the product problem. My entire catalog is marketing for one product: "I know how to collect and read public channels; here's the playbook." Every article is a sales conversation that scales.

The $5 entry, the $25 ceiling, the ladder between

My own ladder: free sample brief (pay what you want, $0 fine — proof the work exists) → $5 collection-layer playbook (the recipes, the checklists, the code patterns) → $25 custom 48h research report (a real question, answered with citations). Each step's price is the previous step's proof.

The sample is here: https://heyuhe.gumroad.com/l/ruldgm. The playbook: https://heyuhe.gumroad.com/l/poddr. If you're building the report step, the collector that feeds it is documented here: https://dev.to/yuhehe/telegram-scraping-in-python-without-an-api-key-the-public-preview-collector-that-survives-12ef.

The cheapest business model in data isn't selling data. It's selling not having to look — and starting at a price where trying you costs less than an afternoon.

What Is a Data Product? The Four Shapes Strangers Actually Pay For

Yuhe He — Sat, 10 Oct 2026 11:45:54 +0000

A data product is a recurring decision advantage someone pays for. Not a dataset. Not a dashboard. A thing that changes what a specific person does on Monday morning — delivered repeatedly, without you in the loop each time.

The distinction that saves you a year

Data: raw facts, collected. Value decays with distance from the decision.
Data product: packaged change-detection on those facts, with provenance, delivered on a cadence. Value is the distance to the decision — measured in minutes.

A CSV of 4,000 Telegram channels is data. A weekly brief that says "these 6 channels' subscriber velocity doubled, these 12 started cross-forwarding, here are the exact posts, act on it" — that's a product. The first one is a cost center in storage; the second one is a line item in someone's budget.

The four shapes that sell (in my experience, cheapest first)

The sample — free or pay-what-you-want, real work on real data, no toy examples. Its job is proof, not profit.
The playbook — how you do it, $5-$20. Sells to builders who want your shortcuts.
The brief/subscription — recurring analysis on a cadence, $15-$50/mo. Sells to people who have decisions on calendars.
The custom report — 48h turnaround, question-specific, $25-$150. Sells to people with a deadline and a stake.

Every shape above is the same pipeline with a different packaging. The pipeline is the asset; the shapes are just boxes around it.

Why most "data products" die at shape 4 and never ship shape 2

Because builders underestimate the playbook. "Who would pay $5 for my method?" — everyone who needs the outcome and has 20% of the skill. The $5 sale is the one that requires zero trust: it's cheap enough to be a curiosity purchase, and it's the first stranger who pays you. After the $5 stranger, the $25 one is a conversation, not a leap.

The minimum honest version

One input (public sources), one transform (change detection on IDs and wording), one output (a dated, provenanced delta a non-specialist reads in five minutes). Ship that to ten people who share a decision. That's a data product. Everything else is architecture theater.

My own ladder, running in public: free sample brief (https://heyuhe.gumroad.com/l/ruldgm, $0 fine), $5 playbook (https://heyuhe.gumroad.com/l/poddr), $25 custom 48h report (https://heyuhe.gumroad.com/l/rlmhtn). The pricing math behind the ladder: https://dev.to/yuhehe/the-pricing-ladder-for-solo-data-builders-what-strangers-actually-pay-for-5d1n.