DEV Community: Mr Zack

Your "only new results" mode has never been tested on the second run

Mr Zack — Fri, 18 Sep 2026 19:48:56 +0000

Half of the scrapers we sell on Apify Store have a monitor switch: onlyNew: true. Schedule the actor, and every run returns only the rows that were not there last time. Class-action settlements that just opened, Kalshi markets that just listed, startups that just hit the marketplace. It is the one feature that turns a "try once" user into a daily schedule.

We shipped it to six actors in one afternoon. Unit tests green, cloud verification green: run 1 returned 136 rows, run 2 returned 0. Done.

Then we tried to write a regression test for it and realised our whole test harness could not express the feature.

The harness only knew about run 1

Every test we had followed the same shape: fresh storage directory, write INPUT.json, run src/main.js, read the dataset, assert. One run, one dataset. There was no way to say "now run it again against the same key-value store and check that the second dataset is empty."

That is not a small gap. The correctness of onlyNew lives entirely in the second run. Run 1 is indistinguishable from onlyNew: false. If the seen-set is never persisted, or is persisted under the wrong key, or is loaded but never applied, run 1 still looks perfect. The bug only exists in a state your harness never reaches.

So we added one field to the spec format:

{
  "name": "onlyNew run-2 → 0 rows",
  "input": { "onlyNew": true },
  "runs": 2,
  "minFirstRows": 50,
  "minRows": 0,
  "maxRows": 0
}

runs: 2 runs the actor twice in the same storage directory. Between runs the dataset is wiped but the key-value stores are kept — exactly what happens between two scheduled runs on the platform. minFirstRows guards the first run so a broken scraper that returns nothing twice does not pass as "correctly returned nothing new."

What the second run caught on day one

The new spec found no bug in the six actors. It found two bugs in how we thought about the feature, which is the more useful kind.

1. "0 new rows" depends on the cap. Our first draft of the spec inherited a base input with maxItems: 50 for the class-action actor. Run 1 returned 50 rows, run 2 returned 50 rows, and the check went red. Not a bug: the board has 194 settlements, the seen-filter runs before the cap (so unsent rows stay "new"), and run 2 correctly delivered the next 50. The golden case "second run is empty" is only true when the first run drained the board. If your cap is smaller than the source, the honest assertion is "run 2 is disjoint from run 1," not "run 2 is empty." We wrote the spec inputs so the board fits, and documented the disjoint case for later.

2. Some actors cannot be tested twice inside a sandbox call. The startup-marketplace scraper walks a 565-URL sitemap; one run is 50 seconds, two runs plus setup is over the time limit of the shell we get. We verified it by hand (409 → 0) and left a note in the spec. A test you cannot run automatically is a test that will silently stop being run; better to know that today than to discover it in a month.

The rule we took from it

If a feature's behaviour depends on state from a previous run — dedupe memory, trend baselines, "since last time" deltas — then the feature has no coverage until your harness can run the actor twice. Not "weak coverage." None. Run 1 exercises the code path that writes the memory, and that path is invisible from the dataset.

We now have seven runs: 2 specs across the catalog (kalshi, YouTube Shorts, MunchEye, class actions, the clipping-campaign aggregator, Microns, Skool). They cost a few seconds each and run with every maintenance pass.

If you sell a scraper with a monitor mode, go check when you last looked at its second run.

We maintain 45+ scrapers on Apify Store. Previous post in this series: Your {} smoke test passes. Your users still crash.

Your `{}` smoke test passes. Your users still crash.

Mr Zack — Thu, 17 Sep 2026 19:51:55 +0000

We run 45 scrapers on Apify Store. Every one of them gets an automated smoke test every day: start the actor with an empty input {}, check that it finishes and returns rows. Green across the board for weeks.

Meanwhile, one of our paid actors — the Amazon one — was failing for real users. Four failed runs in one day, from people who were paying for it.

The crash was a one-line bug that no {} test can ever catch. Here's the shape of it, because you probably have the same one.

The bug

The actor has several modes: keyword search, a list of ASINs, or "Best Sellers" rankings. If the user gives none of them, we fall back to a demo search so the run isn't empty:

const keywords = list(input.keywords);
const asins    = list(input.asins);

if (!keywords.length && !asins.length && !bestSellersCategories.length) {
  keywords.push('air fryer'); // demo
}

// ...eight lines later:
const bestSellersCategories = list(input.bestSellersCategories);

bestSellersCategories is a const declared below the line that reads it. In JavaScript that isn't undefined — it's a ReferenceError (the temporal dead zone). But && short-circuits: as long as keywords has something in it, the bad expression is never evaluated.

So:

{} → the platform injects the schema default keywords: ["air fryer"] → short-circuit → fine.
Our unit tests → parser only, never boot the actor → fine.
Our "all paid flags on" test → keywords present → fine.
A user who opens the Best Sellers mode, deletes the default keyword, and picks a category → keywords: [] → the expression runs → crash on line 1 of the run.

The mode we put in the title of the actor was the one that crashed.

Why every gate was green

Each of our checks answered a question that wasn't the question:

gate	question it answers
`{}` smoke test	"does the default path work?"
unit tests	"does the parser work?"
all-flags-on run	"do the paid options work on top of the default path?"
`node --check`	"is it syntactically valid?" (TDZ is valid syntax)

None of them asked: "does each alternative input mode work when the primary mode is explicitly empty?"

That's not a corner case. It's what a real user does the moment they want the second feature instead of the first one.

Two fixes, one of them permanent

Move the demo line below the declarations. Thirty seconds.
Make sure it can't come back:

A static gate for use-before-declaration. ESLint's no-use-before-define catches this, but we wanted something that runs against the deployed entry point of all 45 actors in one command, with zero config per actor. Forty lines with acorn: parse the module, list top-level let/const/class by statement index, walk every top-level statement (skipping function bodies — they run later) and flag any identifier that resolves to a declaration further down. It found exactly the two amazon lines and nothing else across 160 files.

A third smoke-test scenario. Our user-path audit ran A ({}) and B (paid flags on). It now runs C: for every actor with more than one input mode, an input that empties the primary field and fills the alternative one — {"keywords": [], "bestSellersCategories": ["electronics"]}, {"queries": [], "sources": ["reuters.com"]}, {"videoUrls": [], "channelUrls": [...]}, and so on.

Scenario C paid for itself within the hour. Running it across the catalogue turned up a second, quieter bug in a different actor: a Google News scraper where sources without a query fell back to the top-stories feed and then filtered it against the source list — zero articles, status "Succeeded". Same family: the alternative mode was never exercised on its own.

The takeaway

If your actor (or CLI, or API) has more than one way to specify what to fetch, your smoke tests need one run per way, with the other ways explicitly empty — not absent. Absent gets defaults injected. Empty is what users send.

And add the TDZ gate. It's the cheapest test you'll ever write: it costs nothing to run and it catches the one class of crash that "it works on {}" is structurally unable to see.

The gate and the scenario runner live in the tooling behind our Apify Store actors. Previous post in this series: The scraper said "Succeeded". The column was dead for weeks.

The scraper said 'Succeeded'. The column was dead for weeks.

Mr Zack — Wed, 16 Sep 2026 23:31:29 +0000

Here's a bug class that never trips an alert: the run succeeds, the row count is right, the schema validates — and one numeric column has been 0 on every single row since the source quietly changed its API.

We hit it twice in one morning while auditing our Apify actors.

Case 1: Kalshi `liquidity_dollars`

Our Kalshi weather markets scraper maps liquidity_dollars from Kalshi's /markets endpoint into a liquidity column. At some point Kalshi started returning "liquidity_dollars": "0.0000" for every market — including NFL games with $800k of volume. Nothing failed. 336 rows per run, all green gates, liquidity = 0 everywhere.

The fix wasn't "remove the column". The same payload still carries yes_bid_size_fp, yes_ask_size_fp, yes_bid_dollars and yes_ask_dollars, which is enough to compute resting collateral at the top of the book:

liquidity ≈ yesBidSize × yesBid + yesAskSize × (1 − yesAsk)

Zero extra requests, and the column is alive again. We also added a liquiditySource field (kalshi | topOfBook) so nobody has to guess where the number came from.

Case 2: Kick `likes`

Our Kick clips scraper had likes = 0 on 112/112 rows. We checked the raw API (list and detail endpoints, 41k-view clips): Kick itself now reports 0 likes for everything. Nothing to compute from — so the honest fix is documentation: the field stays for compatibility, the README says "rank by views / viewsPerHour instead".

The gate that catches both

We added one rule to our post-run dataset check, and it fired on the very first run:

A numeric column that is non-null on ≥ 50 rows and equals 0 on 100 % of them is a warning.

Fifteen lines of code. It's cheap because "all zeros" is almost never what a real metric looks like; it's valuable because "succeeded + right row count" is exactly the signal that lets dead columns hide.

If you maintain scrapers, add it today. Skip flags/counters that are legitimately zero (isX, position, errors) and let everything else scream.

Bonus: "cap before filter"

Same audit, different actor — an AliExpress search scraper applied maxProducts first and minSold after, so minSold: 500, maxProducts: 60 could return 16 rows. Filtering while paging (and enforcing AliExpress' own leaky "Choice" filter client-side) turned that into a full 60. Third time we've found this pattern in our own code; it's now a standing test case for every actor with a cap and a filter.

Google News RSS quietly supports after:, before:, site: and -word — I measured what each one actually does

Mr Zack — Sun, 13 Sep 2026 19:47:40 +0000

Google News has no official API, but its RSS search endpoint (news.google.com/rss/search?q=…) accepts the same operators as the web search box. I could not find a single place that documents which ones the feed honours, so I measured them (14 Sep 2026, US/en edition, query nvidia).

Setup

Each request fetched https://news.google.com/rss/search?q=<query>&hl=en-US&gl=US&ceid=US:en with a browser user-agent, and I counted <item> elements and read pubDate / <source url>.

query	items	what came back
`nvidia`	100	mixed sources, mixed dates
`nvidia when:7d`	100	oldest `pubDate` = 7 days ago
`nvidia after:2026-09-10 before:2026-09-12`	100	every `pubDate` in 10–12 Sep (the `before:` day is included)
`nvidia site:reuters.com`	100	only `reuters.com` sources — going back to 2005
`nvidia site:cnbc.com`	100	only `cnbc.com`
`nvidia site:cnbc.com after:2026-09-10 before:2026-09-12`	37	date window narrows the same source
`nvidia -stock`	100	zero titles contain "stock"
`intitle:nvidia`	100	100/100 headlines contain "nvidia"
`nvidia site:cnbc.com -stock`	58	vs 100 without the exclusion
`nvidia site:cnbc.com after:2026-09-10 before:2026-09-12 -stock`	1	(see below)

Four things worth knowing

1. 100 is a hard ceiling per feed. No num=, no pagination. The only way to get more is more feeds — which is exactly what site: gives you: one feed per publisher returns up to 100 per publisher. Ten sources = up to 1,000 articles for one keyword.

2. after:/before: are real date filters. The window is inclusive on both ends (UTC days). They combine with site:, and they override the relative when: window if you send both — so pick one.

3. -word matches the article body, not the headline. nvidia site:cnbc.com in a 3-day window has 37 articles, 23 of them with "stock" in the headline. Add -stock and 36 disappear — the 13 extra ones matched in the body. Google is excluding every article that mentions stocks anywhere. If you want "not in the headline", you have to filter the feed yourself after fetching.

4. Topic and location feeds accept no operators at all. headlines/section/topic/TECHNOLOGY and headlines/section/geo/Chicago ignore everything in the query string except the locale, so date/source/word filters there must be post-fetch.

The links are not the links

Every <link> is a news.google.com/rss/articles/CBMi… redirect. Decoding it to the publisher URL takes one extra request per article to read a signature from the redirect page, then a batched batchexecute call — and from a datacenter IP the signature page sometimes comes back empty for a few minutes (HTTP 200, no signature). Three runs in a row got 0/11 signatures from the same IP; two minutes later the same input resolved 11/11. Retrying immediately from the same IP does not help — wait or switch IP.

Where this landed

I maintain a Google News actor on Apify that already resolved publisher URLs and pulled article text; today it gained publishedAfter / publishedBefore, sources (one feed per source), excludeSources and excludeWords, with every filter also applied post-fetch so topic and location feeds honour them too. Rows dropped by a filter are never billed.

→ https://apify.com/tactful_anvil/google-news-scraper

If you have measured other operators (inurl:, OR, allintext:), I'd love to add them to the table.

I scraped the top 1,000 Skool communities. 85% are paid, the median charges $34, and 10 communities hold 30% of the money.

Mr Zack — Sun, 13 Sep 2026 09:44:09 +0000

Skool's Discovery page ranks communities, but it never tells you what the market looks like as a whole. So I pulled the global top 1,000 (34 pages × 30) on 13 September 2026 — rank, members, price, billing interval, creation date — and ran the numbers. Snapshot, one day, public data only.

Pricing: free is the exception

tier	communities
free	154 (15 %)
$1–9 / month	141
$10–29 / month	227
$30–49 / month	161
$50–99 / month	184
$100+ / month	66
one-time price	67

846 of 1,000 are paid. Median monthly price: $34. In the top 100 the paid share is 95 %; at ranks 901–1000 it's still 85 %. Free communities are bigger on average (median 374 members vs 226 for paid) but they don't dominate the ranking — Skool's algorithm clearly weighs engagement over headcount.

Size: the median community is small

Median members: 241. Median age: 314 days. Median lifetime growth: under 1 member per day. The long tail is very long; a community with 1,000 members is already in the top quarter.

Money: brutal concentration

Multiply members by monthly price and you get an upper bound on gross MRR (not every member pays list price — think of it as "money on the table"). Across the 779 monthly-billed communities that's ≈ $22.4 M / month. Of that:

the top 10 communities hold 30 %,
the top 10 % hold 69 %,
the median community's potential is $7.3 k / month.

The biggest three by this measure: a 71 k-member community at $29, a 44 k one at $27, and a 164 k one at $7 — proof that low price × huge audience beats premium pricing at the top.

Growth: who is actually winning

Lifetime growth (members ÷ age) crowns the giants, but it hides momentum: a 45-day-old trading community is adding 115 members/day, faster than almost anything with a 500-day history. Age-normalised growth is the fairest way to spot rising communities, and it needs two observations, not one — which is why the scraper now has a trackTrends mode: run it daily and each row gets membersPerDay, rankChange, priceChangeUsd and a growthSignal (surging / growing / flat / shrinking / new).

Method

Public Skool Discovery JSON, HTTP only, no login. 1,000 rows, ~5 seconds. monthlyRevenuePotentialUsd = members × monthly price (upper bound). The scraper is on the Apify Store — https://apify.com/tactful_anvil/skool-communities-scraper — with keyword search, category/price filters, owner enrichment (public MRR badge) and the growth tracker above. If it's useful, a bookmark or review there genuinely helps a small developer.

LinkedIn's public job search silently ignores "contract", "remote" and "entry level". I measured it, then fixed it.

Mr Zack — Sat, 12 Sep 2026 19:37:30 +0000

If you scrape LinkedIn jobs without logging in, you probably use the guest search endpoint (/jobs-guest/jobs/api/seeMoreJobPostings/search). It takes the same query parameters the website does: f_JT for job type, f_E for experience level, f_WT for remote, sortBy=DD for newest first.

Since around 11 September 2026 it accepts all of them — and returns exactly the same result set no matter what you send.

The measurement

Query: keywords=data analyst, location=United States. Same request, five variants, first 50 job IDs compared with the unfiltered search (re-run 13 Sep 2026):

variant	overlap with unfiltered
`f_JT=C` (contract)	50 / 50
`f_E=2` (entry level)	50 / 50
`f_WT=2` (remote)	50 / 50
`sortBy=DD`	50 / 50
`f_TPR=r86400` (past 24h)	4 / 50 ← still works

Only f_TPR (posted within), f_AL (Easy Apply) and f_EA change the response. No error, no warning: the server just serves the default page. A scraper that trusts the parameters keeps "succeeding" with wrong data, which is the worst kind of failure.

Why "filter after the fact" isn't enough

The obvious fix is to open each job's detail page (which still has the criteria block: seniority, employment type) and drop what doesn't match. I shipped that first. Then a routine parameter audit found the next bug: with maxJobs: 20 and jobTypes: ["contract"] the run returned 0 rows. Not because there are no contract data-analyst jobs, but because only 20 cards were scanned before filtering, and 0 of those 20 were contracts. In my sample it took 49 detail pages to find the first one.

So the loop is now: page → detail-check → keep matches → keep paging until maxJobs matches or a scan cap is hit (default 5× maxJobs, min 50, max 500). Non-matching jobs cost nothing. A "remote data analyst, 8 jobs" run scans ~30 postings; "contract" scans a lot more and tells you so.

Takeaways for any scraper

Parameters that are accepted are not parameters that are honoured. Test the effect (compare IDs), not the status code.
If a filter has to be enforced client-side, the scan budget has to be decoupled from the result budget, or rare filters return nothing.
Put the measurement in a gate you run every day. Mine is a JSON spec: input variant → expected effect (all rows have employmentType ~ /contract/, ≥8 rows, first 5 ids differ from base). It caught the 0-row bug the day after I introduced it.

The scraper (salary parsing, applicant counts, monitor mode, company hiring signals, and now honest filters) is on the Apify Store: https://apify.com/tactful_anvil/linkedin-jobs-full-details-scraper — if it saves you an afternoon, a bookmark or a review there helps more than you'd think.

I pulled the payout ledger of 658 clipping campaigns. The median one pays 12% of its advertised CPM.

Mr Zack — Fri, 11 Sep 2026 19:38:34 +0000

If you clip for Whop Content Rewards campaigns, you've seen the pitch: "$1 per 1,000 views", "$2 per 1K", budget $100,000. What the board never shows next to that number is how much of it has actually been paid to creators.

It turns out the site's own JSON API does publish that — per campaign: paidOutCents, approvedSubmissionCount, totalViews, and the exact listedAt timestamp. So this week I made my scraper emit those on every row for free (no per-campaign detail fetch) and ran it over the full board: 658 active campaigns, $3.52M of advertised budget.

The numbers (12 Sep 2026, single snapshot)

Paid out so far: $510,929 — 14.5% of advertised budget. "Budget spent" (which includes pending/held amounts) is $1.54M, three times the paid figure.
369 of 658 campaigns (56%) have paid $0 to anyone. 254 of those already show more than 100,000 submitted views.
For the 289 campaigns that have paid something, I computed effectiveRewardPerThousandViews = paidOut / totalViews × 1000 and divided by the headline CPM:
- median: 0.12 — the typical campaign has paid 12 cents for every dollar its CPM promises per view.
- 25th percentile 0.04, 75th percentile 0.28.
- 72% pay under a quarter of headline. 3% pay 80% or more.

That doesn't mean 88% of views are stolen. Views get rejected (rules, watermarks, reposts), payouts lag approvals, and per-submission caps clip viral outliers. But it does mean the CPM on the card is a ceiling, and the floor is usually much closer to the ground.

Same $1/1K, very different campaigns

campaign	headline	paid out	views	effective /1K
Roblox game (ForgeGUI)	$1.00	$96,403	252M	$0.38
French creator (Yomi Denzel)	$1.00	$47,059	1,357M	$0.035
Backyard Breaks	$2.00	$20,123	21M	$0.97
Daimon X Syndicate	$1.00	$18,463	31M	$0.59

Two campaigns, same $1 CPM, 10× difference in what a thousand views has actually been worth. If you're deciding where to spend a week of clipping, that column matters more than the headline.

What I'd do with it as a clipper

Filter to paidOutUsd > 0 — someone has already been paid, so the pipeline works.
Sort by effectiveRewardPerThousandViews and ignore anything under ~0.25× headline unless the budget is huge and young.
Use listedAt (exact now, not "2w") to catch campaigns under 7 days old — 92 on the board today — where you're competing with fewer creators.

How

Everything above is one run of my Apify Actor (Whop Content Rewards Scraper, pay-per-result, default input, ~20 seconds). New this week: listedAt, campaignCreatedAt, approvedSubmissions, paidOutUsd, effectiveRewardPerThousandViews on every row; listedAfter now works without the enrichment mode; and a bug I found while auditing it — category had been rendering as [object Object] for a day because the API started shipping categories as objects. The scraper has two more regression gates now, because "run succeeded" told me nothing about either.

If you use it, a review on the Store page is the thing that actually helps an indie actor get found. Method notes are in the README.

I surveyed 8,758 Apify actors before building anything. Then I shipped three scrapers in one day.

Mr Zack — Sun, 06 Sep 2026 19:47:01 +0000

For a month I built scrapers the way most indie devs do: pick a site I find interesting, build it, publish, hope. Thirty-seven actors later the honest scorecard was: one real earner (a Whop clipping-campaign scraper), a handful with a single paying user each, and a long tail of "zero starts".

So I stopped guessing and pulled the market instead.

Step 1: pull the whole store

Apify's public Store API returns every actor with its 30-day run stats, user counts, rating and price. I pulled the top 8,758 by monthly users (54k listed in total, the rest are dead) and bucketed them by keyword cluster. Monthly active users per cluster, roughly:

Cluster	MAU	Cluster	MAU
Instagram	133k	Ads (Google/Meta)	35k
LinkedIn	114k	YouTube	34k
Jobs	59k	Reddit	16k
Google Maps	57k	Leads / contacts	12k
TikTok	50k	Transcripts	11k

Then the fun part: inside each cluster, is the leader hated (≥250 MAU but ≤3.6★), expensive (≥$10 per 1,000 results) or broken (≥35% failed runs)? Those are the three gaps a newcomer can actually take.

Some examples that fell out of the data:

Airbnb: 1,900 MAU across 48 actors, leader at 754 MAU with 2.5★.
YouTube transcripts: 11k MAU, leader charges $10 / 1,000 and sits at 3.7★.
Google Trends: leader has a 37% failure rate.
LinkedIn jobs: 47k MAU, leader is fine ($2 / 1,000, 16k MAU) — the gap is depth, not quality.

Step 2: probe before building

A gap is worthless if the site blocks datacenter IPs. So each candidate got a 10-minute HTTP probe from a plain server: fetch the listing page, fetch a detail page, hit whatever JSON the front-end calls, count 403s.

Airbnb, LinkedIn's guest job API and YouTube's watch page all answered 200. Google Trends (429), eBay (403 even with browser fingerprints), TikTok Shop (login wall) and Google Jobs did not. They went on a "needs residential relay" list instead of the build list.

One thing I learned the hard way: my sandbox turned out to have a residential IP, so "passed the datacenter probe" only became true after the first cloud run. Verify from the platform you deploy to.

Step 3: build the three that passed

Same day, three actors, each with unit tests against fixtures, a live gate script (10 checks against the real site) and one verification run in the cloud:

Airbnb Market Intel Scraper — search results plus, per listing, the 12-month availability calendar. From that you get occupancy and estimated revenue, which is what investors pay AirDNA for. Live gate today for Austin: median nightly price $166, p25 $130, p75 $222; a sample listing at 57% 30-day occupancy, ~$4.8k estimated monthly revenue. $4 / 1,000 listings, $6 with calendar.

LinkedIn Jobs Scraper — Full Description, Salary & Applicants — the guest API gives the card; the detail page gives description, criteria, salary when posted and applicant count ("Over 200 applicants" on two of three sampled data-analyst roles today). It also emits free "company signal" rows: employers ranked by open roles, which is a lead list for recruiters. $1.60 / 1,000 detailed jobs vs the leader's $2.

YouTube Transcript Scraper — the one that failed the cloud test. YouTube returns the caption track list but the timedtext endpoint answers 200 with 0 bytes from datacenter IPs. Fix: the actor tries direct first, then falls back to a tiny relay running on a Mac mini behind a Tailscale funnel. Verification run: 7 of 8 videos transcribed, 38 seconds, $0.003. $6 / 1,000 transcripts vs $10.

What the survey changed

The old approach optimized for "can I scrape this?". The survey optimizes for "will anyone start this?" — demand first, gap second, feasibility third, build last. One day in: zero paying users, which is exactly what day one looks like. I'll post the 8-week numbers either way.

The survey script is ~150 lines of Python against one public endpoint. If you publish on any marketplace with a public stats API, do this before your next build.

All three are HTTP-only, pay per result, and callable as MCP tools. If you try one, a review on the store page is the only signal other buyers have — and open an issue if a field is wrong; I fix those fast.

211 US class action settlements are open right now, 155 need no receipt. I scrape and score the whole board in one HTTP request.

Mr Zack — Sat, 05 Sep 2026 20:08:15 +0000

Most people find out about a class action settlement the way I used to: a friend forwards a link two days after the claim deadline. The board that lists all of them — ClassAction.org — is public, updates daily, and nobody I know checks it. So I turned it into a dataset.

This morning's run: 211 open settlements, 155 claimable with no proof of purchase, 118 of those pay cash (not vouchers or credits), 43 added recently, 31 closing in the next few days. The top of the board right now is a cluster of data-breach settlements paying $50–$5,000 with no receipt required and 40–56 days left.

Why a scraper and not a bookmark

The board is a wall of cards. Each card has a headline, a dollar range somewhere in the text, a deadline, and a "proof of purchase required?" line buried in the details. Comparing 200 of them by hand is the kind of work that makes you give up after page one — which is exactly why so many settlements end with unclaimed money.

The scraper does three things per card:

Parse the messy bits. Payout text like "Up to $5,000", "$20 - $4,480" or "Varies" becomes payoutMinUsd / payoutMaxUsd (nulls stay null — no fake zeros). Deadlines become deadlineDate + daysLeft + an urgency bucket.
Flag what matters. proofRequired (tri-state — unknown stays null), payoutKind (cash vs non-cash vouchers/credits), and the combination people actually want: isNoProofCash.
Score it. claimScore 0–100 = payout size (log-scaled, 40 pts) + no proof needed (30) + comfortable time window (15) + freshness (15). Sort by it and the "file this tonight" list is the top 20 rows.

Every row also carries the official claim-site URL (not the ClassAction.org page — the actual administrator's site where you file), so the workflow is: run → filter isNoProofCash → click.

The technical part

Nothing exotic. The board is server-rendered HTML — one GET, one parse, ~210 rows, done in a few seconds with no browser and no proxies. The interesting work is in the normalizer:

// payout text → numbers, without inventing data
const m = text.match(/\$([\d,]+)(?:\s*[-–]\s*\$([\d,]+))?/);
const min = m ? Number(m[1].replace(/,/g, '')) : null;
const max = m && m[2] ? Number(m[2].replace(/,/g, '')) : (/up to/i.test(text) ? min : null);

and in the gates I run before every deploy: 13 assertions against the live page (row count within range, ≥60% rows with a parsed payout, no duplicate settlement IDs, daysLeft never negative for open cases, claimScore in [0,100] or null). When ClassAction.org changes its markup — and it will — the gate turns red before a user sees an empty dataset. I learned that lesson the hard way on another Actor that "succeeded" for two days while returning 42 rows instead of 500.

Running it

It's an Apify Actor: Class Action Settlements Scraper | Claim Deadlines & Payouts. Default input {} returns the full board sorted by claimScore. The inputs I actually use:

{ "noProofOnly": true, "cashOnly": true, "sortBy": "claimScore" }

for the weekly "worth filing" list, and

{ "maxDaysLeft": 10, "noProofOnly": true }

on a daily schedule with a Slack/email integration so cash claims closing soon don't slip past. Pay per result — a filtered run is a few cents. It's also MCP-ready if you'd rather have an agent ask "any new no-proof cash settlements this week?" and get a table back.

Honest limitations

It reads one board (ClassAction.org). Settlements that never make it there won't appear.
"No proof required" is what the settlement says; administrators can still ask for an attestation. Always read the official claim site before filing.
This is a data tool, not legal advice.

If you build on it, the changelog and a "found this useful?" section are in the README — a bookmark or a short review on the Store page is what tells me which fields to add next. Issues tab gets answered within 24 hours.

Related Actors from the same series: Bulk Email Verifier, Contact Details Scraper, Microns Startup Listings Scraper.

Kalshi's API is flooded with 8,000+ zero-volume parlay markets. If your scraper reads /markets, it's been empty since Sept 4.

Mr Zack — Fri, 04 Sep 2026 19:46:50 +0000

Yesterday one of my Apify Actors — a cross-platform "biggest movers" feed for Polymarket and Kalshi — ran, succeeded, and delivered 100 rows. Every single row was Polymarket. The Kalshi half of the product had silently gone to zero, and nothing in the run status said so.

Here's what happened, how I caught it, and the one query parameter that fixes it.

What changed on Kalshi

GET https://api.elections.kalshi.com/trade-api/v2/markets?status=open&limit=1000 used to return a sensible mix of open markets. Since September 4, 2026 the first pages look like this:

page 1: 1000 markets, 1000 multivariate, 0 with 24h volume
page 2: 1000 markets, 1000 multivariate, 0 with 24h volume
...
page 8: 1000 markets, 1000 multivariate, 0 with 24h volume

8,000 markets in a row, all with a ticker like KXMVECROSSCATEGORY-SHARD1-S2026…-FE61…, a mve_collection_ticker field, a 26-leg mve_selected_legs array (26 college-football moneylines chained together), volume_24h_fp: "0.00" and last_price_dollars: "0.0000". They are auto-generated multivariate ("parlay") combinations. They're valid markets, they're just not something anyone is trading, and there are thousands of them created every day.

My Actor's Kalshi loop scanned "up to 4× the requested market count" — 800 markets — and stopped. All 800 were parlay shards. Zero Kalshi rows. The Polymarket leg still worked, so the dataset was non-empty, so the run was SUCCEEDED. This is the failure mode I'm most afraid of: success-rate stays 100% while the output quietly loses half its value.

How I caught it

Not from the run status. I run a free local canary for every Actor before I look at the cloud: the production parser hit the live source from my own machine, with gates on counts and field coverage, not just exit codes. Yesterday's gate for this Actor:

PASS polymarket >=150 markets 200
FAIL kalshi >=50 markets w/ 24h vol 0 of 4000 scanned
PASS default filters -> 100 rows 100
FAIL both platforms in top 100 {"polymarket":100}

Two red lines. If I had only checked "did it succeed / did it return 100 rows", I'd have seen nothing.

The fix: `mve_filter=exclude`

Kalshi's API has an (under-documented) filter for exactly this:

GET /trade-api/v2/markets?status=open&limit=1000&mve_filter=exclude

With it, page 1 goes from 0 to 45 markets with 24-hour volume, and by page 4 you have ~600 real, actively traded markets. Ten pages (10k markets) give 2,588 open Kalshi markets with non-zero 24h volume — that's the real universe.

I also made two defensive changes so this can't recur silently:

Skip anything with mve_collection_ticker or mve_selected_legs in the normalizer, regardless of what the API returns.
Paginate by "useful markets collected", not "raw markets scanned" — keep pulling pages (cap 10) until I have at least 2× the requested top-N with volume > 0.

After the fix, the same default run returns 51 Polymarket + 49 Kalshi movers in the top 100, and the Kalshi log line says 618 markets with 24h volume fetched instead of nothing.

Note the /events?with_nested_markets=true endpoint is not flooded (0 MVE events on page 1), so if you're already on events you're fine. It's specifically /markets.

If you scrape Kalshi, check your last few days

Any pipeline built on /markets?status=open that either (a) stops after N pages or (b) filters on volume_24h > 0 after fetching, has most likely been returning a near-empty Kalshi set since Sept 4. Symptoms: dashboards with only Polymarket, arbitrage scanners with zero Kalshi pairs, "market count" charts falling off a cliff. Add mve_filter=exclude and re-run.

The Actor

The fixed feed is live: Prediction Market Movers | Polymarket & Kalshi Catalyst Feed — one ranked list of the biggest 24h odds swings, volume spikes and closing-soon markets across both platforms, each with a 0–100 catalyst score and signals like BIG_SWING, VOLUME_SPIKE, CLOSING_SOON, FAST_MOVER_1H. HTTP-only, public APIs, no keys; you pay per ranked row (~$0.01), and a default run costs about a cent. It also works as an MCP tool if you want an agent to ask "what moved on prediction markets today?"

Sibling Actors if you want the rest of the stack: Polymarket ↔ Kalshi Arbitrage Scanner, Polymarket Top Traders, Kalshi Weather Markets.

If this saved you a debugging session, a ⭐ or bookmark on the Actor page helps a lot — and if you've seen other Kalshi API changes this week, tell me in the comments so I can add gates for them.

I pulled the open positions of Polymarket's 25 most profitable traders. $7.5M is sitting in just 5 bets.

Mr Zack — Thu, 03 Sep 2026 19:33:06 +0000

Polymarket publishes a leaderboard. It also publishes every wallet's positions. Nobody joins the two, so I did — with a scraper that hits the public data-api (no login, no key) and writes one flat row per whale position.

Here is what the top 25 traders by 30-day PnL were holding this morning (September 4, 2026).

The headline numbers

25 traders, $2.03M combined 30-day profit. Median $61.5k. #1 (Sassy-Bucket) made $206k on only $203k of volume — a ~100% return; #3 (BrotherObama) needed $548k of volume for $176k.
318 open positions worth $7.53M, sitting on $980k of unrealized profit. 188 of 318 (59%) are currently green.
Average entry price 0.55, average current price 0.60. They buy at coin-flip odds and hold while the market drifts their way.
Median time to resolution: 60 days. Only 28% of positions resolve within a week. These are not scalpers.

Where the money actually is

Category	Positions	Current value
Crypto price levels ("Will BTC dip to $X by Dec 31?")	69	$4.50M
Macro (Fed rate decisions)	16	$1.11M
Sports (MLB O/U, US Open tennis)	44	$0.52M
Politics / legislation	41	$0.17M
Everything else	148	$1.24M

One wallet (JnStrtPrdctnMrkts) owns the five largest positions on the board — all of them "No" on Bitcoin/Ethereum dip markets: "ETH dips to $1,500 by Dec 31" No at $865k (entered 0.62, now 0.86, +$242k), "BTC dips to $55k" No at $589k (+$209k), "$50k" No at $552k, "$45k" No at $391k, "ETH $1,250" No at $370k. That is a $2.8M ladder betting that crypto simply does not crash this year, bought when the market priced the crash at 25–45%.

The most crowded trades

Markets where several whales independently hold the same side are the closest thing to a consensus signal:

"No change in Fed rates after the September 2026 meeting" — 5 of 25 whales, all Yes, $749k. Plus 3 whales on "No" to a 25 bps hike ($309k). The smart-money book is unanimous: nothing happens in September.
"Clarity Act (H.R.3633) signed into law in 2026?" — 3 whales, all Yes, $108k.
Sports are where they disagree: Giants–Pirates O/U 8.5 has 2 whales Over and 2 Under. Struff–Cerundolo at the US Open: 3 on Cerundolo, 2 on Struff. Sports positions are small ($5–30k) and short-dated — this looks like edge-hunting, not conviction.

What this is useful for

Copy-trading feeds: schedule the scraper hourly, diff conditionId + wallet against the previous run, and you get "whale opened / closed a position" alerts.
Sentiment on macro & crypto: whale positioning on Fed and BTC-level markets moves before the retail price does; this is the cheapest way to watch it.
Research: the avgPrice vs curPrice columns give you each whale's entry, so you can measure who's early and who's chasing.

How it's built

Two public endpoints: GET /v1/leaderboard?window=1m&rankType=pnl&limit=25 for the roster and GET /positions?user=<wallet>&limit=100&sortBy=CURRENT per wallet. Pure HTTP, ~25 requests, 10–20 seconds for the whole board. Output per row: rank, wallet, trader name, 30-day PnL and volume, market, side, shares, avg vs current price, current value, cash PnL, % PnL, end date, and a direct market URL. A watchlist mode accepts your own wallet list, and only position rows are charged.

It runs on Apify as Polymarket Top Traders Scraper | Whale Positions & Copy Feed — schedule it, pipe it to Sheets/Slack, or call it from an MCP client. Siblings if you want the full prediction-market stack: Polymarket Kalshi Arbitrage Scanner (same market, two venues, price gap) and Prediction Market Movers (biggest 24h probability swings on both platforms).

If you run it and it saves you a spreadsheet afternoon, a review or bookmark on the Apify page helps more than you'd think.

Not financial advice. Data is a snapshot; whales close positions too.

I scraped all 545 startups for sale on Microns. Sellers ask 4.3x revenue. Buyers pay 3.3x.

Mr Zack — Wed, 02 Sep 2026 19:44:28 +0000

Every micro-acquisition thread ends with the same question: what multiple is fair for a tiny SaaS? People answer from memory. I'd rather answer from the board.

So I pulled every listing on Microns — the marketplace where indie founders sell $1K–$100K micro-startups — as of September 3, 2026. That's 545 listings: 390 still for sale, 155 marked sold. Here's what the numbers say.

1. The ask/clear gap is a full turn of revenue

	Listings with revenue	Median asking price	Median price ÷ ARR
Available (for sale now)	374	$10,750	4.26x
Sold (market comps)	136	$3,000	3.33x

Sellers ask 4.3x annual revenue. What actually clears is 3.3x. If you're buying, the sold column is your anchor, not the listing page. If you're selling, pricing at 5x+ mostly means you'll still be listed in four months — median time on board for available listings is 110 days.

And it's not a few outliers dragging the ask up: 181 of the 390 available listings (46%) are priced above 5x revenue. Only 27 of 136 sold listings (20%) cleared at more than 10x, and those are the tiny-revenue cases where a $3K price on $200 ARR shows up as "15x".

2. Multiples depend on category more than on the startup

Median sold multiple by category (categories with at least 5 sold comps):

Category	Sold comps	Median multiple that cleared
Directory	12	6.75x
Micro-SaaS	36	5.33x
Newsletter	16	4.20x
Web App	20	3.14x
Mobile App	17	2.95x
E-commerce	7	2.08x
YouTube channel	7	1.12x

Two things surprised me. Directories clear at the highest multiple — low maintenance, recurring, SEO-driven cash. And YouTube channels clear at barely 1x revenue: buyers price in the key-person risk. If you own a directory, you're sitting on a more sellable asset than you think.

3. Where the underpriced deals are hiding

Of 390 available listings, 137 are priced at ≤3x revenue and 79 at ≤2x. Thirty listings were added in the last 7 days. The good ones move in days — the median sold listing had a $3,000 price, which is exactly the size an indie buyer can close on a weekend.

The top of my deal score right now (fresh + fairly priced + real revenue + real customers):

Startup	Category	Asking	ARR	Multiple	Days listed
Fashion Online Store	E-commerce	$109,000	$341,000	0.32x	14
AI Image-to-Video Generator	Web App	$30,000	$55,933	0.54x	10
Amazon FBA Decor Brand	E-commerce	$85,000	$103,000	0.83x	28
Relaxation Gummies Store	E-commerce	$36,000	$38,330	0.94x	12
Gamified Fitness / RPG Workout	Micro-SaaS	$19,000	$18,107	1.05x	3

E-commerce dominates the top because revenue there is gross, not net — which is exactly why you need the seller's expenses and businessModel fields next to the multiple before you get excited. The dataset carries both.

How I get this every morning

I didn't do this by hand. It's an Apify Actor I maintain: Microns Startup Listings Scraper. One run = the whole board, HTTP-only (no browser, no login, no proxies), with these fields per listing:

askingPriceUsd, arrUsd, and revenueMultiple (null when ARR is 0 or unpublished — never a fake number)
status = available or sold, so the sold rows double as comps
customers, pricePerCustomer, publishedDate, daysListed, foundedYear
the seller's own businessModel, expenses, growthOpportunity, reasonForSelling, askingPriceReasoning, techStack, competitors
dealScore 0–100: revenue evidence + price fairness vs ~3x + traction + freshness

Every run also writes a SUMMARY record with board totals, available vs sold, median revenue multiple and new-this-week count — so a weekly listingStatus: "sold" run gives you a live micro-startup valuation index that nobody publishes.

The daily deal-alert input I use, on a schedule with a Slack integration:

{ "listingStatus": "available", "minArrUsd": 1, "maxRevenueMultiple": 4, "sortBy": "dealScore" }

Pay-per-result, so a filtered run like that costs cents. It's also callable from AI agents (MCP) — "find me SaaS listings under $20K priced below 3x revenue, added this week" is one tool call.

Caveats, because this is money

Revenue on Microns is self-reported. Multiples are a screening tool, not diligence.
"Sold" means the seller marked it sold; the clearing price is the listed price at that time, which may differ from the final negotiated number.
Sample sizes per category are small (7–36 comps). Treat category medians as direction, not law.
Not financial advice. Verify on the listing page before you message a founder.

If you buy, sell, or benchmark micro-startups and want a field added, open an Issue on the Actor page — the parser is monitored daily.