DEV Community: Nikita Iakovlev

Email Verification Without Sending a Single Email (And Why Port 25 Is a Dead End)

Nikita Iakovlev — Fri, 04 Sep 2026 06:33:29 +0000

Every email verification service advertises the same thing: paste a list, get back deliverable or not. Almost none of them explain what they actually check, and the difference matters, because one of the checks people assume is happening is usually impossible.

Here's what's really under the hood, what each signal is worth, and the one that quietly doesn't work on any major cloud.

The check that doesn't work anymore

The textbook method is an SMTP handshake. You connect to the recipient's mail server on port 25, say HELO, MAIL FROM, then RCPT TO: <the address>, and read the response code. A 250 means the mailbox exists. You disconnect before DATA, so no mail is ever sent.

It's elegant. It's also unavailable to you.

Outbound port 25 is blocked by default on AWS, Google Cloud, Azure, DigitalOcean, Hetzner and most serverless platforms. It's an anti-spam measure and it has been standard for years. Some providers will unblock it — AWS after a request form, Google automatically for projects it scores as low-risk, Hetzner's dedicated servers by self-service after a month. DigitalOcean blocks 25, 465 and 587 with no path at all.

The point isn't that it's impossible everywhere. It's that on whatever you actually deploy to, it's probably off, and finding out after you've built the feature is expensive. I checked before building anything: on the platform I deploy to, a raw socket to gmail-smtp-in.l.google.com:25 times out from every region I tried.

Two more reasons SMTP verification is worse than it looks even when you can run it:

Catch-all domains accept everything. A large share of corporate domains return 250 for any address at all, so a positive result carries no information.
Greylisting and reputation. Repeated RCPT TO probes from one IP get throttled, then blocked. Providers treat the pattern as harvesting, because it is the same pattern harvesters use.

Which is why the honest services return a three-state answer, not a boolean. Anyone selling you a clean yes/no on a corporate domain is guessing.

What can actually be checked

Five signals, each independently useful:

1. Syntax against the real grammar. Not a regex from Stack Overflow. Length limits (64 characters local part, 254 total), consecutive dots, leading and trailing dots, quoting rules, IDN handling.

2. Domain resolution. Does the domain exist at all? An NXDOMAIN is a definite negative and the only cheap definite negative you get.

3. MX records. A domain can exist and still have no mail exchanger. No MX means nothing can receive mail there, which is again definite. This one catches more than you'd expect, including a surprising number of live business websites whose owners never finished setting up mail.

4. Disposable domain list. Mailinator, Guerrilla Mail, 10minutemail and several thousand throwaway providers, including the typo-squat domains registered specifically to catch mistyped Gmail addresses.

5. Classification. Role address (info@, support@, sales@, noreply@), free mailbox (Gmail, Yahoo, Outlook), and a typo suggestion when the domain is one edit away from a major provider.

The output of those five is not "valid" or "invalid". It's:

Status	Meaning	What to do
deliverable	Domain resolves, has an MX, not disposable, syntax clean	Send
risky	Reachable but flagged — disposable domain, or no MX record	Decide by campaign
undeliverable	Domain doesn't exist, or syntax is invalid	Drop

Role and free-mailbox are flags, not statuses. support@apify.com is deliverable and a role address; whether that disqualifies it depends entirely on what you're sending. The status says whether mail can arrive. The flags say whether you want it to.

The reason a missing MX lands in risky rather than undeliverable is a detail of the standard: with no MX record, a sending server falls back to the domain's A record and tries to deliver there (RFC 5321 §5.1). That almost always fails, because the machine behind the A record is a web server with no mail daemon — but "almost always" isn't "never", so the honest label is risky.

Real output

This is an actual run, not a mockup. Six addresses and four phone numbers went in on 4 September 2026; four of the email results:

[
  { "email": "support@apify.com",
    "emailStatus": "deliverable",
    "emailDomain": "apify.com", "emailIsRole": true,
    "emailIsFree": false, "emailIsDisposable": false,
    "contactScore": 45 },

  { "email": "fake@nonexistentdomain-xyz-99.com",
    "emailStatus": "undeliverable",
    "emailReason": "domain_not_found",
    "contactScore": 0 },

  { "email": "test@mailinator.com",
    "emailStatus": "risky",
    "emailReason": "disposable_domain",
    "emailIsDisposable": true,
    "contactScore": 15 },

  { "email": "noreply@gmial.com",
    "emailStatus": "risky",
    "emailReason": "disposable_domain",
    "emailIsRole": true, "emailIsDisposable": true,
    "contactScore": 15 }
]

That last one is the interesting case. gmial.com is a misspelling of gmail.com, it is registered, it resolves, and it sits on the public disposable-domain blocklists. A syntax check passes it. A domain-resolution check passes it. It takes the blocklist — or, independently, the missing MX record — to flag it, which is the argument for running all five checks rather than the one you think is decisive.

One more from that run, which I'll return to at the end: a live business domain that came back risky, reason no_mx_record. Working website, working DNS, no mail exchanger. Every email sent to that domain bounces, and its owner had no idea.

Phone numbers: the rules are published

North American phone validation has a real specification, which means real verification instead of a regex.

The North American Numbering Plan Administrator publishes the assignment file at reports.nanpa.com/public/npa_report.csv. It lists every area code, whether it's geographic, which state, province or territory it belongs to, and its time zone. In the file dated 3 September 2026 there are 378 geographic area codes in the United States, and 454 across the whole plan once Canada and the Caribbean members are counted.

From that file plus the plan's own rules:

The area code must be assigned and in service. 999 is not: it's one of 79 codes with a 9 in the middle that the file marks Expansion Code, held back for when the plan runs out of numbers. Every one of them fails.
Area code and exchange both start with 2 through 9, never 0 or 1.
The N11 codes are service codes — 911, 411, 211 and the rest — and are never assignable as exchanges. A handful more, 555, 950, 958, 959 and 976, are held back by convention rather than by rule.
Within 555, only the block 555-0100 to 555-0199 is formally set aside for fictional use. This is the part most validators get wrong in the other direction: 555-1212 is a live directory-assistance number, so "reject anything with 555" produces false negatives on real data. Reject the reserved block, not the exchange.

Real output from the same run:

[
  { "phone": "+1 415 555 0132",
    "phoneStatus": "invalid", "phoneReason": "reserved_exchange" },

  { "phone": "+1 999 123 4567",
    "phoneStatus": "invalid", "phoneReason": "nanp_rule" },

  { "phone": "+62 361 123456",
    "phoneStatus": "valid", "phoneType": "international",
    "phoneReason": "structure_only", "phoneE164": "+62361123456" }
]

Note the third one. 0361 is genuinely the Denpasar area code, so the number is correctly formed for Bali, and the honest answer is structure_only — the number is correctly formed, and outside the NANP there is no equivalent public assignment file to check it against. A tool that claims to "validate" any international number without saying that is telling you it checked something it didn't.

For valid North American numbers you also get the state and the time zone, which is what you actually wanted — a lead list sorted by local business hours. One gotcha worth hard-coding: Arizona does not observe daylight saving, so it needs America/Phoenix, not America/Denver, or half your calls land an hour off for eight months of the year.

Who needs this

Anyone buying or scraping lead lists. Bounce rate is what gets a sending domain blacklisted. The clearest published numbers come from Amazon SES: a hard-bounce rate at or above 5% puts an account under review, 10% suspends it. Most other providers publish nothing and act on their own thresholds, which is worse, not better — you find out by being throttled.

Sales teams paying per contact. Enrichment vendors charge whether the contact is alive or not. Filtering before import is cheaper than filtering after.

Anyone with a signup form. Blocking disposable domains at registration removes most throwaway accounts, and the typo suggestion recovers real users who would otherwise never receive the confirmation email.

CRM hygiene. Contact data goes stale continuously as people change jobs, companies fold and domains lapse. The often-quoted "22.5% a year" traces back to a single HubSpot blog post and shouldn't be treated as measured, but the direction isn't in doubt, and a quarterly pass over the whole database catches the decay before a campaign does.

Running it

The Email & Phone Verifier actor does the checks above with no SMTP dependency, which is why it works from anywhere.

Input:

{
  "emails": ["support@apify.com", "test@mailinator.com"],
  "phones": ["+1 415 555 0132", "+1 212 555 0199"],
  "keep": "all"
}

From the API:

curl -X POST "https://api.apify.com/v2/acts/lergassy~email-phone-verifier/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{"emails":["support@apify.com"],"phones":["+1 415 555 0132"]}'

It also reads another actor's dataset directly, which is the part worth knowing. Point inputDatasetId at the output of any scraper that produced contacts and it verifies that dataset in place:

{
  "inputDatasetId": "abc123def456",
  "emailField": "email",
  "phoneField": "phone",
  "keep": "deliverable"
}

Set keep to deliverable and the output contains only contacts that passed. Scrape, verify, import, with nothing in between.

What it does not do

It does not confirm that a specific mailbox exists. Nothing that runs on ordinary cloud infrastructure can, for the port 25 reason above, and on catch-all domains nothing can regardless of where it runs.

It does not tell you whether a phone is mobile or landline. That requires a carrier lookup against paid HLR data, which is a different product at a different price.

It does not check whether a person still works somewhere. That's job-change data, sold by LinkedIn-adjacent vendors, and no amount of DNS will produce it.

What it does is remove the addresses that will definitely bounce and the numbers that cannot possibly ring, and label the rest with the reason. On the scraped lists I've run it through, that has been somewhere between a seventh and a third of the rows — and it's that fraction which costs you a sending domain.

The postscript

That domain with no MX record from the run above? It was my own company's.

The website works, the DNS is fine, mail was never configured. A sender with no MX to aim at falls back to the A record, which here is a web server that has never spoken SMTP — so mail to that domain has been failing quietly for as long as the domain has existed. I found it by running my own address through a tool I built for other people's lists.

Run your own domain through something before you run anyone else's.

Actor: apify.com/lergassy/email-phone-verifier

Related: US New Business Leads delivers newly registered US companies with these same checks already applied to every phone and email in the feed.

Telegram Channel Analytics Without an API Key — Subscriber Counts Lie, Reach Doesn't

Nikita Iakovlev — Fri, 04 Sep 2026 06:33:13 +0000

A Telegram channel with 10.9 million subscribers and a channel with 172,000 subscribers. Which one would you advertise in?

Here are their actual numbers, from a run over the last 100 posts of each on 4 September 2026:

	Pavel Durov	Bloomberg
Subscribers	10,900,000	172,000
Median views per post	5,600,000	39,900
Reach	51.4%	23.2%
Reach trend	−10.4%	+44.9%
Reactions per post	1,030	144
Posts per day	0.3	0.2

Durov's channel reaches half its audience — extraordinary, and 140× more people in absolute terms. But its reach is falling, while Bloomberg's is up 45% over the same window. Depending on whether you're buying one placement or a six-month presence, those two facts point in different directions.

None of this is in the subscriber count, which is the only number most people look at. All of it is available without an API key.

The three ways into Telegram data, and why two of them don't work

Bot API. Requires your bot to be an administrator of the target channel. Perfect for channels you own, useless for every channel you don't.

MTProto / user API. Full access to everything a logged-in user sees. Requires a phone number, a 2FA session and careful rate limiting. Reading public channel history through it is relatively low-risk; bulk member-list exports and messaging strangers are what actually get accounts limited. Either way you are putting a real phone number behind the request, and a restriction lands on the number, not the IP.

The public preview page. Telegram serves t.me/s/<channel> to any visitor, no account required. It's the same HTML a search engine sees.

For monitoring channels you don't own, the third option is the only sane one. What it gives you:

Channel header: title, subscriber count, description, photo, verified badge, media counts
20 posts per request, paged backwards with ?before=<message_id>
Per post: message id, ISO timestamp, full text, view count, reactions with their emoji and counts, media flags, forward source, reply flag, author signature

About 1.4 seconds per page, so 200 posts — ten pages — in roughly fifteen seconds. No token, no proxy required, no ban risk.

Why views are the number that matters

Telegram is unusual in how far it goes here: every post in a public channel carries a view count, visible to anyone, with no login and no API. X and Instagram both surface view counts too, but only inside their own apps, behind authentication and rate limits — which is the difference that matters if you want to compute anything across a list of channels.

That single number lets you compute the thing advertisers actually price against:

reach = median views per post / subscribers

Median, never average. One viral post drags an average up by a factor of three and tells you nothing about what a normal post does. On the Durov channel the average is 6.66M against a 5.6M median, and the best post did 24.2M — if you quoted the average to a client you would be promising reach that a typical post won't deliver.

Reach separates real audiences from bought ones instantly. A channel with 200,000 subscribers and 4% reach was inflated. A channel with 50,000 subscribers and 40% reach is worth more per post than the inflated one, and usually costs less.

What else the timestamps give you

Once you have a few hundred posts with timestamps and view counts, several useful things fall out of simple arithmetic:

Reach trend. Median views of the newer half of the window against the older half. Bloomberg's +44.9% and Durov's −10.4% above come from exactly this. It's a direction, not a forecast, but it's the difference between buying into a channel on the way up and one on the way down.

Best hours and weekdays. Group posts by hour, take the median views of each group, and require at least two posts per group so a single outlier can't win. This is a channel's own data, which beats every generic "best time to post" article ever written, because those articles are about a different audience in a different time zone.

Cadence. Posts per day and per week. A channel posting fifty times a day burns its audience differently from one posting twice.

Content mix. Share of posts with photos, videos, polls, links. Share that are forwards, and which channels they're forwarded from — run that across twenty channels in one niche and you have a map of who feeds whom.

Reaction rate. Reactions per post as a share of median views. Rough, but it separates channels that are read from channels that are merely delivered.

The mistakes worth avoiding

I got most of these wrong before the data corrected me.

A slow channel's 100 posts can span a year. Bloomberg's channel above posts 0.2 times a day, so 100 posts covers well over a year. Any "per day" figure computed over that window is an average across seasons. Always publish the number of days the window covers alongside anything rated per day.

Hashtag extraction needs a letter. A naive #\w+ regex turns #036 from a price into a top hashtag. Require at least one letter in the match.

Reach above 100% is not a bug. Telegram counts a view once per user, including people who saw the post forwarded into another channel or opened it on the web. A post that travels beyond its channel can be seen by more people than the channel has subscribers.

Timestamps are UTC. Convert to the channel's own time zone before concluding anything about a morning slot.

Minimal implementation

The page is server-rendered HTML with stable class names:

import requests, re
from bs4 import BeautifulSoup

def channel_page(name, before=None):
    url = f"https://t.me/s/{name}" + (f"?before={before}" if before else "")
    html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).text
    soup = BeautifulSoup(html, "html.parser")

    posts = []
    for m in soup.select(".tgme_widget_message_wrap"):
        views = m.select_one(".tgme_widget_message_views")
        time_el = m.select_one(".tgme_widget_message_date time")
        posts.append({
            "id": int(m.select_one("[data-post]")["data-post"].split("/")[1]),
            "date": time_el["datetime"] if time_el else None,
            "views": views.get_text(strip=True) if views else None,
        })
    return posts

Two things this snippet doesn't do, and you need both: view counts arrive as 53K and 1.4M, so they need parsing into integers before any arithmetic; and paging requires passing the lowest message id you've seen back as before, then stopping when a page returns nothing new.

The finished version

I packaged the whole thing — paging, the K/M parser, all the metrics above, error rows for channels that have no public page — as Telegram Channel Analytics.

{
  "channels": ["@durov", "@bloomberg"],
  "postsPerChannel": 100,
  "topPosts": 3
}

One row per channel, with a ready-made sentence at the end of it:

{
  "channel": "bloomberg",
  "subscribers": 172000,
  "postsAnalysed": 100,
  "daysCovered": 415.1,
  "viewsMedian": 39900,
  "engagementRate": 23.2,
  "viewsTrendPercent": 44.9,
  "reactionsPerPost": 143.9,
  "bestHours": [{"hour": 8, "posts": 5, "medianViews": 62300}],
  "summary": "Bloomberg has 172,000 subscribers, median post reaches 39,900 views (23.2% of the audience), posts 0.2 times a day, reach up 44.9% across the 100 posts analysed."
}

One naming wrinkle worth flagging: the field is called engagementRate, but it holds what this article calls reach — median views over subscribers. The reaction rate is a separate field.

Switch on includePosts and every post arrives as its own row too, with views, reach, reactions, media flags, hashtags and links.

What it can't do

Private channels and groups are invisible. No public page, no data. Those come back as an explicit error row with the reason, never as a silent zero. Reading private chats requires a real account and a session, which is a different product with different risks.

Subscriber history doesn't exist in the source. Telegram publishes the current count and nothing else. "How fast is this channel growing" can only be answered by running on a schedule and keeping the results — a daily run turns subscribers, viewsMedian and engagementRate into a time series. There is no way to recover last month's number today.

Comments aren't in the preview. Discussion-group replies live elsewhere.

Where this is worth the effort

Buying advertising is the obvious one: compare candidates on reach instead of subscribers and the shortlist reorders immediately.

Selling it is the mirror image. A channel owner quoting reach, reaction rate, cadence and best hours is quoting a media kit. Most don't, because nobody computes it for them.

And competitor tracking, which is the use that compounds: run the same twenty channels weekly, watch whose reach is climbing, then look at what they changed in the same window. The content mix and posting hours are right there in the same row.

Actor: apify.com/lergassy/telegram-channel-analytics

Hotel Review Data in 2026 — Why Agoda Gives You Two Datasets for the Price of One

Nikita Iakovlev — Fri, 04 Sep 2026 06:32:51 +0000

If you're collecting hotel review data, Agoda is the source most people skip and the one with the most on it. Not because it has more reviews than Booking.com, but because it carries both.

Agoda was bought in 2007 by priceline.com, the company that renamed itself Booking Holdings in 2018 and also owns Booking.com, Kayak and OpenTable. One consequence is visible on almost every hotel page: Agoda displays its own guest reviews and Booking.com's, served by separate review providers behind the same page.

A real example, pulled this month:

{
  "type": "hotel",
  "name": "The Magani Hotel And Spa",
  "hotelId": 335650,
  "agodaReviewCount": 1347,
  "bookingReviewCount": 1025
}

That's 2,372 reviews for one mid-sized Bali hotel as of 4 September 2026, from two review pools with different reviewer bases, reachable through one page.

Why hotel review data is harder than it looks

There is no public API. Agoda has an affiliate and partner program with data access for approved partners. For everyone else — an analyst, a hotel owner benchmarking competitors, a developer building a comparison tool — there's the website.

Review counts on the page don't match what the page shows. A hotel page displays a paginated slice of reviews. The provider structure isn't visible in the HTML — you have to know that provider 332 is Agoda's own pool and 3038 is Booking.com's, and request each separately, or you silently collect half the data and never know.

The interesting fields aren't in the review text. Rating and comment are the easy part. Traveler type, room type booked, length of stay and reviewer country are what make the data analysable, and they're attached to the review record rather than rendered as prose.

Prices and availability are dynamic; reviews are not. This is the part that makes reviews the more useful target. A scraped price is stale in an hour. A review stays true to what its author wrote, give or take the occasional edit or deletion, which means a review dataset compounds while a price dataset decays.

What a review record actually contains

Real output, one review, unedited:

{
  "type": "review",
  "hotelId": 335650,
  "hotelName": "The Magani Hotel And Spa",
  "source": "agoda",
  "reviewId": 1175631129,
  "rating": 8.4,
  "title": "Magani",
  "text": "Great location, lovely staff , great breakfast & comfy bed",
  "reviewDate": "September 03, 2026",
  "checkInDate": "August 2026",
  "reviewerName": "Natalie",
  "reviewerCountry": "au",
  "travelerType": "Couple",
  "roomType": "Deluxe Room Double Bed",
  "lengthOfStay": 1,
  "reviewProvider": "Agoda",
  "helpfulVotes": 0
}

Four fields there are worth more than the review text.

reviewerCountry turns a review set into a demand map. Which markets actually book this property, and how do their ratings differ? Australian and Chinese guests rate the same Bali hotel differently, consistently, and if you run a property you need to know in which direction.

travelerType — Couple, Solo, Family with young children, Business — segments the ratings. A hotel averaging 8.4 might be a 9.1 for couples and a 6.8 for families, and the average hides the entire problem.

roomType attributes complaints to inventory. If the 4-star average is being dragged down by one room category, that's an operational fix, not a marketing one.

lengthOfStay separates the one-night airport stopover from the week-long stay. They are different products reviewed on the same page.

checkInDate matters more than reviewDate: it tells you when the guest was actually there, which is what you need to correlate a rating drop with a renovation, a management change or a rainy season.

Who uses this

Hotel operators benchmarking competitors. Five properties in the same district, all their reviews for the last two years, segmented by traveler type. The gaps are specific and actionable in a way that a star rating never is.

Revenue managers. Rating movements lead pricing power. A property whose cleanliness sub-score has fallen half a point over two quarters will lose rate before it loses occupancy.

Investors and acquirers. Reviews are the only public operational data on a private hotel. Trajectory over three years, complaint themes, and whether ratings recovered after the last renovation.

Travel product builders. Aggregating reviews across sources for a comparison site, a recommendation engine, or a chatbot that answers "quiet hotel in Ubud, good for families" from evidence instead of marketing copy.

Sentiment and NLP work. Hotel reviews are unusually good training data: a numeric rating attached to free text, in dozens of languages, with structured metadata about who wrote it.

Running it

Agoda Reviews Scraper takes hotel page URLs and handles both review providers.

{
  "startUrls": [
    { "url": "https://www.agoda.com/the-magani-hotel-and-spa/hotel/bali-id.html" }
  ],
  "maxReviewsPerHotel": 500,
  "reviewsProvider": "all",
  "reviewsSort": "recent"
}

reviewsProvider accepts agoda, booking or all. Use all unless you specifically want one pool. Note that these are the two large providers, not the only ones — some properties also carry smaller pools from regional partners, which this run doesn't pull.

From the API:

curl -X POST "https://api.apify.com/v2/acts/lergassy~agoda-reviews-scraper/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{
    "startUrls": [{"url":"https://www.agoda.com/the-magani-hotel-and-spa/hotel/bali-id.html"}],
    "maxReviewsPerHotel": 200,
    "reviewsProvider": "all"
  }'

Output is one hotel row per property with both review counts, then one review row per review. Sort options are recent, highest rating and lowest rating — and lowest-first is usually the more useful run, because the complaints cluster and the praise doesn't.

What you don't get

No prices. Review extraction and price extraction are separate problems, because prices depend on dates, occupancy and currency. For dated hotel pricing with the rate for your own check-in and check-out, that's Trip.com Scraper.

No reviewer identity beyond a first name and country. That's all Agoda publishes, and it's the right amount.

Hotel replies are hit and miss. Agoda does publish management responses, but adoption varies wildly by property — some hotels answer nearly every review, plenty answer none. Don't build an analysis that assumes replies exist.

Review counts vary by provider availability. Some properties have a large Agoda pool and almost no Booking.com reviews, or the reverse. The hotel row reports both counts before any reviews are pulled, so you can check coverage before committing to a large run.

Build versus buy, honestly

The hard parts here aren't conceptually hard: find the hotel id from a URL slug, call the review endpoint per provider, page through, normalise dates. A competent developer does the first version in a day.

The cost is the second year. Hotel ids get reassigned, the provider list changes per property, date formats differ by locale, and the endpoint shape shifts without notice. It's not difficult work, it's unscheduled work, arriving on the day you need the data.

The actor is free to run while it's new — you pay Apify's platform usage and nothing per review — which makes the build-versus-buy arithmetic hard to argue with. Build it if hotel data is your product. Don't if hotel data is an input to your product.

Actor: apify.com/lergassy/agoda-reviews-scraper

Related: Trip.com Scraper for dated hotel prices and ratings, Google Flights Scraper for airfare on the same trip.

Google Flights Data Without an API — What You Can Actually Get in 2026

Nikita Iakovlev — Fri, 04 Sep 2026 06:32:43 +0000

Google Flights is the best flight search on the internet, and one of the few major Google products with no self-serve API attached to it. QPX Express, the public tier that used to do this, was withdrawn on 10 April 2018. The engine itself never went away — ITA's QPX Enterprise is still sold to airlines — but nothing replaced the tier a developer could just sign up for.

What's left is the page. It's worth more than people assume, because it carries something the airlines themselves don't publish.

Here is a real run, Bali to Singapore, departing 15 October 2026, pulled this month:

{
  "type": "search",
  "origin": "DPS",
  "destination": "SIN",
  "departureDate": "2026-10-15",
  "tripType": "one_way",
  "currency": "USD",
  "flightsFound": 164,
  "lowestPrice": 82,
  "priceLevel": "typical",
  "typicalPrice": 68,
  "typicalPriceLow": 55,
  "typicalPriceHigh": 110,
  "priceHistory": [
    {"date": "2026-07-05", "price": 54},
    {"date": "2026-07-06", "price": 58},
    {"date": "2026-07-07", "price": 58}
  ]
}

Look at priceLevel, typicalPrice and priceHistory. Google tracks what this route normally costs and tells you where today sits in that band. $82 against a typical $68, in a normal range of $55 to $110. It's "typical", not a deal.

No airline gives you that. No aggregator gives you that for free. It's the single most useful field on the page and it exists because Google has been recording every route's fares for years.

Why there's no API

The data isn't simply Google's to hand over. By QPX's own documentation the inputs are schedules from OAG, published fares from ATPCO, and availability supplied directly or indirectly by the carriers. Google can compute and display itineraries from those inputs. Reselling the underlying feeds is a different question with a different set of licensors.

The industry alternatives are expensive. Amadeus and Sabre sell developer access, with per-call pricing, contracts and certification. Duffel and Kiwi are friendlier but priced for booking flows, not research: they expect you to sell tickets, not analyse fares.

The page is a heavy client-side app. Google Flights renders results in the browser from an internal protocol buffer payload. There's no HTML table of flights to parse, which is why most naive attempts return an empty list and conclude the data isn't there.

Results are personalised and regional. Currency, available carriers and sometimes price depend on where the request comes from. A fare pulled from a US datacenter is not necessarily the fare a user in Jakarta sees, and if you're doing price comparison research that difference is your entire result.

What you get per flight

Each flight in the results is a full itinerary, not a price tag:

{
  "type": "flight",
  "price": 82,
  "isBestFlight": true,
  "airline": "Scoot",
  "flightNumbers": ["TR289"],
  "stops": 0,
  "departAirport": "DPS",
  "departAt": "2026-10-15T14:05",
  "arriveAirport": "SIN",
  "arriveAt": "2026-10-15T16:55",
  "totalDurationMinutes": 170,
  "duration": "2 hr 50 min",
  "legs": [
    {
      "flightNumber": "TR289",
      "airline": "Scoot",
      "airlineCode": "TR",
      "from": "DPS",
      "fromName": "I Gusti Ngurah Rai International Airport",
      "to": "SIN",
      "toName": "Singapore Changi Airport",
      "departAt": "2026-10-15T14:05",
      "arriveAt": "2026-10-15T16:55",
      "durationMinutes": 170
    }
  ]
}

Multi-leg itineraries come back with every leg, its operating carrier, and the layover between them — which matters, because a $40 saving that costs a nine-hour layover in Kuala Lumpur isn't a saving.

isBestFlight is Google's own flag combining price, duration and stops. Useful as a shortcut, and useful to ignore deliberately when you're studying what a route actually costs rather than what Google recommends.

What it's genuinely good for

Price monitoring on routes you care about. Run the same route daily, keep the results, and priceLevel plus your own history tells you when to buy. This is the use case that pays for itself in one trip.

Route research. Which carriers fly a city pair, how often, at what times. For anyone planning capacity, a travel business, or a relocation, that's a survey nobody publishes in one place.

Fare benchmarking for travel agencies. Comparing your GDS fare against the public best price on the same route, daily, tells you where your inventory is uncompetitive.

Seasonality studies. Run a route across a range of departure dates and the shape of the year appears — the shoulder seasons, the holiday spikes, the exact week a route gets expensive.

Visa and immigration paperwork. Several consulates want proof of onward travel. Knowing the real price band of the cheapest exit flight, for the right date, is the difference between a $60 ticket and a $300 one.

Running it

Google Flights Scraper takes IATA codes and dates:

{
  "origin": "DPS",
  "destination": "SIN",
  "departureDate": "2026-10-15",
  "adults": 1
}

Round trips take returnDate. Several routes in one run go in routes. Open-jaw and multi-city itineraries go in multiCityLegs. Cabin class, passenger mix and currency are all inputs.

From the API:

curl -X POST "https://api.apify.com/v2/acts/lergassy~google-flights-scraper/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{"origin":"DPS","destination":"SIN","departureDate":"2026-10-15","adults":1}'

Output is one search row per query — with the price band, the history and the totals — followed by one flight row per itinerary. The search row alone answers "is this cheap right now", which for monitoring is the only row you need to keep.

What you don't get

No booking. This is research data. Booking requires an airline or an OTA, and the fare you saw may be gone by the time you get there, which is normal for airline pricing rather than a defect in the data.

No seat availability or fare rules. Change fees, baggage allowance and refundability aren't on the results page. They live in the fare rules, which are a GDS product.

Prices are indicative. Google shows fares from its partners, and the same flight can be cheaper direct or on a regional OTA. Treat the number as a benchmark, not a quote.

Personalisation is real. If you need the fare a specific market sees, the request has to originate in that market. Comparing "the DPS–SIN price" across regions requires deciding which region you mean.

The honest build-versus-buy note

Parsing Google Flights is not a weekend project. The payload is a nested protocol buffer with no field names, positional rather than keyed, and the positions move when Google ships a change. Every published parser is a set of magic array indices that somebody re-derived by hand after the last break.

If flight data is your core product, you'll end up owning that maintenance regardless. If it's one input among many, someone else's parser is the right answer — this one is currently free to run, with only Apify's platform usage to pay — and the two hours you'd spend on the first version are better spent on what you do with the numbers.

Actor: apify.com/lergassy/google-flights-scraper

Related: Agoda Reviews Scraper and Trip.com Scraper for the hotel half of the same trip.

Every US Company That Registered This Week Is Public Data. Here's How to Read It

Nikita Iakovlev — Fri, 04 Sep 2026 06:32:36 +0000

A company that registered on Monday has no accountant, no insurance broker, no payroll provider, no bank relationship, no website and no marketing agency. Four weeks later it has most of those, chosen under time pressure from whoever reached it first.

That window is the most valuable one in B2B sales, and the list of who's in it is public.

Three feeds, three different kinds of company

State registries. Every US state maintains a business registry, though not always under a Secretary of State — Arizona runs it through its Corporation Commission, Michigan through LARA, Virginia through the State Corporation Commission, and Alaska, Hawaii and Utah have no Secretary of State at all. Many publish the registry as open data on a Socrata portal, the platform behind a large share of municipal open-data sites. New LLCs and corporations appear within days of filing. Connecticut is the standout: nearly every new company comes with the email address the founder gave the state.

Municipal licence feeds. Cities publish business licences as they're issued. This catches a different population — the restaurant, the salon, the contractor, businesses that need a licence to open rather than a corporate entity to exist. Owner names and phone numbers appear here far more often than in state data.

SEC EDGAR Form D. When a company raises private capital under Regulation D, it files a notice with the SEC within fifteen days of the first sale. That filing is public and structured, and it names the amount raised, the date of first sale, the number of investors, and the executive officers with a company phone number.

Form D is the highest-intent feed of the three. A company that just closed a round is about to hire, buy software and outsource functions, and it has money on the day you call.

Here's an actual Form D record from a run this month:

{
  "type": "signal",
  "signal": "FUNDING_ROUND",
  "sourceName": "SEC EDGAR Form D",
  "companyName": "AI Planning Logic Inc",
  "cik": "2090528",
  "entityType": "Corporation",
  "jurisdictionOfInc": "DELAWARE",
  "yearOfIncorporation": "2022",
  "incorporatedWithinFiveYears": true,
  "address": "16192 COASTAL HIGHWAY",
  "city": "LEWES", "region": "DE", "postalCode": "19958",
  "contactPhone": "+1707888XXXX",
  "industry": "Other Technology",
  "industryTags": ["Technology & Software"],
  "dateOfFirstSale": "2025-12-05",
  "fundingAmountSold": 240000,
  "investorCount": 3
}

$240,000 raised from three investors, a Delaware corporation under five years old, in technology, with a phone number on the filing. That's a qualified lead assembled entirely from a government record. (The phone is masked here; it's public in EDGAR.)

One caveat visible in that very record: dateOfFirstSale is December 2025, while the filing itself is recent. Form D is filed within fifteen days of the first sale, but amendments and late filings both exist, so read the sale date rather than assuming the filing date is the event date.

Why this isn't as easy as "the data is public"

Every source has a different schema. Connecticut's field names have nothing to do with Chicago's, which have nothing to do with EDGAR's XML. Normalising forty sources into one row shape is the actual work.

Registries are full of things that aren't new businesses. Biennial statements, renewals, amendments, name changes, dissolutions and reinstatements sit in the same feed as formations. A naive pull of "recent filings in New York" returns mostly companies that have existed for a decade. The filing type has to be filtered, and the vocabulary differs by state.

The same business appears more than once. A company files with the state and then takes a city licence. Two rows, one business. Deduplication has to survive punctuation, "LLC" versus "L.L.C.", and suite numbers written four ways.

Dates lie. Future-dated filings, typo'd years, and registries that publish the processing date rather than the filing date. If freshness is your entire value proposition, you have to defend it.

Contact fields are unverified. Registries publish what the filer typed. Nobody checks it. A raw registry feed contains dead numbers and misspelled email domains at a meaningful rate, and each one costs you sender reputation or a wasted dial.

That last one is worth solving properly: every phone in this feed is validated against the North American Numbering Plan and every email domain gets a live MX lookup plus disposable and role-address flags, before the row reaches you. A number in a reserved exchange or a domain with no mail server is labelled, not sold to you as a lead.

Volumes, measured

Actual output over a seven-day window:

Feed	Volume	Contact coverage
New local businesses, 9 cities	~190	100% with phone, ~30% with owner name
New Connecticut companies	~860	100% with email, 99% with NAICS code, 95% with a usable address
Funded startups (Form D)	~250	94% with company phone, 100% with an officer's name

About 1,300 qualified new businesses a week from three presets. Not a database of everyone — a feed of who is new.

Who this is for

Anyone selling to new businesses. Accountants, bookkeepers, insurance brokers, payroll, POS systems, business banking, commercial cleaning, signage. The entire category of vendor a business chooses once, in its first month, and then keeps for years.

Agencies and web developers. A newly registered company with no website is the least ambiguous lead in existence.

B2B SaaS selling to funded startups. Form D plus a five-year incorporation flag plus an amount raised is a better qualification filter than anything in a generic lead database, and it's fresher.

Commercial real estate and equipment. New licences in a city are the leading indicator for space and fit-out demand, by industry and by ZIP.

Market researchers. Formation rates by state, city and industry give you the composition behind the headline number — the Census Bureau publishes Business Formation Statistics weekly, but not broken down the way your question needs it.

Running it

US New Business Leads ships with three presets so the first run needs no configuration:

{ "preset": "funded_startups", "maxResultsPerState": 100 }

Custom targeting takes states, cities, industries, a date floor, and contact-quality filters:

{
  "states": ["CT", "NY"],
  "registeredSince": "2026-08-01",
  "requireContact": true,
  "contactQuality": "deliverable",
  "monitorMode": true
}

monitorMode is what turns it from a scraper into a feed: it remembers what it has already delivered and returns only businesses you haven't seen. Put it on a daily schedule and each run is that day's new companies, deduplicated against every previous run.

From the API:

curl -X POST "https://api.apify.com/v2/acts/lergassy~us-business-filings/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{"preset":"funded_startups","maxResultsPerState":100}'

Every row carries a lead score from 0 to 100 built from freshness, contact completeness, address quality and entity type. Sort by it, or set a floor and never look at the rest.

What you don't get

Not every state. Coverage follows which registries publish machine-readable open data, and several large states don't. Where a state is missing, it's missing because the data isn't published, not because it's hard to scrape.

No decision-maker names beyond what's filed. State filings name a registered agent, sometimes an organiser. Form D names executive officers. Neither gives you a VP of Engineering.

No revenue, headcount or tech stack. This is a formation feed, not an enrichment product. It tells you a company exists and is new. What it does is a separate lookup.

No email for most municipal records. Cities publish phones far more reliably than emails. Connecticut is the exception that makes email-first outreach viable at all.

The thing that makes it work

Freshness is the entire product. A newly-registered-business list that's six weeks old is a list of companies that already chose a vendor, and it's worth close to nothing.

Which is why this belongs on a schedule rather than in a one-off run, and why deduplication across runs matters more than volume in any single one. Reaching a business in week one is a different conversation from reaching it in week seven.

Actor: apify.com/lergassy/us-business-filings

Related: Email & Phone Verifier runs the same checks over any contact list you already have, including the output of another actor.

I Measured 900 Apify Actors and Got the Price Wrong by 100x. Here's the Corrected Data

Nikita Iakovlev — Fri, 04 Sep 2026 06:32:29 +0000

I set out to answer a boring question with data instead of opinion: on the Apify Store, who actually makes money, and from what?

I got an answer, published it internally, and then discovered my central example was wrong by a factor of a hundred. This is the corrected version, including the mistake, because the mistake is the most useful thing in here for anyone doing the same measurement.

The method, and its two limits

Two public endpoints do all the work:

GET /v2/store?limit=1000&offset=N — the catalogue
GET /v2/acts/{username}~{name} — pricing, creation date, monthly users, monthly run counts

Revenue estimate per actor:

runs_30d × price_per_primary_event × 0.8

The 0.8 is the developer's share. This is a lower bound: it assumes one billable result per run, and most runs produce more.

Limit one: you cannot crawl the whole store. The endpoint reports a total of about 56,000 and Apify's own store page advertises more than 67,000 actors. But it stops returning records past an offset of roughly 16,000, and every page comes back short of the limit you asked for. Deduplicated, I ended up with 13,190 actors — under a quarter of the catalogue, and biased toward whatever the default ranking puts first. So: no store-wide survival rates from me. What follows is about the 900 most-used actors, enriched individually, which is where essentially all the revenue is anyway.

Limit two is the interesting one.

The pricing trap that makes every number wrong

Pay-per-event actors can price each event in tiers, keyed to the customer's Apify plan. The API returns them like this:

"eventTieredPricingUsd": {
  "FREE":     { "tieredEventPriceUsd": 0.1 },
  "BRONZE":   { "tieredEventPriceUsd": 0.001 },
  "SILVER":   { "tieredEventPriceUsd": 0.0008 },
  "GOLD":     { "tieredEventPriceUsd": 0.0006 },
  "PLATINUM": { "tieredEventPriceUsd": 0.0006 },
  "DIAMOND":  { "tieredEventPriceUsd": 0.0006 }
}

Read the first key and you conclude this actor charges $100 per 1,000 results. It doesn't. FREE is a deterrent price for people on the free plan — who, as Apify's own docs state, generate no developer revenue at all. Every paying customer is on BRONZE or better: $1 per 1,000, a hundred times less.

My original write-up leaned on exactly this actor as proof that expensive actors win. I had it at $8,600 a month. Corrected, it's about $86.

60% of the top 900 actors use tiered pricing, and 467 of them have a FREE tier priced above their paid tiers. If you take the first key, your revenue total comes out 21% too high and individual actors can be off by up to 100×. The fix is one line — take the maximum across the non-FREE tiers — but you have to know the trap is there.

Everything below is computed the corrected way.

Who takes the money

Five accounts — apify, compass, clockworks, junglee, lukaskrivka — are Apify's own and Apify-affiliated. Among the 900 most-used actors they hold 48% of the users and 44% of the revenue.

That's the single most important correction to make to any niche research. Instagram and Google Maps look enormous until you exclude those accounts, and then most of the money vanishes, because it never reached a third-party developer.

Of roughly $400,000 a month across these 900 actors, about $224,000 is left for everyone else.

The finding that changed how I build

Third-party actors only, grouped by what they charge per 1,000 results:

Price per 1k	Actors	Total/month	Median actor	Median runs/month
under $1	193	$11,409	$5	18,876
$1–3	267	$24,940	$9	5,748
$3–6	199	$82,982	$24	7,153
$6–15	97	$25,018	$23	3,221
over $15	71	$79,725	$111	2,799

Read the last two columns together. Actors priced over $15 per 1,000 do one seventh the traffic of the sub-$1 group and make twenty-two times more money.

The cheap end is a logistics business. 193 actors grinding through nearly nineteen thousand runs a month each, to earn five dollars.

The top earner in the whole sample is a LinkedIn profile search priced at $100 per 1,000 search-page results — genuinely, at every paid tier — doing 651,000 events a month for about $52,000. Four of the top twelve earners are LinkedIn actors, three of them third-party.

The mechanism is not clever. A buyer prices your scraper against the cheapest scraper doing the same thing. A buyer prices your enrichment tool against ZoomInfo, Hunter and Snov, which charge in dollars per contact. Same engineering, different comparison set.

What $500 a month actually requires

Pure arithmetic, after the 20% platform cut:

Price per 1,000	Rows needed per month
$1	625,000
$4	156,000
$20	31,000
$100	6,200
$200	3,100

A 200× spread in what you have to build and operate for the same money.

How many actually get there

Among the 829 third-party actors in the top 900 by usage — the most-used quarter of the store's most-used slice, so this is the optimistic end of the distribution:

Threshold	Actors clearing it
any revenue at all	827
$100/month	175
$500/month	62
$1,000/month	35

Median monthly revenue among those that earn anything: $14.

Remember these are the best-used actors in the store. The tail below them is far worse, and I can't measure it, because the catalogue endpoint won't page that far.

Runs per user, not users

Median runs per user across the sample sits between 15 and 90 depending on the niche.

Hundreds of runs per user means the actor is wired into a pipeline or called in a loop by an agent. Under ten means people tried it once and left, however good the user count looks on the store page. It's the cheapest health check you can compute and almost nobody looks at it.

The mechanics that gate all of this

Worth knowing before you start:

80% of revenue, minus platform usage costs. The split is confirmed in the API as a 20% margin, but Apify's terms deduct platform costs from your share, not from the gross. Compute per run.
Only paying customers count. Free-plan usage earns nothing, which is exactly why the FREE pricing tier exists and exactly why reading it as your price is wrong.
Payout minimum: $20 for PayPal and Wise, $100 for other methods. Below the threshold, the balance rolls over month to month.
KYC is mandatory before any payout: ID, proof of address, tax data.
Money is slow. Invoices generate on the 11th, auto-approve on the 14th, payment follows.
Significant price changes take 14 days to reach existing users — a switch of model, a price increase, a new paid event. Reductions apply immediately.
Rental pricing dies on 30 September 2026. Anything not migrated to pay-per-event collects nothing after that date.

The trend to plan against

Apify has published its developer payouts periodically: about $563,000 a month in September 2025, and $1.5 million a month as of August 2026. That's roughly a 2.7× gain in eleven months.

I'd be careful with any tighter extrapolation than that — the earlier figures I originally used came from banner text and undated tweets, and the banners sat unchanged for months at a time, so a smooth curve fitted through them is fiction. The direction is solid and the driver Apify names is explicit: actors used as tools by AI agents.

Meanwhile new actors keep arriving faster than the pie grows. Which brings it back to the price table — that race isn't won on volume.

What I'd tell myself at the start

Check demand before feasibility. I built several actors for sources that were technically satisfying and commercially empty — open government data is the sharpest version of this, where the API answers instantly, there's no anti-bot, nobody has taken the niche, and the niche leader has three users.

Price against the incumbent SaaS, not against the cheapest scraper.

Pick one vertical and stay in it. My own profile still shows the opposite — travel, Southeast Asian marketplaces, contact verification, B2B lead feeds — which is three themes too many, and cross-linking them is a patch, not a fix.

And verify your own measurement before you build a strategy on it. Mine survived, but only after the correction: the shape of the price table is the same, the magnitudes were not.

The actors are at apify.com/lergassy. The measurement is two endpoints and an afternoon — go compute it for your own niche, and take the maximum of the paid tiers.

Scraping Southeast Asian Marketplaces in 2026 — Shopee and Tokopedia Without a Browser

Nikita Iakovlev — Fri, 04 Sep 2026 06:30:43 +0000

Southeast Asia's e-commerce market runs on platforms most Western developers have never opened. Shopee is the region's largest marketplace and also operates in Taiwan and Brazil. Tokopedia is Indonesia's best-known domestic marketplace, now 75% owned by ByteDance and merged with TikTok Shop. Neither has a public product API, and both moved to client-side rendering, which is where most scraping attempts stop.

They shouldn't. Here's a product pulled this month over plain HTTP, no browser, no token:

{
  "type": "product",
  "title": "Oleh Oleh Bali - Kopi Bubuk Bali Cap Kupu Kupu Bola Dunia kemasan baru 400 Gram",
  "url": "https://shopee.co.id/...-i.21681717.6213583834",
  "shopId": "21681717",
  "itemId": "6213583834",
  "country": "ID",
  "marketplace": "Indonesia",
  "currency": "IDR",
  "price": 74700,
  "rating": 4.9,
  "location": "Denpasar",
  "position": 1,
  "searchTerm": "kopi bali"
}

The rendering problem, and the way around it

In mid-2026 Shopee moved product pages to a client-rendered application whose internal API is guarded by a JavaScript-signed token. The token is computed in the browser. Reproducing it means either running the JavaScript or reverse-engineering a signing routine that changes.

This is why most Shopee tooling today returns only what a product card shows, and marks description, brand and image gallery as "browser only". It's also why those tools are expensive per row — a headless browser costs orders of magnitude more compute than an HTTP request.

The way around it is that Shopee still serves a server-rendered version of the same page to crawlers. Search results, category pages and shop pages come back as HTML with the full product card in it: title, price, original price, discount, rating, units sold, shipping location, badges, image. No browser, no token, no login.

A search for forty products finishes in under a minute on a few megabytes of traffic. The same run through a headless browser burns orders of magnitude more compute and is more likely to be blocked, because a browser fingerprint is a much larger attack surface than an HTTP GET.

Tokopedia is a different shape with the same conclusion. Its review data lives behind its own data endpoint, which answers directly. Thousands of reviews come back in minutes without rendering anything.

The eight live Shopee marketplaces

This is the part that makes Shopee worth more than a single-country scraper:

Country	Domain	Currency
Indonesia	shopee.co.id	IDR
Singapore	shopee.sg	SGD
Malaysia	shopee.com.my	MYR
Philippines	shopee.ph	PHP
Thailand	shopee.co.th	THB
Vietnam	shopee.vn	VND
Taiwan	shopee.tw	TWD
Brazil	shopee.com.br	BRL

Shopee's Latin American footprint has been shrinking: Argentina and the Mexican local marketplace went in 2022, and Chile and Colombia closed on 30 October 2025, leaving cross-border only. Watch out for this — shopee.cl and shopee.com.co still answer with HTTP 200, because a shared CDN shell responds whether or not there's a marketplace behind it. A status code is not evidence that a storefront exists; a product row is. Requesting a Chilean search through the actor now returns an upstream error, which is the honest answer.

Same product, eight catalogues, eight price levels, eight sets of competitors. Every row carries country, marketplace and currency, so one dataset can hold several countries side by side without a currency column guessing game. Prices come back in the marketplace's own currency, never converted — conversion is a decision with a date attached and doesn't belong in extraction.

The cross-market comparison is the analysis nobody does. A product selling at IDR 74,700 in Indonesia and its equivalent in Vietnam and the Philippines tells you which market a seller is underpricing.

What Tokopedia reviews look like

{
  "type": "review",
  "shopName": "Makarizo Advisor",
  "reviewId": "2289257709",
  "rating": 5,
  "reviewText": "Semoga cocok yaaa",
  "reviewTime": "Hari ini",
  "reviewerName": "P***w",
  "isAnonymous": true,
  "sellerReply": "",
  "productName": "Makarizo Advisor Hair & Scalp Tonic 60mL",
  "productUrl": "https://www.tokopedia.com/makarizo-advisor/..."
}

Two details worth knowing before you build on this.

Buyer names are masked at the source. P***w is what Tokopedia publishes, not what the scraper redacted. There is no personal data to collect here even if you wanted it, which makes this dataset unusually clean from a compliance standpoint.

Times are relative. Hari ini means "today". 2 hari lalu means two days ago. If you're building a time series you have to resolve those against the scrape timestamp at collection time, because the phrase stops being meaningful the moment you store it.

Review text is mostly Indonesian, which is a feature for anyone doing multilingual sentiment work and a problem for anyone assuming English.

Who actually buys this data

Sellers benchmarking price. The single most common use. What are the twenty competing listings for my SKU priced at today, in each market, with what discount and what rating.

Brands watching unauthorised resellers. Search your own brand across all eight marketplaces and see who's listing it, at what price, from which warehouse. Grey-market pricing destroys distributor relationships and is invisible from the brand's side without this.

Product research for sourcing. Units sold, rating and review counts across a category tell you where demand actually sits, before committing inventory.

Market entry research. A brand deciding whether to launch in Vietnam or the Philippines wants the current price and competitor density in both. Two runs, one afternoon.

Review mining. Tokopedia's reviews are a large public corpus of Indonesian-language consumer opinion, attached to specific products and star ratings. For a brand selling in Indonesia, that's the actual voice of the customer, and it's not on Twitter.

Running them

Shopee search:

{
  "mode": "search",
  "country": "ID",
  "searchTerms": ["kopi bali"],
  "maxItems": 100
}

Tokopedia reviews, an entire shop at a time:

{
  "startUrls": [{ "url": "https://www.tokopedia.com/makarizo-advisor" }],
  "maxReviewsPerShop": 1000,
  "reviewsSort": "newest"
}

From the API:

curl -X POST "https://api.apify.com/v2/acts/lergassy~shopee-scraper/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{"mode":"search","country":"ID","searchTerms":["kopi bali"],"maxItems":50}'

Shopee also takes startUrls for category, shop or search URLs you already have, and a productDetail mode that takes shopId/itemId pairs. Tokopedia takes sort order and star-rating filters, including photos-only.

What you don't get

Shopee product descriptions and galleries are partial. The listing pages carry the full card. The detail page carries the category path and gallery size, and the full description only when Shopee serves its structured record for that product. This is the residue of the token problem above, and any tool claiming complete detail data over HTTP for every product is claiming something I couldn't reproduce.

No stock quantities. Neither platform publishes real inventory, only "units sold" as a cumulative badge.

No seller contact details. Shop pages give a name, rating and response rate. Anything beyond that isn't published.

Rate limits are real. These are large platforms with anti-bot infrastructure. Modest concurrency works; hammering a search endpoint gets you a challenge page instead of JSON, and the correct response is to slow down rather than to escalate to a browser.

That last point deserves a caveat you rarely see in posts like this one: on platforms in this region, an endpoint that answers cleanly today can start returning an anti-bot page next month with no warning and no change on your side. It has happened to me on a neighbouring marketplace while writing this. Build the retry and the alert before you build the analysis.

The niche, honestly

E-commerce product scraping is one of the most crowded categories on the Apify Store — 147 new actors in six months, median price around $4 per 1,000 rows, and roughly $6,300 a month in total third-party revenue across the whole category. It is not where the money is.

What's differentiated here isn't "scrapes an e-commerce site". It's eight marketplaces in one dataset, no browser in the pipeline, and an Indonesian-language review corpus that the English-speaking tooling ecosystem largely ignores. If you're building in this category, that's the level the difference has to exist at.

Actors: Shopee Scraper · Tokopedia Reviews Scraper

Related: Mercado Libre Scraper for the same job across Latin America.