DEV Community: Nikita Iakovlev

Capterra and Software Advice Share One Review Database and Disagree by 13,058 Reviews

Nikita Iakovlev — Thu, 15 Oct 2026 05:12:38 +0000

I run a visa agency in Bali. The half of my week that has nothing to do with visas goes into Apify Actors, and one of them collects B2B software reviews — Capterra, Software Advice, GetApp and TrustRadius — and returns every site with the same field names.

That last part started as a convenience feature — four scrapers, four field vocabularies, one translation layer I got tired of rewriting. What it turned into was a measuring instrument: once the rows line up you can put the sites side by side and see where they disagree. They disagree a lot, and not in the way I expected.

Here is the measurement. Product: Slack. Builds 0.1.39 and 0.1.40, runs on 13–15 October 2026, no proxy.

Three sites, one database, three different totals

Ask for Slack on Capterra, Software Advice and TrustRadius, ten reviews each, and the product header on those rows reads:

source           productReviewCount   productRating   ratingScale
capterra                     11,166          4.4643            5
softwareadvice               24,224          4.6639            5
getapp                       24,224          4.6639            5
trustradius                  10,268               9           10

Software Advice and GetApp agree to four decimal places. Capterra — the biggest brand of the three, the one whose logo everyone recognises — reports 11,166 reviews and an average two tenths of a star lower.

My first guess was that these are simply three separate review pools that happen to overlap in subject matter. That guess is wrong, and the reviewId field is what killed it:

softwareadvice  Capterra___7219769
softwareadvice  Capterra___7224596
getapp          Capterra___7138404
capterra        Capterra___7209002

Those are reviews scraped from softwareadvice.com and getapp.com. They carry Capterra identifiers because all three sites are Gartner properties running on one review database. The IDs interleave, too — 13 of the 20 Software Advice IDs I collected fall inside the numeric range of the 45 Capterra IDs. It is one sequence, handed out to one pool of reviews, served through three front doors with three different filters on the way out.

So the 13,058-review gap is not a data gap. It is an editorial decision about which of the shared reviews each brand chooses to display, and it is invisible if you scrape one site.

The same review, twice, with a different reviewer on it

Across 45 Capterra reviews and 20 Software Advice reviews I caught exactly one ID on both sites: Capterra___7224596. Field by field:

field                     capterra                softwareadvice
title                     "Slack is the best…"    "Slack is the best…"   same
publishedAt               2026-10-04              2026-10-04             same
pros / cons               identical text          identical text         same
overallRating             5                       5                      same
easeOfUseRating           5                       5                      same
functionalityRating       5                       5                      same
valueForMoneyRating       5                       5                      same
customerSupportRating     4                       4                      same
incentivized              true  (NominalGift)     true  (NominalGift)    same
reviewerIndustry          Construction            Construction           same
--
reviewerName              "Chris S."              "Chris S——" (full)     differs
reviewerJobTitle          "General Manager"       null                   differs
reviewerCompanySize       "51-200 employees"      null                   differs
reviewerCompanySizeCode   null                    "D"                    differs
timeUsed                  null                    "More than 2 years"    differs
usageFrequency            null                    "Daily"                differs

Everything a reviewer typed is byte-identical. Everything about the reviewer is split between the two pages. Capterra publishes the job title and a readable company-size range and abbreviates the surname to an initial. Software Advice publishes the reviewer's full first and last name — I'm redacting it here, since the point is the disclosure difference and not this person — drops the job title, and replaces the company-size range with a single letter.

That letter is worth a paragraph of its own. Software Advice and GetApp publish A through I where Capterra publishes 11-50 employees, and neither site publishes a legend. The temptation is to map it yourself: A is surely the smallest bucket, I the largest, nine letters for nine ranges. I didn't, because on this one shared review the letter is D and the range Capterra prints for the same company is 51-200 employees — a single data point, enough to build a guess on and nowhere near enough to ship one. So the Actor passes the letter through in reviewerCompanySizeCode and leaves reviewerCompanySize empty. One confirmed pair is a hint, not a mapping.

What this does to de-duplication

If you are merging Capterra, Software Advice and GetApp, reviewId is a real join key and you should use it. But the merge is not idempotent in either direction, because neither row is a superset of the other. Prefer the Capterra row and you lose usageFrequency and timeUsed. Prefer the Software Advice row and you lose the job title and the human-readable company size. The only correct merge is field-level: for each column, take whichever side is non-null.

TrustRadius stays out of this entirely. Its IDs are slugs with a timestamp — slack-2026-08-05-00-29-43 — its reviewers are its own, and across ten rows it returned zero sub-ratings while filling in reviewerJobTitle on all ten ("Full-stack developer", "Sr. Director engineering", "Senior Salesforce Administrator"). Different site, different database, genuinely additive sample.

And GetApp is no longer a review source in any useful sense. Its product totals are complete and match Software Advice exactly, but the product page now publishes about six short quotes instead of full reviews, with no ratings at all. Of the five rows I got back, three had pros filled and cons empty, two the reverse — the page quotes one half of each review and doesn't say which half. The first one reads "However, managing constant notifications can be overwhelming at times…", which is a complaint, sitting in a field called pros, because that is the only text GetApp published for it. Use GetApp for product-level numbers; don't build a sentiment set from it.

Page one is not a sample

One more thing I only saw because the rows lined up. Ten reviews from each site, in the order the site publishes them, no sort, no filter:

capterra         5 5 3 3 4 5 5 4 4 5     avg 4.3
softwareadvice   5 5 5 5 5 5 5 5 5 5     avg 5.0

Same database. Software Advice's front page was a wall of five stars; Capterra's front page was not. Five of those ten Software Advice reviews were flagged incentivized — NominalGift or VendorReferredIncentivized — against 14 of 45 on Capterra.

The low ratings do exist in the shared pool, they are just not on page one. Asking for them explicitly finds them:

{
  "searchTerms": ["Slack"],
  "searchSites": ["softwareadvice"],
  "maxRating": 3,
  "maxReviews": 10
}

Ten rows: nine 3-star and one 1-star, the oldest from 2025-10-31, six of the ten incentivized. The filter runs while paging rather than after it, so the run keeps reading pages until it has what you asked for or the site runs out:

def matches_filters(row, *, min_rating, max_rating, published_after, published_before):
    # TrustRadius rates out of 10, the Gartner sites out of 5, so filtering happens on
    # the normalized 1-5 value the user is thinking in.
    rating = row.get('normalizedRating', row.get('overallRating'))

That normalization is why maxRating: 3 means the same band of unhappy customers on TrustRadius as on Capterra. overallRating keeps the site's own number, ratingScale says which scale it was, normalizedRating is raw * 5 / scale. A TrustRadius 9 arrives as overallRating: 9, ratingScale: 10, normalizedRating: 4.5, and averaging that against a Capterra 5 is finally a legal operation.

Calling it

Search by name on Software Advice and TrustRadius, by URL on Capterra and GetApp — Capterra's search page refuses automated requests even through a residential proxy, and GetApp publishes no search page at all. Their product pages are served normally, so a URL is all you need:

curl -X POST "https://api.apify.com/v2/acts/lergassy~software-reviews-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "productUrls": ["https://www.capterra.com/p/135003/Slack/reviews/"],
    "searchTerms": ["Slack"],
    "searchSites": ["trustradius", "softwareadvice"],
    "maxReviews": 200,
    "includeProductRow": true
  }'

Thirty reviews across three sites took 46 seconds; thirty from Capterra alone took 90, because reviews come 25 to a page and each page is a request. Budget roughly three to four seconds per page. The Actor is here if you want to run the comparison on your own category — pricing is per row, $0.0012 a review and $0.003 a product summary, and failed URLs come back as error rows at no charge.

When not to use this

Don't use it to count. If you need "how many reviews does this product have", you now know that question has two different right answers depending on which Gartner brand you ask, and any single number you quote is a brand's choice rather than a fact.

Don't use it for G2 — different site, different gate, not covered here. And if what you want is volume over time, note that publishedAt is reliable but the sites' own ordering is not chronological, so a date histogram needs the full history pulled, not a recent slice.

Do use it when the comparison is the point: the same product across four sites, the same columns, and a normalized rating that makes an average mean something. Just merge field-level, and treat page one as a storefront rather than a sample.

Ten YouTube Comments, One Commenter: What a Reply Budget Does to Your Sample

Nikita Iakovlev — Wed, 14 Oct 2026 21:11:36 +0000

I run a visa agency in Bali. The part of my week that has nothing to do with visas goes into Apify Actors, and one of them pulls YouTube comments — one video, a channel, a playlist, or the top results of a search, one row per comment with likes, reply count, the pinned and hearted flags and the commenter's channel.

People use comment data for sentiment: pull the last twenty videos of a competitor, feed the text to a model, read what their audience actually complains about. That works, and the scraping part is not the hard bit. The hard bit is that two of the three obvious assumptions about a comment sample are wrong, and both of them are wrong quietly — you get a full dataset, the right number of rows, no errors, and a conclusion that isn't about the audience.

Here is the measurement, from runs on 11–14 October 2026, build 0.1.39, no proxy.

One flag, two completely different samples

Same video — jNQXAC9IVRw, "Me at the zoo", the first video on YouTube, 20 years of comments on it. Same limit: 10 comments. Sorted by top. The only difference between the two runs is includeReplies.

With replies off, ten comments from ten different people:

"We're so honored that the first ever YouTube video was filmed here!"   4,800,000 likes
"So we're all here again on October 14th 2026 right"                          746
"Hello"                                                                   303,000
"Is anyone else watching this on a random day in October?"                  1,000
"I'm glad this pops up on my feed every couple years"                         467
"This started it all...... bruh 😳"                                       248,000
"this video lived long enough to see gifs come true"                          459
'Seeing "20 years ago" on a YouTube video makes me feel a certain way'    211,000
'This guys is the only person who has the rights to say "first"'          418,000
"First word on Youtube ever:\nAlright"                                    115,000

With replies on — the default — and the same maxCommentsPerVideo: 10:

isReply  text                                                          likes
false    "We're so honored that the first ever YouTube video was…"  4,800,000
true     "San Diego Zoo That's so awesome! I'm so glad the first…"    206,000
true     "Very cool!"                                                  71,000
true     "cool"                                                        57,000
true     "."                                                           54,000
true     "How did I randomly go to this video and see this comment…"   57,000
true     "Im here before this blows up"                                32,000
true     " @Quetzal_  lol what"                                        17,000
true     "yoo here before thousands of likes"                          18,000
true     "Awesome!"                                                    14,000

One top-level comment and nine replies to it. Ten rows, one conversation, and four of them are a single word or a full stop. Run that through a sentiment model and you have measured one thread on a video with 2.4 million comments.

This is not a bug, and the input field says so in six words — "replies count too when included". It is the interaction that bites. The collector walks the comment list depth-first: take a top-level comment, immediately follow its replies, then move to the next one. Both compete for the same budget:

if (keep(c)) { await pushComment(c, base, null); kept += 1; }
if (includeReplies && c.replyCount && page.replyTokens.has(c.commentId) && kept < wanted) {
    // …up to maxRepliesPerComment replies, each one incrementing the same `kept`
}

The comment at the top of "Me at the zoo" has 985 replies and maxRepliesPerComment defaults to 20. So the first comment plus its reply cap is 21 rows — more than the sample I asked for. The depth-first order is the right default (a thread stays together, and a keyword match inside a thread is still a match), but it means the limit you set is a row budget, not a voice budget.

The arithmetic to set it deliberately:

rows ≈ topLevelComments × (1 + maxRepliesPerComment)

Twenty distinct opinions with their threads at the default reply cap is 420 rows, not 20. If you want breadth, includeReplies: false is the whole fix; the reply count stays on every row, so you still know which comments started something. If you want threads, raise the comment limit by a factor of 21 and expect to pay for it — the Actor bills per comment row, so replies are rows you buy.

Then I ran the identical input four days running

Four runs, 11 to 14 October, same JSON every time: that video, maxCommentsPerVideo: 50, replies on, sort: top. Four datasets of exactly 50 rows.

	Rows	Distinct comments shared with the 14 Oct run
11 Oct	50	21
12 Oct	50	21
13 Oct	50	21
14 Oct	50	—

Across the four runs there are 121 distinct comments, and only 21 appear in all four. Every run returned the same three top-level comments' worth of structure — 3 top-level, 47 replies, the first thread always the 4.8-million-like one — but comments two and three changed almost every day:

11 Oct: "Randomly got recommended this at 11:16 PM…"          3,300 likes
12 Oct: "You will never know why this comment got…"          24,000
13 Oct: 'Seeing "20 years ago" on a YouTube video…'         211,000
14 Oct: "This started it all...... bruh 😳"                  247,000

"Top comments" is a ranking YouTube recomputes, not a stable list, and on a video this size the tail of the top-50 reshuffles daily. The same comment also came back with 417,000 likes on one day and 418,000 on the next, which is the honest behaviour — counts move — but it means a comment is identified by commentId, never by its text plus its like count. If you are tracking a video over time, dedupe on commentId and expect the set to drift under you. If you are sampling once for sentiment, know that you sampled a top-50, not the top-50.

And the dates are arithmetic, not data

YouTube never shows a comment date. It shows an age: "6 years ago", "9 hours ago", "2 days ago". The Actor keeps that string in publishedAgo and also subtracts it from the run clock into publishedAt, because a date column is what every spreadsheet downstream wants. Here is what that does to one comment — the same commentId, five runs:

Run started (UTC)	`publishedAgo`	`publishedAt`
11 Oct 00:33	6 years ago	2020-10-10
12 Oct 21:28	6 years ago	2020-10-12
13 Oct 00:28	6 years ago	2020-10-12
14 Oct 00:45	6 years ago	2020-10-13
14 Oct 21:05	6 years ago	2020-10-14

Two runs on the same calendar day, two different dates, because the subtraction uses the run's clock and a year is 365.25 days. The mapping is not even one-to-one in either direction: the 12 Oct and 13 Oct runs, three hours apart, both landed on 2020-10-12, while the two 14 Oct runs landed a day apart.

The drift is the small problem. The real one is the bucket: "6 years ago" is YouTube rounding down, so the comment's true age is somewhere in [6 years, 7 years). publishedAt: 2020-10-13 is the newest date the comment could have; it could genuinely be from October 2019. For fresh comments the label is tight — "9 hours ago" pins the day — and for anything older than a month it is a floor, not a date. So: filter and group on publishedAgo buckets when the comments are old, use publishedAt for recency windows on fresh comments, and never diff two runs' publishedAt columns and conclude anything.

One more small trap in the same field: an edited comment is labelled "6 years ago (edited)", suffix and all. Across those four runs there were 22 such rows and four distinct label strings — 6 years ago (edited), 5 years ago (edited), 5 months ago (edited), 1 month ago (edited). Matching publishedAgo == "6 years ago" silently drops them. The date parser strips the suffix before doing its arithmetic; a GROUP BY in your sheet does not.

Calling it

One request, and the run-sync endpoint hands back the rows directly — this is the exact call that produced the ten-distinct-commenters sample above:

import requests

rows = requests.post(
    "https://api.apify.com/v2/acts/lergassy~youtube-comments-scraper/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={
        "videoUrls": ["https://www.youtube.com/watch?v=jNQXAC9IVRw"],
        "maxCommentsPerVideo": 10,
        "includeReplies": False,      # breadth, not one thread
        "sort": "top",
        "language": "en",
    },
).json()

Swap videoUrls for channelUrls, playlistUrls or searchQueries and maxVideosPerSource decides how many videos each source contributes; every row carries source so you know where it came from. The Actor is here: YouTube Comments Scraper.

When not to reach for it. If you need exact comment timestamps, nothing scraped from the watch page will give them to you — the official Data API returns real publishedAt values, at the cost of a key and a daily quota. If you need all comments of a viral video, the arithmetic is unkind: twenty comments per request means 2.4 million comments is 120,000 round trips. And if you want to post, reply or moderate, this is read-only; that is the API's job.

The lesson I keep relearning on other people's data: a limit is only meaningful once you know what the thing being limited is. Ten rows looked like ten opinions right up until I printed the isReply column.

One Minute of Audio, Three Runs: 13 Segments, Then 9, One of Them 21.94 Seconds Long

Nikita Iakovlev — Wed, 14 Oct 2026 13:11:52 +0000

I run a visa agency in Bali. The half of my week that has nothing to do with visas goes into Apify Actors, and one of them is a transcriber: give it an audio or video URL, it runs Whisper large v3 and hands back a transcript, timed segments, SRT or WebVTT subtitles, and optional retrieval chunks that keep their timecodes.

The transcript is the part everyone checks. It is also the part that is nearly stable. What is not stable is the structure around it, and almost every pipeline I have seen built on a transcription API quietly assumes the opposite.

Three runs of the same file

One sample: a 60.01-second m4a of someone reading the opening of Pride and Prejudice three times over. Same Actor build (IdudfuIRo8hXFkRgH), same input — quality: fast (Whisper large v3 turbo), language: en. Run on 13 October, 14 October, and again on 14 October twelve hours later.

Run	Segments	Words	Chars	Longest segment	Longest segment text
13 Oct 00:24	9	214	1162	21.94 s	414 chars
14 Oct 00:39	13	215	1163	5.32 s	100 chars
14 Oct 13:09	9	214	1162	21.94 s	414 chars

The words are fine. A diff of the two transcripts that differ at all is two tokens out of 215:

- neighborhood. This truth is so well fixed
+ neighborhood, this truth is so well fixed

- property of some one or other
+ property of someone or other

A comma that became a full stop, and a space that appeared inside "someone". If you are measuring word error rate, this file is stable to 0.5%.

Now look at the structure. Between those same two runs the segment count moved 44%, and the longest unit of output went from five seconds to twenty-two.

And here is my favourite detail, which has nothing to do with run-to-run variance: the recording loops the same sentence three times, so one run contains the same audio three times over. The 14 October run transcribed it as "someone or other" at 15.30 s and as "some one or other" at 34.50 s and 53.74 s. Identical audio, one run, two spellings. Do not build a deduplicator that compares transcript strings.

Why a 21.94-second segment exists

Whisper works in windows of up to 30 seconds and returns segments, not words. How it carves a window into segments depends on where it thinks the sentences end — and that decision is downstream of sampling, so it moves between runs even when the text does not. Nine segments and thirteen segments are both legitimate readings of the same sixty seconds.

The 9-segment run put the final third of the file into one object:

{
  "start": 38.06,
  "duration": 21.94,
  "text": "It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife. However little known…"
}

414 characters with a single start time. Pass segments straight into an SRT writer — the obvious implementation, and the one I shipped first — and a viewer gets a paragraph filling the screen for twenty-two seconds. I caught it on a jfk.flac test last September: 108 characters held for 11 seconds, valid SRT, unwatchable.

So subtitles are not built from segments here. They are rebuilt from them:

LINE_MAX = 42       # the Netflix/BBC reading norm
CUE_MAX = 2 * LINE_MAX
CUE_SECONDS_MAX = 7.0

count = max(1, math.ceil(len(text) / CUE_MAX), math.ceil(duration / CUE_SECONDS_MAX))

Each segment is split into at least that many cues, preferring breaks after a comma or a sentence end, then timed by dividing the segment's duration in proportion to characters. That proportional split is an approximation and I want to be honest about it: the API gives segment timings only, no word timestamps, so there is nothing better available without a forced-alignment pass.

Those 9 segments came out as 21 cues, longest 4.51 seconds:

16
00:00:38,060 --> 00:00:42,405
It is a truth universally acknowledged,
that a single man in possession of a good

17
00:00:42,405 --> 00:00:46,750
fortune, must be in want of a wife.
However little known the feelings or views of

One measured line came out 45 characters rather than 42: the wrapper needs a break point that leaves both halves under the limit, and a long unbroken clause has none, so it falls back to the nearest space. Three characters over beats hyphenating.

The segments also do not cover the file

Add up the durations: 57.12 s, 58.12 s, 58.30 s against a 60.01-second file. Between two and three seconds of every run live in the gaps between segments. On a reading with real pauses that is correct behaviour.

Sometimes it is not. Whisper can skip to its next 30-second window and silently drop the speech in between, and the status still says ok — a 134-second reading came back with roughly 32 seconds missing in one of my golden checks. The transcript reads fluently. Nothing in the output says a third of it is gone.

The Actor now scans for holes of 6 seconds or more, cuts each one out of the prepared audio, and recognises it again with the preceding text as a prompt. Recovered segments are only kept if the service itself is confident it heard speech:

if float(item.get('no_speech_prob') or 0) > 0.6 or float(item.get('avg_logprob') or 0) < -1.0:
    continue

That filter is not optional. Whisper hallucinates over silence, and its favourite invention is a polite "Thank you." Re-check a genuine pause without the confidence gate and you fill real silences with phantom gratitude, which is worse than the gap.

What this breaks downstream

Retrieval chunks inherit the instability. Chunks are assembled from segments, breaking on sentence ends. The 9-segment run gave me three chunks starting at 0, 25.54 and 38.06 seconds, with 491, 255 and 414 characters. The 13-segment run of the same audio cannot give the same boundaries — it has different sentence ends to break on. So: never use chunk index or chunk start time as a stable key in a vector store. Re-embed whole files, keyed on the file, and delete the old vectors rather than diffing them.

Never diff two transcription runs. Any sane diff of these three runs reports changes. One is a real text change of two tokens; the rest is re-segmentation noise. If you need change detection on a recurring recording, compare normalised text — lowercased, punctuation stripped, whitespace collapsed — and never segment counts.

Speaker labels are per-run identities. They are computed inside the Actor from the audio (Silero VAD to cut phrases at pauses, 3D-Speaker CAM++ for one voice print per phrase, then clustering). Clustering happens per file, so Speaker 1 in Monday's run and Speaker 1 in Tuesday's run are not promised to be the same human. For cross-file identity you need enrolled voice prints, which is a different product.

One counter-example, because I only have two data points on it: the other URL in those runs was a 19.02-second YouTube clip, and it came back byte-identical on both days, with segment boundaries on whole seconds (0, 4, 12, 16). Short clean clips may well be stable. I would not build on the assumption.

Calling it

import requests

rows = requests.post(
    "https://api.apify.com/v2/acts/lergassy~speech-to-text/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={"urls": ["https://example.com/episode-12.mp3"],
          "language": "en", "subtitleFormat": "srt",
          "chunkForRag": True, "chunkSize": 600},
).json()

The Actor is Speech to Text. One row per file, no key to set up; the third run in this post cost $0.011 — one billed minute at $0.009 plus $0.002 for the file. Minutes are charged per started minute, so that 19-second YouTube clip billed a full minute, and failed files cost nothing. Direct file links pay no proxy fee; a YouTube, TikTok or Instagram page does, because its audio has to come through a residential exit.

When it is the wrong tool. Live or streaming audio — this is batch, one row per finished file. Word-level timestamps or karaoke-style alignment: segment timings are all the API exposes, and my cue splitter interpolates rather than aligns. Cross-recording speaker identity, per the point above. And if you already run Whisper on your own GPU, nothing here beats that on price; what you are buying is the download, the ffmpeg prep, the splitting of long files, the gap re-check and the retries.

The lesson I keep relearning: when a model returns structure as well as content, the content is the part people test and the structure is the part they build on. Check which of the two your pipeline actually depends on — and if the answer is structure, measure it twice on the same input before you ship.

Asking KAYAK for 10 Flights Instead of 50 Made the Same Seat 30% Dearer

Nikita Iakovlev — Wed, 14 Oct 2026 05:13:08 +0000

I run a visa agency in Bali; the other half of my week goes into Apify Actors. One of them reads KAYAK flight searches into rows: price, who is selling the ticket — airline or agency — flight numbers, times, stops, layovers, aircraft, and a booking link.

A metasearch engine is worth scraping precisely because of that seller column. Google Flights tells you a route costs $57; KAYAK tells you which of eleven sellers is quoting $57 and links to their checkout. So the seller field had better be right.

Last week I noticed it was not — sometimes. Four runs of the same search disagreed about the price, and not randomly: they disagreed in one direction.

Four runs, one search

DPS → SIN, departing 15 March 2027, one way, economy, sorted by KAYAK's "best". Run on 13 and 14 October, two of them three minutes apart. Here is the same Qantas-coded itinerary in each:

Run	Row cap	Took	`price`	`provider`	`providersForThisFlight`
13 Oct 00:18	30	23 s	115	Kiwi.com	2
13 Oct 00:21	50	34 s	80	Justfly.com	28
14 Oct 00:34	30	74 s	115	Kiwi.com	19
14 Oct 00:36	50	23 s	99	Justfly.com	30

Across the 24 itineraries the 13 October pair had in common, the second run was cheaper on 18, identical on 6, and dearer on none. Median gap 7.1%, biggest 43.8%.

A price that only ever moves one way is not noise. It is a partially-read field.

KAYAK's search is a poll, and the payload grows

KAYAK's results page does not fetch a list of flights. It opens a search and then polls it:

POST /i/api/search/dynamic/flights/poll

Each answer carries results[], and each result carries bookingOptions[] — one entry per seller quoting that itinerary, with its own price and deep link. The Actor reads those answers directly instead of scraping the rendered DOM, and picks the cheapest option per itinerary:

const options = (r.bookingOptions || []).filter((o) => o?.displayPrice?.price);
options.sort((a, b) => a.displayPrice.price - b.displayPrice.price);
const chosen = input.allProviders ? options : [options[0]];

That is correct code over an incomplete input. Sellers do not all answer KAYAK at once, so an early poll answer carries one or two bookingOptions per itinerary and a later one carries thirty. price is a minimum over whatever has arrived — and the minimum of a growing set, taken early, is too high every time, by construction.

The run log of the two 13 October runs says it plainly — same search, 180 seconds apart:

00:19:17  DPS-SIN: search API 2 calls, 36 flights in 5 s (page 1 s, search API +2 s)
00:19:17  DPS-SIN 2027-03-15: 29 rows (KAYAK found 29 itineraries)

00:21:55  DPS-SIN: search API 2 calls, 63 flights in 15 s (page 2 s, search API +8 s)
00:21:55  DPS-SIN 2027-03-15: 50 rows (KAYAK found 488 itineraries)

"KAYAK found 29 itineraries" against "KAYAK found 488" — the search's own totalCount is a snapshot too, read five seconds in instead of fifteen.

The stop condition was mine, not KAYAK's

Why did one run stop at five seconds? Because of this line in my wait loop:

if (n >= input.maxResults) break;   // n = flights seen so far

Perfectly reasonable as a row-count guard: the buyer asked for 30 rows, 36 itineraries have arrived, stop waiting. Except the poll answer that first clears your row cap is also the thinnest one on prices. The row cap had quietly become the price-completeness setting, and nothing in the input hinted at that.

Controlled test, same minute

Two runs started in the same second from one account, identical input except the cap, both on 4 GB:

{"origin": "DPS", "destination": "SIN", "departureDate": "2027-03-15", "maxResults": 10}
{"origin": "DPS", "destination": "SIN", "departureDate": "2027-03-15", "maxResults": 50}

Ten itineraries appear in both. Six are cheaper in the 50-row run, four are identical, none is dearer:

Flight	cap 10	sellers	cap 50	sellers	gap
JQ86	$74 Kiwi.com	3	$57 Jetstar	10	29.8%
QF5659	$115 Kiwi.com	8	$91 Justfly.com	32	26.4%
GA842	$129 Kiwi.com	5	$106 Mytrip	18	21.7%
8B553	$99 Kiwi.com	1	$89 Super.com	7	11.2%
SQ937	$151 Singapore Airlines	10	$141 Super.com	30	7.1%
KL836	$83 KLM	9	$83 KLM	9	0%

Median seller count 4 against 8. The $57 Jetstar fare — the actual cheapest way to fly Bali to Singapore that day — is simply absent from the ten-row run. Jetstar had not answered yet.

The bias has a name, and it is Kiwi.com

The seller column is where this gets worse than a wrong price, because the error is not evenly spread. Across all six runs, I split every row by how many sellers had reported:

rows with 1–2 sellers reported (55):  Kiwi.com 45 · CheapOair 3 · VietJet 3 · Booking.com 3 · PAL 1
rows with 10+ sellers reported (80):  Super.com 28 · Booking.com 24 · Mytrip 18 · Justfly 3 · airlines 4

Kiwi.com is named the cheapest seller in 45 of 55 thin rows and in none of the 80 full ones. Kiwi is not cheap here; Kiwi is fast. Read a search early and you are not measuring who sells the route cheapest, you are measuring whose API answers KAYAK first — and then publishing it as a market finding. That is the kind of error that survives review, because every individual row looks plausible.

What to do about it

providersForThisFlight is the confidence field. It ships on every row and I had been treating it as trivia. One or two sellers on a popular route means you caught the search mid-fill; ten-plus means the field has settled. For price monitoring, store it next to the price and drop the thin rows rather than averaging them in.

Never compare runs taken with different caps. Two of my four runs were a fair comparison; the other pair was comparing a five-second read with a fifteen-second one.

The fix I'm making is to stop treating row count as the exit condition. The poll payload already says status: "complete", and seller counts stop growing before the itinerary list does — waiting on those instead of on maxResults costs about ten seconds per search and removes the whole class of error. Until that build is out, pass a high maxResults even if you only intend to keep the first few rows; rows beyond the cap are not charged, and the extra wait is what buys you the right price.

Three other things this Actor had to learn about KAYAK

The layover belongs to the arriving segment. KAYAK attaches the connection wait to the segment that ends at the connecting airport — EI106 JFK→DUB carries the 75 minutes spent in Dublin — so the obvious read gives you every layover airport one hop early.

currency= is a suggestion. Ask kayak.com for a route in SAR and the prices come back in dollars, correctly labelled as dollars by the payload and not by your request. Rows now convert at a daily rate and keep priceOriginal, currencyOriginal and exchangeRate so nothing is silently rebadged.

Whether you get a payload at all is about the IP, not the route. From today's log, the 50-row run on its first address: requests 3 (api 0, failed 52) — page loaded, search never started. A fresh address returned 64 flights in 22 seconds.

Here is the Actor: Kayak Flights Scraper. Calling it is one request; allProviders gives you every seller of an itinerary as its own row, which is the honest shape for seller research:

import requests

rows = requests.post(
    "https://api.apify.com/v2/acts/lergassy~kayak-flights-scraper/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={"origin": "DPS", "destination": "SIN", "departureDate": "2027-03-15",
          "maxResults": 50, "allProviders": True},
).json()

When not to reach for it. A price calendar or an "is today cheap" verdict is Google's data, not KAYAK's. Seat maps, fare rules and baggage allowances are not on the results page at all. And for one route tracked hourly, a metasearch read costs 30–90 seconds of browser time per search — an airline's own API, where you can get one, is steadier.

The lesson I keep paying for: when a site streams its answer, every field you read has a timestamp, whether or not it carries one. Stopping early is not a performance choice. It is a data choice.

Three Supermarkets Spell One Brand Three Ways, So My Price Comparison Never Happened

Nikita Iakovlev — Tue, 13 Oct 2026 21:21:48 +0000

Visas pay my bills — I run an agency in Bali. The other half of my week goes into Apify Actors, and one of them reads the three Australian supermarkets in one run: ask for full cream milk and you get what Coles sells, what Woolworths sells and what ALDI sells, in the same columns.

I assumed the hard part would be getting in. It partly is — Coles sits behind a bot check — but Woolworths answers its own search service in a fraction of a second and ALDI runs a plain product API. A 46-product run finishes in six seconds. The hard part turned out to be the question that sounds trivial: are these two rows the same product?

The run that showed the problem

One keyword, three chains, 20 products each: Coles returned 20, Woolworths 20, ALDI 8. The comparison output is supposed to collapse those into one row per product with three prices side by side.

What came out: 38 comparison rows, of which only 7 had more than one chain on them. Everything else was a "single" — one chain, no comparison. For a tool whose entire reason to exist is putting three prices on one row, that is close to a silent failure: the rows look fine, the schema is right, and the field you came for is null.

Three causes, all in the data rather than the network.

1. One brand, three spellings

Here is the brand field for the same 2L bottle of a2 full cream milk, as the three chains publish it:

Chain	`brand`	`name`	`price`
Coles	`a2 Milk`	a2 Milk Full Cream Milk	7.30
Woolworths	`a2`	a2 Milk Full Cream 2L	7.30
ALDI	`A2 THE A2 MILK COMPANY`	A2 THE A2 MILK COMPANY A2 Full Cream Milk 2L	7.29

My matcher required the brands to agree before it would put two rows together, and "agree" meant this:

const brandKey = (row) => String(row.brand || '').toLowerCase().replace(/[^a-z0-9]/g, '');
// 'a2milk' · 'a2' · 'a2thea2milkcompany'

Three different strings, so three groups of one. The one deliberate exception — house brands, where Coles Full Cream Milk, Woolworths Full Cream Milk and ALDI's Farmdale are treated as comparable — was doing all the visible work. Everything branded fell apart.

The fix is to compare brands as sets of words instead of strings, with the corporate boilerplate dropped:

const BRAND_NOISE = new Set(['the', 'company', 'co', 'pty', 'ltd', 'group', 'brand', 'holdings']);
const brandTokens = (b) => [...new Set(String(b || '').toLowerCase()
    .replace(/[^a-z0-9]+/g, ' ').trim().split(/\s+/).filter((w) => w && !BRAND_NOISE.has(w)))];

export function brandsAgree(a, b) {
    const ta = brandTokens(a); const tb = brandTokens(b);
    if (!ta.length || !tb.length) return false;
    const [shorter, longer] = ta.length <= tb.length ? [ta, tb] : [tb, ta];
    return shorter.every((w) => longer.includes(w));   // {a2} ⊆ {a2, milk}
}

{a2} is a subset of {a2, milk}, and A2 THE A2 MILK COMPANY reduces to {a2, milk} once the and company go. One row, three prices: $7.30, $7.30, $7.29.

The same fix caught something I was not looking for: Coles files the 750 mL Farmhouse Gold bottle under brand Farmhouse, Woolworths under Farmhouse Gold. Subset, so they now meet — $3.90 at Coles against $3.40 at Woolworths, a 13% gap, because Woolworths had it marked down that day. Invisible in every earlier run.

Where I stopped. Same brand, 1.5 L size: Coles calls it Pauls, Woolworths calls it Farmhouse Gold. No shared word, so those stay two single rows unless you ask for the looser equivalent mode. I could close it by reading the brand out of the name — Coles' is "Pauls Farmhouse Gold Full Cream Milk", so the words are right there. I decided not to: Woolworths also sells "Farmhouse Gold Unhomogenised Milk 1.5L", and a matcher that infers brand from names will pair that with the wrong jug. A missing comparison is a gap a buyer can see; a wrong comparison is a number they will believe.

2. Word overlap pairs two different jugs

Which brings me to the opposite failure, one row down in the same output: colesName "Pauls Zymil Lactose Free Low Fat Milk", woolworthsName "Pauls Zymil Lactose Free Full Cream Milk 2L", matchConfidence 0.81, priceGapPercent 14.

Same brand, same normalised 2000 ml, four words in common out of eight, so the overlap score cleared the threshold and the Actor announced a 14% saving between two products that are not the same product. There was a second one: Coles' plain long-life milk paired with ALDI's lactose-free long-life, reported as 12%.

Groceries declare their differences in the title, and the declarations come in two shapes that need opposite treatment:

// Exclusive: conflict only when BOTH names declare a value and the values differ.
// A chain that does not write "full cream" in the title is normal.
const ATTR_EXCLUSIVE = {
    fat: { full: /\b(full cream|whole milk)\b/, light: /\b(light|lite)\b/,
           lowfat: /\b(low fat|hi lo)\b/, skim: /\b(skim|no fat|fat free)\b/ },
    base: { soy: /\bsoy\b/, oat: /\boat\b/, almond: /\balmond\b/, /* … */ },
    flavour: { chocolate: /\b(chocolate|choc)\b/, strawberry: /\bstrawberry\b/, /* … */ },
};

// Flags: conflict when one name carries it and the other does not. Plain milk is
// not lactose-free milk, and the silent name is the plain product.
const ATTR_FLAGS = {
    lactoseFree: /\b(lactose free|free from lactose|zymil)\b/, organic: /\borganic\b/,
    highProtein: /\bprotein\b/, longLife: /\b(long life|uht)\b/, barista: /\bbarista\b/,
    powder: /\bpowder\b/, unhomogenised: /\bunhomogenised\b/, preserved: /\b(condensed|evaporated)\b/,
};

Treating silence correctly is the whole trick. If a missing "full cream" counted as a conflict, ALDI's terse titles would stop matching anything; if a missing "lactose free" counted as agreement, every lactose-free product pairs with its ordinary twin at a fictional discount.

3. The chain lists one product twice and you pay for both

Not a matching problem — a billing one. Woolworths' search API accepts only a few page sizes, so a 20-row request is served as two pages of 12, sorted by relevance, and relevance is not stable across pages:

rows 20
DUP 2 ('208064', 'a2 Milk Full Cream 2L')
DUP 2 ('797407', 'a2 Milk Full Cream 3L')

Stockcodes 208064 and 797407 came back at positions 1 and 2, then again at 19 and 20. Twenty rows, eighteen products, charged as twenty — this Actor bills per row. And a second identical Woolworths row cannot join the group its twin already occupies, so it becomes a bogus single: two of my 38 rows were that. The fix goes where rows are counted and charged:

const idKey = row.productId ? `${row.chain}:${row.productId}` : null;
if (idKey && seenIds.has(idKey)) { stats.duplicates++; continue; }

The run summary reports duplicatesDropped, so nobody has to take my word for it.

Same input, after the three fixes

productsSeen: 46 · duplicatesDropped: 2 · rowsDelivered: 33 · 6 seconds

38 rows became 33, multi-chain rows went 7 → 9, four of them with all three chains:

Size	Coles	Woolworths	ALDI	Match
1000 ml house brand	1.75	1.75	1.75	own-brand
2000 ml house brand	3.40	3.40	3.39	own-brand
3000 ml house brand	4.95	4.95	4.95	own-brand
2000 ml a2	7.30	7.30	7.29	same-brand

Fewer rows, more answers. And the answer is worth noticing: on house-brand full cream milk the three chains sit within one cent of each other at every size, so the cheapest-chain question is settled by a rounding error. The money is in brands and in whatever is marked down that morning — Farmhouse Gold at 13%, Zymil 2L at 14% once it meets the right Coles product.

Fields that are empty on purpose

Three chains, three different silences. ALDI publishes no online stock — inStock is null and availability reads In stores, so filtering on inStock === true drops every ALDI row. availableQuantity exists only at Coles, and with no store number it is a warehouse figure (2,224 for the house-brand 3L), not shelf stock. A row can have no price at all: Woolworths returned "Dairy Farmers Full Cream Milk 1L" with price: null, wasPrice: 3.00, inStock: false, and Coles returned "Geelong Full Cream Milk" with no price while still saying InStoreAndOnline — so a comparison row can legitimately carry cheapestChain: null.

One more thing that is not a bug: ALDI's search is a text match over the whole catalogue. Ask it for milk and 8 of its 30 rows are Maltesers, a Cadbury block, milk arrowroot biscuits and a milk frother. Coles returned 30 of 30 inside Milk and Long-Life Milk, Woolworths 30 of 30 inside DAIRY - MILK and BEVERAGES. The relevance filter you want is the chain's own category, in three vocabularies: Milk / Full Cream Milk, DAIRY - MILK, Dairy, Eggs & Fridge / Milk.

Calling it

No key for the sources, no browser:

import requests

rows = requests.post(
    "https://api.apify.com/v2/acts/lergassy~au-grocery-scraper/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={"keywords": ["full cream milk"], "chains": ["coles", "woolworths", "aldi"],
          "output": "comparison", "maxProductsPerKeyword": 20},
).json()

It lives here: Coles, Woolworths & ALDI Scraper. Give a run a watchName and schedule it, and later runs return only what moved — the mode I actually use.

When not to reach for it. Prices are the national online prices each chain shows a visitor with no store selected; store-level and delivery-zone pricing is not in there. Woolworths' category service refuses anonymous requests, so whole-aisle sweeps work at Coles and ALDI only. And if you only ever need one chain, a single-chain scraper is less machinery.

The lesson I keep relearning: when you join sources that never agreed on a schema, the join key is the product. Everything else is plumbing — and the plumbing was never what was broken.

The Same Song in Two TikToks Came Back as 'Made My Night' and 'Made My Life'

Nikita Iakovlev — Tue, 13 Oct 2026 13:12:44 +0000

I run a visa agency in Bali. The half of my week that has nothing to do with visas goes into Apify Actors, and one of them reads short vertical video: paste TikTok links, Reels, Shorts or a whole profile, get one flat row per video with the transcript, the text burned into the picture, and the first three seconds of both.

The transcript part sounds solved. Speech recognition is a commodity in 2026 — download the audio, post it to a model, get words. What nobody warns you about is that in short-form video, the words you get back are frequently not the words the creator said, and a plain transcript field cannot tell you which case you are in.

Eleven rows from one profile

13 October 2026, profile mode: @duolingo, ten newest videos, default settings, no on-screen text. Eleven rows, 87 seconds end to end.

Duration	Source	`transcriptText`
5.55 s	speech-recognition	`.`
14.63 s	speech-recognition	`Ahem.`
7.41 s	speech-recognition	`Веритесь Майнкрафта! Ему место в Бэкрус!`
17.12 s	speech-recognition	`You just made my night, I'm getting better now…`
21.57 s	speech-recognition	`Time for typical, but what? You just made my life…`
45.6 s	platform-captions	`Boom! Boom! Boom! Boom! Boom! Can you say happy birthday…`
10.54 s	speech-recognition	`Mama, please take a picture. Yes. Thank you…`

Rows carrying an actual script by the account that posted them: six of eleven. One row is a full stop, one is a throat clearing, three are lyrics from borrowed sounds, and one is Russian containing two words that are not Russian words.

That ratio is not a Duolingo quirk. On a labelled set of 26 real videos I measured earlier, only 69% carried a narrated script, 19% had no usable speech, and 12% ran on someone else's trending sound. Treat transcriptText as "what this creator said" and roughly a third of your rows lie to you — and they pass a spot check, because the rows you scroll past first are the narrated ones.

The same song, two different strings

The two lyric rows are the useful ones: both clips ride the same LE SSERAFIM track, and recognition heard it twice.

17.12 s → "You just made my night, I'm getting better now You just made my night,
           got me figured out Feelings got me high, oh baby, it's just science"

21.57 s → "Time for typical, but what? You just made my life, I feel better now
           You just made my life, got me figured out Feeling got me high
           Oh baby, it's just science"

Same line, two spellings: made my night and made my life. The creator's own caption on the second clip reads "made my night, i'm literally an owl" — so we know which is wrong, and we know it only from metadata, never from the audio.

The practical consequence: a transcript is not a usable key for borrowed audio. Group a feed by transcriptText to find every clip riding one sound and you get two groups where there should be one. Use the platform's sound ID for that, not text.

Two sources, and the field that names them

Every row carries transcriptSource with one of two values, and it changes what the row means.

platform-captions — TikTok, Instagram or YouTube already had a caption track, so no video was downloaded and no recognition ran. Fast, and free of the audio charge.
speech-recognition — no track existed, so the audio went through Whisper large v3 turbo.

In the run above it was 1 of 11; in a run over the same account two days earlier, 3 of 11. Platform captions on TikTok are the minority case and an unstable one — whether a video has them depends on the creator's upload settings, not on anything you control.

The two sources also parse differently, which cost me a bug. YouTube's auto-captions arrive rolling: every cue repeats the previous line so the box can scroll, so without dedup the transcript is three times too long. TikTok and Instagram captions are not rolling, and there the same dedup deletes real repetition — a clip with one line said five times came back with three:

def parse_vtt(text: str, rolling: bool = True) -> dict:
    # YouTube: each cue repeats the previous line → drop the repeat.
    # TikTok / Instagram: a repeated line is a repeated line → keep it.
    ...
    if not t or (rolling and t == seen_tail):
        continue

One flag, opposite defaults per platform. There is no way to infer it from the file.

When the platform supplies the captions, it supplies the language too

A seven-second clip captioned "proof that tacos are a language" came back like this:

{
  "durationSeconds": 7.22,
  "language": "vi",
  "transcriptSource": "platform-captions",
  "transcriptText": "ta cô ta có ta cô ta cô ta có ta cô cô ta có ta cô"
}

That is TikTok's own caption generator hearing the English word taco repeated and filing the clip as Vietnamese, tone marks included. My Actor passes it through, because on platform-captions rows the language label is the platform's — inventing a second opinion would be worse. So language is a recognition result only on speech-recognition rows. Elsewhere it is a claim by TikTok, and TikTok is wrong often enough that you should not route anything on it.

YouTube Shorts are cheap and arrive pre-mangled

For YouTube the Actor never downloads the video. YouTube serves Shorts audio only through residential traffic, which at $8/GB costs more on a one-minute clip than I charge for the whole row — so the caption track is the only sane path: a few kilobytes of text, back in seconds, no audio charge. A Short with no track of its own gets an explicit free error row instead of a silent loss.

The catch is formatting. A 40-second MrBeast Short, verbatim:

Find a subscriber for whatever country\nthat lands on, give them 10 grand.
>> Okay, Indonesia.\n>> Hey, you got to go now.
...
>> Subscribe to Mr. [music] Beast.

Three artifacts in one string: >> as a speaker-change marker, hard newlines wrapped at caption-box width instead of at sentence ends, and [music] injected mid-phrase — "Mr. [music] Beast". Strip newlines, markers and bracketed tags before this reaches an LLM or a search index, or your embeddings carry a lot of caption-box geometry.

What recognition says when there is nothing to say

Speech models do not stay quiet over music; they emit their most probable sentence. On wordless clips I have collected "We'll be right back.", "Thanks for watching", and on a music-only TikTok the Korean "구독과 좋아요 부탁드립니다." — "please subscribe and like". The model's own confidence is no help: on one branded clip it reported a false-positive probability of 0.00 for a sentence nobody said.

So there is a filter — short output matching a list of stock lines and music tags becomes an empty transcript, the row is marked as carrying no speech, and the audio is not billed. It works on sentences. It does not work on .: that 5.55-second row above is a silent Taco Bell promo whose transcript is one period, and speechDetected came back true because the string is not empty. Punctuation-only output is the next entry on that list. Until then, treat a transcript with no alphanumeric characters as no speech.

The guard that costs six cents

The Actor can label all of this per row — contentClass of narrated, borrowed_audio, silent or empty, plus speechCoverage and a count of burned-in subtitle tracks. Those fields fill only when on-screen text reading is enabled, because the class is decided by comparing speech against what is written in the frame — which means downloading the video and reading it frame by frame. That is a separate $0.06 event, it was off in the run above, and that is exactly why every contentClass in it is null.

So the trade is $0.004 per clip for text with no verdict, or $0.064 for text plus a verdict on whose words they are. For a one-off competitor sweep, pay it. For 50,000 rows, don't — filter yourself on word count, repeated lines and punctuation-only strings.

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("lergassy/tiktok-reels-shorts-transcript-scraper").call(run_input={
    "profiles": ["@duolingo"],
    "resultsLimitPerProfile": 10,
    "onlyPostsNewerThan": "2026-10-01",
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    words = len((row["transcriptText"] or "").split())
    print(row["transcriptSource"], words, row["firstThreeSecondsSpeech"])

The Actor is here — pay-per-event, $0.002 per video plus $0.001 per started 10 seconds of recognised audio, no login, no proxy fee. Wrong tool if you already hold the media file (use a plain transcriber), if you need Instagram profiles rather than Reel links (those need a login and are not supported), or if you really just want engagement numbers — the rows carry views and likes, but a metrics scraper gets them for a fraction of the price.

One thing to take from this: in short video, log where every transcript came from. The words are the easy part.

A Walkover, a Retirement and a Straight-Sets Win All Arrive With the Same Status Code

Nikita Iakovlev — Tue, 13 Oct 2026 05:11:55 +0000

Visas pay my bills — I run an agency in Bali. The other half of my week goes into Apify Actors, and one of them is a tennis feed: pick a date and a set of tours, get one flat row per match with both players, set scores, tiebreaks and a one-line summary.

I expected the hard part to be blocking. It isn't — the whole professional day of tennis arrives in a single HTTP request, no browser, no key, and a run finishes in under ten seconds. The hard part turned out to be a question that sounds too simple to be a problem: is this match over?

80 rows from yesterday

Here is a real run, taken this morning (13 October 2026): yesterday's results, every tour, capped at 80 rows. 8.1 seconds end to end. The tour mix tells you where professional tennis actually happens on a Monday — 38 Challenger men, 20 Challenger women, 14 ITF men, 4 WTA, 4 ATP.

Breakdown by status:

`status`	Rows
`finished`	73
`retired`	5
`walkover`	2

Seven of eighty — 8.75% — did not end the way a results table assumes. And every single one of those seven carries statusCode: 3. The same number as all 73 completed matches.

Two fields, and the obvious one is useless

The feed carries two state keys per match. The first is coarse: 1 scheduled, 2 live, 3 no longer scheduled. The second is the precise stage, and it is a 70-value enum — finished, retired, walkover, awarded, canceled, postponed, abandoned, suspended, plus live stages down to thirdSetTiebreak.

A cancelled match, a walkover, a retirement and a 6-4 6-2 win are all "no longer scheduled". The coarse field cannot tell them apart. The first version of my mapper read only that field, because it is the one that looks like the status field, and it labelled ten matches out of twenty-eight as finished with no score at all. A results table where a third of the rows are completed matches with an empty score is worse than no table, and it looks fine in a spot check because the first rows you scroll past are real.

So the mapper reads the precise stage and keeps it in the row twice: status for the stage name a human filters on, and played — a boolean that is true for exactly four stages:

const DONE_STAGES = new Set(['finished', 'retired', 'walkover', 'awarded']);

played is the field to filter on, not status == "finished". Retirements and walkovers produced a winner and belong in a results table; filtering on finished silently drops 8.75% of yesterday's results, and those are not random — a disproportionate share of retirements happen at Challenger and ITF level, where bodies are held together with tape and a flight the next morning.

The scoreline that cannot exist

Here is one of the retirements, from the Catania Challenger:

{
  "matchId": "vy2Rjhqo",
  "tour": "CHALLENGER MEN",
  "tournament": "Catania",
  "status": "retired",
  "statusCode": 3,
  "statusStage": "retired",
  "statusStageCode": 8,
  "played": true,
  "homePlayer": "Cuenin S.",
  "awayPlayer": "Almazan Valiente I.",
  "homeSets": 1,
  "awaySets": 1,
  "score": "7-6(4) 3-6 3-4",
  "winner": "away",
  "winnerName": "Almazan Valiente I.",
  "summary": "Cuenin S. vs Almazan Valiente I., CHALLENGER MEN Catania — Almazan Valiente I. won after a retirement 7-6(4) 3-6 3-4"
}

Read score and homeSets/awaySets without reading status and you have a man who won a best-of-three match one set to one, trailing 3-4 in the decider. Feed that into a sets-won model and it is not an outlier you can clip — it is a legal-looking record that drags every average it touches. Another one in the same run: 6-2 2-0, a match "won" in a set and a bit.

The walkovers are the opposite failure. score is null, sets is [], homeSets and awaySets are null — but winner is set and played is true:

{
  "matchId": "hb2HZnSE", "tour": "ATP", "tournament": "Shanghai",
  "status": "walkover", "statusStageCode": 9, "played": true,
  "homePlayer": "Schnaitter J./Wallner M.", "awayPlayer": "Buse I./Vallejo D.",
  "score": null, "sets": [], "homeSets": null, "awaySets": null,
  "winner": "home"
}

A row with a winner and no score is correct here. If your pipeline treats a null score as a scrape failure, it will retry forever on a match that never started.

The mirror image shows up on the schedule side. Today's ATP and WTA card came back as 30 rows: 26 upcoming, 1 walkover, and 3 canceled — Bencic vs Zhang, Kalinskaya vs Zhang and Li vs Zheng, all three cancelled from the Wuhan card, all three still sitting in today's schedule feed with a start time. Publish that as "today's matches" and you have promised three matches that will not be played.

Four more things the source does, and why they bite

The doubles decider is not a set. One Wuhan doubles result came back 7-5 4-6 3-10. That third set is a 10-point match tiebreak, and the feed puts those points in the games column like any other set. homeSets: 1, awaySets: 2 is right; summing the games column for "games won" adds thirteen games that were never played.

There are two player ids and only one of them is stable. The feed carries a per-match entry id — different for the same player in two different matches — and a participant id that the search endpoint returns and the player page URL is built from. My first version used the entry id. Every player link it emitted answered 404 and every id failed to match a search result, which is exactly the class of bug that passes code review: the field is populated, the shape is right, the value is garbage.

Statistics are strings, and the names repeat. A played match carries around 58 statistic rows, in sections Match, Set 1, Set 2, grouped into Service, Return, Points and Games. Of the 116 values in that run, 66 were shaped 74% (29/39), 32 were plain integers, 12 were fractions like 6/7 and 6 were bare percentages. All strings. And the name is not unique — Aces appears three times in one match:

Section	Home	Away
Match	4	9
Set 1	1	5
Set 2	3	4

So the obvious flatten:

stats = {s["name"]: s["home"] for s in match["statistics"]}   # wrong
stats["Aces"]   # "3" — the last set, not the match

Key on (section, group, name) or filter to section == "Match" first. Nothing errors if you don't; you just get the final set's number labelled as the match total.

Point-by-point only names the server twice per set. The block for that match held 2 sets and 18 games. Four of the 18 carried a server value — the first two games of each set. Tennis alternates serve, so the rest are derivable, but you have to do it yourself, and you have to do it per set because the order flips. One more detail worth knowing before you parse the point strings: they are always written home:away, not server:receiver, so 0:15 in a game served by the away player means the server won the point. The strings also carry inline markers like |B1|; I pass those through verbatim rather than invent a meaning for them.

Head-to-head has a related trap that I do fix in the mapper: the winner of each previous meeting is not a flag, it is a leading asterisk on the winning player's name. Left in place it quietly corrupts every join on player names, so it gets stripped into a proper winner field. The h2h scores are set counts only (2:0), not game scores.

Calling it

One POST, no key for the source, pay per row:

import requests

rows = requests.post(
    "https://api.apify.com/v2/acts/lergassy~tennis-scores-api/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={"mode": "results", "tours": ["all"], "daysBack": 1, "maxMatches": 80},
).json()

real = [r for r in rows if r["played"]]
clean = [r for r in real if r["status"] == "finished"]   # completed only
print(len(rows), len(real), len(clean))                 # 80 80 73

It's on the Apify Store as Tennis Scores & Stats API — schedule, live, results, single match, player, rankings and search, one flat row per match at $0.0009 a row.

When this is the wrong tool

You need the round. The day feed does not carry it. A schedule row's round is null rather than a guess reverse-engineered from the draw size, and no amount of filtering will conjure it.
You need odds. Not in this source. Different feed, different article.
You need history. This reads the public live-score data for a date range. It is not an archive — if you want yesterday's closing state preserved, schedule your own runs now.
You are a broadcaster. The official tour feeds are licensed, come with an SLA and cost thousands a month. This is public live-score data turned into rows, and it is priced accordingly.

The portable lesson has nothing to do with tennis: when a feed gives you two fields for the same concept, the one that is easy to read is usually the one that was easy to write — a three-value bucket somebody added for colouring a web page. Find the enum underneath it and spend a day mapping it properly. The cost of getting it wrong is not an error in your logs. It is 8.75% of your rows being confidently, plausibly false.

Kalshi's Market Titled 'HIGHEST Temperature SATX' Settles in Louisville

Nikita Iakovlev — Mon, 12 Oct 2026 21:11:30 +0000

Visas are my day job — I run an agency in Bali. Scrapers are the other half of the week, and one of them reads Kalshi's daily temperature markets: the ladders of brackets where a contract pays $1 if tomorrow's high in a given city lands in a given two-degree band. The prices are a crowd-sourced probability distribution over the weather, which is a genuinely interesting dataset and free to read — Kalshi's market API needs no key and no account.

I thought the hard part would be the bracket arithmetic. It wasn't. The hard part was answering the question which city is this market about, and the reason is that Kalshi answers it three times, in three places, and they do not always agree.

Three location fields, two of them unreliable

Here are three live calls against the public API, made today, 12 October 2026. First the series list:

curl -s 'https://api.elections.kalshi.com/trade-api/v2/series?category=Climate%20and%20Weather'

Two of the 422 entries it returns:

KXHIGHTSDF   | HIGHEST Temperature SATX
KXHIGHTSATX  | San Antonio Daily Maximum Temperature

The series whose ticker ends in SDF — the IATA code for Louisville — is titled with SATX, San Antonio. Now the event under that same series:

curl -s 'https://api.elections.kalshi.com/trade-api/v2/events?series_ticker=KXHIGHTSDF&with_nested_markets=true&status=open&limit=1'

event  KXHIGHTSDF-26OCT12 | Highest temperature in Louisville on Oct 12, 2026?
rules: If the maximum temperature recorded at Louisville (CLISDF) for Oct 12, 2026,
       is less than 82° fahrenheit according to The Weather Company, then the market
       resolves to Yes.

Series title says San Antonio. Event title says Louisville. Settlement rules say Louisville, station CLISDF. Its sibling KXLOWTSDF is titled "Lowest Temperature SDX", which is not a code for anywhere.

So: the only field that is load-bearing is the one inside the resolution rules, because that is the string the exchange pays out against. Everything above it is a label a human typed. My parser is one line, and it reads the rules:

export function stationFromRules(rules) {
    const m = String(rules || '').match(/\(([A-Z]{3}[A-Z0-9]{2,4})\)/);
    return m ? m[1] : null;
}

The city then comes out of a station table, never out of the title. The title is a fallback label and nothing else.

Why the station, not the city, decides your forecast

If you only want prices this is trivia. The moment you want to put a weather forecast next to the price, it becomes the whole problem — because these markets do not settle on "the weather in Chicago". They settle on one climate report from one point on the map:

Market	Settles at	Not at
Chicago	`CLIMDW` — Midway, southwest side	O'Hare
Los Angeles	`CLILAX` — the airport, by the ocean	downtown
Houston	`CLIHOU` — Hobby	Intercontinental (`CLIIAH`)
New York City	`CLINYC` — Central Park	Newark or LaGuardia

Newark and Trenton are their own markets, by the way — KXHIGHTEWR and KXHIGHTTTN. "New York weather" is three separate ladders.

I measured what guessing costs. Same forecast provider (Open-Meteo, also keyless), two coordinate pairs each, daily high for the next three days, pulled this afternoon:

City	Settlement station	The obvious alternative	Biggest gap in 3 days
Los Angeles	LAX 77.6° (14 Oct)	downtown 81.8°	4.2°
San Francisco	SFO 72.5° (13 Oct)	downtown 69.1°	3.4°
Houston	Hobby 93.4° (13 Oct)	Intercontinental 90.3°	3.1°
Chicago	Midway 66.2° (14 Oct)	the Loop 63.3°	2.9°

Phoenix and Washington stayed inside half a degree, so this is not a uniform correction you can shrug off — it is coastal and lake-effect geography, and it is biggest exactly where the volume is. Kalshi's brackets are two degrees wide. A 4.2° error is two whole brackets. You would be comparing the market's price for "77° to 78°" against a forecast that belongs in "81° or above" and calling it an edge.

Observations have the same trap, and a nice shortcut out of it: Kalshi's climate-report code maps onto the NWS station id by swapping the CLI prefix for K. CLIMDW → KMDW, and https://api.weather.gov/stations/KMDW/observations gives you what the settling station has actually recorded, free. (Settlement itself is "according to The Weather Company" per the rules, so treat NWS obs as a sanity check on the same thermometer, not as the official number.)

The field that tells you the forecast is stale, not smart

That observation column turned out to be the most useful thing in the output. A real row from a run I made an hour ago, Chicago high for today:

{
  "city": "Chicago", "station": "CLIMDW", "date": "2026-10-12",
  "favourite": "88° to 89°", "favouriteProbability": 99,
  "marketExpected": 88.4,
  "forecastValue": 86.2,
  "forecastMinusMarket": -2.2,
  "forecastBracket": "86° to 87°",
  "forecastBracketProbability": 1,
  "observedHigh": 88, "observedLow": 62.6,
  "volume": 100723.06
}

Read the first five fields and you have found a disagreement: the model says 86.2, the market is 99% certain of 88–89, and the bracket the model points at is on sale for one cent. Read observedHigh and the story collapses — Midway had already recorded 88° when I pulled this. The high was in. The market wasn't confident, it was informed; the model run was old.

Any forecast-versus-market row for a day already under way needs that column, or you will mistake lag for alpha. For tomorrow, where there is nothing recorded yet, the same actor's Chicago row shows a real gap: market expects 82.2, model says 77.0, and the bracket the model picks is the favourite at 52% — a disagreement about magnitude inside a band everyone agrees on.

One more honesty flag in that example. Tomorrow's favourite is "81° or below", an open-ended end bracket with no true midpoint. When most of the money sits in one of those, a weighted average of midpoints is capped by construction and understates the market. That row comes back with marketExpectedIsBounded: true so you know not to quote the number as a temperature. Use forecastBracketFairProbability instead — what the market charges for the bracket your model actually picked.

Days nobody traded

Kalshi opens an event per city per day whether or not anyone shows up. In yesterday's full snapshot — 48 open events, 288 brackets, 24 cities, highs and lows — 9 of the 48 ladders priced to less than 0.85 in total:

Chicago    low   sum 0.46
San Diego  low   sum 0.58
Seattle    low   sum 0.64
Minneapolis low  sum 0.69
...

A healthy six-bracket ladder sums to a little over 1.00 — the surplus is the exchange's cut. These sum to half of 1.00, because every bracket is sitting at a penny of stale quote on a few dollars of volume. Scale those six pennies to 100% and you have manufactured a confident-looking distribution out of nothing; one of those rows reports an "overround" of −0.31, which is not a tight market, it is an empty one. So they come back flagged illiquid: true, fair probabilities withheld rather than invented, and the forecast mode drops them entirely. Overnight lows are where this clusters — nobody trades them, and that is the data, not a bug.

Calling it

Input is four fields; this is the comparison mode for two cities, one day ahead:

curl -X POST "https://api.apify.com/v2/acts/lergassy~kalshi-weather-scraper/run-sync-get-dataset-items?token=<TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"mode":"forecast","metrics":["high"],"cities":["Louisville","Chicago"],"days":1,"withForecast":true}'

Four rows, 33 seconds, and the Louisville rows are correctly labelled Louisville no matter what the series title says. It's on the Apify Store as Kalshi Weather Markets — brackets, days, forecast-versus-market and the raw series list, pay per row.

When this is the wrong tool

You want to trade from it. It places no orders and holds no positions. It reads public prices and public weather.
You want weather outside the US. The series list includes Hong Kong and Berlin temperature series; my station table covers the 24 US cities that settle on a CLI climate report, and anything else comes back without coordinates.
You want rain, snow or hurricanes. Those series exist in quantity — 30 rain, 23 hurricane, 9 snow in today's list — and usually have no open event. Of the 104 daily high/low series, only 48 had a live event yesterday. Daily highs and lows are what actually trades.
You want to run it from a laptop in the wrong country. Kalshi's API refuses connections from a number of regions, Indonesia among them. That is the honest reason this one lives on a cloud platform instead of my machine.

The portable lesson has nothing to do with prediction markets: when a feed describes the same thing in several fields, find out which field the money is settled against, and parse that one. Here it was a sentence of English prose, three layers down, contradicting the title at the top.

Three Books Said 64.3%, 63.1% and 62.0% on the Same Team. Strip the Vig and They Agree to 0.26 Points

Nikita Iakovlev — Mon, 12 Oct 2026 13:11:51 +0000

I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them is an odds feed: pick leagues and dates, get one row per game × market × book, with DraftKings, Pinnacle and the Kalshi prediction market side by side.

The question I get asked most about it is some version of "which book is right". The question people should ask first is "are these three numbers even on the same scale". They are not, and the gap is bigger than most of the disagreements you will go hunting for.

Here is a run from today — 12 October 2026, NFL, four days ahead, moneyline, spread and total, three sources. Fifteen rows, two games, 2.4 seconds.

The same team, three implied probabilities

Monday night, Buffalo Bills at Los Angeles Rams. Moneyline, Rams to win:

Book	Price	Implied probability	Overround	Fair probability
DraftKings	−180	64.29%	4.29%	61.64%
Pinnacle	−171	63.10%	2.47%	61.58%
Kalshi	−163	62.00%	1.00%	61.38%

Read the third column and the three books look 2.29 percentage points apart on who wins a football game. Read the fifth and they are 0.26 points apart — closer than the rounding on most odds screens.

Nothing moved between those two columns except arithmetic. Implied probability is just 1 / decimal_odds, and for a two-sided market the two implied probabilities sum to more than 100%. That excess is the overround — the bookmaker's margin, the vig, the cut. DraftKings' moneyline added up to 104.29%. Pinnacle's to 102.47%. Kalshi's to 101.00%.

Scale each side down by the sum and you get the fair, vig-free probability:

fair_home = implied_home / (implied_home + implied_away)
61.64%    = 0.6429 / 1.0429

Compare raw implied probabilities across books and you are mostly measuring which book charges more, not which book thinks differently. DraftKings looks like the most confident in the Rams in that table. It isn't. It is the most expensive.

De-vigging does not erase disagreement — it isolates it

The second game in the run, Seattle Seahawks at Denver Broncos, three days out:

Book	Broncos price	Implied	Fair
DraftKings	+130	43.48%	41.70%
Pinnacle	+142	41.32%	40.12%
Kalshi	+150	40.00%	40.00%

Naive gap: 3.48 points. Fair gap: 1.70 points. Half the apparent disagreement was margin, and half of it was real — DraftKings genuinely rates Denver higher than the other two do. That survivor is the part worth modelling.

That is why the rows carry both numbers. homeImpliedProbability is what the price literally says. homeFairProbability and overround are what you need to compare anything to anything. Ship only the first and every cross-book chart is wrong by the margin spread; ship only the second and nobody can reconstruct the price.

Six DraftKings rows in this run ran 4.26% to 4.75% overround. Six Pinnacle rows ran 2.41% to 4.01%. One Kalshi row came back at −0.01%, which is not a bug and not free money: Kalshi is an exchange, the price I quote is the ask on each side, and two asks can sum to a hair under 100% when the book is tight. There is no bookmaker margin there at all, only a bid-ask spread.

The field that says how much the sharp book means it

Pinnacle's public feed carries something nobody else publishes: the risk limit it will accept on each market. Same run, same two games:

Game	Moneyline limit	Spread limit	Total limit
Bills at Rams (tonight)	$50,000	$75,000	$10,000
Seahawks at Broncos (Thursday)	$3,000	$5,000	$3,000

A number a book will take $75,000 on and a number it will take $3,000 on are not the same kind of fact, even though they arrive in the same JSON shape. The limit is a confidence and liquidity signal rolled into one, and it is the honest antidote to treating every row as equally solid.

One caveat before you build on it: part of that spread is time, because Pinnacle raises limits as kickoff approaches and one of these games starts tonight while the other is Thursday. Within a single game the ordering is cleaner — the Rams spread takes $75,000 and the Rams total takes $10,000, so the book is willing to risk 7.5× as much on the spread as on the points.

Kalshi's independent liquidity numbers agree about which market is real: $2,740,094 traded on the Monday-night game, $84,989 on Thursday's. Two unrelated venues, same verdict, 32× apart.

What a cross-book edge actually looks like

With two or more books on the same market the Actor emits edge rows. Across 14 odds rows this run produced exactly one, and zero arbitrage:

{
  "type": "edge",
  "kind": "value",
  "game": "Buffalo Bills at Los Angeles Rams",
  "market": "total",
  "line": 54.5,
  "side": "under",
  "book": "DraftKings",
  "decimal": 1.98,
  "impliedProbability": 0.5051,
  "fairProbability": 0.5114,
  "sharpBook": "Pinnacle",
  "edgePercent": 1.257,
  "sharpLimitUsd": 10000,
  "booksCount": 2
}

The arithmetic, end to end: Pinnacle priced that total over −101 / under −111, which is 1.990 and 1.901 in decimal, implying 50.25% and 52.61% — 102.86% together. Scale down and the sharp book's true read on the under is 51.14%. DraftKings paid −102 on the same under, decimal 1.98. Multiply: 0.5114 × 1.98 = 1.0126, so that price returns 1.26% more than the sharp line says it is worth.

Note sharpLimitUsd: 10000 on the row. A 1.26% edge against a number Pinnacle will take $10,000 on is a different proposition from a 1.26% edge against a number it will take $500 on, so the limit rides along.

And note what did not happen. The other game's totals were DraftKings 41.5 and Pinnacle 42.5. A full point apart, so they were never compared. Grouping is by market and line — total|54.5, not total — because "over at 41.5" and "over at 42.5" are different bets, and comparing their prices produces a confident, meaningless number. If you are merging odds feeds yourself, that grouping key is the bug you will ship first.

Arbitrage being empty is the normal answer, by the way. Books watch each other. An empty arbitrage list means the market is working, not that the scraper failed.

The rest of the potholes

Pinnacle throttles rather than queues. Bursts come back 401, 403 or 429, not a 503 you can politely retry. Grinding through six growing backoffs on the same address pushed a gateway call past its 48-second timeout once; now it gives up after two quick tries and changes address instead.
Pinnacle's feed mixes periods. Every matchup ships full-game, half and quarter markets in one list, all with the same shape. Filter on period 0 or your "NFL spreads" table is a third second-half lines.
Opening lines are DraftKings-only. openLine, openHomePrice and lineMovement come from DraftKings' own feed. Pinnacle does not publish an open, so those fields are null on its rows — it is a source difference, not a gap. Tonight's Rams spread opened −3 at −111 both ways and is now −3.5 at −102; Pinnacle is still on −3.
Nothing stores the past. Movement history beyond what DraftKings itself carries means scheduling your own runs now.

Calling it is one POST:

import requests

rows = requests.post(
    "https://api.apify.com/v2/acts/lergassy~sports-odds-api/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={
        "mode": "odds",
        "leagues": ["nfl"],
        "daysAhead": 4,
        "markets": ["moneyline", "spread", "total"],
        "books": ["draftkings", "pinnacle", "kalshi"],
        "oddsFormat": "american",
        "minEdgePercent": 1,
        "maxItems": 24,
    },
).json()

for r in (x for x in rows if x["type"] == "odds" and x["market"] == "moneyline"):
    print(f'{r["book"]:12} {r["homePrice"]:>5}  implied {r["homeImpliedProbability"]:.4f}'
          f'  fair {r["homeFairProbability"]:.4f}  vig {r["overround"]:.2%}')

When this is the wrong tool

You need every price tick. This is a snapshot per run. Sub-minute movement means a licensed feed with a websocket, not a scraper on a schedule.
You need in-play odds. Full-game pre-match markets only. Live lines move faster than any run interval that makes sense.
You need settlement-grade data for a product that pays out. Public feeds go stale and get throttled. If money moves on the number, buy the number.
You want a tip service. The edge rows are arithmetic on two prices, nothing more. A 1.26% edge against a $10,000 limit is a measurement, not advice.

The Actor is Sports Betting Odds API — 28 leagues, props, futures and Kalshi contracts, no API key and no browser. The whole measurement in this post was 15 rows, about four and a half cents.

Whatever you pull odds from: before you compare two books, make sure the two numbers add up to the same thing.

Booking.com Search Rows Hide 9–16% of the Price in a Field Labelled 21, 22 and 23

Nikita Iakovlev — Mon, 12 Oct 2026 05:15:39 +0000

I run a visa agency in Bali. The other half of my week goes into scrapers on Apify, and one of them reads Booking.com search pages: destination in, one row per property out, with the price for your dates, the matched room, the rating and the location.

The field people build their spreadsheet on is priceTotal. It is the number Booking prints in the search card, it is the number the site sorts by, and it is not the number anybody pays.

On 12 October 2026 I ran the same stay — 20 to 23 November 2026, two adults, one room — against four destinations and read all 24 rows. Here is what the excluded-charges field said.

24 rows, four cities, nothing tax-inclusive

Destination	`priceTotal` range	Excluded charges	Share of the headline price
Lisbon (EUR)	€289.81 – €540.27	flat €24 + 6.00%	10.4% – 14.3%
New York (USD)	$370.15 – $1438.51	flat $10.50 + 14.75%	0% – 16.2%
Tokyo (USD)	$483.42 – $1101.10	9.01% – 10.00%	9.0% – 10.0%
Bali (USD)	$96.09 – $875.27	9.01% or 10.99%	9.0% – 11.0%

Read the New York column slowly. priceTotal for the Hyatt Regency Times Square was $1120.95, and the row also carried $175.84 of charges that price does not contain. The guest pays $1296.79. If your rate-shopping table says $1120.95, your table is wrong by the size of a whole night.

Two of the per-city numbers are suspiciously exact, and that is the useful part:

New York: 14.75%, to the cent, on every row that had it. $110.06 on $746.20. $138.15 on $936.63. $212.18 on $1438.51. A percentage that stable is a tax rate, not an estimate.
Lisbon: a flat €24 on all six properties, regardless of a price range that runs from €290 to €540. Three nights, two guests, €24 — €4 per guest per night, charged per stay rather than per euro spent.

Tokyo came back at 10.00% on five rows and 9.01% on one. Bali is the interesting one: within a single destination, three properties excluded 9.01% and three excluded 10.99%, and which rate applied tracked exactly with the charge code in the field.

The field has amounts but no names

This is what taxesAndFeesExcluded actually looks like:

{
  "name": "Hyatt Regency Times Square",
  "priceTotal": 1120.95,
  "currency": "USD",
  "taxesAndFeesExcluded": ["22: $10.50", "23: $165.34"],
  "taxesAndFeesExcludedTotal": 175.8402,
  "chargesInfo": null
}

22 and 23 are not my invention and not a truncation. Booking's GraphQL search response returns excluded charges as a list of objects carrying chargeMode, chargeInclusion, chargeType and amountPerStay — and for an anonymous caller chargeType is an integer. There is no translation node next to it, the way there is on badges, cancellation copy and review text. The site renders "+ taxes and charges" and hides the breakdown behind a tooltip that is assembled elsewhere.

My first version of this mapper expected names and produced "CITYTAX: US$18.24" in a code comment that never matched reality. The field was empty on every row for a while, because I was reading a translation key that is not there.

Across the four markets the codes behaved consistently enough to describe, though not enough to label:

22 was always a flat amount per stay, identical across every property in a city: €24 in Lisbon, $10.50 in New York, $0 in Tokyo and on three Bali rows.
21 and 23 were always a percentage of priceTotal: 6% in Lisbon, 14.75% in New York, 10% in Tokyo, 9.01% under code 21 and 10.99% under code 23 in Bali.

I ship the integers. I am not going to guess that 22 means "city tax" and put a word in a dataset column on the strength of six rows per market, because the moment I do, someone builds a VAT reclaim report on it. An unexplained 22 makes you go look. A wrong CITYTAX does not.

What it does to a sorted table

The percentage is near-constant inside a city, so within-city ranking mostly survives. I checked all 24 rows: adding the excluded charges did not flip a single pair. Lisbon stayed in the same order; New York stayed in the same order.

Cross-city comparison is where it breaks. A $500 Tokyo stay really costs about $550. A $500 New York stay really costs about $580. Build a "cheapest city for a three-night trip" chart on priceTotal and you have quietly handed New York a 6% discount it does not get.

So compute the honest number before you sort:

import requests

rows = requests.post(
    "https://api.apify.com/v2/acts/lergassy~booking-hotels-scraper/run-sync-get-dataset-items",
    params={"token": TOKEN},
    json={
        "mode": "search",
        "destination": "New York",
        "checkin": "2026-11-20",
        "checkout": "2026-11-23",
        "adults": 2,
        "currency": "USD",
        "sortBy": "price",
        "maxResults": 25,
    },
).json()

for r in sorted(rows, key=lambda r: r["priceTotal"] + (r.get("taxesAndFeesExcludedTotal") or 0)):
    true_total = r["priceTotal"] + (r.get("taxesAndFeesExcludedTotal") or 0)
    print(f'{r["name"][:40]:40} {r["priceTotal"]:8.2f} -> {true_total:8.2f}')

Note the or 0. Which brings me to the bug this article found in my own code.

The zero that looked like a missing value

West Side YMCA, cheapest row in the New York sort at $370.15, came back like this:

{ "taxesAndFeesExcluded": ["23: $0"], "taxesAndFeesExcludedTotal": null }

["23: $0"] is real: Booking returns the charge line with an amount of zero. But my aggregate was null, because the mapper ended in || null — and in JavaScript 0 || null is null. A property with genuinely no extra charges was indistinguishable from a property whose charges I failed to read. On a 25-row New York search, two rows were affected: West Side YMCA and HI New York City Hostel, both hostel-style properties.

One character, || to ??, except I removed the fallback entirely since the coercion already yields null for a missing field. Fixed and live as build 0.1.33 today; those two rows now report 0. If you pulled this Actor's data before 12 October 2026 and treated taxesAndFeesExcludedTotal: null as "unknown, skip the row", you dropped the cheapest properties in the market.

The other half of the row is prose, not data

While confirming the tax numbers I changed the language input and watched the same property come back four ways. MASA Hotel & Spa, Lisbon, same dates:

`language`	`distanceFromCentre`
`en-us` (default)	`2.6 miles from downtown`
`en-gb`	`2.6 miles from centre`
`de`	`Zentrum: 4,2 km`
`fr`	`4,2 km du centre`

Three things worth knowing before you parse that column:

It is a sentence, not a number. There is no numeric distance field in the search response — Booking sends location.mainDistance already rendered.
English means imperial, even British English. en-gb changed "downtown" to "centre" and left the miles alone. Metric arrives only when you leave English, and then with a comma decimal separator: 4,2 km.
The wording changes, so your regex is locale-bound. de puts the label first and a colon in the middle.

Use latitude and longitude instead and compute the distance yourself. They are plain floats and they do not translate.

The same split hits reviewScoreWord: Wonderful in English, Sehr gut in German, Très bien in French. Filter on that string and your pipeline silently empties the day someone sets a language. Filter on reviewScore, which stays 9.3.

What does not translate is the half of the row my own code assembles: propertyType and roomBeds stayed Hotel and 1 king bed under de and fr, because those come from integer→name tables inside the Actor, not from Booking. So a non-English run gives you a row that is half localised and half English, and the seam runs exactly along "who mapped this field". Worth knowing if you ever diff two runs and wonder why only some columns moved.

One more locale trap: priceTotalText is a formatted string — "$147.25" but "€ 524.51", with a non-breaking space and the symbol on the other side. Parse priceTotal, which is a float.

Region searches have no centre

Bali returned destinationType: "REGION" and distanceFromCentre: null on all six rows. Lisbon, New York and Tokyo came back as "CITY" and filled it on every row. That is not a failure — a 5,780 km² island does not have a downtown — but if your dashboard has a "distance from centre" axis, it will be empty for every region, landmark and country-level search you run. Check destinationType before you plot it.

When this is the wrong tool

You need to actually book. This reads public search data. Taking a reservation means Booking's affiliate programme, which is a contract, not a scraper.
You need yesterday's prices. Nothing stores the past. Hotel pricing is only observable going forward, so schedule a daily run now if you want a history in March.
You need the exact final figure at checkout. The excluded charges are Booking's own estimate of what the property will add; resort fees, card fees and local levies collected at the desk are not all in there.
You need every rate of 500 hotels. Rooms and rates mean a request per property. That is fine for fifty and slow for five hundred.

If reading hotel search data as rows is what you want, the Actor is Booking.com Hotel Search Scraper — search, single hotel, reviews and a year-ahead price calendar, no browser and no login. The four-city measurement in this post cost 7 cents.

And whatever you use to get the data: before you sort by price, add the field that says the price is incomplete.

Audio and video to text via API: transcribe a 45-minute podcast for under $0.50

Nikita Iakovlev — Mon, 12 Oct 2026 02:00:46 +0000

Most speech-to-text APIs make you do the plumbing yourself: download the media, strip the audio from video, cut long files under the size limit, send parts, stitch timings back together. If you just have a list of links — podcast episodes, meeting recordings, lecture videos — that's a lot of glue code for one text file.

One link in, one transcript out

Speech to Text on Apify takes MP3, M4A, WAV, FLAC, OGG, MP4, MOV or WEBM — any link or uploaded file with sound — and does the whole pipeline: download, audio extraction, splitting, recognition with Whisper large v3, and re-stitching timings. A three-hour recording is one input and one row of output.

Each row contains:

the clean transcript with timed passages
ready SRT or WebVTT subtitles
retrieval chunks that keep the moment each passage was spoken
language detected, 90+ languages supported
optional speaker labels

curl -X POST "https://api.apify.com/v2/acts/lergassy~speech-to-text/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/episode-42.mp3"]}'

Cost (from 4 October 2026)

$0.009 per started minute with the built-in recognition key — nothing to sign up for
$0.004 per minute if you bring your own Groq key (you then pay Groq directly, a fraction of a cent per minute)
$0.002 per file processed
speaker labels: +$0.006 per minute, only when switched on

A 45-minute episode is about $0.41 with the built-in key (45 × $0.009 + $0.002), or about $0.18 with your own key.

Video pages (YouTube, TikTok, Instagram, X, Vimeo) whose audio has to be pulled through a residential exit node carry an extra $0.09 per link — direct file links never pay it. For YouTube videos that already have captions, the transcript scraper from my earlier post on YouTube transcripts is far cheaper.

When to use it

Podcast show notes and search, meeting and interview archives, turning a course library into a searchable knowledge base, subtitles for your own videos.

Limits

It transcribes what is audible: heavy music or crosstalk lowers accuracy, as with any Whisper-based tool.
Private or login-only media can't be downloaded — use a direct file link or upload the file.

Disclosure: links to Apify in this post use my referral code.

Ten Mercado Libre Results, Four Kinds of Link — and Only Two Carry the Product ID

Nikita Iakovlev — Sun, 11 Oct 2026 21:15:05 +0000

I run a visa agency in Bali. The half of my week that has nothing to do with visas goes into scrapers on Apify, and the newest one reads Mercado Libre — the marketplace that runs 18 separate storefronts across Latin America, each on its own domain and currency: mercadolibre.com.mx, mercadolivre.com.br, mercadolibre.com.ar, .cl, .com.co, and a dozen smaller ones.

I expected the interesting problem to be the bot gate. It is there, and I will get to it. But the thing that actually cost me rewrites is far more boring and far more likely to break your code: a Mercado Libre search page does not hand out one kind of link. On 11 October 2026 I ran a 10-result search for zapatillas on the Argentine site and got back four distinct URL shapes in ten rows.

The four shapes

Positions 1 to 4 — every single one flagged isSponsored: true — looked like this (truncated; the real thing is about a kilobyte):

https://click1.mercadolibre.com.ar/mclics/clicks/external/MLA/count
  ?a=XrvZILjHGldBLoVC1wdirEioJadSenEQxPY7kCoIIJBNUf0R0v3FE0pq…
  &pdp_filters=item_id%3AMLA3183930614

That is not a product page. It is the ad click counter, and the product it points at is nowhere in the path — the only place the identifier survives is the pdp_filters query parameter, URL-encoded.

Positions 5, 7, 8 and 10 were a shape I had not seen in any Mercado Libre scraping write-up:

https://www.mercadolibre.com.ar/zapatilla-topper-sin-genero-lisa-estandar-rocket-tofunegro/up/MLAU3637728744

Note the identifier: MLAU3637728744. Site prefix MLA, then a literal U, then digits. It is not the item ID, and it is not the catalog product ID (/p/MLA…) that older pages use.

Positions 6 and 9 were the classic listing URL everybody's regex knows:

https://articulo.mercadolibre.com.ar/MLA-2815639876-zapatillas-skate-oversize-clasicas-livianas-_JM?searchVariation=189936656896

Four shapes, three different identifier formats, one search page. The practical consequences are immediate:

You cannot pull the ID with one pattern. You need the /p/MLA… path form, the MLA-1234567890 hyphenated form, the wid= and pdp_filters=item_id: query forms, and the /up/MLAU… form — and they do not all mean the same object.
You cannot dedupe by URL. Two ad impressions of the same product carry different a= blobs, so string comparison says they are different products. Dedupe on the extracted ID instead.
You cannot assume the top of the page is organic. In my run, 4 of the first 10 rows were paid placements. If you are tracking a competitor's rank, counting ads as rank is the most common way to get a chart that means nothing. I ship isSponsored as a field for exactly that reason, and includeSponsored: false drops them at the source.

The extraction in my parser is four fallbacks deep, which looks ugly and is the honest shape of the problem:

const ids = idFromUrl(href) || {};                                              // /p/MLA… or MLA-123…
const wid = (href.match(/[?&#]wid=(M[A-Z]{2}\d+)/) || [])[1] || null;           // variation picker
const fromAd = (href.match(/item_id(?:%3A|:)(M[A-Z]{2}\d+)/) || [])[1] || null; // ad redirect
const itemId = ids.kind === 'item' ? ids.id : (wid || fromAd || null);

The honest bug this article found

While gathering the sample for this post I pasted one of those /up/MLAU… URLs straight into reviews mode. My router does not recognise the shape, so it fell through to the "this must be a listing page" branch, opened it in a browser, found no product cards, and wrote this:

{
  "type": "error",
  "url": "https://www.mercadolibre.com.ar/zapatilla-topper…/up/MLAU3637728744",
  "error": "no product cards on listing page",
  "kind": "listing"
}

Two things I will defend about that output, and one I will not.

I will defend the error row: it is in the dataset, it names the URL, and it is billed at zero. A scraper that silently drops the inputs it could not read is worse than one that admits them, because the dataset still looks finished. And I will defend the routing retry visible in the log — plain HTTP got Mercado Libre's login wall (41 KB of "ingresá a tu cuenta"), the browser took over automatically, no human intervention needed.

What I will not defend is the router. The fix is to teach idFromUrl the /up/MLAU… shape; until it ships, feed reviews mode the productId / itemId that the search rows already give you, not the page URL. For the row above that is MLA2796328360. Search first, then reviews — that chain works today and is what the Actor's own example input does.

Where units-sold actually lives

Mercado Libre stopped printing "+5 mil vendidos" on most organic cards sometime before I measured this in late September 2026. The number did not go away; it moved into the page's own tracking payload, which is still in the HTML of every search page:

function soldByItem(html) {
  const map = new Map();
  for (const m of String(html).matchAll(/\{"item_id":"(M[A-Z]{2}\d+)"[^{}]*?"sold_quantity":(\d+)/g)) {
    if (!map.has(m[1])) map.set(m[1], Number(m[2]));
  }
  return map;
}

Read it from there and you get a figure for every result on the page. Read it only from the visible card text and most rows come back null.

One caveat that matters if you plan to do arithmetic: the numbers are Mercado Libre's own rounding, not real sales. My ten rows returned 500, 50, 1000, 100, 5000, 1000, 100, 1000, 500 and 1000. Those are buckets. "Product A sold 1,000 and product B sold 500" means "A is in the 1,000 bucket" — fine for ranking, useless for revenue estimates.

While we are on untrustworthy integers: totalResults came back as 9,999 for zapatillas in Argentina, and 9,999 again for notebook in Chile on the same day. That is a ceiling, not a count. Treat any 9,999 as "more than we are being told", the same way you treat a capped page count.

And prices are never converted. The Chilean row is 1009990 with currency: "CLP"; the Argentine row is 49667 with "ARS". Mixing marketplaces in one run is supported, and if you average that column without grouping by currency you will produce a very confident, very wrong number.

Reviews: the rating total is not the review total

Reviews come from a JSON endpoint rather than the HTML, which is the cheap half of this Actor — no browser, no proxy traffic. A reviews-mode run on 11 October 2026 (maxReviewsPerProduct: 5) produced one summary row and five review rows:

{
  "type": "product_rating",
  "productId": "MLA2815639876",
  "ratingAverage": 4.5,
  "ratingCount": 366,
  "reviewsWithText": 191,
  "ratingBreakdown": {
    "5": { "count": 278, "percentage": 75.96 },
    "4": { "count": 44,  "percentage": 12.02 },
    "3": { "count": 25,  "percentage": 6.83 },
    "2": { "count": 5,   "percentage": 1.37 },
    "1": { "count": 14,  "percentage": 3.83 }
  },
  "picturesCount": 57,
  "videosCount": 18,
  "aiSummaryStatus": "published"
}

ratingCount: 366 but reviewsWithText: 191. Star ratings without a comment are counted in the average and are not text you can read, so a job sized at "366 reviews" will finish at roughly half that and look like it failed. Size review jobs by reviewsWithText.

The breakdown row is the part I would use even if I never read a single comment: it gives you the true star distribution — 76% five-star here — without paginating anything. Scraping page one of the reviews and counting stars gives you the relevance-sorted distribution, which is a different and much happier number.

Three more details from the same run. The endpoint returns 15 reviews per page, hard — ask for 100 and you are making seven requests. Every title field was null; the text instead carries Mercado Libre's guided-form prefixes, which is free structure if you want per-aspect sentiment:

"Diseño: hermoso el estilo y el color negro…
 Comodidad: bastante cómodas por dentro, y son ligeras."

And each review names the variant bought — "Color: Negro · Talle: 44 AR" — plus both a relative date ("Hace 6 meses") and a real timestamp (2026-03-17T15:57:45Z). Use the second one.

Calling it

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("lergassy/mercadolibre-reviews-scraper").call(run_input={
    "mode": "search",
    "country": "AR",
    "searchTerms": ["zapatillas"],
    "maxItems": 100,
    "includeSponsored": False,
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["itemId"], row["price"], row["currency"], row.get("soldQuantity"))

The Actor lives at apify.com/lergassy/mercadolibre-reviews-scraper — search, product detail, reviews, daily deals and official-store modes in one input, dataset rows priced per result with no per-run fee.

When this is the wrong tool

If you are a Mercado Libre seller working with your own listings, orders or stock, use the official API — you have the OAuth credentials and the permissions, and you get write access that no scraper can give you. If you need one price checked once, the product page in a browser is faster than any pipeline. And if you need a guaranteed-complete category, no scraper can promise it: the 9,999 ceiling is real, and the only honest way past it is to slice the query by brand, price band or subcategory until each slice comes back under the cap.

What scraping is genuinely good for here is the public, cross-seller, cross-country view that the seller API will never hand you: what the competition charges in six currencies, which of their listings are paid placements, and what 191 buyers said about a product you do not sell.