<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Matvey</title>
    <description>The latest articles on DEV Community by Matvey (@matveyg).</description>
    <link>https://dev.to/matveyg</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4108814%2F458cdc93-81e5-41b6-966c-33d6453a5c2b.jpg</url>
      <title>DEV Community: Matvey</title>
      <link>https://dev.to/matveyg</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/matveyg"/>
    <language>en</language>
    <item>
      <title>I Asked Instagram for NASA's 12 Newest Reels. Nine Belonged to Other Accounts</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Sun, 18 Oct 2026 13:13:38 +0000</pubDate>
      <link>https://dev.to/matveyg/i-asked-instagram-for-nasas-12-newest-reels-nine-belonged-to-other-accounts-3e7o</link>
      <guid>https://dev.to/matveyg/i-asked-instagram-for-nasas-12-newest-reels-nine-belonged-to-other-accounts-3e7o</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The half of my week that is not visas goes into Apify Actors, and one of them reads short vertical video: give it Reel, TikTok or Shorts links — or a whole profile — and it returns one flat row per video with the transcript, the text burned into the picture, and the engagement numbers the platform showed.&lt;/p&gt;

&lt;p&gt;Profile mode is what people buy it for: &lt;em&gt;watch this creator, transcribe whatever they post.&lt;/em&gt; It is also where Instagram hands you a different dataset than the one you asked for, in two separate ways. I measured both on 18 October 2026, and the second one changes which videos your pipeline calls viral.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;One run, logged out, no cookies, no official API: profiles &lt;code&gt;instagram.com/nasa/&lt;/code&gt; and &lt;code&gt;instagram.com/natgeo/&lt;/code&gt;, twelve newest Reels each, on-screen text off. 24 rows, 24 transcribed, zero failures, 59 seconds end to end.&lt;/p&gt;

&lt;p&gt;Then a second run, 19 seconds, with the ten own-authored Reels of one of those profiles pasted in as plain links. Same videos, different input shape.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Reels tab is a grid, not an authorship list
&lt;/h2&gt;

&lt;p&gt;Here is who actually wrote the twelve Reels that came back for &lt;code&gt;@nasa&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;nasa            3
astro_jessica   2
nasajohnson     2
eaglescheer     1
ravens          1
nasaadmin       1
astro_anil      1
nasaaero        1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three of twelve. The rest are other accounts' Reels sitting on NASA's grid — other NASA centres, individual astronauts, two sports teams. One is from the Philadelphia Eagles cheerleaders, captioned &lt;code&gt;An inspiration to us all, thank you for stopping by @astro_christina 💚&lt;/code&gt;, 864,791 views. A real Reel on a public page, correctly returned. Just not NASA's.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;@natgeo&lt;/code&gt; is tidier and still not clean: ten of twelve are &lt;code&gt;natgeo&lt;/code&gt;, one is &lt;code&gt;natgeofamily&lt;/code&gt;, one is &lt;code&gt;911onabc&lt;/code&gt; — a Disney sibling show.&lt;/p&gt;

&lt;p&gt;The mechanism is collaborator posts: a co-authored Reel is served on both profiles, and the owner field names whoever posted it. The page hands a logged-out visitor the grid as rendered — it has no concept of "only posts this account authored", and neither does any scraper reading it.&lt;/p&gt;

&lt;p&gt;So if your question is &lt;em&gt;how often does this creator post and what do they say&lt;/em&gt;, the profile count is not the answer. The field you need is already in the row:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;mine&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;author&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;nasa&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Which sounds obvious until you see what it costs downstream.&lt;/p&gt;

&lt;h2&gt;
  
  
  The baseline that never formed
&lt;/h2&gt;

&lt;p&gt;Every row carries &lt;code&gt;vsAuthorMedian&lt;/code&gt;: how many times a video beat its own author's usual, computed from the other videos by that author in the same run. At least three peers are required, otherwise the field is &lt;code&gt;null&lt;/code&gt; — a ratio computed from one neighbour is a coin flip wearing a decimal point.&lt;/p&gt;

&lt;p&gt;For &lt;code&gt;@natgeo&lt;/code&gt; the ten own-authored Reels gave each other a baseline and the run flagged the outliers. For &lt;code&gt;@nasa&lt;/code&gt;, three own Reels out of twelve, the baseline never formed: &lt;code&gt;vsAuthorMedian&lt;/code&gt;, &lt;code&gt;authorMedian&lt;/code&gt; and &lt;code&gt;isOutlier&lt;/code&gt; came back &lt;code&gt;null&lt;/code&gt; on all twelve rows. Twelve clean transcripts, no virality signal at all.&lt;/p&gt;

&lt;p&gt;That is the right behaviour — pooling eight accounts into one median produces a number that means nothing — but it is not what you expect from "I asked for twelve videos". On a collab-heavy account you would have to raise the ask until twelve own-authored Reels fit inside it, and logged out you cannot: twelve is the whole ceiling.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ten identical videos, two different virality answers
&lt;/h2&gt;

&lt;p&gt;Now the second finding, and the one I would put on a sticker.&lt;/p&gt;

&lt;p&gt;Instagram shows view counts on the Reels tab of a public profile. It does &lt;strong&gt;not&lt;/strong&gt; show them on a single Reel page to a logged-out visitor. Same ten &lt;code&gt;@natgeo&lt;/code&gt; Reels, run both ways:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;videoId&lt;/th&gt;
&lt;th&gt;views (profile run)&lt;/th&gt;
&lt;th&gt;views (link run)&lt;/th&gt;
&lt;th&gt;likes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;DemLxMGgoL_&lt;/td&gt;
&lt;td&gt;4,072,300&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;205,239&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dec1lmtgGYk&lt;/td&gt;
&lt;td&gt;642,521&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;51,232&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeobYzjga9x&lt;/td&gt;
&lt;td&gt;334,391&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;96,650&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DehZj9wggt0&lt;/td&gt;
&lt;td&gt;286,821&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;49,146&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeckcQnA23l&lt;/td&gt;
&lt;td&gt;238,557&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;21,594&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Delg3cUktih&lt;/td&gt;
&lt;td&gt;189,002&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;10,379&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeeUUHNAqQj&lt;/td&gt;
&lt;td&gt;188,567&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;14,570&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Del2uq1gnwg&lt;/td&gt;
&lt;td&gt;178,700&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;17,062&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DefFZgggApb&lt;/td&gt;
&lt;td&gt;172,808&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;43,590&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DecPSQfoaRe&lt;/td&gt;
&lt;td&gt;110,213&lt;/td&gt;
&lt;td&gt;&lt;code&gt;null&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;21,124&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Likes survive both runs and agree to within 549 on the noisiest row — they are live counters, so small drift between two runs minutes apart is expected. Views simply are not in the single-Reel payload.&lt;/p&gt;

&lt;p&gt;The Actor handles that by falling back: when every video in an author group has views it compares on views, otherwise on likes, and &lt;code&gt;vsAuthorBasis&lt;/code&gt; says which. Honest, and still not interchangeable. Ranking those ten Reels by likes instead of views gives:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;11 of 45 pairs in the wrong order (Spearman ρ = 0.71)&lt;/li&gt;
&lt;li&gt;like-to-view ratio from &lt;strong&gt;5.04% to 28.90%&lt;/strong&gt; across the same ten videos of one account — a 5.7× spread&lt;/li&gt;
&lt;li&gt;a different set of outliers&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last one is the part that reaches your dashboard. At the 3× threshold:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;basis = views  -&amp;gt;  outliers: DemLxMGgoL_ (21.55x), Dec1lmtgGYk (3.40x)
basis = likes  -&amp;gt;  outliers: DemLxMGgoL_  (9.51x), DeobYzjga9x (4.50x)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One video in, one video out, on identical input videos. &lt;code&gt;Dec1lmtgGYk&lt;/code&gt; pulled 642k views on a 7.97% like rate — broad reach, ordinary engagement. &lt;code&gt;DeobYzjga9x&lt;/code&gt; got a third of that reach at a 28.90% like rate — a small, very warm audience. Views call the first a breakout, likes call the second. Both are defensible; neither substitutes for the other, and if half your rows arrived by link and half by profile, your "top videos" list is mixing two scales.&lt;/p&gt;

&lt;p&gt;Practical rule: do not mix input shapes inside one analysis. Pull by profile when you need reach; if a Reel has to arrive by link, label its &lt;code&gt;vsAuthorMedian&lt;/code&gt; as an engagement figure — &lt;code&gt;vsAuthorBasis&lt;/code&gt; is in the row precisely so you can.&lt;/p&gt;

&lt;h2&gt;
  
  
  The twelve-item ceiling is a scheduling constraint
&lt;/h2&gt;

&lt;p&gt;Instagram's logged-out Reels tab renders twelve items. That is not a parameter the Actor could raise; it is the page. Which means how often you need to run is set by how fast the creator fills twelve slots.&lt;/p&gt;

&lt;p&gt;In my two runs the twelve newest Reels covered:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;@nasa&lt;/code&gt; — 14 September to 10 October, 26 days&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;@natgeo&lt;/code&gt; — 13 October to 18 October, &lt;strong&gt;5 days&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A weekly watch on &lt;code&gt;@natgeo&lt;/code&gt; would have silently missed Reels; on &lt;code&gt;@nasa&lt;/code&gt; it would have been comfortable. Same ceiling, 5× difference in how long the window lasts. Measure the span of your first run, set the schedule to a fraction of it, and turn on "only videos not processed before" so the overlap costs nothing — already-seen videos are skipped before download and not billed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the whole exercise cost
&lt;/h2&gt;

&lt;p&gt;24 Reels transcribed, speech recognition on, on-screen text off: &lt;strong&gt;$0.2422&lt;/strong&gt; total, per-row from $0.0028 to $0.0364. The cheap rows are the silent ones: four of the ten &lt;code&gt;@natgeo&lt;/code&gt; Reels had no speech, came back with &lt;code&gt;transcriptSource: no-speech&lt;/code&gt;, and were billed for the video event only, no audio charge. The expensive row was 236 seconds long. Re-running the ten Reels as links cost $0.0658.&lt;/p&gt;

&lt;p&gt;Calling it is a POST and a read:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;

&lt;span class="n"&gt;TOKEN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;your apify token&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.apify.com/v2/acts/lergassy~instagram-reels-transcript-scraper&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/run-sync-get-dataset-items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;token&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;TOKEN&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;profiles&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.instagram.com/natgeo/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resultsLimitPerProfile&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;includeOnScreenText&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;onlyNewVideos&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;watchlistName&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;natgeo-weekly&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;mine&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;reel&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;author&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;natgeo&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows -&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mine&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;authored by the profile&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;basis&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;vsAuthorBasis&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;mine&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;comparison basis:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;basis&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;# {'views'} by profile, {'likes'} by link
&lt;/span&gt;
&lt;span class="n"&gt;span&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;postedAt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;mine&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;window:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;span&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;  &lt;span class="c1"&gt;# set your schedule inside this
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three lines of hygiene, in this order, before anything aggregates: filter on &lt;code&gt;author&lt;/code&gt;, assert a single &lt;code&gt;vsAuthorBasis&lt;/code&gt;, read the date span. Full field list and the input form: &lt;a href="https://apify.com/lergassy/instagram-reels-transcript-scraper?fpr=wmeplu" rel="noopener noreferrer"&gt;Instagram Reels Transcript Scraper&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  When this is the wrong tool
&lt;/h2&gt;

&lt;p&gt;If you need more than twelve Reels of a profile in one go, this is not it — and no logged-out scraper is, because the page stops there. Older Reels need their own links, which drops the views again. If you need historical view counts, also stop: Instagram reports a live counter, so a time series only exists if you start snapshotting now. And if your question is purely &lt;em&gt;what was said&lt;/em&gt;, skip profile mode and paste links — authorship and reach stop mattering, and the whole problem above goes away.&lt;/p&gt;

&lt;p&gt;The general lesson is not about Instagram. A profile endpoint answers "what is displayed on this page", and you read it as "what this account produced". Those two differ by nine rows out of twelve on a heavily collaborating account, and the field that tells you so is sitting in the response, ignored.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>api</category>
      <category>dataanalysis</category>
    </item>
    <item>
      <title>Airbnb Snapped 'Seminyak' to 'Kuta': 90 of 116 Listings Were Labelled Somewhere Else</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Sun, 18 Oct 2026 05:12:59 +0000</pubDate>
      <link>https://dev.to/matveyg/airbnb-snapped-seminyak-to-kuta-90-of-116-listings-were-labelled-somewhere-else-34f</link>
      <guid>https://dev.to/matveyg/airbnb-snapped-seminyak-to-kuta-90-of-116-listings-were-labelled-somewhere-else-34f</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them reads Airbnb search pages: you give it a place, two dates and a party size, and it returns one flat row per listing — total price for the stay, price per night, rating, review count, beds, coordinates.&lt;/p&gt;

&lt;p&gt;The question people actually buy that for is a pricing question. &lt;em&gt;What does a two-bedroom in Seminyak go for in mid-November?&lt;/em&gt; I live twenty minutes from Seminyak, which gave me an advantage over most of my own users: I could tell that several villas in the answer were not in Seminyak at all.&lt;/p&gt;

&lt;p&gt;Here is what is going on, measured, because it changes every average you compute downstream.&lt;/p&gt;

&lt;h2&gt;
  
  
  The place you type is not the place that gets searched
&lt;/h2&gt;

&lt;p&gt;Three scheduled runs, 16, 17 and 18 October 2026, same input every time: &lt;code&gt;Seminyak, Bali, Indonesia&lt;/code&gt;, a three-night stay a month out, two adults, prices in USD, 60 listings. 180 rows, 116 unique listings across the three days.&lt;/p&gt;

&lt;p&gt;Every row carries both the string I asked for and the place Airbnb decided I meant:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="err"&gt;query&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Seminyak, Bali, Indonesia"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="err"&gt;place&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Kuta, Indonesia"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Kuta is the &lt;em&gt;kecamatan&lt;/em&gt; — the district. Seminyak is one village inside it. The search ran across the whole district, and the row says so, in a field most people never read.&lt;/p&gt;

&lt;p&gt;What that does to the result set, counting the &lt;code&gt;area&lt;/code&gt; label on all 116 unique listings:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Kuta                     37
Seminyak                 26
Kuta Utara               24
Kecamatan Kuta Utara     15
South Kuta                3
Umalas                    3
Kabupaten Badung          3
Pererenan                 2
Denpasar                  1
Berawa                    1
Kecamatan Denpasar Barat  1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Twenty-six of 116. Pererenan and Berawa are Canggu — a different beach, a different market, a 25-minute drive in traffic. Denpasar is the provincial capital, inland.&lt;/p&gt;

&lt;p&gt;Coordinates confirm it. Taking Seminyak Beach as −8.6900 / 115.1685 and measuring great-circle distance per run:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;n&lt;/th&gt;
&lt;th&gt;median&lt;/th&gt;
&lt;th&gt;p90&lt;/th&gt;
&lt;th&gt;max&lt;/th&gt;
&lt;th&gt;beyond 5 km&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;16 Oct&lt;/td&gt;
&lt;td&gt;60&lt;/td&gt;
&lt;td&gt;1.5 km&lt;/td&gt;
&lt;td&gt;4.0 km&lt;/td&gt;
&lt;td&gt;7.4 km&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;17 Oct&lt;/td&gt;
&lt;td&gt;60&lt;/td&gt;
&lt;td&gt;1.3 km&lt;/td&gt;
&lt;td&gt;3.7 km&lt;/td&gt;
&lt;td&gt;7.1 km&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;18 Oct&lt;/td&gt;
&lt;td&gt;60&lt;/td&gt;
&lt;td&gt;1.4 km&lt;/td&gt;
&lt;td&gt;3.9 km&lt;/td&gt;
&lt;td&gt;7.1 km&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Stable across days, so this isn't one odd page. The two farthest rows:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; 7.4 km  Pererenan    $116.00/night  Villa Kayu: Chic Minimalist Retreat Near Beach
 7.1 km  Pererenan    $186.33/night  Super Unique &amp;amp; Stunning Architecture @Bocoa Villas
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Airbnb obfuscates coordinates slightly before you book, so treat the sub-kilometre figures as soft. It does not obfuscate by seven kilometres.&lt;/p&gt;

&lt;h2&gt;
  
  
  The text label is the wrong filter
&lt;/h2&gt;

&lt;p&gt;The obvious fix is to throw away everything where &lt;code&gt;area != "Seminyak"&lt;/code&gt;. That is worse than doing nothing, and the same 116 listings show why.&lt;/p&gt;

&lt;p&gt;Sort by distance and look at the three &lt;em&gt;closest&lt;/em&gt; rows in the whole sample:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; 0.2 km  Kuta   $152.00/night  One Bedroom Villa with Private Pool in Seminyak
 0.2 km  Kuta   $ 78.33/night  The Catur Villa - Macaca 2BR Villa
 0.3 km  Kuta   $122.33/night  Calladi Villa 2BR Luxury with 300m² Big Garden
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The nearest listing to the centre of Seminyak is labelled &lt;code&gt;Kuta&lt;/code&gt; and has the word Seminyak in its own title. The label is whatever administrative string Airbnb happened to render for that card, and the list above shows it is not even one consistent level: village (&lt;code&gt;Seminyak&lt;/code&gt;, &lt;code&gt;Umalas&lt;/code&gt;, &lt;code&gt;Pererenan&lt;/code&gt;), district (&lt;code&gt;Kuta Utara&lt;/code&gt;), the same district again with the Indonesian prefix attached (&lt;code&gt;Kecamatan Kuta Utara&lt;/code&gt; — 24 rows one way, 15 the other), and regency (&lt;code&gt;Kabupaten Badung&lt;/code&gt;, which contains all of the above). You cannot &lt;code&gt;GROUP BY&lt;/code&gt; that column.&lt;/p&gt;

&lt;p&gt;So filter on geometry instead. Same 116 listings, two definitions of "in Seminyak":&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Filter&lt;/th&gt;
&lt;th&gt;n&lt;/th&gt;
&lt;th&gt;median $/night&lt;/th&gt;
&lt;th&gt;mean $/night&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;everything returned&lt;/td&gt;
&lt;td&gt;116&lt;/td&gt;
&lt;td&gt;112.00&lt;/td&gt;
&lt;td&gt;113.48&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;area == "Seminyak"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;127.67&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;137.47&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;within 1.5 km of the beach&lt;/td&gt;
&lt;td&gt;68&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;108.33&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;105.32&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;beyond 1.5 km&lt;/td&gt;
&lt;td&gt;48&lt;/td&gt;
&lt;td&gt;116.00&lt;/td&gt;
&lt;td&gt;125.03&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;beyond 3 km&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;119.33&lt;/td&gt;
&lt;td&gt;131.69&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The label filter reports a median 18% above the coordinate filter, on one dataset, for the same week and the same party size. It keeps 26 rows where the honest answer keeps 68, and the 26 are not a random subsample — Airbnb's label skews toward the expensive main-strip inventory. If you are a host setting a rate off that number you are pricing against a market that does not exist.&lt;/p&gt;

&lt;p&gt;Mean rating barely moves between the bands (4.90 against 4.93) — the tell that the price gap is geography, not a quality split.&lt;/p&gt;

&lt;h2&gt;
  
  
  It isn't a Bali thing
&lt;/h2&gt;

&lt;p&gt;Two more searches, 18 October 2026, same shape, from a throwaway account:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="err"&gt;query&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Shoreditch, London, United Kingdom"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="err"&gt;place&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Greater London, United Kingdom"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Greater London is roughly 1,570 km². Those 20 rows are much tighter than Bali's — median 0.4 km from Shoreditch High Street, 18 of 20 inside 3 km — but the two that escaped are the two that matter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;11.8 km  London    $ 36.33  A home away from home,
 6.9 km  Newham    $ 40.00  3 bedroom house
 0.7 km  London    $341.00  2BR Shoreditch Flat with Balcony, near the tube
 0.6 km  London    $297.67  Stylish 2-Bed Apartment In Shoreditch
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ten percent of the rows, and they set your floor: $36 against a Shoreditch core around $275–340. Any "cheapest available in Shoreditch" built on this is answering a question about Newham.&lt;/p&gt;

&lt;p&gt;And the one that surprised me most:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="err"&gt;query&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Williamsburg, Brooklyn, New York"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="err"&gt;place&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"New York City, NY"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A neighborhood of Brooklyn widened to the entire city. Five of those ten rows were not homes at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Pod Brooklyn Hotel"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"propertyType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Pod Brooklyn Hotel"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"area"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"priceTotal"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pricePerNight"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"badges"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Featured hotel"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"bedInfo"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rating"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;4.76&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"reviewCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;629&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Airbnb's hotel inventory. No price in the search card, no area, no beds — and one of them, Pod 39, sits in Murray Hill in Manhattan, 4.3 km from Williamsburg. I deliberately emit &lt;code&gt;null&lt;/code&gt; rather than &lt;code&gt;0&lt;/code&gt; for those, because a zero would quietly drag an average down and nobody would notice. But &lt;code&gt;len(dataset)&lt;/code&gt; is still not your sample size: asking for ten listings got me five priced homes.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to actually do with the rows
&lt;/h2&gt;

&lt;p&gt;Three lines, in this order, before any aggregation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;TOKEN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;your apify token&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.apify.com/v2/acts/lergassy~airbnb-scraper/run-sync-get-dataset-items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;token&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;TOKEN&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;locationQueries&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Seminyak, Bali, Indonesia&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;checkIn&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-11-17&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;checkOut&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2026-11-20&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;adults&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;currency&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;USD&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;maxListings&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;km&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lon&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;alat&lt;/span&gt;&lt;span class="o"&gt;=-&lt;/span&gt;&lt;span class="mf"&gt;8.6900&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;alon&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;115.1685&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pi&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;180&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;6371&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;asin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sin&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;lat&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;alat&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
        &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cos&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;alat&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cos&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lat&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sin&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;lon&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;alon&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="n"&gt;homes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;
         &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;priceTotal&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;                  &lt;span class="c1"&gt;# drops hotel cards
&lt;/span&gt;         &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Featured hotel&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;badges&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;
         &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;km&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;latitude&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;longitude&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mf"&gt;1.5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;        &lt;span class="c1"&gt;# drops the next district
&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows -&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;homes&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;in scope&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Read &lt;code&gt;place&lt;/code&gt;, not just &lt;code&gt;query&lt;/code&gt;.&lt;/strong&gt; If they differ, the search was widened and you now need a geographic cut. In my runs they differed in all three cities.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cut on coordinates, with a radius you chose.&lt;/strong&gt; Not on &lt;code&gt;area&lt;/code&gt;, ever.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Drop the hotel cards&lt;/strong&gt; before you average anything, or report them separately.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Pasting an Airbnb search URL into &lt;code&gt;startUrls&lt;/code&gt; instead of typing a place does not save you — Airbnb resolves the place server-side either way, and the URL's own &lt;code&gt;map bounds&lt;/code&gt; are advisory. The only honest boundary is the one you draw on the coordinates you got back. Actor and full field list: &lt;a href="https://apify.com/lergassy/airbnb-scraper?fpr=wmeplu" rel="noopener noreferrer"&gt;Airbnb Scraper&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  When this is the wrong tool
&lt;/h2&gt;

&lt;p&gt;If you need the &lt;em&gt;exact&lt;/em&gt; street address, stop here — Airbnb only reveals it after booking, and no scraper changes that. If you need historic nightly rates, also stop: Airbnb quotes live availability, so a price series only exists if you start collecting now and keep the snapshots. And if your question is a single listing's amenities rather than a market's prices, run the listing URL directly — a search is the one thing that can hand you a price, and the one thing that will hand you the wrong neighborhood while doing it.&lt;/p&gt;

&lt;p&gt;The general lesson has nothing to do with Airbnb. Any search endpoint that accepts free-text place names is doing geocoding you cannot see, and the polite ones tell you what they decided. Read that field. It is the difference between a market report and a confident average of the wrong town.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>api</category>
      <category>python</category>
      <category>dataanalysis</category>
    </item>
    <item>
      <title>398 of 692 Products Were Published in One Afternoon. Zero Were Created in 90 Days</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Sat, 17 Oct 2026 21:11:51 +0000</pubDate>
      <link>https://dev.to/matveyg/398-of-692-products-were-published-in-one-afternoon-zero-were-created-in-90-days-54i4</link>
      <guid>https://dev.to/matveyg/398-of-692-products-were-published-in-one-afternoon-zero-were-created-in-90-days-54i4</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them profiles Shopify stores: you hand it a list of domains and it returns one flat row per shop — who the merchant is, the e-mails and socials they published, the theme and apps the storefront runs, catalogue size, price band.&lt;/p&gt;

&lt;p&gt;Nobody buys that row for the price band. They buy it for one question that decides whether a store is worth a pitch, an outreach e-mail or a line in a market sizing: &lt;strong&gt;is anyone still working on this shop?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Shopify answers that question in public, for free, on every store that hasn't switched it off. It just answers it with two fields that look interchangeable and are not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The measurement
&lt;/h2&gt;

&lt;p&gt;On 17 October 2026 I read &lt;code&gt;/products.json&lt;/code&gt; from four Shopify stores — no login, no API key, no proxy, the same JSON the storefront downloads for itself — and counted products two ways: how many were &lt;em&gt;created&lt;/em&gt; in the last 90 days, and how many were &lt;em&gt;published&lt;/em&gt; in the last 90 days.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Store&lt;/th&gt;
&lt;th&gt;Sample&lt;/th&gt;
&lt;th&gt;Created in 90 days&lt;/th&gt;
&lt;th&gt;Published in 90 days&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;allbirds.com&lt;/td&gt;
&lt;td&gt;692&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;404&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ridge.com&lt;/td&gt;
&lt;td&gt;750&lt;/td&gt;
&lt;td&gt;157&lt;/td&gt;
&lt;td&gt;295&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;deathwishcoffee.com&lt;/td&gt;
&lt;td&gt;147 (whole catalogue)&lt;/td&gt;
&lt;td&gt;17&lt;/td&gt;
&lt;td&gt;37&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;kith.com&lt;/td&gt;
&lt;td&gt;750&lt;/td&gt;
&lt;td&gt;410&lt;/td&gt;
&lt;td&gt;500&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every store reports more recent publishing than recent creation. That gap is the whole post. On a genuinely busy shop it's roughly a doubling and both numbers point the same way. On Allbirds the two fields disagree about whether the catalogue has moved at all this quarter: one says 404 products, the other says none.&lt;/p&gt;

&lt;p&gt;Sort the Allbirds sample by &lt;code&gt;published_at&lt;/code&gt; and the reason shows up immediately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;top published_at days: 2026-09-25 → 398 products
                       2026-06-16 → 222
                       2026-06-11 →  50
top created_at days:   2025-08-06 → 116
                       2025-07-23 →  44
                       2025-07-03 →  20
created_at by year:    2018: 5 · 2019: 7 · 2020: 11 · 2021: 28 · 2022: 35
                       2023: 85 · 2024: 154 · 2025: 340 · 2026: 27
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;398 of the 692 products in that sample carry a &lt;code&gt;published_at&lt;/code&gt; inside one afternoon — the earliest at 14:51:05, the latest at 16:58:13 on 25 September 2026. Two hours and seven minutes for 58% of a catalogue that took eight years to build. Here is one of them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="s"&gt;Men's Canvas Runner NZ - Deep Navy Stripes&lt;/span&gt;
  &lt;span class="s"&gt;created_at&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;   &lt;span class="s"&gt;2025-12-04T12:15:25-08:00&lt;/span&gt;
  &lt;span class="s"&gt;published_at&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2026-09-25T16:58:09-07:00&lt;/span&gt;
  &lt;span class="s"&gt;updated_at&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;   &lt;span class="s"&gt;2026-10-17T14:08:56-07:00&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A product that has existed for ten months, stamped as published three weeks ago. The newest &lt;code&gt;created_at&lt;/code&gt; anywhere in those 692 products is 2026-06-05 — four months before the run. The oldest is 2018-11-12.&lt;/p&gt;

&lt;p&gt;I can't see from outside which admin operation did that: a sales-channel change, a market or theme migration, a bulk edit, a re-publish of unpublished seasonal lines. It doesn't matter for the consumer of the data. What matters is that &lt;code&gt;published_at&lt;/code&gt; is not a creation timestamp and nothing in the payload says so. An activity metric built on it reports a store that added nothing in a quarter as the most active shop in your list.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;updated_at&lt;/code&gt; is worse
&lt;/h2&gt;

&lt;p&gt;The other candidate is the one most people reach for second. Don't:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;allbirds.com        250 of 250 products · updated_at = 2026-10-17
deathwishcoffee.com 147 of 147 products · updated_at = 2026-10-17
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every product in both catalogues, 100% of each, updated today. Inventory sync, price feeds and app writes touch the record constantly, so &lt;code&gt;updated_at&lt;/code&gt; is a heartbeat of Shopify's own plumbing, not of the merchant. As a freshness signal it has no variance at all, which makes it exactly as informative as a constant.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the Actor counts instead
&lt;/h2&gt;

&lt;p&gt;Nine lines, and the comment is the whole argument:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="c1"&gt;// Count on created_at, not published_at: Shopify bumps published_at whenever a product is&lt;/span&gt;
&lt;span class="c1"&gt;// re-published, so a store that changed nothing all year still looks brand new.&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;fresh&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;products&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;created_at&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;created_at&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;864&lt;/span&gt;&lt;span class="nx"&gt;e5&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;created_at&lt;/code&gt; is the field that only moves once. It survives re-publishing, channel changes and theme migrations, because none of those create a product record — they touch one that already exists.&lt;/p&gt;

&lt;p&gt;That lands in the store row as three fields: &lt;code&gt;productsCreatedLast90Days&lt;/code&gt;, &lt;code&gt;newestProductAt&lt;/code&gt; and &lt;code&gt;oldestProductAt&lt;/code&gt;. Here is the real Allbirds row from a run of the published Actor on 17 October, trimmed to the parts under discussion:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rowType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"store"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"shopName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Allbirds"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"myshopifyDomain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"weareallbirds.myshopify.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"help@allbirds.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"emails"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"help@allbirds.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"travis@allbirds.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"press@allbirds.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"info@afterpay.com"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"theme"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"[DNAM Theme July 2026]"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"productCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;250&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"totalProducts"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;692&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"catalogueComplete"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"variantCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2720&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"inStockVariants"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;495&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"outOfStockVariants"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2225&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"priceMin"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"priceMedian"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"priceMax"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;165&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"newestProductAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-05-26T11:10:21-07:00"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"oldestProductAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2018-11-12T14:25:43-08:00"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"productsCreatedLast90Days"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pageCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;175&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"firstPageAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2017-04-25T14:22:24-07:00"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three other numbers in that row point the same direction as the zero: 2,225 of 2,720 variants out of stock, and a catalogue whose oldest content page dates to April 2017. That's a store with eight years of history and nothing new on the shelf this quarter — which is a perfectly good thing to know before you spend a sales hour on it, and the exact opposite of what &lt;code&gt;published_at&lt;/code&gt; would have told you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two caveats, because this is not a law of nature
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;created_at&lt;/code&gt; lies about replatforming.&lt;/strong&gt; It records when the &lt;em&gt;record&lt;/em&gt; was made, not when the &lt;em&gt;product&lt;/em&gt; existed. Kith shows 665 of 750 sampled products created in 2026 and 1 in 2018. Some of that is a genuinely fast fashion catalogue; some of it is what happens when a catalogue is rebuilt or bulk-imported, because every imported product gets a brand-new &lt;code&gt;created_at&lt;/code&gt;. A store that migrated last month looks newborn on this metric. There is no public field that survives that, and I'm not going to pretend otherwise — the honest read is "this record is young", not "this business is young".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The sample is not the catalogue.&lt;/strong&gt; &lt;code&gt;/products.json&lt;/code&gt; pages out at whatever cap you set — 250 by default. Allbirds returned &lt;code&gt;productCount: 250&lt;/code&gt; against &lt;code&gt;totalProducts: 692&lt;/code&gt;, counted separately from the store's product sitemap, with &lt;code&gt;catalogueComplete: false&lt;/code&gt; saying so out loud. Kith came back 250 against &lt;strong&gt;34,970&lt;/strong&gt;. Every catalogue figure in the row, including the 90-day count, describes the sample; the Actor's own &lt;code&gt;newestProductAt&lt;/code&gt; for Allbirds reads 2026-05-26 from its 250-product sample against 2026-06-05 across all 692. If the activity number is what you're buying, raise the per-store product cap and read &lt;code&gt;catalogueComplete&lt;/code&gt; before you trust it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fourth store
&lt;/h2&gt;

&lt;p&gt;One of the four addresses in that run never produced a profile:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rowType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"storeUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://ridge.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"The shop refused the request (HTTP 403), so it could not be checked. Slow the run down with fewer parallel stores, or try again later."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"scrapedAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-10-17T21:09:32.400Z"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;ridge.com rate-limited the Actor during the run, having answered a plain request from the same machine a few minutes earlier — that's the 157/295 row in the table above. The thing I care about is that it comes back as &lt;em&gt;refused&lt;/em&gt;, not as &lt;code&gt;isShopify: false&lt;/code&gt;. A scraper that reports a 403 as "not a Shopify store" silently deletes real shops from a prospect list and gives you no way to notice. Error rows cost nothing and one bad address doesn't stop the run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it
&lt;/h2&gt;

&lt;p&gt;One HTTP call, rows back in the same request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/lergassy~shopify-store-intel/run-sync-get-dataset-items?token=&amp;lt;TOKEN&amp;gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"storeUrls":["allbirds.com","deathwishcoffee.com"],"mode":"store","maxProductsPerStore":250}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four addresses in store mode took 22 seconds end to end. The Actor is &lt;a href="https://apify.com/lergassy/shopify-store-intel?fpr=wmeplu" rel="noopener noreferrer"&gt;shopify-store-intel&lt;/a&gt;; the dates, the 403 and the Allbirds row above all come from that run.&lt;/p&gt;

&lt;h2&gt;
  
  
  When not to bother
&lt;/h2&gt;

&lt;p&gt;If you only need to know &lt;em&gt;whether&lt;/em&gt; a domain is Shopify, don't profile it — detect mode answers that from one request. If you need stock levels or sales volume, this is the wrong data source entirely: &lt;code&gt;/products.json&lt;/code&gt; publishes an &lt;code&gt;available&lt;/code&gt; boolean per variant and no quantities, and nothing public tells you what sold. And if your list is stores you already know are active, the activity fields are dead weight — buy the contacts and skip the catalogue.&lt;/p&gt;

&lt;p&gt;But if the list is 5,000 domains scraped off a directory and your real question is which 400 are worth a human reading them, count creations. Counting publications tells you when somebody last clicked Save.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>ecommerce</category>
      <category>api</category>
      <category>node</category>
    </item>
    <item>
      <title>Three of Five Founder Mailboxes Rejected first.last, the Layout Everyone Defaults To</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Sat, 17 Oct 2026 13:12:16 +0000</pubDate>
      <link>https://dev.to/matveyg/three-of-five-founder-mailboxes-rejected-firstlast-the-layout-everyone-defaults-to-2n7p</link>
      <guid>https://dev.to/matveyg/three-of-five-founder-mailboxes-rejected-firstlast-the-layout-everyone-defaults-to-2n7p</guid>
      <description>&lt;p&gt;I run a visa agency in Bali, and the other half of my week goes into Apify Actors. One of them is narrow on purpose: you give it a LinkedIn profile URL, it gives you back the person's name, title, current company, location — and their work e-mail address with the mailbox checked over SMTP.&lt;/p&gt;

&lt;p&gt;The part that looks like the hard problem is finding the address. It isn't. Writing &lt;code&gt;first.last@domain.com&lt;/code&gt; takes one line of code, and that is exactly what most of this category ships, with a label like &lt;code&gt;deliverable&lt;/code&gt; next to it. The hard problem is the one that appears the moment you actually ask a mail server whether the thing you just wrote down exists: &lt;strong&gt;you only get a couple of questions, so the order you ask them in is the product.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here is what that costs, measured on nine runs between 10 and 17 October 2026 across six public profiles of founders and executives (builds 0.1.32 to 0.1.41).&lt;/p&gt;

&lt;h2&gt;
  
  
  The ranked guess was wrong more often than it was right
&lt;/h2&gt;

&lt;p&gt;Every one of the six people got the same treatment: read the public profile, resolve the company to a domain, build candidate addresses in the company's own layout, then check mailboxes over SMTP — at most two per person — and keep the first one the server confirms.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Person&lt;/th&gt;
&lt;th&gt;Domain&lt;/th&gt;
&lt;th&gt;Check 1 (ranked first)&lt;/th&gt;
&lt;th&gt;Check 2&lt;/th&gt;
&lt;th&gt;Delivered&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Bill Gates&lt;/td&gt;
&lt;td&gt;gatesfoundation.org&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;bill.gates@&lt;/code&gt; → &lt;code&gt;safe&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;code&gt;bill.gates@&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Garry Tan&lt;/td&gt;
&lt;td&gt;ycombinator.com&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;garry.tan@&lt;/code&gt; → &lt;code&gt;safe&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;&lt;code&gt;garry.tan@&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Satya Nadella&lt;/td&gt;
&lt;td&gt;microsoft.com&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;satya.nadella@&lt;/code&gt; → &lt;code&gt;invalid&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;satya@&lt;/code&gt; → &lt;code&gt;safe&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;satya@&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rand Fishkin&lt;/td&gt;
&lt;td&gt;sparktoro.com&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;rand.fishkin@&lt;/code&gt; → &lt;code&gt;invalid&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;rand@&lt;/code&gt; → &lt;code&gt;safe&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;rand@&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Jeff Weiner&lt;/td&gt;
&lt;td&gt;nextplayventures.com&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;jeff.weiner@&lt;/code&gt; → &lt;code&gt;invalid&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;jeff@&lt;/code&gt; → &lt;code&gt;safe&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;jeff@&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Patrick Collison&lt;/td&gt;
&lt;td&gt;stripe.com&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;patrick.collison@&lt;/code&gt; → &lt;code&gt;catch_all&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;patrick.collison@arcinstitute.org&lt;/code&gt; → &lt;code&gt;catch_all&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;nothing&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Five of the six ended with a mailbox the server confirmed. In three of those five the highest-ranked candidate came back &lt;code&gt;invalid&lt;/code&gt; — not "unconfirmed", not "probably", but a mail server actively saying there is no such mailbox. The address that worked was the plain first name.&lt;/p&gt;

&lt;p&gt;Six people is not a statistical law about corporate mail, and I'm not going to dress it up as one. It is, however, a direct cost argument. If your highest-ranked candidate fails 60% of the time on the kind of person people actually pay to reach, then a tool with no mailbox check hands out three wrong addresses in five with a confident label on each, and a tool with a mailbox check spends 60% of its check budget on check 1.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the default led in all six cases
&lt;/h2&gt;

&lt;p&gt;The Actor's first choice of layout is not supposed to be a default. It's supposed to be evidence:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// The whole product rests on one observation: a company that publishes any personal&lt;/span&gt;
&lt;span class="c1"&gt;// address on its own website has told you the rule for all of them. jane.doe@acme.com&lt;/span&gt;
&lt;span class="c1"&gt;// on the team page means the sales director is firstname.lastname too.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the company site publishes &lt;code&gt;j.diaz@acme.com&lt;/code&gt; and &lt;code&gt;m.kent@acme.com&lt;/code&gt;, you don't guess — you've been told the layout, and you write the one address you were asked for in it. That path works, and I wrote about the evidence side of it separately. It just didn't fire once in this batch. All six rows came back with &lt;code&gt;patternEvidenceCount: 0&lt;/code&gt; and &lt;code&gt;emailPatternSource: "default"&lt;/code&gt;: gatesfoundation.org, microsoft.com, stripe.com, ycombinator.com, sparktoro.com and nextplayventures.com publish no personal address a crawler can read. Big companies route you to a contact form; two-person funds have a &lt;code&gt;hello@&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;With nothing to learn from, the ranked prior takes over, and the prior is this list:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;PATTERNS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;first.last&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;first&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;flast&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;firstlast&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;first_last&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;firstl&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;f.last&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;last.first&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;lastfirst&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;first-last&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;last&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;fl&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;];&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;first.last&lt;/code&gt; is roughly half of corporate mail, so it leads. But look at who rejected it: the CEO who joined in 1992, the founder of a two-person company, a founding partner of his own fund. People who were employee number small tend to hold the short mailbox, and the short mailbox is &lt;code&gt;first@&lt;/code&gt;. The population you want from LinkedIn — founders, partners, heads of things — is enriched in exactly the people for whom the most common layout is the wrong one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The budget is two checks, and that's the whole design
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;REOON_MAX_CHECKS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="err"&gt;…&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;pool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;candidatesFor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;first&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;last&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;company&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;company&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;pattern&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;first.last&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="err"&gt;…&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;order&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[...(&lt;/span&gt;&lt;span class="nx"&gt;onSite&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;onSite&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[]),&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;email&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nx"&gt;onSite&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;email&lt;/span&gt;&lt;span class="p"&gt;)];&lt;/span&gt;
&lt;span class="nx"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;verifyFirst&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;order&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;email&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;maxChecks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;REOON_MAX_CHECKS&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two, not ten, for two unglamorous reasons. Each check is a paid API call, and the Actor charges per &lt;strong&gt;found&lt;/strong&gt; e-mail — so checks that find nothing come out of my margin, not the buyer's bill. And hammering one domain with ten mailbox probes per person is how a verification provider's IPs stop being trusted by that domain's mail server.&lt;/p&gt;

&lt;p&gt;Which means the ranking is not a nicety. With a budget of two, "the right layout is third in my list" and "I have no address for this person" are the same outcome. The three saves above all happened on check 2. There is no check 3.&lt;/p&gt;

&lt;p&gt;That also explains a field people ask about. &lt;code&gt;alternatives&lt;/code&gt; in every row is not padding — it's the rest of the ranked list that never got measured, labelled as such, so a buyer whose first address bounces has somewhere to go:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"satya@microsoft.com"&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"emailPattern"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"first"&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"confidence"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"verified"&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"emailStatus"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"mailbox-verified"&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"emailVerification"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"provider"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"reoon"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"safe"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"score"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;98&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"checks"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"tried"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"satya.nadella@microsoft.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"invalid"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
             &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"satya@microsoft.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"safe"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"alternatives"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"satya.nadella@microsoft.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"pattern"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"first.last"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
                  &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"snadella@microsoft.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"pattern"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"flast"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Where no amount of check budget helps
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;stripe.com&lt;/code&gt; answered &lt;code&gt;catch_all&lt;/code&gt; to &lt;code&gt;patrick.collison@stripe.com&lt;/code&gt; in all six runs it appeared in, with a score of 75 against the 98 a confirmed mailbox gets. A catch-all server accepts every address you offer it, including ones you invent, so nothing can be confirmed there — not by me, not by any vendor, no matter what their UI says. The remaining check rolled to the person's other current company from the profile, &lt;code&gt;arcinstitute.org&lt;/code&gt;, which is also catch-all. Two checks spent, nothing learned:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"emailGuess"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"patrick.collison@stripe.com"&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"emailCatchAll"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"emailStatus"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"catch-all"&lt;/span&gt;&lt;span class="err"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="nl"&gt;"summary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Not delivered as an e-mail: the mail server of stripe.com accepts any address
            (catch-all), so no mailbox can be confirmed there; best unverified guess
            patrick.collison@stripe.com is in &lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;emailGuess&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;. This row was not charged."&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The guess still ships, in a differently-named field, free. If you're building this yourself, that split is the only honest shape I've found: one field for "a server confirmed this", a different field for "this is my best construction", and never the same field for both.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure that only shows up when you run the same thing twice
&lt;/h2&gt;

&lt;p&gt;One run out of nine broke the pattern. On 13 October, &lt;code&gt;microsoft.com&lt;/code&gt; answered &lt;code&gt;unknown&lt;/code&gt; to &lt;code&gt;satya.nadella@microsoft.com&lt;/code&gt; after a single check — no score, no verdict — where the eight other runs got a flat &lt;code&gt;invalid&lt;/code&gt; on that same address. The row came back free, no e-mail, status &lt;code&gt;mailbox-unknown&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That's the right outcome for billing and the wrong one for the buyer, and it exposes a limit I'd rather print than hide: because the run stopped at &lt;code&gt;unknown&lt;/code&gt; instead of &lt;code&gt;invalid&lt;/code&gt;, it never reached &lt;code&gt;satya@&lt;/code&gt;, and &lt;code&gt;emailGuess&lt;/code&gt; was filled with &lt;code&gt;satya.nadella@microsoft.com&lt;/code&gt; — the exact address eight other runs proved does not exist. Nothing in a single row is wrong. The row simply can't know what a different run knew. If you're consuming SMTP verdicts in a pipeline, treat &lt;code&gt;unknown&lt;/code&gt; as "ask again tomorrow", never as "no mailbox", and keep your own memory of addresses that came back &lt;code&gt;invalid&lt;/code&gt; across runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/lergassy~linkedin-email-finder/run-sync-get-dataset-items?token=&lt;/span&gt;&lt;span class="nv"&gt;$APIFY_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type: application/json'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"profileUrls":["https://www.linkedin.com/in/satyanadella",
                      "https://www.linkedin.com/in/patrickcollison",
                      "https://www.linkedin.com/in/williamhgates"]}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those three profiles took 14.6 seconds end to end in the 17 October run these numbers come from, and about $0.0006 of platform compute; the price is per e-mail found, and the two catch-all and unknown rows above cost nothing. It's &lt;a href="https://apify.com/lergassy/linkedin-email-finder?fpr=wmeplu" rel="noopener noreferrer"&gt;LinkedIn Email Finder on Apify&lt;/a&gt; if you want the rows rather than the reasoning.&lt;/p&gt;

&lt;p&gt;Where it doesn't fit: there's no login, so the roughly one profile in five that LinkedIn hides from anonymous readers needs an unblocking route and sometimes still doesn't open. If your target list is all large-enterprise domains, expect a pile of catch-all rows and plan on guesses. If you already have a name and a company domain and no LinkedIn URL, the profile read is wasted work — the name-plus-domain path is a different Actor. And if your list is twenty people, reading twenty company websites by hand will beat any of this.&lt;/p&gt;

&lt;p&gt;What I'd keep even if you never run it: a mailbox check turns e-mail finding from a text-formatting exercise into a search problem with a budget. Spend the budget on the layout the company told you about. When it hasn't told you anything, know that your prior is tuned for the median employee, and the people you're looking for aren't median.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>api</category>
      <category>node</category>
      <category>sales</category>
    </item>
    <item>
      <title>Allbirds' Home Page Carries No Ad Pixel. Its Tag Container Carries 23 Account IDs</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Sat, 17 Oct 2026 05:14:01 +0000</pubDate>
      <link>https://dev.to/matveyg/allbirds-home-page-carries-no-ad-pixel-its-tag-container-carries-23-account-ids-1pdb</link>
      <guid>https://dev.to/matveyg/allbirds-home-page-carries-no-ad-pixel-its-tag-container-carries-23-account-ids-1pdb</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them does something narrower than a general tech-stack lookup: you give it a list of domains, it gives you back one row per domain with the technologies the site runs, &lt;strong&gt;the capabilities it lacks&lt;/strong&gt;, whether the company is spending on traffic right now, and the e-mail and phone to reach it.&lt;/p&gt;

&lt;p&gt;The "lacks" column is the whole point. Nobody replies to "I see you use WordPress". People reply to "you're running Meta ads but a visitor can't book an appointment on your site". The problem is that a missing-capability list is a negation, and a negation is only as trustworthy as your detection. Every tool you fail to see becomes a gap you'll put in an e-mail and be wrong about.&lt;/p&gt;

&lt;p&gt;Here is the single biggest source of those wrong gaps, measured on 23 live domains on 17 October 2026.&lt;/p&gt;

&lt;h2&gt;
  
  
  The home page knows nothing
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;allbirds.com&lt;/code&gt;, home page, about 700 kB of HTML. I grepped it for every advertising identifier I could think of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GTM containers:        GTM-TH8KRSBJ
Google Ads (AW-…):     (none)
GA4 (G-…):             (none)
fbq('init', …):        (none)
connect.facebook.net:  0 occurrences
analytics.tiktok.com:  0 occurrences
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Allbirds is not a company that has stopped advertising. The page just doesn't contain its advertising. All it contains is the loader for container &lt;code&gt;GTM-TH8KRSBJ&lt;/code&gt;. So I fetched that container — it's a public JavaScript file, no auth, &lt;code&gt;https://www.googletagmanager.com/gtm.js?id=GTM-TH8KRSBJ&lt;/code&gt;, 656 kB — and grepped the same patterns:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Google Ads:  AW-604448474  AW-671743546  AW-736740096  AW-786649265
             AW-798041621  AW-822752074  AW-822752224
GA4:         G-KJL05B1DJZ  G-KB1068K0E6  G-K4JD0P61SJ  G-X7QTYLMX8J … (10 unique)
Meta pixels: 1758221267590067  1689016458013442  1569821056431325 … (8 unique)
TikTok:      2 occurrences of analytics.tiktok.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's the whole asymmetry in two greps. The tags are not in the page; they are in a file the page tells the browser to go and get. A scraper that reads HTML and stops is reading the table of contents.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does to the output
&lt;/h2&gt;

&lt;p&gt;So the scanner fetches the container for every domain that loads one and runs detection over HTML plus container body. To find out what that's worth I ran the same 23 domains twice, once with &lt;code&gt;expandTagManager: true&lt;/code&gt; and once with it off — 11 e-commerce brands, 8 local businesses (dental and roofing) and 4 vertical SaaS sites. 20 answered, 3 were blocked.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Over the 20 reachable sites&lt;/th&gt;
&lt;th&gt;HTML only&lt;/th&gt;
&lt;th&gt;HTML + container&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Sites flagged as paying for traffic&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Distinct ad platforms detected&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;31&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Technologies detected&lt;/td&gt;
&lt;td&gt;79&lt;/td&gt;
&lt;td&gt;124&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gaps reported&lt;/td&gt;
&lt;td&gt;126&lt;/td&gt;
&lt;td&gt;106&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sites reported as &lt;code&gt;no_ads_pixel&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;11 of the 20 sites load a GTM container, and on 8 of them the container flipped &lt;code&gt;runsPaidAds&lt;/code&gt; from &lt;code&gt;false&lt;/code&gt; to &lt;code&gt;true&lt;/code&gt;: &lt;code&gt;allbirds.com&lt;/code&gt;, &lt;code&gt;warbyparker.com&lt;/code&gt;, &lt;code&gt;ruggable.com&lt;/code&gt;, &lt;code&gt;chubbies.com&lt;/code&gt;, &lt;code&gt;bostondentalgroup.com&lt;/code&gt;, &lt;code&gt;mrroof.com&lt;/code&gt;, &lt;code&gt;housecallpro.com&lt;/code&gt;, &lt;code&gt;nexhealth.com&lt;/code&gt;. Every one of those 8 would otherwise have gone out with &lt;code&gt;no_ads_pixel&lt;/code&gt; in its gap list — a brand running five pixels, described to a prospect as a company with no tracking.&lt;/p&gt;

&lt;p&gt;The 20 gaps that disappeared weren't all ad pixels either:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;no_ads_pixel        8
no_analytics        7
no_reviews_widget   1
no_online_payments  1
no_email_capture    1
no_crm              1
booking_unverified  1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two of those are worth naming. &lt;code&gt;bostondentalgroup.com&lt;/code&gt; loads &lt;strong&gt;Sunbit&lt;/strong&gt; — patient financing, i.e. an online payment capability — from inside its container; HTML-only called it &lt;code&gt;no_online_payments&lt;/code&gt;. And &lt;code&gt;mrroof.com&lt;/code&gt; went from 7 gaps to 3, because the container held both its CRM tag and a booking vendor, so &lt;code&gt;booking_unverified&lt;/code&gt; resolved into an actual identified vendor. On that one domain, four of seven "opportunities" were fiction.&lt;/p&gt;

&lt;p&gt;Dropping 20 of 126 gaps sounds like a modest 16% correction until you remember which direction it runs. A gap list is a buy signal. The error mode of HTML-only scanning is never "I missed an opportunity" — it's always "I invented one", and you find out after you've sent the mail.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cost of the extra fetch
&lt;/h2&gt;

&lt;p&gt;One more HTTP request per domain, not a browser. That matters for anybody building the same thing: the honest-looking solution here is "render the page in Chrome and watch the network tab", and it works, and it costs an order of magnitude more per row. The container is a static file; you don't need to execute it to read which tags it carries.&lt;/p&gt;

&lt;p&gt;The guard rails in the implementation are boring on purpose — at most 2 containers per site, a 3 MB cap on each, a shared timeout, and a &lt;code&gt;Promise.all&lt;/code&gt; so the containers for one site come in parallel:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;expandTagManager&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;timeoutMs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;findContainerIds&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;MAX_CONTAINERS&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;containerIds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;adAccountIds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;bodies&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;fetchContainer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;timeoutMs&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;bodies&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;uniq&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;re&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[...&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;([...&lt;/span&gt;&lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;matchAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;re&lt;/span&gt;&lt;span class="p"&gt;)].&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]))].&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;adAccountIds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;metaPixelIds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;uniq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\b\d{15,16}\b&lt;/span&gt;&lt;span class="sr"&gt;/g&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="na"&gt;googleAdsIds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;uniq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\b&lt;/span&gt;&lt;span class="sr"&gt;AW-&lt;/span&gt;&lt;span class="se"&gt;\d{6,}\b&lt;/span&gt;&lt;span class="sr"&gt;/g&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="na"&gt;ga4Ids&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;uniq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\b&lt;/span&gt;&lt;span class="sr"&gt;G-&lt;/span&gt;&lt;span class="se"&gt;[&lt;/span&gt;&lt;span class="sr"&gt;A-Z0-9&lt;/span&gt;&lt;span class="se"&gt;]{8,12}\b&lt;/span&gt;&lt;span class="sr"&gt;/g&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="err"&gt;…&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Where this is still wrong, in writing
&lt;/h2&gt;

&lt;p&gt;Four limits I'd want to know about if I were reading someone else's row:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;adAccountIds&lt;/code&gt; is capped at 8 per type.&lt;/strong&gt; My own grep found 10 unique GA4 IDs in the Allbirds container; the row carries 8. One of the two dropped is &lt;code&gt;G-D6E2XB4D2P&lt;/code&gt;, sitting in a lookup table keyed to &lt;code&gt;allbirds.co.nz&lt;/code&gt; — the container holds the whole international estate, not one property. Treat the field as evidence that accounts exist, not as an inventory. Same regex, same caveat: a literal &lt;code&gt;G-XXXXXXXXXX&lt;/code&gt; placeholder in a container matches the GA4 pattern as happily as a real ID.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;adAccountIds&lt;/code&gt; reads only the container.&lt;/strong&gt; &lt;code&gt;roofingbybruce.com&lt;/code&gt; has a hardcoded &lt;code&gt;gtag/js?id=G-41VCL68TF2&lt;/code&gt; in its HTML plus two other GA4 IDs in its container. The technology "Google Analytics 4" is detected correctly; the ID from the page itself never appears in &lt;code&gt;ga4Ids&lt;/code&gt;. The categorical answer and the identifier list come from different places, and only one of them sees the HTML.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Other injectors aren't expanded.&lt;/strong&gt; Server-side GTM and Segment load tags somewhere this approach can't follow. A site whose tags arrive that way still reads as a gap.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's one page.&lt;/strong&gt; The scanner reads the home page. A booking widget living only on &lt;code&gt;/appointments&lt;/code&gt; reads as &lt;code&gt;booking_unverified&lt;/code&gt; if the page mentions scheduling and as a flat gap if it doesn't — which is why those are two different values rather than one.&lt;/p&gt;

&lt;p&gt;Three of the 23 domains never produced a row at all, and they're instructive about the other half of this job: &lt;code&gt;bombas.com&lt;/code&gt; answered 429 to both fetch engines, and &lt;code&gt;clio.com&lt;/code&gt; answered 403 to both. &lt;code&gt;baker-roofing.com&lt;/code&gt; and &lt;code&gt;getjobber.com&lt;/code&gt; also answered 403 to a plain &lt;code&gt;fetch&lt;/code&gt; — then returned 200 with 227 kB and 537 kB to a retry with a browser-shaped TLS handshake, same machine, same moment. Node's TLS fingerprint is recognisable; the header set isn't the thing being read. Blocked rows come back as &lt;code&gt;status: "unreachable"&lt;/code&gt; with the code, because a list where failures vanish silently is worse than a list with holes in it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/lergassy~website-tech-gap-scanner/run-sync-get-dataset-items?token=&lt;/span&gt;&lt;span class="nv"&gt;$APIFY_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type: application/json'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"domains":["allbirds.com","bostondentalgroup.com"],"onlyPaidTraffic":true}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three domains took 4.3 seconds end to end in the run these numbers come from. A trimmed row:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"domain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"bostondentalgroup.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"finalUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://www.bdgdentalservices.com/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"confidence"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"high"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"prospectScore"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"prospectTier"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"hot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"runsPaidAds"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"adPlatforms"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Meta Pixel"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"tagManagerContainers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"GTM-MFVLQ3J"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"adAccountIds"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"metaPixelIds"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"473376469722887"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"421033821700845"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
                    &lt;/span&gt;&lt;span class="nl"&gt;"ga4Ids"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"G-FKKWNQNGDB"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"technologies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Google Tag Manager"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Google Analytics 4"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Universal Analytics"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                   &lt;/span&gt;&lt;span class="s2"&gt;"Meta Pixel"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"WordPress"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Sunbit"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"gaps"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"booking_unverified"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"no_live_chat"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"no_crm"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"no_email_capture"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
           &lt;/span&gt;&lt;span class="s2"&gt;"no_reviews_widget"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"no_contact_form"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"bookingSignals"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"schedule appointment"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"schedule an appointment"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"city"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Las Vegas"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"stateCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"NV"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"locationSource"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"page text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"phones"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"7023998888"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"7024324329"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"7025222888"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Actor is &lt;a href="https://apify.com/lergassy/website-tech-gap-scanner?fpr=wmeplu" rel="noopener noreferrer"&gt;Website Tech Gap Scanner on Apify&lt;/a&gt;. Where it doesn't fit: if you want a complete inventory of what a company runs across its whole estate, this is a home-page snapshot and you want BuiltWith's historical database instead; if you want contacts rather than stack, a dedicated contact crawler reads more than one page and will do better; and if your list is 50 domains you can check by hand, checking by hand is more accurate than any of this.&lt;/p&gt;

&lt;p&gt;What I'd take away even if you never touch the Actor: if you're writing your own detector, the HTML is the loader and the container is the payload. Deciding what a website &lt;em&gt;doesn't&lt;/em&gt; have on the strength of the loader alone will hand you a pile of confident, specific, wrong openings.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>javascript</category>
      <category>data</category>
      <category>api</category>
    </item>
    <item>
      <title>8% of My Capterra Reviews Arrived as the Three Characters $62</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Fri, 16 Oct 2026 21:22:16 +0000</pubDate>
      <link>https://dev.to/matveyg/8-of-my-capterra-reviews-arrived-as-the-three-characters-62-8e2</link>
      <guid>https://dev.to/matveyg/8-of-my-capterra-reviews-arrived-as-the-three-characters-62-8e2</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The half of my week that is not visas goes into Apify Actors, and one of them reads B2B software reviews — Capterra, Software Advice, GetApp, TrustRadius — and returns all four with the same field names.&lt;/p&gt;

&lt;p&gt;Last night I was grepping my own output for junk. Not looking for anything in particular, just running a regex for field values that look like machinery rather than text. One hit came back often enough to stop me:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"reviewId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Capterra___7209002"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Slack: Essential for remote work productivity...and employee engagement"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"overallRating"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pros"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"$62"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"cons"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Honestly, not much. The only time I get frustrated with it is when I need to invite partners (from external companies) into our workspace for targeted deal collaboration…"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"comments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"$61"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;cons&lt;/code&gt; is a real review. &lt;code&gt;pros&lt;/code&gt; is the string &lt;code&gt;$62&lt;/code&gt;. Not an empty field, not a truncation, not a price — a three-character pointer where 1,309 characters of someone's opinion of Slack should be.&lt;/p&gt;

&lt;h2&gt;
  
  
  How often, and to whom
&lt;/h2&gt;

&lt;p&gt;I keep every run's dataset, so this is measurable rather than anecdotal. Across 16 runs and &lt;strong&gt;409 Capterra review rows&lt;/strong&gt;, 35 rows — &lt;strong&gt;8.6%&lt;/strong&gt; — carried a pointer in at least one text field: 34 in &lt;code&gt;pros&lt;/code&gt;, 12 in &lt;code&gt;comments&lt;/code&gt;, never in &lt;code&gt;cons&lt;/code&gt;. On Software Advice, 5 pointer fields across 150 rows. On TrustRadius, 0 across 1,330.&lt;/p&gt;

&lt;p&gt;The TrustRadius zero is the clue. TrustRadius goes through a completely different code path in my Actor — one that reassembles each review out of a slice of markup. Capterra and Software Advice share a path that reads structured objects out of the page. So this was not a site change. It was my parser, and it had been wrong since at least late September.&lt;/p&gt;

&lt;h2&gt;
  
  
  What &lt;code&gt;$62&lt;/code&gt; actually is
&lt;/h2&gt;

&lt;p&gt;Capterra runs on Next.js App Router, which means the page does not ship a single JSON blob. It ships a React Server Components stream, pushed into the document in pieces:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;script&amp;gt;&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;__next_f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;1:&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;$Sreact.fragment&lt;/span&gt;&lt;span class="se"&gt;\"\n&lt;/span&gt;&lt;span class="s2"&gt;3:I[339756,[&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;/assets-bx-capterra/…&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="nt"&gt;&amp;lt;/script&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Concatenate those pushes and you get a flat wire format of numbered rows. Mostly they are component references, but some are text:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;61:T425,Slack has been part of the corporate toolkits of the companies I've worked for since 2017…
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Row &lt;code&gt;61&lt;/code&gt;, type &lt;code&gt;T&lt;/code&gt;, length &lt;code&gt;0x425&lt;/code&gt; = 1,061 bytes, then the text. And in the review object that owns that text, React leaves this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"reviewId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"Capterra___7209002"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"overallRating"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"5.0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nl"&gt;"consText"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"Honestly, not much. The only time I get frustrated with it is when…"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nl"&gt;"generalComments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"$61"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"prosText"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"$62"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is React's reference syntax. A &lt;code&gt;$&lt;/code&gt;-prefixed string in an RSC payload is a pointer to another row, not a value. The serializer &lt;em&gt;outlines&lt;/em&gt; a long string — hoists it out of the object graph into its own row — and substitutes the pointer. The threshold in React's flight serializer is 1,024 characters.&lt;/p&gt;

&lt;p&gt;I had read none of this. My parser pulled balanced JSON objects out of the stream, found &lt;code&gt;"prosText": "$62"&lt;/code&gt;, and dutifully wrote &lt;code&gt;$62&lt;/code&gt; into the dataset as the review.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a line-based reader misses the rows
&lt;/h2&gt;

&lt;p&gt;The obvious fix is to build a lookup table of rows and resolve pointers against it. My first attempt did it by splitting the stream on newlines and matching &lt;code&gt;^([0-9a-f]+):T([0-9a-f]+),&lt;/code&gt;. It found row &lt;code&gt;61&lt;/code&gt; and reported rows &lt;code&gt;62&lt;/code&gt; and &lt;code&gt;63&lt;/code&gt; as missing — even though &lt;code&gt;62:T51d,&lt;/code&gt; is right there in the HTML.&lt;/p&gt;

&lt;p&gt;Here is the reason, lifted from the raw stream:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;…companies should seriously consider adopting it.62:T51d,Since 2020, I've worked remotely as…
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A text row declares its length, so it does not need a terminator — and it does not get one. The next row id begins immediately after the last byte of the previous row's text. On a line-based split, the only text rows you find are the ones that happen to follow a newline-terminated row. Every text row that follows another text row disappears, which is exactly the case when a page has several long reviews in a row.&lt;/p&gt;

&lt;p&gt;So the rows have to be walked in order, and the length has to be honoured in bytes, not characters — &lt;code&gt;T&lt;/code&gt; counts UTF-8 bytes, and review text is full of smart quotes and emoji:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;_RSC_ROW&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;rb&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;([0-9a-f]{1,8}):&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;_RSC_TEXT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;rb&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;T([0-9a-f]+),&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rsc_text_rows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;blob&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Map every length-prefixed text row in the stream to the text it carries.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;position&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;head&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_RSC_ROW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;head&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;newline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;break&lt;/span&gt;
            &lt;span class="n"&gt;position&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_RSC_TEXT&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;newline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;position&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setdefault&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ignore&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;position&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then resolving is a recursive walk over the parsed object, replacing any string that is exactly &lt;code&gt;$&amp;lt;row id&amp;gt;&lt;/code&gt; with that row's text. One detail worth stealing: React escapes a string that genuinely starts with &lt;code&gt;$&lt;/code&gt; by doubling it, so a reviewer who wrote &lt;code&gt;$62 a month is fair&lt;/code&gt; arrives as &lt;code&gt;$$62 a month is fair&lt;/code&gt;. That means a single-&lt;code&gt;$&lt;/code&gt; value is &lt;strong&gt;always&lt;/strong&gt; a pointer and never literal text — and that a parser which ignores all of this also silently doubles the dollar sign on any review starting with a price. I had zero of those in 2,947 rows, so that half is read from React's source rather than measured, but it costs one line to handle:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;$$&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The bias is the real damage
&lt;/h2&gt;

&lt;p&gt;8.6% of rows losing a field sounds survivable. It isn't, because the loss is not random.&lt;/p&gt;

&lt;p&gt;Here are the longest text values on one Capterra page for Slack — the 25 reviews on page 1 — split by whether React kept them inline or outlined them:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Characters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Longest &lt;strong&gt;inline&lt;/strong&gt; value&lt;/td&gt;
&lt;td&gt;927&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Shortest &lt;strong&gt;outlined&lt;/strong&gt; value&lt;/td&gt;
&lt;td&gt;1,061&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Other outlined values&lt;/td&gt;
&lt;td&gt;1,309 and 1,966&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There is no overlap, and there can't be: the cut is React's 1,024-character rule. So the fields that went missing were precisely the three longest pieces of writing on the page. The 150-character "great tool, love it" reviews all came through perfectly.&lt;/p&gt;

&lt;p&gt;If you are scraping reviews to feed a summarisation prompt, sentiment model or RAG index, that is the worst possible sample to lose. Short reviews are noise; the 2,000-character one from the IT executive explaining what Slack replaced and why is the entire reason to buy review data. My dataset was quietly dropping the signal and keeping the filler, and every run finished green with the expected row count.&lt;/p&gt;

&lt;p&gt;That is the part I keep coming back to. The run succeeded. The row count was right. Nothing errored, nothing timed out, no site blocked anything. The only symptom was a two-byte string in one field of one row in twelve, and the only reason I found it is that I ran a regex looking for values that don't look like prose.&lt;/p&gt;

&lt;h2&gt;
  
  
  After
&lt;/h2&gt;

&lt;p&gt;Same page, same 25 reviews, after the fix — and verified on the platform, not just on my laptop:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;reviews: 25   pointer fields: 0
longest pros lengths: 1966, 1309, 926, 843, 805
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Rows 1,966 and 1,309 are the two that used to be &lt;code&gt;$63&lt;/code&gt; and &lt;code&gt;$62&lt;/code&gt;. The 926 was always fine. Both Actors that share this code path — &lt;a href="https://apify.com/lergassy/capterra-reviews-scraper?fpr=wmeplu" rel="noopener noreferrer"&gt;Capterra &amp;amp; SaaS Reviews Scraper&lt;/a&gt; and its sibling — went out the same evening.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to take from this if you scrape Next.js
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Any &lt;code&gt;$&lt;/code&gt;-prefixed string value in an RSC payload is a pointer.&lt;/strong&gt; If your scraper reads App Router pages and never resolves them, you are losing your longest strings right now and your row counts look fine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Length-prefixed rows are not newline-terminated.&lt;/strong&gt; Walk the stream; don't split it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Add a junk canary to your validator.&lt;/strong&gt; Mine is one regex over every string field: &lt;code&gt;^\$[0-9a-f]{1,8}$&lt;/code&gt;. Values that look like machinery — pointers, &lt;code&gt;$undefined&lt;/code&gt;, bare &lt;code&gt;[object Object]&lt;/code&gt;, &lt;code&gt;NaN&lt;/code&gt; — are cheap to detect and are the ones that pass every other check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Headless browsers don't have this problem.&lt;/strong&gt; Read the rendered DOM and React has already resolved its own pointers for you. You pay a browser per page for it, which is why I parse the stream instead — but if you're scraping a handful of pages a day, that trade may not be yours.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check field-length distributions, not just fill rates.&lt;/strong&gt; A fill-rate check would have given this a clean bill of health: &lt;code&gt;pros&lt;/code&gt; was populated in 100% of rows. It was populated with &lt;code&gt;$62&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The reviews were there the whole time, 1,061 bytes further down the same response. I just never followed the pointer.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>react</category>
      <category>nextjs</category>
    </item>
    <item>
      <title>PrizePicks Gave Me Nine Pass-Yard Lines for One Quarterback. Eight Are Over-Only</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Fri, 16 Oct 2026 13:16:00 +0000</pubDate>
      <link>https://dev.to/matveyg/prizepicks-gave-me-nine-pass-yard-lines-for-one-quarterback-eight-are-over-only-3fed</link>
      <guid>https://dev.to/matveyg/prizepicks-gave-me-nine-pass-yard-lines-for-one-quarterback-eight-are-over-only-3fed</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them reads the two big pick'em apps — PrizePicks and Underdog Fantasy — and returns both boards as one table.&lt;/p&gt;

&lt;p&gt;Pick'em is not sportsbook betting. There is no moneyline and no spread; the app posts a number for a player and a stat, and you say over or under. The number &lt;em&gt;is&lt;/em&gt; the product. So the obvious thing to build is a comparison: same player, same stat, two apps, who asks for less.&lt;/p&gt;

&lt;p&gt;That join is where everyone — me included, on the first try — gets it wrong. Not because the data is dirty, but because the two boards are not the same shape, and nothing in either payload announces that.&lt;/p&gt;

&lt;p&gt;Here is a run from this afternoon, 16 October 2026: props mode, both apps, NFL, 600 rows, one run.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shape problem in one table
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;PrizePicks&lt;/th&gt;
&lt;th&gt;Underdog&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Rows returned&lt;/td&gt;
&lt;td&gt;473&lt;/td&gt;
&lt;td&gt;127&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Distinct player + stat pairs&lt;/td&gt;
&lt;td&gt;125&lt;/td&gt;
&lt;td&gt;127&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rows per pair&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.78&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.00&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Prices published&lt;/td&gt;
&lt;td&gt;none&lt;/td&gt;
&lt;td&gt;all 127 rows&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Underdog shipped exactly one row for every player-and-stat pair — 127 of 127, no duplicates, every one two-sided and priced. PrizePicks shipped 473 rows covering 125 pairs.&lt;/p&gt;

&lt;p&gt;Those extra 348 rows are not a paging bug. PrizePicks posts a &lt;em&gt;ladder&lt;/em&gt;: the board line plus boosted alternates at other numbers. &lt;code&gt;oddsType&lt;/code&gt; names them — &lt;code&gt;standard&lt;/code&gt; is the board line, a &lt;strong&gt;demon&lt;/strong&gt; is a harder number that pays more, a &lt;strong&gt;goblin&lt;/strong&gt; an easier one that pays less.&lt;/p&gt;

&lt;p&gt;Across the whole pull:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;code&gt;oddsType&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;demon&lt;/td&gt;
&lt;td&gt;249&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;goblin&lt;/td&gt;
&lt;td&gt;128&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;standard&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;96&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;balanced (all Underdog)&lt;/td&gt;
&lt;td&gt;127&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Only 96 of 473 PrizePicks rows are the number on the board. If you average &lt;code&gt;line&lt;/code&gt; grouped by player and stat, you are averaging a marketing ladder.&lt;/p&gt;

&lt;h2&gt;
  
  
  Nine rungs, one board number
&lt;/h2&gt;

&lt;p&gt;The deepest ladder in the run, C.J. Stroud's passing yards:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;code&gt;line&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;oddsType&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;allowedPicks&lt;/code&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;189.5&lt;/td&gt;
&lt;td&gt;goblin&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;199.5&lt;/td&gt;
&lt;td&gt;goblin&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;224.5&lt;/td&gt;
&lt;td&gt;goblin&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;246.5&lt;/td&gt;
&lt;td&gt;goblin&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;256.5&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;standard&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;under_or_over&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;264.5&lt;/td&gt;
&lt;td&gt;demon&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;266.5&lt;/td&gt;
&lt;td&gt;demon&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;269.5&lt;/td&gt;
&lt;td&gt;demon&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;374.5&lt;/td&gt;
&lt;td&gt;demon&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Nine rows, one player, one stat, a 185-yard spread from bottom to top. The board number is 256.5. Everything else is a priced-up offer sitting on the same player.&lt;/p&gt;

&lt;p&gt;And look at the third column, because that is the one that actually bites. Eight of those nine rows are &lt;strong&gt;over-only&lt;/strong&gt;. You cannot take the under on a goblin at 189.5. It isn't a line in the sense you mean when you say "line" — it is a one-sided offer that happens to be rendered as a number. Pull these rows into a model as symmetric over/under lines and you have invented bets the app does not sell.&lt;/p&gt;

&lt;p&gt;Ladder depth across the 125 pairs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Rows for the pair&lt;/th&gt;
&lt;th&gt;1&lt;/th&gt;
&lt;th&gt;2&lt;/th&gt;
&lt;th&gt;3&lt;/th&gt;
&lt;th&gt;4&lt;/th&gt;
&lt;th&gt;5&lt;/th&gt;
&lt;th&gt;6&lt;/th&gt;
&lt;th&gt;7&lt;/th&gt;
&lt;th&gt;8&lt;/th&gt;
&lt;th&gt;9&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Pairs&lt;/td&gt;
&lt;td&gt;38&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;17&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Nearly 70% of pairs carry at least two rows, and in the compare mode only 2 of 46 standard lines had no alternates behind them at all. The median standard line was hiding six or seven.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;oddsType&lt;/code&gt; and &lt;code&gt;allowedPicks&lt;/code&gt; disagree, so read both
&lt;/h2&gt;

&lt;p&gt;The tempting shortcut is &lt;code&gt;oddsType == 'standard'&lt;/code&gt;. I used it, then cross-tabbed the two fields on the same 473 rows:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;code&gt;oddsType&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;allowedPicks&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;demon&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;td&gt;240&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;goblin&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;td&gt;117&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;standard&lt;/td&gt;
&lt;td&gt;under_or_over&lt;/td&gt;
&lt;td&gt;75&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;standard&lt;/td&gt;
&lt;td&gt;&lt;em&gt;null&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;goblin&lt;/td&gt;
&lt;td&gt;under_or_over&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;demon&lt;/td&gt;
&lt;td&gt;under_or_over&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;standard&lt;/td&gt;
&lt;td&gt;over&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three surprises in that table. Twenty boosted alternates &lt;strong&gt;are&lt;/strong&gt; two-sided. Five standard lines are &lt;strong&gt;not&lt;/strong&gt; — they are over-only despite being the board number. And 16 rows don't say at all: &lt;code&gt;allowedPicks&lt;/code&gt; comes back null on every &lt;code&gt;Anytime TDs (Combo)&lt;/code&gt; row and, less explicably, on seven plain &lt;code&gt;Rec Targets&lt;/code&gt; rows.&lt;/p&gt;

&lt;p&gt;So the payout type and the permitted sides are independent facts, and one of them is sometimes missing. Treat null as unknown, never as two-sided.&lt;/p&gt;

&lt;p&gt;The other thing you lose with a &lt;code&gt;standard&lt;/code&gt; filter: &lt;strong&gt;29 of the 125 pairs have no standard row at all.&lt;/strong&gt; Chris Rodriguez Jr.'s longest rush, Nico Collins' receptions — demons and goblins only. Filter to standard and 23% of the board silently disappears, which is worse than keeping the ladder, because a short list looks like a correct list.&lt;/p&gt;

&lt;h2&gt;
  
  
  Seven of thirty stat names match
&lt;/h2&gt;

&lt;p&gt;Now the join. PrizePicks used 30 distinct stat names in this run, Underdog 31. Names that match &lt;strong&gt;exactly&lt;/strong&gt;: seven.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;FG Made&lt;/code&gt;, &lt;code&gt;Kicking Points&lt;/code&gt;, &lt;code&gt;Longest Rush&lt;/code&gt;, &lt;code&gt;Pass Attempts&lt;/code&gt;, &lt;code&gt;Pass TDs&lt;/code&gt;, &lt;code&gt;Pass Yards&lt;/code&gt;, &lt;code&gt;Rush Yards&lt;/code&gt;. That's it. Everything else needs translating:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;PrizePicks&lt;/th&gt;
&lt;th&gt;Underdog&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Rec Yards&lt;/td&gt;
&lt;td&gt;Receiving Yards&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recs&lt;/td&gt;
&lt;td&gt;Receptions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rec Targets&lt;/td&gt;
&lt;td&gt;Targets&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rush Atts&lt;/td&gt;
&lt;td&gt;Rush Attempts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Longest Rec&lt;/td&gt;
&lt;td&gt;Longest Reception&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rush+Rec Yds&lt;/td&gt;
&lt;td&gt;Rush + Rec Yards&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Thirty of the 46 comparison rows in my compare run pair two differently-spelled names. A join on raw strings would have returned 16 rows and looked like it worked.&lt;/p&gt;

&lt;p&gt;Fuzzy matching is worse than useless here, and this run has the proof sitting right in it. PrizePicks publishes eight &lt;code&gt;(Combo)&lt;/code&gt; markets — multi-player parlay-style props — and one of them is literally named &lt;code&gt;Receiving Yards (Combo)&lt;/code&gt;. Any substring or token-overlap matcher pairs that against Underdog's single-player &lt;code&gt;Receiving Yards&lt;/code&gt;. Meanwhile Underdog's plain NFL board carries eight partial-game markets inline — &lt;code&gt;1H Rush Yards&lt;/code&gt;, &lt;code&gt;1Q Receptions&lt;/code&gt;, &lt;code&gt;1H Rush + Rec TDs&lt;/code&gt; — which share every meaningful token with the full-game stat and are a completely different bet. &lt;code&gt;Rush+Rec Yds&lt;/code&gt; versus &lt;code&gt;Receiving Yards&lt;/code&gt; scores high on most similarity metrics too.&lt;/p&gt;

&lt;p&gt;The only thing that works is an explicit alias table and an explicit exclusion list. Boring, finite, and it does not quietly invent pairs on a slate you didn't look at.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the comparison actually found
&lt;/h2&gt;

&lt;p&gt;Compare mode, same slate, 46 pairs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Line difference&lt;/th&gt;
&lt;th&gt;Pairs&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;17&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.5 – 2&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3 – 5&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6 – 7&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The marketing claim about these apps is that they rarely agree. On this slate they agreed exactly on 37% of shared props. Where they differed, PrizePicks had the easier over 16 times and Underdog 13.&lt;/p&gt;

&lt;p&gt;The widest gaps, and the thing to notice about them:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Player&lt;/th&gt;
&lt;th&gt;Stat&lt;/th&gt;
&lt;th&gt;PrizePicks&lt;/th&gt;
&lt;th&gt;Underdog&lt;/th&gt;
&lt;th&gt;Gap&lt;/th&gt;
&lt;th&gt;Easier over&lt;/th&gt;
&lt;th&gt;PP alternates&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;C.J. Stroud&lt;/td&gt;
&lt;td&gt;Pass Yards&lt;/td&gt;
&lt;td&gt;256.5&lt;/td&gt;
&lt;td&gt;249.5&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;underdog&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dalton Schultz&lt;/td&gt;
&lt;td&gt;Rec Yards&lt;/td&gt;
&lt;td&gt;31.5&lt;/td&gt;
&lt;td&gt;24.5&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;underdog&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bijan Robinson&lt;/td&gt;
&lt;td&gt;Rush+Rec Yds&lt;/td&gt;
&lt;td&gt;123.5&lt;/td&gt;
&lt;td&gt;129.5&lt;/td&gt;
&lt;td&gt;−6&lt;/td&gt;
&lt;td&gt;prizepicks&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bhayshul Tuten&lt;/td&gt;
&lt;td&gt;Rec Yards&lt;/td&gt;
&lt;td&gt;8.5&lt;/td&gt;
&lt;td&gt;14.5&lt;/td&gt;
&lt;td&gt;−6&lt;/td&gt;
&lt;td&gt;prizepicks&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Seven yards on Stroud's 250 is noise. Seven yards on Schultz's 24.5 is a 29% disagreement about a tight end. Absolute line difference is the wrong unit for comparing across stats, and the field is deliberately raw so you can divide by whatever you like.&lt;/p&gt;

&lt;p&gt;Also: &lt;code&gt;easierOver&lt;/code&gt; is null on all 17 ties rather than naming a winner. Those are ties, not wins.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it
&lt;/h2&gt;

&lt;p&gt;One player, both boards, everything including the ladder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/lergassy~dfs-props-scraper/run-sync-get-dataset-items?token=&lt;/span&gt;&lt;span class="nv"&gt;$APIFY_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"mode":"props","players":["Stroud"],"maxRows":50}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Board lines only, with the ladder depth kept as a number:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;TOKEN&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;lergassy/dfs-props-scraper&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mode&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;compare&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;leagues&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;NFL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;maxRows&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;defaultDatasetId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]).&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;rel&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;lineDifference&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;underdogLine&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;player&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;market&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;prizePicksLine&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
          &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;underdogLine&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;rel&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;prizePicksAlternateLines&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both endpoints are public JSON — no account, no cookies, no proxy, no browser. PrizePicks answers 403 to a filtered request on its main host and 429 to bursts, so requests go to the partner host and get spaced out; a rate limit is retried slowly rather than hammered. My three runs today — 11 league rows, 600 props, 46 comparisons — cost about $0.15 in total, with no per-run fee. The Actor is &lt;a href="https://apify.com/lergassy/dfs-props-scraper?fpr=wmeplu" rel="noopener noreferrer"&gt;DFS Player Props API&lt;/a&gt; if you want to point it at tonight's slate.&lt;/p&gt;

&lt;h2&gt;
  
  
  When this is the wrong tool
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;You want sportsbook odds.&lt;/strong&gt; Pick'em apps don't publish a moneyline or a vig. For prices across DraftKings, Pinnacle and the like you need an odds feed and a de-vig step — a different tool (&lt;a href="https://apify.com/lergassy/sports-odds-api?fpr=wmeplu" rel="noopener noreferrer"&gt;this one&lt;/a&gt;, in my case).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You want implied probabilities on both apps.&lt;/strong&gt; PrizePicks publishes a payout &lt;em&gt;type&lt;/em&gt;, not a price. There is no honest implied probability to compute from &lt;code&gt;goblin&lt;/code&gt;. Quote &lt;code&gt;oddsType&lt;/code&gt; for PrizePicks rows and leave the percentage column empty.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You want the whole board, historically.&lt;/strong&gt; Boards move all day and these endpoints serve now. Schedule the run if you want a time series; nobody will hand you yesterday.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You want picks.&lt;/strong&gt; This returns a table. It has no opinion.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One run, two boards, one asymmetry worth internalising: Underdog hands you a row per bet, PrizePicks hands you a row per &lt;em&gt;offer&lt;/em&gt;. Count rows and you will think PrizePicks has four times the board. It has roughly the same board, sold four ways.&lt;/p&gt;

</description>
      <category>api</category>
      <category>data</category>
      <category>python</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Four Law Firms Published 79 Email Addresses. Four Tech Companies Published 11</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Fri, 16 Oct 2026 05:11:22 +0000</pubDate>
      <link>https://dev.to/matveyg/four-law-firms-published-79-email-addresses-four-tech-companies-published-11-38i6</link>
      <guid>https://dev.to/matveyg/four-law-firms-published-79-email-addresses-four-tech-companies-published-11-38i6</guid>
      <description>&lt;p&gt;I run a visa agency in Bali, and the other half of my week goes into Apify Actors. One of them does the thing every sales tool claims to do: you give it &lt;code&gt;Jane Doe, acme.com&lt;/code&gt;, it gives you back Jane's work address.&lt;/p&gt;

&lt;p&gt;Everybody solves this the same way — apply &lt;code&gt;first.last&lt;/code&gt;, because roughly half of corporate mail is &lt;code&gt;first.last&lt;/code&gt;, and ship it. That is a coin flip dressed as a lookup. The approach I wanted instead rests on one observation: &lt;strong&gt;a company that publishes any personal address on its own website has told you the rule for all of them.&lt;/strong&gt; If the team page says &lt;code&gt;michael.hart@acme.com&lt;/code&gt;, you no longer need to guess what the sales director's address looks like.&lt;/p&gt;

&lt;p&gt;So the interesting question stopped being "which format should I guess" and became "how often does a company actually publish something I can read". I ran eight companies through it to find out, and the answer splits so hard by industry that it changes who this kind of tool is for.&lt;/p&gt;

&lt;h2&gt;
  
  
  The run
&lt;/h2&gt;

&lt;p&gt;Eight people at eight companies, company rows on, four pages read per site:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"people"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Patrick Collison, stripe.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Jason Fried, basecamp.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Maria Bennett, cliffordchance.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Daniel Weber, freshfields.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Daniel Weber, linklaters.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Daniel Weber, bakermckenzie.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Daniel Weber, n8n.io"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Daniel Weber, hubspot.com"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"candidatesPerPerson"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"checkMx"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"includeCompanyRows"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"maxPagesPerCompany"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two of those names are real people; "Maria Bennett" and "Daniel Weber" are probes I invented, because the person is not what is being measured here. The layout belongs to the company, and the same probe name at eight companies isolates it cleanly. 32 pages, plain HTTP, no browser, no proxy: 7.1 seconds and $0.29 for the whole run on 16 October 2026.&lt;/p&gt;

&lt;p&gt;Here is what came back, per company:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Domain&lt;/th&gt;
&lt;th&gt;Addresses on site&lt;/th&gt;
&lt;th&gt;Usable as evidence&lt;/th&gt;
&lt;th&gt;Layout&lt;/th&gt;
&lt;th&gt;Confidence&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;cliffordchance.com&lt;/td&gt;
&lt;td&gt;48&lt;/td&gt;
&lt;td&gt;22&lt;/td&gt;
&lt;td&gt;&lt;code&gt;first.last&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;high&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;freshfields.com&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;13&lt;/td&gt;
&lt;td&gt;&lt;code&gt;first.last&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;high&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;linklaters.com&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;&lt;code&gt;first.last&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;high&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;bakermckenzie.com&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;n8n.io&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;stripe.com&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;first.last&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;medium&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;basecamp.com&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;first&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;confirmed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;hubspot.com&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;low&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;79 of the 90 published addresses came from the four law firms. 42 of the 44 usable pieces of evidence did too.&lt;/p&gt;

&lt;p&gt;That is not a coincidence and it is not about law. Professional services firms publish staff directories because clients need to reach a named partner — and the same is true of agencies, clinics, brokerages, accountants and consultancies. Companies that sell software route everything through a form and a support queue on purpose. Which means: if your lead list is made of firms, this method mostly works. If it is made of SaaS companies, you are back to guessing, and the honest move is to say so in the output rather than to hide it.&lt;/p&gt;

&lt;h2&gt;
  
  
  90 addresses found, 44 that say anything
&lt;/h2&gt;

&lt;p&gt;The gap between those two columns is where all the work is. An address only counts as evidence if you can tell which half is the first name — and that is harder than it sounds, because &lt;code&gt;jane.diaz@&lt;/code&gt; and &lt;code&gt;complaints-in@&lt;/code&gt; have the &lt;strong&gt;identical shape&lt;/strong&gt;. One token, a separator, one token. A tool that reads shape alone will conclude that Stripe writes &lt;code&gt;firstname-lastname&lt;/code&gt; on the strength of a complaints inbox, which is exactly what an early version of mine did.&lt;/p&gt;

&lt;p&gt;So every candidate local part is tested against a list of common given names instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;jane.diaz                    {"shapes":["first.last"],"strength":1}
esther.chancatlin            {"shapes":["first.last"],"strength":1}
victoria.brewertonowen       {"shapes":["first.last"],"strength":1}
jason                        {"shapes":["first"],"strength":1}
complaints-in                {"shapes":[],"strength":0}
nodes                        {"shapes":[],"strength":0}
nlinklaters                  {"shapes":[],"strength":0}
freshfieldsalumninetwork     {"shapes":[],"strength":0}
20iva.celic                  {"shapes":[],"strength":0}
esther.chan2                 {"shapes":[],"strength":0}
anne-marie.nistoroiu         {"shapes":[],"strength":0}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are all real locals from this run. A few things worth pointing at:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;esther.chancatlin&lt;/code&gt; and &lt;code&gt;victoria.brewertonowen&lt;/code&gt; pass because the &lt;em&gt;first&lt;/em&gt; half is a recognised name — the surname is never checked against anything, which is what lets compound surnames through intact.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;20iva.celic&lt;/code&gt; is a scraping artifact: a page number glued to an address by the surrounding text. Clifford Chance's site produced both it and the clean &lt;code&gt;iva.celic&lt;/code&gt;. The dirty one is refused by a leading-character rule, which is cheap and right. The clean one is &lt;em&gt;also&lt;/em&gt; refused, because neither "iva" nor "celic" is in the given-name list — a false negative, and one I am fine with. The site offered 22 other usable addresses. Being wrong about the format is expensive; skipping one address out of 48 costs nothing.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;anne-marie.nistoroiu&lt;/code&gt; is the one that annoys me. It contains both a dot and a hyphen, the dot wins the split, and &lt;code&gt;anne-marie&lt;/code&gt; is not in the name list as a hyphenated token. It should be read as &lt;code&gt;first.last&lt;/code&gt; and is not.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;nlinklaters@linklaters.com&lt;/code&gt; is the kind of thing that makes you add rules you did not plan for: a token that starts with a letter, contains no separator, and is not a name. Read it as initial-plus-surname and you have "N. Linklaters" working at Linklaters.&lt;/p&gt;

&lt;p&gt;The output ships &lt;code&gt;patternEvidence&lt;/code&gt; and &lt;code&gt;patternEvidenceCount&lt;/code&gt; for exactly this reason. 22 agreeing addresses and 1 are both &lt;code&gt;patternSource: "site"&lt;/code&gt;, and they are not the same claim.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stripe's one piece of evidence is a placeholder
&lt;/h2&gt;

&lt;p&gt;Stripe published four addresses: &lt;code&gt;dpo@&lt;/code&gt;, &lt;code&gt;privacy@&lt;/code&gt;, &lt;code&gt;complaints-in@&lt;/code&gt; — all discarded as department mail — and &lt;code&gt;jane.diaz@stripe.com&lt;/code&gt;. So the Actor read Stripe's layout off a single address and returned:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"fullName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Patrick Collison"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"patrick.collison@stripe.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"emailPattern"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"first.last"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"emailPatternSource"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"site"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"confidence"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"medium"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"patternEvidence"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"jane.diaz@stripe.com"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"patternEvidenceCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"emailStatus"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"deliverable"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"emailProvider"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"google"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"emailMxHost"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"aspmx.l.google.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"mailboxVerified"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"alternatives"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"patrick@stripe.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"pattern"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"first"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"pcollison@stripe.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"pattern"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"flast"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Jane Diaz does not work at Stripe. She is a string in the home page's checkout graphic:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight properties"&gt;&lt;code&gt;&lt;span class="err"&gt;payment-graphic.checkout.email-address-us":"jane.diaz@stripe.com"&lt;/span&gt;
&lt;span class="err"&gt;payment-graphic.checkout.email-address-de":"damian.michelfelder@example.com"&lt;/span&gt;
&lt;span class="err"&gt;payment-graphic.checkout.email-address-jp":"taro.yamada@example.com"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The German and Japanese variants of the same mockup use &lt;code&gt;@example.com&lt;/code&gt;, which any sane filter drops. The US one uses the company's live domain. That slip is the entire reason Stripe has a &lt;code&gt;site&lt;/code&gt; layout rather than a default guess.&lt;/p&gt;

&lt;p&gt;The answer is probably still correct — &lt;code&gt;first.last&lt;/code&gt; is a common layout and Stripe is a large US company. But it is right by luck, and a tool that reported it as "confirmed from the company's website" would be lying with a straight face. &lt;code&gt;medium&lt;/code&gt; plus one visible piece of evidence lets you decide. That is the difference between a confidence column that means something and one that is decoration.&lt;/p&gt;

&lt;p&gt;While I was in there: Freshfields came back as &lt;code&gt;"companyName": "Leading global law firm"&lt;/code&gt;. The name falls back through JSON-LD &lt;code&gt;Organization&lt;/code&gt; → &lt;code&gt;og:site_name&lt;/code&gt; → the first segment of &lt;code&gt;&amp;lt;title&amp;gt;&lt;/code&gt;, and Freshfields has neither of the first two, so an SEO title became the company name. Two scraped fields, two reminders that "it parsed" and "it's true" are different states.&lt;/p&gt;

&lt;h2&gt;
  
  
  What zero evidence looks like
&lt;/h2&gt;

&lt;p&gt;Baker McKenzie and HubSpot published nothing on four pages. n8n published six addresses — &lt;code&gt;marketing@&lt;/code&gt;, &lt;code&gt;license@&lt;/code&gt;, &lt;code&gt;nodes@&lt;/code&gt;, &lt;code&gt;community@&lt;/code&gt;, &lt;code&gt;partners@&lt;/code&gt;, &lt;code&gt;legal@&lt;/code&gt; — and not one of them says anything about how a person's address is written. "How many addresses did you find" is the wrong metric; "how many contained a human name" is the one that matters.&lt;/p&gt;

&lt;p&gt;All three got &lt;code&gt;first.last&lt;/code&gt;, &lt;code&gt;patternSource: "default"&lt;/code&gt;, &lt;code&gt;confidence: "low"&lt;/code&gt;, and empty &lt;code&gt;patternEvidence&lt;/code&gt;. Same shaped answer, honestly labelled as a guess. Sort by confidence and those rows go to the bottom of the sequence, or nowhere.&lt;/p&gt;

&lt;p&gt;One more field earns its place here. The four law firms sit behind mail gateways — Mimecast twice, Proofpoint twice — while Stripe, HubSpot and n8n are on Google. &lt;code&gt;emailProvider&lt;/code&gt; and &lt;code&gt;emailMxHost&lt;/code&gt; come from an MX lookup, which also means &lt;code&gt;mailboxVerified&lt;/code&gt; is permanently &lt;code&gt;false&lt;/code&gt;: checking whether a specific mailbox exists requires an SMTP conversation on port 25, and outbound 25 is blocked from Apify's cloud for every Actor there, whatever a listing claims. Domain-level truth and published evidence is what is actually obtainable. If you need address hygiene on a list you already have — role flags, disposable domains, deliverability — that is a different job, and &lt;a href="https://apify.com/lergassy/email-phone-verifier" rel="noopener noreferrer"&gt;Email Verifier &amp;amp; Phone Number Validator&lt;/a&gt; is the one I use for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;TOKEN&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;lergassy/email-finder-api&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;people&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Daniel Weber, linklaters.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;includeCompanyRows&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;defaultDatasetId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]).&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;person&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;confidence&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;confirmed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;high&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;email&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;patternEvidenceCount&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;peopleList&lt;/code&gt; takes pre-split fields from a CRM export, and &lt;code&gt;inputDatasetId&lt;/code&gt; reads the people straight out of another Actor's dataset. Error rows are free, which matters when a third of a real list has an unresolvable company.&lt;/p&gt;

&lt;h2&gt;
  
  
  When not to use this
&lt;/h2&gt;

&lt;p&gt;If your targets are consumer-facing tech companies, expect mostly &lt;code&gt;low&lt;/code&gt; and treat the output as a guess with a mail-server check attached. If you need a confirmed mailbox before sending, no Apify Actor can give you that — send to the &lt;code&gt;confirmed&lt;/code&gt; and &lt;code&gt;high&lt;/code&gt; rows and let the bounce be the verification, with &lt;code&gt;alternatives&lt;/code&gt; as the retry. And if what you actually want is every public address on a set of websites rather than one person's address, this is the wrong shape of tool entirely.&lt;/p&gt;

&lt;p&gt;The Actor is &lt;a href="https://apify.com/lergassy/email-finder-api?fpr=wmeplu" rel="noopener noreferrer"&gt;Email Finder API&lt;/a&gt;. If you find a company whose layout it reads wrongly, the company and the expected address is a genuinely useful bug report — the given-name list and the rules above grew almost entirely out of those. &lt;code&gt;anne-marie.nistoroiu&lt;/code&gt; is next.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>api</category>
      <category>node</category>
      <category>data</category>
    </item>
    <item>
      <title>Every Threads Profile Page Embeds Four Threads: Two Hours of CNN, Seventeen Days of @zuck</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Thu, 15 Oct 2026 21:16:14 +0000</pubDate>
      <link>https://dev.to/matveyg/every-threads-profile-page-embeds-four-threads-two-hours-of-cnn-seventeen-days-of-zuck-3mb3</link>
      <guid>https://dev.to/matveyg/every-threads-profile-page-embeds-four-threads-two-hours-of-cnn-seventeen-days-of-zuck-3mb3</guid>
      <description>&lt;p&gt;Half my week is a visa agency in Bali. The other half is Apify Actors, and the newest one is not a scraper but a &lt;strong&gt;monitor&lt;/strong&gt;: you name a few Threads accounts or brand keywords, give the watch a name, put it on a schedule, and every run hands you only the posts it has never handed you before. Same name, same memory, one row per new post.&lt;/p&gt;

&lt;p&gt;That shape sounds like a scraper with a &lt;code&gt;Set&lt;/code&gt; bolted on, and for about a week I believed it was. The thing that breaks the illusion is a question a scraper never has to ask: &lt;strong&gt;how far back does the page I am diffing against actually go?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Because that is the whole game. A monitor compares "what is on the page now" with "what I have already delivered". Anything that fell off the bottom of the page between two checks was never compared to anything. It is not late. It is gone.&lt;/p&gt;

&lt;p&gt;All numbers below are from runs I made on 15 October 2026, build 0.1.29, plus one run from 14 October on build 0.1.27. No login, no cookies, no browser — these are the pages Threads serves to a visitor who is not signed in.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three accounts, one window
&lt;/h2&gt;

&lt;p&gt;I ran the monitor in preview mode (no watch name, nothing remembered) against three accounts, asking for up to 25 posts each. What came back:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Account&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;th&gt;Distinct threads&lt;/th&gt;
&lt;th&gt;Oldest → newest row&lt;/th&gt;
&lt;th&gt;Depth&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;@cnn&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;18:59:07 → 21:00:21 UTC, same day&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2.02 hours&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;@mosseri&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2026-10-02 13:32 → 2026-10-12 17:09&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;10.15 days&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;@zuck&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2026-09-25 16:50 → 2026-10-12 16:56&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;17.0 days&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The row counts are 4, 5 and 10. The thread counts are 4, 4 and 4.&lt;/p&gt;

&lt;p&gt;The profile HTML does not embed a number of posts. It embeds &lt;strong&gt;four threads&lt;/strong&gt;, and a thread is one post or a chain of them. &lt;code&gt;@zuck&lt;/code&gt;'s four threads happen to be 4 + 1 + 4 + 1 posts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-10-12T16:56:29Z  thread=4006434140014576097  pos=0/4
2026-10-12T16:57:09Z  thread=4006434140014576097  pos=1/4
2026-10-12T16:56:28Z  thread=4006434140014576097  pos=2/4
2026-10-12T16:56:28Z  thread=4006434140014576097  pos=3/4
2026-10-12T15:01:17Z  thread=4006376160339141718  pos=0/1
2026-09-28T12:35:32Z  thread=3996155940894885511  pos=0/4
…
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;@cnn&lt;/code&gt;'s four threads are four single posts, which is why a news desk that posts all day gives you a two-hour window and a Meta executive who posts twice a week gives you seventeen days. Same markup, same four slots, a 200× difference in what they cover.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your schedule is in minutes; the window is in threads
&lt;/h2&gt;

&lt;p&gt;This is the part I had wrong when I wrote the first version. I had reasoned about the check interval in the only unit I controlled — minutes — and picked an hour, because most Threads accounts post a few times a day and an hourly check is plainly enough for them.&lt;/p&gt;

&lt;p&gt;It is plainly enough for &lt;code&gt;@mosseri&lt;/code&gt;. For &lt;code&gt;@cnn&lt;/code&gt; the same hour buys you a window with &lt;strong&gt;two slots of headroom&lt;/strong&gt;: the account was running at roughly two posts an hour, four threads deep. A check every three hours against the embedded page alone would have silently skipped posts, and the monitor would have reported nothing wrong, because every post it &lt;em&gt;did&lt;/em&gt; see was correctly marked new. Silent, plausible, wrong — the worst failure a scheduled job can have.&lt;/p&gt;

&lt;p&gt;The conversion factor between "four threads" and "how long I may sleep" is the account's posting rate, and that is the one variable in the system that belongs to someone else.&lt;/p&gt;

&lt;h2&gt;
  
  
  The signal that you fell off the end
&lt;/h2&gt;

&lt;p&gt;You cannot know the posts you did not see. But there is a tell, and it is cheap: &lt;strong&gt;if every single thread on the page is new to the watch, the page no longer overlaps your memory&lt;/strong&gt;, and an account that posted exactly four times since the last check is far less likely than an account that posted more.&lt;/p&gt;

&lt;p&gt;So the normal check reads one page. Only when the overlap is empty does it ask for the next page of the profile feed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;gap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;isPreview&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;isFirstCheck&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;
    &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;isPinned&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;every&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;seenBefore&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;gap&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;tab&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;page_info&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;has_next_page&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// one refetch, 25 posts, stop at the first id this watch already delivered&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Seeded a watch with a memory that deliberately did not overlap and pointed it at &lt;code&gt;@cnn&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;INFO  Watch "devto-gap2": 1 post(s) already delivered by earlier runs.
INFO  @cnn: 10 new of 25 checked
INFO  Done: {"checked":1,"newPosts":10,"pages":2,"requests":2,"bytes":1309270}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two requests instead of one, 25 posts instead of 4. The ten rows it delivered (my &lt;code&gt;maxItems&lt;/code&gt; cap) span 5.65 hours; the 25 it read reach back about fourteen hours at that posting rate. A quiet account never triggers this and keeps paying one request per check — which matters when a schedule fires 720 times a month.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the cursor is an id and never a timestamp
&lt;/h2&gt;

&lt;p&gt;Look again at &lt;code&gt;@zuck&lt;/code&gt;'s four rows in feed order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;16:56:29   16:57:09   16:56:28   16:56:28
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Not descending. Not ascending. The &lt;em&gt;feed&lt;/em&gt; is newest-thread-first, but a thread's own posts come out in reply order, and the author wrote the second link of that chain forty seconds after the first and then back-filled. Store "the newest &lt;code&gt;createdAt&lt;/code&gt; I have delivered" as your cursor — the obvious design, the one every incremental-sync tutorial shows you — and the watermark lands at 16:57:09. The two posts at 16:56:28 that follow it in the feed are then older than the watermark, so they are dropped. &lt;strong&gt;Two of ten posts lost on a page where nothing went wrong.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The memory is therefore a set of post ids, and the gap scan stops at the first id already in that set rather than at a time. The same rule makes the second page safe: everything after the first familiar id is older than the last check and must not come back as new.&lt;/p&gt;

&lt;p&gt;One consequence worth stating plainly: filtered-out posts go into that set too. A post that failed your &lt;code&gt;minLikes&lt;/code&gt; floor today has been seen, and should not arrive tomorrow as breaking news because it finally got its tenth like.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cheapest useful thing a monitor does is refuse to look
&lt;/h2&gt;

&lt;p&gt;A watch will not check the same target more than once an hour. Fire it sooner and it stops before any request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Watch "devto-demo-15oct" was checked 29 s ago. A watch is checked at most once an hour,
so this run did not look again. Nothing is lost — anything new arrives with the next check.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That run finished in 1.245 seconds, made zero HTTP requests and delivered zero rows. Given the four-thread window, a tighter interval is not obviously wrong — it is what a crisis watch wants — but it has to be a decision, not an accident of someone copying a &lt;code&gt;*/5 * * * *&lt;/code&gt; cron line.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it
&lt;/h2&gt;

&lt;p&gt;Named watch, two targets, one schedule — the whole thing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/lergassy~threads-monitor/runs?token=&lt;/span&gt;&lt;span class="nv"&gt;$APIFY_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type: application/json'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"watchName":"brand-mentions","usernames":["cnn"],"keywords":["my brand"],"maxPostsPerCheck":25}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reuse &lt;code&gt;watchName&lt;/code&gt; and every later run returns the difference. Leave it empty and you get a preview that remembers nothing, which is how the three measurements above were taken: &lt;a href="https://apify.com/lergassy/threads-monitor?fpr=wmeplu" rel="noopener noreferrer"&gt;Threads Monitor&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  When this is the wrong tool
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;You want history.&lt;/strong&gt; A monitor's answer to "what did they post in June" is nothing at all. That is a &lt;a href="https://apify.com/lergassy/threads-scraper" rel="noopener noreferrer"&gt;Threads Scraper&lt;/a&gt; job — search, full accounts, whole threads, profiles.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You want sub-minute latency.&lt;/strong&gt; Four threads deep and a once-an-hour floor is not a firehose, and public pages will never be one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You want your own account's numbers.&lt;/strong&gt; Reach, impressions, follower demographics — that is the official API, which exists precisely and only for the account you own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You already have a dedup table you trust.&lt;/strong&gt; Then you have a monitor; it is just made of a scraper plus your table, and you are paying for the overlap you throw away.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And do not hardcode four. It is a server-side rendering choice that owes me nothing: three accounts, one day, one shape. The heuristic that says &lt;em&gt;"no overlap means read further"&lt;/em&gt; survives Meta changing that number. A constant does not.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>javascript</category>
      <category>node</category>
      <category>monitoring</category>
    </item>
    <item>
      <title>Basecamp's Home Page Gave Me Two Valid Phone Numbers. Both Were Calendar Day Numbers</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Thu, 15 Oct 2026 13:12:18 +0000</pubDate>
      <link>https://dev.to/matveyg/basecamps-home-page-gave-me-two-valid-phone-numbers-both-were-calendar-day-numbers-oj3</link>
      <guid>https://dev.to/matveyg/basecamps-home-page-gave-me-two-valid-phone-numbers-both-were-calendar-day-numbers-oj3</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them does something that sounds trivial until you try it: you give it a list of websites, it gives you back one row per website with the company's e-mail addresses, phone numbers, socials, contact page and postal address, each contact carrying the page it was found on.&lt;/p&gt;

&lt;p&gt;Fetching the pages is the easy part: a business site is a handful of HTML documents and no login. The part that took the most work — and still isn't finished — is that &lt;strong&gt;marketing pages are full of contact data that is completely fake and completely well-formed&lt;/strong&gt;. Not malformed strings you can regex away. Valid E.164 phone numbers. Addresses on the company's own domain that pass a DNS deliverability check.&lt;/p&gt;

&lt;p&gt;Here are three of them, from real runs on 13–14 October 2026, all reproducible against the live pages as I write this.&lt;/p&gt;

&lt;h2&gt;
  
  
  A calendar is ten digits long
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;basecamp.com&lt;/code&gt;, home page only, one page read, 568 ms. Two phone numbers came back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+1 301 516 1718   fixed_line_or_mobile   source: text
+1 301 213 1415   fixed_line_or_mobile   source: text
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Basecamp does not publish a phone number. These came out of the page text, and I only found out where by printing the matched substring instead of the parsed number:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+13015161718  MATCHED=&amp;gt; "30 15 16 17 18"
+13012131415  MATCHED=&amp;gt; "30 12 13 14 15"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Basecamp's home page shows off its calendar view, and that screenshot is not an image — it is rendered HTML. Strip the tags and a month grid collapses into a run of day numbers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;… 29 30 31 Sep 1, 2026 Team Sync 11:00 - 11:30 2 3 Product Meeting 10:00 - 11:00 …
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;30 12 13 14 15&lt;/code&gt; is ten digits with plausible spacing. I was running candidates through &lt;code&gt;libphonenumber-js&lt;/code&gt;, which is the right tool and gave the right answer: in the North American plan, area code 301 (Maryland) with exchange 213 is a valid number. It &lt;em&gt;is&lt;/em&gt; a valid number. It is just not anybody's.&lt;/p&gt;

&lt;p&gt;My guard against this was a length floor — a number found in prose needs at least seven national digits, so that order IDs and dates don't become phones:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;source&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;text&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;pn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;nationalNumber&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ten digits sails past it. A length floor cannot distinguish a phone number from a calendar, and neither can a numbering-plan validator. The only thing that separates them is &lt;strong&gt;where the digits were standing&lt;/strong&gt;, which is why every phone in the output carries &lt;code&gt;source&lt;/code&gt; and &lt;code&gt;sourceUrl&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tel: 4     a tel: link — a human put this here to be dialled
schema.org: 3
whatsapp: 3
text: 1    digits that happened to be shaped like a phone
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Rank 1 means "shaped like". Basecamp's real contact on that same page is rank 4 and unambiguous: &lt;code&gt;mailto:jason@basecamp.com&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stripe's mock checkout, and a filter that fired four times out of five
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;stripe.com&lt;/code&gt;, same run. One e-mail address came back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"jane.diaz@stripe.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"personal"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"isOwnDomain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"sourceUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://stripe.com/"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is not a Stripe mailbox. Grep the home page for every address on it and the story is immediate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight codeowners"&gt;&lt;code&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;jane.smith@example.com&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;taro.yamada@example.com&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;damian.michelfelder@example.com&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;billing@example.com&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;jane.diaz@stripe.com&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;They all live in one place — the i18n bundle for the mock checkout graphic on the home page:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight properties"&gt;&lt;code&gt;&lt;span class="err"&gt;mkt-ssr.homepage.modular-solutions.payment-graphic.checkout.email-address-us&lt;/span&gt; &lt;span class="err"&gt;→&lt;/span&gt; &lt;span class="err"&gt;"jane.diaz@stripe.com"&lt;/span&gt;
&lt;span class="err"&gt;mkt-ssr.homepage.modular-solutions.payment-graphic.checkout.email-address-de&lt;/span&gt; &lt;span class="err"&gt;→&lt;/span&gt; &lt;span class="err"&gt;"damian.michelfelder@example.com"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Whoever wrote the German string used &lt;code&gt;example.com&lt;/code&gt;, the domain reserved by RFC 2606 for exactly this. Whoever wrote the US string used &lt;code&gt;stripe.com&lt;/code&gt;. My placeholder filter caught four of the five:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;PLACEHOLDER_DOMAIN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Set&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;example.com&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;example.org&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;yourdomain.com&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;test.com&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="err"&gt;…&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;PLACEHOLDER_LOCAL&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sr"&gt;/^&lt;/span&gt;&lt;span class="se"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;example|sample|test|yourname|john&lt;/span&gt;&lt;span class="se"&gt;\.?&lt;/span&gt;&lt;span class="sr"&gt;doe|jane&lt;/span&gt;&lt;span class="se"&gt;\.?&lt;/span&gt;&lt;span class="sr"&gt;doe|demo|dummy|…&lt;/span&gt;&lt;span class="se"&gt;)&lt;/span&gt;&lt;span class="sr"&gt;$/i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;jane.doe&lt;/code&gt; is in there. &lt;code&gt;jane.diaz&lt;/code&gt; is not, and never will be — the list is of placeholder &lt;em&gt;conventions&lt;/em&gt;, and a convention nobody agreed on is not detectable.&lt;/p&gt;

&lt;p&gt;Then the ranking promoted it. The score is deliberately biased toward the company's own domain, because for lead data that is almost always right:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;emailScore&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
      &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;isOwnDomain&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;isFree&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;rank&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
    &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;PRIMARY_LOCAL&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;email&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;type&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;personal&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;type&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;role&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here that bias is exactly backwards. The one fake address on the page is the one on &lt;code&gt;stripe.com&lt;/code&gt;, so it scores 100 while its four honest siblings have already been dropped, and &lt;code&gt;bestEmail&lt;/code&gt; is a person who does not exist.&lt;/p&gt;

&lt;p&gt;The deliverability add-on does not save you either:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"jane.diaz@stripe.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"deliverable"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"provider"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"google"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"mxHost"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"aspmx.l.google.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"isRole"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That verdict is correct and useless at once. A DNS-level check asks whether the &lt;em&gt;domain&lt;/em&gt; accepts mail, and &lt;code&gt;stripe.com&lt;/code&gt; obviously does. Nothing short of sending mail knows whether the mailbox exists, and I am not doing that from a scraper.&lt;/p&gt;

&lt;p&gt;The honest answer for Stripe is in a different run. Pointed straight at &lt;code&gt;https://stripe.com/contact&lt;/code&gt; — three pages read, including &lt;code&gt;support.stripe.com&lt;/code&gt; — the row came back with &lt;strong&gt;zero e-mails&lt;/strong&gt; and one phone:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"domain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"stripe.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"emails"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"phones"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"phone"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"+18889262289"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"kind"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"toll_free"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
               &lt;/span&gt;&lt;span class="nl"&gt;"sourceUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://stripe.com/contact"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"summary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Stripe: +18889262289."&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Stripe publishes a support line and no e-mail address. The home page's only "e-mail" is a prop. Both rows are accurate reports of what the page said; only one of them is a contact.&lt;/p&gt;

&lt;h2&gt;
  
  
  Five real addresses, and the wrong desk wins
&lt;/h2&gt;

&lt;p&gt;Not every miss is fake data. &lt;code&gt;zendesk.com&lt;/code&gt;, three pages, five addresses — all of them genuine &lt;code&gt;mailto:&lt;/code&gt; links off the contact page:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ask.philippines@zendesk.com   personal   mailto
ask.thailand@zendesk.com      personal   mailto
ask.indonesia@zendesk.com     personal   mailto
ask.malaysia@zendesk.com      personal   mailto
ask.gcr@zendesk.com           personal   mailto
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;bestEmail&lt;/code&gt; is the Philippines desk — for no reason at all, just first in page order among five identical scores. And all five are typed &lt;code&gt;personal&lt;/code&gt;, which is wrong: these are regional team mailboxes. The classifier is two lines long:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ROLE_PREFIXES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;l&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;ROLE_PREFIXES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;[&lt;/span&gt;&lt;span class="sr"&gt;._-&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;role&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/^&lt;/span&gt;&lt;span class="se"&gt;[&lt;/span&gt;&lt;span class="sr"&gt;a-z&lt;/span&gt;&lt;span class="se"&gt;]{2,}[&lt;/span&gt;&lt;span class="sr"&gt;._&lt;/span&gt;&lt;span class="se"&gt;][&lt;/span&gt;&lt;span class="sr"&gt;a-z&lt;/span&gt;&lt;span class="se"&gt;]{2,}&lt;/span&gt;&lt;span class="sr"&gt;$/&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;l&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;personal&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;// jane.diaz, j_smith&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;generic&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;ask&lt;/code&gt; is in &lt;code&gt;ROLE_PREFIXES&lt;/code&gt;. &lt;code&gt;ask.philippines&lt;/code&gt; collapses to &lt;code&gt;askphilippines&lt;/code&gt;, which is not, so the "two words joined by a dot" rule wins and calls it a person. The same rule is what typed &lt;code&gt;jane.diaz&lt;/code&gt; as personal and handed it the +4. Telling a person's name from a department name without a names list is guesswork, and I document it as guesswork rather than pretending otherwise.&lt;/p&gt;

&lt;p&gt;Compare &lt;code&gt;apify.com&lt;/code&gt; in the same run — four addresses, all &lt;code&gt;role&lt;/code&gt;, all &lt;code&gt;mailto&lt;/code&gt;, &lt;code&gt;hello@apify.com&lt;/code&gt; on top because &lt;code&gt;hello&lt;/code&gt; is in &lt;code&gt;PRIMARY_LOCAL&lt;/code&gt;. That is what the scoring is built for, and on a site that publishes contacts like a normal business it works.&lt;/p&gt;

&lt;h2&gt;
  
  
  The other failure mode: nothing at all
&lt;/h2&gt;

&lt;p&gt;Worth saying plainly, because lead-gen tooling rarely does. Out of the big SaaS names I tested, these returned &lt;code&gt;status: "no_contacts"&lt;/code&gt; after reading three to six pages each:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;monzo.com  notion.com  zapier.com  vercel.com  ycombinator.com  bbc.co.uk
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Vercel's row read every door — &lt;code&gt;/contact&lt;/code&gt;, &lt;code&gt;/contact/sales&lt;/code&gt;, &lt;code&gt;/about&lt;/code&gt; — and came back &lt;code&gt;"Vercel: no e-mail or phone; linkedin, facebook, instagram, x, youtube."&lt;/code&gt; There is no scraping trick behind that. A company that routes everything through a form and a support portal has no e-mail address to find, and a tool that returns one for them is making it up. Those rows are also free in this Actor — you are billed per website where a contact was actually found, not per website submitted — which is the only honest way to price a job whose hit rate depends on the list.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd tell you to do with the output
&lt;/h2&gt;

&lt;p&gt;Use &lt;code&gt;source&lt;/code&gt; as a trust column, not decoration. For cold outreach, keep &lt;code&gt;mailto&lt;/code&gt; and &lt;code&gt;schema.org&lt;/code&gt; and treat &lt;code&gt;text&lt;/code&gt; as a lead to check by hand — every contact ships with &lt;code&gt;sourceUrl&lt;/code&gt;, so checking is one click. The same goes for phones: &lt;code&gt;tel&lt;/code&gt; is dialled by a human somewhere, &lt;code&gt;text&lt;/code&gt; is a pattern match. If you are enriching a CRM rather than mailing, &lt;code&gt;isOwnDomain&lt;/code&gt; and &lt;code&gt;isFree&lt;/code&gt; matter more, since a sole trader's only address really is on gmail.com.&lt;/p&gt;

&lt;p&gt;Calling it is a list of domains and nothing else:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/lergassy~website-contacts-api/run-sync-get-dataset-items?token=&lt;/span&gt;&lt;span class="nv"&gt;$APIFY_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type: application/json'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"domains":["basecamp.com","zendesk.com","apify.com"],"maxPagesPerSite":3,"stopEarly":true}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;stopEarly&lt;/code&gt; stops reading a site once a page has yielded both an own-domain e-mail and a valid phone — which is why Basecamp's row says &lt;code&gt;pagesScannedCount: 1&lt;/code&gt;. Turn it off and you get every address on every page; the price does not change with pages read either way.&lt;/p&gt;

&lt;p&gt;The Actor is &lt;a href="https://apify.com/lergassy/website-contacts-api?fpr=wmeplu" rel="noopener noreferrer"&gt;website-contacts-api on Apify&lt;/a&gt;. Where it does not fit: it reads HTML, so a site whose contact block only exists after JavaScript runs will come back thin (the row says so — &lt;code&gt;note: "javascript_rendered_site"&lt;/code&gt;); it verifies e-mails at the domain level and never probes a mailbox; and it will not find a person's direct address that the company has not published. It finds what a website publishes, which — as Basecamp's calendar demonstrates — occasionally includes things that merely look published.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>javascript</category>
      <category>data</category>
      <category>node</category>
    </item>
    <item>
      <title>Capterra and Software Advice Share One Review Database and Disagree by 13,058 Reviews</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Thu, 15 Oct 2026 05:12:38 +0000</pubDate>
      <link>https://dev.to/matveyg/capterra-and-software-advice-share-one-review-database-and-disagree-by-13058-reviews-jd5</link>
      <guid>https://dev.to/matveyg/capterra-and-software-advice-share-one-review-database-and-disagree-by-13058-reviews-jd5</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The half of my week that has nothing to do with visas goes into Apify Actors, and one of them collects B2B software reviews — Capterra, Software Advice, GetApp and TrustRadius — and returns every site with the same field names.&lt;/p&gt;

&lt;p&gt;That last part started as a convenience feature — four scrapers, four field vocabularies, one translation layer I got tired of rewriting. What it turned into was a measuring instrument: once the rows line up you can put the sites side by side and see where they disagree. They disagree a lot, and not in the way I expected.&lt;/p&gt;

&lt;p&gt;Here is the measurement. Product: Slack. Builds &lt;code&gt;0.1.39&lt;/code&gt; and &lt;code&gt;0.1.40&lt;/code&gt;, runs on 13–15 October 2026, no proxy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three sites, one database, three different totals
&lt;/h2&gt;

&lt;p&gt;Ask for Slack on Capterra, Software Advice and TrustRadius, ten reviews each, and the product header on those rows reads:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csvs"&gt;&lt;code&gt;&lt;span class="k"&gt;source&lt;/span&gt;           &lt;span class="k"&gt;productReviewCount&lt;/span&gt;   &lt;span class="k"&gt;productRating&lt;/span&gt;   &lt;span class="k"&gt;ratingScale&lt;/span&gt;
&lt;span class="k"&gt;capterra&lt;/span&gt;                     &lt;span class="mf"&gt;11&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;166&lt;/span&gt;          &lt;span class="mf"&gt;4.4643&lt;/span&gt;            &lt;span class="mf"&gt;5&lt;/span&gt;
&lt;span class="k"&gt;softwareadvice&lt;/span&gt;               &lt;span class="mf"&gt;24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;224&lt;/span&gt;          &lt;span class="mf"&gt;4.6639&lt;/span&gt;            &lt;span class="mf"&gt;5&lt;/span&gt;
&lt;span class="k"&gt;getapp&lt;/span&gt;                       &lt;span class="mf"&gt;24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;224&lt;/span&gt;          &lt;span class="mf"&gt;4.6639&lt;/span&gt;            &lt;span class="mf"&gt;5&lt;/span&gt;
&lt;span class="k"&gt;trustradius&lt;/span&gt;                  &lt;span class="mf"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;268&lt;/span&gt;               &lt;span class="mf"&gt;9&lt;/span&gt;           &lt;span class="mf"&gt;10&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Software Advice and GetApp agree to four decimal places. Capterra — the biggest brand of the three, the one whose logo everyone recognises — reports 11,166 reviews and an average two tenths of a star lower.&lt;/p&gt;

&lt;p&gt;My first guess was that these are simply three separate review pools that happen to overlap in subject matter. That guess is wrong, and the &lt;code&gt;reviewId&lt;/code&gt; field is what killed it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;softwareadvice  Capterra___7219769
softwareadvice  Capterra___7224596
getapp          Capterra___7138404
capterra        Capterra___7209002
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are reviews scraped from &lt;code&gt;softwareadvice.com&lt;/code&gt; and &lt;code&gt;getapp.com&lt;/code&gt;. They carry Capterra identifiers because all three sites are Gartner properties running on one review database. The IDs interleave, too — 13 of the 20 Software Advice IDs I collected fall &lt;em&gt;inside&lt;/em&gt; the numeric range of the 45 Capterra IDs. It is one sequence, handed out to one pool of reviews, served through three front doors with three different filters on the way out.&lt;/p&gt;

&lt;p&gt;So the 13,058-review gap is not a data gap. It is an editorial decision about which of the shared reviews each brand chooses to display, and it is invisible if you scrape one site.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same review, twice, with a different reviewer on it
&lt;/h2&gt;

&lt;p&gt;Across 45 Capterra reviews and 20 Software Advice reviews I caught exactly one ID on both sites: &lt;code&gt;Capterra___7224596&lt;/code&gt;. Field by field:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;field                     capterra                softwareadvice
title                     "Slack is the best…"    "Slack is the best…"   same
publishedAt               2026-10-04              2026-10-04             same
pros / cons               identical text          identical text         same
overallRating             5                       5                      same
easeOfUseRating           5                       5                      same
functionalityRating       5                       5                      same
valueForMoneyRating       5                       5                      same
customerSupportRating     4                       4                      same
incentivized              true  (NominalGift)     true  (NominalGift)    same
reviewerIndustry          Construction            Construction           same
--
reviewerName              "Chris S."              "Chris S——" (full)     differs
reviewerJobTitle          "General Manager"       null                   differs
reviewerCompanySize       "51-200 employees"      null                   differs
reviewerCompanySizeCode   null                    "D"                    differs
timeUsed                  null                    "More than 2 years"    differs
usageFrequency            null                    "Daily"                differs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything a reviewer typed is byte-identical. Everything &lt;em&gt;about&lt;/em&gt; the reviewer is split between the two pages. Capterra publishes the job title and a readable company-size range and abbreviates the surname to an initial. Software Advice publishes the reviewer's full first and last name — I'm redacting it here, since the point is the disclosure difference and not this person — drops the job title, and replaces the company-size range with a single letter.&lt;/p&gt;

&lt;p&gt;That letter is worth a paragraph of its own. Software Advice and GetApp publish &lt;code&gt;A&lt;/code&gt; through &lt;code&gt;I&lt;/code&gt; where Capterra publishes &lt;code&gt;11-50 employees&lt;/code&gt;, and neither site publishes a legend. The temptation is to map it yourself: &lt;code&gt;A&lt;/code&gt; is surely the smallest bucket, &lt;code&gt;I&lt;/code&gt; the largest, nine letters for nine ranges. I didn't, because on this one shared review the letter is &lt;code&gt;D&lt;/code&gt; and the range Capterra prints for the same company is &lt;code&gt;51-200 employees&lt;/code&gt; — a single data point, enough to build a guess on and nowhere near enough to ship one. So the Actor passes the letter through in &lt;code&gt;reviewerCompanySizeCode&lt;/code&gt; and leaves &lt;code&gt;reviewerCompanySize&lt;/code&gt; empty. One confirmed pair is a hint, not a mapping.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does to de-duplication
&lt;/h2&gt;

&lt;p&gt;If you are merging Capterra, Software Advice and GetApp, &lt;code&gt;reviewId&lt;/code&gt; is a real join key and you should use it. But the merge is not idempotent in either direction, because neither row is a superset of the other. Prefer the Capterra row and you lose &lt;code&gt;usageFrequency&lt;/code&gt; and &lt;code&gt;timeUsed&lt;/code&gt;. Prefer the Software Advice row and you lose the job title and the human-readable company size. The only correct merge is field-level: for each column, take whichever side is non-null.&lt;/p&gt;

&lt;p&gt;TrustRadius stays out of this entirely. Its IDs are slugs with a timestamp — &lt;code&gt;slack-2026-08-05-00-29-43&lt;/code&gt; — its reviewers are its own, and across ten rows it returned zero sub-ratings while filling in &lt;code&gt;reviewerJobTitle&lt;/code&gt; on all ten ("Full-stack developer", "Sr. Director engineering", "Senior Salesforce Administrator"). Different site, different database, genuinely additive sample.&lt;/p&gt;

&lt;p&gt;And GetApp is no longer a review source in any useful sense. Its product totals are complete and match Software Advice exactly, but the product page now publishes about six short quotes instead of full reviews, with no ratings at all. Of the five rows I got back, three had &lt;code&gt;pros&lt;/code&gt; filled and &lt;code&gt;cons&lt;/code&gt; empty, two the reverse — the page quotes one half of each review and doesn't say which half. The first one reads &lt;em&gt;"However, managing constant notifications can be overwhelming at times…"&lt;/em&gt;, which is a complaint, sitting in a field called &lt;code&gt;pros&lt;/code&gt;, because that is the only text GetApp published for it. Use GetApp for product-level numbers; don't build a sentiment set from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Page one is not a sample
&lt;/h2&gt;

&lt;p&gt;One more thing I only saw because the rows lined up. Ten reviews from each site, in the order the site publishes them, no sort, no filter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;capterra         5 5 3 3 4 5 5 4 4 5     avg 4.3
softwareadvice   5 5 5 5 5 5 5 5 5 5     avg 5.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same database. Software Advice's front page was a wall of five stars; Capterra's front page was not. Five of those ten Software Advice reviews were flagged &lt;code&gt;incentivized&lt;/code&gt; — &lt;code&gt;NominalGift&lt;/code&gt; or &lt;code&gt;VendorReferredIncentivized&lt;/code&gt; — against 14 of 45 on Capterra.&lt;/p&gt;

&lt;p&gt;The low ratings do exist in the shared pool, they are just not on page one. Asking for them explicitly finds them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"searchTerms"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Slack"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"searchSites"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"softwareadvice"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"maxRating"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"maxReviews"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ten rows: nine 3-star and one 1-star, the oldest from 2025-10-31, six of the ten incentivized. The filter runs &lt;em&gt;while&lt;/em&gt; paging rather than after it, so the run keeps reading pages until it has what you asked for or the site runs out:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;matches_filters&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;min_rating&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_rating&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;published_after&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;published_before&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# TrustRadius rates out of 10, the Gartner sites out of 5, so filtering happens on
&lt;/span&gt;    &lt;span class="c1"&gt;# the normalized 1-5 value the user is thinking in.
&lt;/span&gt;    &lt;span class="n"&gt;rating&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;normalizedRating&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;overallRating&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That normalization is why &lt;code&gt;maxRating: 3&lt;/code&gt; means the same band of unhappy customers on TrustRadius as on Capterra. &lt;code&gt;overallRating&lt;/code&gt; keeps the site's own number, &lt;code&gt;ratingScale&lt;/code&gt; says which scale it was, &lt;code&gt;normalizedRating&lt;/code&gt; is &lt;code&gt;raw * 5 / scale&lt;/code&gt;. A TrustRadius 9 arrives as &lt;code&gt;overallRating: 9, ratingScale: 10, normalizedRating: 4.5&lt;/code&gt;, and averaging that against a Capterra 5 is finally a legal operation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it
&lt;/h2&gt;

&lt;p&gt;Search by name on Software Advice and TrustRadius, by URL on Capterra and GetApp — Capterra's search page refuses automated requests even through a residential proxy, and GetApp publishes no search page at all. Their product pages are served normally, so a URL is all you need:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/lergassy~software-reviews-scraper/run-sync-get-dataset-items?token=&lt;/span&gt;&lt;span class="nv"&gt;$APIFY_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type: application/json'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{
    "productUrls": ["https://www.capterra.com/p/135003/Slack/reviews/"],
    "searchTerms": ["Slack"],
    "searchSites": ["trustradius", "softwareadvice"],
    "maxReviews": 200,
    "includeProductRow": true
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Thirty reviews across three sites took 46 seconds; thirty from Capterra alone took 90, because reviews come 25 to a page and each page is a request. Budget roughly three to four seconds per page. The &lt;a href="https://apify.com/lergassy/software-reviews-scraper?fpr=wmeplu" rel="noopener noreferrer"&gt;Actor is here&lt;/a&gt; if you want to run the comparison on your own category — pricing is per row, $0.0012 a review and $0.003 a product summary, and failed URLs come back as &lt;code&gt;error&lt;/code&gt; rows at no charge.&lt;/p&gt;

&lt;h2&gt;
  
  
  When not to use this
&lt;/h2&gt;

&lt;p&gt;Don't use it to count. If you need "how many reviews does this product have", you now know that question has two different right answers depending on which Gartner brand you ask, and any single number you quote is a brand's choice rather than a fact.&lt;/p&gt;

&lt;p&gt;Don't use it for G2 — different site, different gate, not covered here. And if what you want is volume over time, note that &lt;code&gt;publishedAt&lt;/code&gt; is reliable but the sites' own ordering is not chronological, so a date histogram needs the full history pulled, not a recent slice.&lt;/p&gt;

&lt;p&gt;Do use it when the comparison is the point: the same product across four sites, the same columns, and a normalized rating that makes an average mean something. Just merge field-level, and treat page one as a storefront rather than a sample.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>api</category>
      <category>data</category>
    </item>
    <item>
      <title>Ten YouTube Comments, One Commenter: What a Reply Budget Does to Your Sample</title>
      <dc:creator>Matvey</dc:creator>
      <pubDate>Wed, 14 Oct 2026 21:11:36 +0000</pubDate>
      <link>https://dev.to/matveyg/ten-youtube-comments-one-commenter-what-a-reply-budget-does-to-your-sample-31ii</link>
      <guid>https://dev.to/matveyg/ten-youtube-comments-one-commenter-what-a-reply-budget-does-to-your-sample-31ii</guid>
      <description>&lt;p&gt;I run a visa agency in Bali. The part of my week that has nothing to do with visas goes into Apify Actors, and one of them pulls YouTube comments — one video, a channel, a playlist, or the top results of a search, one row per comment with likes, reply count, the pinned and hearted flags and the commenter's channel.&lt;/p&gt;

&lt;p&gt;People use comment data for sentiment: pull the last twenty videos of a competitor, feed the text to a model, read what their audience actually complains about. That works, and the scraping part is not the hard bit. The hard bit is that two of the three obvious assumptions about a comment sample are wrong, and both of them are wrong &lt;em&gt;quietly&lt;/em&gt; — you get a full dataset, the right number of rows, no errors, and a conclusion that isn't about the audience.&lt;/p&gt;

&lt;p&gt;Here is the measurement, from runs on 11–14 October 2026, build &lt;code&gt;0.1.39&lt;/code&gt;, no proxy.&lt;/p&gt;

&lt;h2&gt;
  
  
  One flag, two completely different samples
&lt;/h2&gt;

&lt;p&gt;Same video — &lt;code&gt;jNQXAC9IVRw&lt;/code&gt;, "Me at the zoo", the first video on YouTube, 20 years of comments on it. Same limit: 10 comments. Sorted by top. The only difference between the two runs is &lt;code&gt;includeReplies&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;With replies &lt;strong&gt;off&lt;/strong&gt;, ten comments from ten different people:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;"We're so honored that the first ever YouTube video was filmed here!"   4,800,000 likes
"So we're all here again on October 14th 2026 right"                          746
"Hello"                                                                   303,000
"Is anyone else watching this on a random day in October?"                  1,000
"I'm glad this pops up on my feed every couple years"                         467
"This started it all...... bruh 😳"                                       248,000
"this video lived long enough to see gifs come true"                          459
'Seeing "20 years ago" on a YouTube video makes me feel a certain way'    211,000
'This guys is the only person who has the rights to say "first"'          418,000
"First word on Youtube ever:\nAlright"                                    115,000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With replies &lt;strong&gt;on&lt;/strong&gt; — the default — and the same &lt;code&gt;maxCommentsPerVideo: 10&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csvs"&gt;&lt;code&gt;&lt;span class="k"&gt;isReply&lt;/span&gt;  &lt;span class="k"&gt;text&lt;/span&gt;                                                          &lt;span class="k"&gt;likes&lt;/span&gt;
&lt;span class="k"&gt;false&lt;/span&gt;    &lt;span class="s2"&gt;"We're so honored that the first ever YouTube video was…"&lt;/span&gt;  &lt;span class="mf"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;800&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;"San Diego Zoo That's so awesome! I'm so glad the first…"&lt;/span&gt;    &lt;span class="mf"&gt;206&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;"Very cool!"&lt;/span&gt;                                                  &lt;span class="mf"&gt;71&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;"cool"&lt;/span&gt;                                                        &lt;span class="mf"&gt;57&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;"."&lt;/span&gt;                                                           &lt;span class="mf"&gt;54&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;"How did I randomly go to this video and see this comment…"&lt;/span&gt;   &lt;span class="mf"&gt;57&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;"Im here before this blows up"&lt;/span&gt;                                &lt;span class="mf"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;" @Quetzal_  lol what"&lt;/span&gt;                                        &lt;span class="mf"&gt;17&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;"yoo here before thousands of likes"&lt;/span&gt;                          &lt;span class="mf"&gt;18&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;span class="k"&gt;true&lt;/span&gt;     &lt;span class="s2"&gt;"Awesome!"&lt;/span&gt;                                                    &lt;span class="mf"&gt;14&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mf"&gt;000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One top-level comment and nine replies to it. Ten rows, one conversation, and four of them are a single word or a full stop. Run that through a sentiment model and you have measured one thread on a video with 2.4 million comments.&lt;/p&gt;

&lt;p&gt;This is not a bug, and the input field says so in six words — &lt;em&gt;"replies count too when included"&lt;/em&gt;. It is the interaction that bites. The collector walks the comment list depth-first: take a top-level comment, immediately follow its replies, then move to the next one. Both compete for the same budget:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;keep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;pushComment&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;base&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;kept&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;includeReplies&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;replyCount&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;replyTokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;commentId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;kept&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;wanted&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// …up to maxRepliesPerComment replies, each one incrementing the same `kept`&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The comment at the top of "Me at the zoo" has 985 replies and &lt;code&gt;maxRepliesPerComment&lt;/code&gt; defaults to 20. So the first comment plus its reply cap is 21 rows — more than the sample I asked for. The depth-first order is the right default (a thread stays together, and a keyword match inside a thread is still a match), but it means the limit you set is a &lt;em&gt;row&lt;/em&gt; budget, not a &lt;em&gt;voice&lt;/em&gt; budget.&lt;/p&gt;

&lt;p&gt;The arithmetic to set it deliberately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rows ≈ topLevelComments × (1 + maxRepliesPerComment)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Twenty distinct opinions with their threads at the default reply cap is 420 rows, not 20. If you want breadth, &lt;code&gt;includeReplies: false&lt;/code&gt; is the whole fix; the reply count stays on every row, so you still know which comments started something. If you want threads, raise the comment limit by a factor of 21 and expect to pay for it — the Actor bills per comment row, so replies are rows you buy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then I ran the identical input four days running
&lt;/h2&gt;

&lt;p&gt;Four runs, 11 to 14 October, same JSON every time: that video, &lt;code&gt;maxCommentsPerVideo: 50&lt;/code&gt;, replies on, &lt;code&gt;sort: top&lt;/code&gt;. Four datasets of exactly 50 rows.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Rows&lt;/th&gt;
&lt;th&gt;Distinct comments shared with the 14 Oct run&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;11 Oct&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12 Oct&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13 Oct&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14 Oct&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Across the four runs there are &lt;strong&gt;121 distinct comments&lt;/strong&gt;, and only &lt;strong&gt;21 appear in all four&lt;/strong&gt;. Every run returned the same three top-level comments' worth of structure — 3 top-level, 47 replies, the first thread always the 4.8-million-like one — but comments two and three changed almost every day:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;11 Oct: "Randomly got recommended this at 11:16 PM…"          3,300 likes
12 Oct: "You will never know why this comment got…"          24,000
13 Oct: 'Seeing "20 years ago" on a YouTube video…'         211,000
14 Oct: "This started it all...... bruh 😳"                  247,000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;"Top comments" is a ranking YouTube recomputes, not a stable list, and on a video this size the tail of the top-50 reshuffles daily. The same comment also came back with 417,000 likes on one day and 418,000 on the next, which is the honest behaviour — counts move — but it means a comment is identified by &lt;code&gt;commentId&lt;/code&gt;, never by its text plus its like count. If you are tracking a video over time, dedupe on &lt;code&gt;commentId&lt;/code&gt; and expect the set to drift under you. If you are sampling once for sentiment, know that you sampled &lt;em&gt;a&lt;/em&gt; top-50, not &lt;em&gt;the&lt;/em&gt; top-50.&lt;/p&gt;

&lt;h2&gt;
  
  
  And the dates are arithmetic, not data
&lt;/h2&gt;

&lt;p&gt;YouTube never shows a comment date. It shows an age: "6 years ago", "9 hours ago", "2 days ago". The Actor keeps that string in &lt;code&gt;publishedAgo&lt;/code&gt; and also subtracts it from the run clock into &lt;code&gt;publishedAt&lt;/code&gt;, because a date column is what every spreadsheet downstream wants. Here is what that does to one comment — the same &lt;code&gt;commentId&lt;/code&gt;, five runs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run started (UTC)&lt;/th&gt;
&lt;th&gt;&lt;code&gt;publishedAgo&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;&lt;code&gt;publishedAt&lt;/code&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;11 Oct 00:33&lt;/td&gt;
&lt;td&gt;6 years ago&lt;/td&gt;
&lt;td&gt;2020-10-10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12 Oct 21:28&lt;/td&gt;
&lt;td&gt;6 years ago&lt;/td&gt;
&lt;td&gt;2020-10-12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;13 Oct 00:28&lt;/td&gt;
&lt;td&gt;6 years ago&lt;/td&gt;
&lt;td&gt;2020-10-12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14 Oct 00:45&lt;/td&gt;
&lt;td&gt;6 years ago&lt;/td&gt;
&lt;td&gt;2020-10-13&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;14 Oct 21:05&lt;/td&gt;
&lt;td&gt;6 years ago&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2020-10-14&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two runs on the same calendar day, two different dates, because the subtraction uses the run's clock and a year is 365.25 days. The mapping is not even one-to-one in either direction: the 12 Oct and 13 Oct runs, three hours apart, both landed on 2020-10-12, while the two 14 Oct runs landed a day apart.&lt;/p&gt;

&lt;p&gt;The drift is the small problem. The real one is the bucket: "6 years ago" is YouTube rounding down, so the comment's true age is somewhere in &lt;code&gt;[6 years, 7 years)&lt;/code&gt;. &lt;code&gt;publishedAt: 2020-10-13&lt;/code&gt; is the &lt;strong&gt;newest date the comment could have&lt;/strong&gt;; it could genuinely be from October 2019. For fresh comments the label is tight — "9 hours ago" pins the day — and for anything older than a month it is a floor, not a date. So: filter and group on &lt;code&gt;publishedAgo&lt;/code&gt; buckets when the comments are old, use &lt;code&gt;publishedAt&lt;/code&gt; for recency windows on fresh comments, and never diff two runs' &lt;code&gt;publishedAt&lt;/code&gt; columns and conclude anything.&lt;/p&gt;

&lt;p&gt;One more small trap in the same field: an edited comment is labelled &lt;code&gt;"6 years ago (edited)"&lt;/code&gt;, suffix and all. Across those four runs there were 22 such rows and four distinct label strings — &lt;code&gt;6 years ago (edited)&lt;/code&gt;, &lt;code&gt;5 years ago (edited)&lt;/code&gt;, &lt;code&gt;5 months ago (edited)&lt;/code&gt;, &lt;code&gt;1 month ago (edited)&lt;/code&gt;. Matching &lt;code&gt;publishedAgo == "6 years ago"&lt;/code&gt; silently drops them. The date parser strips the suffix before doing its arithmetic; a &lt;code&gt;GROUP BY&lt;/code&gt; in your sheet does not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Calling it
&lt;/h2&gt;

&lt;p&gt;One request, and the run-sync endpoint hands back the rows directly — this is the exact call that produced the ten-distinct-commenters sample above:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.apify.com/v2/acts/lergassy~youtube-comments-scraper/run-sync-get-dataset-items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;token&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;TOKEN&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;videoUrls&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.youtube.com/watch?v=jNQXAC9IVRw&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;maxCommentsPerVideo&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;includeReplies&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;      &lt;span class="c1"&gt;# breadth, not one thread
&lt;/span&gt;        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sort&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;language&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;en&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Swap &lt;code&gt;videoUrls&lt;/code&gt; for &lt;code&gt;channelUrls&lt;/code&gt;, &lt;code&gt;playlistUrls&lt;/code&gt; or &lt;code&gt;searchQueries&lt;/code&gt; and &lt;code&gt;maxVideosPerSource&lt;/code&gt; decides how many videos each source contributes; every row carries &lt;code&gt;source&lt;/code&gt; so you know where it came from. The Actor is here: &lt;a href="https://apify.com/lergassy/youtube-comments-scraper?fpr=wmeplu" rel="noopener noreferrer"&gt;YouTube Comments Scraper&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When not to reach for it.&lt;/strong&gt; If you need exact comment timestamps, nothing scraped from the watch page will give them to you — the official Data API returns real &lt;code&gt;publishedAt&lt;/code&gt; values, at the cost of a key and a daily quota. If you need &lt;em&gt;all&lt;/em&gt; comments of a viral video, the arithmetic is unkind: twenty comments per request means 2.4 million comments is 120,000 round trips. And if you want to post, reply or moderate, this is read-only; that is the API's job.&lt;/p&gt;

&lt;p&gt;The lesson I keep relearning on other people's data: a limit is only meaningful once you know what the thing being limited is. Ten rows looked like ten opinions right up until I printed the &lt;code&gt;isReply&lt;/code&gt; column.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>api</category>
      <category>youtube</category>
    </item>
  </channel>
</rss>
