<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ben Barazandeh</title>
    <description>The latest articles on DEV Community by Ben Barazandeh (@dedicated_crest).</description>
    <link>https://dev.to/dedicated_crest</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4207142%2F33dbbacd-8d9a-4b55-90a3-03ad7330b025.png</url>
      <title>DEV Community: Ben Barazandeh</title>
      <link>https://dev.to/dedicated_crest</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/dedicated_crest"/>
    <language>en</language>
    <item>
      <title>Querying the Grants.gov API in Python: Open Federal Grants by Keyword and Eligibility</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Sun, 08 Nov 2026 00:27:44 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/querying-the-grantsgov-api-in-python-open-federal-grants-by-keyword-and-eligibility-4b14</link>
      <guid>https://dev.to/dedicated_crest/querying-the-grantsgov-api-in-python-open-federal-grants-by-keyword-and-eligibility-4b14</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. The tutorial part needs only Python and the free Grants.gov API.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Every federal grant opportunity (NOFO) is posted on Grants.gov, and Grants.gov has a public JSON API with no key. Hardly anyone uses it directly, partly because the useful parts (award ceiling, total funding, eligibility, agency contact) aren't in the search results. Here's how to get them, and the quirks I hit.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two endpoints
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;POST https://api.grants.gov/v1/api/search2&lt;/code&gt; returns matching opportunities: id, number, title, agency, status, open and close dates. Up to &lt;code&gt;rows&lt;/code&gt; per page, with &lt;code&gt;startRecordNum&lt;/code&gt; for paging.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;POST https://api.grants.gov/v1/api/fetchOpportunity&lt;/code&gt; returns one opportunity's full detail: the synopsis with award ceiling and floor, estimated total funding, expected awards, cost sharing, eligible applicant types, Assistance Listings (CFDA) and the contact.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1: Search, then fetch details
&lt;/h2&gt;

&lt;p&gt;This ran as-is on 25 Oct 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.grants.gov/v1/api/search2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;keyword&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;artificial intelligence&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;oppStatuses&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;forecasted|posted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hitCount&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;matches&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;oppHits&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.grants.gov/v1/api/fetchOpportunity&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                      &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;opportunityId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])},&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;syn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;synopsis&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;number&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;hit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;closeDate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;syn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;awardCeiling&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;syn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;agencyContactEmail&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output (I've trimmed one individual's email):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;167 matches
PD-19-127Y 02/10/2027 none grantsgovsupport@nsf.gov
W519TC-25-S-0001 - None None
G27AS00079 11/14/2026 150000 &amp;lt;agency contact&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 2: Filter by who can apply
&lt;/h2&gt;

&lt;p&gt;This is the filter that matters most for nonprofits, universities and small businesses. Grants.gov tags each opportunity with eligible applicant codes, and &lt;code&gt;search2&lt;/code&gt; accepts them pipe-separated in &lt;code&gt;eligibilities&lt;/code&gt;. A few: &lt;code&gt;12&lt;/code&gt; = 501(c)(3) nonprofits, &lt;code&gt;13&lt;/code&gt; = nonprofits without 501(c)(3), &lt;code&gt;20&lt;/code&gt; = private institutions of higher education, &lt;code&gt;06&lt;/code&gt; = public and state-controlled institutions of higher education, &lt;code&gt;23&lt;/code&gt; = small businesses, &lt;code&gt;25&lt;/code&gt; = others.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.grants.gov/v1/api/search2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;oppStatuses&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;eligibilities&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;12&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rows&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hitCount&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;  &lt;span class="c1"&gt;# 440 on 25 Oct 2026
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So on that day, 440 posted opportunities accepted 501(c)(3) applicants.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: The quirks
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Inconsistent types.&lt;/strong&gt; &lt;code&gt;awardCeiling&lt;/code&gt; came back as the string &lt;code&gt;"none"&lt;/code&gt; for the NSF program and as a number for the USGS one. A forecasted DoD notice had no synopsis at all (it lives in a different &lt;code&gt;forecast&lt;/code&gt; block). Parse defensively and normalise to &lt;code&gt;null&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;US dates.&lt;/strong&gt; &lt;code&gt;closeDate&lt;/code&gt; is &lt;code&gt;MM/DD/YYYY&lt;/code&gt;, and blank for forecasts and rolling programs. Convert before sorting by "closing soon", or &lt;code&gt;01/07/2028&lt;/code&gt; sorts before &lt;code&gt;11/17/2026&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Counts move daily.&lt;/strong&gt; My keyword search had 171 matches on 18 Oct and 167 on 25 Oct. Opportunities close, and new ones are posted. Store the opportunity &lt;code&gt;id&lt;/code&gt; and diff, rather than trusting counts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One detail call per result.&lt;/strong&gt; Search is cheap, but anything useful needs &lt;code&gt;fetchOpportunity&lt;/code&gt;. For 400+ results that's 400+ requests. Be gentle and cache.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Contacts are often individuals.&lt;/strong&gt; The agency contact is sometimes a shared inbox, sometimes a named program officer. Use it to ask about the program, not as a mailing list.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 4: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged this as an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/grants-gov-opportunity-finder" rel="noopener noreferrer"&gt;Grants.gov Scraper – Federal &amp;amp; Nonprofit Grants Finder&lt;/a&gt;. It runs the search, fetches every detail, normalises the types and dates, and returns one clean row per opportunity. It also adds &lt;code&gt;daysUntilClose&lt;/code&gt;, and "posted in the last N days" / "closing within N days" filters for alerts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run-ready link:&lt;/strong&gt; &lt;a href="https://console.apify.com/actors/imbcI522UkGM5yadr/input" rel="noopener noreferrer"&gt;open it in Apify Console with an example input already filled in&lt;/a&gt; (&lt;code&gt;{"keyword": "artificial intelligence", "maxItems": 20}&lt;/code&gt;). Click Start, or change the input first.&lt;/p&gt;

&lt;p&gt;The pre-filled input (keyword &lt;code&gt;artificial intelligence&lt;/code&gt;, forecasted + posted, 20 results) ran on 18 Oct 2026 in about 6 seconds and saved 20 of 171 matches:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Number&lt;/th&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;th&gt;Agency&lt;/th&gt;
&lt;th&gt;Deadline&lt;/th&gt;
&lt;th&gt;Award ceiling&lt;/th&gt;
&lt;th&gt;Total funding&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;26-512&lt;/td&gt;
&lt;td&gt;Unlocking Dataset Value for AI-Enabled Scientific Discovery…&lt;/td&gt;
&lt;td&gt;NSF&lt;/td&gt;
&lt;td&gt;2026-11-04&lt;/td&gt;
&lt;td&gt;–&lt;/td&gt;
&lt;td&gt;$100M&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;25-531&lt;/td&gt;
&lt;td&gt;Cybersecurity Innovation for Cyberinfrastructure (CICI)&lt;/td&gt;
&lt;td&gt;NSF&lt;/td&gt;
&lt;td&gt;2027-01-20&lt;/td&gt;
&lt;td&gt;$1.2M&lt;/td&gt;
&lt;td&gt;$12M&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;G27AS00079&lt;/td&gt;
&lt;td&gt;Cooperative Agreement for Affiliated Partner with the Great…&lt;/td&gt;
&lt;td&gt;DOI-USGS1&lt;/td&gt;
&lt;td&gt;2026-11-14&lt;/td&gt;
&lt;td&gt;$150K&lt;/td&gt;
&lt;td&gt;$150K&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The nonprofit version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"statuses"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"posted"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"eligibilities"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"12"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"sortBy"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"closingSoon"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"maxItems"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That ran on 19 Oct 2026 and saved 10 of 443 matching opportunities. One was HRSA-27-005, the FY2027 Health Center Service Area Competition, with an award ceiling of $10,116,100, a floor of $650,000, $267,773,700 total, 63 expected awards, and &lt;code&gt;daysUntilClose: 0&lt;/code&gt;. Sorting by closing date surfaces exactly the ones you're about to miss.&lt;/p&gt;

&lt;p&gt;Each row has: id, number, title, agency, status, open and close dates, days left, CFDA numbers, award ceiling and floor, estimated total funding, expected awards, cost sharing, funding instruments, categories, eligible applicants, description, contact and the grants.gov link.&lt;/p&gt;

&lt;h2&gt;
  
  
  As a grant alert
&lt;/h2&gt;

&lt;p&gt;Schedule a weekly run with "posted in the last 7 days", your eligibility codes and keywords, and push the new rows to email, Slack or a sheet. That replaces scanning the Grants.gov site by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Federal only.&lt;/strong&gt; Grants.gov covers federal opportunities. Foundation, corporate and most state grants aren't there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It returns what Grants.gov publishes.&lt;/strong&gt; If an agency leaves the ceiling or funding blank, so does the row. Forecasts usually have no deadline yet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not a full NOFO.&lt;/strong&gt; Eligibility rules, page limits and review criteria are in the attached documents, which you'll still need to read.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Partial results.&lt;/strong&gt; If Grants.gov errors partway through a large run, the Actor keeps what it saved and stops instead of failing the whole run.&lt;/li&gt;
&lt;li&gt;Not affiliated with Grants.gov.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.003 per opportunity saved&lt;/strong&gt; ($3 per 1,000), no start fee, and it stops at your maximum cost per run. The 20-result test was 6 cents. If you only need one search, the Python above is free and works fine.&lt;/p&gt;

&lt;p&gt;If you look for grants regularly: what do you filter on, and what's missing from Grants.gov's own search? I'd like to know what to add next, so tell me in the comments.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>opendata</category>
      <category>nonprofit</category>
    </item>
    <item>
      <title>Building Healthcare Provider Lists From the Free NPI Registry (and Getting Past Its 1,200-Result Cap)</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Fri, 06 Nov 2026 19:38:48 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/building-healthcare-provider-lists-from-the-free-npi-registry-and-getting-past-its-1200-result-8a</link>
      <guid>https://dev.to/dedicated_crest/building-healthcare-provider-lists-from-the-free-npi-registry-and-getting-past-its-1200-result-8a</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. Everything in the tutorial part works with free tools too.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If you sell to dentists, clinics, chiropractors or therapists, or you need to verify a list of providers, the best free source in the US is the &lt;strong&gt;NPI Registry&lt;/strong&gt; (NPPES), run by CMS. Every HIPAA-covered provider has a National Provider Identifier, about 8 million of them, and the registry has a public API with no key.&lt;/p&gt;

&lt;p&gt;Each record has the NPI, name, credential, taxonomies (specialties), license, practice and mailing addresses with &lt;strong&gt;phone and fax&lt;/strong&gt;, the date the NPI was issued, and for organizations the &lt;strong&gt;authorized official&lt;/strong&gt; (often the owner or CEO) with title and phone.&lt;/p&gt;

&lt;p&gt;What it doesn't have: &lt;strong&gt;email addresses&lt;/strong&gt;. Plan around that from the start.&lt;/p&gt;

&lt;p&gt;If you need &lt;em&gt;everything&lt;/em&gt;, CMS also publishes the full NPPES data dissemination file (monthly, plus weekly updates). It's free and complete, and it's a very large CSV with hundreds of columns. For targeted lists, the API is easier.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Query the API
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://npiregistry.cms.hhs.gov/api/?version=2.1&amp;amp;taxonomy_description=Dentist&amp;amp;city=San Diego&amp;amp;state=CA&amp;amp;limit=200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;limit&lt;/code&gt; is at most 200, and you page with &lt;code&gt;skip&lt;/code&gt;. Here's a pager that also keeps only providers whose &lt;em&gt;primary&lt;/em&gt; taxonomy is a dentist one. It ran as-is on 25 Oct 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;npi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;skip&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://npiregistry.cms.hhs.gov/api/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;2.1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;limit&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;skip&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;skip&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;results&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;skip&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="c1"&gt;# skip is capped at 1000 -&amp;gt; max 1,200 per query
&lt;/span&gt;            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;
        &lt;span class="n"&gt;skip&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;
&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;npi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;taxonomy_description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Dentist&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;city&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;San Diego&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CA&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;primary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;primary&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Dent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;desc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;taxonomies&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;primary&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;with a dentist taxonomy as primary&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1200
1193 with a dentist taxonomy as primary
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exactly 1,200 is suspicious, and it should be.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: The pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Every query stops at 1,200 results.&lt;/strong&gt; &lt;code&gt;skip&lt;/code&gt; is capped at 1000, and a page holds 200. Worse, it fails silently: I asked for &lt;code&gt;skip=1000&lt;/code&gt;, &lt;code&gt;skip=1200&lt;/code&gt; and &lt;code&gt;skip=2000&lt;/code&gt; on a California dentist search, and all three returned the same page. If you just keep paging, you collect duplicates and think you're done. To get a whole city or state, split the search (by city, or by ZIP prefix, and for a state by both) until each piece is under 1,200, then dedupe on NPI.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Specialty matches &lt;em&gt;any&lt;/em&gt; taxonomy.&lt;/strong&gt; A provider with Dentist as a secondary taxonomy still matches. Above, 7 of 1,200 didn't have a dentist taxonomy as primary. For strict lists, check the &lt;code&gt;primary&lt;/code&gt; flag.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two addresses.&lt;/strong&gt; Each record has a &lt;code&gt;MAILING&lt;/code&gt; and a &lt;code&gt;LOCATION&lt;/code&gt; address, and in the record I checked the mailing address came first in the list. Use &lt;code&gt;address_purpose&lt;/code&gt;, not position. City and state searches can match either one, so a "San Diego" search can include providers who only &lt;em&gt;mail&lt;/em&gt; from there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"New practices" isn't a query parameter.&lt;/strong&gt; The API can't filter on the NPI issue date (&lt;code&gt;enumeration_date&lt;/code&gt;). To find newly enumerated providers, you pull and filter client-side.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Self-reported data.&lt;/strong&gt; Phones and fax numbers are what providers entered, and some records haven't been updated in years. &lt;code&gt;last_updated&lt;/code&gt; tells you how stale a record is.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged this as an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/npi-healthcare-provider-leads" rel="noopener noreferrer"&gt;NPI Registry Scraper – NPPES Healthcare Provider Leads&lt;/a&gt;. The main thing it adds is &lt;strong&gt;auto-splitting&lt;/strong&gt;: large searches are split by state and ZIP prefix, so you can pull a whole state past the 1,200 cap. When it splits by state, it keeps providers whose &lt;em&gt;practice&lt;/em&gt; location is in that state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run-ready link:&lt;/strong&gt; &lt;a href="https://console.apify.com/actors/AeZTbhypXKAXC4S18/input" rel="noopener noreferrer"&gt;open it in Apify Console with an example input already filled in&lt;/a&gt; (&lt;code&gt;{"specialties": ["Dentist"], "states": ["CA"], "cities": ["San Diego, CA"], "maxItems": 50}&lt;/code&gt;). Click Start, or change the input first.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"specialties"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Dentist"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"states"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"CA"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"cities"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"San Diego, CA"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"maxItems"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Other inputs: individual vs organization, ZIPs, names, a list of NPIs to verify, and filters for &lt;strong&gt;NPI issued after&lt;/strong&gt;, record updated after, only with phone, and primary specialty must match. Output is deduplicated, with formatted phones and ZIPs and separate practice and mailing addresses.&lt;/p&gt;

&lt;p&gt;That input ran on 18 Oct 2026 in about 18 seconds and returned 50 providers. Some rows:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;NPI&lt;/th&gt;
&lt;th&gt;Name&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Primary specialty&lt;/th&gt;
&lt;th&gt;NPI issued&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1326171653&lt;/td&gt;
&lt;td&gt;Dr. WILLIAM JOSEPH DI ZINNO, DMD&lt;/td&gt;
&lt;td&gt;Individual&lt;/td&gt;
&lt;td&gt;Dentist, General Practice&lt;/td&gt;
&lt;td&gt;2007-03-14&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1699214478&lt;/td&gt;
&lt;td&gt;GLENN C. ASHMORE DDS, INC.&lt;/td&gt;
&lt;td&gt;Organization&lt;/td&gt;
&lt;td&gt;Dentist&lt;/td&gt;
&lt;td&gt;2017-02-14&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1730639188&lt;/td&gt;
&lt;td&gt;A.S. KIM DDS, INC.&lt;/td&gt;
&lt;td&gt;Organization&lt;/td&gt;
&lt;td&gt;Dentist, General Practice&lt;/td&gt;
&lt;td&gt;2016-10-10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1487251294&lt;/td&gt;
&lt;td&gt;Dr. ZEKI ABDULAALI, DDS&lt;/td&gt;
&lt;td&gt;Individual&lt;/td&gt;
&lt;td&gt;Dentist, General Practice&lt;/td&gt;
&lt;td&gt;2020-10-05&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Organization rows are the useful ones for B2B. For &lt;code&gt;A.S. KIM DDS, INC.&lt;/code&gt;, the row has the authorized official (a DDS, title "CEO") with a phone, the practice phone, license number, other names (DBA "VILLAGE HILLCREST DENTAL"), and the count of additional practice locations.&lt;/p&gt;

&lt;p&gt;A bigger real example from my testing: all individual chiropractors in California with a phone came to 3,000 providers in about 3 minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  A monthly "new practices" list
&lt;/h2&gt;

&lt;p&gt;Set &lt;em&gt;NPI issued after&lt;/em&gt; (&lt;code&gt;enumeratedAfter&lt;/code&gt;) to 90 days ago, pick your specialties and states, and schedule it monthly. New clinicians and new group practices show up as they get their NPIs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No emails.&lt;/strong&gt; NPPES doesn't publish them. You get practice phone and fax, the mailing address and, for organizations, the authorized official. If you need an email, you'll need the practice's website.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Self-reported and sometimes stale.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;NPI holders only.&lt;/strong&gt; It covers HIPAA-covered providers, not every wellness or cosmetic business.&lt;/li&gt;
&lt;li&gt;Not affiliated with CMS or HHS.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.001 per provider&lt;/strong&gt; ($1 per 1,000), no start fee. Duplicates and filtered-out records are free, and runs stop at your "maximum cost per run". The 50-provider test was 5 cents. The API and the full CMS file are free.&lt;/p&gt;

&lt;h2&gt;
  
  
  Play fair
&lt;/h2&gt;

&lt;p&gt;Practice phones are business lines, but authorized officials are people. TCPA rules apply to calls and texts, CAN-SPAM to email, and some providers will rightly be annoyed by cold outreach. Be relevant and honour opt-outs.&lt;/p&gt;

&lt;p&gt;Do you work with NPPES data? I'd like to hear how you deal with stale records, so tell me in the comments.&lt;/p&gt;

</description>
      <category>healthcare</category>
      <category>opendata</category>
      <category>python</category>
      <category>api</category>
    </item>
    <item>
      <title>Building a Trucking Company List From FMCSA Open Data (SoQL Pitfalls Included)</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Fri, 06 Nov 2026 17:01:42 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/building-a-trucking-company-list-from-fmcsa-open-data-soql-pitfalls-included-4jo9</link>
      <guid>https://dev.to/dedicated_crest/building-a-trucking-company-list-from-fmcsa-open-data-soql-pitfalls-included-4jo9</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. The tutorial part uses only free public data.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Freight insurance agents, factoring companies, ELD and fuel-card sellers, and brokers looking for capacity all need the same thing: a list of trucking companies with a phone number and fleet size. People pay for those lists, but the source behind most of them is free. Every US interstate carrier files an &lt;strong&gt;MCS-150&lt;/strong&gt; with FMCSA, and US DOT publishes the result as open data, refreshed daily.&lt;/p&gt;

&lt;p&gt;Here's how to query it yourself, and what tripped me up.&lt;/p&gt;

&lt;h2&gt;
  
  
  The datasets
&lt;/h2&gt;

&lt;p&gt;All on data.transportation.gov (Socrata), no key needed for light use:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Company Census File&lt;/strong&gt; (&lt;code&gt;az4n-8mr2&lt;/code&gt;): one row per USDOT number, with legal name, address, phone, email (if filed), power units, drivers, cargo, operation type and status.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;L&amp;amp;I "Carrier – All With History"&lt;/strong&gt; (&lt;code&gt;6eyk-hxee&lt;/code&gt;): MC dockets, authority and insurance.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;L&amp;amp;I "AuthHist – All With History"&lt;/strong&gt; (&lt;code&gt;9mw4-x3tu&lt;/code&gt;): authority grants, revocations and reinstatements.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1: A SoQL query
&lt;/h2&gt;

&lt;p&gt;Small Dallas carriers (5–50 power units), active, with an email on file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-s&lt;/span&gt; &lt;span class="nt"&gt;-G&lt;/span&gt; &lt;span class="s2"&gt;"https://data.transportation.gov/resource/az4n-8mr2.json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\$&lt;/span&gt;&lt;span class="s2"&gt;select=dot_number,legal_name,phone,email_address,power_units,total_drivers"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\$&lt;/span&gt;&lt;span class="s2"&gt;where=phy_state='TX' AND phy_city='DALLAS' AND status_code='A' AND power_units ::number BETWEEN 5 AND 50 AND email_address IS NOT NULL"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\$&lt;/span&gt;&lt;span class="s2"&gt;order=dot_number"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\$&lt;/span&gt;&lt;span class="s2"&gt;limit=5"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run on 25 Oct 2026 (emails left out here):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;dot_number&lt;/th&gt;
&lt;th&gt;legal_name&lt;/th&gt;
&lt;th&gt;phone&lt;/th&gt;
&lt;th&gt;power_units&lt;/th&gt;
&lt;th&gt;total_drivers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;58702&lt;/td&gt;
&lt;td&gt;ECOBAT LOGISTICS US INC&lt;/td&gt;
&lt;td&gt;2145830296&lt;/td&gt;
&lt;td&gt;46&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;59903&lt;/td&gt;
&lt;td&gt;BISHOP INTERNATIONAL INC&lt;/td&gt;
&lt;td&gt;2144266449&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;60111&lt;/td&gt;
&lt;td&gt;DAVIS-HAWN LUMBER COMPANY&lt;/td&gt;
&lt;td&gt;2149460734&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;109485&lt;/td&gt;
&lt;td&gt;TEXAS INSTRUMENTS INCORPORATED&lt;/td&gt;
&lt;td&gt;2145674884&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;164404&lt;/td&gt;
&lt;td&gt;WALNUT HILL WRECKER SERVICEINC&lt;/td&gt;
&lt;td&gt;9726202160&lt;/td&gt;
&lt;td&gt;17&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Step 2: The pitfalls
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Numbers are stored as text.&lt;/strong&gt; My first version used &lt;code&gt;power_units BETWEEN 5 AND 50&lt;/code&gt; and Socrata rejected it with a type mismatch. You need the &lt;code&gt;::number&lt;/code&gt; cast. Without a cast, a comparison like &lt;code&gt;&amp;gt;&lt;/code&gt; on a text column runs as a string comparison, so &lt;code&gt;"9" &amp;gt; "50"&lt;/code&gt; is true.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cities are free text, uppercase.&lt;/strong&gt; &lt;code&gt;phy_city='DALLAS'&lt;/code&gt; works, but a filer who typed a suburb or a typo won't match. For full coverage, filter by ZIP prefix (&lt;code&gt;phy_zip like '752%'&lt;/code&gt;) as well as city.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Active" isn't one thing.&lt;/strong&gt; &lt;code&gt;status_code='A'&lt;/code&gt; is the USDOT status. Whether a company can legally haul freight for hire is a separate question, the &lt;strong&gt;operating authority&lt;/strong&gt; (MC number), and that lives in the L&amp;amp;I datasets. The census also has private fleets: Texas Instruments and a lumber company are in my list because they run their own trucks. If you're selling to for-hire carriers, filter on the classification and join authority status.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The join is the slow part.&lt;/strong&gt; Authority history is per docket, with multiple grant, revocation and reinstatement rows. You have to fetch the dockets for your candidate USDOT numbers and work out the current state. My Actor's test run (below) used the same city and fleet filters plus "active authority", and it left out Bishop International and Davis-Hawn Lumber, which my raw query includes. So the census alone and census + authority give different lists.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Formatting.&lt;/strong&gt; Phones come back as raw digits, and names like &lt;code&gt;SERVICEINC&lt;/code&gt; are as filed. Email exists only when the company put one on its MCS-150.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged this as an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/fmcsa-trucking-carrier-leads" rel="noopener noreferrer"&gt;FMCSA Carrier Leads – Trucking Companies &amp;amp; Freight Brokers List&lt;/a&gt;. It builds the SoQL for you, joins the census with both L&amp;amp;I datasets, and returns one clean row per company.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run-ready link:&lt;/strong&gt; &lt;a href="https://console.apify.com/actors/OYGIEL4x0GmbUh4rc/input" rel="noopener noreferrer"&gt;open it in Apify Console with an example input already filled in&lt;/a&gt; (&lt;code&gt;{"states": ["TX"], "cities": ["Dallas, TX"], "maxItems": 25}&lt;/code&gt;). Click Start, or change the input first.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"states"&lt;/span&gt;&lt;span class="p"&gt;:[&lt;/span&gt;&lt;span class="s2"&gt;"TX"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="nl"&gt;"cities"&lt;/span&gt;&lt;span class="p"&gt;:[&lt;/span&gt;&lt;span class="s2"&gt;"Dallas, TX"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="nl"&gt;"entityType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"carrier"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"minPowerUnits"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"maxPowerUnits"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"authorityStatus"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"active"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"requireEmail"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"maxItems"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Filters: states, cities, ZIP prefixes, USDOT or MC numbers, name contains, entity type (carrier, broker, both, freight forwarder, shipper), operation type, classification (for-hire, private…), USDOT status, authority status, power units and driver ranges, cargo types, hazmat, safety rating, USDOT add date, and "only with email".&lt;/p&gt;

&lt;p&gt;That input ran on Apify on 22 Oct 2026 and returned 10 carriers. A few rows (emails omitted):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;USDOT&lt;/th&gt;
&lt;th&gt;MC&lt;/th&gt;
&lt;th&gt;Legal name&lt;/th&gt;
&lt;th&gt;Phone&lt;/th&gt;
&lt;th&gt;Power units&lt;/th&gt;
&lt;th&gt;Safety rating&lt;/th&gt;
&lt;th&gt;First authority&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;58702&lt;/td&gt;
&lt;td&gt;MC1250042&lt;/td&gt;
&lt;td&gt;ECOBAT LOGISTICS US INC&lt;/td&gt;
&lt;td&gt;(214) 583-0296&lt;/td&gt;
&lt;td&gt;46&lt;/td&gt;
&lt;td&gt;Satisfactory&lt;/td&gt;
&lt;td&gt;2021-06-10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;164404&lt;/td&gt;
&lt;td&gt;MC142549&lt;/td&gt;
&lt;td&gt;WALNUT HILL WRECKER SERVICEINC&lt;/td&gt;
&lt;td&gt;(972) 620-2160&lt;/td&gt;
&lt;td&gt;17&lt;/td&gt;
&lt;td&gt;Satisfactory&lt;/td&gt;
&lt;td&gt;1978-01-25&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;275346&lt;/td&gt;
&lt;td&gt;MC193564&lt;/td&gt;
&lt;td&gt;JACK KEY AUTO TRANSPORT INC&lt;/td&gt;
&lt;td&gt;(972) 557-8210&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;Satisfactory&lt;/td&gt;
&lt;td&gt;1990-12-21&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each record has formatted phones, the full address, company officers, fleet and driver counts, MCS-150 mileage, cargo types, safety rating and review dates, common/contract/broker authority, pending and revocation flags, BIPD insurance on file vs required, and the authority history. For Ecobat, the history shows the authority &lt;strong&gt;revoked on 2022-12-19 and reinstated on 2022-12-23&lt;/strong&gt;, the kind of detail that matters if you're vetting. Every record also links to the SAFER snapshot. A &lt;code&gt;SUMMARY&lt;/code&gt; record shows records scanned, API calls and the exact SoQL filter used, so you can reproduce it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Recipes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;New carriers for insurance/factoring&lt;/strong&gt;: USDOT added in the last 90 days, for-hire, 1–10 power units, your states, with email. Schedule it weekly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Capacity sourcing&lt;/strong&gt;: cargo type "Refrigerated Food", 5–50 power units, Satisfactory rating, active authority.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Broker list&lt;/strong&gt;: entity type broker, active broker authority, by state.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Limits, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It's only as good as the MCS-150.&lt;/strong&gt; Small carriers often file infrequently, so phones, emails and fleet sizes can be stale. Each record has &lt;code&gt;mcs150Date&lt;/code&gt; so you can judge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Email is often missing.&lt;/strong&gt; "Only with email" shrinks lists a lot.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The authority filter is slow&lt;/strong&gt; because it runs after the census query. Narrow by state, city or fleet size for big searches.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No inspection or crash detail (SMS/CSA scores).&lt;/strong&gt; Only the safety rating and review dates from the census.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compliance is on you.&lt;/strong&gt; It's public government data, but TCPA and CAN-SPAM still apply when you call or email.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.002 per company saved&lt;/strong&gt; ($2 per 1,000). No start fee, and the run stops cleanly at your maximum charge. The 10-row test cost 2 cents. For a one-off pull of one city, the curl above is free.&lt;/p&gt;

&lt;p&gt;If you work in freight: what do you actually filter on when you build a carrier list? Fleet size, cargo, authority age? I'd like to know which filters matter in practice, so tell me in the comments.&lt;/p&gt;

</description>
      <category>opendata</category>
      <category>api</category>
      <category>sql</category>
      <category>logistics</category>
    </item>
    <item>
      <title>Scraping Capterra Reviews Without Logging In: Next.js Page Data, Cloudflare and the 375-Review Wall</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Thu, 05 Nov 2026 15:05:14 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/scraping-capterra-reviews-without-logging-in-nextjs-page-data-cloudflare-and-the-375-review-wall-3fla</link>
      <guid>https://dev.to/dedicated_crest/scraping-capterra-reviews-without-logging-in-nextjs-page-data-cloudflare-and-the-375-review-wall-3fla</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned at the end. I charge per review for it. The how-it-works part applies whatever tool you use.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I wanted competitor reviews in a spreadsheet: star rating, the four sub-scores, pros, cons, the reviewer's job title, and the date. Capterra has all of that on public pages, but getting it out cleanly took a few tries. Here's what I found.&lt;/p&gt;

&lt;h2&gt;
  
  
  The reviews are in the Next.js page data
&lt;/h2&gt;

&lt;p&gt;Capterra's review pages (&lt;code&gt;/p/&amp;lt;id&amp;gt;/&amp;lt;Product&amp;gt;/reviews/&lt;/code&gt;) are a Next.js app. You don't have to parse the layout. The page ships its data in &lt;code&gt;self.__next_f.push([...])&lt;/code&gt; script calls, and if you join the string parts you get the React Server Components payload. Search it for &lt;code&gt;"hybridReviewPage":[&lt;/code&gt; and bracket-match the array. Each of the 25 entries looks roughly like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"kind"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"primary"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"publishedAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-11-02T05:27:06Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"overallRating"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"ratingDetails"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"easeOfUse"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"functionality"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"qualityOfSupport"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"valueForMoney"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"recommendation"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"pros"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"cons"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"generalComments"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"reviewer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"fullName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Hannah M."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"jobTitle"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Creative arts coordinator"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"timeUsedProduct"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"3-5 years"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Next to it there's a &lt;code&gt;reviewSummary&lt;/code&gt; object with the product's overall rating, total review count and the four average sub-scores. For Slack on Nov 2, 2026 that was 4.7 overall from 24,177 reviews.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ask for the data, not the page
&lt;/h2&gt;

&lt;p&gt;Once you have a browser session, you can request the next page with the header &lt;code&gt;RSC: 1&lt;/code&gt; and Next.js sends back just the server components payload instead of the full HTML. For a Slack reviews page that came to about 217 KB of data. When you pay for residential proxy by the GB, skipping the HTML, scripts and images around it adds up fast.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;?page=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;n&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;credentials&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;include&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;RSC&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="c1"&gt;// same "hybridReviewPage" array, no HTML around it&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Cloudflare
&lt;/h2&gt;

&lt;p&gt;A plain request from a server gets Cloudflare's "Just a moment..." page with a 403. What worked:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A stealth-patched Firefox via Playwright on a US residential proxy.&lt;/li&gt;
&lt;li&gt;Open the first reviews page and wait until the title is the real page title. Careful: while the check runs, the title can be "Loading &lt;a href="https://www.capterra.com/.." rel="noopener noreferrer"&gt;https://www.capterra.com/..&lt;/a&gt;.", which contains "capterra". My first version matched on that, thought it was through, and every fetch after it failed.&lt;/li&gt;
&lt;li&gt;After that, fetch the other pages from inside the same tab with &lt;code&gt;RSC: 1&lt;/code&gt;, one at a time with a short gap.&lt;/li&gt;
&lt;li&gt;Block images, fonts, ads and trackers, but let Cloudflare's own challenge files through.&lt;/li&gt;
&lt;li&gt;If the check doesn't clear, throw the session away and start again with a new IP.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The 375-review wall
&lt;/h2&gt;

&lt;p&gt;This one surprised me. &lt;code&gt;?page=N&lt;/code&gt; works, but only up to page 15. Page 16 comes back with an empty review list, even for Slack with its 24,177 reviews. And if you ask for a page way past the end, like &lt;code&gt;?page=99999&lt;/code&gt;, you get page 1 again, so check for repeated review IDs or you'll loop.&lt;/p&gt;

&lt;p&gt;So a logged-out visitor sees 15 pages of 25, which is 375 reviews per product in Capterra's default order. Most products have fewer than that, so you get everything. For the big ones you get the first 375. I didn't try to get around it, and I stay out of &lt;code&gt;/search&lt;/code&gt; and the sort parameters, which Capterra's robots.txt disallows.&lt;/p&gt;

&lt;h2&gt;
  
  
  Real numbers
&lt;/h2&gt;

&lt;p&gt;From my test runs on Nov 2, 2026:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;50 Slack reviews (2 pages): about $0.056 in platform cost, most of it the browser start and the Cloudflare check.&lt;/li&gt;
&lt;li&gt;All 15 pages of Discord (375 reviews scanned) with a 2-star filter: about $0.031 in total, under 48 seconds.&lt;/li&gt;
&lt;li&gt;Filtering helps: in those 375 Discord reviews, 7 had 2 stars or fewer and 5 had 1 star.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  If you don't want to build it
&lt;/h2&gt;

&lt;p&gt;I packaged this as an Apify Actor: &lt;a href="https://apify.com/dedicated_crest/capterra-reviews-scraper" rel="noopener noreferrer"&gt;Capterra Reviews Scraper&lt;/a&gt;. Paste product URLs, optionally set a star range or a "reviewed after" date, and download JSON, CSV or Excel. It's $1 per 1,000 reviews plus a $0.005 start fee per run, proxy included. Reviews your filters drop aren't charged.&lt;/p&gt;

&lt;p&gt;If Capterra changes something and it breaks, open an issue on the Actor page and I'll take a look.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>javascript</category>
      <category>playwright</category>
      <category>saas</category>
    </item>
    <item>
      <title>Scraping Public Quora Questions and Answers Without Logging In (What Works and What Doesn't)</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Wed, 04 Nov 2026 22:04:02 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/scraping-public-quora-questions-and-answers-without-logging-in-what-works-and-what-doesnt-49ai</link>
      <guid>https://dev.to/dedicated_crest/scraping-public-quora-questions-and-answers-without-logging-in-what-works-and-what-doesnt-49ai</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned at the end. I charge per result for it. The how-it-works part applies whatever tool you use.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I wanted a simple thing: give it a Quora question, get back the question, the top answers with their view and upvote counts, and the related questions. No account, no cookies. It turned out to be more fiddly than most sites I've scraped, so here's what I learned.&lt;/p&gt;

&lt;h2&gt;
  
  
  The data is already in the page
&lt;/h2&gt;

&lt;p&gt;You don't need to parse Quora's HTML layout. Every question, answer and profile page ships its data as JSON strings inside script tags, pushed onto an array like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="p"&gt;....&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;{&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;data&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;:{&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;question&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;:{...}}}&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each string is JSON encoded twice, so you &lt;code&gt;JSON.parse&lt;/code&gt; it twice. Inside you'll find objects with a &lt;code&gt;__typename&lt;/code&gt; of &lt;code&gt;Question&lt;/code&gt;, &lt;code&gt;Answer&lt;/code&gt; or &lt;code&gt;User&lt;/code&gt;. Walk the whole tree, group by &lt;code&gt;qid&lt;/code&gt;, &lt;code&gt;aid&lt;/code&gt; and &lt;code&gt;uid&lt;/code&gt;, and merge the partial copies you see along the way. That gets you things that aren't easy to read off the screen:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;answers: &lt;code&gt;numViews&lt;/code&gt;, &lt;code&gt;numUpvotes&lt;/code&gt;, &lt;code&gt;numShares&lt;/code&gt;, comment count, creation time (in microseconds, careful), the author's credential, and even an &lt;code&gt;isMachineAnswer&lt;/code&gt; flag&lt;/li&gt;
&lt;li&gt;questions: answer count, follower count, topics, related questions&lt;/li&gt;
&lt;li&gt;profiles: followers, number of answers, all-time content views, views last month, "Most viewed writer" years&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Answer text is stored as a small rich text document (&lt;code&gt;sections&lt;/code&gt; of &lt;code&gt;spans&lt;/code&gt;), so you join the span texts to get plain text. Skip anything whose typename ends in &lt;code&gt;Ad&lt;/code&gt;, or you'll scrape promoted content too.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hard part: Cloudflare
&lt;/h2&gt;

&lt;p&gt;A plain &lt;code&gt;fetch&lt;/code&gt; or &lt;code&gt;curl&lt;/code&gt; from a server gets the "Just a moment..." Cloudflare page. Datacenter IPs basically never get through. What worked for me:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A real (stealth-patched) Firefox via Playwright, on a US residential proxy.&lt;/li&gt;
&lt;li&gt;Open the first page normally and wait until the challenge clears and the &lt;code&gt;cf_clearance&lt;/code&gt; cookie shows up.&lt;/li&gt;
&lt;li&gt;After that, &lt;em&gt;don't&lt;/em&gt; keep navigating. Call &lt;code&gt;fetch()&lt;/code&gt; for the next page from inside that same page, with the session's cookies. You get the full HTML with the JSON blobs, and you skip loading Quora's JS bundles, images and ads.&lt;/li&gt;
&lt;li&gt;Block everything except the document, XHR and Cloudflare's own challenge files. Residential proxy is billed per GB, so this matters.&lt;/li&gt;
&lt;li&gt;If a page still gets blocked, throw the session away and start a new one with a new IP. Give up on that page after three tries.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Go slowly, one page at a time with a short gap. That's kinder to Quora and also what keeps the session alive.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits you can't get around (logged out)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Only a few answers per question.&lt;/strong&gt; A logged-out visitor sees the first handful, usually 2 to 6. On "How do I learn Python?" (198 answers) I got 4. Some of those are pulled in from a very similar question, so check each answer's own question ID; I flag those as &lt;code&gt;fromSimilarQuestion&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No keyword search.&lt;/strong&gt; &lt;code&gt;/search&lt;/code&gt; is disallowed in Quora's robots.txt, so I stay out of it. Instead you start from question URLs and follow the related questions list to grow a topic dataset.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't use it for AI training.&lt;/strong&gt; Quora's terms say no to that without a license. Market research, content ideas, finding experts: fine, with the usual care for authors' copyright.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Real numbers
&lt;/h2&gt;

&lt;p&gt;Some records from my test runs on Nov 1, 2026:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Record&lt;/th&gt;
&lt;th&gt;Numbers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Top answer on "How do I learn Python?"&lt;/td&gt;
&lt;td&gt;921,089 views, 3,160 upvotes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;An answer on "What are the best ways to save money?"&lt;/td&gt;
&lt;td&gt;449,994 views, 4,904 upvotes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Jimmy Wales' profile&lt;/td&gt;
&lt;td&gt;387,617 followers, 1,003 answers, 64,155,244 content views&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;My last QA run (1 seed question with related questions on, 4 question pages, 1 profile) saved 20 records (4 questions, 15 answers, 1 profile) in about 71 seconds. Across my test runs the platform cost (browser + residential proxy) came out around $0.0012 per record, almost all of it proxy traffic and compute for the browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you don't want to build it
&lt;/h2&gt;

&lt;p&gt;I packaged all of this as an Apify Actor: &lt;a href="https://apify.com/dedicated_crest/quora-questions-answers-scraper" rel="noopener noreferrer"&gt;Quora Scraper: Questions, Answers and Profiles&lt;/a&gt;. Paste question or profile URLs, optionally turn on "Follow related questions", and download JSON/CSV/Excel. It's $0.0019 per record with the proxy included, and you can set a max cost per run so it stops cleanly.&lt;/p&gt;

&lt;p&gt;If something breaks (Quora changes things often), open an issue on the Actor page and I'll take a look.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>javascript</category>
      <category>playwright</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Reading Medium Articles by Tag, Author and Publication With RSS (and Where It Stops)</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Wed, 04 Nov 2026 18:04:56 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/reading-medium-articles-by-tag-author-and-publication-with-rss-and-where-it-stops-1hp</link>
      <guid>https://dev.to/dedicated_crest/reading-medium-articles-by-tag-author-and-publication-with-rss-and-where-it-stops-1hp</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned at the end of this post. I charge per result for it. Everything in the tutorial part works with free tools too.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Medium has no public API for reading articles. What it does have is a set of RSS feeds that work without a login or an API key. If you want to track a topic, follow a few writers, or watch a publication, they get you most of the way there, as long as you know what's missing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three feeds
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://medium.com/feed/tag/{tag}            # latest articles in a tag
https://medium.com/feed/@{username}          # one author's latest stories
https://medium.com/feed/{publication-slug}   # one publication's latest posts
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The tag is the slug from &lt;code&gt;medium.com/tag/...&lt;/code&gt; (for example &lt;code&gt;programming&lt;/code&gt; or &lt;code&gt;artificial-intelligence&lt;/code&gt;). The publication slug is the part after &lt;code&gt;medium.com/&lt;/code&gt;, such as &lt;code&gt;better-programming&lt;/code&gt;. Publications with a custom domain still answer on the &lt;code&gt;medium.com/feed/{slug}&lt;/code&gt; URL.&lt;/p&gt;

&lt;p&gt;Each &lt;code&gt;&amp;lt;item&amp;gt;&lt;/code&gt; gives you the title, link, a &lt;code&gt;guid&lt;/code&gt; of the form &lt;code&gt;https://medium.com/p/{postId}&lt;/code&gt;, the author in &lt;code&gt;dc:creator&lt;/code&gt;, &lt;code&gt;pubDate&lt;/code&gt;, &lt;code&gt;atom:updated&lt;/code&gt; and the tags as &lt;code&gt;&amp;lt;category&amp;gt;&lt;/code&gt; elements.&lt;/p&gt;

&lt;h2&gt;
  
  
  What each feed actually contains
&lt;/h2&gt;

&lt;p&gt;I pulled one of each on 31 Oct 2026:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Feed&lt;/th&gt;
&lt;th&gt;Items&lt;/th&gt;
&lt;th&gt;Items with full text&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;tag/programming&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;@ev&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;better-programming&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That table covers the three things you need to know:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Every feed returns 10 items.&lt;/strong&gt; There's no pagination: adding &lt;code&gt;?page=2&lt;/code&gt; just gives you the same 10 again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tag feeds never include the body.&lt;/strong&gt; You get a &lt;code&gt;medium-feed-snippet&lt;/code&gt; paragraph in &lt;code&gt;&amp;lt;description&amp;gt;&lt;/code&gt; (about 140 characters, cut off with "â€¦") and a "Continue reading on {Publication}" link. Use that link text to get the publication name.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Author and publication feeds include the full HTML&lt;/strong&gt; in &lt;code&gt;content:encoded&lt;/code&gt;, &lt;em&gt;except&lt;/em&gt; for member-only stories. Those come through with only the snippet, which is why 6 of the 10 Better Programming items had no body. In practice, "author or publication item without &lt;code&gt;content:encoded&lt;/code&gt;" is a good member-only signal.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A few smaller details:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An unknown username returns &lt;strong&gt;HTTP 404&lt;/strong&gt;, so you can tell a typo apart from an empty feed.&lt;/li&gt;
&lt;li&gt;Links carry a &lt;code&gt;?source=rss-...&lt;/code&gt; tracking parameter. Strip it before you dedupe.&lt;/li&gt;
&lt;li&gt;Full-text bodies include a 1Ã—1 tracking image (&lt;code&gt;medium.com/_/stat?...&lt;/code&gt;). Skip it when you pick a cover image.&lt;/li&gt;
&lt;li&gt;Feed items have &lt;strong&gt;no clap or response counts&lt;/strong&gt;. Those are only on the article page and Medium's internal GraphQL endpoint, and both sit behind Cloudflare bot protection. From a server, I got &lt;code&gt;403 "Just a moment..."&lt;/code&gt; on every request from datacenter IPs and only occasional success through residential proxies. Treat claps as best effort.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A parser you can run today (Python, standard library only)
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;xml.etree.ElementTree&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;ET&lt;/span&gt;

&lt;span class="n"&gt;NS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://purl.org/dc/elements/1.1/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://purl.org/rss/1.0/modules/content/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;UA&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0 (compatible; medium-feed-reader/0.1)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;feed_url&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tag&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://medium.com/feed/tag/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;author&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://medium.com/feed/@&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;@&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;publication&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://medium.com/feed/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}[&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;text_of&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;(figure|script|style)[\s\S]*?&amp;lt;/\1&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\s+&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;unescape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;[^&amp;gt;]+&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;))).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;read_feed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;feed_url&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;UA&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;root&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ET&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fromstring&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;item&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findtext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content:encoded&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;namespaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;NS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;desc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findtext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;description&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;snippet&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;medium-feed-snippet&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;gt;(.*?)&amp;lt;/p&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;desc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;text_of&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findtext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;guid&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;rsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/p/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findtext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findtext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;link&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;?source=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;author&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findtext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dc:creator&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;namespaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;NS&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;published&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findtext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pubDate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tags&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;category&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)],&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;summary&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;text_of&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;snippet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;snippet&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;280&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tag&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;programming&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;author&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ev&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;publication&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;better-programming&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)]:&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;read_feed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;words&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output on 31 Oct 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tag programming 10 0 7 Python Automation Projects That Save Hours Every Week
author ev 10 10 Making â€œSocialâ€ Social Again
publication better-programming 10 4 Let a thousand programming publications bloom.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a rough reading time, divide &lt;code&gt;words&lt;/code&gt; by 265 (an average adult reading speed in words per minute) and round up.&lt;/p&gt;

&lt;h3&gt;
  
  
  The same idea in JavaScript (Node 18+)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://medium.com/feed/@ev&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;User-Agent&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;medium-feed-reader/0.1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;xml&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;pick&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;RegExp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&amp;lt;&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;[^&amp;gt;]*&amp;gt;([&lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="s2"&gt;s&lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="s2"&gt;S]*?)&amp;lt;/&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;gt;`&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="p"&gt;[])[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]?.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;lt;!&lt;/span&gt;&lt;span class="se"&gt;\[&lt;/span&gt;&lt;span class="sr"&gt;CDATA&lt;/span&gt;&lt;span class="se"&gt;\[&lt;/span&gt;&lt;span class="sr"&gt;|&lt;/span&gt;&lt;span class="se"&gt;\]\]&lt;/span&gt;&lt;span class="sr"&gt;&amp;gt;/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;items&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;xml&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;&amp;lt;item&amp;gt;&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;it&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;pick&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;it&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;title&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;pick&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;it&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;link&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)?.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;?source=&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
  &lt;span class="na"&gt;author&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;pick&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;it&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;dc:creator&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="na"&gt;tags&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[...&lt;/span&gt;&lt;span class="nx"&gt;it&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;matchAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&amp;lt;category&amp;gt;&amp;lt;!&lt;/span&gt;&lt;span class="se"&gt;\[&lt;/span&gt;&lt;span class="sr"&gt;CDATA&lt;/span&gt;&lt;span class="se"&gt;\[(&lt;/span&gt;&lt;span class="sr"&gt;.*&lt;/span&gt;&lt;span class="se"&gt;?)\]\]&lt;/span&gt;&lt;span class="sr"&gt;&amp;gt;&amp;lt;&lt;/span&gt;&lt;span class="se"&gt;\/&lt;/span&gt;&lt;span class="sr"&gt;category&amp;gt;/g&lt;/span&gt;&lt;span class="p"&gt;)].&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt;
  &lt;span class="na"&gt;hasFullText&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;it&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;&amp;lt;content:encoded&amp;gt;&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;}));&lt;/span&gt;
&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;items&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;items&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Getting more than 10 articles
&lt;/h2&gt;

&lt;p&gt;Since the feeds don't page, there are two honest options:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Widen&lt;/strong&gt;: more tags, authors and publications per run. The same story often shows up in several, so dedupe on the post ID from &lt;code&gt;guid&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Poll&lt;/strong&gt;: run it on a schedule (hourly for busy tags, daily for authors) and keep every ID you've seen. A history builds up quickly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Be polite: one request every second or so is plenty, back off on 429 and 5xx, and send a real User-Agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you don't want to maintain it
&lt;/h2&gt;

&lt;p&gt;I wrapped this in an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/medium-articles-scraper" rel="noopener noreferrer"&gt;Medium Scraper: Articles by Tag, Author and Publication&lt;/a&gt;. It takes tags, usernames, publication slugs and article URLs, dedupes across them, and outputs one clean record per article: title, URL, author, publication, dates, tags, summary or full text, word count, reading time and a member-only flag. Claps and responses are an opt-in, best-effort extra. It costs &lt;strong&gt;$0.001 per article&lt;/strong&gt;, and you can set a maximum cost per run. My QA run (one tag, one author, one publication) returned 30 articles in about 7 seconds. It has the same 10-per-feed limit described above, because that's Medium's limit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run-ready link:&lt;/strong&gt; &lt;a href="https://console.apify.com/actors/i2aJxAc59YbSECM47/input" rel="noopener noreferrer"&gt;open it in Apify Console with an example input already filled in&lt;/a&gt; (&lt;code&gt;{"tags": ["artificial-intelligence"], "maxItems": 30}&lt;/code&gt;). Click Start, or change the input first.&lt;/p&gt;

&lt;p&gt;What do you track on Medium: topics, writers, or competitors' publications? Tell me in the comments.&lt;/p&gt;

</description>
      <category>python</category>
      <category>rss</category>
      <category>webscraping</category>
      <category>javascript</category>
    </item>
    <item>
      <title>Who Just Won Federal Contracts in Your NAICS? Querying the USAspending.gov API</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Wed, 04 Nov 2026 13:25:00 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/who-just-won-federal-contracts-in-your-naics-querying-the-usaspendinggov-api-3ih7</link>
      <guid>https://dev.to/dedicated_crest/who-just-won-federal-contracts-in-your-naics-querying-the-usaspendinggov-api-3ih7</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. Everything in the tutorial part works with free tools too.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Every US federal contract and grant award ends up on &lt;strong&gt;USAspending.gov&lt;/strong&gt;, Treasury's public spending database. If you do GovCon business development, look for teaming partners, track competitors, or sell to companies that just received federal money, it's the place to look. It shows &lt;strong&gt;who won&lt;/strong&gt;, how much, from which office, under which NAICS and contract vehicle.&lt;/p&gt;

&lt;p&gt;(It isn't where you find &lt;em&gt;open&lt;/em&gt; solicitations. Those are on SAM.gov. USAspending is awards that were already made.)&lt;/p&gt;

&lt;p&gt;The website's Award Search is free and has CSV downloads, and for one-off questions it's all you need. When you want the same filtered list every week, the API is better. It's public and needs no key.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Search awards with the API
&lt;/h2&gt;

&lt;p&gt;This returns the three largest contract awards matching "artificial intelligence" with activity in a 30-day window. It ran as-is on 25 Oct 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-s&lt;/span&gt; &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.usaspending.gov/api/v2/search/spending_by_award/"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{
    "filters": {
      "award_type_codes": ["A", "B", "C", "D"],
      "keywords": ["artificial intelligence"],
      "time_period": [{"start_date": "2026-09-25", "end_date": "2026-10-24"}]
    },
    "fields": ["Award ID", "Recipient Name", "Award Amount", "Awarding Agency"],
    "sort": "Award Amount", "order": "desc", "limit": 3
  }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Top results:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Award ID&lt;/th&gt;
&lt;th&gt;Recipient&lt;/th&gt;
&lt;th&gt;Amount&lt;/th&gt;
&lt;th&gt;Awarding agency&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;47QFHA26F0005&lt;/td&gt;
&lt;td&gt;BOOZ ALLEN HAMILTON INC&lt;/td&gt;
&lt;td&gt;$30.72M&lt;/td&gt;
&lt;td&gt;General Services Administration&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;75FCMC23F0001&lt;/td&gt;
&lt;td&gt;INDEX ANALYTICS LLC&lt;/td&gt;
&lt;td&gt;$22.94M&lt;/td&gt;
&lt;td&gt;Department of Health and Human Services&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;70B04C26F00001201&lt;/td&gt;
&lt;td&gt;BOOZ ALLEN HAMILTON INC&lt;/td&gt;
&lt;td&gt;$14.18M&lt;/td&gt;
&lt;td&gt;Department of Homeland Security&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each result has a &lt;code&gt;generated_internal_id&lt;/code&gt;. Pass it to &lt;code&gt;GET /api/v2/awards/{generated_internal_id}/&lt;/code&gt; for the detail record: recipient UEI and parent company, number of offers, set-aside, parent award (the IDIQ or GWAC it was ordered under), subaward count and period of performance. For the first Booz Allen award, that call returned UEI &lt;code&gt;JCBMLGPE6Z71&lt;/code&gt;, parent "BOOZ ALLEN HAMILTON HOLDING CORPORATION", 3 offers received, parent award &lt;code&gt;47QFHA24D0005&lt;/code&gt;, and a performance period from 2026-09-15 with a potential end date of 2030-09-14.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: The pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One award-type group per request.&lt;/strong&gt; Contracts (&lt;code&gt;A&lt;/code&gt;–&lt;code&gt;D&lt;/code&gt;), IDVs, grants, loans, direct payments and other assistance are separate groups. Mix &lt;code&gt;A&lt;/code&gt; with grant code &lt;code&gt;02&lt;/code&gt; and the API replies: &lt;em&gt;"'award_type_codes' must only contain types from one group."&lt;/em&gt; Run one search per group and merge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"In the last 30 days" isn't "new".&lt;/strong&gt; &lt;code&gt;time_period&lt;/code&gt; matches awards with activity in the window. That's why the Index Analytics award above, a 2023 order, shows up in a September–October 2026 search. Modifications and option exercises count as activity. If you want new wins only, filter on the award's start or signed date.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Amount vs ceiling.&lt;/strong&gt; &lt;code&gt;Award Amount&lt;/code&gt; is obligations so far. The potential value ("base and all options") can be far larger. In one real award I pulled on 17 Oct, Accenture Federal Services' CDAO delivery order, the amount was $36.7M while base-and-all-options was $821.3M.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DoD data lags.&lt;/strong&gt; Defense contract data has a 90-day publication delay, so recent DoD wins won't be there yet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The detail record is one call per award.&lt;/strong&gt; For a list of 500 awards, that's 500 more requests. Add pacing and retries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Agency names must match&lt;/strong&gt; USAspending's top-tier names exactly ("Department of Defense", not "DoD").&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Result ceilings.&lt;/strong&gt; A single search returns roughly 10,000 awards per award type. Split big pulls by date range.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged this into an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/usaspending-federal-awards" rel="noopener noreferrer"&gt;USAspending Scraper – Federal Contracts &amp;amp; Awards Data&lt;/a&gt;. It runs one search per award type you pick (contracts, IDVs, grants, loans, direct payments, other assistance), paginates, joins in the detail record for every award (on by default), and flattens it all into one row.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run-ready link:&lt;/strong&gt; &lt;a href="https://console.apify.com/actors/1gBWYbXAbdVPHOBz7/input" rel="noopener noreferrer"&gt;open it in Apify Console with an example input already filled in&lt;/a&gt; (&lt;code&gt;{"awardTypes": ["contracts"], "keywords": ["artificial intelligence"], "daysBack": 30, "maxItems": 25}&lt;/code&gt;). Click Start, or change the input first.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"awardTypes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"contracts"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"naicsCodes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"541512"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"setAsideTypes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"SBA"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"daysBack"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"dateType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"new_awards_only"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"maxItems"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Filters: keywords, recipient name or UEI, awarding agency, NAICS, PSC, place-of-performance state, recipient state, set-aside (SBA, 8(a), HUBZone, SDVOSB, WOSB…), min/max amount, last N days or a custom range, and the date type (action date, signed date, last modified, or &lt;strong&gt;new awards only&lt;/strong&gt;).&lt;/p&gt;

&lt;p&gt;That Accenture award is a good example of what one row holds (trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"awardId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"47QFCA26F0023"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"awardType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"DELIVERY ORDER"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"amount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;36709970&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"baseAndAllOptions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;821270642&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"actionDate"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-06-25"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"lastModified"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-09-30 17:02:48"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"awardingSubAgency"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Federal Acquisition Service"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"fundingAgency"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Department of Defense"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"recipientName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ACCENTURE FEDERAL SERVICES LLC"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"recipientUei"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"C47BNA8GM833"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"parentRecipientName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ACCENTURE PLC"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"naicsCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"541512"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"extentCompeted"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"FULL AND OPEN COMPETITION"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"numberOfOffers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"solicitationId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"47QFCA26R0014"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"parentAwardType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"GWAC"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"subawardCount"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;17&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the dates: action date in June, last modified on 30 Sept. That's how a June award lands in a "last 30 days" search on 18 Oct, and why &lt;code&gt;new_awards_only&lt;/code&gt; exists.&lt;/p&gt;

&lt;p&gt;The default test (contracts, "artificial intelligence", last 30 days, 25 results, sorted by amount) ran on 18 Oct 2026 in about 6 seconds and returned 25 awards totalling $190.0M. Each run also writes a SUMMARY record with the filters, matching counts by type, items saved and total dollars.&lt;/p&gt;

&lt;h2&gt;
  
  
  Uses I had in mind
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Teaming and subcontracting:&lt;/strong&gt; primes winning work in your NAICS and state&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Competitor tracking:&lt;/strong&gt; a weekly run filtered by a competitor's UEI&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;B2B sales:&lt;/strong&gt; companies that just received federal money, with address, parent company and business categories&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Limits, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Awards only.&lt;/strong&gt; No open opportunities (SAM.gov's opportunity API needs a key).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;As reported by agencies&lt;/strong&gt; to FPDS/FABS. Contract actions usually appear within days, except DoD's 90-day delay.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;About 10,000 awards per type per search.&lt;/strong&gt; Split by date for more.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No contact people.&lt;/strong&gt; You get the recipient's address and UEI, not a named BD contact.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.002 per award&lt;/strong&gt; ($2 per 1,000), detail enrichment included, no start fee. Runs stop at your "maximum cost per run". The 25-award test was 5 cents. The free website search and the API above cost nothing.&lt;/p&gt;

&lt;p&gt;GovCon folks: which field do you look at first when you size up a competitor's win? Tell me in the comments, I'd like to know what's missing.&lt;/p&gt;

</description>
      <category>opendata</category>
      <category>api</category>
      <category>govtech</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Turn HN's "Who is hiring?" Thread Into a Spreadsheet With the Algolia API</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Tue, 03 Nov 2026 15:01:58 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/turn-hns-who-is-hiring-thread-into-a-spreadsheet-with-the-algolia-api-3kpd</link>
      <guid>https://dev.to/dedicated_crest/turn-hns-who-is-hiring-thread-into-a-spreadsheet-with-the-algolia-api-3kpd</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. Everything in the tutorial part works with free tools too.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;On the first weekday of every month, the &lt;code&gt;whoishiring&lt;/code&gt; account posts "Ask HN: Who is hiring?". Hundreds of companies reply with one top-level comment each. The October 2026 thread had &lt;strong&gt;266 top-level posts&lt;/strong&gt; when I counted on 25 Oct 2026. It's one of the best job sources for engineers, and it's a wall of text.&lt;/p&gt;

&lt;p&gt;There's a loose convention for the first line of each post:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csvs"&gt;&lt;code&gt;&lt;span class="k"&gt;Company&lt;/span&gt; &lt;span class="err"&gt;|&lt;/span&gt; &lt;span class="k"&gt;Role&lt;/span&gt; &lt;span class="err"&gt;|&lt;/span&gt; &lt;span class="k"&gt;Location&lt;/span&gt; &lt;span class="err"&gt;|&lt;/span&gt; &lt;span class="k"&gt;ONSITE&lt;/span&gt;&lt;span class="err"&gt;/&lt;/span&gt;&lt;span class="k"&gt;REMOTE&lt;/span&gt; &lt;span class="err"&gt;|&lt;/span&gt; &lt;span class="k"&gt;Salary&lt;/span&gt; &lt;span class="err"&gt;|&lt;/span&gt; &lt;span class="err"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That convention, plus two free official APIs, gets you most of the way to a spreadsheet.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two APIs
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The official Hacker News API&lt;/strong&gt; (Firebase): &lt;code&gt;hacker-news.firebaseio.com/v0/item/{id}.json&lt;/code&gt;. It's exact, but you have to walk the thread item by item.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HN Search by Algolia&lt;/strong&gt;: &lt;code&gt;hn.algolia.com/api/v1/...&lt;/code&gt;. Full-text search over stories and comments, with tags such as &lt;code&gt;story_{id}&lt;/code&gt; and &lt;code&gt;author_{name}&lt;/code&gt;, and up to 1,000 hits per page.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Algolia is the practical one for a whole thread.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Fetch the thread's top-level posts
&lt;/h2&gt;

&lt;p&gt;This ran as-is on 25 Oct 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="n"&gt;A&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://hn.algolia.com/api/v1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;thread&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;A&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/search_by_date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tags&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;story,author_whoishiring&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;query&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Who is hiring&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hitsPerPage&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;}).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hits&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;tid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;thread&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;objectID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;thread&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;tid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;posts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;A&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/search_by_date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tags&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;comment,story_&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tid&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hitsPerPage&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;page&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;}).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;posts&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hits&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;parent_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;tid&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;  &lt;span class="c1"&gt;# top-level only
&lt;/span&gt;    &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;nbPages&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="k"&gt;break&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;posts&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top-level posts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;first_line&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;posts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;comment_text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;p&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;first_line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Ask HN: Who is hiring? (October 2026) 49922569
266 top-level posts
['Treat Health', 'Full-stack Engineer (Elixir, Phoenix LiveView, React)', 'Helsinki, Finland', 'ONSITE', 'Full-time', 'Compensation in the link below']
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The trick is the filter: &lt;code&gt;tags=comment,story_{id}&lt;/code&gt; returns &lt;em&gt;every&lt;/em&gt; comment in the thread, including replies, so you keep only those whose &lt;code&gt;parent_id&lt;/code&gt; is the thread itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: The pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The convention is loose.&lt;/strong&gt; I counted the October first lines on 25 Oct: 77 had 5 pipe-separated fields, 69 had 4, 60 had 6, 23 had 3, and &lt;strong&gt;28 of 266 had no pipe at all&lt;/strong&gt;. Extra fields ("Full-time", "Compensation in the link below") and missing ones are both normal, so you can't map fields by position alone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Remote" is spread around.&lt;/strong&gt; It can be its own field (&lt;code&gt;REMOTE&lt;/code&gt;, &lt;code&gt;ONSITE&lt;/code&gt;) or buried in the location (&lt;code&gt;Remote US or Ontario, Canada&lt;/code&gt;, &lt;code&gt;REMOTE (Europe / US Eastern)&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Salary formats vary.&lt;/strong&gt; &lt;code&gt;$100k-$180k&lt;/code&gt;, &lt;code&gt;â‚¬75kâ€“80k&lt;/code&gt; (with an en dash), &lt;code&gt;$104Kâ€“$150K&lt;/code&gt;, or "compensation in the link".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HTML.&lt;/strong&gt; &lt;code&gt;comment_text&lt;/code&gt; is HTML with &lt;code&gt;&amp;lt;p&amp;gt;&lt;/code&gt;, &lt;code&gt;&amp;lt;a&amp;gt;&lt;/code&gt; and escaped entities. Decode it before you parse, and keep the links.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Relevance search caps at 1,000 results&lt;/strong&gt; on Algolia. For bigger searches, sort by date and walk back through time windows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sibling threads.&lt;/strong&gt; The same account posts "Who wants to be hired?" at the same time (IDs 49922568 and 49922569 in October), so match the title exactly.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged the parser as an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/hacker-news-search-hiring" rel="noopener noreferrer"&gt;Hacker News Scraper â€“ Who Is Hiring, Search &amp;amp; Comments&lt;/a&gt;. In &lt;code&gt;whoIsHiring&lt;/code&gt; mode it reads the latest (or last N) monthly threads and returns one row per post: company, roles, location, remote flag, salary text, visa mention, all URLs, author, timestamp, HN link and the &lt;strong&gt;full original text&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run-ready link:&lt;/strong&gt; &lt;a href="https://console.apify.com/actors/UtYvToTq7p3Ay2nRf/input" rel="noopener noreferrer"&gt;open it in Apify Console with an example input already filled in&lt;/a&gt; (&lt;code&gt;{"mode": "whoIsHiring", "keywords": ["engineer"], "maxItems": 50}&lt;/code&gt;). Click Start, or change the input first.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"whoIsHiring"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"keywords"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"engineer"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"maxItems"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A real run with that input on 17 Oct 2026 took about 6 seconds:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Company&lt;/th&gt;
&lt;th&gt;Roles&lt;/th&gt;
&lt;th&gt;Location&lt;/th&gt;
&lt;th&gt;Remote&lt;/th&gt;
&lt;th&gt;Salary&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;PrairieLearn&lt;/td&gt;
&lt;td&gt;Full-Stack Software Engineer&lt;/td&gt;
&lt;td&gt;â€“&lt;/td&gt;
&lt;td&gt;True&lt;/td&gt;
&lt;td&gt;$100k-$180k&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PAGNOS&lt;/td&gt;
&lt;td&gt;Cloud Engineer, AWS Serverless (TypeScript/Node.js)&lt;/td&gt;
&lt;td&gt;â€“&lt;/td&gt;
&lt;td&gt;True&lt;/td&gt;
&lt;td&gt;â‚¬75kâ€“80k&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Snout&lt;/td&gt;
&lt;td&gt;Multiple Engineering + Product Roles&lt;/td&gt;
&lt;td&gt;Remote US or Ontario, Canada&lt;/td&gt;
&lt;td&gt;True&lt;/td&gt;
&lt;td&gt;â€“&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Planlab.ai&lt;/td&gt;
&lt;td&gt;Product Engineer (Design) &amp;amp; Platform Engineer&lt;/td&gt;
&lt;td&gt;ONSITE in London, UK&lt;/td&gt;
&lt;td&gt;False&lt;/td&gt;
&lt;td&gt;â€“&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Checkly&lt;/td&gt;
&lt;td&gt;Senior Product Engineer, Product Manager, â€¦&lt;/td&gt;
&lt;td&gt;REMOTE (Europe / US Eastern)&lt;/td&gt;
&lt;td&gt;True&lt;/td&gt;
&lt;td&gt;â€“&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Other modes use the same APIs: &lt;code&gt;search&lt;/code&gt; (stories and comments with date, points, comments and author filters, walking date windows past the 1,000 cap), &lt;code&gt;feed&lt;/code&gt; (top/new/best/ask/show/jobs) and &lt;code&gt;items&lt;/code&gt; (specific IDs with full comment trees). I use &lt;code&gt;search&lt;/code&gt; with &lt;code&gt;lastDays: 7&lt;/code&gt; for brand mentions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Parsing accuracy, honestly
&lt;/h2&gt;

&lt;p&gt;Of the 50 posts in that run, &lt;strong&gt;37 had a parsed location, 31 had parsed roles and 25 had a parsed salary range&lt;/strong&gt;. The misses are mostly posts that don't follow the pipe format, or that genuinely don't mention salary. Those fields come back &lt;code&gt;null&lt;/code&gt;, and the raw text and links are always included, so nothing is lost. If you need perfect structure, run an LLM over the &lt;code&gt;text&lt;/code&gt; field afterwards. The parser gets you a sortable sheet first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Only what's posted on HN.&lt;/strong&gt; It's a self-selected set of mostly startups. It isn't a job market sample.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Timing.&lt;/strong&gt; The thread is posted on the first weekday of the month and most posts seem to land early: October had 266 top-level posts both when I tested on 17 Oct and when I recounted on 25 Oct. A run on day 1 will see fewer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Visa is a mention flag&lt;/strong&gt;, not a verified sponsorship policy.&lt;/li&gt;
&lt;li&gt;Not affiliated with Y Combinator, Hacker News or Algolia.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.003 per result&lt;/strong&gt; ($3 per 1,000), no start fee. The run stops at your "maximum cost per run". In my test, a run capped at $0.02 saved exactly 6 results. The 50-post run above was 15 cents. The script above is free, and honestly, for one month's thread it's all you need. The tool is for people who want the parser maintained, the filters and the other modes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Play fair
&lt;/h2&gt;

&lt;p&gt;People post in these threads to hire, not to be added to a sales sequence. Please don't mass-email the posters.&lt;/p&gt;

&lt;p&gt;Which posts does a pipe parser get wrong for you? Paste an example (or the HN link) in the comments and I'll test it.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>career</category>
      <category>hackernews</category>
    </item>
    <item>
      <title>A Free Google Alerts Alternative: Monitoring Global News With the GDELT DOC API</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Mon, 02 Nov 2026 15:12:40 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/a-free-google-alerts-alternative-monitoring-global-news-with-the-gdelt-doc-api-51fh</link>
      <guid>https://dev.to/dedicated_crest/a-free-google-alerts-alternative-monitoring-global-news-with-the-gdelt-doc-api-51fh</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. Everything in the tutorial part works with free tools too.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If you track mentions of a brand, a competitor, a supplier or a topic, you've probably tried Google Alerts and found it hit-and-miss. There's a free, open alternative that a lot of developers haven't met: &lt;strong&gt;the GDELT Project&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;GDELT monitors online news from sites in over 100 countries, in 65+ languages, and updates every 15 minutes. Its &lt;strong&gt;DOC 2.0 API&lt;/strong&gt; lets you search roughly the last three months of articles by keyword, and filter by things Google News doesn't expose: the outlet's &lt;strong&gt;country&lt;/strong&gt;, the article's &lt;strong&gt;language&lt;/strong&gt;, specific domains, GDELT's &lt;strong&gt;themes&lt;/strong&gt; (like &lt;code&gt;CYBER_ATTACK&lt;/code&gt; or &lt;code&gt;ECON_BANKRUPTCY&lt;/code&gt;) and &lt;strong&gt;tone&lt;/strong&gt;. No key, no signup.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: A query
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://api.gdeltproject.org/api/v2/doc/doc?query="data breach" sourcelang:english&amp;amp;mode=artlist&amp;amp;format=json&amp;amp;maxrecords=250&amp;amp;timespan=24h&amp;amp;sort=datedesc
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Operators go inside &lt;code&gt;query&lt;/code&gt;: &lt;code&gt;sourcelang:&lt;/code&gt;, &lt;code&gt;sourcecountry:&lt;/code&gt;, &lt;code&gt;domain:&lt;/code&gt;, &lt;code&gt;theme:&lt;/code&gt;, &lt;code&gt;tone&amp;lt;-5&lt;/code&gt;, &lt;code&gt;near20:"a b"&lt;/code&gt; and more. Each article comes back with &lt;code&gt;url&lt;/code&gt;, &lt;code&gt;title&lt;/code&gt;, &lt;code&gt;seendate&lt;/code&gt;, &lt;code&gt;domain&lt;/code&gt;, &lt;code&gt;language&lt;/code&gt;, &lt;code&gt;sourcecountry&lt;/code&gt; and &lt;code&gt;socialimage&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Respect the rate limit (seriously)
&lt;/h2&gt;

&lt;p&gt;GDELT asks clients to send &lt;strong&gt;at most one request every 5 seconds&lt;/strong&gt;, and when you don't, it doesn't always send a polite 429. While I was writing this, a quick burst of test calls got this back as plain text instead of JSON:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Please limit requests to one every 5 seconds or contact kalev.leetaru5@gmail.com for larger queries. ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So a naive &lt;code&gt;r.json()&lt;/code&gt; crashes. Here's a version that treats anything non-JSON as "busy" and backs off. It ran as-is on 25 Oct 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;gdelt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timespan&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;24h&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_records&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;250&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;query&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mode&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;artlist&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;format&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;maxrecords&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;max_records&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timespan&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;timespan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sort&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;datedesc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tries&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.gdeltproject.org/api/v2/doc/doc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;articles&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;busy:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="c1"&gt;# throttled: GDELT answers 429 or a plain-text "Please limit requests..." message
&lt;/span&gt;        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;GDELT kept saying busy&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;arts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;gdelt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="s"&gt;data breach&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; sourcelang:english&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;unique&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;arts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;          &lt;span class="c1"&gt;# syndicated copies share a headline
&lt;/span&gt;        &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;arts&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;articles,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unique headlines&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;seendate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;domain&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sourcecountry&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][:&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;9 articles, 8 unique headlines
20261025T111500Z thehindubusinessline.com India | Think . Over the week - The HinduBusinessLine
20261025T110000Z themarketsdaily.com United States | CocaCola Company ( The ) $KO Shares Purchased by Signaturefd
20261025T103000Z koreatimes.co.kr South Korea | Korean Air breach exposes blind spot in airline cybersecurit
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 3: The pitfalls (several are visible in that output)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Matches aren't headline matches.&lt;/strong&gt; The Coca-Cola share-purchase headline above says nothing about a breach. GDELT matched the phrase somewhere in the page. Add must-have terms, exclude terms, or a tone filter if you want tighter results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Low counts can be real.&lt;/strong&gt; Only 9 English articles matched &lt;code&gt;"data breach"&lt;/code&gt; in that 24-hour window. GDELT is broad, but it isn't every site on the web, so don't treat one query as "all the news".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Syndication.&lt;/strong&gt; The same wire story appears on dozens of sites with the same headline. Dedupe on a normalised title (and canonical URL), or you'll count one story many times.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Titles are tokenised.&lt;/strong&gt; Note &lt;code&gt;Think . Over the week&lt;/code&gt; and &lt;code&gt;( The )&lt;/code&gt;: punctuation comes back space-padded. Clean it for display.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;250 per request.&lt;/strong&gt; For busy topics or long ranges, split the time range into windows and merge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dates&lt;/strong&gt; are &lt;code&gt;YYYYMMDDTHHMMSSZ&lt;/code&gt; in UTC, and &lt;code&gt;seendate&lt;/code&gt; is when GDELT saw the article, not necessarily when it was published.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Three months back.&lt;/strong&gt; The DOC API covers roughly the last 3 months.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 4: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged all of this as an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/gdelt-news-monitor" rel="noopener noreferrer"&gt;GDELT News Scraper – Global News Monitor &amp;amp; Search API&lt;/a&gt;. It waits the 5 seconds between requests, retries patiently when GDELT is busy (you aren't charged for waiting), splits large ranges into time windows to go past 250 per query, canonicalises URLs and removes syndicated duplicates &lt;strong&gt;before&lt;/strong&gt; charging.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run-ready link:&lt;/strong&gt; &lt;a href="https://console.apify.com/actors/C0gVEJJhVsTMIFJmq/input" rel="noopener noreferrer"&gt;open it in Apify Console with an example input already filled in&lt;/a&gt; (&lt;code&gt;{"keywords": ["artificial intelligence"], "languages": ["english"], "maxItems": 50}&lt;/code&gt;). Click Start, or change the input first.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"keywords"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"data breach"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ransomware"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"languages"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"english"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"countries"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"UK"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"timespan"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"24h"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"sentiment"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"maxItems"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Inputs: keywords (any), must-contain (all), exclude terms, only/exclude domains, source countries, languages, GDELT themes, a raw query box for any operator, sentiment labels (negative/neutral/positive from GDELT's tone) or min/max tone, and a timespan or exact dates. Optionally it visits each article for its meta description, author, published time and image, and up to 5,000 characters of opening text.&lt;/p&gt;

&lt;p&gt;The default test (keyword &lt;code&gt;artificial intelligence&lt;/code&gt;, English, last 24 hours, 50 articles) ran on 18 Oct 2026. It saved 50 articles and skipped 74 duplicate headlines, which weren't charged. Most of its 3.7 minutes were spent waiting for GDELT. A few rows:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Seen (UTC)&lt;/th&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;th&gt;Outlet&lt;/th&gt;
&lt;th&gt;Outlet country&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2026-10-18 16:45&lt;/td&gt;
&lt;td&gt;S. Korea growth momentum to strengthen amid chip boom: Finance minister&lt;/td&gt;
&lt;td&gt;prokerala.com&lt;/td&gt;
&lt;td&gt;India&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-10-18 16:45&lt;/td&gt;
&lt;td&gt;Aboitiz Group builds future - ready talent as AI transforms business&lt;/td&gt;
&lt;td&gt;bworldonline.com&lt;/td&gt;
&lt;td&gt;Philippines&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2026-10-18 16:45&lt;/td&gt;
&lt;td&gt;Bangkok Meetings Highlight Overlapping Shocks to Global Economy&lt;/td&gt;
&lt;td&gt;english.aawsat.com&lt;/td&gt;
&lt;td&gt;Syria&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;74 duplicates for 50 unique articles gives you an idea how much syndication there is.&lt;/p&gt;

&lt;h2&gt;
  
  
  As a Google Alerts replacement
&lt;/h2&gt;

&lt;p&gt;Schedule a run with &lt;code&gt;timespan&lt;/code&gt; matching the interval (&lt;code&gt;24h&lt;/code&gt; daily, &lt;code&gt;1h&lt;/code&gt; hourly) and send new rows to Slack, email or a sheet. Use &lt;code&gt;sentiment&lt;/code&gt; or a max tone if you only care about negative coverage, which is the common setup for adverse-media screening.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It's slow by design.&lt;/strong&gt; Runs take from about 30 seconds to a few minutes, longer when GDELT is busy. Sentiment labelling runs 3 queries per window, so it's about 3× slower.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GDELT's coverage, not Google's.&lt;/strong&gt; It's strong on global and non-English outlets. If you want "what Google News shows for this query", a Google News tool fits better.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;About 3 months of history.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Relevance is GDELT's.&lt;/strong&gt; Full-text matching produces some off-topic hits (see the Coca-Cola example above).&lt;/li&gt;
&lt;li&gt;Not affiliated with the GDELT Project. Respect publishers' copyright if you enable text snippets.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.002 per unique article&lt;/strong&gt; ($2 per 1,000), no start fee, duplicates free. Runs stop at your "maximum cost per run". The 50-article test was 10 cents. The script above is free, and it's what I'd use for a single daily query.&lt;/p&gt;

&lt;p&gt;What do you use for news monitoring today, and what does it miss? I'm curious whether GDELT's country and language filters matter to anyone but me, so tell me in the comments.&lt;/p&gt;

</description>
      <category>python</category>
      <category>opendata</category>
      <category>api</category>
      <category>news</category>
    </item>
    <item>
      <title>Collecting App Store Reviews by Country and Version With Apple's Public Feed</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Fri, 30 Oct 2026 14:12:14 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/collecting-app-store-reviews-by-country-and-version-with-apples-public-feed-2fod</link>
      <guid>https://dev.to/dedicated_crest/collecting-app-store-reviews-by-country-and-version-with-apples-public-feed-2fod</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. Everything in the tutorial part works with free tools too.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A star average tells you almost nothing. "4.6 stars" hides the fact that version 7.144.0 broke streak freezes for half your users in the UK. What product teams actually want is reviews &lt;strong&gt;by version and by country&lt;/strong&gt;, for their own app and for competitors.&lt;/p&gt;

&lt;p&gt;For your own app, App Store Connect has the reviews and lets you reply. For any other app, there's no official reviews API, but Apple does publish a public &lt;strong&gt;customer-reviews feed&lt;/strong&gt; per storefront, and the iTunes Search/Lookup API covers the app metadata.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the feed works
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://itunes.apple.com/{country}/rss/customerreviews/page={n}/id={appId}/sortby={mostrecent|mosthelpful}/json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;country&lt;/code&gt; is a storefront code (&lt;code&gt;us&lt;/code&gt;, &lt;code&gt;gb&lt;/code&gt;, &lt;code&gt;de&lt;/code&gt;, &lt;code&gt;jp&lt;/code&gt;, &lt;code&gt;br&lt;/code&gt;…). &lt;strong&gt;Every storefront has its own reviews.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Each page has up to 50 reviews, and you get &lt;strong&gt;at most 10 pages per sort order&lt;/strong&gt;. That's 500 for &lt;code&gt;mostrecent&lt;/code&gt; and 500 for &lt;code&gt;mosthelpful&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Each entry has the rating, title, text, author, &lt;strong&gt;app version&lt;/strong&gt; and date.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The app ID is the number after &lt;code&gt;id&lt;/code&gt; in the App Store URL (&lt;code&gt;389801252&lt;/code&gt; for Instagram). If you only have a name, the iTunes Search API turns it into an ID, and Lookup gives you the average rating, rating count and current version.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: A collector you can run today
&lt;/h2&gt;

&lt;p&gt;This pulls both sort orders for one storefront and dedupes on review ID. It ran as-is on 25 Oct 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;app_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;country&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;us&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;seen&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;sort&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mostrecent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mosthelpful&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;  &lt;span class="c1"&gt;# Apple serves at most 10 pages of 50 per sort order
&lt;/span&gt;            &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://itunes.apple.com/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;country&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/rss/customerreviews/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                   &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;page=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/id=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;app_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/sortby=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;sort&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;entries&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;_&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;())},&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; \
                              &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;feed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{}).&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;entry&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;  &lt;span class="c1"&gt;# defensive: normalise a lone object to a list
&lt;/span&gt;                &lt;span class="n"&gt;entries&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;im:rating&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                    &lt;span class="k"&gt;continue&lt;/span&gt;  &lt;span class="c1"&gt;# skip anything that isn't a review
&lt;/span&gt;                &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;label&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rating&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;im:rating&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;label&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;im:version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;label&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
                    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;label&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;updated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;label&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;break&lt;/span&gt;
            &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;389801252&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# Instagram, UK storefront
&lt;/span&gt;&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unique reviews&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rating&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;are 1-2 stars&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I ran it twice, a few minutes apart. The first run printed &lt;code&gt;749 unique reviews / 492 are 1-2 stars&lt;/code&gt;. The second printed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;996 unique reviews
611 are 1-2 stars
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same app, same storefront, same code. The difference is the feed's habit of returning an empty page before the real end: my loop treats an empty page as "done", so the first run stopped early on at least one sort order. That's the most important pitfall in this whole post, and the fix is to retry empty pages (with a fresh cache-buster) before giving up.&lt;/p&gt;

&lt;p&gt;Also notice the skew: the newest and "most helpful" reviews lean negative (611 of 996 here). That describes this sample, not every review ever written.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: The pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The cap is per sort order and per country.&lt;/strong&gt; To see more, add storefronts. A US-only pull misses most of the feedback for an app that's big in Germany, Japan or Brazil.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;History needs polling.&lt;/strong&gt; You only ever see roughly the newest 500. To build a full history, run on a schedule and keep what you've already stored.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Empty pages that aren't the end.&lt;/strong&gt; As above. Retry an empty page a couple of times with a new cache-busting parameter before you treat it as the last page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Structure.&lt;/strong&gt; Every value is wrapped in &lt;code&gt;{"label": ...}&lt;/code&gt;, and dates come with a UTC offset (e.g. &lt;code&gt;-07:00&lt;/code&gt;). Normalise to UTC before you compare countries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Version is the useful field.&lt;/strong&gt; Group 1★ reviews by &lt;code&gt;im:version&lt;/code&gt; and release-related spikes jump out.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged this as an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/app-store-reviews-scraper" rel="noopener noreferrer"&gt;App Store Reviews Scraper&lt;/a&gt;. It takes App Store URLs, numeric IDs or plain app names, any number of countries, and &lt;code&gt;sort&lt;/code&gt; set to most recent, most helpful or both (deduplicated, up to about 1,000 unique per app per country).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"apps"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://apps.apple.com/us/app/instagram/id389801252"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"duolingo"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"countries"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"gb"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"maxReviewsPerApp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can filter by min/max stars, "only reviews since" a date, and keywords in the title or text. Filtered-out reviews aren't charged. Optional app details go on every row: name, developer, bundle ID, genre, average rating, rating count, current version.&lt;/p&gt;

&lt;p&gt;That input returned 200 reviews from 4 app/country combinations in about 22 seconds on 18 Oct 2026. Some rows:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;App&lt;/th&gt;
&lt;th&gt;Store&lt;/th&gt;
&lt;th&gt;Rating&lt;/th&gt;
&lt;th&gt;Review title&lt;/th&gt;
&lt;th&gt;Version&lt;/th&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Instagram&lt;/td&gt;
&lt;td&gt;GB&lt;/td&gt;
&lt;td&gt;★★★★★&lt;/td&gt;
&lt;td&gt;Like a digital scrap book&lt;/td&gt;
&lt;td&gt;451.0.0&lt;/td&gt;
&lt;td&gt;2026-10-17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Instagram&lt;/td&gt;
&lt;td&gt;GB&lt;/td&gt;
&lt;td&gt;★&lt;/td&gt;
&lt;td&gt;Suspended exile&lt;/td&gt;
&lt;td&gt;451.0.0&lt;/td&gt;
&lt;td&gt;2026-10-16&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Duolingo: Learn Languages&lt;/td&gt;
&lt;td&gt;US&lt;/td&gt;
&lt;td&gt;★&lt;/td&gt;
&lt;td&gt;Annoying streak freezes&lt;/td&gt;
&lt;td&gt;7.144.0&lt;/td&gt;
&lt;td&gt;2026-10-17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Duolingo: Language &amp;amp; Chess&lt;/td&gt;
&lt;td&gt;GB&lt;/td&gt;
&lt;td&gt;★★★★★&lt;/td&gt;
&lt;td&gt;So good&lt;/td&gt;
&lt;td&gt;7.144.0&lt;/td&gt;
&lt;td&gt;2026-10-17&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Note the name difference: the same Duolingo app has a different title in the US and UK storefronts.&lt;/p&gt;

&lt;h2&gt;
  
  
  The workflow I'd use
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Your app plus 3–5 competitors, in the 5–10 countries that matter to you.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;maxRating: 2&lt;/code&gt; for a complaints feed, or no filter for everything.&lt;/li&gt;
&lt;li&gt;A weekly schedule with "only reviews since" set to last week.&lt;/li&gt;
&lt;li&gt;Send the rows to a sheet, or to an LLM step that clusters complaints by version.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Limits, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Apple's cap is the cap.&lt;/strong&gt; About 500 per sort order per country, so around 1,000 per country at best. Anything older is only reachable if you started polling earlier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No Google Play.&lt;/strong&gt; There's no official public reviews API for other people's apps on Play.&lt;/li&gt;
&lt;li&gt;Not affiliated with Apple.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.0002 per review&lt;/strong&gt; ($0.20 per 1,000). App lookups and filtered-out reviews are free, there's no start fee, and runs stop at your "maximum cost per run". The 200-review test above came to 4 cents. Some Store Actors are slightly cheaper per review. The script above is free.&lt;/p&gt;

&lt;h2&gt;
  
  
  Play fair
&lt;/h2&gt;

&lt;p&gt;Reviews are public, but they're written by people. Don't republish them with names attached, and follow Apple's terms and privacy laws.&lt;/p&gt;

&lt;p&gt;How do you analyse reviews today: by hand, with tags, or with an LLM? I'm curious what's actually worked for you, so tell me in the comments.&lt;/p&gt;

</description>
      <category>ios</category>
      <category>python</category>
      <category>api</category>
      <category>productmanagement</category>
    </item>
    <item>
      <title>Finding Contact Emails on Company Websites Politely: robots.txt, mailto, JSON-LD and the Traps</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Thu, 29 Oct 2026 14:03:03 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/finding-contact-emails-on-company-websites-politely-robotstxt-mailto-json-ld-and-the-traps-35l3</link>
      <guid>https://dev.to/dedicated_crest/finding-contact-emails-on-company-websites-politely-robotstxt-mailto-json-ld-and-the-traps-35l3</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. Everything in the tutorial part works with free tools too.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You have a list of company domains (a CRM export, a lead list, a spreadsheet from an event) and you need a contact email, a phone number and maybe the company's Instagram or LinkedIn. The details are usually right there on the company's own site. Getting them reliably, and politely, takes more care than a regex over the homepage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: A minimal, polite extractor
&lt;/h2&gt;

&lt;p&gt;This is roughly the smallest version I'd be comfortable running against small-business sites. It checks robots.txt with its own User-Agent, reads the homepage, pulls &lt;code&gt;mailto:&lt;/code&gt; and &lt;code&gt;tel:&lt;/code&gt; links, and lists same-domain contact/about pages to visit next. It ran as-is on 25 Oct 2026:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;robotparser&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib.parse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urlparse&lt;/span&gt;

&lt;span class="n"&gt;UA&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;MyContactBot/0.1 (+mailto:you@example.com)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UA&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;geturl&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ignore&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;contacts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;site&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;rp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;robotparser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RobotFileParser&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="c1"&gt;# fetch robots.txt with *your* UA; RobotFileParser.read() uses urllib's default
&lt;/span&gt;        &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;site&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/robots.txt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;([])&lt;/span&gt;  &lt;span class="c1"&gt;# no robots.txt (404) means allowed
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;can_fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;UA&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;site&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;site&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;site&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;blocked_by_robots_txt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;final&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;site&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;emails&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;mailto:([^&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;\'?&amp;gt;\s]+)&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;phones&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;tel:([+\d()\-. ]{7,})&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;host&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urlparse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;final&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt;
    &lt;span class="n"&gt;contact_links&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;final&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;href=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;([^&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]+)&lt;/span&gt;&lt;span class="sh"&gt;"'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                     &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;contact|about|impressum&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;I&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                     &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;urlparse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;final&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;site&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;final&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;emails&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;emails&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;phones&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;phones&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;contactPages&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;contact_links&lt;/span&gt;&lt;span class="p"&gt;)[:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;contacts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.aloha-dental.com/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="err"&gt;'site':&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'https://www.aloha-dental.com/'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'emails':&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'phones':&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="err"&gt;'(&lt;/span&gt;&lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="err"&gt;)&lt;/span&gt;&lt;span class="mi"&gt;707-7300&lt;/span&gt;&lt;span class="err"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="err"&gt;'contactPages':&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="err"&gt;'https://www.aloha-dental.com/about-us/'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'https://www.aloha-dental.com/contact-us/'&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two lessons from that one run.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First, the homepage had no email.&lt;/strong&gt; The phone was there as a &lt;code&gt;tel:&lt;/code&gt; link, but the address lives deeper in the site. My packaged tool (below) found &lt;code&gt;riverside@aloha-dental.com&lt;/code&gt; on the same site after visiting 5 pages. You almost always need to follow the contact/about links.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Second, my first version wrongly reported this site as blocked by robots.txt.&lt;/strong&gt; Python's &lt;code&gt;RobotFileParser.read()&lt;/code&gt; fetches robots.txt with urllib's default User-Agent, the server answered 403, and the parser treats an access error as "disallow everything". Fetching robots.txt yourself with your real UA fixed it. If your crawler says "blocked" a lot, check that before blaming the sites.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: The traps you'll hit next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Emails aren't only in &lt;code&gt;mailto:&lt;/code&gt;.&lt;/strong&gt; Look in page text, and in schema.org JSON-LD (&lt;code&gt;"email"&lt;/code&gt;, &lt;code&gt;"telephone"&lt;/code&gt;, &lt;code&gt;"address"&lt;/code&gt;), which a lot of small-business sites ship through their CMS or SEO plugin.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Obfuscation.&lt;/strong&gt; Cloudflare's email protection replaces addresses with a &lt;code&gt;data-cfemail&lt;/code&gt; hex string (a simple XOR you can decode). People also write &lt;code&gt;name [at] domain [dot] com&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Junk addresses.&lt;/strong&gt; Placeholder and tracking addresses (&lt;code&gt;example.com&lt;/code&gt;, Sentry DSNs, &lt;code&gt;wixpress&lt;/code&gt;) show up in page source. Filter them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dead domains.&lt;/strong&gt; An email whose domain has no MX record won't receive mail. A DNS lookup is cheap and doesn't touch anyone's mailbox.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Phone false positives.&lt;/strong&gt; Order numbers, dates and prices look like phone numbers in text. Trust &lt;code&gt;tel:&lt;/code&gt; links and schema.org more than regexes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No contact link at all.&lt;/strong&gt; Fall back to trying &lt;code&gt;/contact&lt;/code&gt;, &lt;code&gt;/contact-us&lt;/code&gt; and &lt;code&gt;/about&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Be slow.&lt;/strong&gt; These are often tiny shared-hosting sites. One request per second per site, an identifiable User-Agent, and robots.txt &lt;code&gt;Crawl-delay&lt;/code&gt; if they set one.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged this into an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/website-contact-enrichment" rel="noopener noreferrer"&gt;Website Contact Scraper&lt;/a&gt;. You paste domains, URLs or emails, and get one row per company:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Emails&lt;/strong&gt; from &lt;code&gt;mailto:&lt;/code&gt;, text, JSON-LD, Cloudflare-protected and "[at]/[dot]" forms. Each is tagged &lt;code&gt;sameDomain&lt;/code&gt;, &lt;code&gt;role&lt;/code&gt; (info@, sales@…) and &lt;code&gt;domainAcceptsMail&lt;/code&gt; (the MX check).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Phones&lt;/strong&gt; from &lt;code&gt;tel:&lt;/code&gt; and schema.org. Numbers found only in text go into a separate &lt;code&gt;phonesUncertain&lt;/code&gt; list.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;13 social networks&lt;/strong&gt; (LinkedIn, Facebook, Instagram, X, YouTube, TikTok, Pinterest, GitHub, Threads, Telegram, WhatsApp, Discord, Yelp), plus &lt;code&gt;contactPageUrls&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Company name and postal address when the site publishes them in schema.org or OpenGraph&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It crawls the homepage plus up to 19 contact/about/impressum/team pages on the same domain, honours robots.txt (including &lt;code&gt;Crawl-delay&lt;/code&gt;), goes 1 request per second per site and identifies itself as &lt;code&gt;ContactEnrichmentBot&lt;/code&gt;. It doesn't run JavaScript.&lt;/p&gt;

&lt;p&gt;The real result for the site above (trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"domain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"aloha-dental.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"companyName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Aloha Dental"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"primaryEmail"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"riverside@aloha-dental.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"phones"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"(512)707-7300"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"phonesUncertain"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"(512) 494-5655"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"address"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1614 E. Riverside DR., Austin, Texas, 78741, US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"contactPageUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://www.aloha-dental.com/contact-us/"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"instagram"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://www.instagram.com/alohadentaltx"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pagesCrawled"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"charged"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Real fill rates
&lt;/h2&gt;

&lt;p&gt;From live runs on 17 Oct 2026, measured over sites that were actually reached:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Websites&lt;/th&gt;
&lt;th&gt;Reached&lt;/th&gt;
&lt;th&gt;Email&lt;/th&gt;
&lt;th&gt;Phone&lt;/th&gt;
&lt;th&gt;Instagram&lt;/th&gt;
&lt;th&gt;LinkedIn&lt;/th&gt;
&lt;th&gt;Contact page&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Manchester restaurant/café sites&lt;/td&gt;
&lt;td&gt;90&lt;/td&gt;
&lt;td&gt;72&lt;/td&gt;
&lt;td&gt;68%&lt;/td&gt;
&lt;td&gt;62%&lt;/td&gt;
&lt;td&gt;76%&lt;/td&gt;
&lt;td&gt;11%&lt;/td&gt;
&lt;td&gt;70%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mixed: US trades/dentists, Paris cafés, 12 SaaS companies&lt;/td&gt;
&lt;td&gt;55&lt;/td&gt;
&lt;td&gt;49&lt;/td&gt;
&lt;td&gt;55%&lt;/td&gt;
&lt;td&gt;57%&lt;/td&gt;
&lt;td&gt;69%&lt;/td&gt;
&lt;td&gt;59%&lt;/td&gt;
&lt;td&gt;89%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The sites that weren't reached were blocked by robots.txt (7 and 4 sites, never crawled), returned HTTP 403/404, had no DNS, or timed out.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No JavaScript rendering.&lt;/strong&gt; Sites that only render contact details client-side return fewer results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Some sites block data-centre IPs.&lt;/strong&gt; You can set an Apify Proxy, which Apify bills separately.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Email isn't guaranteed.&lt;/strong&gt; Between a third and almost half of the reached sites in the tests had no email at all (32% and 45%). Many small businesses only offer a contact form, which is why &lt;code&gt;contactPageUrls&lt;/code&gt; is there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The MX check isn't verification.&lt;/strong&gt; It confirms the domain accepts mail, not that the mailbox exists. The tool never sends mail or probes mailboxes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.002 per website where at least one email, phone or social profile was found&lt;/strong&gt; ($2 per 1,000). Websites that are unreachable, blocked by robots.txt or empty are free (&lt;code&gt;charged: false&lt;/code&gt;). In the mixed test, 47 of the 55 sites were charged, which is 9.4 cents. With a $0.01 max-cost cap, it charged exactly 5 sites and stopped. The script above is free.&lt;/p&gt;

&lt;h2&gt;
  
  
  Play fair
&lt;/h2&gt;

&lt;p&gt;Named employees' emails and phones are personal data under GDPR, UK GDPR and CCPA, and you're the data controller. Have a lawful basis, honour opt-outs, follow CAN-SPAM, CASL and PECR, and if you want to minimise personal data, keep only the role addresses (&lt;code&gt;role: true&lt;/code&gt;). Please don't build spam lists with any of this.&lt;/p&gt;

&lt;p&gt;What obfuscation trick have you run into that I haven't listed? Tell me in the comments and I'll add a decoder for it.&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>leadgeneration</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Domain Age, Email Provider and DMARC From a List of Domains: RDAP + DNS-over-HTTPS in Python</title>
      <dc:creator>Ben Barazandeh</dc:creator>
      <pubDate>Wed, 28 Oct 2026 14:08:21 +0000</pubDate>
      <link>https://dev.to/dedicated_crest/domain-age-email-provider-and-dmarc-from-a-list-of-domains-rdap-dns-over-https-in-python-5ge6</link>
      <guid>https://dev.to/dedicated_crest/domain-age-email-provider-and-dmarc-from-a-list-of-domains-rdap-dns-over-https-in-python-5ge6</guid>
      <description>&lt;p&gt;&lt;em&gt;Disclosure: I'm Ben, and I built the Apify Actor mentioned later in this post. I charge per result for it. Everything in the tutorial part works with free tools too.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A list of domains says a lot more than it looks. From public data alone you can tell how old a domain is, when it expires, who hosts its DNS, whether the company is on Google Workspace or Microsoft 365, and whether it has a DMARC policy. That's useful for fraud checks ("this domain is 12 days old"), deliverability audits, and segmenting a lead list ("Microsoft 365 shops with no DMARC").&lt;/p&gt;

&lt;p&gt;You don't need a WHOIS API subscription for most of it. You need two open protocols: &lt;strong&gt;RDAP&lt;/strong&gt; and &lt;strong&gt;DNS&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  RDAP in one paragraph
&lt;/h2&gt;

&lt;p&gt;RDAP is the official, structured JSON successor to WHOIS, run by the same registries. IANA publishes a bootstrap file (&lt;code&gt;data.iana.org/rdap/dns.json&lt;/code&gt;) that tells you which RDAP server handles each TLD. You look up the TLD, then call &lt;code&gt;{server}/domain/{name}&lt;/code&gt;. A 404 means the registry has no record, so the domain is probably available.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Registration dates + DNS in ~30 lines
&lt;/h2&gt;

&lt;p&gt;This ran as-is on 25 Oct 2026. DNS goes through Cloudflare's public DNS-over-HTTPS JSON endpoint, so there's nothing to install beyond &lt;code&gt;requests&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rdap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;tld&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;services&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://data.iana.org/rdap/dns.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;services&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;next&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;tlds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urls&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;services&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tld&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;tlds&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;  &lt;span class="c1"&gt;# this TLD has no RDAP server: fall back to port-43 WHOIS
&lt;/span&gt;    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/domain/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;404&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registered&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;ev&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;eventAction&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;eventDate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;events&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[])}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registered&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;created&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ev&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;registration&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expires&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ev&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expiration&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;dns&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rtype&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://cloudflare-dns.com/dns-query&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;name&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rtype&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
                     &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;accept&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;application/dns-json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Answer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[])]&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stripe.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;spiegel.de&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;rdap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  MX   &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;dns&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;MX&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[:&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  DMARC&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;dns&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;_dmarc.&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;TXT&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output (trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;stripe.com {'registered': True, 'created': '1995-09-12T04:00:00Z', 'expires': '2027-09-11T04:00:00Z'}
  MX    ['10 aspmx.l.google.com.', '20 alt1.aspmx.l.google.com.']
  DMARC ['"v=DMARC1; p=reject; pct=100; fo=1; rua=mailto:dmarc-reports@stripe.com; ...']
spiegel.de None
  MX    ['0 spiegel-de.mail.protection.outlook.com.']
  DMARC ['"v=DMARC1; p=quarantine; rua=mailto:report.dmarc@spiegel.de; ...']
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two facts fall straight out: Stripe's mail goes to Google (&lt;code&gt;aspmx.l.google.com&lt;/code&gt;), Der Spiegel's to Microsoft 365 (&lt;code&gt;mail.protection.outlook.com&lt;/code&gt;). And &lt;code&gt;.de&lt;/code&gt; has no RDAP server in IANA's bootstrap, so &lt;code&gt;rdap()&lt;/code&gt; returned &lt;code&gt;None&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: The pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;ccTLDs are a patchwork.&lt;/strong&gt; Some have RDAP, some only port-43 WHOIS with their own text format, and some publish no dates at all. &lt;code&gt;.de&lt;/code&gt; doesn't publish registration dates. A few registries (for example &lt;code&gt;.es&lt;/code&gt;) offer no public RDAP or WHOIS.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Registrable domain" isn't "last two labels".&lt;/strong&gt; &lt;code&gt;news.bbc.co.uk&lt;/code&gt; belongs to &lt;code&gt;bbc.co.uk&lt;/code&gt;, not &lt;code&gt;co.uk&lt;/code&gt;. Use the Public Suffix List, not string splitting. The same goes for turning emails and URLs into domains.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Registrant data is mostly redacted&lt;/strong&gt; since GDPR. Expect the organisation and country at best.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DNS answers need cleaning.&lt;/strong&gt; MX records come with a priority prefix and trailing dot. TXT records come quoted and can be split into chunks.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Email provider is inference.&lt;/strong&gt; You map MX hosts (&lt;code&gt;google.com&lt;/code&gt;, &lt;code&gt;outlook.com&lt;/code&gt;, &lt;code&gt;zoho&lt;/code&gt;…) to providers, and SPF &lt;code&gt;include:&lt;/code&gt;s to sending tools (HubSpot, Salesforce, SendGrid…). It's reliable for the big providers and fuzzy for self-hosted setups.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DKIM can't be listed.&lt;/strong&gt; You can only probe common selectors, and some domains answer every selector (wildcards), which looks like a false positive.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rate limits.&lt;/strong&gt; Registry RDAP servers throttle bulk lookups. Add concurrency limits and backoff.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 3: Or use the packaged version
&lt;/h2&gt;

&lt;p&gt;I packaged all of this as an Apify Actor, &lt;a href="https://apify.com/dedicated_crest/domain-whois-dns-intel" rel="noopener noreferrer"&gt;Bulk WHOIS &amp;amp; DNS Lookup&lt;/a&gt;. You paste domains, URLs or email addresses, and it normalises them to the registrable domain (&lt;code&gt;co.uk&lt;/code&gt;, &lt;code&gt;com.au&lt;/code&gt;…) and dedupes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"domains"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"stripe.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://www.apify.com/store"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"jane@hubspot.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"bbc.co.uk"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"notion.so"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"includeWhois"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"includeDns"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"includeWebsite"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"concurrency"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One row per domain: registrar and IANA ID, created/updated/expiry dates, &lt;strong&gt;domain age in days&lt;/strong&gt;, days until expiry, EPP status, nameservers, A/AAAA/MX/NS/TXT, SPF and its includes, DMARC record and policy, optional DKIM probing (with wildcard detection), &lt;strong&gt;email provider&lt;/strong&gt;, email security gateway, email senders, DNS provider, about 50 SaaS tools detected from DNS verification records, and an optional website check (status, redirects, title, basic tech like WordPress or Shopify). It's RDAP first with WHOIS fallback.&lt;/p&gt;

&lt;p&gt;A real run on 18 Oct 2026 with 7 inputs and every check on took about 46 seconds:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Domain&lt;/th&gt;
&lt;th&gt;Registrar&lt;/th&gt;
&lt;th&gt;Created&lt;/th&gt;
&lt;th&gt;Email provider&lt;/th&gt;
&lt;th&gt;DMARC&lt;/th&gt;
&lt;th&gt;DNS provider&lt;/th&gt;
&lt;th&gt;SaaS found&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;stripe.com&lt;/td&gt;
&lt;td&gt;SafeNames Ltd.&lt;/td&gt;
&lt;td&gt;1995-09-12&lt;/td&gt;
&lt;td&gt;Google Workspace&lt;/td&gt;
&lt;td&gt;reject&lt;/td&gt;
&lt;td&gt;Amazon Route 53&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;openai.com&lt;/td&gt;
&lt;td&gt;MarkMonitor Inc.&lt;/td&gt;
&lt;td&gt;2007-01-19&lt;/td&gt;
&lt;td&gt;Google Workspace&lt;/td&gt;
&lt;td&gt;reject&lt;/td&gt;
&lt;td&gt;Azure DNS&lt;/td&gt;
&lt;td&gt;19&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;hubspot.com&lt;/td&gt;
&lt;td&gt;MarkMonitor Inc.&lt;/td&gt;
&lt;td&gt;2005-02-06&lt;/td&gt;
&lt;td&gt;Google Workspace&lt;/td&gt;
&lt;td&gt;reject&lt;/td&gt;
&lt;td&gt;Cloudflare&lt;/td&gt;
&lt;td&gt;17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;bbc.co.uk&lt;/td&gt;
&lt;td&gt;British Broadcasting Corpor…&lt;/td&gt;
&lt;td&gt;1994-12-13&lt;/td&gt;
&lt;td&gt;Other / self-hosted&lt;/td&gt;
&lt;td&gt;reject&lt;/td&gt;
&lt;td&gt;–&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;spiegel.de&lt;/td&gt;
&lt;td&gt;–&lt;/td&gt;
&lt;td&gt;–&lt;/td&gt;
&lt;td&gt;Microsoft 365&lt;/td&gt;
&lt;td&gt;quarantine&lt;/td&gt;
&lt;td&gt;–&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;ceo@hubspot.com&lt;/code&gt; and &lt;code&gt;https://www.apify.com&lt;/code&gt; in the input were normalised to their domains. Same &lt;code&gt;.de&lt;/code&gt; story as my script: DNS and email data, no WHOIS dates.&lt;/p&gt;

&lt;p&gt;Useful filters once it's in a sheet: &lt;code&gt;domainAgeDays &amp;lt; 90&lt;/code&gt; for new-domain risk, &lt;code&gt;daysUntilExpiry &amp;lt; 30&lt;/code&gt; for portfolio renewals, &lt;code&gt;emailProvider&lt;/code&gt; plus &lt;code&gt;dmarcPolicy&lt;/code&gt; for deliverability or security outreach.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits, honestly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;WHOIS coverage depends on the registry.&lt;/strong&gt; No dates for &lt;code&gt;.de&lt;/code&gt;, nothing for TLDs with no public RDAP/WHOIS (you get a &lt;code&gt;whoisError&lt;/code&gt; note plus DNS data).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Registrant fields are usually null&lt;/strong&gt; because of privacy redaction.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Provider and SaaS detection come from DNS records&lt;/strong&gt;, so they show what a domain has &lt;em&gt;verified&lt;/em&gt; or &lt;em&gt;routed&lt;/em&gt;, not what it actively uses.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The website check is slower.&lt;/strong&gt; Turn it off for big lists and raise concurrency (up to 20).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;registered: false&lt;/code&gt; is a strong hint, not a guarantee. Confirm at a registrar before you buy.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;$0.003 per domain&lt;/strong&gt; ($3 per 1,000), with WHOIS/RDAP, DNS, email intelligence and the website check included. No start fee, and runs stop at your "maximum cost per run". The 7-domain test was about 2 cents. If you only need registrar and expiry, a WHOIS-only tool can be cheaper. And the script above is free.&lt;/p&gt;

&lt;h2&gt;
  
  
  Play fair
&lt;/h2&gt;

&lt;p&gt;Some registries restrict reusing WHOIS data for marketing. Read their terms, and keep privacy laws in mind when you combine domain data with people.&lt;/p&gt;

&lt;p&gt;What signal would you want added: MTA-STS, BIMI, CAA, something else? Tell me in the comments.&lt;/p&gt;

</description>
      <category>python</category>
      <category>dns</category>
      <category>security</category>
      <category>api</category>
    </item>
  </channel>
</rss>
