<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Daniel Meshulam</title>
    <description>The latest articles on DEV Community by Daniel Meshulam (@glitchbound).</description>
    <link>https://dev.to/glitchbound</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4055417%2F57611ce1-a653-43cc-a2cc-478624eb75e9.png</url>
      <title>DEV Community: Daniel Meshulam</title>
      <link>https://dev.to/glitchbound</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/glitchbound"/>
    <language>en</language>
    <item>
      <title>Google's News API has been gone since 2016, and people still search for it</title>
      <dc:creator>Daniel Meshulam</dc:creator>
      <pubDate>Thu, 30 Jul 2026 18:44:36 +0000</pubDate>
      <link>https://dev.to/glitchbound/googles-news-api-has-been-gone-since-2016-and-people-still-search-for-it-3544</link>
      <guid>https://dev.to/glitchbound/googles-news-api-has-been-gone-since-2016-and-people-still-search-for-it-3544</guid>
      <description>&lt;p&gt;Search for "google news api" and a striking share of the results are people&lt;br&gt;
looking for something that no longer exists. Google deprecated the News Search&lt;br&gt;
API in May 2011 as part of a wider AJAX Search API cull, kept it limping until&lt;br&gt;
the final shutdown in February 2016, and never shipped a replacement. A decade&lt;br&gt;
later the searches are still there.&lt;/p&gt;

&lt;p&gt;What almost nobody mentions is that Google News still exposes a perfectly good&lt;br&gt;
public endpoint. It is just an RSS feed rather than a JSON API, it needs no key,&lt;br&gt;
and it has quietly outlived the API that was supposed to be the real product.&lt;/p&gt;
&lt;h2&gt;
  
  
  The endpoint
&lt;/h2&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://news.google.com/rss/search?q=&amp;lt;query&amp;gt;&amp;amp;hl=en-US&amp;amp;gl=US&amp;amp;ceid=US:en
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;That is the whole thing. Swap &lt;code&gt;hl&lt;/code&gt;, &lt;code&gt;gl&lt;/code&gt; and &lt;code&gt;ceid&lt;/code&gt; for a different language and&lt;br&gt;
country and you get that edition instead. There is no key, no quota page, and no&lt;br&gt;
sign-up.&lt;/p&gt;

&lt;p&gt;The catches are real but small:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It is RSS, so you get XML.&lt;/strong&gt; Title, link, publication date, source name and
a short description. No article body, and no way to ask for one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The links are Google redirects&lt;/strong&gt;, not publisher URLs. You have to follow them
if you want the destination.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The publisher name lives in a nested &lt;code&gt;&amp;lt;source&amp;gt;&lt;/code&gt; element&lt;/strong&gt;, not where a normal
feed parser looks for it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Roughly 100 items per query.&lt;/strong&gt; Deeper history is not available through it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of that is a reason to go and pay for a news API. It is a reason to write&lt;br&gt;
about forty lines once.&lt;/p&gt;
&lt;h2&gt;
  
  
  Two sources are better than one
&lt;/h2&gt;

&lt;p&gt;Bing News publishes its own RSS in the same shape at a different URL. Running&lt;br&gt;
both and merging costs one extra request and meaningfully changes what you see,&lt;br&gt;
because the two indexes disagree about what is worth surfacing.&lt;/p&gt;

&lt;p&gt;Here is a real run over both:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;16 articles
sources: {'google:semiconductor export controls': 8,
          'bing:semiconductor export controls': 8}

2026-07-09  Just Security          It Takes More Than Two to Tango: Creating...
2026-07-20  Reuters                China considers tighter export controls on...
2026-03-24  CSIS                   China's Localization Drive in Semiconductors...
2026-07-28  EU Today               China's Domestic DUV Push Puts ASML's Export...
2026-06-24  Purdue University      Intellectual Property Protection and Export...
2025-11-20  Cfr.org                Protecting the Foundation: Strengthening...
2026-07-19  The Motley Fool        Semiconductor Trade Statistics: U.S. Imports...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every row carries &lt;code&gt;title&lt;/code&gt;, &lt;code&gt;url&lt;/code&gt;, &lt;code&gt;publisher&lt;/code&gt;, &lt;code&gt;publishedAt&lt;/code&gt;, &lt;code&gt;summary&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;source&lt;/code&gt;, &lt;code&gt;image&lt;/code&gt; and &lt;code&gt;guid&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;One thing worth knowing if you parse Bing's feed yourself. It carries the&lt;br&gt;
publisher in &lt;code&gt;&amp;lt;News:Source&amp;gt;&lt;/code&gt;, and declares that prefix as&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;xmlns:News="https://www.bing.com/news/search?q=&amp;lt;your query&amp;gt;&amp;amp;format=RSS"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;The namespace URI is the request URL, so it changes on every query and no fixed&lt;br&gt;
namespace constant will ever match it. Match on the local element name instead.&lt;br&gt;
I found this the way you would expect: &lt;code&gt;publisher&lt;/code&gt; was null on exactly the Bing&lt;br&gt;
half of a two-source run while the value was sitting in the feed the whole time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The code
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.request&lt;/span&gt;

&lt;span class="n"&gt;ACTOR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;glitchbound~news-scraper&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;API&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.apify.com/v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;headlines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;per_source&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;searchQueries&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;maxArticlesPerSource&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;per_source&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;language&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;en&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;includeBingNews&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dedupeHeadlines&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/acts/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ACTOR&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/run-sync-get-dataset-items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
           &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;?token=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;APIFY_TOKEN&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;maxTotalChargeUsd=0.50&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;POST&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Content-Type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;application/json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# A feed that fails comes back as a row with an `error` field rather than
&lt;/span&gt;    &lt;span class="c1"&gt;# failing the run, and those are not charged.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;


&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;headlines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;semiconductor export controls&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;per_source&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# Fields can be absent on any given row, so read them defensively rather
&lt;/span&gt;    &lt;span class="c1"&gt;# than subscripting. Feeds are not schemas.
&lt;/span&gt;    &lt;span class="n"&gt;when&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;publishedAt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)[:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no date&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;who&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;publisher&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[:&lt;/span&gt;&lt;span class="mi"&gt;28&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;when&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;who&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;28&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That runs &lt;a href="https://apify.com/glitchbound/news-scraper" rel="noopener noreferrer"&gt;News Scraper&lt;/a&gt; on&lt;br&gt;
&lt;a href="https://apify.com" rel="noopener noreferrer"&gt;Apify&lt;/a&gt;, which is the same forty lines plus the parts that are&lt;br&gt;
tedious rather than hard: both feeds, the nested publisher element, the date&lt;br&gt;
normalisation, and deduplication.&lt;/p&gt;

&lt;p&gt;If you would rather own the code, the endpoint above is public and this post has&lt;br&gt;
told you everything you need.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deduplication is most of the value
&lt;/h2&gt;

&lt;p&gt;One wire story appears on dozens of syndicating domains, and Google and Bing will&lt;br&gt;
both hand you several copies. A hundred raw results is frequently thirty stories.&lt;/p&gt;

&lt;p&gt;Dedupe on the URL first, then on a normalised headline, because the same Reuters&lt;br&gt;
piece appears under slightly different titles across outlets. Doing it on&lt;br&gt;
headline alone merges genuinely different stories that share a phrasing; doing it&lt;br&gt;
on URL alone leaves most of the duplication in place.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this does and does not fit
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Fits:&lt;/strong&gt; brand and competitor monitoring, tracking a topic across outlets,&lt;br&gt;
building a corpus of headlines, seeing which publishers cover a beat, feeding a&lt;br&gt;
summarisation pipeline.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does not fit:&lt;/strong&gt; anything needing article text, anything needing more than a few&lt;br&gt;
months of history, or academic archival work. For those you want a licensed&lt;br&gt;
provider, and the honest answer is that no free endpoint replaces one.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;If you write it yourself, nothing beyond your own hosting.&lt;/p&gt;

&lt;p&gt;Through the Actor it is $1.50 per 1,000 articles on the free tier, so a daily run&lt;br&gt;
pulling 50 articles is about seven cents a month, plus a $0.02 Actor-start event&lt;br&gt;
per run. Duplicates that get dropped are not charged, and neither are feeds that&lt;br&gt;
fail.&lt;/p&gt;

&lt;h2&gt;
  
  
  The wider point
&lt;/h2&gt;

&lt;p&gt;An enormous amount of "you need an API for this" is habit. Google News, Apple's&lt;br&gt;
app charts, the SEC, the World Bank, the ECB and Open Food Facts all publish&lt;br&gt;
usable public endpoints that need no key and get very little attention, largely&lt;br&gt;
because there is no vendor with an incentive to tell you about them.&lt;/p&gt;

&lt;p&gt;Worth checking before you pay for access to public data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;Runnable examples are here:&lt;br&gt;
&lt;a href="https://github.com/danielhagever/apify-data-actors" rel="noopener noreferrer"&gt;github.com/danielhagever/apify-data-actors&lt;/a&gt;&lt;/p&gt;

</description>
      <category>api</category>
      <category>python</category>
      <category>webdev</category>
      <category>datascience</category>
    </item>
    <item>
      <title>WHOIS is gone, RDAP replaced it, and a 404 does not mean what you think</title>
      <dc:creator>Daniel Meshulam</dc:creator>
      <pubDate>Thu, 30 Jul 2026 18:02:58 +0000</pubDate>
      <link>https://dev.to/glitchbound/whois-is-gone-rdap-replaced-it-and-a-404-does-not-mean-what-you-think-3alh</link>
      <guid>https://dev.to/glitchbound/whois-is-gone-rdap-replaced-it-and-a-404-does-not-mean-what-you-think-3alh</guid>
      <description>&lt;p&gt;If you last touched domain lookups a few years ago, the thing you remember is&lt;br&gt;
WHOIS: a plaintext protocol on port 43 that returned an unparseable wall of text&lt;br&gt;
in a different shape from every registry.&lt;/p&gt;

&lt;p&gt;It is effectively gone. RDAP replaced it, it is RFC 9083, it returns JSON, and&lt;br&gt;
every gTLD registry is required to run one. This is one of the rare cases where&lt;br&gt;
the standards process produced something strictly better and nobody noticed.&lt;/p&gt;

&lt;p&gt;Here is what a lookup looks like now, what you can actually get out of it after&lt;br&gt;
GDPR, and the one trap that had my own tool confidently reporting registered&lt;br&gt;
domains as free to buy.&lt;/p&gt;
&lt;h2&gt;
  
  
  What RDAP gives you, and what it does not
&lt;/h2&gt;

&lt;p&gt;Post-GDPR, RDAP returns no registrant personal data for virtually every TLD. If&lt;br&gt;
you are looking for who owns a domain, stop: that information is not there and&lt;br&gt;
no legitimate source has it.&lt;/p&gt;

&lt;p&gt;What is there is the operational picture, and that turns out to be the more&lt;br&gt;
useful half:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;registrar, registration date, expiry date, status codes, DNSSEC&lt;/li&gt;
&lt;li&gt;whether the name is locked against transfer&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Combine that with a DNS lookup and you get the thing people actually want, which&lt;br&gt;
is what a company runs on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;domain        registrar                expires       age mail       dns         dmarc
stripe.com    SafeNames Ltd.           2027-09-11   30.9 google     aws-route53 reject
linear.app    CloudFlare, Inc.         2030-05-09    8.2 google     cloudflare  reject
vercel.com    Amazon Registrar, Inc.   2029-10-04   26.8 google     vercel      quarantine
github.io     -                        -               - -          aws-route53 -
angel.co      -                        -               - google     cloudflare  reject
notion.so     -                        -               - -          cloudflare  quarantine
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is a real run. Mail provider and DNS provider come from MX and NS records,&lt;br&gt;
DMARC policy from a TXT lookup at &lt;code&gt;_dmarc.&amp;lt;domain&amp;gt;&lt;/code&gt;. For sales qualification,&lt;br&gt;
security auditing or competitive research, those three columns are worth more&lt;br&gt;
than any registrant field ever was.&lt;/p&gt;

&lt;p&gt;Note the bottom three rows have DNS but no registrar and no expiry. Hold that&lt;br&gt;
thought.&lt;/p&gt;
&lt;h2&gt;
  
  
  The code
&lt;/h2&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.request&lt;/span&gt;

&lt;span class="n"&gt;ACTOR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;glitchbound~whois-domain-lookup&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;API&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.apify.com/v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;profile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domains&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;domains&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;domains&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;includeDns&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;recordTypes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;A&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;MX&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;NS&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;TXT&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;includeEmailSecurity&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/acts/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ACTOR&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/run-sync-get-dataset-items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
           &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;?token=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;APIFY_TOKEN&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;maxTotalChargeUsd=0.50&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;POST&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Content-Type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;application/json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# A domain that could not be looked up comes back as a row with an `error`
&lt;/span&gt;    &lt;span class="c1"&gt;# field rather than failing the run, and those are not charged.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;


&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;profile&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stripe.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;linear.app&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;vercel.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]):&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;domain&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mailProvider&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dnsProvider&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; dmarc=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dmarcPolicy&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;That runs an &lt;a href="https://apify.com" rel="noopener noreferrer"&gt;Apify&lt;/a&gt; Actor I maintain,&lt;br&gt;
&lt;a href="https://apify.com/glitchbound/whois-domain-lookup" rel="noopener noreferrer"&gt;WHOIS Domain Lookup&lt;/a&gt;, because the&lt;br&gt;
tedious parts are the RDAP bootstrap, the vcard unpacking and the provider&lt;br&gt;
fingerprinting rather than the HTTP call. If you would rather do it yourself,&lt;br&gt;
&lt;code&gt;https://rdap.org/domain/&amp;lt;name&amp;gt;&lt;/code&gt; is the whole API and it needs no key.&lt;/p&gt;
&lt;h2&gt;
  
  
  The trap: a 404 is ambiguous
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;rdap.org&lt;/code&gt; is a router. It reads IANA's bootstrap registry, finds the RDAP&lt;br&gt;
server for your TLD, and redirects. A 404 back from it looks like it means one&lt;br&gt;
thing: no registry holds this name, therefore the domain is available.&lt;/p&gt;

&lt;p&gt;That is what my tool assumed. It was wrong, and the failure is ugly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rdap.org/domain/github.io    404
rdap.org/domain/notion.so    404
rdap.org/domain/segment.io   404
rdap.org/domain/angel.co     404
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All four are registered. All four returned 404.&lt;/p&gt;

&lt;p&gt;The reason is that &lt;strong&gt;IANA's bootstrap does not cover every TLD.&lt;/strong&gt; It lists RDAP&lt;br&gt;
services for around 1,200 of them, and some of the most common startup TLDs are&lt;br&gt;
not in there:&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;.io   .co   .de   .me   .so   .gg   .sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;p&gt;For those, &lt;code&gt;rdap.org&lt;/code&gt; has nowhere to send the query, so it 404s for every name&lt;br&gt;
under them, registered or not. My Actor turned that into &lt;code&gt;isAvailable: true&lt;/code&gt; and&lt;br&gt;
would have told you &lt;code&gt;github.io&lt;/code&gt; was free to register.&lt;/p&gt;

&lt;p&gt;On a tool whose headline feature is availability checking, that is the worst&lt;br&gt;
possible way to be wrong: confident, plausible, and acted upon. Nobody&lt;br&gt;
double-checks a clear answer.&lt;/p&gt;
&lt;h2&gt;
  
  
  The fix, in two parts
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Ask whether there is a registry at all.&lt;/strong&gt; Fetch&lt;br&gt;
&lt;code&gt;https://data.iana.org/rdap/dns.json&lt;/code&gt; once, build the set of TLDs that have an&lt;br&gt;
RDAP service, and only treat a 404 as "available" when the TLD is in it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rdap_tlds&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://data.iana.org/rdap/dns.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;services&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Let DNS settle the rest.&lt;/strong&gt; A name with NS records has been delegated by the&lt;br&gt;
registry, and a registry does not delegate a name nobody registered. So for the&lt;br&gt;
TLDs with no RDAP, NS records prove registration even though RDAP cannot.&lt;/p&gt;

&lt;p&gt;The reverse does not hold, and this is the part worth being careful about: a&lt;br&gt;
registered domain can have no NS at all. So "no NS and no RDAP" is not&lt;br&gt;
"available", it is &lt;strong&gt;unknown&lt;/strong&gt;, and it should be returned as null with an&lt;br&gt;
explanation rather than as a guess. A null is a fact the caller can handle. A&lt;br&gt;
plausible wrong answer is not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this generalises
&lt;/h2&gt;

&lt;p&gt;The bug was not really about RDAP. It was about reading an absence as a&lt;br&gt;
definite answer.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;404&lt;/code&gt; means "I have nothing for you". Whether that is because the thing does not&lt;br&gt;
exist, or because you asked somewhere that was never going to know, is not in&lt;br&gt;
the response. Any time you convert a missing result into a positive claim, check&lt;br&gt;
that you actually asked something capable of answering.&lt;/p&gt;

&lt;p&gt;I have now hit the same shape four times in a month in different tools: an API&lt;br&gt;
silently ignoring a filter it does not support, a page parser taking the first&lt;br&gt;
number in a plausible range, this, and crt.sh, which answers 404 when it is&lt;br&gt;
refusing traffic and reports "nothing found" as an empty list with a 200. All of&lt;br&gt;
them produced output that looked completely normal.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;Charged per domain returned, at $2.00 per 1,000 on the free tier, so a hundred&lt;br&gt;
domains is 20 cents plus a $0.02 Actor-start event. Domains removed by the&lt;br&gt;
built-in filters, like &lt;code&gt;onlyRegistered&lt;/code&gt; or &lt;code&gt;expiringWithinDays&lt;/code&gt;, are not charged,&lt;br&gt;
and neither are lookups that fail.&lt;/p&gt;

&lt;h2&gt;
  
  
  Notes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Some ccTLDs run thin RDAP servers that return fewer fields. You get whatever
the registry publishes rather than an error.&lt;/li&gt;
&lt;li&gt;DNS answers are returned sorted. Resolvers round-robin, and unsorted output
makes a scheduled run look like it changed every time when nothing did.&lt;/li&gt;
&lt;li&gt;Availability comes from the registry itself, not a reseller API, so there is no
incentive for anyone to tell you a name is taken when it is not.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;Runnable examples are here:&lt;br&gt;
&lt;a href="https://github.com/danielhagever/apify-data-actors" rel="noopener noreferrer"&gt;github.com/danielhagever/apify-data-actors&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Every example input in that repo is generated from the Actor's own input schema,&lt;br&gt;
so it is valid by construction rather than by my memory of the field names.&lt;/p&gt;

</description>
      <category>api</category>
      <category>python</category>
      <category>devops</category>
      <category>webdev</category>
    </item>
    <item>
      <title>The App Store top charts are a public API, and nobody archives them</title>
      <dc:creator>Daniel Meshulam</dc:creator>
      <pubDate>Thu, 30 Jul 2026 16:19:14 +0000</pubDate>
      <link>https://dev.to/glitchbound/the-app-store-top-charts-are-a-public-api-and-nobody-archives-them-n7n</link>
      <guid>https://dev.to/glitchbound/the-app-store-top-charts-are-a-public-api-and-nobody-archives-them-n7n</guid>
      <description>&lt;p&gt;Apple's top charts are one of the few genuinely useful public rankings left.&lt;br&gt;
They are free, they need no key, and they update daily.&lt;/p&gt;

&lt;p&gt;They also have one property that makes them frustrating: &lt;strong&gt;there is no history&lt;/strong&gt;.&lt;br&gt;
The feed shows you today. Yesterday is gone. If you want to know whether an app&lt;br&gt;
climbed or slid, or which category a publisher quietly took over last month, that&lt;br&gt;
data does not exist anywhere unless somebody was writing it down.&lt;/p&gt;

&lt;p&gt;Nobody is writing it down, which means the archive is worth more than the feed.&lt;/p&gt;

&lt;p&gt;This post covers both halves: reading the charts today, and the small amount of&lt;br&gt;
work it takes to have a year of history in a year's time.&lt;/p&gt;
&lt;h2&gt;
  
  
  The feed
&lt;/h2&gt;

&lt;p&gt;Apple's RSS feeds sit at a predictable URL per country, chart type and category.&lt;br&gt;
No key, no login, and no bot wall. That is unusual enough in 2026 to be worth&lt;br&gt;
saying out loud.&lt;/p&gt;

&lt;p&gt;What is awkward is the shape. Each entry buries the app id under&lt;br&gt;
&lt;code&gt;id.attributes["im:id"]&lt;/code&gt;, the price under &lt;code&gt;im:price.attributes&lt;/code&gt;, the developer&lt;br&gt;
under &lt;code&gt;im:artist.label&lt;/code&gt;, and the icon in an array where you want the last&lt;br&gt;
element. Parsing it correctly is 30 lines of nested &lt;code&gt;.get()&lt;/code&gt; calls that you write&lt;br&gt;
once and never enjoy.&lt;/p&gt;

&lt;p&gt;I put that in an &lt;a href="https://apify.com" rel="noopener noreferrer"&gt;Apify&lt;/a&gt; Actor,&lt;br&gt;
&lt;a href="https://apify.com/glitchbound/app-charts-scraper" rel="noopener noreferrer"&gt;App Store Top Charts Scraper&lt;/a&gt;,&lt;br&gt;
so I could stop rewriting it. One HTTP call runs it and returns flat rows.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.request&lt;/span&gt;

&lt;span class="n"&gt;ACTOR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;glitchbound~app-charts-scraper&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;API&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.apify.com/v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;top_charts&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;countries&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;charts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top-free&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,),&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;countries&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;countries&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;charts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;charts&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;categories&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;all&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;limit&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/acts/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ACTOR&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/run-sync-get-dataset-items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
           &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;?token=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;APIFY_TOKEN&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;maxTotalChargeUsd=0.50&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;POST&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Content-Type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;application/json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# A country or chart that fails comes back as a row with an `error` field
&lt;/span&gt;    &lt;span class="c1"&gt;# rather than failing the run, and those are not charged. Split them out or a
&lt;/span&gt;    &lt;span class="c1"&gt;# run that fetched nothing still looks successful.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;


&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;top_charts&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;us&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;de&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;charts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top-free&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;country&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])):&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;country&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  #&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rank&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;developer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Real output, from an actual run, top of the German chart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DE  #1   Claude by Anthropic                        Anthropic PBC
DE  #2   ChatGPT                                    OpenAI OpCo, LLC
DE  #3   Nect Wallet                                Nect
DE  #4   Google Gemini                              Google
DE  #5   EasyPark parken - Park App                 Easy Park AS
DE  #6   AusweisApp Bund                            Governikus GmbH &amp;amp; Co. KG
DE  #7   Mein dm                                    dm-drogerie markt GmbH +
DE  #8   Post &amp;amp; DHL                                 Deutsche Post AG
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That German top ten is itself the argument for archiving. Four of those eight are&lt;br&gt;
government or logistics apps that nobody would predict from the US chart, and the&lt;br&gt;
top two are AI assistants that were not in this chart at all two years ago. You&lt;br&gt;
cannot see either of those facts in a single pull.&lt;/p&gt;

&lt;p&gt;Every row also carries &lt;code&gt;appId&lt;/code&gt;, &lt;code&gt;bundleId&lt;/code&gt;, &lt;code&gt;price&lt;/code&gt;, &lt;code&gt;currency&lt;/code&gt;, &lt;code&gt;isFree&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;genre&lt;/code&gt;, &lt;code&gt;releaseDate&lt;/code&gt;, &lt;code&gt;icon&lt;/code&gt; and &lt;code&gt;url&lt;/code&gt;, so a chart pull doubles as an app&lt;br&gt;
metadata pull.&lt;/p&gt;
&lt;h2&gt;
  
  
  The part that is actually valuable
&lt;/h2&gt;

&lt;p&gt;One pull tells you the ranking today, which you could have read off your phone.&lt;br&gt;
The interesting queries all need two pulls separated by time:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which apps entered the top 20 this week&lt;/li&gt;
&lt;li&gt;Which publisher gained the most chart slots this month&lt;/li&gt;
&lt;li&gt;How long a given app has held its position&lt;/li&gt;
&lt;li&gt;Whether a competitor's launch actually moved anything&lt;/li&gt;
&lt;li&gt;Which categories are churning and which are frozen&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of that is answerable from the feed. All of it is answerable from a table&lt;br&gt;
you append to daily.&lt;/p&gt;

&lt;p&gt;So the whole trick is: run it on a schedule and never delete anything.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;top_charts&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;us&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;charts&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top-free&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top-paid&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;stamp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;isoformat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timespec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;seconds&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;capturedAt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;stamp&lt;/span&gt;          &lt;span class="c1"&gt;# the feed has no timestamp; add your own
# append to wherever you keep things: Postgres, SQLite, a CSV, a dataset
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Apify you get this without writing the storage layer: put the Actor on a&lt;br&gt;
Schedule and every run appends to the same dataset, each with its own timestamp.&lt;br&gt;
Three months later you have something nobody sells.&lt;/p&gt;

&lt;p&gt;One detail that matters if you build this yourself: &lt;strong&gt;the feed carries no&lt;br&gt;
timestamp of its own.&lt;/strong&gt; If you do not stamp each capture at write time, a year&lt;br&gt;
later you have a pile of rankings and no way to order them. Stamp it on the way&lt;br&gt;
in, not on the way out.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;Charged per chart entry returned. A 100-app chart is 100 results, so at $3.00 per&lt;br&gt;
1,000 on the free tier a daily 100-app capture for one country is about 30 cents&lt;br&gt;
a month, plus a $0.02 Actor-start event per run. Countries and chart types&lt;br&gt;
multiply that, so pick the markets you actually care about rather than sweeping&lt;br&gt;
all of them.&lt;/p&gt;

&lt;p&gt;Rows that come back as errors are not charged.&lt;/p&gt;

&lt;h2&gt;
  
  
  Caveats worth stating plainly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Apple only.&lt;/strong&gt; Google Play publishes no equivalent public chart feed. Anything
claiming a Play "top chart" API is scraping a rendered page, which breaks on
redesign.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Charts are per country, per chart type, per category.&lt;/strong&gt; An app can be absent
from &lt;code&gt;top-free&lt;/code&gt; while sitting high in &lt;code&gt;top-grossing&lt;/code&gt;. "Not in the chart" is
always relative to which chart you asked for.&lt;/li&gt;
&lt;li&gt;Apple caps a feed at 200 entries. Beyond that the answer does not exist in the
feed.&lt;/li&gt;
&lt;li&gt;Rankings are Apple's, computed however Apple computes them. This reads them; it
does not model them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;Runnable examples, including this one, are here:&lt;br&gt;
&lt;a href="https://github.com/danielhagever/apify-data-actors" rel="noopener noreferrer"&gt;github.com/danielhagever/apify-data-actors&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Every example input in that repo is generated from the Actor's own input schema,&lt;br&gt;
so they are valid by construction rather than by my memory of the field names.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>ios</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Find startups the day they raise, before the press release, with 30 lines of Python</title>
      <dc:creator>Daniel Meshulam</dc:creator>
      <pubDate>Thu, 30 Jul 2026 16:16:48 +0000</pubDate>
      <link>https://dev.to/glitchbound/find-startups-the-day-they-raise-before-the-press-release-with-30-lines-of-python-46o6</link>
      <guid>https://dev.to/glitchbound/find-startups-the-day-they-raise-before-the-press-release-with-30-lines-of-python-46o6</guid>
      <description>&lt;p&gt;Funding announcements are old news by the time you read them. There's usually a&lt;br&gt;
gap of weeks between a company closing a round and telling anyone about it. In&lt;br&gt;
that gap, the round is already a matter of public record.&lt;/p&gt;

&lt;p&gt;That's because of Form D. When a US company sells securities in a private&lt;br&gt;
placement, it has to file one with the SEC, generally within 15 days of the first&lt;br&gt;
sale. It's a legal requirement, it's public the moment it's accepted, and almost&lt;br&gt;
nobody reads it.&lt;/p&gt;

&lt;p&gt;So the question is not &lt;em&gt;whether&lt;/em&gt; you can see funding rounds early. It's how to&lt;br&gt;
filter thousands of Form Ds a month down to the ones you care about.&lt;/p&gt;

&lt;p&gt;This post is the working version of that, end to end: ~30 lines of Python, real&lt;br&gt;
output, and the two traps that cost me a couple of hours.&lt;/p&gt;
&lt;h2&gt;
  
  
  Why you can't just list Form Ds
&lt;/h2&gt;

&lt;p&gt;The naive approach is "give me every Form D from the last 90 days." That's&lt;br&gt;
thousands of filings: real estate syndicates, hedge funds, family offices, a&lt;br&gt;
long tail of LLCs. Most of it is not what anyone means by "a startup raised."&lt;/p&gt;

&lt;p&gt;You also can't filter by company name, because &lt;strong&gt;you don't know the company&lt;br&gt;
names yet&lt;/strong&gt;. That's the entire point of the exercise.&lt;/p&gt;

&lt;p&gt;What you want is a full-text search over the &lt;em&gt;body&lt;/em&gt; of the filings, scoped to&lt;br&gt;
Form D and to a recent window. EDGAR supports exactly that, over every filing&lt;br&gt;
since 2001.&lt;/p&gt;
&lt;h2&gt;
  
  
  The code
&lt;/h2&gt;

&lt;p&gt;I'm using an &lt;a href="https://apify.com" rel="noopener noreferrer"&gt;Apify&lt;/a&gt; Actor I built for EDGAR,&lt;br&gt;
&lt;a href="https://apify.com/glitchbound/sec-filings-scraper" rel="noopener noreferrer"&gt;SEC EDGAR Scraper&lt;/a&gt;, because&lt;br&gt;
it handles the parts of EDGAR that are tedious: CIK resolution, the fair-access&lt;br&gt;
rate limit, XBRL, and the date-range bug I'll get to below. One HTTP call runs it&lt;br&gt;
and returns rows.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.request&lt;/span&gt;

&lt;span class="n"&gt;ACTOR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;glitchbound~sec-filings-scraper&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;API&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.apify.com/v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;form_d_by_sector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;since&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;today&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;dt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;isoformat&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;# Quote the phrase or EDGAR matches the words separately.
&lt;/span&gt;        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;searchQuery&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;sector&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;formTypes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;D&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;filedSince&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;since&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;maxSearchResults&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/acts/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ACTOR&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/run-sync-get-dataset-items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
           &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;?token=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;APIFY_TOKEN&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;amp;maxTotalChargeUsd=0.50&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;POST&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Content-Type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;application/json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# A target the Actor couldn't resolve comes back as a row with an `error`
&lt;/span&gt;    &lt;span class="c1"&gt;# field instead of failing the whole run, and those aren't charged. Count
&lt;/span&gt;    &lt;span class="c1"&gt;# them separately or a run that fetched nothing still looks successful.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;


&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;form_d_by_sector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;artificial intelligence&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;filingDate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;filingDate&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;companyName&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;55&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;state&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;--&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;            &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;filingUrl&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Set &lt;code&gt;APIFY_TOKEN&lt;/code&gt; from&lt;br&gt;
&lt;a href="https://console.apify.com/settings/integrations" rel="noopener noreferrer"&gt;console.apify.com/settings/integrations&lt;/a&gt;&lt;br&gt;
and run it.&lt;/p&gt;
&lt;h2&gt;
  
  
  Real output
&lt;/h2&gt;

&lt;p&gt;This is an actual run, not a mockup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-07-06  Flo Artificial Intelligence, Inc.  (CIK 0001798621)      CA
            .../data/1798621/000179862126000003/primary_doc.xml
2026-06-29  Artificial Intelligence Financial Services Market Inc.   DE
            .../data/2142624/000214262426000001/primary_doc.xml
2026-06-29  Artificial Intelligence Financial Services Market Inc.   DE
            .../data/2142624/000214262426000002/primary_doc.xml
2026-06-18  Panorama Artificial Intelligence Corp.                   WA
            .../data/2001638/000200163826000001/primary_doc.xml
2026-05-26  Artificial Intelligence Progress Research, LP            --
            .../data/2133240/000213324026000002/primary_doc.xml
2026-05-15  Allied Innovation Artificial Intelligence Fund, LLC      NY
            .../data/2021379/000202137926000001/primary_doc.xml
2026-05-05  Artificial Intelligence Progress Research, LP            --
            .../data/2133240/000213324026000001/primary_doc.xml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(URLs truncated for width; the real ones are absolute &lt;code&gt;sec.gov&lt;/code&gt; links.)&lt;/p&gt;

&lt;p&gt;Seven filings, each linking straight to the primary document on sec.gov. Every one&lt;br&gt;
is a real entity that filed a real Form D in that window.&lt;/p&gt;

&lt;p&gt;Note that the same company can appear twice, with two separate accession numbers&lt;br&gt;
on 2026-06-29. Those are genuinely two filings, not a duplicate row: an amended Form&lt;br&gt;
D gets its own accession. Dedupe on &lt;code&gt;companyName&lt;/code&gt; only if you actually want to.&lt;/p&gt;

&lt;p&gt;You'll notice something honest about this list: it is heavy on entities with&lt;br&gt;
"Artificial Intelligence" &lt;em&gt;in the company name&lt;/em&gt;, plus a couple of funds. That's&lt;br&gt;
what a full-text phrase search gives you. It matches the text of the filing, and&lt;br&gt;
Form D is a short form without a business description. It's a strong signal,&lt;br&gt;
not a magic sector classifier. More on that below.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 1: EDGAR silently ignores a half-open date range
&lt;/h2&gt;

&lt;p&gt;This one cost me real money before I caught it.&lt;/p&gt;

&lt;p&gt;EDGAR's full-text endpoint takes &lt;code&gt;startdt&lt;/code&gt; and &lt;code&gt;enddt&lt;/code&gt; alongside&lt;br&gt;
&lt;code&gt;dateRange=custom&lt;/code&gt;. Send only &lt;code&gt;startdt&lt;/code&gt; and it does not narrow anything, and it&lt;br&gt;
does not tell you either. Measured against the live endpoint, same query, &lt;code&gt;forms=D&lt;/code&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Parameters&lt;/th&gt;
&lt;th&gt;Hits&lt;/th&gt;
&lt;th&gt;Oldest result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;startdt&lt;/code&gt; only&lt;/td&gt;
&lt;td&gt;53&lt;/td&gt;
&lt;td&gt;2015-07-28&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;no date params at all&lt;/td&gt;
&lt;td&gt;53&lt;/td&gt;
&lt;td&gt;2015-07-28&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;startdt&lt;/code&gt; + &lt;code&gt;enddt&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;2026-05-05&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;startdt&lt;/code&gt; alone is byte-for-byte identical to sending no filter. If you're paying&lt;br&gt;
per result, you're paying for eleven years of filings outside the window you&lt;br&gt;
asked for, and the response looks perfectly normal.&lt;/p&gt;

&lt;p&gt;If you're calling EDGAR directly: &lt;strong&gt;always send both bounds.&lt;/strong&gt; Default the&lt;br&gt;
missing one: &lt;code&gt;2001-01-01&lt;/code&gt; is the floor for full-text search, and today for the&lt;br&gt;
upper bound. (This is fixed in the Actor as of build 0.1.4; that's why the run&lt;br&gt;
above returns 7 and not 53.)&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 2: "recent" isn't the sort you want for anything else
&lt;/h2&gt;

&lt;p&gt;For funding discovery, newest-first is right. For almost every other EDGAR&lt;br&gt;
question it isn't. If you're looking for the filing that &lt;em&gt;matters&lt;/em&gt; on a topic,&lt;br&gt;
recency is noise. Worth knowing before you build the wrong dashboard.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this costs
&lt;/h2&gt;

&lt;p&gt;The Actor charges per result: $3.00 per 1,000 on the free tier, so seven filings&lt;br&gt;
is 2.1 cents, plus a one-time $0.02 Actor-start event, so about &lt;strong&gt;4 cents&lt;/strong&gt; for&lt;br&gt;
the run. Paid Apify tiers pay less per result, down to half that on Gold.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;maxTotalChargeUsd=0.50&lt;/code&gt; in the code above is a hard&lt;br&gt;
ceiling the platform enforces. The run stops cleanly at the limit rather than&lt;br&gt;
being killed mid-write, so you can't accidentally spend more than you meant to.&lt;/p&gt;

&lt;p&gt;Rows that come back as errors, such as a ticker that doesn't resolve or a query&lt;br&gt;
EDGAR rejects, aren't charged.&lt;/p&gt;

&lt;h2&gt;
  
  
  Going further
&lt;/h2&gt;

&lt;p&gt;A few directions that work well from here:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Watch a sector on a schedule.&lt;/strong&gt; Run it daily with &lt;code&gt;days=2&lt;/code&gt; and diff against&lt;br&gt;
what you've already seen. New Form Ds show up the day they're accepted.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pivot to the company.&lt;/strong&gt; Once a name appears, feed its CIK back in with&lt;br&gt;
&lt;code&gt;includeFinancials: true&lt;/code&gt; to get XBRL figures: revenue, net income, total assets,&lt;br&gt;
straight from what the company tagged, not scraped off a rendered page. A full&lt;br&gt;
20-year history is one request, because it comes from &lt;code&gt;companyfacts&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Search for other things entirely.&lt;/strong&gt; &lt;code&gt;formTypes: ["8-K"]&lt;/code&gt; with a phrase like&lt;br&gt;
&lt;code&gt;"material weakness"&lt;/code&gt; or &lt;code&gt;"going concern"&lt;/code&gt; finds every company that disclosed it.&lt;br&gt;
&lt;code&gt;formTypes: ["4"]&lt;/code&gt; is insider trades. &lt;code&gt;formTypes: ["13F-HR"]&lt;/code&gt; is what the big&lt;br&gt;
funds hold.&lt;/p&gt;

&lt;h2&gt;
  
  
  Caveats worth stating plainly
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Form D is a filing, not a validated funding announcement.&lt;/strong&gt; It tells you
securities were sold under a Reg D exemption. It does not confirm a
venture-style round, and the amounts on the form are frequently the maximum
offering rather than what closed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Full-text phrase search matches text, not sectors.&lt;/strong&gt; As the output shows, it
finds companies whose filings contain your phrase, which correlates with the
sector but isn't the same thing. For real classification you'd want to pull the
filing and read it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;This is public disclosure, and that's the whole appeal.&lt;/strong&gt; No login, no key,
no scraping of rendered pages. The SEC asks callers to identify themselves and
stay under 10 requests/second, which the Actor does.&lt;/li&gt;
&lt;li&gt;Not investment advice, obviously.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;Full runnable scripts, including this one and a few others, are here:&lt;br&gt;
&lt;a href="https://github.com/danielhagever/apify-data-actors" rel="noopener noreferrer"&gt;github.com/danielhagever/apify-data-actors&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Every example input in that repo is generated from the Actor's own input schema,&lt;br&gt;
so they're valid by construction rather than by my memory of the field names.&lt;br&gt;
That's a habit I picked up after the &lt;code&gt;reviewedAt&lt;/code&gt;-vs-&lt;code&gt;date&lt;/code&gt; mistake in a&lt;br&gt;
different script cost me an afternoon of blank output.&lt;/p&gt;

&lt;p&gt;If you find a case where EDGAR behaves differently than described here, I'd&lt;br&gt;
genuinely like to know. That date-range trap only surfaced because I ran the&lt;br&gt;
example instead of just writing it.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>webdev</category>
      <category>datascience</category>
    </item>
  </channel>
</rss>
