<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Rafi-siddiq</title>
    <description>The latest articles on DEV Community by Rafi-siddiq (@rafisiddiq).</description>
    <link>https://dev.to/rafisiddiq</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4219347%2F1df0d8b6-f686-4e9b-a90b-d80f1b87b1ab.png</url>
      <title>DEV Community: Rafi-siddiq</title>
      <link>https://dev.to/rafisiddiq</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/rafisiddiq"/>
    <language>en</language>
    <item>
      <title>Build a ZIP file in Node.js with zero dependencies (and four bugs I hit downloading files in bulk)</title>
      <dc:creator>Rafi-siddiq</dc:creator>
      <pubDate>Mon, 09 Nov 2026 04:35:03 +0000</pubDate>
      <link>https://dev.to/rafisiddiq/build-a-zip-file-in-nodejs-with-zero-dependencies-and-four-bugs-i-hit-downloading-files-in-bulk-14dh</link>
      <guid>https://dev.to/rafisiddiq/build-a-zip-file-in-nodejs-with-zero-dependencies-and-four-bugs-i-hit-downloading-files-in-bulk-14dh</guid>
      <description>&lt;p&gt;I recently built a bulk file downloader: a list of URLs in, every file stored, plus one ZIP of everything at the end. The ZIP part turned out to need no library at all on modern Node, and the downloading part had more traps than I expected. Here are both. &lt;em&gt;Disclosure: the finished tool is linked at the end, and I built it.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  A ZIP writer in about 60 lines
&lt;/h2&gt;

&lt;p&gt;A ZIP file is simple: for each file, a 30-byte &lt;strong&gt;local header&lt;/strong&gt;, the name, and the data. Then a &lt;strong&gt;central directory&lt;/strong&gt; with a 46-byte entry per file, and a 22-byte &lt;strong&gt;end record&lt;/strong&gt; pointing at it. Every number is little-endian.&lt;/p&gt;

&lt;p&gt;The only hard part used to be CRC-32, and Node 22.2 added &lt;code&gt;zlib.crc32&lt;/code&gt;. With &lt;code&gt;deflateRawSync&lt;/code&gt; for compression, that's everything:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;crc32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;deflateRawSync&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;node:zlib&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;buildZip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;time&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;now&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getUTCHours&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;now&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getUTCMinutes&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;now&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getUTCSeconds&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;date&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;now&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getUTCFullYear&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1980&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;now&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getUTCMonth&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nx"&gt;now&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getUTCDate&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;locals&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;centrals&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
    &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;data&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;nameBuf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;utf8&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;deflated&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;deflateRawSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;useDeflate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;deflated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// JPGs and PNGs are already compressed&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;useDeflate&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;deflated&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;crc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;crc32&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;head&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;alloc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mh"&gt;0x04034b50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;   &lt;span class="c1"&gt;// local file header signature&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;           &lt;span class="c1"&gt;// version needed&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mh"&gt;0x0800&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;       &lt;span class="c1"&gt;// flag: names are UTF-8&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;useDeflate&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;crc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// compressed size&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// uncompressed size&lt;/span&gt;
        &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;nameBuf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;26&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;locals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;nameBuf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;cen&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;alloc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;46&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mh"&gt;0x02014b50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;    &lt;span class="c1"&gt;// central directory signature&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mh"&gt;0x0800&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;useDeflate&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;crc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;nameBuf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;28&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;offset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;       &lt;span class="c1"&gt;// where this file's local header starts&lt;/span&gt;
        &lt;span class="nx"&gt;centrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;cen&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;nameBuf&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nx"&gt;head&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;nameBuf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;cd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;concat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;centrals&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;alloc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mh"&gt;0x06054b50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt16LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;cd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeUInt32LE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;offset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;concat&lt;/span&gt;&lt;span class="p"&gt;([...&lt;/span&gt;&lt;span class="nx"&gt;locals&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;cd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Limits: this is classic ZIP, not ZIP64, so stay under 4 GB and 65,535 files, and it builds in memory. I capped mine at 200 MB. I tested it by extracting the output with Windows' built-in &lt;code&gt;Expand-Archive&lt;/code&gt;, which rejects bad CRCs, and comparing every file's SHA-256 with the original.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four bugs from downloading files in bulk
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Duplicate detection raced itself.&lt;/strong&gt; I hashed each file with SHA-256 and skipped hashes I'd already stored. With ten downloads in parallel, two copies of the same PDF both checked the map &lt;em&gt;before&lt;/em&gt; either finished uploading, so both were stored. The fix is to claim the hash (and the storage key) synchronously, before the first &lt;code&gt;await&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;storageKey&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;fileName&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;contentType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;taken&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;hashes&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;??=&lt;/span&gt; &lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;                       &lt;span class="c1"&gt;// claim first&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setValue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;contentType&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt; &lt;span class="c1"&gt;// then upload&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;2. Don't trust Content-Type.&lt;/strong&gt; Plenty of servers send images as &lt;code&gt;application/octet-stream&lt;/code&gt; or &lt;code&gt;binary/octet-stream&lt;/code&gt;. Check the first bytes instead: &lt;code&gt;ffd8ff&lt;/code&gt; is JPEG, &lt;code&gt;89504e47&lt;/code&gt; is PNG, &lt;code&gt;%PDF&lt;/code&gt; is PDF, &lt;code&gt;RIFF....WEBP&lt;/code&gt; is WebP. Then use the URL's extension, and only then the header.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Cap the size while streaming, not after.&lt;/strong&gt; &lt;code&gt;Content-Length&lt;/code&gt; is often missing, so read &lt;code&gt;res.body&lt;/code&gt; chunk by chunk and cancel once you pass the limit. Otherwise one 4 GB video takes down the whole run.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Your storage may "helpfully" parse files.&lt;/strong&gt; When building the ZIP, I read each stored file back from Apify's key-value store, and the JSON file came back as a parsed object rather than bytes, which crashed &lt;code&gt;deflateRawSync&lt;/code&gt;. Ask for the raw buffer explicitly (&lt;code&gt;getRecord(key, { buffer: true })&lt;/code&gt; in the Apify client).&lt;/p&gt;

&lt;h2&gt;
  
  
  The finished tool
&lt;/h2&gt;

&lt;p&gt;All of this is packaged as &lt;strong&gt;&lt;a href="https://apify.com/rsiddiq95/bulk-file-downloader" rel="noopener noreferrer"&gt;Bulk File &amp;amp; Image Downloader&lt;/a&gt;&lt;/strong&gt; on Apify: paste URLs, or point it at another scraper's dataset, and you get every file stored with its name, type, size and SHA-256, duplicates skipped, optional type and size filters, retries on 429 and 5xx errors, and &lt;code&gt;ALL_FILES.zip&lt;/code&gt; at the end. It costs $2 per 1,000 files, and failed, skipped and duplicate URLs are free.&lt;/p&gt;

</description>
      <category>node</category>
      <category>javascript</category>
      <category>webscraping</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>How to check domain age, expiry and DMARC for thousands of domains with free official APIs</title>
      <dc:creator>Rafi-siddiq</dc:creator>
      <pubDate>Mon, 09 Nov 2026 04:34:14 +0000</pubDate>
      <link>https://dev.to/rafisiddiq/how-to-check-domain-age-expiry-and-dmarc-for-thousands-of-domains-with-free-official-apis-4pco</link>
      <guid>https://dev.to/rafisiddiq/how-to-check-domain-age-expiry-and-dmarc-for-thousands-of-domains-with-free-official-apis-4pco</guid>
      <description>&lt;p&gt;WHOIS is on its way out. For generic domains like .com, .net and .org, ICANN now requires registries and registrars to answer over &lt;strong&gt;RDAP&lt;/strong&gt;, the Registration Data Access Protocol: plain HTTPS that returns JSON instead of free-form text that every registry formats differently.&lt;/p&gt;

&lt;p&gt;Put RDAP together with DNS over HTTPS and the Wayback Machine's CDX API, and you can answer most "what is this domain?" questions for free, from official sources, without parsing a single WHOIS blob. This post shows each API, the gotchas I hit, and a hosted version that does all of it in bulk. &lt;em&gt;Disclosure: I built the tool at the end.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Registration date, registrar and expiry: RDAP
&lt;/h2&gt;

&lt;p&gt;There's no single RDAP server. Each top-level domain has its own, and IANA publishes the list:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl https://data.iana.org/rdap/dns.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;services&lt;/code&gt; is a list of &lt;code&gt;[[tlds], [base URLs]]&lt;/code&gt;. For &lt;code&gt;com&lt;/code&gt; it points to Verisign:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl https://rdap.verisign.com/com/v1/domain/stripe.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The answer is structured JSON. The useful parts:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;rdap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://rdap.verisign.com/com/v1/domain/stripe.com&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;date&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;action&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;rdap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;eventAction&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;action&lt;/span&gt;&lt;span class="p"&gt;)?.&lt;/span&gt;&lt;span class="nx"&gt;eventDate&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;registrar&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;rdap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;entities&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;e&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;roles&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;registrar&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;

&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;registered&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;registration&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="c1"&gt;// 1995-09-12&lt;/span&gt;
    &lt;span class="na"&gt;expires&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;expiration&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;      &lt;span class="c1"&gt;// 2027-09-11&lt;/span&gt;
    &lt;span class="na"&gt;registrar&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;registrar&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;vcardArray&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;fn&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="c1"&gt;// SafeNames Ltd.&lt;/span&gt;
    &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;rdap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;nameservers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;rdap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;nameservers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;n&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ldhName&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toLowerCase&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
    &lt;span class="na"&gt;dnssec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;rdap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;secureDNS&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;delegationSigned&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Gotchas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A 404 means the domain isn't registered.&lt;/strong&gt; That makes RDAP a clean availability check too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not every country domain has RDAP.&lt;/strong&gt; &lt;code&gt;.io&lt;/code&gt;, &lt;code&gt;.es&lt;/code&gt; and &lt;code&gt;.so&lt;/code&gt; weren't in IANA's bootstrap file when I checked, so for those you only get DNS-level facts. Don't fall back to guessing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use the registrable domain&lt;/strong&gt;, not the host: &lt;code&gt;news.bbc.co.uk&lt;/code&gt; → &lt;code&gt;bbc.co.uk&lt;/code&gt;, so you need a small list of two-level suffixes like &lt;code&gt;co.uk&lt;/code&gt; and &lt;code&gt;com.au&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Owner names and emails are mostly redacted for privacy. You get dates, registrar and status, not people.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  2. Email provider, SPF and DMARC: DNS over HTTPS
&lt;/h2&gt;

&lt;p&gt;Google's public resolver answers DNS queries as JSON:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="s2"&gt;"https://dns.google/resolve?name=stripe.com&amp;amp;type=MX"&lt;/span&gt;
curl &lt;span class="s2"&gt;"https://dns.google/resolve?name=_dmarc.stripe.com&amp;amp;type=TXT"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From MX hosts you can tell the email provider: &lt;code&gt;aspmx.l.google.com&lt;/code&gt; is Google Workspace, &lt;code&gt;*.mail.protection.outlook.com&lt;/code&gt; is Microsoft 365, &lt;code&gt;*.pphosted.com&lt;/code&gt; is Proofpoint, and so on. The SPF record is the TXT record starting with &lt;code&gt;v=spf1&lt;/code&gt;, and the DMARC policy is the &lt;code&gt;p=&lt;/code&gt; tag on &lt;code&gt;_dmarc.&amp;lt;domain&amp;gt;&lt;/code&gt;: &lt;code&gt;none&lt;/code&gt;, &lt;code&gt;quarantine&lt;/code&gt; or &lt;code&gt;reject&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Two details that cost me time: long TXT records come back as several quoted chunks (&lt;code&gt;"v=spf1 include:a" " -all"&lt;/code&gt;) that you have to join, and the answer list can contain CNAME records mixed in with the type you asked for, so filter by the numeric &lt;code&gt;type&lt;/code&gt; (MX is 15, TXT is 16).&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Site history: the Wayback Machine CDX API
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# first capture&lt;/span&gt;
curl &lt;span class="s2"&gt;"https://web.archive.org/cdx/search/cdx?url=stripe.com&amp;amp;output=json&amp;amp;limit=1&amp;amp;fl=timestamp,original&amp;amp;filter=statuscode:200"&lt;/span&gt;
&lt;span class="c"&gt;# most recent capture&lt;/span&gt;
curl &lt;span class="s2"&gt;"https://web.archive.org/cdx/search/cdx?url=stripe.com&amp;amp;output=json&amp;amp;limit=-1&amp;amp;fastLatest=true&amp;amp;fl=timestamp,original&amp;amp;filter=statuscode:200"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Row 0 is the header; row 1 is the capture, with a &lt;code&gt;yyyyMMddHHmmss&lt;/code&gt; timestamp. The archive link is &lt;code&gt;https://web.archive.org/web/{timestamp}/{original}&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The interesting signal is comparing this with the RDAP date. &lt;code&gt;apify.com&lt;/code&gt; was registered (in its current registration) in &lt;strong&gt;2009&lt;/strong&gt;, but its first Wayback capture is from &lt;strong&gt;2007&lt;/strong&gt;. A first capture years before the registration date usually means the name had an earlier owner, which matters if you're buying aged or expired domains.&lt;/p&gt;

&lt;p&gt;The CDX API is slow and rate-limited, so pace it (about one request a second) and back off on errors.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Website status and SSL certificate
&lt;/h2&gt;

&lt;p&gt;A normal &lt;code&gt;fetch&lt;/code&gt; with &lt;code&gt;redirect: 'follow'&lt;/code&gt; gives you the status code, final URL and page title. For the certificate, Node's &lt;code&gt;tls&lt;/code&gt; module reads it directly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;tls&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;node:tls&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;socket&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;tls&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;host&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;stripe.com&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;servername&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;stripe.com&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;cert&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getPeerCertificate&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;cert&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;issuer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;O&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;cert&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;valid_to&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// DigiCert, Inc.  May 2027&lt;/span&gt;
    &lt;span class="nx"&gt;socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Doing it in bulk
&lt;/h2&gt;

&lt;p&gt;Each source needs its own pacing (RDAP servers and the Wayback Machine are much stricter than Google DNS), and you want one row per domain, not four API responses. I packaged all of the above as an Apify Actor, &lt;strong&gt;&lt;a href="https://apify.com/rsiddiq95/domain-intelligence" rel="noopener noreferrer"&gt;Domain Intelligence&lt;/a&gt;&lt;/strong&gt;: paste domains, URLs or email addresses, and each gets one row with registration date, age, registrar, expiry, status, name servers, DNSSEC, IPs, MX hosts, email provider, SPF, DMARC policy, first and last Wayback capture, HTTP status, final URL, title, and SSL issuer and days left. It costs $5 per 1,000 domains and invalid entries are free.&lt;/p&gt;

&lt;p&gt;If you'd rather run it yourself, everything above is free and official. The main work is the pacing and the edge cases.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>dns</category>
      <category>api</category>
      <category>javascript</category>
    </item>
    <item>
      <title>How to get every open job from a company's careers page (Greenhouse, Lever, Ashby)</title>
      <dc:creator>Rafi-siddiq</dc:creator>
      <pubDate>Fri, 06 Nov 2026 16:17:08 +0000</pubDate>
      <link>https://dev.to/rafisiddiq/how-to-get-every-open-job-from-a-companys-careers-page-greenhouse-lever-ashby-c7j</link>
      <guid>https://dev.to/rafisiddiq/how-to-get-every-open-job-from-a-companys-careers-page-greenhouse-lever-ashby-c7j</guid>
      <description>&lt;p&gt;Most tech companies don't host their own careers page. It's run by an applicant tracking system, usually &lt;strong&gt;Greenhouse&lt;/strong&gt;, &lt;strong&gt;Lever&lt;/strong&gt; or &lt;strong&gt;Ashby&lt;/strong&gt;. You can tell by the address: &lt;code&gt;boards.greenhouse.io/airbnb&lt;/code&gt;, &lt;code&gt;jobs.lever.co/palantir&lt;/code&gt;, &lt;code&gt;jobs.ashbyhq.com/ramp&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The useful part: all three publish a free, public job-board API so companies can show their jobs on their own sites. No login, no scraping HTML, no blocking. If you want a company's open roles in a spreadsheet, a job alert, or a sales prospect list, this is the cleanest source there is.&lt;/p&gt;

&lt;p&gt;This post shows the three APIs, the differences between them, and a hosted tool that merges them into one format. &lt;em&gt;Disclosure: I built the tool at the end.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The three APIs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Greenhouse&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="s2"&gt;"https://boards-api.greenhouse.io/v1/boards/airbnb/jobs?content=true"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Returns &lt;code&gt;{ "jobs": [...] }&lt;/code&gt; with title, location, departments, offices, &lt;code&gt;first_published&lt;/code&gt;, &lt;code&gt;updated_at&lt;/code&gt; and the job page URL. Two gotchas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;content&lt;/code&gt; (the description) is HTML that is &lt;strong&gt;itself HTML-escaped&lt;/strong&gt;, so you have to decode it before stripping tags.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Salary isn't in the list.&lt;/strong&gt; Greenhouse only returns pay ranges on each job's own endpoint, with &lt;code&gt;?pay_transparency=true&lt;/code&gt;:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="s2"&gt;"https://boards-api.greenhouse.io/v1/boards/airbnb/jobs/&amp;lt;job id&amp;gt;?pay_transparency=true"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That gives &lt;code&gt;pay_input_ranges&lt;/code&gt; in cents, for example &lt;code&gt;min_cents: 17900000&lt;/code&gt;, &lt;code&gt;max_cents: 21000000&lt;/code&gt;, &lt;code&gt;currency_type: "USD"&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lever&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="s2"&gt;"https://api.lever.co/v0/postings/palantir?mode=json"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Returns a plain array. The title is in &lt;code&gt;text&lt;/code&gt;, location, team and commitment are under &lt;code&gt;categories&lt;/code&gt;, and &lt;code&gt;createdAt&lt;/code&gt; is in milliseconds. European companies use &lt;code&gt;api.eu.lever.co&lt;/code&gt;. Salary is in &lt;code&gt;salaryRange&lt;/code&gt; when the company sets it, which many don't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ashby&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="s2"&gt;"https://api.ashbyhq.com/posting-api/job-board/ramp?includeCompensation=true"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Returns &lt;code&gt;{ "jobs": [...] }&lt;/code&gt; with department, team, &lt;code&gt;employmentType&lt;/code&gt; (&lt;code&gt;FullTime&lt;/code&gt;, &lt;code&gt;PartTime&lt;/code&gt;, …), &lt;code&gt;workplaceType&lt;/code&gt;, secondary locations and, with &lt;code&gt;includeCompensation=true&lt;/code&gt;, a ready-made summary like &lt;code&gt;"$189K – $330K • Offers Equity"&lt;/code&gt;. Watch for titles with a leading space.&lt;/p&gt;

&lt;h2&gt;
  
  
  The annoying parts
&lt;/h2&gt;

&lt;p&gt;If you're collecting from many companies, you end up writing the same glue:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Three different shapes&lt;/strong&gt; to map into one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Finding the system.&lt;/strong&gt; A company name like &lt;code&gt;airbnb&lt;/code&gt; could be on any of the three, so you try each.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Same slug, different company.&lt;/strong&gt; &lt;code&gt;ramp&lt;/code&gt; on Ashby is the fintech with 171 jobs. &lt;code&gt;ramp&lt;/code&gt; on Greenhouse is a different company with one job. Guess wrong and your data is wrong.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Greenhouse salaries&lt;/strong&gt; need one extra request per job.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Filtering&lt;/strong&gt; by title, location, department, remote and date, consistently across all three.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  A hosted version that merges all three
&lt;/h2&gt;

&lt;p&gt;I wrapped this up as the &lt;a href="https://apify.com/rsiddiq95/career-pages-jobs-scraper" rel="noopener noreferrer"&gt;Company Career Pages Jobs Scraper&lt;/a&gt; on Apify. You give it careers-page URLs or just company names:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"companies"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"airbnb"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ramp"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://jobs.lever.co/palantir"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"keywords"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"data engineer"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"data scientist"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"locations"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"United States"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Remote"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"postedSince"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-09-01"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A plain name is checked on all three systems. If it exists on more than one, it uses the board with the most open jobs and says so in the log; a URL always picks the exact board.&lt;/p&gt;

&lt;p&gt;Every job comes back in the same shape. A real row from that run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"company"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Airbnb"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"system"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"greenhouse"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Senior Staff Data Engineer, Foundational Data"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"location"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"United States"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"workplaceType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Remote"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"isRemote"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"salary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"USD 248,000–310,000 (Pay Range)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"postedAt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-09-22T18:58:28.000Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"jobUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://careers.airbnb.com/positions/8224032?gh_jid=8224032"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"applyUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://careers.airbnb.com/positions/8224032?gh_jid=8224032"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Other fields: &lt;code&gt;department&lt;/code&gt;, &lt;code&gt;team&lt;/code&gt;, &lt;code&gt;allLocations&lt;/code&gt;, &lt;code&gt;employmentType&lt;/code&gt;, &lt;code&gt;updatedAt&lt;/code&gt;, and the full description as plain text if you turn it on.&lt;/p&gt;

&lt;p&gt;From code, one HTTP call runs it and returns the jobs (keep your Apify token in an environment variable):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/rsiddiq95~career-pages-jobs-scraper/run-sync-get-dataset-items"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &lt;/span&gt;&lt;span class="nv"&gt;$APIFY_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"companies": ["airbnb", "ramp"], "keywords": ["engineer"], "remoteOnly": true}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It costs $5 per 1,000 jobs. Companies that aren't found, or have nothing matching your filters, are free. A 200-job company takes a few seconds.&lt;/p&gt;

&lt;h2&gt;
  
  
  What people use this for
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Job alerts for a shortlist of companies.&lt;/strong&gt; Schedule it daily with &lt;code&gt;postedSince&lt;/code&gt; set to yesterday.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sales prospecting.&lt;/strong&gt; Companies hiring "RevOps", "data engineer" or "security engineer" are telling you what they're investing in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recruiting and market research.&lt;/strong&gt; Which teams a company is growing, where, and at what pay.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;From an AI assistant.&lt;/strong&gt; It also runs as a tool in Claude, ChatGPT or Cursor through &lt;a href="https://docs.apify.com/platform/integrations/mcp" rel="noopener noreferrer"&gt;Apify's MCP server&lt;/a&gt;: add &lt;code&gt;https://mcp.apify.com?tools=rsiddiq95/career-pages-jobs-scraper&lt;/code&gt; and ask "which engineering jobs is Airbnb hiring for in the US, and what do they pay?"&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;Only companies on Greenhouse, Lever or Ashby. Workday, iCIMS, SmartRecruiters and others aren't covered. If a company's careers page is on its own domain, click any job: the apply link usually reveals which system it uses.&lt;/p&gt;

&lt;p&gt;Questions or a system you'd like added? Open an issue on the Actor page.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>api</category>
      <category>javascript</category>
      <category>career</category>
    </item>
    <item>
      <title>A pytrends alternative that handles Google Trends rate limits</title>
      <dc:creator>Rafi-siddiq</dc:creator>
      <pubDate>Sat, 31 Oct 2026 17:32:09 +0000</pubDate>
      <link>https://dev.to/rafisiddiq/a-pytrends-alternative-that-handles-google-trends-rate-limits-537p</link>
      <guid>https://dev.to/rafisiddiq/a-pytrends-alternative-that-handles-google-trends-rate-limits-537p</guid>
      <description>&lt;p&gt;For years, the default way to get Google Trends data in code was &lt;a href="https://github.com/GeneralMills/pytrends" rel="noopener noreferrer"&gt;pytrends&lt;/a&gt;. Today it's a frustrating choice: the repository is archived, it no longer gets fixes, and many people now see &lt;code&gt;429 Too Many Requests&lt;/code&gt; on the very first call.&lt;/p&gt;

&lt;p&gt;This post covers why that happens, what your options are, and how to get Trends data without managing it yourself. &lt;em&gt;Disclosure: the hosted option at the end is a tool I built.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Google Trends returns 429s
&lt;/h2&gt;

&lt;p&gt;Google Trends has no public API that anyone can use (more on the official one below). Libraries like pytrends call the same internal endpoints the Trends website uses. Google limits how often those endpoints can be called, and it doesn't publish the limits. When you go over, you get a 429 and have to wait.&lt;/p&gt;

&lt;p&gt;The limits are stricter for traffic from cloud servers than from home connections, which is why a script that works on your laptop often fails on a server or in a notebook environment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your options
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Keep a library and slow it down.&lt;/strong&gt; Space requests out by several seconds, retry with exponential backoff (wait 2 s, then 4 s, then 8 s…), and cache results so you don't ask twice. This works for small, occasional jobs. It gets painful when you need hundreds of keywords.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Apply for the official Google Trends API.&lt;/strong&gt; Google opened an &lt;a href="https://developers.google.com/search/apis/trends" rel="noopener noreferrer"&gt;alpha of an official Trends API&lt;/a&gt; in 2025. It returns consistently scaled data, which the website doesn't. Access is by application, it's limited to approved testers, and there's no general release date yet. If Trends data matters to your product, apply now.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Use a hosted scraper.&lt;/strong&gt; Someone else deals with retries, pacing, and changes to the site, and you call an API.&lt;/p&gt;

&lt;h2&gt;
  
  
  Using a hosted Google Trends scraper
&lt;/h2&gt;

&lt;p&gt;I built the &lt;a href="https://apify.com/rsiddiq95/google-trends-scraper" rel="noopener noreferrer"&gt;Google Trends Scraper&lt;/a&gt; on Apify for my own keyword research. For each search term it returns:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Interest over time:&lt;/strong&gt; the 0–100 timeline for any period, from the past hour to 2004 onward, or exact dates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Interest by region:&lt;/strong&gt; countries, states, cities, or US metro areas (DMA).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Related searches:&lt;/strong&gt; top and rising queries, with a flag for "Breakout" (growth over 5,000%).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Comparisons:&lt;/strong&gt; up to 5 terms on the same scale.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trending Now:&lt;/strong&gt; today's trending searches per country, with traffic estimates and news links.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It paces itself, backs off when Google rate-limits it, and only charges for terms that return data: failed lookups and terms with no data are free.&lt;/p&gt;

&lt;h3&gt;
  
  
  From the web UI
&lt;/h3&gt;

&lt;p&gt;Enter your terms, a location such as &lt;code&gt;US&lt;/code&gt; or &lt;code&gt;US-CA&lt;/code&gt;, and a time range, then click &lt;strong&gt;Start&lt;/strong&gt;. You can download the results as JSON, CSV or Excel.&lt;/p&gt;

&lt;h3&gt;
  
  
  From code
&lt;/h3&gt;

&lt;p&gt;Apify can run it and return the results in one HTTP call. Get your API token from &lt;strong&gt;Settings → API &amp;amp; Integrations&lt;/strong&gt; in Apify Console, and keep it out of your code and repositories (use an environment variable).&lt;/p&gt;

&lt;p&gt;With curl:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/rsiddiq95~google-trends-scraper/run-sync-get-dataset-items"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &lt;/span&gt;&lt;span class="nv"&gt;$APIFY_TOKEN&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"searchTerms": ["air fryer", "standing desk"], "geo": "US", "timeRange": "today 12-m"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With Python:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.apify.com/v2/acts/rsiddiq95~google-trends-scraper/run-sync-get-dataset-items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Authorization&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Bearer &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;APIFY_TOKEN&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;searchTerms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;air fryer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;standing desk&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;geo&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;US&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timeRange&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;today 12-m&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;resultType&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;keyword&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;searchTerm&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;averageInterest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each keyword comes back as one record:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"resultType"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"keyword"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"hasData"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"searchTerm"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"air fryer"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"geo"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"timeRange"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"today 12-m"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"averageInterest"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"interestOverTime"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"date"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2025-10-26T00:00:00.000Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;52&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"isPartial"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"interestByRegion"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"geoCode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"US-WY"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"geoName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Wyoming"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"relatedQueries"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"top"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"query"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"air fryer chicken"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"rising"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"query"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"best air fryer 2026"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"formattedValue"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Breakout"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"isBreakout"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"trendsUrl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://trends.google.com/trends/explore?q=air+fryer&amp;amp;date=today+12-m&amp;amp;geo=US"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It costs $2.50 per 1,000 keywords and $1 per 1,000 trending searches, with no platform fees on top. Apify's free plan includes monthly credit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things to know about Trends data (whatever tool you use)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Values are relative.&lt;/strong&gt; 100 is the peak within your query, not a search volume. If you want two keywords on the same scale, compare them in one query.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Results can shift slightly between runs,&lt;/strong&gt; because Google re-scales each request. Use exact date ranges for reports you want to reproduce.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Related topics aren't available&lt;/strong&gt; from my scraper, because Google doesn't provide them to automated tools.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Classic Explore is going away.&lt;/strong&gt; Google shows a notice on the Trends site that the "classic Explore" page is expiring soon, with no date announced. The new version requires signing in. Every unofficial Trends tool reads the classic version, so expect disruption when it shuts down. That's another reason to apply for the official API if you depend on this data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you have questions or need a feature, open an issue on the Actor page.&lt;/p&gt;

</description>
      <category>python</category>
      <category>javascript</category>
      <category>seo</category>
      <category>webscraping</category>
    </item>
    <item>
      <title>How to track competitors' Google ads in a Google Sheet automatically</title>
      <dc:creator>Rafi-siddiq</dc:creator>
      <pubDate>Sat, 31 Oct 2026 17:31:44 +0000</pubDate>
      <link>https://dev.to/rafisiddiq/how-to-track-competitors-google-ads-in-a-google-sheet-automatically-1nnj</link>
      <guid>https://dev.to/rafisiddiq/how-to-track-competitors-google-ads-in-a-google-sheet-automatically-1nnj</guid>
      <description>&lt;p&gt;If you run paid search, you've probably opened the &lt;a href="https://adstransparency.google.com/" rel="noopener noreferrer"&gt;Google Ads Transparency Center&lt;/a&gt; to see what a competitor is running. It's useful, but it shows one ad at a time, and there's no export. If you want to notice when a competitor changes their messaging, you have to remember to check.&lt;/p&gt;

&lt;p&gt;This tutorial sets up a small pipeline that does it for you:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Every morning, collect the latest Google ads for a list of competitors, including the ad copy.&lt;/li&gt;
&lt;li&gt;Write them to a Google Sheet, updating ads you already have instead of duplicating them.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It takes about 15 minutes and runs on &lt;a href="https://apify.com" rel="noopener noreferrer"&gt;Apify&lt;/a&gt;. &lt;em&gt;Disclosure: I built both Actors used here.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What you'll end up with
&lt;/h2&gt;

&lt;p&gt;A sheet with one row per ad, kept up to date:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Column&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;advertiserName&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Nike, Inc.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;format&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;TEXT&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adText&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Nike, nike.com, Nike® Official Site, Shop exclusive new styles…&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;firstShownAt&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;2023-11-17T00:26:00Z&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;lastShownAt&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;2026-10-30T00:29:17Z&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;daysShown&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;911&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;landingDomain&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;nike.com&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adUrl&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;link to the ad in the Transparency Center&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Because &lt;code&gt;lastShownAt&lt;/code&gt; updates on every run, sorting by it shows which ads are still live, and new rows at the bottom are ads that appeared since yesterday.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Collect the ads
&lt;/h2&gt;

&lt;p&gt;Open the &lt;a href="https://apify.com/rsiddiq95/google-ads-transparency-scraper" rel="noopener noreferrer"&gt;Google Ads Transparency Scraper&lt;/a&gt; and click &lt;strong&gt;Try for free&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;You can find a competitor's ads three ways:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Website:&lt;/strong&gt; &lt;code&gt;competitor.com&lt;/code&gt;. This also finds resellers and agencies running ads that point to the same site.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verified advertiser name:&lt;/strong&gt; for example &lt;code&gt;Nike, Inc.&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Advertiser ID or URL:&lt;/strong&gt; paste a link from the Transparency Center.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here's an input that tracks two competitors' US search ads:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"advertiserDomains"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"competitor-one.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"competitor-two.com"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"platform"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"SEARCH"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"maxAdsPerTarget"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"extractAdCopy"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ads come back newest first, so 50 per competitor is plenty for a daily check: anything new shows up at the top. For the very first run you might raise it to a few hundred to get the full history, then lower it again.&lt;/p&gt;

&lt;p&gt;Click &lt;strong&gt;Start&lt;/strong&gt;. 100 ads take about a minute.&lt;/p&gt;

&lt;p&gt;A note on ad copy: Google stores a text preview for many ads, but some only as images. When there's no text preview, &lt;code&gt;adText&lt;/code&gt; is empty and you still get the image URL.&lt;/p&gt;

&lt;p&gt;When the run finishes, click &lt;strong&gt;Save as a new task&lt;/strong&gt; and name it something like &lt;code&gt;competitor-ads-daily&lt;/code&gt;. A task is just a saved input you can schedule.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Create a Google service account (one time)
&lt;/h2&gt;

&lt;p&gt;To write to your sheet, the sync Actor uses a Google service account: a separate login that can only open sheets you share with it.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;In &lt;a href="https://console.cloud.google.com/" rel="noopener noreferrer"&gt;Google Cloud Console&lt;/a&gt;, create a project.&lt;/li&gt;
&lt;li&gt;Go to &lt;strong&gt;APIs &amp;amp; Services → Library&lt;/strong&gt; and enable &lt;strong&gt;Google Sheets API&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Go to &lt;strong&gt;APIs &amp;amp; Services → Credentials → Create credentials → Service account&lt;/strong&gt;, give it a name, and click &lt;strong&gt;Done&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Open the service account, go to &lt;strong&gt;Keys → Add key → Create new key → JSON&lt;/strong&gt;. A key file downloads.&lt;/li&gt;
&lt;li&gt;Create a Google Sheet, click &lt;strong&gt;Share&lt;/strong&gt;, and add the service account's email (the &lt;code&gt;client_email&lt;/code&gt; in the key file) as an &lt;strong&gt;Editor&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Step 3: Send the ads to the sheet automatically
&lt;/h2&gt;

&lt;p&gt;Open your &lt;code&gt;competitor-ads-daily&lt;/code&gt; task, go to &lt;strong&gt;Integrations → Add integration → Apify Actor&lt;/strong&gt;, and pick &lt;a href="https://apify.com/rsiddiq95/google-sheets-sync" rel="noopener noreferrer"&gt;Google Sheets Sync&lt;/a&gt;. Use this input:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"upsert"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"spreadsheet"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://docs.google.com/spreadsheets/d/your-sheet-id/edit"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"sheetName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Competitor ads"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"datasetId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"{{resource.defaultDatasetId}}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"keyColumns"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"creativeId"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"omitFields"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"previewScriptUrl"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"resultType"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"serviceAccountKey"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"paste the contents of your key file here"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What each part does:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;upsert&lt;/code&gt; with &lt;code&gt;creativeId&lt;/code&gt; as the key&lt;/strong&gt; means an ad already in the sheet gets its row updated (new &lt;code&gt;lastShownAt&lt;/code&gt;, new &lt;code&gt;daysShown&lt;/code&gt;) instead of being added again. New ads are appended.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;{{resource.defaultDatasetId}}&lt;/code&gt;&lt;/strong&gt; is filled in by Apify with the results of each run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Columns you add yourself,&lt;/strong&gt; such as a "Notes" column, are left alone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Values are written as plain text,&lt;/strong&gt; so ad text that happens to start with &lt;code&gt;=&lt;/code&gt; can't run as a formula in your sheet.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The key is stored encrypted in Apify.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: Schedule it
&lt;/h2&gt;

&lt;p&gt;Go to &lt;strong&gt;Schedules → Create new&lt;/strong&gt;, set it to run daily (for example at 7 AM), and add the &lt;code&gt;competitor-ads-daily&lt;/code&gt; task. That's it: the scraper runs, and when it finishes, the sync writes the results to your sheet.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;Both Actors charge per result, and you don't pay Apify platform fees on top:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ads: $3 per 1,000 ads. Two competitors at 50 ads each, every day, is about 3,000 ads a month, or about $9.&lt;/li&gt;
&lt;li&gt;Sheets sync: $0.05 per 1,000 rows, so about 15 cents a month for the same data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Checking weekly instead of daily cuts that to about $2 a month.&lt;/p&gt;

&lt;p&gt;Lookups that find nothing aren't charged, and Apify's free plan includes monthly credit to try it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ideas to extend it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Spot new messaging:&lt;/strong&gt; if &lt;code&gt;firstShownAt&lt;/code&gt; is in column F, add a column with &lt;code&gt;=IF(DATEVALUE(LEFT(F2,10))&amp;gt;TODAY()-7,"NEW","")&lt;/code&gt; to flag ads first shown in the last week.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Track YouTube creative:&lt;/strong&gt; change &lt;code&gt;platform&lt;/code&gt; to &lt;code&gt;YOUTUBE&lt;/code&gt; and &lt;code&gt;adFormat&lt;/code&gt; to &lt;code&gt;VIDEO&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch your own brand:&lt;/strong&gt; search your domain to see who else runs ads pointing to your site.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you try it and something's missing, open an issue on the Actor page. I read every one.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>marketing</category>
      <category>googleads</category>
      <category>automation</category>
    </item>
  </channel>
</rss>
