<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Pulse Extract</title>
    <description>The latest articles on DEV Community by Pulse Extract (@pulse-extract).</description>
    <link>https://dev.to/pulse-extract</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254880%2F68e192ae-e77c-464d-9814-4e67dd91695b.png</url>
      <title>DEV Community: Pulse Extract</title>
      <link>https://dev.to/pulse-extract</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/pulse-extract"/>
    <language>en</language>
    <item>
      <title>Stop Spinning Up Headless Chrome for Public JSON: HTTP-First Extraction Patterns for Shopify, Multi-ATS Boards, and Substack (with n8n)</title>
      <dc:creator>Pulse Extract</dc:creator>
      <pubDate>Sat, 21 Nov 2026 22:40:27 +0000</pubDate>
      <link>https://dev.to/pulse-extract/stop-spinning-up-headless-chrome-for-public-json-http-first-extraction-patterns-for-shopify-1mi5</link>
      <guid>https://dev.to/pulse-extract/stop-spinning-up-headless-chrome-for-public-json-http-first-extraction-patterns-for-shopify-1mi5</guid>
      <description>&lt;p&gt;A common source of fragility is a cron job that starts a headless browser to parse HTML cards on a site that already serves structured public JSON.&lt;/p&gt;

&lt;p&gt;When a store renames a CSS class or loads prices asynchronously, DOM selectors break silently, browser memory spikes inside small n8n containers, and empty arrays look identical to HTTP 429 rate limits.&lt;/p&gt;

&lt;p&gt;These three HTTP-first patterns cover catalog changes, incomplete runs, and feeds with published transcripts.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Shopify Storefronts: Diff &lt;code&gt;/products.json&lt;/code&gt; by Variant ID, Never by Page Card
&lt;/h2&gt;

&lt;p&gt;Some Shopify storefronts expose a paginated JSON feed at the catalog level. Check availability and usage terms for the store you plan to monitor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;GET https://&amp;lt;store-domain&amp;gt;/products.json?limit=250&amp;amp;page=1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three rules prevent silent monitor bugs when polling &lt;code&gt;/products.json&lt;/code&gt;:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Key your state on immutable variant IDs, not titles or handles.&lt;/strong&gt; When a single size (&lt;code&gt;M&lt;/code&gt; or &lt;code&gt;EU 42&lt;/code&gt;) restocks or changes price, the parent product title and URL never move — only the nested &lt;code&gt;variants[].id&lt;/code&gt;, &lt;code&gt;variants[].available&lt;/code&gt;, and &lt;code&gt;variants[].price&lt;/code&gt; / &lt;code&gt;compare_at_price&lt;/code&gt; flip.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat &lt;code&gt;http://&lt;/code&gt; pasted URLs, percent-encoded handles, and Shopify Markets locales carefully.&lt;/strong&gt; Normalizing pasted store URLs to &lt;code&gt;https://&lt;/code&gt; avoids mixed-content blocks, decoding collection handles preserves non-ASCII paths cleanly, and keeping Markets locale prefixes (like &lt;code&gt;/fr/&lt;/code&gt; or &lt;code&gt;/en-ca/&lt;/code&gt;) stops price checks from reading the wrong currency feed. Virtual collection paths (&lt;code&gt;/collections/vendors?...&lt;/code&gt;, &lt;code&gt;/collections/all&lt;/code&gt;) also behave differently from handle feeds (&lt;code&gt;/collections/{handle}/products.json&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never confuse an HTTP &lt;code&gt;429 Too Many Requests&lt;/code&gt; with an empty catalog.&lt;/strong&gt; Honor &lt;code&gt;Retry-After&lt;/code&gt; headers for a bounded backoff, and flag throttled runs as incomplete (&lt;code&gt;rate_limited&lt;/code&gt;) instead of overwriting last-known stock with zero items.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  2. Multi-ATS Job Boards: Separate "Zero Openings" from Lookup Failures
&lt;/h2&gt;

&lt;p&gt;Many public job boards on Greenhouse, Lever, Ashby, Workable, or SmartRecruiters expose structured job listings. Check which endpoint is available for each provider before choosing a browser workflow.&lt;/p&gt;

&lt;p&gt;One failure mode in job trackers is &lt;strong&gt;overwriting state on a partial failure&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If &lt;code&gt;source A&lt;/code&gt; returns 220 jobs cleanly and &lt;code&gt;source B&lt;/code&gt; times out or returns HTTP &lt;code&gt;403&lt;/code&gt;, overwriting your snapshot as &lt;code&gt;0 open roles&lt;/code&gt; for &lt;code&gt;source B&lt;/code&gt; triggers bogus "closed" removals on the current run and bogus "new role" alerts on the next healthy run.&lt;/li&gt;
&lt;li&gt;Always pair &lt;code&gt;(ats_provider, board_slug, immutable_job_id)&lt;/code&gt; as the deduplication key (titles and URL slugs get edited mid-posting) and isolate per-board error states from genuine empty boards (&lt;code&gt;200 OK&lt;/code&gt; with &lt;code&gt;[]&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  3. Substack &amp;amp; Audio Archives: Exhaust XML/JSON Metadata Before Touching Whisper or Browsers
&lt;/h2&gt;

&lt;p&gt;Two checks can avoid a browser launch or an audio transcription job when the publisher already provides the content:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Substack:&lt;/strong&gt; RSS feeds are great for new issues, while archive endpoints/sitemaps let you backfill older posts as clean text/Markdown without rendering full browser pages (preserving inline links and separating subscriber-only previews from full free essays cleanly).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Podcast transcripts:&lt;/strong&gt; Always inspect &lt;code&gt;&amp;lt;podcast:transcript&amp;gt;&lt;/code&gt; tags inside the show's RSS &lt;code&gt;&amp;lt;item&amp;gt;&lt;/code&gt; nodes before scheduling an audio download + Whisper pass. Shows that use the tag can link to timestamped VTT/SRT or JSON transcripts directly from the feed.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Plug-and-Play n8n Templates &amp;amp; Hosted Pay-Per-Event Endpoints
&lt;/h2&gt;

&lt;p&gt;We open-sourced MIT-licensed n8n JSON workflow templates (&lt;code&gt;Shopify Price Monitor&lt;/code&gt;, &lt;code&gt;Multi-ATS Startup Jobs Tracker&lt;/code&gt;, &lt;code&gt;Vinted Underpriced Comps Alert&lt;/code&gt;) plus Claude Desktop / Cursor MCP configs on GitHub:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Open Templates &amp;amp; MCP Configs:&lt;/strong&gt; &lt;a href="https://github.com/atelierprojetscontact/pulse-extract-templates" rel="noopener noreferrer"&gt;https://github.com/atelierprojetscontact/pulse-extract-templates&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hosted HTTP-first Actors &amp;amp; Example Tasks (pay-per-event; check each listing for current pricing):&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://apify.com/pulse_extract/multi-ats-jobs-scraper" rel="noopener noreferrer"&gt;Multi-ATS Jobs Scraper&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://apify.com/pulse_extract/shopify-catalog-monitor" rel="noopener noreferrer"&gt;Shopify Catalog, Price &amp;amp; Stock Monitor&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://apify.com/pulse_extract/vinted-catalog-scraper" rel="noopener noreferrer"&gt;Vinted EU &amp;amp; UK Catalog &amp;amp; Price Comps Scraper&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://apify.com/pulse_extract/substack-newsletter-scraper" rel="noopener noreferrer"&gt;Substack Newsletter &amp;amp; Archive Scraper&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://apify.com/pulse_extract/podcast-transcript-extractor" rel="noopener noreferrer"&gt;Podcast &amp;amp; Audio RSS Transcript Extractor&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Example cURL request to a hosted endpoint that returns normalized JSON. Running the request uses your Apify account and can incur charges:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -X POST \
  &lt;span class="s2"&gt;"https://api.apify.com/v2/acts/pulse_extract~multi-ats-jobs-scraper/run-sync-get-dataset-items"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"Authorization: Bearer $APIFY_TOKEN"&lt;/span&gt; \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{"targets":["ashby:linear"],"maxJobsPerCompany":5,"includeDescription":false}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Keep heavy HTML or JSON diffing out of your main n8n memory process, persist explicit failure states separate from empty arrays, and reserve browser rendering for pages that need client-side JavaScript and permit automated access.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>automation</category>
      <category>javascript</category>
    </item>
  </channel>
</rss>
