<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ruslan Shkarin</title>
    <description>The latest articles on DEV Community by Ruslan Shkarin (@rslndev).</description>
    <link>https://dev.to/rslndev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4194398%2F630c9463-22ae-4b0b-9cb2-c58a554cb0a1.JPG</url>
      <title>DEV Community: Ruslan Shkarin</title>
      <link>https://dev.to/rslndev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/rslndev"/>
    <language>en</language>
    <item>
      <title>JSON-LD, __NEXT_DATA__ and __PRELOADED_STATE__: Where E-commerce Data Actually Lives</title>
      <dc:creator>Ruslan Shkarin</dc:creator>
      <pubDate>Tue, 27 Oct 2026 16:45:23 +0000</pubDate>
      <link>https://dev.to/rslndev/json-ld-nextdata-and-preloadedstate-where-e-commerce-data-actually-lives-5bm3</link>
      <guid>https://dev.to/rslndev/json-ld-nextdata-and-preloadedstate-where-e-commerce-data-actually-lives-5bm3</guid>
      <description>&lt;p&gt;Modern store pages ship their data twice: once as HTML for the shopper, and once as JSON for the JavaScript that runs the page. The JSON is usually richer, cleaner and more stable than the markup. Finding it is the single biggest shortcut in e-commerce scraping, as long as you know which of the several JSON blobs to trust.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key takeaways&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Look in four places, in order: JSON-LD, embedded app state, rendered HTML, and APIs called after load.&lt;/li&gt;
&lt;li&gt;JSON-LD is easiest to parse but often absent: of five big US retailers we checked, only Best Buy had a schema.org &lt;code&gt;Product&lt;/code&gt; with a price.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;__NEXT_DATA__&lt;/code&gt; exists only on Next.js "Pages Router" sites. App Router sites stream data through &lt;code&gt;self.__next_f.push()&lt;/code&gt; instead.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;__PRELOADED_STATE__&lt;/code&gt; is a naming convention from the Redux docs, not a standard. Each site's state has its own shape.&lt;/li&gt;
&lt;li&gt;Pages carry many prices. On Best Buy, the first &lt;code&gt;regularPrice&lt;/code&gt; in the source belonged to a mouse in a recommendations widget, not the laptop on the page.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What we found on five big US retailers
&lt;/h2&gt;

&lt;p&gt;On October 8, 2026 we fetched one product page from each of five large US retailers in a regular browser session and inspected the server HTML, before any client-side JavaScript ran.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Retailer&lt;/th&gt;
&lt;th&gt;JSON-LD on the product page&lt;/th&gt;
&lt;th&gt;Embedded state&lt;/th&gt;
&lt;th&gt;Price in server HTML?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Walmart&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;WebPage&lt;/code&gt; only, no Product&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;__NEXT_DATA__&lt;/code&gt;, 523 KB&lt;/td&gt;
&lt;td&gt;Yes, inside &lt;code&gt;__NEXT_DATA__&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Target&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;__NEXT_DATA__&lt;/code&gt; (156 KB) + &lt;code&gt;__TGT_DATA__&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;No. Loaded later from an API&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Best Buy&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;Product&lt;/code&gt; with price, &lt;code&gt;BreadcrumbList&lt;/code&gt;, &lt;code&gt;ItemList&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;App Router flight data (&lt;code&gt;self.__next_f&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Yes, in both&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Amazon&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;Inline JavaScript for variations&lt;/td&gt;
&lt;td&gt;Yes, in the HTML price block&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Newegg&lt;/td&gt;
&lt;td&gt;Breadcrumbs, images, FAQ, item lists; no Product&lt;/td&gt;
&lt;td&gt;&lt;code&gt;window.__initialState__&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Yes, in state and HTML&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;One product page per retailer, viewed without login. Structures change; treat this as a snapshot.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Three different conclusions fall out of five sites. Structured data is not something you can count on at the biggest stores. Next.js is common, but in two different flavors. And at least one major retailer, Target, doesn't put the price in the initial page at all.&lt;/p&gt;

&lt;p&gt;Server HTML weight for a single product page, which is what you actually pay for per gigabyte of proxy traffic:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Retailer&lt;/th&gt;
&lt;th&gt;Server HTML&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Amazon&lt;/td&gt;
&lt;td&gt;1,988 KB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Best Buy&lt;/td&gt;
&lt;td&gt;1,332 KB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Walmart&lt;/td&gt;
&lt;td&gt;711 KB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Newegg&lt;/td&gt;
&lt;td&gt;450 KB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Target&lt;/td&gt;
&lt;td&gt;339 KB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On Walmart, 74% of that HTML was the &lt;code&gt;__NEXT_DATA__&lt;/code&gt; payload.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. JSON-LD: the standard, when it exists
&lt;/h2&gt;

&lt;p&gt;JSON-LD is structured data in a &lt;code&gt;&amp;lt;script type="application/ld+json"&amp;gt;&lt;/code&gt; tag, using schema.org vocabulary. Retailers add it for search engines; Google's &lt;a href="https://developers.google.com/search/docs/appearance/structured-data/product" rel="noopener noreferrer"&gt;Product structured data guide&lt;/a&gt; lists price, availability, ratings and shipping among the supported properties. When it's present it's the cleanest source on the page: documented field names, typed values, one object per product.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;bs4&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BeautifulSoup&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;jsonld_products&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;soup&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;BeautifulSoup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;html.parser&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;found&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;soup&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find_all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;script&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;application/ld+json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;string&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;JSONDecodeError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;                          &lt;span class="c1"&gt;# broken JSON-LD is common
&lt;/span&gt;        &lt;span class="n"&gt;items&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;@graph&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;types&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;@type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;types&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;types&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Product&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;types&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ProductGroup&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Price semantics: which price is it?
&lt;/h3&gt;

&lt;p&gt;A JSON-LD &lt;code&gt;offers&lt;/code&gt; block can hold more than one price. Google's &lt;a href="https://developers.google.com/search/docs/appearance/structured-data/merchant-listing" rel="noopener noreferrer"&gt;merchant listing documentation&lt;/a&gt; distinguishes the active selling price from a strikethrough (previous) price and from member-only prices. A parser that grabs the first &lt;code&gt;price&lt;/code&gt; it sees can record a crossed-out price as current, or a loyalty price as the public one. Read the price specification type, not just the number, and keep each type in its own column.&lt;/p&gt;

&lt;p&gt;Also watch for &lt;code&gt;AggregateOffer&lt;/code&gt; with &lt;code&gt;lowPrice&lt;/code&gt; and &lt;code&gt;highPrice&lt;/code&gt;: that's a range across variants or sellers, not a price.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. &lt;code&gt;__NEXT_DATA__&lt;/code&gt;: Next.js Pages Router
&lt;/h2&gt;

&lt;p&gt;Sites built on Next.js with the older Pages Router serialize each page's data into a single JSON script: &lt;code&gt;&amp;lt;script id="__NEXT_DATA__" type="application/json"&amp;gt;&lt;/code&gt;. The browser uses it to hydrate React; you can use it to skip HTML parsing entirely.&lt;/p&gt;

&lt;p&gt;These payloads get big. Next.js itself warns when page data exceeds 128 kB, according to its &lt;a href="https://nextjs.org/docs/messages/large-page-data" rel="noopener noreferrer"&gt;large-page-data notice&lt;/a&gt;. That's a performance warning, not a limit: the Walmart page we checked carried 523 KB. Big payloads are good news for completeness and bad news for bandwidth.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;bs4&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BeautifulSoup&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;next_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;BeautifulSoup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;html.parser&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;script&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__NEXT_DATA__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;find_paths&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Yield every JSON path that ends in `key`. Use it to explore.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
            &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="nf"&gt;find_paths&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="nf"&gt;find_paths&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;[&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# for p, v in find_paths(next_data(html), "currentPrice"): print(p, v)
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exploration with a helper like &lt;code&gt;find_paths&lt;/code&gt; is how you discover that Walmart's price sits at &lt;code&gt;props.pageProps.initialData.data.product.priceInfo.currentPrice&lt;/code&gt;. Once you know the path, hard-code it and alert when it disappears.&lt;/p&gt;

&lt;h3&gt;
  
  
  No &lt;code&gt;__NEXT_DATA__&lt;/code&gt;? It may be the App Router
&lt;/h3&gt;

&lt;p&gt;Next.js's newer App Router uses React Server Components and doesn't emit &lt;code&gt;__NEXT_DATA__&lt;/code&gt;. Instead the page contains many small scripts calling &lt;code&gt;self.__next_f.push([...])&lt;/code&gt;, which together carry a serialized component tree in React's own wire format. Best Buy's product page worked this way in our check. The data is there, including the price, but it's not one JSON document: values are split across chunks and reference each other.&lt;/p&gt;

&lt;p&gt;For quick extraction, people search the concatenated chunk strings for a known key. That works, and it's exactly where the most common mistake happens.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The first price on the page is often someone else's.&lt;/strong&gt; On the Best Buy laptop page we checked, the first &lt;code&gt;"regularPrice"&lt;/code&gt; in the source was &lt;code&gt;79.99&lt;/code&gt;. It belonged to an Apple Magic Mouse in a recommendations block. The laptop's own &lt;code&gt;customerPrice&lt;/code&gt; was &lt;code&gt;1099&lt;/code&gt;, matching the JSON-LD. Always anchor extraction to the product's ID or SKU, never to the first match of a key name.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For anything beyond spot checks, use a parser built for the format (the open-source &lt;code&gt;njsparser&lt;/code&gt; Python package is one) and select objects by product identifier.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. &lt;code&gt;__PRELOADED_STATE__&lt;/code&gt; and other window globals
&lt;/h2&gt;

&lt;p&gt;The name &lt;code&gt;__PRELOADED_STATE__&lt;/code&gt; comes from Redux's &lt;a href="https://redux.js.org/usage/server-rendering" rel="noopener noreferrer"&gt;server rendering guide&lt;/a&gt;, which shows the server writing the store's initial state into &lt;code&gt;window.__PRELOADED_STATE__&lt;/code&gt;, escaping &lt;code&gt;&amp;lt;&lt;/code&gt; to prevent script injection. Many sites followed the pattern under their own names: &lt;code&gt;__INITIAL_STATE__&lt;/code&gt;, &lt;code&gt;__APOLLO_STATE__&lt;/code&gt; for Apollo GraphQL caches, &lt;code&gt;__NUXT__&lt;/code&gt; or &lt;code&gt;__NUXT_DATA__&lt;/code&gt; for Nuxt, Newegg's &lt;code&gt;window.__initialState__&lt;/code&gt;, Target's &lt;code&gt;__TGT_DATA__&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;There is no shared schema. Newegg's state carried &lt;code&gt;"FinalPrice":279.99&lt;/code&gt; for the processor we checked; Target's held location variables and no price. Some are plain object literals, others are JSON strings wrapped in &lt;code&gt;JSON.parse("...")&lt;/code&gt; with escaped quotes. A generic extractor handles the common case and leaves the rest to per-site code:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="n"&gt;STATE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;window\.(__[A-Za-z0-9_]+__)\s*=\s*&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;window_states&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;JSONDecoder&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;STATE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;finditer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;                          &lt;span class="c1"&gt;# JSON.parse("..."), function, etc.
&lt;/span&gt;        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raw_decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# stops at the end of the object
&lt;/span&gt;            &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;JSONDecodeError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;pass&lt;/span&gt;                              &lt;span class="c1"&gt;# JS literal, not strict JSON
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;raw_decode&lt;/code&gt; parses one JSON value starting at a position and ignores what follows, so you don't need to find the closing brace yourself. It fails on JavaScript-only syntax such as unquoted keys or &lt;code&gt;undefined&lt;/code&gt;; for those, a small JavaScript runtime or a per-site regex is the pragmatic fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. When the price isn't in the page at all
&lt;/h2&gt;

&lt;p&gt;Target's server HTML for the product we checked contained no dollar amounts at all. The price arrives afterwards from an internal API keyed by the item and the store. Open the browser's developer tools, filter the Network tab by "Fetch/XHR," reload, and search responses for the price you see on screen.&lt;/p&gt;

&lt;p&gt;Internal APIs return compact, well-structured JSON, which makes them tempting. They are also undocumented, often require tokens or visitor IDs issued by the page, and tend to be the most closely watched endpoints on a site. Use them when the page gives you no alternative, keep volumes low, and expect them to change.&lt;/p&gt;

&lt;h2&gt;
  
  
  A working order of operations
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;View source (not the inspector) and search for the price you see on screen.&lt;/li&gt;
&lt;li&gt;If it's in JSON-LD with a clear price type, start there.&lt;/li&gt;
&lt;li&gt;Otherwise look for &lt;code&gt;__NEXT_DATA__&lt;/code&gt;, &lt;code&gt;self.__next_f&lt;/code&gt; or &lt;code&gt;window.__…__&lt;/code&gt; state, and find the product object by its ID.&lt;/li&gt;
&lt;li&gt;Fall back to rendered HTML selectors only for fields that exist nowhere else.&lt;/li&gt;
&lt;li&gt;Where two sources exist, extract both and alert when they disagree. Disagreement is your earliest warning that a layout changed or a promotion started.&lt;/li&gt;
&lt;li&gt;Record the store or location you found in the same payload — embedded state often reveals a location you never set, and prices on grocery and home-improvement sites depend on it.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Quick answers
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What is &lt;code&gt;__NEXT_DATA__&lt;/code&gt;?&lt;/strong&gt;&lt;br&gt;
A JSON script that Next.js Pages Router sites embed in every server-rendered page. It contains the props used to render the page, which on stores usually includes product, price and availability data.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why can't I find &lt;code&gt;__NEXT_DATA__&lt;/code&gt; on a Next.js site?&lt;/strong&gt;&lt;br&gt;
The site probably uses the App Router, which streams React Server Component data through &lt;code&gt;self.__next_f.push()&lt;/code&gt; calls instead of one JSON document.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is JSON-LD price data reliable?&lt;/strong&gt;&lt;br&gt;
It's reliable when present and correctly typed, but many large retailers don't publish a Product block, and offers can mix current, strikethrough and member prices. Check the price type and cross-check against the visible page.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is &lt;code&gt;__PRELOADED_STATE__&lt;/code&gt;?&lt;/strong&gt;&lt;br&gt;
A convention from the Redux documentation for passing a server-generated store state to the browser. Sites use it and similar names, but each has its own structure; there's no standard product schema.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do I need a headless browser to read embedded JSON?&lt;/strong&gt;&lt;br&gt;
No. Embedded JSON is in the server HTML, so a plain HTTP request is enough. You need a browser only when data is loaded after the page renders and you can't call that source directly.&lt;/p&gt;




&lt;p&gt;We run scrapers for e-commerce data at JsonEngine, which is why these five pages got taken apart in the first place.&lt;/p&gt;

&lt;p&gt;If you've hit a site where the data lives somewhere stranger than these four places, we'd like to hear about it in the comments.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>nextjs</category>
      <category>webdev</category>
    </item>
    <item>
      <title>We Tested Shopify’s /products.json on 68 Stores: The Limits Tutorials Don’t Mention</title>
      <dc:creator>Ruslan Shkarin</dc:creator>
      <pubDate>Mon, 19 Oct 2026 15:23:14 +0000</pubDate>
      <link>https://dev.to/rslndev/we-tested-shopifys-productsjson-on-68-stores-the-limits-tutorials-dont-mention-1ofj</link>
      <guid>https://dev.to/rslndev/we-tested-shopifys-productsjson-on-68-stores-the-limits-tutorials-dont-mention-1ofj</guid>
      <description>&lt;p&gt;Shopify merchants processed &lt;a href="https://www.shopify.com/news/shopify-q4-2025-financial-results" rel="noopener noreferrer"&gt;$378.4 billion in gross merchandise volume&lt;/a&gt; in 2025, and a large share of US direct-to-consumer brands sell through it. Most of those stores answer a simple request, &lt;code&gt;/products.json&lt;/code&gt;, with their full public catalog in JSON.&lt;/p&gt;

&lt;p&gt;Tutorials explain the endpoint in two paragraphs. We wanted numbers, so we tested it on 68 brand domains.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;50 of 68&lt;/strong&gt; brand domains returned a usable catalog from &lt;code&gt;/products.json&lt;/code&gt;. The rest blocked us, returned 404 or didn't answer a cross-origin request.&lt;/li&gt;
&lt;li&gt;No store ever returned &lt;strong&gt;more than 250 products per page&lt;/strong&gt;, even when we asked for 500.&lt;/li&gt;
&lt;li&gt;There is a &lt;strong&gt;hard ceiling&lt;/strong&gt;: when page × limit passes 25,000, Shopify answers HTTP 400 with &lt;code&gt;"Page * Limit exceeds the 25000 limit."&lt;/code&gt; Two stores hit it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;4 of 50 stores returned short pages&lt;/strong&gt; before the end of the catalog. A loop that stops at the first short page would have lost up to 93% of one store's products.&lt;/li&gt;
&lt;li&gt;Prices come in the &lt;strong&gt;shop's base currency&lt;/strong&gt; (we saw USD, CAD, GBP and EUR), not the currency a shopper in another country would see.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The collector and the raw results are on GitHub: &lt;a href="https://github.com/JsonEngine/shopify-products-json" rel="noopener noreferrer"&gt;JsonEngine/shopify-products-json&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What /products.json is, and what it isn't
&lt;/h2&gt;

&lt;p&gt;Shopify stores expose several ways to read product data. They are easy to confuse because they return similar JSON, and advice written about one is often applied to another.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgt2zbdej90deadw90grt.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgt2zbdej90deadw90grt.webp" alt="Comparison of three Shopify interfaces: public /products.json storefront feed, the Ajax Product API at /products/handle.js, and the authenticated Admin API" width="800" height="400"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Only the first lists a whole catalog without credentials. Pagination rules from the Admin API docs do not apply to it.&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;/products.json&lt;/code&gt;&lt;/strong&gt; is the public storefront feed. No credentials, paginated with &lt;code&gt;?limit=&lt;/code&gt; and &lt;code&gt;?page=&lt;/code&gt;. It is not formally documented as an API, which is why its limits are folklore. That's what we measured.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;/products/{handle}.js&lt;/code&gt;&lt;/strong&gt; is the &lt;a href="https://shopify.dev/docs/api/ajax/reference/product" rel="noopener noreferrer"&gt;Ajax Product API&lt;/a&gt;, documented by Shopify. One product per request, prices in cents, and monetary values in the buyer's presentment currency.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Admin API&lt;/strong&gt; needs the merchant's access token. Its &lt;a href="https://shopify.dev/docs/api/admin-rest/usage/pagination" rel="noopener noreferrer"&gt;pagination docs&lt;/a&gt; describe cursor-based &lt;code&gt;page_info&lt;/code&gt; paging with a 250-item maximum. That is a different interface; don't expect &lt;code&gt;page_info&lt;/code&gt; on the public feed.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  Fields you get
&lt;/h3&gt;

&lt;p&gt;Each product carries &lt;code&gt;id&lt;/code&gt;, &lt;code&gt;title&lt;/code&gt;, &lt;code&gt;handle&lt;/code&gt;, &lt;code&gt;body_html&lt;/code&gt;, &lt;code&gt;vendor&lt;/code&gt;, &lt;code&gt;product_type&lt;/code&gt;, &lt;code&gt;tags&lt;/code&gt;, &lt;code&gt;created_at&lt;/code&gt;, &lt;code&gt;published_at&lt;/code&gt;, &lt;code&gt;updated_at&lt;/code&gt;, &lt;code&gt;options&lt;/code&gt;, &lt;code&gt;images&lt;/code&gt; and &lt;code&gt;variants&lt;/code&gt;. Each variant carries &lt;code&gt;id&lt;/code&gt;, &lt;code&gt;sku&lt;/code&gt;, &lt;code&gt;price&lt;/code&gt;, &lt;code&gt;compare_at_price&lt;/code&gt;, &lt;code&gt;available&lt;/code&gt;, &lt;code&gt;option1&lt;/code&gt; to &lt;code&gt;option3&lt;/code&gt;, &lt;code&gt;grams&lt;/code&gt; and timestamps.&lt;/p&gt;

&lt;p&gt;Equally important is what's missing: no inventory quantity (only an &lt;code&gt;available&lt;/code&gt; true/false), no currency field, no reviews, no barcode/GTIN, and nothing about prices for other markets or B2B catalogs.&lt;/p&gt;
&lt;h2&gt;
  
  
  Our test: 68 brand stores, one afternoon
&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Method.&lt;/strong&gt; On October 8, 2026 we requested &lt;code&gt;/products.json?limit=250&amp;amp;page=N&lt;/code&gt; from 68 well-known US-facing brand domains we believed to run on Shopify, from a desktop browser in Europe. For each store that answered, we paged until an empty page or an error, waited 250 ms between pages, recorded page sizes and unique product IDs, compared the total with the store's &lt;code&gt;/meta.json&lt;/code&gt; product count, and repeated page 1 with &lt;code&gt;limit=500&lt;/code&gt;. One pass per store, no retries. We did not verify the platform of the 18 domains that didn't answer, so don't read the failure share as a Shopify-wide rate. The raw results are in the &lt;a href="https://github.com/JsonEngine/shopify-products-json/tree/main/data" rel="noopener noreferrer"&gt;repo's data folder&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Domains that returned a usable catalog&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;50 / 68&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Maximum products per page (even with &lt;code&gt;limit=500&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;250&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hard cap on page × limit (HTTP 400 beyond it)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;25,000&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Median catalog size among the 50 stores&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;366&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh7xs2aviqn0swpruvmlx.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh7xs2aviqn0swpruvmlx.webp" alt="Bar chart of 50 Shopify brand stores grouped by catalog size: 9 under 100 products, 21 with 100 to 499, 6 with 500 to 999, 10 with 1,000 to 4,999, 2 with 5,000 to 24,999, and 2 at the 25,000 cap" width="800" height="363"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Most brand catalogs are small enough to fetch in one to four requests. Two large fashion stores hit the 25,000-product ceiling.&lt;/em&gt;&lt;/p&gt;
&lt;h3&gt;
  
  
  The 25,000-product ceiling
&lt;/h3&gt;

&lt;p&gt;Kith and Fashion Nova returned full pages of 250 all the way to page 100. Page 101 returned HTTP 400 with this body:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"errors"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"Page * Limit exceeds the 25000 limit."&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both stores' &lt;code&gt;/meta.json&lt;/code&gt; reported &lt;code&gt;25001&lt;/code&gt; published products, which looks like a capped counter rather than a real total. In other words, for the largest Shopify stores the public feed cannot tell you how many products exist, let alone list them all.&lt;/p&gt;

&lt;p&gt;Products came back newest first: on Kith, page 1 started with items created the previous day. So the 25,000 you can reach are the most recent ones, and older catalog depth is what you lose.&lt;/p&gt;

&lt;h3&gt;
  
  
  Short pages that aren't the end
&lt;/h3&gt;

&lt;p&gt;Most tutorials say "stop when a page returns fewer items than the limit." On 46 of the 50 stores that works. On four it doesn't:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frqz387ns0a6zydhnqbm0.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frqz387ns0a6zydhnqbm0.webp" alt="Bar chart of products returned per page by ridge.com for limit=250: 58, 190, 163, 208, 229, 7 and 0 on pages one through seven" width="800" height="399"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The first page held 58 products. The catalog had 855.&lt;/em&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Store&lt;/th&gt;
&lt;th&gt;Items per page (limit=250)&lt;/th&gt;
&lt;th&gt;Real total&lt;/th&gt;
&lt;th&gt;Lost by "stop on short page"&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;ridge.com&lt;/td&gt;
&lt;td&gt;58, 190, 163, 208, 229, 7, 0&lt;/td&gt;
&lt;td&gt;855&lt;/td&gt;
&lt;td&gt;797 (93%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;drsquatch.com&lt;/td&gt;
&lt;td&gt;221, 213, 62, 0&lt;/td&gt;
&lt;td&gt;496&lt;/td&gt;
&lt;td&gt;275 (55%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;mejuri.com&lt;/td&gt;
&lt;td&gt;250, 248, 206, 0&lt;/td&gt;
&lt;td&gt;704&lt;/td&gt;
&lt;td&gt;206 (29%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;rains.com&lt;/td&gt;
&lt;td&gt;230, 78, 0&lt;/td&gt;
&lt;td&gt;308&lt;/td&gt;
&lt;td&gt;78 (25%)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Our best guess is that these stores filter some products out of each page after pagination is applied, for example items hidden from the online store channel, so a page of 250 slots arrives partly empty. That's our interpretation of the pattern, not something Shopify documents. Whatever the cause, the fix is simple: &lt;strong&gt;stop on an empty page, never on a short one.&lt;/strong&gt;&lt;/p&gt;
&lt;h3&gt;
  
  
  Totals you can check against
&lt;/h3&gt;

&lt;p&gt;Every store that answered also served &lt;code&gt;/meta.json&lt;/code&gt; with a &lt;code&gt;published_products_count&lt;/code&gt;. For 41 of the 48 stores below the cap our count matched exactly, six were within three products, and one differed by 23. That makes &lt;code&gt;meta.json&lt;/code&gt; a cheap completeness check after each run, as long as the store is under the cap.&lt;/p&gt;
&lt;h2&gt;
  
  
  A collector that handles all of this
&lt;/h2&gt;

&lt;p&gt;The script below pages correctly, respects rate limits, detects the 25,000 cap and checks completeness. It uses only &lt;code&gt;requests&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;S&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0 (compatible; catalog-check/1.0; +https://example.com/bot)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tries&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;                 &lt;span class="c1"&gt;# rate limited: wait and retry
&lt;/span&gt;            &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Retry-After&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))))&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;meta&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/meta.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="n"&gt;capped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;page&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/products.json?limit=250&amp;amp;page=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;25000&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;capped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;                         &lt;span class="c1"&gt;# page * limit &amp;gt; 25,000
&lt;/span&gt;            &lt;span class="k"&gt;break&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;products&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;                             &lt;span class="c1"&gt;# stop on EMPTY, not short
&lt;/span&gt;            &lt;span class="k"&gt;break&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;                 &lt;span class="c1"&gt;# dedupe by product id
&lt;/span&gt;        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;expected&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;published_products_count&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; products, meta says &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;, &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
          &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;currency &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;currency&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;, capped=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;capped&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;to_rows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;variants&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;domain&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;currency&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;currency&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;product_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;handle&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;handle&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;vendor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;vendor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;product_type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;product_type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;variant_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sku&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sku&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;variant&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;compare_at_price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;compare_at_price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;available&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;available&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;updated_at&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;updated_at&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;domain&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;www.allbirds.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;products&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;collect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;to_rows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;meta&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;catalog.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DictWriter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fieldnames&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;keys&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeheader&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A one-second delay is deliberately conservative. Most catalogs in our sample need one to four requests, so politeness costs you seconds, not hours.&lt;/p&gt;

&lt;p&gt;A command-line version with the same logic (domain and output file as arguments, progress output, MIT license) is in the &lt;a href="https://github.com/JsonEngine/shopify-products-json" rel="noopener noreferrer"&gt;GitHub repo&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; requirements.txt
python collect.py www.allbirds.com &lt;span class="nt"&gt;--out&lt;/span&gt; allbirds.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Getting past the 25,000 cap
&lt;/h2&gt;

&lt;p&gt;For the handful of stores above the ceiling, the cap applies per listing, not per store. Two routes help:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Per collection.&lt;/strong&gt; &lt;code&gt;/collections.json&lt;/code&gt; lists collections (also 250 per page), and &lt;code&gt;/collections/{handle}/products.json&lt;/code&gt; pages through each one under its own 25,000 limit. Merge and deduplicate on product ID. Note that &lt;code&gt;/collections/all/products.json&lt;/code&gt; is capped at the same page 100 as the main feed; we checked.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per product.&lt;/strong&gt; Product sitemaps list every product URL. You can enumerate handles from them and fetch &lt;code&gt;/products/{handle}.js&lt;/code&gt; for each. It's one request per product, so use it to fill gaps, not as the main route.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Measure what each route adds in unique IDs rather than assuming it is complete. The same idea applies to any retailer with a cap on search or listing results: split the query into slices that each stay under the cap, then deduplicate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prices: base currency vs what a shopper sees
&lt;/h2&gt;

&lt;p&gt;All 50 stores returned prices in their shop currency: 47 in USD, and one each in CAD, GBP and EUR. None adjusted to our European location. That is correct behavior for a base-price feed, and it is a trap if you compare stores across countries.&lt;/p&gt;

&lt;p&gt;Shopify's own &lt;a href="https://shopify.dev/docs/apps/build/markets" rel="noopener noreferrer"&gt;Markets documentation&lt;/a&gt; warns developers not to compute an international market's price from base variant prices and to request each market's prices from Shopify instead, because markets can have their own price lists, rounding and currency. The Ajax &lt;code&gt;.js&lt;/code&gt; endpoint returns presentment-currency prices and supports locale-prefixed URLs, which makes it the better tool for checking what a shopper in a given market pays.&lt;/p&gt;

&lt;p&gt;Two more price details trip people up:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;price&lt;/code&gt; is a string like &lt;code&gt;"25.00"&lt;/code&gt; in the JSON feed but an integer in cents (&lt;code&gt;2500&lt;/code&gt;) in the Ajax API. Normalize before comparing.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;compare_at_price&lt;/code&gt; usage varies enormously. In our sample, the share of variants with a compare-at price ranged from 0% to 98% depending on the store. Some brands use it for real markdowns, others as a permanent "was" price. Treat it as a store-specific signal, not a universal discount flag.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What /products.json won't give you
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Stock levels.&lt;/strong&gt; Only an &lt;code&gt;available&lt;/code&gt; boolean per variant. For stock movement over time, track availability changes between runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Identifiers for matching.&lt;/strong&gt; SKUs are merchant-specific and barcodes aren't in the feed, so matching the same product across stores needs titles, options and images, not just IDs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every store.&lt;/strong&gt; Some brands block the endpoint (Gymshark returned 403), run a headless storefront on another domain, or sit behind a bot challenge. For those, the fallback is the product pages themselves and the structured data embedded in them (JSON-LD and framework state like &lt;code&gt;__NEXT_DATA__&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Is it OK to use /products.json?
&lt;/h2&gt;

&lt;p&gt;The endpoint is served to any visitor without a login, and the Shopify robots.txt files we looked at do not disallow it. That puts it on the low-risk end of public data collection, and it is also the lightest possible load on the store: one request instead of 250 page views.&lt;/p&gt;

&lt;p&gt;Still, keep requests slow, don't republish product descriptions or photos, and stop if a store asks you to. This isn't legal advice; if you're building something commercial, check the terms of the stores you collect from.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;/p&gt;
  How many products can /products.json return?
  &lt;br&gt;
Up to 250 per page. In our October 2026 test, asking for &lt;code&gt;limit=500&lt;/code&gt; still returned 250. The total reachable through page-based paging is capped at 25,000 products: beyond page × limit = 25,000 the endpoint returns HTTP 400.&lt;br&gt;


&lt;p&gt;&lt;/p&gt;

&lt;p&gt;&lt;/p&gt;
  Does every Shopify store have /products.json?
  &lt;br&gt;
Most do, but not all respond. In our test 50 of 68 brand domains returned a catalog; others returned 403 or 404, or ran headless storefronts on other domains.&lt;br&gt;


&lt;p&gt;&lt;/p&gt;

&lt;p&gt;&lt;/p&gt;
  When should my scraper stop paginating?
  &lt;br&gt;
When a page returns zero products, or on an HTTP 400 at the 25,000 cap. Do not stop on a page with fewer than 250 items: four of the 50 stores we tested returned short pages mid-catalog.&lt;br&gt;


&lt;p&gt;&lt;/p&gt;

&lt;p&gt;&lt;/p&gt;
  Does /products.json use page_info cursors?
  &lt;br&gt;
No. Cursor-based &lt;code&gt;page_info&lt;/code&gt; pagination belongs to the authenticated Admin API. The public storefront feed uses &lt;code&gt;?page=N&lt;/code&gt;.&lt;br&gt;


&lt;p&gt;&lt;/p&gt;

&lt;p&gt;&lt;/p&gt;
  Are /products.json prices in USD?
  &lt;br&gt;
They are in the shop's base currency, which is USD for most US brands but not all. Read the currency from &lt;code&gt;/meta.json&lt;/code&gt;, and use the Ajax &lt;code&gt;.js&lt;/code&gt; endpoint or market-specific URLs if you need the price shown in another market.&lt;br&gt;


&lt;p&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;The collector (MIT) and the raw test results (CC BY 4.0) are on GitHub: &lt;a href="https://github.com/JsonEngine/shopify-products-json" rel="noopener noreferrer"&gt;JsonEngine/shopify-products-json&lt;/a&gt;. If you've seen other &lt;code&gt;/products.json&lt;/code&gt; quirks, We'd like to hear about them in the comments.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>shopify</category>
      <category>python</category>
      <category>webscraping</category>
      <category>ecommerce</category>
    </item>
  </channel>
</rss>
