The latest articles on DEV Community by Abdulwahab (@abdulwhab95). https://dev.to/abdulwhab95 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4145294%2F9e90f489-801a-40f3-9d8d-e725841415fc.png https://dev.to/abdulwhab95 en Abdulwahab Sun, 27 Sep 2026 18:49:20 +0000 https://dev.to/abdulwhab95/html-to-clean-markdown-chunks-in-python-and-spotting-what-really-changed-1ibm https://dev.to/abdulwhab95/html-to-clean-markdown-chunks-in-python-and-spotting-what-really-changed-1ibm <p>If you feed web pages to a search index or a retrieval-augmented generation (RAG) pipeline, you usually want three things: only the main content, as plain Markdown, split into sections that have stable names. When the page is updated, you also want to know <strong>which</strong> sections changed, so you re-embed a few chunks instead of the whole page.</p> <p>This post builds that with Python's standard library only. The output is CommonMark, plus GitHub Flavored Markdown pipe tables for HTML tables. The script:</p> <ol> <li>parses HTML into a small tree,</li> <li>keeps the main content and drops navigation, scripts and decorative noise,</li> <li>renders headings, paragraphs, lists, code blocks, tables and links as Markdown,</li> <li>splits the result into one chunk per section, named by its heading path,</li> <li>compares two versions and sorts each section into added, removed, renamed, changed, links-only or unchanged.</li> </ol> <p>The real run below uses two versions of the same page: the <code>json</code> module documentation for Python 3.13 and for Python 3.14, fetched on 27 September 2026 (UTC). <code>docs.python.org/robots.txt</code> disallows <code>/dev</code>, <code>/release</code> and end-of-life versions such as <code>/3.9/</code>; <code>/3.13/</code> and <code>/3.14/</code> are allowed. That gives a genuine before and after without waiting for a page to change.</p> <h2> Step 1: parse into a small tree </h2> <p><code>html.parser</code> is an event parser: it reports start tags, end tags and text. Building a tree from those events takes a stack. Void elements such as <code>&lt;br&gt;</code> and <code>&lt;img&gt;</code> never get an end tag, so they are not pushed. An end tag closes the nearest open element with the same name; a stray end tag with no match is ignored.</p> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 29-50:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">class</span> <span class="nc">TreeBuilder</span><span class="p">(</span><span class="n">HTMLParser</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Build a small tree of {</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="s">, </span><span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="s">, </span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="s">} dicts; text stays as plain strings.</span><span class="sh">"""</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="nf">super</span><span class="p">().</span><span class="nf">__init__</span><span class="p">(</span><span class="n">convert_charrefs</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="n">root</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">#root</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">:</span> <span class="p">{},</span> <span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">:</span> <span class="p">[]}</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span> <span class="o">=</span> <span class="p">[</span><span class="n">self</span><span class="p">.</span><span class="n">root</span><span class="p">]</span> <span class="k">def</span> <span class="nf">handle_starttag</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">tag</span><span class="p">,</span> <span class="n">attrs</span><span class="p">):</span> <span class="n">node</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">:</span> <span class="n">tag</span><span class="p">,</span> <span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">:</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span> <span class="ow">or</span> <span class="sh">""</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">attrs</span><span class="p">},</span> <span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">:</span> <span class="p">[]}</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">].</span><span class="nf">append</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="k">if</span> <span class="n">tag</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">VOID</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="k">def</span> <span class="nf">handle_endtag</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">tag</span><span class="p">):</span> <span class="k">for</span> <span class="n">depth</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">):</span> <span class="c1"># close the nearest open match, if any </span> <span class="k">if</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">[</span><span class="n">depth</span><span class="p">][</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="n">tag</span><span class="p">:</span> <span class="k">del</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">[</span><span class="n">depth</span><span class="p">:]</span> <span class="k">break</span> <span class="k">def</span> <span class="nf">handle_data</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">].</span><span class="nf">append</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> </code></pre> </div> <p>This is deliberately simple. It is not a full HTML5 parser: it doesn't apply the rules that implicitly close a <code>&lt;p&gt;</code> when a <code>&lt;div&gt;</code> starts, for example. For well-formed pages, such as most documentation sites, that is fine. For messy HTML, parse with <code>html5lib</code> or <code>lxml</code> and keep the rendering part of this post.</p> <h2> Step 2: keep the main content, drop the noise </h2> <p>The script looks for <code>&lt;main&gt;</code>, then <code>&lt;article&gt;</code>, then any element with <code>role="main"</code>, then <code>&lt;body&gt;</code>. Inside it, whole elements are skipped by tag (scripts, styles, navigation, footers, forms) and by class.</p> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 20-27:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="n">USER_AGENT</span> <span class="o">=</span> <span class="sh">"</span><span class="s">html-to-chunks-example/1.0 (tutorial script; one request per page)</span><span class="sh">"</span> <span class="n">VOID</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">area</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">base</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">br</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">col</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">embed</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">hr</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">img</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">input</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">link</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">meta</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">source</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">track</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">wbr</span><span class="sh">"</span><span class="p">}</span> <span class="n">SKIP</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">script</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">style</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">noscript</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">template</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">svg</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">nav</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">footer</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">form</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">button</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">iframe</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">aside</span><span class="sh">"</span><span class="p">}</span> <span class="n">NOISE_CLASSES</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">headerlink</span><span class="sh">"</span><span class="p">}</span> <span class="c1"># e.g. the pilcrow permalinks Sphinx adds after headings </span><span class="n">BLOCKS</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">p</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">div</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">section</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">article</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">blockquote</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">pre</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">ul</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">ol</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">table</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">dl</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">dt</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">dd</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">hr</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">figure</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h2</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h3</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h4</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h5</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h6</span><span class="sh">"</span><span class="p">}</span> <span class="o">|</span> <span class="n">SKIP</span> </code></pre> </div> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 64-78:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">main_content</span><span class="p">(</span><span class="n">root</span><span class="p">):</span> <span class="k">for</span> <span class="n">test</span> <span class="ow">in</span> <span class="p">(</span><span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="sh">"</span><span class="s">article</span><span class="sh">"</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span><span class="p">[</span><span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">role</span><span class="sh">"</span><span class="p">)</span> <span class="o">==</span> <span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="sh">"</span><span class="s">body</span><span class="sh">"</span><span class="p">):</span> <span class="n">hit</span> <span class="o">=</span> <span class="nf">find</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">test</span><span class="p">)</span> <span class="k">if</span> <span class="n">hit</span><span class="p">:</span> <span class="k">return</span> <span class="n">hit</span> <span class="k">return</span> <span class="n">root</span> <span class="k">def</span> <span class="nf">ignored</span><span class="p">(</span><span class="n">node</span><span class="p">):</span> <span class="k">return</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="n">SKIP</span> <span class="ow">or</span> <span class="nf">bool</span><span class="p">(</span><span class="n">NOISE_CLASSES</span> <span class="o">&amp;</span> <span class="nf">set</span><span class="p">(</span><span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">class</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">).</span><span class="nf">split</span><span class="p">()))</span> <span class="k">def</span> <span class="nf">collapse</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">re</span><span class="p">.</span><span class="nf">sub</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">\s+</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span><span class="p">,</span> <span class="n">text</span><span class="p">).</span><span class="nf">strip</span><span class="p">()</span> </code></pre> </div> <p>The class list is where site-specific cleanup goes. Sphinx, the documentation generator behind the Python docs, puts a <code>¶</code> permalink after every heading with the class <code>headerlink</code>. Without that one entry, every heading in the output would end in <code>¶</code> and every chunk name would carry it.</p> <h2> Step 3: inline Markdown </h2> <p>Inline content becomes one line: bold and italics, inline code, image alt text, and links. Links are made absolute with <code>urljoin</code>, because a relative link is useless once the text leaves the page. Links to anchors on the same page (<code>#...</code>) keep only their text.</p> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 81-108:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">text_of</span><span class="p">(</span><span class="n">node</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Raw text with whitespace kept (for &lt;pre&gt;), minus ignored elements.</span><span class="sh">"""</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span> <span class="k">return</span> <span class="n">node</span> <span class="k">return</span> <span class="sh">""</span> <span class="k">if</span> <span class="nf">ignored</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="k">else</span> <span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="nf">text_of</span><span class="p">(</span><span class="n">child</span><span class="p">)</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">])</span> <span class="k">def</span> <span class="nf">inline</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">base</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Render inline content as Markdown on one line.</span><span class="sh">"""</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span> <span class="k">return</span> <span class="n">node</span> <span class="k">if</span> <span class="nf">ignored</span><span class="p">(</span><span class="n">node</span><span class="p">):</span> <span class="k">return</span> <span class="sh">""</span> <span class="n">tag</span><span class="p">,</span> <span class="n">attrs</span> <span class="o">=</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">],</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">br</span><span class="sh">"</span><span class="p">:</span> <span class="k">return</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">img</span><span class="sh">"</span><span class="p">:</span> <span class="k">return</span> <span class="n">attrs</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">alt</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="n">inner</span> <span class="o">=</span> <span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="nf">inline</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">)</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">])</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">code</span><span class="sh">"</span><span class="p">:</span> <span class="k">return</span> <span class="sa">f</span><span class="sh">"</span><span class="s">`</span><span class="si">{</span><span class="nf">collapse</span><span class="p">(</span><span class="nf">text_of</span><span class="p">(</span><span class="n">node</span><span class="p">))</span><span class="si">}</span><span class="s">`</span><span class="sh">"</span> <span class="k">if</span> <span class="nf">collapse</span><span class="p">(</span><span class="nf">text_of</span><span class="p">(</span><span class="n">node</span><span class="p">))</span> <span class="k">else</span> <span class="sh">""</span> <span class="k">if</span> <span class="n">tag</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">strong</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">b</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">em</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">i</span><span class="sh">"</span><span class="p">)</span> <span class="ow">and</span> <span class="nf">collapse</span><span class="p">(</span><span class="n">inner</span><span class="p">):</span> <span class="n">mark</span> <span class="o">=</span> <span class="sh">"</span><span class="s">**</span><span class="sh">"</span> <span class="k">if</span> <span class="n">tag</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">strong</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">b</span><span class="sh">"</span><span class="p">)</span> <span class="k">else</span> <span class="sh">"</span><span class="s">*</span><span class="sh">"</span> <span class="k">return</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">mark</span><span class="si">}{</span><span class="nf">collapse</span><span class="p">(</span><span class="n">inner</span><span class="p">)</span><span class="si">}{</span><span class="n">mark</span><span class="si">}</span><span class="sh">"</span> <span class="n">href</span> <span class="o">=</span> <span class="n">attrs</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">href</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">a</span><span class="sh">"</span> <span class="ow">and</span> <span class="nf">collapse</span><span class="p">(</span><span class="n">inner</span><span class="p">)</span> <span class="ow">and</span> <span class="n">href</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">href</span><span class="p">.</span><span class="nf">startswith</span><span class="p">((</span><span class="sh">"</span><span class="s">#</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">javascript:</span><span class="sh">"</span><span class="p">)):</span> <span class="k">return</span> <span class="sa">f</span><span class="sh">"</span><span class="s">[</span><span class="si">{</span><span class="nf">collapse</span><span class="p">(</span><span class="n">inner</span><span class="p">)</span><span class="si">}</span><span class="s">](</span><span class="si">{</span><span class="n">urllib</span><span class="p">.</span><span class="n">parse</span><span class="p">.</span><span class="nf">urljoin</span><span class="p">(</span><span class="n">base</span><span class="p">,</span> <span class="n">href</span><span class="p">)</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span> <span class="k">return</span> <span class="n">inner</span> </code></pre> </div> <h2> Step 4: blocks, lists and tables </h2> <p><code>render()</code> walks the children of a node. Text and inline elements are collected into a "run" until a block element appears; then the run is flushed as a paragraph. Headings, code blocks, lists, tables and definition lists each get their own branch. Code blocks keep their whitespace; everything else is collapsed.</p> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 149-185:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">out</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Append Markdown blocks (strings) for the children of `node` to `out`.</span><span class="sh">"""</span> <span class="n">run</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">def</span> <span class="nf">flush</span><span class="p">():</span> <span class="n">text</span> <span class="o">=</span> <span class="nf">collapse</span><span class="p">(</span><span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">run</span><span class="p">))</span> <span class="n">run</span><span class="p">.</span><span class="nf">clear</span><span class="p">()</span> <span class="k">if</span> <span class="n">text</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">]:</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="ow">or</span> <span class="n">child</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">BLOCKS</span><span class="p">:</span> <span class="n">run</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="nf">inline</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">))</span> <span class="k">continue</span> <span class="nf">flush</span><span class="p">()</span> <span class="n">tag</span> <span class="o">=</span> <span class="n">child</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="nf">ignored</span><span class="p">(</span><span class="n">child</span><span class="p">)</span> <span class="ow">or</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">hr</span><span class="sh">"</span><span class="p">:</span> <span class="k">continue</span> <span class="k">if</span> <span class="n">re</span><span class="p">.</span><span class="nf">fullmatch</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">h[1-6]</span><span class="sh">"</span><span class="p">,</span> <span class="n">tag</span><span class="p">):</span> <span class="n">title</span> <span class="o">=</span> <span class="nf">collapse</span><span class="p">(</span><span class="nf">inline</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">))</span> <span class="k">if</span> <span class="n">title</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">"</span><span class="s">#</span><span class="sh">"</span> <span class="o">*</span> <span class="nf">int</span><span class="p">(</span><span class="n">tag</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span> <span class="o">+</span> <span class="n">title</span><span class="p">)</span> <span class="k">elif</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">pre</span><span class="sh">"</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">"</span><span class="s">``` </span><span class="se">\n</span><span class="sh">"</span> <span class="o">+</span> <span class="nf">text_of</span><span class="p">(</span><span class="n">child</span><span class="p">).</span><span class="nf">strip</span><span class="p">(</span><span class="sh">"</span><span class="se">\n</span><span class="sh">"</span><span class="p">)</span> <span class="o">+</span> <span class="sh">"</span><span class="se">\n</span><span class="s"> ```</span><span class="sh">"</span><span class="p">)</span> <span class="k">elif</span> <span class="n">tag</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">ul</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">ol</span><span class="sh">"</span><span class="p">):</span> <span class="nf">render_list</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">out</span><span class="p">)</span> <span class="k">elif</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">table</span><span class="sh">"</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="nf">render_table</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">))</span> <span class="k">elif</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">dt</span><span class="sh">"</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="nf">collapse</span><span class="p">(</span><span class="nf">text_of</span><span class="p">(</span><span class="n">child</span><span class="p">)))</span> <span class="k">elif</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">blockquote</span><span class="sh">"</span><span class="p">:</span> <span class="n">quoted</span> <span class="o">=</span> <span class="p">[]</span> <span class="nf">render</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">quoted</span><span class="p">)</span> <span class="n">out</span><span class="p">.</span><span class="nf">extend</span><span class="p">(</span><span class="sh">"</span><span class="s">&gt; </span><span class="sh">"</span> <span class="o">+</span> <span class="n">block</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="se">\n</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="se">\n</span><span class="s">&gt; </span><span class="sh">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">block</span> <span class="ow">in</span> <span class="n">quoted</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="nf">render</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">out</span><span class="p">)</span> <span class="nf">flush</span><span class="p">()</span> </code></pre> </div> <p>Definition terms (<code>&lt;dt&gt;</code>) become plain text. In Sphinx output they hold function signatures, and the <code>*</code> in a signature such as <code>json.dump(obj, fp, *, ...)</code> would break Markdown emphasis if it were wrapped in <code>**</code>.</p> <p>Tables become pipe tables. Pipes inside cells are escaped, short rows are padded, and <code>colspan</code>/<code>rowspan</code> are ignored, so complex tables lose their structure. That is a known limit.</p> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 125-137:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">render_table</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">base</span><span class="p">):</span> <span class="n">rows</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="nf">find_all</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="sh">"</span><span class="s">tr</span><span class="sh">"</span><span class="p">):</span> <span class="n">cells</span> <span class="o">=</span> <span class="p">[</span><span class="nf">collapse</span><span class="p">(</span><span class="nf">inline</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">base</span><span class="p">)).</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="s">|</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="se">\\</span><span class="s">|</span><span class="sh">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="ow">and</span> <span class="n">cell</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">th</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">td</span><span class="sh">"</span><span class="p">)]</span> <span class="k">if</span> <span class="n">cells</span><span class="p">:</span> <span class="n">rows</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">cells</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">rows</span><span class="p">:</span> <span class="k">return</span> <span class="sh">""</span> <span class="n">width</span> <span class="o">=</span> <span class="nf">max</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">row</span><span class="p">)</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">)</span> <span class="n">rows</span> <span class="o">=</span> <span class="p">[</span><span class="n">row</span> <span class="o">+</span> <span class="p">[</span><span class="sh">""</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="n">width</span> <span class="o">-</span> <span class="nf">len</span><span class="p">(</span><span class="n">row</span><span class="p">))</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">]</span> <span class="n">lines</span> <span class="o">=</span> <span class="p">[</span><span class="sh">"</span><span class="s">| </span><span class="sh">"</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> | </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">rows</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> |</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">|</span><span class="sh">"</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> --- |</span><span class="sh">"</span> <span class="o">*</span> <span class="n">width</span><span class="p">]</span> <span class="k">return</span> <span class="sh">"</span><span class="se">\n</span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">lines</span> <span class="o">+</span> <span class="p">[</span><span class="sh">"</span><span class="s">| </span><span class="sh">"</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> | </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">row</span><span class="p">)</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> |</span><span class="sh">"</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">[</span><span class="mi">1</span><span class="p">:]])</span> </code></pre> </div> <h2> Step 5: one chunk per section </h2> <p>Each heading up to level 3 starts a new chunk. The chunk's id is its heading path, such as <code>Guide &gt; Setup</code>. Deeper headings stay inside their parent's chunk. When two sections share a path, the second gets <code>#2</code>.</p> <p>Every chunk gets two hashes:</p> <ul> <li> <code>sha256</code> of the exact text, to answer "is this byte-for-byte the same?",</li> <li> <code>fingerprint</code>, the hash of the text with link targets removed and whitespace collapsed, to answer "did the words change?".</li> </ul> <p>The real run shows why the second one matters.</p> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 197-229:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">prose</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Text with link targets and whitespace differences removed.</span><span class="sh">"""</span> <span class="k">return</span> <span class="nf">collapse</span><span class="p">(</span><span class="n">re</span><span class="p">.</span><span class="nf">sub</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">\]\([^)]*\)</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">]</span><span class="sh">"</span><span class="p">,</span> <span class="n">text</span><span class="p">))</span> <span class="k">def</span> <span class="nf">fingerprint</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">hashlib</span><span class="p">.</span><span class="nf">sha256</span><span class="p">(</span><span class="nf">prose</span><span class="p">(</span><span class="n">text</span><span class="p">).</span><span class="nf">encode</span><span class="p">(</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)).</span><span class="nf">hexdigest</span><span class="p">()[:</span><span class="mi">16</span><span class="p">]</span> <span class="k">def</span> <span class="nf">chunk</span><span class="p">(</span><span class="n">blocks</span><span class="p">,</span> <span class="n">max_level</span><span class="o">=</span><span class="mi">3</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Group blocks under their nearest heading (h1..h{max_level}); one chunk per section.</span><span class="sh">"""</span> <span class="n">chunks</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">body</span><span class="p">,</span> <span class="n">seen</span> <span class="o">=</span> <span class="p">[],</span> <span class="p">[],</span> <span class="p">[],</span> <span class="p">{}</span> <span class="k">def</span> <span class="nf">emit</span><span class="p">():</span> <span class="n">text</span> <span class="o">=</span> <span class="sh">"</span><span class="se">\n\n</span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">body</span><span class="p">).</span><span class="nf">strip</span><span class="p">()</span> <span class="n">body</span><span class="p">.</span><span class="nf">clear</span><span class="p">()</span> <span class="k">if</span> <span class="n">text</span><span class="p">:</span> <span class="n">name</span> <span class="o">=</span> <span class="sh">"</span><span class="s"> &gt; </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">"</span><span class="s">(top)</span><span class="sh">"</span> <span class="n">seen</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">seen</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span> <span class="n">chunk_id</span> <span class="o">=</span> <span class="n">name</span> <span class="k">if</span> <span class="n">seen</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">else</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s"> #</span><span class="si">{</span><span class="n">seen</span><span class="p">[</span><span class="n">name</span><span class="p">]</span><span class="si">}</span><span class="sh">"</span> <span class="n">chunks</span><span class="p">.</span><span class="nf">append</span><span class="p">({</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">:</span> <span class="n">chunk_id</span><span class="p">,</span> <span class="sh">"</span><span class="s">words</span><span class="sh">"</span><span class="p">:</span> <span class="nf">len</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="nf">split</span><span class="p">()),</span> <span class="sh">"</span><span class="s">sha256</span><span class="sh">"</span><span class="p">:</span> <span class="n">hashlib</span><span class="p">.</span><span class="nf">sha256</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="nf">encode</span><span class="p">(</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)).</span><span class="nf">hexdigest</span><span class="p">()[:</span><span class="mi">16</span><span class="p">],</span> <span class="sh">"</span><span class="s">fingerprint</span><span class="sh">"</span><span class="p">:</span> <span class="nf">fingerprint</span><span class="p">(</span><span class="n">text</span><span class="p">),</span> <span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">:</span> <span class="n">text</span><span class="p">})</span> <span class="k">for</span> <span class="n">block</span> <span class="ow">in</span> <span class="n">blocks</span><span class="p">:</span> <span class="n">heading</span> <span class="o">=</span> <span class="n">re</span><span class="p">.</span><span class="nf">match</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">(#{1,6}) (.*)</span><span class="sh">"</span><span class="p">,</span> <span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="n">heading</span> <span class="ow">and</span> <span class="nf">len</span><span class="p">(</span><span class="n">heading</span><span class="p">.</span><span class="nf">group</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span> <span class="o">&lt;=</span> <span class="n">max_level</span><span class="p">:</span> <span class="nf">emit</span><span class="p">()</span> <span class="k">del</span> <span class="n">path</span><span class="p">[</span><span class="nf">len</span><span class="p">(</span><span class="n">heading</span><span class="p">.</span><span class="nf">group</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span> <span class="o">-</span> <span class="mi">1</span><span class="p">:]</span> <span class="n">path</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">heading</span><span class="p">.</span><span class="nf">group</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span> <span class="n">body</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">block</span><span class="p">)</span> <span class="nf">emit</span><span class="p">()</span> <span class="k">return</span> <span class="n">chunks</span> </code></pre> </div> <h2> Step 6: compare two versions </h2> <p>Sections are matched by id. A renamed heading changes the id, so it first shows up as one removed and one added section. <code>compare()</code> pairs those up when their text is at least 60% similar according to <code>difflib.SequenceMatcher</code>, and reports them as renamed. The 0.6 threshold is a judgment call; the test below also checks that a strict threshold turns pairing off.</p> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 232-253:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">,</span> <span class="n">rename_threshold</span><span class="o">=</span><span class="mf">0.6</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Classify chunks by id: added, removed, renamed, changed (prose), links-only, unchanged.</span><span class="sh">"""</span> <span class="n">before</span> <span class="o">=</span> <span class="p">{</span><span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]:</span> <span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">old</span><span class="p">}</span> <span class="n">after</span> <span class="o">=</span> <span class="p">{</span><span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]:</span> <span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">new</span><span class="p">}</span> <span class="n">result</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">:</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">after</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">-</span> <span class="n">before</span><span class="p">.</span><span class="nf">keys</span><span class="p">()),</span> <span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">:</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">before</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">-</span> <span class="n">after</span><span class="p">.</span><span class="nf">keys</span><span class="p">()),</span> <span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">links-only</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">unchanged</span><span class="sh">"</span><span class="p">:</span> <span class="p">[]}</span> <span class="k">for</span> <span class="n">old_id</span> <span class="ow">in</span> <span class="nf">list</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">]):</span> <span class="c1"># a renamed heading looks like removed + added </span> <span class="n">scored</span> <span class="o">=</span> <span class="p">[(</span><span class="n">difflib</span><span class="p">.</span><span class="nc">SequenceMatcher</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="nf">prose</span><span class="p">(</span><span class="n">before</span><span class="p">[</span><span class="n">old_id</span><span class="p">][</span><span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">]),</span> <span class="nf">prose</span><span class="p">(</span><span class="n">after</span><span class="p">[</span><span class="n">new_id</span><span class="p">][</span><span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">])).</span><span class="nf">ratio</span><span class="p">(),</span> <span class="n">new_id</span><span class="p">)</span> <span class="k">for</span> <span class="n">new_id</span> <span class="ow">in</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">]]</span> <span class="k">if</span> <span class="n">scored</span> <span class="ow">and</span> <span class="nf">max</span><span class="p">(</span><span class="n">scored</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&gt;=</span> <span class="n">rename_threshold</span><span class="p">:</span> <span class="n">ratio</span><span class="p">,</span> <span class="n">new_id</span> <span class="o">=</span> <span class="nf">max</span><span class="p">(</span><span class="n">scored</span><span class="p">)</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">].</span><span class="nf">remove</span><span class="p">(</span><span class="n">old_id</span><span class="p">)</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">].</span><span class="nf">remove</span><span class="p">(</span><span class="n">new_id</span><span class="p">)</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">].</span><span class="nf">append</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">old_id</span><span class="si">}</span><span class="s"> -&gt; </span><span class="si">{</span><span class="n">new_id</span><span class="si">}</span><span class="s"> (similarity </span><span class="si">{</span><span class="n">ratio</span><span class="si">:</span><span class="p">.</span><span class="mi">3</span><span class="n">f</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">chunk_id</span> <span class="ow">in</span> <span class="p">[</span><span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">new</span> <span class="k">if</span> <span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="n">before</span><span class="p">]:</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="o">=</span> <span class="n">before</span><span class="p">[</span><span class="n">chunk_id</span><span class="p">],</span> <span class="n">after</span><span class="p">[</span><span class="n">chunk_id</span><span class="p">]</span> <span class="n">kind</span> <span class="o">=</span> <span class="p">(</span><span class="sh">"</span><span class="s">unchanged</span><span class="sh">"</span> <span class="k">if</span> <span class="n">a</span><span class="p">[</span><span class="sh">"</span><span class="s">sha256</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="n">b</span><span class="p">[</span><span class="sh">"</span><span class="s">sha256</span><span class="sh">"</span><span class="p">]</span> <span class="k">else</span> <span class="sh">"</span><span class="s">links-only</span><span class="sh">"</span> <span class="k">if</span> <span class="n">a</span><span class="p">[</span><span class="sh">"</span><span class="s">fingerprint</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="n">b</span><span class="p">[</span><span class="sh">"</span><span class="s">fingerprint</span><span class="sh">"</span><span class="p">]</span> <span class="k">else</span> <span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">)</span> <span class="n">result</span><span class="p">[</span><span class="n">kind</span><span class="p">].</span><span class="nf">append</span><span class="p">(</span><span class="n">chunk_id</span><span class="p">)</span> <span class="k">return</span> <span class="n">result</span> </code></pre> </div> <h2> Fetching politely </h2> <p>One request per page, a descriptive <code>User-Agent</code>, <code>robots.txt</code> read first, and at least a one-second pause, or the site's <code>Crawl-delay</code> if it sets one. If <code>robots.txt</code> disallows the page, the script stops. One trap: <code>urllib.robotparser</code> ends a group at a blank line, while <a href="https://www.rfc-editor.org/rfc/rfc9309" rel="noopener noreferrer">RFC 9309</a> does not, and <code>docs.python.org/robots.txt</code> has a blank line before its end-of-life rules. Parsed as is, <code>/3.9/</code> comes out allowed, so <code>robots_rules()</code> drops blank lines first. The standard parser still differs from RFC 9309 in other ways; for example, it applies the first matching rule rather than the longest one.</p> <p><em>Excerpt of <code>page_to_chunks.py</code>, lines 256-281:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Parse robots.txt. urllib.robotparser ends a group at a blank line; RFC 9309 does not.</span><span class="sh">"""</span> <span class="n">rules</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">robotparser</span><span class="p">.</span><span class="nc">RobotFileParser</span><span class="p">()</span> <span class="n">rules</span><span class="p">.</span><span class="nf">parse</span><span class="p">([</span><span class="n">line</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">text</span><span class="p">.</span><span class="nf">splitlines</span><span class="p">()</span> <span class="k">if</span> <span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()])</span> <span class="k">return</span> <span class="n">rules</span> <span class="k">def</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">url</span><span class="p">):</span> <span class="n">parts</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">parse</span><span class="p">.</span><span class="nf">urlsplit</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="n">text</span> <span class="o">=</span> <span class="nf">get</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">parts</span><span class="p">.</span><span class="n">scheme</span><span class="si">}</span><span class="s">://</span><span class="si">{</span><span class="n">parts</span><span class="p">.</span><span class="n">netloc</span><span class="si">}</span><span class="s">/robots.txt</span><span class="sh">"</span><span class="p">)</span> <span class="k">except</span> <span class="n">urllib</span><span class="p">.</span><span class="n">error</span><span class="p">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="k">if</span> <span class="n">err</span><span class="p">.</span><span class="n">code</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">404</span><span class="p">,</span> <span class="mi">410</span><span class="p">):</span> <span class="k">raise</span> <span class="n">text</span> <span class="o">=</span> <span class="sh">""</span> <span class="c1"># no robots.txt: nothing is disallowed </span> <span class="n">rules</span> <span class="o">=</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">rules</span><span class="p">.</span><span class="nf">can_fetch</span><span class="p">(</span><span class="n">USER_AGENT</span><span class="p">,</span> <span class="n">url</span><span class="p">):</span> <span class="n">sys</span><span class="p">.</span><span class="nf">exit</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">robots.txt disallows </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s">; stopping</span><span class="sh">"</span><span class="p">)</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="nf">max</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">rules</span><span class="p">.</span><span class="nf">crawl_delay</span><span class="p">(</span><span class="n">USER_AGENT</span><span class="p">)</span> <span class="ow">or</span> <span class="mi">0</span><span class="p">))</span> <span class="k">return</span> <span class="nf">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">url</span><span class="p">):</span> <span class="n">request</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nc">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="sh">"</span><span class="s">User-Agent</span><span class="sh">"</span><span class="p">:</span> <span class="n">USER_AGENT</span><span class="p">})</span> <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nf">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">30</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span> <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="nf">read</span><span class="p">().</span><span class="nf">decode</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">headers</span><span class="p">.</span><span class="nf">get_content_charset</span><span class="p">()</span> <span class="ow">or</span> <span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">replace</span><span class="sh">"</span><span class="p">)</span> </code></pre> </div> <h2> Tests </h2> <p>Twelve <code>unittest</code> tests cover the parser, the renderer, the comparison and the <code>robots.txt</code> parsing. Two of them:</p> <p><em>Excerpt of <code>test_page_to_chunks.py</code>, lines 13-18:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code> <span class="k">def</span> <span class="nf">test_main_content_only_and_noise_removed</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">html</span> <span class="o">=</span> <span class="sh">"""</span><span class="s">&lt;html&gt;&lt;body&gt;&lt;nav&gt;Menu&lt;/nav&gt;&lt;div role=</span><span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="s">&gt; &lt;h1&gt;Title&lt;a class=</span><span class="sh">"</span><span class="s">headerlink</span><span class="sh">"</span><span class="s"> href=</span><span class="sh">"</span><span class="s">#t</span><span class="sh">"</span><span class="s">&gt;¶&lt;/a&gt;&lt;/h1&gt; &lt;p&gt;Hello &lt;b&gt;bold&lt;/b&gt; world.&lt;/p&gt;&lt;script&gt;track()&lt;/script&gt;&lt;/div&gt;&lt;footer&gt;Legal&lt;/footer&gt;&lt;/body&gt;&lt;/html&gt;</span><span class="sh">"""</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">md</span><span class="p">(</span><span class="n">html</span><span class="p">),</span> <span class="p">[</span><span class="sh">"</span><span class="s"># Title</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Hello **bold** world.</span><span class="sh">"</span><span class="p">])</span> </code></pre> </div> <p><em>Excerpt of <code>test_page_to_chunks.py</code>, lines 59-73:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code> <span class="k">def</span> <span class="nf">test_compare</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">old</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">([</span><span class="sh">"</span><span class="s"># A</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">same</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># B</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">[x](https://v1)</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># C</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">old text</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># D</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">gone</span><span class="sh">"</span><span class="p">])</span> <span class="n">new</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">([</span><span class="sh">"</span><span class="s"># A</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">same</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># B</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">[x](https://v2)</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># C</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">new text</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># E</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">fresh</span><span class="sh">"</span><span class="p">])</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">),</span> <span class="p">{</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">E</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">D</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">C</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">links-only</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">B</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">unchanged</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">A</span><span class="sh">"</span><span class="p">]})</span> <span class="k">def</span> <span class="nf">test_renamed_heading_is_paired</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">body</span> <span class="o">=</span> <span class="sh">"</span><span class="s">The json module can be run from the shell to validate and pretty-print input.</span><span class="sh">"</span> <span class="n">old</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">([</span><span class="sh">"</span><span class="s"># Command Line Interface</span><span class="sh">"</span><span class="p">,</span> <span class="n">body</span><span class="p">])</span> <span class="n">new</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">([</span><span class="sh">"</span><span class="s"># Command-line interface</span><span class="sh">"</span><span class="p">,</span> <span class="n">body</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> Also as python -m json.</span><span class="sh">"</span><span class="p">])</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">],</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">]),</span> <span class="p">([],</span> <span class="p">[]))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">]),</span> <span class="mi">1</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertTrue</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">][</span><span class="mi">0</span><span class="p">].</span><span class="nf">startswith</span><span class="p">(</span><span class="sh">"</span><span class="s">Command Line Interface -&gt; Command-line interface (</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">,</span> <span class="n">rename_threshold</span><span class="o">=</span><span class="mf">0.99</span><span class="p">)[</span><span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">],</span> <span class="p">[])</span> </code></pre> </div> <p><em>Excerpt of <code>test-log.txt</code>, lines 14-17:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>---------------------------------------------------------------------- Ran 12 tests in 0.002s OK </code></pre> </div> <h2> A real run </h2> <p>Convert both versions:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python page_to_chunks.py convert https://docs.python.org/3.13/library/json.html json-3.13.md json-3.13.json https://docs.python.org/3.13/library/json.html at 2026-09-27T13:07:49Z: 110,495 characters of HTML -&gt; 30,089 characters of Markdown in 12 chunks $ python page_to_chunks.py convert https://docs.python.org/3.14/library/json.html json-3.14.md json-3.14.json https://docs.python.org/3.14/library/json.html at 2026-09-27T13:07:51Z: 112,591 characters of HTML -&gt; 30,296 characters of Markdown in 12 chunks </code></pre> </div> <p>About 110,000 characters of HTML became about 30,000 characters of Markdown: navigation, sidebars, scripts, styles and tags are gone. The 3.14 page splits into these twelve chunks:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python list_chunks.py json-3.14.json 530 words 7748786f6be31bbd `json` — JSON encoder and decoder 1092 words 9e39ef8ad501eb91 `json` — JSON encoder and decoder &gt; Basic Usage 1061 words 16519e3a65bd9a4f `json` — JSON encoder and decoder &gt; Encoders and Decoders 50 words b697d5281fe94b36 `json` — JSON encoder and decoder &gt; Exceptions 120 words 33131999719582ed `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability 194 words de4623597eb4dc0c `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability &gt; Character Encodings 102 words b48c164cc407d4a2 `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability &gt; Infinite and NaN Number Values 79 words 065b393d5fffcd15 `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability &gt; Repeated Names Within an Object 86 words f97d8636fc31a9fc `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability &gt; Top-level Non-Object, Non-Array Values 133 words 9f09bd8d5c282095 `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability &gt; Implementation Limitations 138 words a6742132e3ec15c4 `json` — JSON encoder and decoder &gt; Command-line interface 129 words e253fc9726be8808 `json` — JSON encoder and decoder &gt; Command-line interface &gt; Command-line options </code></pre> </div> <p>A piece of the Markdown, the decoding table from the 3.14 page, shows how an HTML table comes out:</p> <p><em>Excerpt of <code>json-3.14.md</code>, lines 245-254:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight markdown"><code>| JSON | Python | | --- | --- | | object | dict | | array | list | | string | str | | number (int) | int | | number (real) | float | | true | True | | false | False | | null | None | </code></pre> </div> <p>Now compare the two versions:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python page_to_chunks.py compare json-3.13.json json-3.14.json https://docs.python.org/3.13/library/json.html -&gt; https://docs.python.org/3.14/library/json.html added 0 removed 0 renamed 2 `json` — JSON encoder and decoder &gt; Command Line Interface -&gt; `json` — JSON encoder and decoder &gt; Command-line interface (similarity 0.758) `json` — JSON encoder and decoder &gt; Command Line Interface &gt; Command line options -&gt; `json` — JSON encoder and decoder &gt; Command-line interface &gt; Command-line options (similarity 0.996) changed 1 `json` — JSON encoder and decoder links-only 6 `json` — JSON encoder and decoder &gt; Basic Usage `json` — JSON encoder and decoder &gt; Encoders and Decoders `json` — JSON encoder and decoder &gt; Exceptions `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability &gt; Character Encodings `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability &gt; Top-level Non-Object, Non-Array Values `json` — JSON encoder and decoder &gt; Standard Compliance and Interoperability &gt; Implementation Limitations unchanged 3 </code></pre> </div> <p>Reading that result:</p> <ul> <li> <strong>Six sections are links-only.</strong> Internal links in the 3.13 page point to <code>/3.13/...</code>; in the 3.14 page they point to <code>/3.14/...</code>. A plain byte hash marks all six as changed. The fingerprint, which ignores link targets, shows that the words are identical. The check below confirms it: in each of the six, swapping <code>/3.13/</code> for <code>/3.14/</code> gives exactly the 3.14 text. </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python check_links_only.py Basic Usage: 50 links, 49 differ; identical once /3.13/ is replaced by /3.14/: True Encoders and Decoders: 19 links, 18 differ; identical once /3.13/ is replaced by /3.14/: True Exceptions: 1 links, 1 differ; identical once /3.13/ is replaced by /3.14/: True Standard Compliance and Interoperability &gt; Character Encodings: 3 links, 3 differ; identical once /3.13/ is replaced by /3.14/: True Standard Compliance and Interoperability &gt; Top-level Non-Object, Non-Array Values: 4 links, 2 differ; identical once /3.13/ is replaced by /3.14/: True Standard Compliance and Interoperability &gt; Implementation Limitations: 2 links, 2 differ; identical once /3.13/ is replaced by /3.14/: True unchanged: Standard Compliance and Interoperability | links: 2 unchanged: Standard Compliance and Interoperability &gt; Infinite and NaN Number Values | links: 0 unchanged: Standard Compliance and Interoperability &gt; Repeated Names Within an Object | links: 0 </code></pre> </div> <ul> <li> <strong>Two sections were renamed.</strong> "Command Line Interface" became "Command-line interface", and its subsection "Command line options" became "Command-line options". Matched by heading alone, they would be reported as two removals and two additions. The subsection's text is almost unchanged (similarity 0.996); the parent section's text changed more (0.758), because in 3.14 the module can be run directly as <code>python -m json</code>, with <code>python -m json.tool</code> kept for backwards compatibility. A renamed section can still have changed content, so re-embed it.</li> <li> <strong>One section changed for real.</strong> The introduction's shell example now runs <code>python -m json</code> instead of <code>python -m json.tool</code>.</li> <li> <strong>Three sections are unchanged</strong>, byte for byte.</li> </ul> <p>For a RAG index, that means re-embedding three chunks (one changed, two renamed) out of twelve, and only updating the stored link targets of six others.</p> <h2> Limits and common mistakes </h2> <ul> <li> <strong>JavaScript-rendered pages.</strong> The script sees the HTML the server sends. If the content is built in the browser, there is little to convert.</li> <li> <strong>Messy HTML.</strong> The tree builder doesn't implement HTML5's implicit-close rules. Use <code>html5lib</code> or <code>lxml</code> for parsing if the output looks wrong.</li> <li> <strong>Hashing raw text only.</strong> Link targets, tracking parameters and version numbers in URLs make every chunk look changed. Keep a fingerprint that ignores them.</li> <li> <strong>Uneven chunk sizes.</strong> Sections are natural units, but they vary: in this page from 50 words ("Exceptions") to 1,092 words ("Basic Usage"). If your embedding model has a size limit, split long sections further at paragraph boundaries, never inside a code block.</li> <li> <strong>Ids from headings.</strong> They are readable and stable until someone renames a heading. The rename pairing covers the common case; a restructured page will still show up as removals and additions.</li> <li> <strong>Content licences.</strong> Converting a page doesn't change who owns it. Check the licence before storing or redistributing the text.</li> </ul> <h2> Official sources </h2> <ul> <li>Python <code>html.parser</code>: <a href="https://docs.python.org/3/library/html.parser.html" rel="noopener noreferrer">https://docs.python.org/3/library/html.parser.html</a> </li> <li>Python <code>difflib.SequenceMatcher</code>: <a href="https://docs.python.org/3/library/difflib.html" rel="noopener noreferrer">https://docs.python.org/3/library/difflib.html</a> </li> <li>Python <code>hashlib</code>: <a href="https://docs.python.org/3/library/hashlib.html" rel="noopener noreferrer">https://docs.python.org/3/library/hashlib.html</a> </li> <li>Python <code>urllib.robotparser</code>: <a href="https://docs.python.org/3/library/urllib.robotparser.html" rel="noopener noreferrer">https://docs.python.org/3/library/urllib.robotparser.html</a> </li> <li>Robots Exclusion Protocol, RFC 9309: <a href="https://www.rfc-editor.org/rfc/rfc9309" rel="noopener noreferrer">https://www.rfc-editor.org/rfc/rfc9309</a> </li> <li>CommonMark specification: <a href="https://spec.commonmark.org/" rel="noopener noreferrer">https://spec.commonmark.org/</a> </li> <li>GitHub Flavored Markdown specification (tables): <a href="https://github.github.com/gfm/" rel="noopener noreferrer">https://github.github.com/gfm/</a> </li> <li>The pages used here: <a href="https://docs.python.org/3.13/library/json.html" rel="noopener noreferrer">https://docs.python.org/3.13/library/json.html</a> and <a href="https://docs.python.org/3.14/library/json.html" rel="noopener noreferrer">https://docs.python.org/3.14/library/json.html</a> </li> </ul> <h2> The complete script and tests </h2> <p></p> page_to_chunks.py (309 lines) <br> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="sh">"""</span><span class="s">Turn an HTML page into clean Markdown chunks, and compare two chunk sets. Usage: python page_to_chunks.py convert URL out.md out.json python page_to_chunks.py compare old.json new.json </span><span class="sh">"""</span> <span class="kn">import</span> <span class="n">difflib</span> <span class="kn">import</span> <span class="n">hashlib</span> <span class="kn">import</span> <span class="n">json</span> <span class="kn">import</span> <span class="n">re</span> <span class="kn">import</span> <span class="n">sys</span> <span class="kn">import</span> <span class="n">time</span> <span class="kn">import</span> <span class="n">urllib.error</span> <span class="kn">import</span> <span class="n">urllib.parse</span> <span class="kn">import</span> <span class="n">urllib.request</span> <span class="kn">import</span> <span class="n">urllib.robotparser</span> <span class="kn">from</span> <span class="n">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timezone</span> <span class="kn">from</span> <span class="n">html.parser</span> <span class="kn">import</span> <span class="n">HTMLParser</span> <span class="n">USER_AGENT</span> <span class="o">=</span> <span class="sh">"</span><span class="s">html-to-chunks-example/1.0 (tutorial script; one request per page)</span><span class="sh">"</span> <span class="n">VOID</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">area</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">base</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">br</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">col</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">embed</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">hr</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">img</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">input</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">link</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">meta</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">source</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">track</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">wbr</span><span class="sh">"</span><span class="p">}</span> <span class="n">SKIP</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">script</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">style</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">noscript</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">template</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">svg</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">nav</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">footer</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">form</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">button</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">iframe</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">aside</span><span class="sh">"</span><span class="p">}</span> <span class="n">NOISE_CLASSES</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">headerlink</span><span class="sh">"</span><span class="p">}</span> <span class="c1"># e.g. the pilcrow permalinks Sphinx adds after headings </span><span class="n">BLOCKS</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">p</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">div</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">section</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">article</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">blockquote</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">pre</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">ul</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">ol</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">table</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">dl</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">dt</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">dd</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">hr</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">figure</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h2</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h3</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h4</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h5</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">h6</span><span class="sh">"</span><span class="p">}</span> <span class="o">|</span> <span class="n">SKIP</span> <span class="k">class</span> <span class="nc">TreeBuilder</span><span class="p">(</span><span class="n">HTMLParser</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Build a small tree of {</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="s">, </span><span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="s">, </span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="s">} dicts; text stays as plain strings.</span><span class="sh">"""</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="nf">super</span><span class="p">().</span><span class="nf">__init__</span><span class="p">(</span><span class="n">convert_charrefs</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="n">root</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">#root</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">:</span> <span class="p">{},</span> <span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">:</span> <span class="p">[]}</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span> <span class="o">=</span> <span class="p">[</span><span class="n">self</span><span class="p">.</span><span class="n">root</span><span class="p">]</span> <span class="k">def</span> <span class="nf">handle_starttag</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">tag</span><span class="p">,</span> <span class="n">attrs</span><span class="p">):</span> <span class="n">node</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">:</span> <span class="n">tag</span><span class="p">,</span> <span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">:</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span> <span class="ow">or</span> <span class="sh">""</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">attrs</span><span class="p">},</span> <span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">:</span> <span class="p">[]}</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">].</span><span class="nf">append</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="k">if</span> <span class="n">tag</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">VOID</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="k">def</span> <span class="nf">handle_endtag</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">tag</span><span class="p">):</span> <span class="k">for</span> <span class="n">depth</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">):</span> <span class="c1"># close the nearest open match, if any </span> <span class="k">if</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">[</span><span class="n">depth</span><span class="p">][</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="n">tag</span><span class="p">:</span> <span class="k">del</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">[</span><span class="n">depth</span><span class="p">:]</span> <span class="k">break</span> <span class="k">def</span> <span class="nf">handle_data</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="n">stack</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">].</span><span class="nf">append</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="k">def</span> <span class="nf">find</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">test</span><span class="p">):</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span> <span class="k">if</span> <span class="nf">test</span><span class="p">(</span><span class="n">node</span><span class="p">):</span> <span class="k">return</span> <span class="n">node</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">]:</span> <span class="n">hit</span> <span class="o">=</span> <span class="nf">find</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">test</span><span class="p">)</span> <span class="k">if</span> <span class="n">hit</span><span class="p">:</span> <span class="k">return</span> <span class="n">hit</span> <span class="k">return</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">main_content</span><span class="p">(</span><span class="n">root</span><span class="p">):</span> <span class="k">for</span> <span class="n">test</span> <span class="ow">in</span> <span class="p">(</span><span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="sh">"</span><span class="s">article</span><span class="sh">"</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span><span class="p">[</span><span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">role</span><span class="sh">"</span><span class="p">)</span> <span class="o">==</span> <span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">n</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="sh">"</span><span class="s">body</span><span class="sh">"</span><span class="p">):</span> <span class="n">hit</span> <span class="o">=</span> <span class="nf">find</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">test</span><span class="p">)</span> <span class="k">if</span> <span class="n">hit</span><span class="p">:</span> <span class="k">return</span> <span class="n">hit</span> <span class="k">return</span> <span class="n">root</span> <span class="k">def</span> <span class="nf">ignored</span><span class="p">(</span><span class="n">node</span><span class="p">):</span> <span class="k">return</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="n">SKIP</span> <span class="ow">or</span> <span class="nf">bool</span><span class="p">(</span><span class="n">NOISE_CLASSES</span> <span class="o">&amp;</span> <span class="nf">set</span><span class="p">(</span><span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">class</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">).</span><span class="nf">split</span><span class="p">()))</span> <span class="k">def</span> <span class="nf">collapse</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">re</span><span class="p">.</span><span class="nf">sub</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">\s+</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span><span class="p">,</span> <span class="n">text</span><span class="p">).</span><span class="nf">strip</span><span class="p">()</span> <span class="k">def</span> <span class="nf">text_of</span><span class="p">(</span><span class="n">node</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Raw text with whitespace kept (for &lt;pre&gt;), minus ignored elements.</span><span class="sh">"""</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span> <span class="k">return</span> <span class="n">node</span> <span class="k">return</span> <span class="sh">""</span> <span class="k">if</span> <span class="nf">ignored</span><span class="p">(</span><span class="n">node</span><span class="p">)</span> <span class="k">else</span> <span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="nf">text_of</span><span class="p">(</span><span class="n">child</span><span class="p">)</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">])</span> <span class="k">def</span> <span class="nf">inline</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">base</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Render inline content as Markdown on one line.</span><span class="sh">"""</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span> <span class="k">return</span> <span class="n">node</span> <span class="k">if</span> <span class="nf">ignored</span><span class="p">(</span><span class="n">node</span><span class="p">):</span> <span class="k">return</span> <span class="sh">""</span> <span class="n">tag</span><span class="p">,</span> <span class="n">attrs</span> <span class="o">=</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">],</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">attrs</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">br</span><span class="sh">"</span><span class="p">:</span> <span class="k">return</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">img</span><span class="sh">"</span><span class="p">:</span> <span class="k">return</span> <span class="n">attrs</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">alt</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="n">inner</span> <span class="o">=</span> <span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="nf">inline</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">)</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">])</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">code</span><span class="sh">"</span><span class="p">:</span> <span class="k">return</span> <span class="sa">f</span><span class="sh">"</span><span class="s">`</span><span class="si">{</span><span class="nf">collapse</span><span class="p">(</span><span class="nf">text_of</span><span class="p">(</span><span class="n">node</span><span class="p">))</span><span class="si">}</span><span class="s">`</span><span class="sh">"</span> <span class="k">if</span> <span class="nf">collapse</span><span class="p">(</span><span class="nf">text_of</span><span class="p">(</span><span class="n">node</span><span class="p">))</span> <span class="k">else</span> <span class="sh">""</span> <span class="k">if</span> <span class="n">tag</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">strong</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">b</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">em</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">i</span><span class="sh">"</span><span class="p">)</span> <span class="ow">and</span> <span class="nf">collapse</span><span class="p">(</span><span class="n">inner</span><span class="p">):</span> <span class="n">mark</span> <span class="o">=</span> <span class="sh">"</span><span class="s">**</span><span class="sh">"</span> <span class="k">if</span> <span class="n">tag</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">strong</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">b</span><span class="sh">"</span><span class="p">)</span> <span class="k">else</span> <span class="sh">"</span><span class="s">*</span><span class="sh">"</span> <span class="k">return</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">mark</span><span class="si">}{</span><span class="nf">collapse</span><span class="p">(</span><span class="n">inner</span><span class="p">)</span><span class="si">}{</span><span class="n">mark</span><span class="si">}</span><span class="sh">"</span> <span class="n">href</span> <span class="o">=</span> <span class="n">attrs</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">href</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">a</span><span class="sh">"</span> <span class="ow">and</span> <span class="nf">collapse</span><span class="p">(</span><span class="n">inner</span><span class="p">)</span> <span class="ow">and</span> <span class="n">href</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">href</span><span class="p">.</span><span class="nf">startswith</span><span class="p">((</span><span class="sh">"</span><span class="s">#</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">javascript:</span><span class="sh">"</span><span class="p">)):</span> <span class="k">return</span> <span class="sa">f</span><span class="sh">"</span><span class="s">[</span><span class="si">{</span><span class="nf">collapse</span><span class="p">(</span><span class="n">inner</span><span class="p">)</span><span class="si">}</span><span class="s">](</span><span class="si">{</span><span class="n">urllib</span><span class="p">.</span><span class="n">parse</span><span class="p">.</span><span class="nf">urljoin</span><span class="p">(</span><span class="n">base</span><span class="p">,</span> <span class="n">href</span><span class="p">)</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span> <span class="k">return</span> <span class="n">inner</span> <span class="k">def</span> <span class="nf">render_list</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">out</span><span class="p">):</span> <span class="n">number</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">]:</span> <span class="k">if</span> <span class="ow">not</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">item</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="ow">or</span> <span class="n">item</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">!=</span> <span class="sh">"</span><span class="s">li</span><span class="sh">"</span><span class="p">:</span> <span class="k">continue</span> <span class="n">number</span> <span class="o">+=</span> <span class="mi">1</span> <span class="n">marker</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">number</span><span class="si">}</span><span class="s">.</span><span class="sh">"</span> <span class="k">if</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="sh">"</span><span class="s">ol</span><span class="sh">"</span> <span class="k">else</span> <span class="sh">"</span><span class="s">-</span><span class="sh">"</span> <span class="n">parts</span> <span class="o">=</span> <span class="p">[]</span> <span class="nf">render</span><span class="p">(</span><span class="n">item</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">parts</span><span class="p">)</span> <span class="k">if</span> <span class="n">parts</span><span class="p">:</span> <span class="n">indent</span> <span class="o">=</span> <span class="sh">"</span><span class="se">\n</span><span class="sh">"</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span> <span class="o">*</span> <span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">marker</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">marker</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span> <span class="o">+</span> <span class="n">indent</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">part</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="se">\n</span><span class="sh">"</span><span class="p">,</span> <span class="n">indent</span><span class="p">)</span> <span class="k">for</span> <span class="n">part</span> <span class="ow">in</span> <span class="n">parts</span><span class="p">))</span> <span class="k">def</span> <span class="nf">render_table</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">base</span><span class="p">):</span> <span class="n">rows</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="nf">find_all</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="sh">"</span><span class="s">tr</span><span class="sh">"</span><span class="p">):</span> <span class="n">cells</span> <span class="o">=</span> <span class="p">[</span><span class="nf">collapse</span><span class="p">(</span><span class="nf">inline</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">base</span><span class="p">)).</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="s">|</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="se">\\</span><span class="s">|</span><span class="sh">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="ow">and</span> <span class="n">cell</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">th</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">td</span><span class="sh">"</span><span class="p">)]</span> <span class="k">if</span> <span class="n">cells</span><span class="p">:</span> <span class="n">rows</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">cells</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">rows</span><span class="p">:</span> <span class="k">return</span> <span class="sh">""</span> <span class="n">width</span> <span class="o">=</span> <span class="nf">max</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">row</span><span class="p">)</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">)</span> <span class="n">rows</span> <span class="o">=</span> <span class="p">[</span><span class="n">row</span> <span class="o">+</span> <span class="p">[</span><span class="sh">""</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="n">width</span> <span class="o">-</span> <span class="nf">len</span><span class="p">(</span><span class="n">row</span><span class="p">))</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">]</span> <span class="n">lines</span> <span class="o">=</span> <span class="p">[</span><span class="sh">"</span><span class="s">| </span><span class="sh">"</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> | </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">rows</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> |</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">|</span><span class="sh">"</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> --- |</span><span class="sh">"</span> <span class="o">*</span> <span class="n">width</span><span class="p">]</span> <span class="k">return</span> <span class="sh">"</span><span class="se">\n</span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">lines</span> <span class="o">+</span> <span class="p">[</span><span class="sh">"</span><span class="s">| </span><span class="sh">"</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> | </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">row</span><span class="p">)</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> |</span><span class="sh">"</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">[</span><span class="mi">1</span><span class="p">:]])</span> <span class="k">def</span> <span class="nf">find_all</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">tag</span><span class="p">):</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">]:</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span> <span class="k">if</span> <span class="n">child</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="n">tag</span><span class="p">:</span> <span class="k">yield</span> <span class="n">child</span> <span class="k">elif</span> <span class="n">child</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="o">!=</span> <span class="sh">"</span><span class="s">table</span><span class="sh">"</span><span class="p">:</span> <span class="c1"># do not descend into nested tables </span> <span class="k">yield</span> <span class="k">from</span> <span class="nf">find_all</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">tag</span><span class="p">)</span> <span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">out</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Append Markdown blocks (strings) for the children of `node` to `out`.</span><span class="sh">"""</span> <span class="n">run</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">def</span> <span class="nf">flush</span><span class="p">():</span> <span class="n">text</span> <span class="o">=</span> <span class="nf">collapse</span><span class="p">(</span><span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">run</span><span class="p">))</span> <span class="n">run</span><span class="p">.</span><span class="nf">clear</span><span class="p">()</span> <span class="k">if</span> <span class="n">text</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="k">for</span> <span class="n">child</span> <span class="ow">in</span> <span class="n">node</span><span class="p">[</span><span class="sh">"</span><span class="s">children</span><span class="sh">"</span><span class="p">]:</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="ow">or</span> <span class="n">child</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">BLOCKS</span><span class="p">:</span> <span class="n">run</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="nf">inline</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">))</span> <span class="k">continue</span> <span class="nf">flush</span><span class="p">()</span> <span class="n">tag</span> <span class="o">=</span> <span class="n">child</span><span class="p">[</span><span class="sh">"</span><span class="s">tag</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="nf">ignored</span><span class="p">(</span><span class="n">child</span><span class="p">)</span> <span class="ow">or</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">hr</span><span class="sh">"</span><span class="p">:</span> <span class="k">continue</span> <span class="k">if</span> <span class="n">re</span><span class="p">.</span><span class="nf">fullmatch</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">h[1-6]</span><span class="sh">"</span><span class="p">,</span> <span class="n">tag</span><span class="p">):</span> <span class="n">title</span> <span class="o">=</span> <span class="nf">collapse</span><span class="p">(</span><span class="nf">inline</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">))</span> <span class="k">if</span> <span class="n">title</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">"</span><span class="s">#</span><span class="sh">"</span> <span class="o">*</span> <span class="nf">int</span><span class="p">(</span><span class="n">tag</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span> <span class="o">+</span> <span class="n">title</span><span class="p">)</span> <span class="k">elif</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">pre</span><span class="sh">"</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">"</span><span class="s">``` </span><span class="se">\n</span><span class="sh">"</span> <span class="o">+</span> <span class="nf">text_of</span><span class="p">(</span><span class="n">child</span><span class="p">).</span><span class="nf">strip</span><span class="p">(</span><span class="sh">"</span><span class="se">\n</span><span class="sh">"</span><span class="p">)</span> <span class="o">+</span> <span class="sh">"</span><span class="se">\n</span><span class="s"> ```</span><span class="sh">"</span><span class="p">)</span> <span class="k">elif</span> <span class="n">tag</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">ul</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">ol</span><span class="sh">"</span><span class="p">):</span> <span class="nf">render_list</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">out</span><span class="p">)</span> <span class="k">elif</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">table</span><span class="sh">"</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="nf">render_table</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">))</span> <span class="k">elif</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">dt</span><span class="sh">"</span><span class="p">:</span> <span class="n">out</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="nf">collapse</span><span class="p">(</span><span class="nf">text_of</span><span class="p">(</span><span class="n">child</span><span class="p">)))</span> <span class="k">elif</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">blockquote</span><span class="sh">"</span><span class="p">:</span> <span class="n">quoted</span> <span class="o">=</span> <span class="p">[]</span> <span class="nf">render</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">quoted</span><span class="p">)</span> <span class="n">out</span><span class="p">.</span><span class="nf">extend</span><span class="p">(</span><span class="sh">"</span><span class="s">&gt; </span><span class="sh">"</span> <span class="o">+</span> <span class="n">block</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="se">\n</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="se">\n</span><span class="s">&gt; </span><span class="sh">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">block</span> <span class="ow">in</span> <span class="n">quoted</span><span class="p">)</span> <span class="k">else</span><span class="p">:</span> <span class="nf">render</span><span class="p">(</span><span class="n">child</span><span class="p">,</span> <span class="n">base</span><span class="p">,</span> <span class="n">out</span><span class="p">)</span> <span class="nf">flush</span><span class="p">()</span> <span class="k">def</span> <span class="nf">to_markdown</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">base</span><span class="p">):</span> <span class="n">builder</span> <span class="o">=</span> <span class="nc">TreeBuilder</span><span class="p">()</span> <span class="n">builder</span><span class="p">.</span><span class="nf">feed</span><span class="p">(</span><span class="n">html</span><span class="p">)</span> <span class="n">builder</span><span class="p">.</span><span class="nf">close</span><span class="p">()</span> <span class="n">blocks</span> <span class="o">=</span> <span class="p">[]</span> <span class="nf">render</span><span class="p">(</span><span class="nf">main_content</span><span class="p">(</span><span class="n">builder</span><span class="p">.</span><span class="n">root</span><span class="p">),</span> <span class="n">base</span><span class="p">,</span> <span class="n">blocks</span><span class="p">)</span> <span class="k">return</span> <span class="p">[</span><span class="n">block</span> <span class="k">for</span> <span class="n">block</span> <span class="ow">in</span> <span class="n">blocks</span> <span class="k">if</span> <span class="n">block</span><span class="p">]</span> <span class="k">def</span> <span class="nf">prose</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Text with link targets and whitespace differences removed.</span><span class="sh">"""</span> <span class="k">return</span> <span class="nf">collapse</span><span class="p">(</span><span class="n">re</span><span class="p">.</span><span class="nf">sub</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">\]\([^)]*\)</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">]</span><span class="sh">"</span><span class="p">,</span> <span class="n">text</span><span class="p">))</span> <span class="k">def</span> <span class="nf">fingerprint</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="k">return</span> <span class="n">hashlib</span><span class="p">.</span><span class="nf">sha256</span><span class="p">(</span><span class="nf">prose</span><span class="p">(</span><span class="n">text</span><span class="p">).</span><span class="nf">encode</span><span class="p">(</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)).</span><span class="nf">hexdigest</span><span class="p">()[:</span><span class="mi">16</span><span class="p">]</span> <span class="k">def</span> <span class="nf">chunk</span><span class="p">(</span><span class="n">blocks</span><span class="p">,</span> <span class="n">max_level</span><span class="o">=</span><span class="mi">3</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Group blocks under their nearest heading (h1..h{max_level}); one chunk per section.</span><span class="sh">"""</span> <span class="n">chunks</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">body</span><span class="p">,</span> <span class="n">seen</span> <span class="o">=</span> <span class="p">[],</span> <span class="p">[],</span> <span class="p">[],</span> <span class="p">{}</span> <span class="k">def</span> <span class="nf">emit</span><span class="p">():</span> <span class="n">text</span> <span class="o">=</span> <span class="sh">"</span><span class="se">\n\n</span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">body</span><span class="p">).</span><span class="nf">strip</span><span class="p">()</span> <span class="n">body</span><span class="p">.</span><span class="nf">clear</span><span class="p">()</span> <span class="k">if</span> <span class="n">text</span><span class="p">:</span> <span class="n">name</span> <span class="o">=</span> <span class="sh">"</span><span class="s"> &gt; </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">"</span><span class="s">(top)</span><span class="sh">"</span> <span class="n">seen</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">seen</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span> <span class="n">chunk_id</span> <span class="o">=</span> <span class="n">name</span> <span class="k">if</span> <span class="n">seen</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">else</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s"> #</span><span class="si">{</span><span class="n">seen</span><span class="p">[</span><span class="n">name</span><span class="p">]</span><span class="si">}</span><span class="sh">"</span> <span class="n">chunks</span><span class="p">.</span><span class="nf">append</span><span class="p">({</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">:</span> <span class="n">chunk_id</span><span class="p">,</span> <span class="sh">"</span><span class="s">words</span><span class="sh">"</span><span class="p">:</span> <span class="nf">len</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="nf">split</span><span class="p">()),</span> <span class="sh">"</span><span class="s">sha256</span><span class="sh">"</span><span class="p">:</span> <span class="n">hashlib</span><span class="p">.</span><span class="nf">sha256</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="nf">encode</span><span class="p">(</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)).</span><span class="nf">hexdigest</span><span class="p">()[:</span><span class="mi">16</span><span class="p">],</span> <span class="sh">"</span><span class="s">fingerprint</span><span class="sh">"</span><span class="p">:</span> <span class="nf">fingerprint</span><span class="p">(</span><span class="n">text</span><span class="p">),</span> <span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">:</span> <span class="n">text</span><span class="p">})</span> <span class="k">for</span> <span class="n">block</span> <span class="ow">in</span> <span class="n">blocks</span><span class="p">:</span> <span class="n">heading</span> <span class="o">=</span> <span class="n">re</span><span class="p">.</span><span class="nf">match</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">(#{1,6}) (.*)</span><span class="sh">"</span><span class="p">,</span> <span class="n">block</span><span class="p">)</span> <span class="k">if</span> <span class="n">heading</span> <span class="ow">and</span> <span class="nf">len</span><span class="p">(</span><span class="n">heading</span><span class="p">.</span><span class="nf">group</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span> <span class="o">&lt;=</span> <span class="n">max_level</span><span class="p">:</span> <span class="nf">emit</span><span class="p">()</span> <span class="k">del</span> <span class="n">path</span><span class="p">[</span><span class="nf">len</span><span class="p">(</span><span class="n">heading</span><span class="p">.</span><span class="nf">group</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span> <span class="o">-</span> <span class="mi">1</span><span class="p">:]</span> <span class="n">path</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">heading</span><span class="p">.</span><span class="nf">group</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span> <span class="n">body</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">block</span><span class="p">)</span> <span class="nf">emit</span><span class="p">()</span> <span class="k">return</span> <span class="n">chunks</span> <span class="k">def</span> <span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">,</span> <span class="n">rename_threshold</span><span class="o">=</span><span class="mf">0.6</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Classify chunks by id: added, removed, renamed, changed (prose), links-only, unchanged.</span><span class="sh">"""</span> <span class="n">before</span> <span class="o">=</span> <span class="p">{</span><span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]:</span> <span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">old</span><span class="p">}</span> <span class="n">after</span> <span class="o">=</span> <span class="p">{</span><span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]:</span> <span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">new</span><span class="p">}</span> <span class="n">result</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">:</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">after</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">-</span> <span class="n">before</span><span class="p">.</span><span class="nf">keys</span><span class="p">()),</span> <span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">:</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">before</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">-</span> <span class="n">after</span><span class="p">.</span><span class="nf">keys</span><span class="p">()),</span> <span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">links-only</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">unchanged</span><span class="sh">"</span><span class="p">:</span> <span class="p">[]}</span> <span class="k">for</span> <span class="n">old_id</span> <span class="ow">in</span> <span class="nf">list</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">]):</span> <span class="c1"># a renamed heading looks like removed + added </span> <span class="n">scored</span> <span class="o">=</span> <span class="p">[(</span><span class="n">difflib</span><span class="p">.</span><span class="nc">SequenceMatcher</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="nf">prose</span><span class="p">(</span><span class="n">before</span><span class="p">[</span><span class="n">old_id</span><span class="p">][</span><span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">]),</span> <span class="nf">prose</span><span class="p">(</span><span class="n">after</span><span class="p">[</span><span class="n">new_id</span><span class="p">][</span><span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">])).</span><span class="nf">ratio</span><span class="p">(),</span> <span class="n">new_id</span><span class="p">)</span> <span class="k">for</span> <span class="n">new_id</span> <span class="ow">in</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">]]</span> <span class="k">if</span> <span class="n">scored</span> <span class="ow">and</span> <span class="nf">max</span><span class="p">(</span><span class="n">scored</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&gt;=</span> <span class="n">rename_threshold</span><span class="p">:</span> <span class="n">ratio</span><span class="p">,</span> <span class="n">new_id</span> <span class="o">=</span> <span class="nf">max</span><span class="p">(</span><span class="n">scored</span><span class="p">)</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">].</span><span class="nf">remove</span><span class="p">(</span><span class="n">old_id</span><span class="p">)</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">].</span><span class="nf">remove</span><span class="p">(</span><span class="n">new_id</span><span class="p">)</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">].</span><span class="nf">append</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">old_id</span><span class="si">}</span><span class="s"> -&gt; </span><span class="si">{</span><span class="n">new_id</span><span class="si">}</span><span class="s"> (similarity </span><span class="si">{</span><span class="n">ratio</span><span class="si">:</span><span class="p">.</span><span class="mi">3</span><span class="n">f</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">chunk_id</span> <span class="ow">in</span> <span class="p">[</span><span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">new</span> <span class="k">if</span> <span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="n">before</span><span class="p">]:</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="o">=</span> <span class="n">before</span><span class="p">[</span><span class="n">chunk_id</span><span class="p">],</span> <span class="n">after</span><span class="p">[</span><span class="n">chunk_id</span><span class="p">]</span> <span class="n">kind</span> <span class="o">=</span> <span class="p">(</span><span class="sh">"</span><span class="s">unchanged</span><span class="sh">"</span> <span class="k">if</span> <span class="n">a</span><span class="p">[</span><span class="sh">"</span><span class="s">sha256</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="n">b</span><span class="p">[</span><span class="sh">"</span><span class="s">sha256</span><span class="sh">"</span><span class="p">]</span> <span class="k">else</span> <span class="sh">"</span><span class="s">links-only</span><span class="sh">"</span> <span class="k">if</span> <span class="n">a</span><span class="p">[</span><span class="sh">"</span><span class="s">fingerprint</span><span class="sh">"</span><span class="p">]</span> <span class="o">==</span> <span class="n">b</span><span class="p">[</span><span class="sh">"</span><span class="s">fingerprint</span><span class="sh">"</span><span class="p">]</span> <span class="k">else</span> <span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">)</span> <span class="n">result</span><span class="p">[</span><span class="n">kind</span><span class="p">].</span><span class="nf">append</span><span class="p">(</span><span class="n">chunk_id</span><span class="p">)</span> <span class="k">return</span> <span class="n">result</span> <span class="k">def</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Parse robots.txt. urllib.robotparser ends a group at a blank line; RFC 9309 does not.</span><span class="sh">"""</span> <span class="n">rules</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">robotparser</span><span class="p">.</span><span class="nc">RobotFileParser</span><span class="p">()</span> <span class="n">rules</span><span class="p">.</span><span class="nf">parse</span><span class="p">([</span><span class="n">line</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">text</span><span class="p">.</span><span class="nf">splitlines</span><span class="p">()</span> <span class="k">if</span> <span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()])</span> <span class="k">return</span> <span class="n">rules</span> <span class="k">def</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">url</span><span class="p">):</span> <span class="n">parts</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">parse</span><span class="p">.</span><span class="nf">urlsplit</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="n">text</span> <span class="o">=</span> <span class="nf">get</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">parts</span><span class="p">.</span><span class="n">scheme</span><span class="si">}</span><span class="s">://</span><span class="si">{</span><span class="n">parts</span><span class="p">.</span><span class="n">netloc</span><span class="si">}</span><span class="s">/robots.txt</span><span class="sh">"</span><span class="p">)</span> <span class="k">except</span> <span class="n">urllib</span><span class="p">.</span><span class="n">error</span><span class="p">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="k">if</span> <span class="n">err</span><span class="p">.</span><span class="n">code</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">404</span><span class="p">,</span> <span class="mi">410</span><span class="p">):</span> <span class="k">raise</span> <span class="n">text</span> <span class="o">=</span> <span class="sh">""</span> <span class="c1"># no robots.txt: nothing is disallowed </span> <span class="n">rules</span> <span class="o">=</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">rules</span><span class="p">.</span><span class="nf">can_fetch</span><span class="p">(</span><span class="n">USER_AGENT</span><span class="p">,</span> <span class="n">url</span><span class="p">):</span> <span class="n">sys</span><span class="p">.</span><span class="nf">exit</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">robots.txt disallows </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s">; stopping</span><span class="sh">"</span><span class="p">)</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="nf">max</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">rules</span><span class="p">.</span><span class="nf">crawl_delay</span><span class="p">(</span><span class="n">USER_AGENT</span><span class="p">)</span> <span class="ow">or</span> <span class="mi">0</span><span class="p">))</span> <span class="k">return</span> <span class="nf">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">url</span><span class="p">):</span> <span class="n">request</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nc">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="sh">"</span><span class="s">User-Agent</span><span class="sh">"</span><span class="p">:</span> <span class="n">USER_AGENT</span><span class="p">})</span> <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nf">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">30</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span> <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="nf">read</span><span class="p">().</span><span class="nf">decode</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="n">headers</span><span class="p">.</span><span class="nf">get_content_charset</span><span class="p">()</span> <span class="ow">or</span> <span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">replace</span><span class="sh">"</span><span class="p">)</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">):</span> <span class="k">if</span> <span class="n">args</span><span class="p">[:</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="p">[</span><span class="sh">"</span><span class="s">convert</span><span class="sh">"</span><span class="p">]</span> <span class="ow">and</span> <span class="nf">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="o">==</span> <span class="mi">4</span><span class="p">:</span> <span class="n">url</span><span class="p">,</span> <span class="n">md_file</span><span class="p">,</span> <span class="n">json_file</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span> <span class="n">html</span> <span class="o">=</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="n">captured</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="nf">now</span><span class="p">(</span><span class="n">timezone</span><span class="p">.</span><span class="n">utc</span><span class="p">).</span><span class="nf">strftime</span><span class="p">(</span><span class="sh">"</span><span class="s">%Y-%m-%dT%H:%M:%SZ</span><span class="sh">"</span><span class="p">)</span> <span class="n">blocks</span> <span class="o">=</span> <span class="nf">to_markdown</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span> <span class="n">chunks</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">(</span><span class="n">blocks</span><span class="p">)</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">md_file</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">handle</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="sh">"</span><span class="se">\n\n</span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">blocks</span><span class="p">)</span> <span class="o">+</span> <span class="sh">"</span><span class="se">\n</span><span class="sh">"</span><span class="p">)</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">json_file</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="nf">dump</span><span class="p">({</span><span class="sh">"</span><span class="s">url</span><span class="sh">"</span><span class="p">:</span> <span class="n">url</span><span class="p">,</span> <span class="sh">"</span><span class="s">captured_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">captured</span><span class="p">,</span> <span class="sh">"</span><span class="s">chunks</span><span class="sh">"</span><span class="p">:</span> <span class="n">chunks</span><span class="p">},</span> <span class="n">handle</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s"> at </span><span class="si">{</span><span class="n">captured</span><span class="si">}</span><span class="s">: </span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">html</span><span class="p">)</span><span class="si">:</span><span class="p">,</span><span class="si">}</span><span class="s"> characters of HTML -&gt; </span><span class="sh">"</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="nf">sum</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">blocks</span><span class="p">)</span><span class="si">:</span><span class="p">,</span><span class="si">}</span><span class="s"> characters of Markdown in </span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">chunks</span><span class="p">)</span><span class="si">}</span><span class="s"> chunks</span><span class="sh">"</span><span class="p">)</span> <span class="k">elif</span> <span class="n">args</span><span class="p">[:</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="p">[</span><span class="sh">"</span><span class="s">compare</span><span class="sh">"</span><span class="p">]</span> <span class="ow">and</span> <span class="nf">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="o">==</span> <span class="mi">3</span><span class="p">:</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">a</span><span class="p">,</span> <span class="nf">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">b</span><span class="p">:</span> <span class="n">old</span><span class="p">,</span> <span class="n">new</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="n">json</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">[</span><span class="sh">"</span><span class="s">chunks</span><span class="sh">"</span><span class="p">],</span> <span class="n">new</span><span class="p">[</span><span class="sh">"</span><span class="s">chunks</span><span class="sh">"</span><span class="p">])</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">old</span><span class="p">[</span><span class="sh">'</span><span class="s">url</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="s"> -&gt; </span><span class="si">{</span><span class="n">new</span><span class="p">[</span><span class="sh">'</span><span class="s">url</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">kind</span><span class="p">,</span> <span class="n">ids</span> <span class="ow">in</span> <span class="n">result</span><span class="p">.</span><span class="nf">items</span><span class="p">():</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">kind</span><span class="si">:</span><span class="o">&lt;</span><span class="mi">10</span><span class="si">}</span><span class="s"> </span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">ids</span><span class="p">)</span><span class="si">:</span><span class="o">&gt;</span><span class="mi">3</span><span class="si">}</span><span class="sh">"</span> <span class="o">+</span> <span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="se">\n</span><span class="s"> </span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="sh">"</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">ids</span> <span class="k">if</span> <span class="n">kind</span> <span class="o">!=</span> <span class="sh">"</span><span class="s">unchanged</span><span class="sh">"</span><span class="p">))</span> <span class="k">else</span><span class="p">:</span> <span class="n">sys</span><span class="p">.</span><span class="nf">exit</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="sh">"</span><span class="s">__main__</span><span class="sh">"</span><span class="p">:</span> <span class="nf">main</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span> </code></pre> </div> <p></p> <p></p> test_page_to_chunks.py (85 lines) <br> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="kn">import</span> <span class="n">unittest</span> <span class="kn">from</span> <span class="n">page_to_chunks</span> <span class="kn">import</span> <span class="n">chunk</span><span class="p">,</span> <span class="n">compare</span><span class="p">,</span> <span class="n">fingerprint</span><span class="p">,</span> <span class="n">robots_rules</span><span class="p">,</span> <span class="n">to_markdown</span> <span class="n">BASE</span> <span class="o">=</span> <span class="sh">"</span><span class="s">https://example.org/docs/page.html</span><span class="sh">"</span> <span class="k">def</span> <span class="nf">md</span><span class="p">(</span><span class="n">html</span><span class="p">):</span> <span class="k">return</span> <span class="nf">to_markdown</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">BASE</span><span class="p">)</span> <span class="k">class</span> <span class="nc">Markdown</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_main_content_only_and_noise_removed</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">html</span> <span class="o">=</span> <span class="sh">"""</span><span class="s">&lt;html&gt;&lt;body&gt;&lt;nav&gt;Menu&lt;/nav&gt;&lt;div role=</span><span class="sh">"</span><span class="s">main</span><span class="sh">"</span><span class="s">&gt; &lt;h1&gt;Title&lt;a class=</span><span class="sh">"</span><span class="s">headerlink</span><span class="sh">"</span><span class="s"> href=</span><span class="sh">"</span><span class="s">#t</span><span class="sh">"</span><span class="s">&gt;¶&lt;/a&gt;&lt;/h1&gt; &lt;p&gt;Hello &lt;b&gt;bold&lt;/b&gt; world.&lt;/p&gt;&lt;script&gt;track()&lt;/script&gt;&lt;/div&gt;&lt;footer&gt;Legal&lt;/footer&gt;&lt;/body&gt;&lt;/html&gt;</span><span class="sh">"""</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">md</span><span class="p">(</span><span class="n">html</span><span class="p">),</span> <span class="p">[</span><span class="sh">"</span><span class="s"># Title</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Hello **bold** world.</span><span class="sh">"</span><span class="p">])</span> <span class="k">def</span> <span class="nf">test_links_become_absolute_and_anchors_become_text</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">html</span> <span class="o">=</span> <span class="sh">'</span><span class="s">&lt;main&gt;&lt;p&gt;See &lt;a href=</span><span class="sh">"</span><span class="s">../api.html#x</span><span class="sh">"</span><span class="s">&gt;the API&lt;/a&gt; and &lt;a href=</span><span class="sh">"</span><span class="s">#top</span><span class="sh">"</span><span class="s">&gt;top&lt;/a&gt;.&lt;/p&gt;&lt;/main&gt;</span><span class="sh">'</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">md</span><span class="p">(</span><span class="n">html</span><span class="p">),</span> <span class="p">[</span><span class="sh">"</span><span class="s">See [the API](https://example.org/api.html#x) and top.</span><span class="sh">"</span><span class="p">])</span> <span class="k">def</span> <span class="nf">test_code_inline_and_pre_whitespace</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">html</span> <span class="o">=</span> <span class="sh">"</span><span class="s">&lt;main&gt;&lt;p&gt;Call &lt;code&gt;json.dumps( )&lt;/code&gt;:&lt;/p&gt;&lt;pre&gt;&amp;gt;&amp;gt;&amp;gt; x = 1</span><span class="se">\n</span><span class="s"> y&lt;/pre&gt;&lt;/main&gt;</span><span class="sh">"</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">md</span><span class="p">(</span><span class="n">html</span><span class="p">),</span> <span class="p">[</span><span class="sh">"</span><span class="s">Call `json.dumps( )`:</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">``` </span><span class="se">\n</span><span class="s">&gt;&gt;&gt; x = 1</span><span class="se">\n</span><span class="s"> y</span><span class="se">\n</span><span class="s"> ```</span><span class="sh">"</span><span class="p">])</span> <span class="k">def</span> <span class="nf">test_nested_lists</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">html</span> <span class="o">=</span> <span class="sh">"</span><span class="s">&lt;main&gt;&lt;ol&gt;&lt;li&gt;&lt;p&gt;One&lt;/p&gt;&lt;ul&gt;&lt;li&gt;a&lt;/li&gt;&lt;li&gt;b&lt;/li&gt;&lt;/ul&gt;&lt;/li&gt;&lt;li&gt;Two&lt;/li&gt;&lt;/ol&gt;&lt;/main&gt;</span><span class="sh">"</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">md</span><span class="p">(</span><span class="n">html</span><span class="p">),</span> <span class="p">[</span><span class="sh">"</span><span class="s">1. One</span><span class="se">\n</span><span class="s"> - a</span><span class="se">\n</span><span class="s"> - b</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2. Two</span><span class="sh">"</span><span class="p">])</span> <span class="k">def</span> <span class="nf">test_table_with_pipes_and_ragged_rows</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">html</span> <span class="o">=</span> <span class="p">(</span><span class="sh">"</span><span class="s">&lt;main&gt;&lt;table&gt;&lt;tr&gt;&lt;th&gt;JSON&lt;/th&gt;&lt;th&gt;Python&lt;/th&gt;&lt;/tr&gt;</span><span class="sh">"</span> <span class="sh">"</span><span class="s">&lt;tr&gt;&lt;td&gt;&lt;p&gt;object&lt;/p&gt;&lt;/td&gt;&lt;td&gt;dict&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;a|b&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/main&gt;</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">md</span><span class="p">(</span><span class="n">html</span><span class="p">),</span> <span class="p">[</span><span class="sh">"</span><span class="s">| JSON | Python |</span><span class="se">\n</span><span class="s">| --- | --- |</span><span class="se">\n</span><span class="s">| object | dict |</span><span class="se">\n</span><span class="s">| a</span><span class="se">\\</span><span class="s">|b | |</span><span class="sh">"</span><span class="p">])</span> <span class="k">def</span> <span class="nf">test_definition_list_and_blockquote</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">html</span> <span class="o">=</span> <span class="p">(</span><span class="sh">"</span><span class="s">&lt;main&gt;&lt;dl&gt;&lt;dt&gt;json.dump(&lt;em&gt;obj&lt;/em&gt;, &lt;em&gt;fp&lt;/em&gt;)&lt;a class=</span><span class="sh">'</span><span class="s">headerlink</span><span class="sh">'</span><span class="s">&gt;¶&lt;/a&gt;&lt;/dt&gt;</span><span class="sh">"</span> <span class="sh">"</span><span class="s">&lt;dd&gt;&lt;p&gt;Serialize.&lt;/p&gt;&lt;/dd&gt;&lt;/dl&gt;&lt;blockquote&gt;&lt;p&gt;Quoted&lt;/p&gt;&lt;/blockquote&gt;&lt;/main&gt;</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">md</span><span class="p">(</span><span class="n">html</span><span class="p">),</span> <span class="p">[</span><span class="sh">"</span><span class="s">json.dump(obj, fp)</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Serialize.</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">&gt; Quoted</span><span class="sh">"</span><span class="p">])</span> <span class="k">def</span> <span class="nf">test_unclosed_paragraphs_do_not_swallow_the_page</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">md</span><span class="p">(</span><span class="sh">"</span><span class="s">&lt;main&gt;&lt;p&gt;First&lt;p&gt;Second&lt;/main&gt;&lt;p&gt;outside</span><span class="sh">"</span><span class="p">),</span> <span class="p">[</span><span class="sh">"</span><span class="s">First</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Second</span><span class="sh">"</span><span class="p">])</span> <span class="k">class</span> <span class="nc">Chunks</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_heading_paths_levels_and_duplicates</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">blocks</span> <span class="o">=</span> <span class="p">[</span><span class="sh">"</span><span class="s">intro</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># Guide</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">## Setup</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">b</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">#### Deep</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">c</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">## Setup</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">d</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">### Notes</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">e</span><span class="sh">"</span><span class="p">]</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">(</span><span class="n">blocks</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">([</span><span class="n">c</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">result</span><span class="p">],</span> <span class="p">[</span><span class="sh">"</span><span class="s">(top)</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Guide</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Guide &gt; Setup</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Guide &gt; Setup #2</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Guide &gt; Setup &gt; Notes</span><span class="sh">"</span><span class="p">])</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="mi">2</span><span class="p">][</span><span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">## Setup</span><span class="se">\n\n</span><span class="s">b</span><span class="se">\n\n</span><span class="s">#### Deep</span><span class="se">\n\n</span><span class="s">c</span><span class="sh">"</span><span class="p">)</span> <span class="k">def</span> <span class="nf">test_fingerprint_ignores_link_targets</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">fingerprint</span><span class="p">(</span><span class="sh">"</span><span class="s">See [docs](https://x/3.13/a.html).</span><span class="sh">"</span><span class="p">),</span> <span class="nf">fingerprint</span><span class="p">(</span><span class="sh">"</span><span class="s">See &lt;a href=</span><span class="sh">"</span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">x</span><span class="o">/</span><span class="mf">3.14</span><span class="o">/</span><span class="n">a</span><span class="p">.</span><span class="n">html</span><span class="sh">"</span><span class="s">&gt;docs&lt;/a&gt;.</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertNotEqual</span><span class="p">(</span><span class="nf">fingerprint</span><span class="p">(</span><span class="sh">"</span><span class="s">See [docs](u).</span><span class="sh">"</span><span class="p">),</span> <span class="nf">fingerprint</span><span class="p">(</span><span class="sh">"</span><span class="s">See [the docs](u).</span><span class="sh">"</span><span class="p">))</span> <span class="k">def</span> <span class="nf">test_compare</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">old</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">([</span><span class="sh">"</span><span class="s"># A</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">same</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># B</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">[x](https://v1)</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># C</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">old text</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># D</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">gone</span><span class="sh">"</span><span class="p">])</span> <span class="n">new</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">([</span><span class="sh">"</span><span class="s"># A</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">same</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># B</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">[x](https://v2)</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># C</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">new text</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s"># E</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">fresh</span><span class="sh">"</span><span class="p">])</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">),</span> <span class="p">{</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">E</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">D</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">C</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">links-only</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">B</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">unchanged</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">A</span><span class="sh">"</span><span class="p">]})</span> <span class="k">def</span> <span class="nf">test_renamed_heading_is_paired</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">body</span> <span class="o">=</span> <span class="sh">"</span><span class="s">The json module can be run from the shell to validate and pretty-print input.</span><span class="sh">"</span> <span class="n">old</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">([</span><span class="sh">"</span><span class="s"># Command Line Interface</span><span class="sh">"</span><span class="p">,</span> <span class="n">body</span><span class="p">])</span> <span class="n">new</span> <span class="o">=</span> <span class="nf">chunk</span><span class="p">([</span><span class="sh">"</span><span class="s"># Command-line interface</span><span class="sh">"</span><span class="p">,</span> <span class="n">body</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> Also as python -m json.</span><span class="sh">"</span><span class="p">])</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">],</span> <span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">]),</span> <span class="p">([],</span> <span class="p">[]))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">]),</span> <span class="mi">1</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertTrue</span><span class="p">(</span><span class="n">result</span><span class="p">[</span><span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">][</span><span class="mi">0</span><span class="p">].</span><span class="nf">startswith</span><span class="p">(</span><span class="sh">"</span><span class="s">Command Line Interface -&gt; Command-line interface (</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">compare</span><span class="p">(</span><span class="n">old</span><span class="p">,</span> <span class="n">new</span><span class="p">,</span> <span class="n">rename_threshold</span><span class="o">=</span><span class="mf">0.99</span><span class="p">)[</span><span class="sh">"</span><span class="s">renamed</span><span class="sh">"</span><span class="p">],</span> <span class="p">[])</span> <span class="k">class</span> <span class="nc">Robots</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_blank_line_does_not_end_the_group</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">rules</span> <span class="o">=</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="sh">"</span><span class="s">User-agent: *</span><span class="se">\n</span><span class="s">Disallow: /dev</span><span class="se">\n\n</span><span class="s"># EOL versions</span><span class="se">\n</span><span class="s">Disallow: /3.9/</span><span class="se">\n</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertFalse</span><span class="p">(</span><span class="n">rules</span><span class="p">.</span><span class="nf">can_fetch</span><span class="p">(</span><span class="sh">"</span><span class="s">page-to-chunks</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">https://x/3.9/library/json.html</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertFalse</span><span class="p">(</span><span class="n">rules</span><span class="p">.</span><span class="nf">can_fetch</span><span class="p">(</span><span class="sh">"</span><span class="s">page-to-chunks</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">https://x/dev/</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertTrue</span><span class="p">(</span><span class="n">rules</span><span class="p">.</span><span class="nf">can_fetch</span><span class="p">(</span><span class="sh">"</span><span class="s">page-to-chunks</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">https://x/3.14/library/json.html</span><span class="sh">"</span><span class="p">))</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="sh">"</span><span class="s">__main__</span><span class="sh">"</span><span class="p">:</span> <span class="n">unittest</span><span class="p">.</span><span class="nf">main</span><span class="p">()</span> </code></pre> </div> <p></p> <p>Run the tests with <code>python -m unittest -v test_page_to_chunks</code>.</p> <p><em>This article and its code were drafted by an AI assistant at the account owner's request. The code was run against docs.python.org on 27 September 2026 at 13:07:49 and 13:07:51 UTC, and the tests passed the same day. The blank-line handling for <code>robots.txt</code> was added later the same day after a review; it makes the same <code>robots.txt</code> decisions for both pages, so it does not change the run shown above.</em></p> python rag markdown tutorial Abdulwahab Sun, 27 Sep 2026 18:47:26 +0000 https://dev.to/abdulwhab95/tracking-product-price-and-stock-changes-from-json-ld-in-python-4mm6 https://dev.to/abdulwhab95/tracking-product-price-and-stock-changes-from-json-ld-in-python-4mm6 <p>Many product pages carry a machine-readable copy of their own data: a <code>&lt;script type="application/ld+json"&gt;</code> block with a schema.org <code>Product</code>, written for search engines. It holds the name, the price or price range, the currency and the stock status as JSON. That makes it a steadier thing to read than CSS classes, which change whenever the theme changes.</p> <p>This post builds a small standard-library Python script that:</p> <ol> <li>finds the JSON-LD blocks in a page,</li> <li>picks out every <code>Product</code>, whatever shape the markup takes,</li> <li>turns them into flat rows with exact decimal prices, one per <code>sku</code> (or one per page when there is no <code>sku</code>),</li> <li>saves a snapshot of several pages, politely,</li> <li>compares two snapshots and reports what was added, removed or changed, without calling a failed download a "removal".</li> </ol> <p>At the end there is a real run from 27 September 2026 (UTC) and the complete script.</p> <h2> Before you point this at a real store </h2> <p>Read the site's terms and its <code>robots.txt</code> first. The script checks <code>robots.txt</code> and honours <code>Crawl-delay</code>, but it cannot read terms for you, and many shops offer an official feed or API that is a better source than their pages.</p> <p>The examples use two public sources:</p> <ul> <li> <strong>web-scraping.dev</strong>, a sandbox shop built for scraping practice. Its home page calls itself "Safe, legal, and designed for learning". Its <code>robots.txt</code> blocks only <code>/robots-disallowed</code> for all agents and asks for <code>Crawl-delay: 2</code>.</li> <li> <strong>The Internet Archive's Wayback Machine</strong>, for older copies of the same pages, so there is a real "before" to compare with.</li> </ul> <h2> What the markup looks like </h2> <p>This is the <code>offers</code> part of the JSON-LD on <code>https://web-scraping.dev/product/1</code>, fetched on 27 September 2026 at 12:54 UTC. The page has one JSON-LD block; the full block also carries a description, an image, a brand, a rating and five reviews.</p> <p><em>Excerpt of <code>product-1-jsonld.txt</code>, lines 5-17:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight json"><code><span class="p">{</span><span class="w"> </span><span class="nl">"@type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Product"</span><span class="p">,</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Box of Chocolate Candy"</span><span class="p">,</span><span class="w"> </span><span class="nl">"offers"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"@type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"AggregateOffer"</span><span class="p">,</span><span class="w"> </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://web-scraping.dev/product/1"</span><span class="p">,</span><span class="w"> </span><span class="nl">"priceCurrency"</span><span class="p">:</span><span class="w"> </span><span class="s2">"USD"</span><span class="p">,</span><span class="w"> </span><span class="nl">"lowPrice"</span><span class="p">:</span><span class="w"> </span><span class="s2">"9.99"</span><span class="p">,</span><span class="w"> </span><span class="nl">"highPrice"</span><span class="p">:</span><span class="w"> </span><span class="s2">"19.99"</span><span class="p">,</span><span class="w"> </span><span class="nl">"offerCount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"6"</span><span class="p">,</span><span class="w"> </span><span class="nl">"availability"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://schema.org/InStock"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <p>That is one shape. Real pages vary, and a parser has to cope with all of these:</p> <ul> <li> <code>offers</code> can be a single <code>Offer</code> with <code>price</code>, an <code>AggregateOffer</code> with <code>lowPrice</code> and <code>highPrice</code>, or a list of <code>Offer</code> objects.</li> <li>The price can sit in <code>offers.priceSpecification.price</code>. Google's product snippet documentation says that when both <code>offers.price</code> and <code>offers.priceSpecification</code> are present, it uses <code>offers.price</code>. The script does the same.</li> <li> <code>@type</code> can be a string, a list such as <code>["Product", "Thing"]</code>, or a compact form such as <code>schema:Product</code>.</li> <li> <code>availability</code> is normally a full URL such as <code>https://schema.org/InStock</code>; the code also accepts the <code>http://</code> form and a bare <code>InStock</code>.</li> <li>The data can be wrapped in a top-level array or an <code>@graph</code> list, and a page can hold several blocks, one of them broken.</li> <li> <code>price</code> can be a JSON number or a string. schema.org asks for a <code>.</code> decimal point and no currency symbols or thousands separators, but not every page follows that.</li> </ul> <h2> Step 1: collect the JSON-LD blocks </h2> <p>Python's <code>html.parser</code> treats the inside of a <code>&lt;script&gt;</code> element as raw text, so the JSON arrives in <code>handle_data</code> untouched. The <code>type</code> attribute is compared without any parameters, so a value such as <code>application/ld+json; charset=utf-8</code> still matches.</p> <p><em>Excerpt of <code>catalog_snapshot.py</code>, lines 24-43:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">class</span> <span class="nc">JsonLdScripts</span><span class="p">(</span><span class="n">HTMLParser</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Collect the raw text of every &lt;script type=</span><span class="sh">"</span><span class="s">application/ld+json</span><span class="sh">"</span><span class="s">&gt; element.</span><span class="sh">"""</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="nf">super</span><span class="p">().</span><span class="nf">__init__</span><span class="p">()</span> <span class="n">self</span><span class="p">.</span><span class="n">blocks</span><span class="p">,</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="o">=</span> <span class="p">[],</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">handle_starttag</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">tag</span><span class="p">,</span> <span class="n">attrs</span><span class="p">):</span> <span class="n">kind</span> <span class="o">=</span> <span class="p">(</span><span class="nf">dict</span><span class="p">(</span><span class="n">attrs</span><span class="p">).</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">type</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">""</span><span class="p">).</span><span class="nf">split</span><span class="p">(</span><span class="sh">"</span><span class="s">;</span><span class="sh">"</span><span class="p">)[</span><span class="mi">0</span><span class="p">].</span><span class="nf">strip</span><span class="p">().</span><span class="nf">lower</span><span class="p">()</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">script</span><span class="sh">"</span> <span class="ow">and</span> <span class="n">kind</span> <span class="o">==</span> <span class="sh">"</span><span class="s">application/ld+json</span><span class="sh">"</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">def</span> <span class="nf">handle_data</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="k">if</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="k">def</span> <span class="nf">handle_endtag</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">tag</span><span class="p">):</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">script</span><span class="sh">"</span> <span class="ow">and</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">blocks</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">_parts</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="o">=</span> <span class="bp">None</span> </code></pre> </div> <h2> Step 2: walk the graph and find products </h2> <p><code>nodes()</code> yields every JSON object in a block, at any depth. That covers <code>@graph</code>, top-level arrays, and products nested inside other objects. <code>short()</code> reduces a schema.org term to its last part, so the type and availability checks don't care about <code>https://</code>, <code>http://</code> or <code>schema:</code>.</p> <p><em>Excerpt of <code>catalog_snapshot.py</code>, lines 46-64:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">nodes</span><span class="p">(</span><span class="n">value</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Yield every JSON object in a JSON-LD value: top-level arrays, @graph, nesting.</span><span class="sh">"""</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">value</span><span class="p">:</span> <span class="k">yield</span> <span class="k">from</span> <span class="nf">nodes</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="k">elif</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span> <span class="k">yield</span> <span class="n">value</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">value</span><span class="p">.</span><span class="nf">values</span><span class="p">():</span> <span class="k">yield</span> <span class="k">from</span> <span class="nf">nodes</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="k">def</span> <span class="nf">short</span><span class="p">(</span><span class="n">term</span><span class="p">):</span> <span class="sh">"""'</span><span class="s">https://schema.org/InStock</span><span class="sh">'</span><span class="s"> -&gt; </span><span class="sh">'</span><span class="s">InStock</span><span class="sh">'</span><span class="s">, </span><span class="sh">'</span><span class="s">schema:Product</span><span class="sh">'</span><span class="s"> -&gt; </span><span class="sh">'</span><span class="s">Product</span><span class="sh">'</span><span class="s">.</span><span class="sh">"""</span> <span class="k">return</span> <span class="n">re</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">[/:#]</span><span class="sh">"</span><span class="p">,</span> <span class="n">term</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">term</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">else</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">is_type</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">wanted</span><span class="p">):</span> <span class="n">types</span> <span class="o">=</span> <span class="n">node</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="n">wanted</span> <span class="ow">in</span> <span class="p">[</span><span class="nf">short</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="p">(</span><span class="n">types</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">types</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">types</span><span class="p">])]</span> </code></pre> </div> <h2> Step 3: one flat row per product, with exact prices </h2> <p>Prices stay as text produced from <code>Decimal</code>, never <code>float</code>, so <code>19.99</code> stays <code>19.99</code> and <code>20.00</code> and <code>20</code> compare equal. A value that isn't a clean number, such as <code>1,299.00</code>, becomes <code>None</code> rather than a wrong number.</p> <p><em>Excerpt of <code>catalog_snapshot.py</code>, lines 67-104:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">clean_price</span><span class="p">(</span><span class="n">value</span><span class="p">):</span> <span class="sh">"""'</span><span class="s">19.990</span><span class="sh">'</span><span class="s"> -&gt; </span><span class="sh">'</span><span class="s">19.99</span><span class="sh">'</span><span class="s">, 20 -&gt; </span><span class="sh">'</span><span class="s">20</span><span class="sh">'</span><span class="s">, </span><span class="sh">''</span><span class="s"> or </span><span class="sh">'</span><span class="s">1,299.00</span><span class="sh">'</span><span class="s"> -&gt; None. Text, not float.</span><span class="sh">"""</span> <span class="k">if</span> <span class="n">value</span> <span class="ow">is</span> <span class="bp">None</span> <span class="ow">or</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">bool</span><span class="p">):</span> <span class="k">return</span> <span class="bp">None</span> <span class="k">try</span><span class="p">:</span> <span class="n">number</span> <span class="o">=</span> <span class="nc">Decimal</span><span class="p">(</span><span class="nf">str</span><span class="p">(</span><span class="n">value</span><span class="p">).</span><span class="nf">strip</span><span class="p">())</span> <span class="k">except</span> <span class="n">InvalidOperation</span><span class="p">:</span> <span class="k">return</span> <span class="bp">None</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">number</span><span class="p">.</span><span class="nf">is_finite</span><span class="p">():</span> <span class="k">return</span> <span class="bp">None</span> <span class="n">text</span> <span class="o">=</span> <span class="nf">format</span><span class="p">(</span><span class="n">number</span><span class="p">,</span> <span class="sh">"</span><span class="s">f</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="n">text</span><span class="p">.</span><span class="nf">rstrip</span><span class="p">(</span><span class="sh">"</span><span class="s">0</span><span class="sh">"</span><span class="p">).</span><span class="nf">rstrip</span><span class="p">(</span><span class="sh">"</span><span class="s">.</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="sh">"</span><span class="s">.</span><span class="sh">"</span> <span class="ow">in</span> <span class="n">text</span> <span class="k">else</span> <span class="n">text</span> <span class="k">def</span> <span class="nf">to_row</span><span class="p">(</span><span class="n">product</span><span class="p">,</span> <span class="n">page_url</span><span class="p">):</span> <span class="n">offers</span> <span class="o">=</span> <span class="n">product</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">offers</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]</span> <span class="n">offers</span> <span class="o">=</span> <span class="p">[</span><span class="n">o</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="p">(</span><span class="n">offers</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">offers</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">offers</span><span class="p">])</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">o</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)]</span> <span class="n">lows</span><span class="p">,</span> <span class="n">highs</span> <span class="o">=</span> <span class="p">[],</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">offer</span> <span class="ow">in</span> <span class="n">offers</span><span class="p">:</span> <span class="n">spec</span> <span class="o">=</span> <span class="n">offer</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">priceSpecification</span><span class="sh">"</span><span class="p">)</span> <span class="n">price</span> <span class="o">=</span> <span class="n">offer</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">price</span><span class="sh">"</span><span class="p">,</span> <span class="n">spec</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">price</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">spec</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="k">else</span> <span class="bp">None</span><span class="p">)</span> <span class="n">low</span><span class="p">,</span> <span class="n">high</span> <span class="o">=</span> <span class="nf">clean_price</span><span class="p">(</span><span class="n">offer</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">lowPrice</span><span class="sh">"</span><span class="p">,</span> <span class="n">price</span><span class="p">)),</span> <span class="nf">clean_price</span><span class="p">(</span><span class="n">offer</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">highPrice</span><span class="sh">"</span><span class="p">,</span> <span class="n">price</span><span class="p">))</span> <span class="n">lows</span> <span class="o">+=</span> <span class="p">[</span><span class="nc">Decimal</span><span class="p">(</span><span class="n">low</span><span class="p">)]</span> <span class="k">if</span> <span class="n">low</span> <span class="k">else</span> <span class="p">[]</span> <span class="n">highs</span> <span class="o">+=</span> <span class="p">[</span><span class="nc">Decimal</span><span class="p">(</span><span class="n">high</span><span class="p">)]</span> <span class="k">if</span> <span class="n">high</span> <span class="k">else</span> <span class="p">[]</span> <span class="n">stock</span> <span class="o">=</span> <span class="nf">sorted</span><span class="p">({</span><span class="nf">short</span><span class="p">(</span><span class="n">o</span><span class="p">[</span><span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">])</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="n">offers</span> <span class="k">if</span> <span class="n">o</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">)})</span> <span class="n">currency</span> <span class="o">=</span> <span class="nf">sorted</span><span class="p">({</span><span class="nf">str</span><span class="p">(</span><span class="n">o</span><span class="p">[</span><span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">])</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="n">offers</span> <span class="k">if</span> <span class="n">o</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">)})</span> <span class="n">count</span> <span class="o">=</span> <span class="n">offers</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">offerCount</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="nf">len</span><span class="p">(</span><span class="n">offers</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">else</span> <span class="bp">None</span> <span class="n">sku</span> <span class="o">=</span> <span class="n">product</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">sku</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="p">{</span> <span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">:</span> <span class="nf">str</span><span class="p">(</span><span class="n">sku</span><span class="p">)</span> <span class="k">if</span> <span class="n">sku</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="k">else</span> <span class="n">page_url</span><span class="p">,</span> <span class="sh">"</span><span class="s">page</span><span class="sh">"</span><span class="p">:</span> <span class="n">page_url</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="nf">str</span><span class="p">(</span><span class="n">product</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">""</span><span class="p">).</span><span class="nf">strip</span><span class="p">()</span> <span class="ow">or</span> <span class="bp">None</span><span class="p">,</span> <span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">/</span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">currency</span><span class="p">)</span> <span class="ow">or</span> <span class="bp">None</span><span class="p">,</span> <span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">:</span> <span class="nf">str</span><span class="p">(</span><span class="nf">min</span><span class="p">(</span><span class="n">lows</span><span class="p">))</span> <span class="k">if</span> <span class="n">lows</span> <span class="k">else</span> <span class="bp">None</span><span class="p">,</span> <span class="sh">"</span><span class="s">high_price</span><span class="sh">"</span><span class="p">:</span> <span class="nf">str</span><span class="p">(</span><span class="nf">max</span><span class="p">(</span><span class="n">highs</span><span class="p">))</span> <span class="k">if</span> <span class="n">highs</span> <span class="k">else</span> <span class="bp">None</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="n">stock</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">if</span> <span class="nf">len</span><span class="p">(</span><span class="n">stock</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="nf">else </span><span class="p">(</span><span class="sh">"</span><span class="s">mixed</span><span class="sh">"</span> <span class="k">if</span> <span class="n">stock</span> <span class="k">else</span> <span class="bp">None</span><span class="p">),</span> <span class="sh">"</span><span class="s">offer_count</span><span class="sh">"</span><span class="p">:</span> <span class="nf">int</span><span class="p">(</span><span class="n">count</span><span class="p">)</span> <span class="k">if</span> <span class="nf">str</span><span class="p">(</span><span class="n">count</span> <span class="ow">or</span> <span class="sh">""</span><span class="p">).</span><span class="nf">isdigit</span><span class="p">()</span> <span class="k">else</span> <span class="nf">len</span><span class="p">(</span><span class="n">offers</span><span class="p">),</span> <span class="p">}</span> </code></pre> </div> <p>A few decisions in <code>to_row()</code> matter later:</p> <ul> <li> <strong>The key</strong> is the <code>sku</code> when the page has one, otherwise the URL you asked for. Not the product name and not <code>offers.url</code>; the real run below shows why.</li> <li> <strong>Several offers</strong> are reduced to the lowest and highest price. <code>availability</code> becomes <code>mixed</code> when the offers disagree.</li> <li> <strong>Several currencies</strong> are joined (<code>EUR/USD</code>). The script still takes a min and max, which means nothing across currencies, so treat a joined currency as a warning.</li> </ul> <h2> Step 4: extract, and report problems instead of hiding them </h2> <p>A broken block is reported and skipped; the other blocks on the page are still read. A page with no JSON-LD at all is reported too, because "no data" and "no products" are different situations.</p> <p><em>Excerpt of <code>catalog_snapshot.py</code>, lines 107-127:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">extract</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">page_url</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Return (rows, problems) for one HTML page.</span><span class="sh">"""</span> <span class="n">parser</span> <span class="o">=</span> <span class="nc">JsonLdScripts</span><span class="p">()</span> <span class="n">parser</span><span class="p">.</span><span class="nf">feed</span><span class="p">(</span><span class="n">html</span><span class="p">)</span> <span class="n">parser</span><span class="p">.</span><span class="nf">close</span><span class="p">()</span> <span class="n">rows</span><span class="p">,</span> <span class="n">problems</span> <span class="o">=</span> <span class="p">{},</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">number</span><span class="p">,</span> <span class="n">block</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="n">parser</span><span class="p">.</span><span class="n">blocks</span><span class="p">,</span> <span class="mi">1</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="nf">loads</span><span class="p">(</span><span class="n">block</span><span class="p">)</span> <span class="k">except</span> <span class="n">json</span><span class="p">.</span><span class="n">JSONDecodeError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="n">problems</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">JSON-LD block </span><span class="si">{</span><span class="n">number</span><span class="si">}</span><span class="s"> is not valid JSON (</span><span class="si">{</span><span class="n">err</span><span class="p">.</span><span class="n">msg</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span><span class="p">)</span> <span class="k">continue</span> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="nf">nodes</span><span class="p">(</span><span class="n">data</span><span class="p">):</span> <span class="k">if</span> <span class="nf">is_type</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="sh">"</span><span class="s">Product</span><span class="sh">"</span><span class="p">):</span> <span class="n">row</span> <span class="o">=</span> <span class="nf">to_row</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">page_url</span><span class="p">)</span> <span class="n">rows</span><span class="p">.</span><span class="nf">setdefault</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">parser</span><span class="p">.</span><span class="n">blocks</span><span class="p">:</span> <span class="n">problems</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">"</span><span class="s">no JSON-LD on the page</span><span class="sh">"</span><span class="p">)</span> <span class="k">elif</span> <span class="ow">not</span> <span class="n">rows</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">problems</span><span class="p">:</span> <span class="n">problems</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">"</span><span class="s">JSON-LD present but no Product</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="nf">list</span><span class="p">(</span><span class="n">rows</span><span class="p">.</span><span class="nf">values</span><span class="p">()),</span> <span class="n">problems</span> </code></pre> </div> <h2> Step 5: fetch politely </h2> <p><code>robots.txt</code> is fetched once per host, with the same <code>User-Agent</code> as the pages. A 404 for <code>robots.txt</code> means nothing is disallowed. The function returns the host's <code>Crawl-delay</code> so the caller can wait at least that long. <code>robots_rules()</code> drops blank lines before parsing: <code>urllib.robotparser</code> ends a group at a blank line, while <a href="https://www.rfc-editor.org/rfc/rfc9309" rel="noopener noreferrer">RFC 9309</a> does not, so without this step any rules after a blank line inside the <code>User-agent: *</code> group would be ignored.</p> <p><em>Excerpt of <code>catalog_snapshot.py</code>, lines 130-158:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">url</span><span class="p">):</span> <span class="n">request</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nc">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="sh">"</span><span class="s">User-Agent</span><span class="sh">"</span><span class="p">:</span> <span class="n">USER_AGENT</span><span class="p">})</span> <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nf">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">30</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span> <span class="n">charset</span> <span class="o">=</span> <span class="n">response</span><span class="p">.</span><span class="n">headers</span><span class="p">.</span><span class="nf">get_content_charset</span><span class="p">()</span> <span class="ow">or</span> <span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span> <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="nf">read</span><span class="p">().</span><span class="nf">decode</span><span class="p">(</span><span class="n">charset</span><span class="p">,</span> <span class="sh">"</span><span class="s">replace</span><span class="sh">"</span><span class="p">),</span> <span class="n">response</span><span class="p">.</span><span class="nf">geturl</span><span class="p">()</span> <span class="k">def</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Parse robots.txt. urllib.robotparser ends a group at a blank line; RFC 9309 does not.</span><span class="sh">"""</span> <span class="n">rules</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">robotparser</span><span class="p">.</span><span class="nc">RobotFileParser</span><span class="p">()</span> <span class="n">rules</span><span class="p">.</span><span class="nf">parse</span><span class="p">([</span><span class="n">line</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">text</span><span class="p">.</span><span class="nf">splitlines</span><span class="p">()</span> <span class="k">if</span> <span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()])</span> <span class="k">return</span> <span class="n">rules</span> <span class="k">def</span> <span class="nf">allowed</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">cache</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Check robots.txt (fetched once per host, with our User-Agent). Returns the crawl delay.</span><span class="sh">"""</span> <span class="n">parts</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">parse</span><span class="p">.</span><span class="nf">urlsplit</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="n">base</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">parts</span><span class="p">.</span><span class="n">scheme</span><span class="si">}</span><span class="s">://</span><span class="si">{</span><span class="n">parts</span><span class="p">.</span><span class="n">netloc</span><span class="si">}</span><span class="sh">"</span> <span class="k">if</span> <span class="n">base</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">cache</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="n">text</span> <span class="o">=</span> <span class="nf">get</span><span class="p">(</span><span class="n">base</span> <span class="o">+</span> <span class="sh">"</span><span class="s">/robots.txt</span><span class="sh">"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="k">except</span> <span class="n">urllib</span><span class="p">.</span><span class="n">error</span><span class="p">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="k">if</span> <span class="n">err</span><span class="p">.</span><span class="n">code</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">404</span><span class="p">,</span> <span class="mi">410</span><span class="p">):</span> <span class="k">raise</span> <span class="n">text</span> <span class="o">=</span> <span class="sh">""</span> <span class="c1"># no robots.txt: nothing is disallowed </span> <span class="n">cache</span><span class="p">[</span><span class="n">base</span><span class="p">]</span> <span class="o">=</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">cache</span><span class="p">[</span><span class="n">base</span><span class="p">].</span><span class="nf">can_fetch</span><span class="p">(</span><span class="n">USER_AGENT</span><span class="p">,</span> <span class="n">url</span><span class="p">):</span> <span class="k">raise</span> <span class="nc">PermissionError</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">robots.txt disallows </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="n">cache</span><span class="p">[</span><span class="n">base</span><span class="p">].</span><span class="nf">crawl_delay</span><span class="p">(</span><span class="n">USER_AGENT</span><span class="p">)</span> <span class="ow">or</span> <span class="mi">0</span> </code></pre> </div> <p>The snapshot loop fetches each URL once, waits at least two seconds before every request, and records errors per page instead of stopping. With <code>--wayback</code>, it asks the Wayback Machine for the capture of each URL nearest to the given date. The <code>id_</code> after the timestamp returned the page as it was captured, without the archive's banner, and its JSON-LD parsed exactly like the live page.</p> <p><em>Excerpt of <code>catalog_snapshot.py</code>, lines 161-180:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">snapshot</span><span class="p">(</span><span class="n">url_file</span><span class="p">,</span> <span class="n">out_file</span><span class="p">,</span> <span class="n">wayback</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">url_file</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">urls</span> <span class="o">=</span> <span class="p">[</span><span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">handle</span> <span class="k">if</span> <span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">line</span><span class="p">.</span><span class="nf">startswith</span><span class="p">(</span><span class="sh">"</span><span class="s">#</span><span class="sh">"</span><span class="p">)]</span> <span class="n">robots</span><span class="p">,</span> <span class="n">rows</span><span class="p">,</span> <span class="n">pages</span> <span class="o">=</span> <span class="p">{},</span> <span class="p">[],</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="n">urls</span><span class="p">:</span> <span class="n">target</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="s">https://web.archive.org/web/</span><span class="si">{</span><span class="n">wayback</span><span class="si">}</span><span class="s">id_/</span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="sh">"</span> <span class="k">if</span> <span class="n">wayback</span> <span class="k">else</span> <span class="n">url</span> <span class="n">page</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">url</span><span class="sh">"</span><span class="p">:</span> <span class="n">url</span><span class="p">}</span> <span class="k">try</span><span class="p">:</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="nf">max</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="nf">allowed</span><span class="p">(</span><span class="n">target</span><span class="p">,</span> <span class="n">robots</span><span class="p">)))</span> <span class="n">html</span><span class="p">,</span> <span class="n">page</span><span class="p">[</span><span class="sh">"</span><span class="s">fetched</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="nf">get</span><span class="p">(</span><span class="n">target</span><span class="p">)</span> <span class="n">found</span><span class="p">,</span> <span class="n">page</span><span class="p">[</span><span class="sh">"</span><span class="s">problems</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="nf">extract</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span> <span class="n">rows</span> <span class="o">+=</span> <span class="n">found</span> <span class="k">except</span> <span class="nb">OSError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="c1"># HTTP errors, timeouts, robots refusals (PermissionError) </span> <span class="n">page</span><span class="p">[</span><span class="sh">"</span><span class="s">error</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="nf">str</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="n">pages</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">page</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s">: </span><span class="si">{</span><span class="n">page</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">'</span><span class="s">error</span><span class="sh">'</span><span class="p">)</span> <span class="ow">or</span> <span class="sa">f</span><span class="sh">'</span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">found</span><span class="p">)</span><span class="si">}</span> <span class="nf">product</span><span class="p">(</span><span class="n">s</span><span class="p">)</span><span class="sh">'</span><span class="si">}</span><span class="s"> </span><span class="si">{</span><span class="n">page</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">'</span><span class="s">problems</span><span class="sh">'</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">''</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="n">captured</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="nf">now</span><span class="p">(</span><span class="n">timezone</span><span class="p">.</span><span class="n">utc</span><span class="p">).</span><span class="nf">strftime</span><span class="p">(</span><span class="sh">"</span><span class="s">%Y-%m-%dT%H:%M:%SZ</span><span class="sh">"</span><span class="p">)</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">out_file</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="nf">dump</span><span class="p">({</span><span class="sh">"</span><span class="s">captured_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">captured</span><span class="p">,</span> <span class="sh">"</span><span class="s">pages</span><span class="sh">"</span><span class="p">:</span> <span class="n">pages</span><span class="p">,</span> <span class="sh">"</span><span class="s">rows</span><span class="sh">"</span><span class="p">:</span> <span class="n">rows</span><span class="p">},</span> <span class="n">handle</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span><span class="si">}</span><span class="s"> product rows from </span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">pages</span><span class="p">)</span><span class="si">}</span><span class="s"> pages -&gt; </span><span class="si">{</span><span class="n">out_file</span><span class="si">}</span><span class="s"> (</span><span class="si">{</span><span class="n">captured</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span><span class="p">)</span> </code></pre> </div> <h2> Step 6: compare two snapshots </h2> <p>The diff is set arithmetic on the keys, plus a field-by-field comparison for keys present in both. One rule prevents the classic false alarm: if a page failed to load in the new snapshot, its products are reported as <code>unknown</code>, not <code>removed</code>. A timeout is not a discontinued product.</p> <p><em>Excerpt of <code>catalog_snapshot.py</code>, lines 183-196:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">diff</span><span class="p">(</span><span class="n">before</span><span class="p">,</span> <span class="n">after</span><span class="p">):</span> <span class="sh">"""</span><span class="s">List (change, key, field, old, new). A page that failed to load is </span><span class="sh">'</span><span class="s">unknown</span><span class="sh">'</span><span class="s">, not </span><span class="sh">'</span><span class="s">removed</span><span class="sh">'</span><span class="s">.</span><span class="sh">"""</span> <span class="n">old</span> <span class="o">=</span> <span class="p">{</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">]:</span> <span class="n">row</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">before</span><span class="p">[</span><span class="sh">"</span><span class="s">rows</span><span class="sh">"</span><span class="p">]}</span> <span class="n">new</span> <span class="o">=</span> <span class="p">{</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">]:</span> <span class="n">row</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">after</span><span class="p">[</span><span class="sh">"</span><span class="s">rows</span><span class="sh">"</span><span class="p">]}</span> <span class="n">failed</span> <span class="o">=</span> <span class="p">{</span><span class="n">page</span><span class="p">[</span><span class="sh">"</span><span class="s">url</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">page</span> <span class="ow">in</span> <span class="n">after</span><span class="p">[</span><span class="sh">"</span><span class="s">pages</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="n">page</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">error</span><span class="sh">"</span><span class="p">)}</span> <span class="n">changes</span> <span class="o">=</span> <span class="p">[(</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="n">new</span><span class="p">[</span><span class="n">key</span><span class="p">][</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">])</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">new</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">-</span> <span class="n">old</span><span class="p">.</span><span class="nf">keys</span><span class="p">())]</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">old</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">-</span> <span class="n">new</span><span class="p">.</span><span class="nf">keys</span><span class="p">()):</span> <span class="n">kind</span> <span class="o">=</span> <span class="sh">"</span><span class="s">unknown</span><span class="sh">"</span> <span class="k">if</span> <span class="n">old</span><span class="p">[</span><span class="n">key</span><span class="p">][</span><span class="sh">"</span><span class="s">page</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="n">failed</span> <span class="k">else</span> <span class="sh">"</span><span class="s">removed</span><span class="sh">"</span> <span class="n">changes</span><span class="p">.</span><span class="nf">append</span><span class="p">((</span><span class="n">kind</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="n">old</span><span class="p">[</span><span class="n">key</span><span class="p">][</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">],</span> <span class="sh">""</span><span class="p">))</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">old</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">&amp;</span> <span class="n">new</span><span class="p">.</span><span class="nf">keys</span><span class="p">()):</span> <span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">WATCHED</span><span class="p">:</span> <span class="k">if</span> <span class="n">old</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="n">field</span><span class="p">)</span> <span class="o">!=</span> <span class="n">new</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="n">field</span><span class="p">):</span> <span class="n">changes</span><span class="p">.</span><span class="nf">append</span><span class="p">((</span><span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">field</span><span class="p">,</span> <span class="n">old</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="n">field</span><span class="p">),</span> <span class="n">new</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="n">field</span><span class="p">)))</span> <span class="k">return</span> <span class="n">changes</span> </code></pre> </div> <h2> Tests </h2> <p>Eleven <code>unittest</code> tests cover the branches: <code>Offer</code> versus <code>AggregateOffer</code> versus a list of offers, <code>priceSpecification</code>, <code>@graph</code>, list and compact types, a broken block next to a good one, pages without JSON-LD, the diff rules, and a blank line inside a <code>robots.txt</code> group. For example:</p> <p><em>Excerpt of <code>test_catalog_snapshot.py</code>, lines 37-48:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code> <span class="k">def</span> <span class="nf">test_graph_type_list_sku_and_offer_list</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">rows</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="nf">extract</span><span class="p">(</span><span class="nf">page</span><span class="p">({</span><span class="sh">"</span><span class="s">@context</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://schema.org</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">@graph</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">WebPage</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">ignored</span><span class="sh">"</span><span class="p">},</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">Product</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Thing</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">sku</span><span class="sh">"</span><span class="p">:</span> <span class="mi">42</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Mug</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">offers</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Offer</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">price</span><span class="sh">"</span><span class="p">:</span> <span class="mi">12</span><span class="p">,</span> <span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">EUR</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">http://schema.org/InStock</span><span class="sh">"</span><span class="p">},</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Offer</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">priceSpecification</span><span class="sh">"</span><span class="p">:</span> <span class="p">{</span><span class="sh">"</span><span class="s">price</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">10.50</span><span class="sh">"</span><span class="p">},</span> <span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">EUR</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">OutOfStock</span><span class="sh">"</span><span class="p">}]}]}),</span> <span class="sh">"</span><span class="s">https://x/p/2</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">rows</span><span class="p">),</span> <span class="mi">1</span><span class="p">)</span> <span class="n">row</span> <span class="o">=</span> <span class="n">rows</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">high_price</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="sh">"</span><span class="s">42</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">10.5</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">12</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">offer_count</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="sh">"</span><span class="s">mixed</span><span class="sh">"</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="sh">"</span><span class="s">EUR</span><span class="sh">"</span><span class="p">))</span> </code></pre> </div> <p><em>Excerpt of <code>test_catalog_snapshot.py</code>, lines 94-95:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code> <span class="k">def</span> <span class="nf">test_failed_page_is_unknown_not_removed</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">diff</span><span class="p">(</span><span class="nf">snap</span><span class="p">([</span><span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">)]),</span> <span class="nf">snap</span><span class="p">([],</span> <span class="n">failed</span><span class="o">=</span><span class="p">[</span><span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">])),</span> <span class="p">[(</span><span class="sh">"</span><span class="s">unknown</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="sh">"</span><span class="s">A</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)])</span> </code></pre> </div> <p><em>Excerpt of <code>test-log.txt</code>, lines 13-16:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>---------------------------------------------------------------------- Ran 11 tests in 0.001s OK </code></pre> </div> <h2> A real run </h2> <p>The input file lists six product pages on the sandbox. These six were chosen because the Wayback Machine holds 2026 captures of them.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code># Product pages on web-scraping.dev, a public sandbox built for scraping practice https://web-scraping.dev/product/1 https://web-scraping.dev/product/2 https://web-scraping.dev/product/10 https://web-scraping.dev/product/11 https://web-scraping.dev/product/12 https://web-scraping.dev/product/13 </code></pre> </div> <p>First the "before" snapshot, from the archive. The date asks for captures near 1 May 2026:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python catalog_snapshot.py snapshot urls.txt before.json --wayback 20260501 https://web-scraping.dev/product/1: 1 product(s) https://web-scraping.dev/product/2: 1 product(s) https://web-scraping.dev/product/10: 1 product(s) https://web-scraping.dev/product/11: 1 product(s) https://web-scraping.dev/product/12: 1 product(s) https://web-scraping.dev/product/13: 1 product(s) 6 product rows from 6 pages -&gt; before.json (2026-09-27T13:00:09Z) </code></pre> </div> <p>Then the live pages:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python catalog_snapshot.py snapshot urls.txt after.json https://web-scraping.dev/product/1: 1 product(s) https://web-scraping.dev/product/2: 1 product(s) https://web-scraping.dev/product/10: 1 product(s) https://web-scraping.dev/product/11: 1 product(s) https://web-scraping.dev/product/12: 1 product(s) https://web-scraping.dev/product/13: 1 product(s) 6 product rows from 6 pages -&gt; after.json (2026-09-27T13:00:37Z) </code></pre> </div> <p>The rows in the live snapshot, next to the date of the archive capture used for each page:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Page</th> <th>name</th> <th>low</th> <th>high</th> <th>currency</th> <th>availability</th> <th>offers</th> <th>archive capture</th> </tr> </thead> <tbody> <tr> <td><code>/product/1</code></td> <td>Box of Chocolate Candy</td> <td>9.99</td> <td>19.99</td> <td>USD</td> <td>InStock</td> <td>6</td> <td>2026-03-28</td> </tr> <tr> <td><code>/product/2</code></td> <td>Dark Red Energy Potion</td> <td>4.99</td> <td>25.99</td> <td>USD</td> <td>InStock</td> <td>2</td> <td>2026-04-12</td> </tr> <tr> <td><code>/product/10</code></td> <td>Kids' Light-Up Sneakers</td> <td>29.99</td> <td>29.99</td> <td>USD</td> <td>InStock</td> <td>4</td> <td>2026-05-17</td> </tr> <tr> <td><code>/product/11</code></td> <td>Classic Leather Sneakers</td> <td>79.99</td> <td>79.99</td> <td>USD</td> <td>InStock</td> <td>6</td> <td>2026-06-13</td> </tr> <tr> <td><code>/product/12</code></td> <td>Cat-Ear Beanie</td> <td>14.99</td> <td>14.99</td> <td>USD</td> <td>InStock</td> <td>8</td> <td>2026-05-17</td> </tr> <tr> <td><code>/product/13</code></td> <td>Box of Chocolate Candy</td> <td>9.99</td> <td>19.99</td> <td>USD</td> <td>InStock</td> <td>6</td> <td>2026-05-17</td> </tr> </tbody> </table></div> <p>And the comparison:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python catalog_snapshot.py diff before.json after.json changes.csv before 2026-09-27T13:00:09Z (6 rows), after 2026-09-27T13:00:37Z (6 rows): 0 change(s) </code></pre> </div> <p>Zero changes. For a sandbox that is the expected answer, and it is still a useful result: six captures from between March and June 2026 and six live pages, parsed independently, produced identical rows. It also shows that rows built from two different sources, the archive and the live site, are directly comparable.</p> <p>What the run shows that matters for a real catalog:</p> <ul> <li> <strong>Names are not identities.</strong> <code>/product/1</code> and <code>/product/13</code> have the same name, the same price range and the same offer count. Keyed by name, they would collapse into one row.</li> <li> <strong>None of the six pages had a <code>sku</code></strong>, so the page URL is the key.</li> <li> <strong><code>offers.url</code> is not a safe key either.</strong> On an archived variant page, <code>/product/13?variant=cherry-large</code> (captured on 12 February 2026), <code>offers.url</code> carried the <code>?variant=cherry-large</code> query.</li> <li> <strong>The archive gives the nearest capture, not the date you ask for.</strong> Asking for 1 May 2026 returned captures from 28 March to 13 June. The capture of <code>/product/2</code> is stored under <code>www.web-scraping.dev</code>. Keying rows by the URL you asked for, not the URL that was served, keeps them comparable. The <code>captured_at</code> field is when the script ran; each capture's date is in the <code>fetched</code> URL of <code>before.json</code>.</li> <li> <strong>An <code>AggregateOffer</code> hides variants.</strong> <code>/product/1</code> has six offers, reduced to 9.99 to 19.99. If a middle variant's price changed, this row would not change. For per-variant tracking, read the variant pages or an official feed.</li> </ul> <h2> Limits and common mistakes </h2> <ul> <li> <strong>Without a <code>sku</code>, a page gives one row.</strong> Every <code>Product</code> without a <code>sku</code> is keyed by the page URL, and <code>extract()</code> keeps only the first one. That suits single-product pages like the six above, but on a category page, or a page whose JSON-LD lists several products, the other products are dropped without a warning. Products nested inside another product, such as related items, are found too, and each one that has a <code>sku</code> becomes its own row. For such pages, key rows on <code>sku</code>, <code>gtin</code> or <code>@id</code> plus the page URL, or report a page with more than one product as a problem.</li> <li> <strong>JavaScript-injected JSON-LD is invisible here.</strong> The script reads the HTML the server sends. If a site adds its structured data in the browser, this approach returns "no JSON-LD on the page".</li> <li> <strong>The markup can disagree with the page.</strong> Google's structured data guidelines tell publishers not to mark up content that is not visible to readers, but that is a rule for publishers, not a guarantee. Spot-check a few pages against what a person sees.</li> <li> <strong>Comparing floats.</strong> <code>0.1 + 0.2 != 0.3</code>. Keep prices as decimal text.</li> <li> <strong>Treating a failed download as a removal.</strong> The <code>unknown</code> rule above exists for this.</li> <li> <strong>Forgetting the currency.</strong> A price without its currency is not comparable.</li> <li> <strong>Fetching too fast or too much.</strong> One request per page, a fixed delay, a descriptive <code>User-Agent</code>, and <code>robots.txt</code> respected. If a site refuses access, stop; do not work around it.</li> </ul> <h2> Official sources </h2> <ul> <li>schema.org <code>Product</code>: <a href="https://schema.org/Product" rel="noopener noreferrer">https://schema.org/Product</a> </li> <li>schema.org <code>Offer</code>: <a href="https://schema.org/Offer" rel="noopener noreferrer">https://schema.org/Offer</a> and <code>AggregateOffer</code>: <a href="https://schema.org/AggregateOffer" rel="noopener noreferrer">https://schema.org/AggregateOffer</a> </li> <li>schema.org <code>price</code> usage guidelines: <a href="https://schema.org/price" rel="noopener noreferrer">https://schema.org/price</a> </li> <li>Google, product snippet structured data: <a href="https://developers.google.com/search/docs/appearance/structured-data/product-snippet" rel="noopener noreferrer">https://developers.google.com/search/docs/appearance/structured-data/product-snippet</a> </li> <li>Google, general structured data guidelines: <a href="https://developers.google.com/search/docs/appearance/structured-data/sd-policies" rel="noopener noreferrer">https://developers.google.com/search/docs/appearance/structured-data/sd-policies</a> </li> <li>JSON-LD 1.1 (W3C): <a href="https://www.w3.org/TR/json-ld11/" rel="noopener noreferrer">https://www.w3.org/TR/json-ld11/</a> </li> <li>Wayback Machine availability API: <a href="https://archive.org/help/wayback_api.php" rel="noopener noreferrer">https://archive.org/help/wayback_api.php</a> </li> <li>Wayback CDX server (listing captures): <a href="https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md" rel="noopener noreferrer">https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md</a> </li> <li>Python <code>html.parser</code>: <a href="https://docs.python.org/3/library/html.parser.html" rel="noopener noreferrer">https://docs.python.org/3/library/html.parser.html</a> </li> <li>Python <code>urllib.robotparser</code>: <a href="https://docs.python.org/3/library/urllib.robotparser.html" rel="noopener noreferrer">https://docs.python.org/3/library/urllib.robotparser.html</a> </li> <li>Robots Exclusion Protocol, RFC 9309: <a href="https://www.rfc-editor.org/rfc/rfc9309" rel="noopener noreferrer">https://www.rfc-editor.org/rfc/rfc9309</a> </li> <li>Python <code>decimal</code>: <a href="https://docs.python.org/3/library/decimal.html" rel="noopener noreferrer">https://docs.python.org/3/library/decimal.html</a> </li> <li>The sandbox used here: <a href="https://web-scraping.dev/" rel="noopener noreferrer">https://web-scraping.dev/</a> </li> </ul> <h2> The complete script and tests </h2> <p></p> catalog_snapshot.py (219 lines) <br> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="sh">"""</span><span class="s">Snapshot schema.org Product data from JSON-LD, then diff two snapshots. Usage: python catalog_snapshot.py snapshot urls.txt out.json [--wayback YYYYMMDD] python catalog_snapshot.py diff before.json after.json changes.csv </span><span class="sh">"""</span> <span class="kn">import</span> <span class="n">csv</span> <span class="kn">import</span> <span class="n">json</span> <span class="kn">import</span> <span class="n">re</span> <span class="kn">import</span> <span class="n">sys</span> <span class="kn">import</span> <span class="n">time</span> <span class="kn">import</span> <span class="n">urllib.error</span> <span class="kn">import</span> <span class="n">urllib.parse</span> <span class="kn">import</span> <span class="n">urllib.request</span> <span class="kn">import</span> <span class="n">urllib.robotparser</span> <span class="kn">from</span> <span class="n">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timezone</span> <span class="kn">from</span> <span class="n">decimal</span> <span class="kn">import</span> <span class="n">Decimal</span><span class="p">,</span> <span class="n">InvalidOperation</span> <span class="kn">from</span> <span class="n">html.parser</span> <span class="kn">import</span> <span class="n">HTMLParser</span> <span class="n">USER_AGENT</span> <span class="o">=</span> <span class="sh">"</span><span class="s">jsonld-snapshot-example/1.0 (tutorial script; one request per page)</span><span class="sh">"</span> <span class="n">WATCHED</span> <span class="o">=</span> <span class="p">(</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">high_price</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">offer_count</span><span class="sh">"</span><span class="p">)</span> <span class="k">class</span> <span class="nc">JsonLdScripts</span><span class="p">(</span><span class="n">HTMLParser</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Collect the raw text of every &lt;script type=</span><span class="sh">"</span><span class="s">application/ld+json</span><span class="sh">"</span><span class="s">&gt; element.</span><span class="sh">"""</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="nf">super</span><span class="p">().</span><span class="nf">__init__</span><span class="p">()</span> <span class="n">self</span><span class="p">.</span><span class="n">blocks</span><span class="p">,</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="o">=</span> <span class="p">[],</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">handle_starttag</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">tag</span><span class="p">,</span> <span class="n">attrs</span><span class="p">):</span> <span class="n">kind</span> <span class="o">=</span> <span class="p">(</span><span class="nf">dict</span><span class="p">(</span><span class="n">attrs</span><span class="p">).</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">type</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">""</span><span class="p">).</span><span class="nf">split</span><span class="p">(</span><span class="sh">"</span><span class="s">;</span><span class="sh">"</span><span class="p">)[</span><span class="mi">0</span><span class="p">].</span><span class="nf">strip</span><span class="p">().</span><span class="nf">lower</span><span class="p">()</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">script</span><span class="sh">"</span> <span class="ow">and</span> <span class="n">kind</span> <span class="o">==</span> <span class="sh">"</span><span class="s">application/ld+json</span><span class="sh">"</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">def</span> <span class="nf">handle_data</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span> <span class="k">if</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="k">def</span> <span class="nf">handle_endtag</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">tag</span><span class="p">):</span> <span class="k">if</span> <span class="n">tag</span> <span class="o">==</span> <span class="sh">"</span><span class="s">script</span><span class="sh">"</span> <span class="ow">and</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">blocks</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">_parts</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="n">_parts</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">nodes</span><span class="p">(</span><span class="n">value</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Yield every JSON object in a JSON-LD value: top-level arrays, @graph, nesting.</span><span class="sh">"""</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">value</span><span class="p">:</span> <span class="k">yield</span> <span class="k">from</span> <span class="nf">nodes</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="k">elif</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span> <span class="k">yield</span> <span class="n">value</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">value</span><span class="p">.</span><span class="nf">values</span><span class="p">():</span> <span class="k">yield</span> <span class="k">from</span> <span class="nf">nodes</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="k">def</span> <span class="nf">short</span><span class="p">(</span><span class="n">term</span><span class="p">):</span> <span class="sh">"""'</span><span class="s">https://schema.org/InStock</span><span class="sh">'</span><span class="s"> -&gt; </span><span class="sh">'</span><span class="s">InStock</span><span class="sh">'</span><span class="s">, </span><span class="sh">'</span><span class="s">schema:Product</span><span class="sh">'</span><span class="s"> -&gt; </span><span class="sh">'</span><span class="s">Product</span><span class="sh">'</span><span class="s">.</span><span class="sh">"""</span> <span class="k">return</span> <span class="n">re</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="sa">r</span><span class="sh">"</span><span class="s">[/:#]</span><span class="sh">"</span><span class="p">,</span> <span class="n">term</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">term</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">else</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">is_type</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">wanted</span><span class="p">):</span> <span class="n">types</span> <span class="o">=</span> <span class="n">node</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="n">wanted</span> <span class="ow">in</span> <span class="p">[</span><span class="nf">short</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="p">(</span><span class="n">types</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">types</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">types</span><span class="p">])]</span> <span class="k">def</span> <span class="nf">clean_price</span><span class="p">(</span><span class="n">value</span><span class="p">):</span> <span class="sh">"""'</span><span class="s">19.990</span><span class="sh">'</span><span class="s"> -&gt; </span><span class="sh">'</span><span class="s">19.99</span><span class="sh">'</span><span class="s">, 20 -&gt; </span><span class="sh">'</span><span class="s">20</span><span class="sh">'</span><span class="s">, </span><span class="sh">''</span><span class="s"> or </span><span class="sh">'</span><span class="s">1,299.00</span><span class="sh">'</span><span class="s"> -&gt; None. Text, not float.</span><span class="sh">"""</span> <span class="k">if</span> <span class="n">value</span> <span class="ow">is</span> <span class="bp">None</span> <span class="ow">or</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">bool</span><span class="p">):</span> <span class="k">return</span> <span class="bp">None</span> <span class="k">try</span><span class="p">:</span> <span class="n">number</span> <span class="o">=</span> <span class="nc">Decimal</span><span class="p">(</span><span class="nf">str</span><span class="p">(</span><span class="n">value</span><span class="p">).</span><span class="nf">strip</span><span class="p">())</span> <span class="k">except</span> <span class="n">InvalidOperation</span><span class="p">:</span> <span class="k">return</span> <span class="bp">None</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">number</span><span class="p">.</span><span class="nf">is_finite</span><span class="p">():</span> <span class="k">return</span> <span class="bp">None</span> <span class="n">text</span> <span class="o">=</span> <span class="nf">format</span><span class="p">(</span><span class="n">number</span><span class="p">,</span> <span class="sh">"</span><span class="s">f</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="n">text</span><span class="p">.</span><span class="nf">rstrip</span><span class="p">(</span><span class="sh">"</span><span class="s">0</span><span class="sh">"</span><span class="p">).</span><span class="nf">rstrip</span><span class="p">(</span><span class="sh">"</span><span class="s">.</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="sh">"</span><span class="s">.</span><span class="sh">"</span> <span class="ow">in</span> <span class="n">text</span> <span class="k">else</span> <span class="n">text</span> <span class="k">def</span> <span class="nf">to_row</span><span class="p">(</span><span class="n">product</span><span class="p">,</span> <span class="n">page_url</span><span class="p">):</span> <span class="n">offers</span> <span class="o">=</span> <span class="n">product</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">offers</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]</span> <span class="n">offers</span> <span class="o">=</span> <span class="p">[</span><span class="n">o</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="p">(</span><span class="n">offers</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">offers</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">offers</span><span class="p">])</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">o</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)]</span> <span class="n">lows</span><span class="p">,</span> <span class="n">highs</span> <span class="o">=</span> <span class="p">[],</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">offer</span> <span class="ow">in</span> <span class="n">offers</span><span class="p">:</span> <span class="n">spec</span> <span class="o">=</span> <span class="n">offer</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">priceSpecification</span><span class="sh">"</span><span class="p">)</span> <span class="n">price</span> <span class="o">=</span> <span class="n">offer</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">price</span><span class="sh">"</span><span class="p">,</span> <span class="n">spec</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">price</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">spec</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="k">else</span> <span class="bp">None</span><span class="p">)</span> <span class="n">low</span><span class="p">,</span> <span class="n">high</span> <span class="o">=</span> <span class="nf">clean_price</span><span class="p">(</span><span class="n">offer</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">lowPrice</span><span class="sh">"</span><span class="p">,</span> <span class="n">price</span><span class="p">)),</span> <span class="nf">clean_price</span><span class="p">(</span><span class="n">offer</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">highPrice</span><span class="sh">"</span><span class="p">,</span> <span class="n">price</span><span class="p">))</span> <span class="n">lows</span> <span class="o">+=</span> <span class="p">[</span><span class="nc">Decimal</span><span class="p">(</span><span class="n">low</span><span class="p">)]</span> <span class="k">if</span> <span class="n">low</span> <span class="k">else</span> <span class="p">[]</span> <span class="n">highs</span> <span class="o">+=</span> <span class="p">[</span><span class="nc">Decimal</span><span class="p">(</span><span class="n">high</span><span class="p">)]</span> <span class="k">if</span> <span class="n">high</span> <span class="k">else</span> <span class="p">[]</span> <span class="n">stock</span> <span class="o">=</span> <span class="nf">sorted</span><span class="p">({</span><span class="nf">short</span><span class="p">(</span><span class="n">o</span><span class="p">[</span><span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">])</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="n">offers</span> <span class="k">if</span> <span class="n">o</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">)})</span> <span class="n">currency</span> <span class="o">=</span> <span class="nf">sorted</span><span class="p">({</span><span class="nf">str</span><span class="p">(</span><span class="n">o</span><span class="p">[</span><span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">])</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="n">offers</span> <span class="k">if</span> <span class="n">o</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">)})</span> <span class="n">count</span> <span class="o">=</span> <span class="n">offers</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">offerCount</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="nf">len</span><span class="p">(</span><span class="n">offers</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">else</span> <span class="bp">None</span> <span class="n">sku</span> <span class="o">=</span> <span class="n">product</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">sku</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="p">{</span> <span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">:</span> <span class="nf">str</span><span class="p">(</span><span class="n">sku</span><span class="p">)</span> <span class="k">if</span> <span class="n">sku</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="k">else</span> <span class="n">page_url</span><span class="p">,</span> <span class="sh">"</span><span class="s">page</span><span class="sh">"</span><span class="p">:</span> <span class="n">page_url</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="nf">str</span><span class="p">(</span><span class="n">product</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">""</span><span class="p">).</span><span class="nf">strip</span><span class="p">()</span> <span class="ow">or</span> <span class="bp">None</span><span class="p">,</span> <span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">/</span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">currency</span><span class="p">)</span> <span class="ow">or</span> <span class="bp">None</span><span class="p">,</span> <span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">:</span> <span class="nf">str</span><span class="p">(</span><span class="nf">min</span><span class="p">(</span><span class="n">lows</span><span class="p">))</span> <span class="k">if</span> <span class="n">lows</span> <span class="k">else</span> <span class="bp">None</span><span class="p">,</span> <span class="sh">"</span><span class="s">high_price</span><span class="sh">"</span><span class="p">:</span> <span class="nf">str</span><span class="p">(</span><span class="nf">max</span><span class="p">(</span><span class="n">highs</span><span class="p">))</span> <span class="k">if</span> <span class="n">highs</span> <span class="k">else</span> <span class="bp">None</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="n">stock</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">if</span> <span class="nf">len</span><span class="p">(</span><span class="n">stock</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="nf">else </span><span class="p">(</span><span class="sh">"</span><span class="s">mixed</span><span class="sh">"</span> <span class="k">if</span> <span class="n">stock</span> <span class="k">else</span> <span class="bp">None</span><span class="p">),</span> <span class="sh">"</span><span class="s">offer_count</span><span class="sh">"</span><span class="p">:</span> <span class="nf">int</span><span class="p">(</span><span class="n">count</span><span class="p">)</span> <span class="k">if</span> <span class="nf">str</span><span class="p">(</span><span class="n">count</span> <span class="ow">or</span> <span class="sh">""</span><span class="p">).</span><span class="nf">isdigit</span><span class="p">()</span> <span class="k">else</span> <span class="nf">len</span><span class="p">(</span><span class="n">offers</span><span class="p">),</span> <span class="p">}</span> <span class="k">def</span> <span class="nf">extract</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">page_url</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Return (rows, problems) for one HTML page.</span><span class="sh">"""</span> <span class="n">parser</span> <span class="o">=</span> <span class="nc">JsonLdScripts</span><span class="p">()</span> <span class="n">parser</span><span class="p">.</span><span class="nf">feed</span><span class="p">(</span><span class="n">html</span><span class="p">)</span> <span class="n">parser</span><span class="p">.</span><span class="nf">close</span><span class="p">()</span> <span class="n">rows</span><span class="p">,</span> <span class="n">problems</span> <span class="o">=</span> <span class="p">{},</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">number</span><span class="p">,</span> <span class="n">block</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="n">parser</span><span class="p">.</span><span class="n">blocks</span><span class="p">,</span> <span class="mi">1</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="nf">loads</span><span class="p">(</span><span class="n">block</span><span class="p">)</span> <span class="k">except</span> <span class="n">json</span><span class="p">.</span><span class="n">JSONDecodeError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="n">problems</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">JSON-LD block </span><span class="si">{</span><span class="n">number</span><span class="si">}</span><span class="s"> is not valid JSON (</span><span class="si">{</span><span class="n">err</span><span class="p">.</span><span class="n">msg</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span><span class="p">)</span> <span class="k">continue</span> <span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="nf">nodes</span><span class="p">(</span><span class="n">data</span><span class="p">):</span> <span class="k">if</span> <span class="nf">is_type</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="sh">"</span><span class="s">Product</span><span class="sh">"</span><span class="p">):</span> <span class="n">row</span> <span class="o">=</span> <span class="nf">to_row</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">page_url</span><span class="p">)</span> <span class="n">rows</span><span class="p">.</span><span class="nf">setdefault</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">parser</span><span class="p">.</span><span class="n">blocks</span><span class="p">:</span> <span class="n">problems</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">"</span><span class="s">no JSON-LD on the page</span><span class="sh">"</span><span class="p">)</span> <span class="k">elif</span> <span class="ow">not</span> <span class="n">rows</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">problems</span><span class="p">:</span> <span class="n">problems</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="sh">"</span><span class="s">JSON-LD present but no Product</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="nf">list</span><span class="p">(</span><span class="n">rows</span><span class="p">.</span><span class="nf">values</span><span class="p">()),</span> <span class="n">problems</span> <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">url</span><span class="p">):</span> <span class="n">request</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nc">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="sh">"</span><span class="s">User-Agent</span><span class="sh">"</span><span class="p">:</span> <span class="n">USER_AGENT</span><span class="p">})</span> <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nf">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">30</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span> <span class="n">charset</span> <span class="o">=</span> <span class="n">response</span><span class="p">.</span><span class="n">headers</span><span class="p">.</span><span class="nf">get_content_charset</span><span class="p">()</span> <span class="ow">or</span> <span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span> <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="nf">read</span><span class="p">().</span><span class="nf">decode</span><span class="p">(</span><span class="n">charset</span><span class="p">,</span> <span class="sh">"</span><span class="s">replace</span><span class="sh">"</span><span class="p">),</span> <span class="n">response</span><span class="p">.</span><span class="nf">geturl</span><span class="p">()</span> <span class="k">def</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="n">text</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Parse robots.txt. urllib.robotparser ends a group at a blank line; RFC 9309 does not.</span><span class="sh">"""</span> <span class="n">rules</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">robotparser</span><span class="p">.</span><span class="nc">RobotFileParser</span><span class="p">()</span> <span class="n">rules</span><span class="p">.</span><span class="nf">parse</span><span class="p">([</span><span class="n">line</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">text</span><span class="p">.</span><span class="nf">splitlines</span><span class="p">()</span> <span class="k">if</span> <span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()])</span> <span class="k">return</span> <span class="n">rules</span> <span class="k">def</span> <span class="nf">allowed</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">cache</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Check robots.txt (fetched once per host, with our User-Agent). Returns the crawl delay.</span><span class="sh">"""</span> <span class="n">parts</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">parse</span><span class="p">.</span><span class="nf">urlsplit</span><span class="p">(</span><span class="n">url</span><span class="p">)</span> <span class="n">base</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">parts</span><span class="p">.</span><span class="n">scheme</span><span class="si">}</span><span class="s">://</span><span class="si">{</span><span class="n">parts</span><span class="p">.</span><span class="n">netloc</span><span class="si">}</span><span class="sh">"</span> <span class="k">if</span> <span class="n">base</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">cache</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="n">text</span> <span class="o">=</span> <span class="nf">get</span><span class="p">(</span><span class="n">base</span> <span class="o">+</span> <span class="sh">"</span><span class="s">/robots.txt</span><span class="sh">"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="k">except</span> <span class="n">urllib</span><span class="p">.</span><span class="n">error</span><span class="p">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="k">if</span> <span class="n">err</span><span class="p">.</span><span class="n">code</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">404</span><span class="p">,</span> <span class="mi">410</span><span class="p">):</span> <span class="k">raise</span> <span class="n">text</span> <span class="o">=</span> <span class="sh">""</span> <span class="c1"># no robots.txt: nothing is disallowed </span> <span class="n">cache</span><span class="p">[</span><span class="n">base</span><span class="p">]</span> <span class="o">=</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="n">text</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">cache</span><span class="p">[</span><span class="n">base</span><span class="p">].</span><span class="nf">can_fetch</span><span class="p">(</span><span class="n">USER_AGENT</span><span class="p">,</span> <span class="n">url</span><span class="p">):</span> <span class="k">raise</span> <span class="nc">PermissionError</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">robots.txt disallows </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="n">cache</span><span class="p">[</span><span class="n">base</span><span class="p">].</span><span class="nf">crawl_delay</span><span class="p">(</span><span class="n">USER_AGENT</span><span class="p">)</span> <span class="ow">or</span> <span class="mi">0</span> <span class="k">def</span> <span class="nf">snapshot</span><span class="p">(</span><span class="n">url_file</span><span class="p">,</span> <span class="n">out_file</span><span class="p">,</span> <span class="n">wayback</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">url_file</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">urls</span> <span class="o">=</span> <span class="p">[</span><span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">handle</span> <span class="k">if</span> <span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">line</span><span class="p">.</span><span class="nf">startswith</span><span class="p">(</span><span class="sh">"</span><span class="s">#</span><span class="sh">"</span><span class="p">)]</span> <span class="n">robots</span><span class="p">,</span> <span class="n">rows</span><span class="p">,</span> <span class="n">pages</span> <span class="o">=</span> <span class="p">{},</span> <span class="p">[],</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="n">urls</span><span class="p">:</span> <span class="n">target</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="s">https://web.archive.org/web/</span><span class="si">{</span><span class="n">wayback</span><span class="si">}</span><span class="s">id_/</span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="sh">"</span> <span class="k">if</span> <span class="n">wayback</span> <span class="k">else</span> <span class="n">url</span> <span class="n">page</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">url</span><span class="sh">"</span><span class="p">:</span> <span class="n">url</span><span class="p">}</span> <span class="k">try</span><span class="p">:</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="nf">max</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="nf">allowed</span><span class="p">(</span><span class="n">target</span><span class="p">,</span> <span class="n">robots</span><span class="p">)))</span> <span class="n">html</span><span class="p">,</span> <span class="n">page</span><span class="p">[</span><span class="sh">"</span><span class="s">fetched</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="nf">get</span><span class="p">(</span><span class="n">target</span><span class="p">)</span> <span class="n">found</span><span class="p">,</span> <span class="n">page</span><span class="p">[</span><span class="sh">"</span><span class="s">problems</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="nf">extract</span><span class="p">(</span><span class="n">html</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span> <span class="n">rows</span> <span class="o">+=</span> <span class="n">found</span> <span class="k">except</span> <span class="nb">OSError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="c1"># HTTP errors, timeouts, robots refusals (PermissionError) </span> <span class="n">page</span><span class="p">[</span><span class="sh">"</span><span class="s">error</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="nf">str</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="n">pages</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">page</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s">: </span><span class="si">{</span><span class="n">page</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">'</span><span class="s">error</span><span class="sh">'</span><span class="p">)</span> <span class="ow">or</span> <span class="sa">f</span><span class="sh">'</span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">found</span><span class="p">)</span><span class="si">}</span> <span class="nf">product</span><span class="p">(</span><span class="n">s</span><span class="p">)</span><span class="sh">'</span><span class="si">}</span><span class="s"> </span><span class="si">{</span><span class="n">page</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">'</span><span class="s">problems</span><span class="sh">'</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">''</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="n">captured</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="nf">now</span><span class="p">(</span><span class="n">timezone</span><span class="p">.</span><span class="n">utc</span><span class="p">).</span><span class="nf">strftime</span><span class="p">(</span><span class="sh">"</span><span class="s">%Y-%m-%dT%H:%M:%SZ</span><span class="sh">"</span><span class="p">)</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">out_file</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="nf">dump</span><span class="p">({</span><span class="sh">"</span><span class="s">captured_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">captured</span><span class="p">,</span> <span class="sh">"</span><span class="s">pages</span><span class="sh">"</span><span class="p">:</span> <span class="n">pages</span><span class="p">,</span> <span class="sh">"</span><span class="s">rows</span><span class="sh">"</span><span class="p">:</span> <span class="n">rows</span><span class="p">},</span> <span class="n">handle</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span><span class="si">}</span><span class="s"> product rows from </span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">pages</span><span class="p">)</span><span class="si">}</span><span class="s"> pages -&gt; </span><span class="si">{</span><span class="n">out_file</span><span class="si">}</span><span class="s"> (</span><span class="si">{</span><span class="n">captured</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span><span class="p">)</span> <span class="k">def</span> <span class="nf">diff</span><span class="p">(</span><span class="n">before</span><span class="p">,</span> <span class="n">after</span><span class="p">):</span> <span class="sh">"""</span><span class="s">List (change, key, field, old, new). A page that failed to load is </span><span class="sh">'</span><span class="s">unknown</span><span class="sh">'</span><span class="s">, not </span><span class="sh">'</span><span class="s">removed</span><span class="sh">'</span><span class="s">.</span><span class="sh">"""</span> <span class="n">old</span> <span class="o">=</span> <span class="p">{</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">]:</span> <span class="n">row</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">before</span><span class="p">[</span><span class="sh">"</span><span class="s">rows</span><span class="sh">"</span><span class="p">]}</span> <span class="n">new</span> <span class="o">=</span> <span class="p">{</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">]:</span> <span class="n">row</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">after</span><span class="p">[</span><span class="sh">"</span><span class="s">rows</span><span class="sh">"</span><span class="p">]}</span> <span class="n">failed</span> <span class="o">=</span> <span class="p">{</span><span class="n">page</span><span class="p">[</span><span class="sh">"</span><span class="s">url</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">page</span> <span class="ow">in</span> <span class="n">after</span><span class="p">[</span><span class="sh">"</span><span class="s">pages</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="n">page</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">error</span><span class="sh">"</span><span class="p">)}</span> <span class="n">changes</span> <span class="o">=</span> <span class="p">[(</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="n">new</span><span class="p">[</span><span class="n">key</span><span class="p">][</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">])</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">new</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">-</span> <span class="n">old</span><span class="p">.</span><span class="nf">keys</span><span class="p">())]</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">old</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">-</span> <span class="n">new</span><span class="p">.</span><span class="nf">keys</span><span class="p">()):</span> <span class="n">kind</span> <span class="o">=</span> <span class="sh">"</span><span class="s">unknown</span><span class="sh">"</span> <span class="k">if</span> <span class="n">old</span><span class="p">[</span><span class="n">key</span><span class="p">][</span><span class="sh">"</span><span class="s">page</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="n">failed</span> <span class="k">else</span> <span class="sh">"</span><span class="s">removed</span><span class="sh">"</span> <span class="n">changes</span><span class="p">.</span><span class="nf">append</span><span class="p">((</span><span class="n">kind</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="n">old</span><span class="p">[</span><span class="n">key</span><span class="p">][</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">],</span> <span class="sh">""</span><span class="p">))</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">old</span><span class="p">.</span><span class="nf">keys</span><span class="p">()</span> <span class="o">&amp;</span> <span class="n">new</span><span class="p">.</span><span class="nf">keys</span><span class="p">()):</span> <span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">WATCHED</span><span class="p">:</span> <span class="k">if</span> <span class="n">old</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="n">field</span><span class="p">)</span> <span class="o">!=</span> <span class="n">new</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="n">field</span><span class="p">):</span> <span class="n">changes</span><span class="p">.</span><span class="nf">append</span><span class="p">((</span><span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">field</span><span class="p">,</span> <span class="n">old</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="n">field</span><span class="p">),</span> <span class="n">new</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">get</span><span class="p">(</span><span class="n">field</span><span class="p">)))</span> <span class="k">return</span> <span class="n">changes</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">):</span> <span class="k">if</span> <span class="n">args</span><span class="p">[:</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="p">[</span><span class="sh">"</span><span class="s">snapshot</span><span class="sh">"</span><span class="p">]</span> <span class="ow">and</span> <span class="nf">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">):</span> <span class="nf">snapshot</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">args</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="n">args</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span> <span class="k">if</span> <span class="n">args</span><span class="p">[</span><span class="mi">3</span><span class="p">:</span><span class="mi">4</span><span class="p">]</span> <span class="o">==</span> <span class="p">[</span><span class="sh">"</span><span class="s">--wayback</span><span class="sh">"</span><span class="p">]</span> <span class="k">else</span> <span class="bp">None</span><span class="p">)</span> <span class="k">elif</span> <span class="n">args</span><span class="p">[:</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="p">[</span><span class="sh">"</span><span class="s">diff</span><span class="sh">"</span><span class="p">]</span> <span class="ow">and</span> <span class="nf">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="o">==</span> <span class="mi">4</span><span class="p">:</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">a</span><span class="p">,</span> <span class="nf">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">b</span><span class="p">:</span> <span class="n">before</span><span class="p">,</span> <span class="n">after</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="n">json</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">b</span><span class="p">)</span> <span class="n">changes</span> <span class="o">=</span> <span class="nf">diff</span><span class="p">(</span><span class="n">before</span><span class="p">,</span> <span class="n">after</span><span class="p">)</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">3</span><span class="p">],</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">newline</span><span class="o">=</span><span class="sh">""</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">writer</span> <span class="o">=</span> <span class="n">csv</span><span class="p">.</span><span class="nf">writer</span><span class="p">(</span><span class="n">handle</span><span class="p">)</span> <span class="n">writer</span><span class="p">.</span><span class="nf">writerow</span><span class="p">([</span><span class="sh">"</span><span class="s">change</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">field</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">old</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">])</span> <span class="n">writer</span><span class="p">.</span><span class="nf">writerows</span><span class="p">(</span><span class="n">changes</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">before </span><span class="si">{</span><span class="n">before</span><span class="p">[</span><span class="sh">'</span><span class="s">captured_at</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="s"> (</span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">before</span><span class="p">[</span><span class="sh">'</span><span class="s">rows</span><span class="sh">'</span><span class="p">])</span><span class="si">}</span><span class="s"> rows), </span><span class="sh">"</span> <span class="sa">f</span><span class="sh">"</span><span class="s">after </span><span class="si">{</span><span class="n">after</span><span class="p">[</span><span class="sh">'</span><span class="s">captured_at</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="s"> (</span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">after</span><span class="p">[</span><span class="sh">'</span><span class="s">rows</span><span class="sh">'</span><span class="p">])</span><span class="si">}</span><span class="s"> rows): </span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">changes</span><span class="p">)</span><span class="si">}</span><span class="s"> change(s)</span><span class="sh">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">change</span> <span class="ow">in</span> <span class="n">changes</span><span class="p">:</span> <span class="nf">print</span><span class="p">(</span><span class="sh">"</span><span class="s"> </span><span class="sh">"</span> <span class="o">+</span> <span class="sh">"</span><span class="s"> | </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="nf">str</span><span class="p">(</span><span class="n">part</span><span class="p">)</span> <span class="k">for</span> <span class="n">part</span> <span class="ow">in</span> <span class="n">change</span><span class="p">))</span> <span class="k">else</span><span class="p">:</span> <span class="n">sys</span><span class="p">.</span><span class="nf">exit</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="sh">"</span><span class="s">__main__</span><span class="sh">"</span><span class="p">:</span> <span class="nf">main</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span> </code></pre> </div> <p></p> <p></p> test_catalog_snapshot.py (110 lines) <br> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="kn">import</span> <span class="n">json</span> <span class="kn">import</span> <span class="n">unittest</span> <span class="kn">from</span> <span class="n">catalog_snapshot</span> <span class="kn">import</span> <span class="n">clean_price</span><span class="p">,</span> <span class="n">diff</span><span class="p">,</span> <span class="n">extract</span><span class="p">,</span> <span class="n">robots_rules</span> <span class="k">def</span> <span class="nf">page</span><span class="p">(</span><span class="o">*</span><span class="n">blocks</span><span class="p">,</span> <span class="n">kind</span><span class="o">=</span><span class="sh">"</span><span class="s">application/ld+json</span><span class="sh">"</span><span class="p">):</span> <span class="n">scripts</span> <span class="o">=</span> <span class="sh">""</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="sa">f</span><span class="sh">'</span><span class="s">&lt;script type=</span><span class="sh">"</span><span class="si">{</span><span class="n">kind</span><span class="si">}</span><span class="sh">"</span><span class="s">&gt;</span><span class="si">{</span><span class="n">b</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">b</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">else</span> <span class="n">json</span><span class="p">.</span><span class="nf">dumps</span><span class="p">(</span><span class="n">b</span><span class="p">)</span><span class="si">}</span><span class="s">&lt;/script&gt;</span><span class="sh">'</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">blocks</span><span class="p">)</span> <span class="k">return</span> <span class="sa">f</span><span class="sh">"</span><span class="s">&lt;html&gt;&lt;head&gt;</span><span class="si">{</span><span class="n">scripts</span><span class="si">}</span><span class="s">&lt;/head&gt;&lt;body&gt;&lt;p&gt;hi&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</span><span class="sh">"</span> <span class="k">class</span> <span class="nc">CleanPrice</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_values</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">clean_price</span><span class="p">(</span><span class="sh">"</span><span class="s">19.990</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">19.99</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">clean_price</span><span class="p">(</span><span class="mi">20</span><span class="p">),</span> <span class="sh">"</span><span class="s">20</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">clean_price</span><span class="p">(</span><span class="sh">"</span><span class="s">20.00</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">20</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">clean_price</span><span class="p">(</span><span class="sh">"</span><span class="s"> 9.5 </span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">9.5</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertIsNone</span><span class="p">(</span><span class="nf">clean_price</span><span class="p">(</span><span class="sh">""</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertIsNone</span><span class="p">(</span><span class="nf">clean_price</span><span class="p">(</span><span class="sh">"</span><span class="s">1,299.00</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertIsNone</span><span class="p">(</span><span class="nf">clean_price</span><span class="p">(</span><span class="sh">"</span><span class="s">NaN</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertIsNone</span><span class="p">(</span><span class="nf">clean_price</span><span class="p">(</span><span class="bp">True</span><span class="p">))</span> <span class="k">class</span> <span class="nc">Extract</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_aggregate_offer</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">rows</span><span class="p">,</span> <span class="n">problems</span> <span class="o">=</span> <span class="nf">extract</span><span class="p">(</span><span class="nf">page</span><span class="p">({</span> <span class="sh">"</span><span class="s">@context</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://schema.org/</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Product</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s"> Tea </span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">offers</span><span class="sh">"</span><span class="p">:</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">AggregateOffer</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">USD</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">lowPrice</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">9.99</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">highPrice</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">19.99</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">offerCount</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">6</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://schema.org/InStock</span><span class="sh">"</span><span class="p">}}),</span> <span class="sh">"</span><span class="s">https://x/p/1</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">problems</span><span class="p">,</span> <span class="p">[])</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">rows</span><span class="p">,</span> <span class="p">[{</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://x/p/1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">page</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://x/p/1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Tea</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">USD</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">9.99</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">high_price</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">19.99</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">InStock</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">offer_count</span><span class="sh">"</span><span class="p">:</span> <span class="mi">6</span><span class="p">}])</span> <span class="k">def</span> <span class="nf">test_graph_type_list_sku_and_offer_list</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">rows</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="nf">extract</span><span class="p">(</span><span class="nf">page</span><span class="p">({</span><span class="sh">"</span><span class="s">@context</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://schema.org</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">@graph</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">WebPage</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">ignored</span><span class="sh">"</span><span class="p">},</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="sh">"</span><span class="s">Product</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">Thing</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">sku</span><span class="sh">"</span><span class="p">:</span> <span class="mi">42</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Mug</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">offers</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Offer</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">price</span><span class="sh">"</span><span class="p">:</span> <span class="mi">12</span><span class="p">,</span> <span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">EUR</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">http://schema.org/InStock</span><span class="sh">"</span><span class="p">},</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Offer</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">priceSpecification</span><span class="sh">"</span><span class="p">:</span> <span class="p">{</span><span class="sh">"</span><span class="s">price</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">10.50</span><span class="sh">"</span><span class="p">},</span> <span class="sh">"</span><span class="s">priceCurrency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">EUR</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">OutOfStock</span><span class="sh">"</span><span class="p">}]}]}),</span> <span class="sh">"</span><span class="s">https://x/p/2</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">rows</span><span class="p">),</span> <span class="mi">1</span><span class="p">)</span> <span class="n">row</span> <span class="o">=</span> <span class="n">rows</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">high_price</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="sh">"</span><span class="s">42</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">10.5</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">12</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">offer_count</span><span class="sh">"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="sh">"</span><span class="s">mixed</span><span class="sh">"</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="sh">"</span><span class="s">EUR</span><span class="sh">"</span><span class="p">))</span> <span class="k">def</span> <span class="nf">test_compact_type_and_charset_parameter</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">rows</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="nf">extract</span><span class="p">(</span><span class="nf">page</span><span class="p">({</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">schema:Product</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Cap</span><span class="sh">"</span><span class="p">},</span> <span class="n">kind</span><span class="o">=</span><span class="sh">"</span><span class="s">application/ld+json; charset=utf-8</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">https://x/p/3</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">rows</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">Cap</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">rows</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">],</span> <span class="n">rows</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="sh">"</span><span class="s">offer_count</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="mi">0</span><span class="p">))</span> <span class="k">def</span> <span class="nf">test_bad_block_does_not_hide_good_block</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">rows</span><span class="p">,</span> <span class="n">problems</span> <span class="o">=</span> <span class="nf">extract</span><span class="p">(</span><span class="nf">page</span><span class="p">(</span><span class="sh">'</span><span class="s">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="s">: </span><span class="sh">"</span><span class="s">Product</span><span class="sh">"</span><span class="s">,}</span><span class="sh">'</span><span class="p">,</span> <span class="p">{</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Product</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Ok</span><span class="sh">"</span><span class="p">}),</span> <span class="sh">"</span><span class="s">u</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">([</span><span class="n">r</span><span class="p">[</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">],</span> <span class="p">[</span><span class="sh">"</span><span class="s">Ok</span><span class="sh">"</span><span class="p">])</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">problems</span><span class="p">),</span> <span class="mi">1</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertIn</span><span class="p">(</span><span class="sh">"</span><span class="s">not valid JSON</span><span class="sh">"</span><span class="p">,</span> <span class="n">problems</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="k">def</span> <span class="nf">test_no_jsonld_and_no_product</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">extract</span><span class="p">(</span><span class="sh">"</span><span class="s">&lt;p&gt;plain&lt;/p&gt;</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">u</span><span class="sh">"</span><span class="p">),</span> <span class="p">([],</span> <span class="p">[</span><span class="sh">"</span><span class="s">no JSON-LD on the page</span><span class="sh">"</span><span class="p">]))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">extract</span><span class="p">(</span><span class="nf">page</span><span class="p">({</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Organization</span><span class="sh">"</span><span class="p">}),</span> <span class="sh">"</span><span class="s">u</span><span class="sh">"</span><span class="p">),</span> <span class="p">([],</span> <span class="p">[</span><span class="sh">"</span><span class="s">JSON-LD present but no Product</span><span class="sh">"</span><span class="p">]))</span> <span class="k">def</span> <span class="nf">test_other_script_types_ignored</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">rows</span><span class="p">,</span> <span class="n">problems</span> <span class="o">=</span> <span class="nf">extract</span><span class="p">(</span><span class="nf">page</span><span class="p">({</span><span class="sh">"</span><span class="s">@type</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">Product</span><span class="sh">"</span><span class="p">},</span> <span class="n">kind</span><span class="o">=</span><span class="sh">"</span><span class="s">application/json</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">u</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">rows</span><span class="p">,</span> <span class="n">problems</span><span class="p">),</span> <span class="p">([],</span> <span class="p">[</span><span class="sh">"</span><span class="s">no JSON-LD on the page</span><span class="sh">"</span><span class="p">]))</span> <span class="k">def</span> <span class="nf">snap</span><span class="p">(</span><span class="n">rows</span><span class="p">,</span> <span class="n">failed</span><span class="o">=</span><span class="p">()):</span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">captured_at</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">t</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">rows</span><span class="sh">"</span><span class="p">:</span> <span class="n">rows</span><span class="p">,</span> <span class="sh">"</span><span class="s">pages</span><span class="sh">"</span><span class="p">:</span> <span class="p">[{</span><span class="sh">"</span><span class="s">url</span><span class="sh">"</span><span class="p">:</span> <span class="n">u</span><span class="p">,</span> <span class="sh">"</span><span class="s">error</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">HTTP Error 503</span><span class="sh">"</span><span class="p">}</span> <span class="k">for</span> <span class="n">u</span> <span class="ow">in</span> <span class="n">failed</span><span class="p">]}</span> <span class="k">def</span> <span class="nf">row</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="o">**</span><span class="n">fields</span><span class="p">):</span> <span class="n">base</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">key</span><span class="sh">"</span><span class="p">:</span> <span class="n">key</span><span class="p">,</span> <span class="sh">"</span><span class="s">page</span><span class="sh">"</span><span class="p">:</span> <span class="n">key</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="n">key</span><span class="p">.</span><span class="nf">upper</span><span class="p">(),</span> <span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">USD</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">high_price</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">InStock</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">offer_count</span><span class="sh">"</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span> <span class="k">return</span> <span class="p">{</span><span class="o">**</span><span class="n">base</span><span class="p">,</span> <span class="o">**</span><span class="n">fields</span><span class="p">}</span> <span class="k">class</span> <span class="nc">Diff</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_added_removed_changed</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">before</span> <span class="o">=</span> <span class="nf">snap</span><span class="p">([</span><span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">),</span> <span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">b</span><span class="sh">"</span><span class="p">),</span> <span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">c</span><span class="sh">"</span><span class="p">)])</span> <span class="n">after</span> <span class="o">=</span> <span class="nf">snap</span><span class="p">([</span><span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">,</span> <span class="n">low_price</span><span class="o">=</span><span class="sh">"</span><span class="s">0.9</span><span class="sh">"</span><span class="p">,</span> <span class="n">availability</span><span class="o">=</span><span class="sh">"</span><span class="s">OutOfStock</span><span class="sh">"</span><span class="p">),</span> <span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">c</span><span class="sh">"</span><span class="p">),</span> <span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">d</span><span class="sh">"</span><span class="p">)])</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">diff</span><span class="p">(</span><span class="n">before</span><span class="p">,</span> <span class="n">after</span><span class="p">),</span> <span class="p">[</span> <span class="p">(</span><span class="sh">"</span><span class="s">added</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">d</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="sh">"</span><span class="s">D</span><span class="sh">"</span><span class="p">),</span> <span class="p">(</span><span class="sh">"</span><span class="s">removed</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">b</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="sh">"</span><span class="s">B</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">),</span> <span class="p">(</span><span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">low_price</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">0.9</span><span class="sh">"</span><span class="p">),</span> <span class="p">(</span><span class="sh">"</span><span class="s">changed</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">availability</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">InStock</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">OutOfStock</span><span class="sh">"</span><span class="p">),</span> <span class="p">])</span> <span class="k">def</span> <span class="nf">test_failed_page_is_unknown_not_removed</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">diff</span><span class="p">(</span><span class="nf">snap</span><span class="p">([</span><span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">)]),</span> <span class="nf">snap</span><span class="p">([],</span> <span class="n">failed</span><span class="o">=</span><span class="p">[</span><span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">])),</span> <span class="p">[(</span><span class="sh">"</span><span class="s">unknown</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">,</span> <span class="sh">"</span><span class="s">A</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)])</span> <span class="k">def</span> <span class="nf">test_identical_snapshots</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">diff</span><span class="p">(</span><span class="nf">snap</span><span class="p">([</span><span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">)]),</span> <span class="nf">snap</span><span class="p">([</span><span class="nf">row</span><span class="p">(</span><span class="sh">"</span><span class="s">a</span><span class="sh">"</span><span class="p">)])),</span> <span class="p">[])</span> <span class="k">class</span> <span class="nc">Robots</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_blank_line_does_not_end_the_group</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">rules</span> <span class="o">=</span> <span class="nf">robots_rules</span><span class="p">(</span><span class="sh">"</span><span class="s">User-agent: *</span><span class="se">\n</span><span class="s">Disallow: /cart</span><span class="se">\n\n</span><span class="s"># checkout</span><span class="se">\n</span><span class="s">Disallow: /checkout</span><span class="se">\n</span><span class="sh">"</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertFalse</span><span class="p">(</span><span class="n">rules</span><span class="p">.</span><span class="nf">can_fetch</span><span class="p">(</span><span class="sh">"</span><span class="s">catalog-snapshot</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">https://x/checkout/1</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertFalse</span><span class="p">(</span><span class="n">rules</span><span class="p">.</span><span class="nf">can_fetch</span><span class="p">(</span><span class="sh">"</span><span class="s">catalog-snapshot</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">https://x/cart</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertTrue</span><span class="p">(</span><span class="n">rules</span><span class="p">.</span><span class="nf">can_fetch</span><span class="p">(</span><span class="sh">"</span><span class="s">catalog-snapshot</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">https://x/product/1</span><span class="sh">"</span><span class="p">))</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="sh">"</span><span class="s">__main__</span><span class="sh">"</span><span class="p">:</span> <span class="n">unittest</span><span class="p">.</span><span class="nf">main</span><span class="p">()</span> </code></pre> </div> <p></p> <p>Run the tests with <code>python -m unittest -v test_catalog_snapshot</code>.</p> <p><em>This article and its code were drafted by an AI assistant at the account owner's request. The code was run on 27 September 2026: the archived snapshot finished at 13:00:09 UTC, the live snapshot at 13:00:37 UTC, and the tests passed the same day. The blank-line handling for <code>robots.txt</code> was added later the same day after a review; it makes the same <code>robots.txt</code> decisions for all twelve URLs in the run, so it does not change the output shown above.</em></p> python webscraping ecommerce tutorial Abdulwahab Sun, 27 Sep 2026 18:40:06 +0000 https://dev.to/abdulwhab95/watching-github-releases-with-etags-and-304s-in-python-4721 https://dev.to/abdulwhab95/watching-github-releases-with-etags-and-304s-in-python-4721 <p>If you depend on a handful of open-source projects, you probably want to know when one of them ships a release. When you administer the repository, a webhook is the right tool, and GitHub's own best-practice guide recommends webhook events over polling. For repositories you don't control, polling the REST API on a schedule is the usual answer, and it can be done cheaply.</p> <p>This post builds a small standard-library Python script that:</p> <ol> <li>asks the releases endpoint of each repository once per run,</li> <li>sends the <code>ETag</code> from last time, so an unchanged list comes back as <code>304 Not Modified</code> with no body,</li> <li>keeps a small JSON state file with the release IDs it has already seen,</li> <li>stays silent on the first run instead of announcing twenty "new" releases,</li> <li>keeps the old state when a request fails, and stops when it is rate limited.</li> </ol> <p>At the end there is a real, unauthenticated run from 27 September 2026 (UTC) and the complete script.</p> <h2> What the documentation says </h2> <p>These points come from GitHub's REST API documentation, read on 27 September 2026:</p> <ul> <li> <strong>Endpoint:</strong> <code>GET /repos/{owner}/{repo}/releases</code>. <code>per_page</code> defaults to 30, maximum 100. It works without authentication for public repositories. Draft releases are only listed for users with push access.</li> <li> <strong>Rate limit:</strong> unauthenticated requests get 60 per hour, counted per IP address. A personal access token raises that to 5,000 per hour.</li> <li> <strong>Conditional requests:</strong> most endpoints return an <code>etag</code> header. Send it back in <code>if-none-match</code>; if nothing changed you get <code>304 Not Modified</code>. The docs say a 304 does not count against the primary rate limit <strong>if the request was made with an <code>Authorization</code> header</strong>.</li> <li> <strong>Versioning:</strong> requests without an <code>X-GitHub-Api-Version</code> header use version <code>2022-11-28</code>. The current version is <code>2026-03-10</code>. Pinning the version keeps a breaking change from reaching a script silently.</li> <li> <strong>Polling etiquette:</strong> poll only as often as you need, on a fixed schedule; make requests one after another, not concurrently.</li> </ul> <h2> Step 1: build the request </h2> <p>The <code>If-None-Match</code> header is only sent when there is a saved ETag. The token is optional and read from the environment, never from the code.</p> <p><em>Excerpt of <code>release_watch.py</code>, lines 14-26:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="n">API</span> <span class="o">=</span> <span class="sh">"</span><span class="s">https://api.github.com/repos/{}/releases?per_page=20</span><span class="sh">"</span> <span class="n">USER_AGENT</span> <span class="o">=</span> <span class="sh">"</span><span class="s">release-watch-example/1.0 (tutorial script)</span><span class="sh">"</span> <span class="n">KEEP_IDS</span> <span class="o">=</span> <span class="mi">200</span> <span class="c1"># enough history to recognise releases we already reported </span> <span class="k">def</span> <span class="nf">build_request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">etag</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">User-Agent</span><span class="sh">"</span><span class="p">:</span> <span class="n">USER_AGENT</span><span class="p">,</span> <span class="sh">"</span><span class="s">Accept</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">application/vnd.github+json</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">X-GitHub-Api-Version</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">2026-03-10</span><span class="sh">"</span><span class="p">}</span> <span class="k">if</span> <span class="n">etag</span><span class="p">:</span> <span class="n">headers</span><span class="p">[</span><span class="sh">"</span><span class="s">If-None-Match</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="n">etag</span> <span class="k">if</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">GITHUB_TOKEN</span><span class="sh">"</span><span class="p">):</span> <span class="n">headers</span><span class="p">[</span><span class="sh">"</span><span class="s">Authorization</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="sh">"</span><span class="s">Bearer </span><span class="sh">"</span> <span class="o">+</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">[</span><span class="sh">"</span><span class="s">GITHUB_TOKEN</span><span class="sh">"</span><span class="p">]</span> <span class="k">return</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nc">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span> </code></pre> </div> <h2> Step 2: treat 304 as an answer, not an error </h2> <p><code>urllib</code> raises <code>HTTPError</code> for a 304, exactly as it does for a 404 or a 503. Catching it and returning the status keeps the calling code simple: one status code to branch on.</p> <p><em>Excerpt of <code>release_watch.py</code>, lines 29-35:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">etag</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Return (status, headers, JSON body or None). A 304 is a status here, not an exception.</span><span class="sh">"""</span> <span class="k">try</span><span class="p">:</span> <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nf">urlopen</span><span class="p">(</span><span class="nf">build_request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">etag</span><span class="p">),</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">30</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span> <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="n">status</span><span class="p">,</span> <span class="n">response</span><span class="p">.</span><span class="n">headers</span><span class="p">,</span> <span class="n">json</span><span class="p">.</span><span class="nf">loads</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="nf">read</span><span class="p">())</span> <span class="k">except</span> <span class="n">urllib</span><span class="p">.</span><span class="n">error</span><span class="p">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="c1"># urllib raises for 304 and for every 4xx/5xx </span> <span class="k">return</span> <span class="n">err</span><span class="p">.</span><span class="n">code</span><span class="p">,</span> <span class="n">err</span><span class="p">.</span><span class="n">headers</span><span class="p">,</span> <span class="bp">None</span> </code></pre> </div> <h2> Step 3: decide what is new </h2> <p>This is the core of the script. The rules:</p> <ul> <li> <strong>304:</strong> nothing changed, the saved state stays as it is.</li> <li> <strong>Anything other than 200 or 304</strong> (a 404, a 5xx, a rate limit): keep the saved state untouched. A failed request must never look like "no releases".</li> <li> <strong>First run for a repository:</strong> save every release ID and report nothing. Otherwise the first run would announce the project's whole history.</li> <li> <strong>Later runs:</strong> a release is new when its ID is not in the saved list. Compare IDs, not the fact that the response was a 200: the ETag can change for reasons other than a new release, as the real run below shows.</li> </ul> <p><em>Excerpt of <code>release_watch.py</code>, lines 38-58:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">summary</span><span class="p">(</span><span class="n">release</span><span class="p">):</span> <span class="k">return</span> <span class="p">{</span><span class="n">key</span><span class="p">:</span> <span class="n">release</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">tag_name</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">published_at</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">prerelease</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">html_url</span><span class="sh">"</span><span class="p">)}</span> <span class="k">def</span> <span class="nf">check</span><span class="p">(</span><span class="n">repo</span><span class="p">,</span> <span class="n">entry</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="n">fetch</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Check one repo against its saved state. Returns (new state entry, report).</span><span class="sh">"""</span> <span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">body</span> <span class="o">=</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">API</span><span class="p">.</span><span class="nf">format</span><span class="p">(</span><span class="n">repo</span><span class="p">),</span> <span class="n">entry</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">))</span> <span class="n">report</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">repo</span><span class="sh">"</span><span class="p">:</span> <span class="n">repo</span><span class="p">,</span> <span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">:</span> <span class="n">status</span><span class="p">,</span> <span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">remaining</span><span class="sh">"</span><span class="p">:</span> <span class="n">headers</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">x-ratelimit-remaining</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="n">headers</span> <span class="k">else</span> <span class="bp">None</span><span class="p">}</span> <span class="k">if</span> <span class="n">status</span> <span class="o">!=</span> <span class="mi">200</span><span class="p">:</span> <span class="c1"># 304: nothing changed. Anything else: keep the old state untouched. </span> <span class="k">return</span> <span class="n">entry</span><span class="p">,</span> <span class="n">report</span> <span class="n">releases</span> <span class="o">=</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">body</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">r</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">draft</span><span class="sh">"</span><span class="p">)]</span> <span class="k">if</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span> <span class="ow">in</span> <span class="n">entry</span><span class="p">:</span> <span class="n">seen</span> <span class="o">=</span> <span class="nf">set</span><span class="p">(</span><span class="n">entry</span><span class="p">[</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">])</span> <span class="n">fresh</span> <span class="o">=</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">releases</span> <span class="k">if</span> <span class="n">r</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">seen</span><span class="p">]</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nf">summary</span><span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">fresh</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">r</span><span class="p">:</span> <span class="n">r</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">published_at</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">""</span><span class="p">)]</span> <span class="k">else</span><span class="p">:</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">baseline</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="nf">len</span><span class="p">(</span><span class="n">releases</span><span class="p">)</span> <span class="c1"># first run: remember everything, alert on nothing </span> <span class="n">current</span> <span class="o">=</span> <span class="p">[</span><span class="n">r</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">releases</span><span class="p">]</span> <span class="n">older</span> <span class="o">=</span> <span class="p">[</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">entry</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">,</span> <span class="p">[])</span> <span class="k">if</span> <span class="n">i</span> <span class="ow">not</span> <span class="ow">in</span> <span class="nf">set</span><span class="p">(</span><span class="n">current</span><span class="p">)]</span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">:</span> <span class="n">headers</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">ETag</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="p">(</span><span class="n">current</span> <span class="o">+</span> <span class="n">older</span><span class="p">)[:</span><span class="n">KEEP_IDS</span><span class="p">]},</span> <span class="n">report</span> </code></pre> </div> <p>The saved list keeps the newest IDs first and is capped at 200 entries. <code>per_page=20</code> means the script sees the 20 newest releases per request. If a project can publish more than 20 releases between two runs, raise <code>per_page</code> (up to 100) or follow the pagination links.</p> <h2> Step 4: the run loop and the state file </h2> <p>Requests go out one per second, in order. On a 403 or 429 the run stops rather than pushing on. The state file is written to a temporary file first and then moved into place with <code>os.replace</code>, so an interrupted run cannot leave half a JSON file behind.</p> <p><em>Excerpt of <code>release_watch.py</code>, lines 74-97:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">repo_file</span><span class="p">,</span> <span class="n">state_file</span><span class="p">):</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">repo_file</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">repos</span> <span class="o">=</span> <span class="p">[</span><span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">handle</span> <span class="k">if</span> <span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">line</span><span class="p">.</span><span class="nf">startswith</span><span class="p">(</span><span class="sh">"</span><span class="s">#</span><span class="sh">"</span><span class="p">)]</span> <span class="k">try</span><span class="p">:</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">state_file</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">state</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">handle</span><span class="p">)</span> <span class="k">except</span> <span class="nb">FileNotFoundError</span><span class="p">:</span> <span class="n">state</span> <span class="o">=</span> <span class="p">{}</span> <span class="nf">print</span><span class="p">(</span><span class="sh">"</span><span class="s">run at</span><span class="sh">"</span><span class="p">,</span> <span class="n">datetime</span><span class="p">.</span><span class="nf">now</span><span class="p">(</span><span class="n">timezone</span><span class="p">.</span><span class="n">utc</span><span class="p">).</span><span class="nf">strftime</span><span class="p">(</span><span class="sh">"</span><span class="s">%Y-%m-%dT%H:%M:%SZ</span><span class="sh">"</span><span class="p">))</span> <span class="k">for</span> <span class="n">number</span><span class="p">,</span> <span class="n">repo</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="n">repos</span><span class="p">):</span> <span class="k">if</span> <span class="n">number</span><span class="p">:</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># serial requests, one per second </span> <span class="n">state</span><span class="p">[</span><span class="n">repo</span><span class="p">],</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="n">repo</span><span class="p">,</span> <span class="n">state</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="n">repo</span><span class="p">,</span> <span class="p">{}))</span> <span class="nf">print</span><span class="p">(</span><span class="nf">describe</span><span class="p">(</span><span class="n">report</span><span class="p">))</span> <span class="k">for</span> <span class="n">release</span> <span class="ow">in</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">]:</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s"> NEW </span><span class="si">{</span><span class="n">release</span><span class="p">[</span><span class="sh">'</span><span class="s">tag_name</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="s"> published </span><span class="si">{</span><span class="n">release</span><span class="p">[</span><span class="sh">'</span><span class="s">published_at</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="sh">"</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="sh">'</span><span class="s"> (pre-release)</span><span class="sh">'</span> <span class="k">if</span> <span class="n">release</span><span class="p">[</span><span class="sh">'</span><span class="s">prerelease</span><span class="sh">'</span><span class="p">]</span> <span class="k">else</span> <span class="sh">''</span><span class="si">}</span><span class="s"> </span><span class="si">{</span><span class="n">release</span><span class="p">[</span><span class="sh">'</span><span class="s">html_url</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">403</span><span class="p">,</span> <span class="mi">429</span><span class="p">):</span> <span class="c1"># primary or secondary rate limit: stop, do not push on </span> <span class="nf">print</span><span class="p">(</span><span class="sh">"</span><span class="s">rate limited; stopping this run (state for earlier repos is still saved)</span><span class="sh">"</span><span class="p">)</span> <span class="k">break</span> <span class="n">temporary</span> <span class="o">=</span> <span class="n">state_file</span> <span class="o">+</span> <span class="sh">"</span><span class="s">.tmp</span><span class="sh">"</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">temporary</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="nf">dump</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">handle</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="n">os</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="n">temporary</span><span class="p">,</span> <span class="n">state_file</span><span class="p">)</span> <span class="c1"># never leave a half-written state file </span></code></pre> </div> <p>A network error that isn't an HTTP status (DNS failure, timeout) stops the whole run with a traceback before the state is written, so the previous state file stays intact.</p> <h2> Tests </h2> <p>Nine <code>unittest</code> tests use a fake API object instead of the network. The fake answers 304 when it receives its current ETag, so the conditional-request logic can be tested without sending a single request:</p> <p><em>Excerpt of <code>test_release_watch.py</code>, lines 13-26:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">class</span> <span class="nc">FakeApi</span><span class="p">:</span> <span class="sh">"""</span><span class="s">Stands in for fetch(): answers 304 when the caller sends the current ETag.</span><span class="sh">"""</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">releases</span><span class="p">,</span> <span class="n">etag</span><span class="o">=</span><span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">v1</span><span class="sh">"'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="n">releases</span><span class="p">,</span> <span class="n">self</span><span class="p">.</span><span class="n">etag</span><span class="p">,</span> <span class="n">self</span><span class="p">.</span><span class="n">status</span><span class="p">,</span> <span class="n">self</span><span class="p">.</span><span class="n">calls</span> <span class="o">=</span> <span class="n">releases</span><span class="p">,</span> <span class="n">etag</span><span class="p">,</span> <span class="n">status</span><span class="p">,</span> <span class="p">[]</span> <span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">etag</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="n">calls</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">etag</span><span class="p">)</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">ETag</span><span class="sh">"</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">etag</span><span class="p">,</span> <span class="sh">"</span><span class="s">x-ratelimit-remaining</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">59</span><span class="sh">"</span><span class="p">}</span> <span class="k">if</span> <span class="n">self</span><span class="p">.</span><span class="n">status</span> <span class="o">!=</span> <span class="mi">200</span><span class="p">:</span> <span class="k">return</span> <span class="n">self</span><span class="p">.</span><span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="bp">None</span> <span class="k">if</span> <span class="n">etag</span> <span class="o">==</span> <span class="n">self</span><span class="p">.</span><span class="n">etag</span><span class="p">:</span> <span class="k">return</span> <span class="mi">304</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="bp">None</span> <span class="k">return</span> <span class="mi">200</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="nf">list</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">releases</span><span class="p">)</span> </code></pre> </div> <p><em>Excerpt of <code>test_release_watch.py</code>, lines 36-41:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code> <span class="k">def</span> <span class="nf">test_unchanged_repo_returns_304_and_keeps_state</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">api</span> <span class="o">=</span> <span class="nc">FakeApi</span><span class="p">([</span><span class="nf">release</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="sh">"</span><span class="s">v1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-01-01T00:00:00Z</span><span class="sh">"</span><span class="p">)])</span> <span class="n">entry</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="p">{},</span> <span class="n">fetch</span><span class="o">=</span><span class="n">api</span><span class="p">)</span> <span class="n">again</span><span class="p">,</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="n">entry</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="n">api</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">],</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">],</span> <span class="n">again</span><span class="p">),</span> <span class="p">(</span><span class="mi">304</span><span class="p">,</span> <span class="p">[],</span> <span class="n">entry</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">api</span><span class="p">.</span><span class="n">calls</span><span class="p">,</span> <span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">v1</span><span class="sh">"'</span><span class="p">])</span> </code></pre> </div> <p><em>Excerpt of <code>test_release_watch.py</code>, lines 60-64:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code> <span class="k">def</span> <span class="nf">test_errors_keep_previous_state</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">previous</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">:</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">x</span><span class="sh">"'</span><span class="p">,</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]}</span> <span class="k">for</span> <span class="n">status</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">404</span><span class="p">,</span> <span class="mi">403</span><span class="p">,</span> <span class="mi">429</span><span class="p">,</span> <span class="mi">502</span><span class="p">):</span> <span class="n">entry</span><span class="p">,</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="n">previous</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="nc">FakeApi</span><span class="p">([],</span> <span class="n">status</span><span class="o">=</span><span class="n">status</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">entry</span><span class="p">,</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">],</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="n">previous</span><span class="p">,</span> <span class="p">[],</span> <span class="n">status</span><span class="p">))</span> </code></pre> </div> <p><em>Excerpt of <code>test-log.txt</code>, lines 11-14:</em><br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>---------------------------------------------------------------------- Ran 9 tests in 0.002s OK </code></pre> </div> <h2> A real run </h2> <p>Four public repositories, chosen because they differ: three publish GitHub releases, one does not.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code># owner/repo, one per line pallets/flask psf/black astral-sh/ruff python/cpython </code></pre> </div> <p>The runs below were made without a token. <code>GITHUB_TOKEN</code> was removed from the environment for each run.</p> <p><strong>Run 1</strong>, with no state file:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python release_watch.py repos.txt state.json # run 1, no state yet run at 2026-09-27T13:13:50Z pallets/flask HTTP 200 baseline: 20 releases saved, nothing reported (rate limit left: 51) psf/black HTTP 200 baseline: 20 releases saved, nothing reported (rate limit left: 50) astral-sh/ruff HTTP 200 baseline: 20 releases saved, nothing reported (rate limit left: 49) python/cpython HTTP 200 baseline: 0 releases saved, nothing reported (rate limit left: 48) </code></pre> </div> <p><code>python/cpython</code> answered 200 with an empty list. That repository publishes no GitHub release objects, so a release watcher sees nothing there; the releases endpoint lists releases, not Git tags.</p> <p><strong>Run 2</strong>, eleven seconds later, with the ETags saved by run 1:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python release_watch.py repos.txt state.json # run 2, nothing touched run at 2026-09-27T13:14:01Z pallets/flask HTTP 304 not modified (rate limit left: 47) psf/black HTTP 304 not modified (rate limit left: 46) astral-sh/ruff HTTP 304 not modified (rate limit left: 45) python/cpython HTTP 304 not modified (rate limit left: 44) </code></pre> </div> <p>Four 304s, no bodies, nothing reported.</p> <p><strong>Run 3.</strong> Waiting for a real release could take days, so to exercise the alert path, a small helper edits the state file: it removes the newest <code>astral-sh/ruff</code> release ID and the saved ETag, as if the last check had happened before that release. This is the only manipulated input in this post.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="sh">"""</span><span class="s">Demo helper: make the saved state look as if the last check ran before the newest release.</span><span class="sh">"""</span> <span class="kn">import</span> <span class="n">json</span> <span class="kn">import</span> <span class="n">sys</span> <span class="n">state_file</span><span class="p">,</span> <span class="n">repo</span> <span class="o">=</span> <span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">state_file</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">state</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">handle</span><span class="p">)</span> <span class="n">dropped</span> <span class="o">=</span> <span class="n">state</span><span class="p">[</span><span class="n">repo</span><span class="p">][</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">].</span><span class="nf">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># the API lists the newest release first </span><span class="k">del</span> <span class="n">state</span><span class="p">[</span><span class="n">repo</span><span class="p">][</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">]</span> <span class="c1"># without this, the next request would just get a 304 </span><span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">state_file</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="nf">dump</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">handle</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">repo</span><span class="si">}</span><span class="s">: forgot release id </span><span class="si">{</span><span class="n">dropped</span><span class="si">}</span><span class="s"> and the saved ETag</span><span class="sh">"</span><span class="p">)</span> </code></pre> </div> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python forget_newest.py state.json astral-sh/ruff astral-sh/ruff: forgot release id 396061666 and the saved ETag $ python release_watch.py repos.txt state.json # run 3 run at 2026-09-27T13:14:09Z pallets/flask HTTP 304 not modified (rate limit left: 43) psf/black HTTP 304 not modified (rate limit left: 42) astral-sh/ruff HTTP 200 1 new release(s) (rate limit left: 41) NEW 0.16.9 published 2026-09-24T20:38:52Z https://github.com/astral-sh/ruff/releases/tag/0.16.9 python/cpython HTTP 304 not modified (rate limit left: 40) </code></pre> </div> <p>The watcher found exactly the one release that had been removed from its memory (published on 24 September 2026) and nothing else.</p> <h2> What the run shows </h2> <p><strong>Unauthenticated 304s still cost a request.</strong> Look at the "rate limit left" column in run 2: 47, 46, 45, 44. Every 304 took one request from the 60-per-hour budget. That matches the documentation, which only promises free 304s for requests with an <code>Authorization</code> header. Without a token, conditional requests save bandwidth and parsing, not rate limit. With a token, they save both.</p> <p><strong>The ETag can change without a new release.</strong> The same four endpoints were also queried ten minutes earlier, at 13:03 UTC. Comparing those ETags with the ones saved at 13:13 to 13:14 UTC:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>ETag at 13:03:03Z (earlier run) vs ETag in state.json after the 13:13:50Z-13:14:09Z runs pallets/flask same W/"a15e35fe2fa... -&gt; W/"a15e35fe2fa... psf/black CHANGED W/"be5a64cb397... -&gt; W/"eb174407a1d... astral-sh/ruff CHANGED W/"804e242aa83... -&gt; W/"50f608ba2e0... python/cpython same "74d1db57f0fec... -&gt; "74d1db57f0fec... psf/black newest release id at 13:03:03Z 324430137, now 324430137: same astral-sh/ruff newest release id at 13:03:03Z 396061666, now 396061666: same </code></pre> </div> <p>Two ETags changed although the newest release ID of both repositories was the same at both times. Something else in the response changed. The run doesn't show what; release objects include fields that change without a new release, such as the <code>download_count</code> of each release asset, and an edited release note would also do it. That is why the script compares release IDs and never treats a 200 as proof of a new release.</p> <p><strong>Weak and strong ETags both work as-is.</strong> Three repositories returned weak ETags (<code>W/"..."</code>) and <code>python/cpython</code> a strong one. The script stores the header value unchanged and sends it back unchanged.</p> <h2> Limits and common mistakes </h2> <ul> <li> <strong>Announcing on the first run.</strong> Without the silent baseline, the first run reports every release the project ever made.</li> <li> <strong>Treating a failed request as "no releases".</strong> A 404, 5xx or 429 must leave the saved state alone.</li> <li> <strong>Treating any 200 as news.</strong> Compare IDs.</li> <li> <strong>Assuming 304s are free.</strong> Only with authentication, per the docs, and the run above confirms that unauthenticated 304s are counted.</li> <li> <strong>Pagination.</strong> With <code>per_page=20</code>, a burst of more than 20 releases between runs is partly missed.</li> <li> <strong>Polling too often.</strong> 60 requests per hour unauthenticated is shared by everything on your IP address. Pick a schedule that fits the number of repositories, and use a token if you need more.</li> <li> <strong>Retrying while rate limited.</strong> GitHub answers 403 or 429 for both the primary and the secondary rate limits. Its docs say to wait until <code>x-ratelimit-reset</code> when <code>x-ratelimit-remaining</code> is 0, to wait <code>retry-after</code> seconds when that header is present, and otherwise to wait at least a minute and back off exponentially; continuing to send requests while limited can get an integration banned. The script simply stops the run. A longer-lived tool should implement those waits.</li> </ul> <h2> Official sources </h2> <ul> <li>REST API, releases: <a href="https://docs.github.com/en/rest/releases/releases" rel="noopener noreferrer">https://docs.github.com/en/rest/releases/releases</a> </li> <li>Best practices, including conditional requests and polling: <a href="https://docs.github.com/en/rest/using-the-rest-api/best-practices-for-using-the-rest-api" rel="noopener noreferrer">https://docs.github.com/en/rest/using-the-rest-api/best-practices-for-using-the-rest-api</a> </li> <li>Rate limits for the REST API: <a href="https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api" rel="noopener noreferrer">https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api</a> </li> <li>API versions: <a href="https://docs.github.com/en/rest/about-the-rest-api/api-versions" rel="noopener noreferrer">https://docs.github.com/en/rest/about-the-rest-api/api-versions</a> </li> <li>Python <code>urllib.request</code>: <a href="https://docs.python.org/3/library/urllib.request.html" rel="noopener noreferrer">https://docs.python.org/3/library/urllib.request.html</a> </li> <li>Python <code>os.replace</code>: <a href="https://docs.python.org/3/library/os.html#os.replace" rel="noopener noreferrer">https://docs.python.org/3/library/os.html#os.replace</a> </li> </ul> <h2> The complete script and tests </h2> <p></p> release_watch.py (103 lines) <br> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="sh">"""</span><span class="s">Watch GitHub repositories for new releases, using conditional requests. Usage: python release_watch.py repos.txt state.json Optional: set GITHUB_TOKEN in the environment to use your own (higher) rate limit. </span><span class="sh">"""</span> <span class="kn">import</span> <span class="n">json</span> <span class="kn">import</span> <span class="n">os</span> <span class="kn">import</span> <span class="n">sys</span> <span class="kn">import</span> <span class="n">time</span> <span class="kn">import</span> <span class="n">urllib.error</span> <span class="kn">import</span> <span class="n">urllib.request</span> <span class="kn">from</span> <span class="n">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timezone</span> <span class="n">API</span> <span class="o">=</span> <span class="sh">"</span><span class="s">https://api.github.com/repos/{}/releases?per_page=20</span><span class="sh">"</span> <span class="n">USER_AGENT</span> <span class="o">=</span> <span class="sh">"</span><span class="s">release-watch-example/1.0 (tutorial script)</span><span class="sh">"</span> <span class="n">KEEP_IDS</span> <span class="o">=</span> <span class="mi">200</span> <span class="c1"># enough history to recognise releases we already reported </span> <span class="k">def</span> <span class="nf">build_request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">etag</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">User-Agent</span><span class="sh">"</span><span class="p">:</span> <span class="n">USER_AGENT</span><span class="p">,</span> <span class="sh">"</span><span class="s">Accept</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">application/vnd.github+json</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">X-GitHub-Api-Version</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">2026-03-10</span><span class="sh">"</span><span class="p">}</span> <span class="k">if</span> <span class="n">etag</span><span class="p">:</span> <span class="n">headers</span><span class="p">[</span><span class="sh">"</span><span class="s">If-None-Match</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="n">etag</span> <span class="k">if</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">GITHUB_TOKEN</span><span class="sh">"</span><span class="p">):</span> <span class="n">headers</span><span class="p">[</span><span class="sh">"</span><span class="s">Authorization</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="sh">"</span><span class="s">Bearer </span><span class="sh">"</span> <span class="o">+</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">[</span><span class="sh">"</span><span class="s">GITHUB_TOKEN</span><span class="sh">"</span><span class="p">]</span> <span class="k">return</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nc">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span> <span class="k">def</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">etag</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Return (status, headers, JSON body or None). A 304 is a status here, not an exception.</span><span class="sh">"""</span> <span class="k">try</span><span class="p">:</span> <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nf">urlopen</span><span class="p">(</span><span class="nf">build_request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">etag</span><span class="p">),</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">30</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span> <span class="k">return</span> <span class="n">response</span><span class="p">.</span><span class="n">status</span><span class="p">,</span> <span class="n">response</span><span class="p">.</span><span class="n">headers</span><span class="p">,</span> <span class="n">json</span><span class="p">.</span><span class="nf">loads</span><span class="p">(</span><span class="n">response</span><span class="p">.</span><span class="nf">read</span><span class="p">())</span> <span class="k">except</span> <span class="n">urllib</span><span class="p">.</span><span class="n">error</span><span class="p">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="c1"># urllib raises for 304 and for every 4xx/5xx </span> <span class="k">return</span> <span class="n">err</span><span class="p">.</span><span class="n">code</span><span class="p">,</span> <span class="n">err</span><span class="p">.</span><span class="n">headers</span><span class="p">,</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">summary</span><span class="p">(</span><span class="n">release</span><span class="p">):</span> <span class="k">return</span> <span class="p">{</span><span class="n">key</span><span class="p">:</span> <span class="n">release</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">tag_name</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">published_at</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">prerelease</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">html_url</span><span class="sh">"</span><span class="p">)}</span> <span class="k">def</span> <span class="nf">check</span><span class="p">(</span><span class="n">repo</span><span class="p">,</span> <span class="n">entry</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="n">fetch</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Check one repo against its saved state. Returns (new state entry, report).</span><span class="sh">"""</span> <span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">body</span> <span class="o">=</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">API</span><span class="p">.</span><span class="nf">format</span><span class="p">(</span><span class="n">repo</span><span class="p">),</span> <span class="n">entry</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">))</span> <span class="n">report</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">repo</span><span class="sh">"</span><span class="p">:</span> <span class="n">repo</span><span class="p">,</span> <span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">:</span> <span class="n">status</span><span class="p">,</span> <span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">:</span> <span class="p">[],</span> <span class="sh">"</span><span class="s">remaining</span><span class="sh">"</span><span class="p">:</span> <span class="n">headers</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">x-ratelimit-remaining</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="n">headers</span> <span class="k">else</span> <span class="bp">None</span><span class="p">}</span> <span class="k">if</span> <span class="n">status</span> <span class="o">!=</span> <span class="mi">200</span><span class="p">:</span> <span class="c1"># 304: nothing changed. Anything else: keep the old state untouched. </span> <span class="k">return</span> <span class="n">entry</span><span class="p">,</span> <span class="n">report</span> <span class="n">releases</span> <span class="o">=</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">body</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">r</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">draft</span><span class="sh">"</span><span class="p">)]</span> <span class="k">if</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span> <span class="ow">in</span> <span class="n">entry</span><span class="p">:</span> <span class="n">seen</span> <span class="o">=</span> <span class="nf">set</span><span class="p">(</span><span class="n">entry</span><span class="p">[</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">])</span> <span class="n">fresh</span> <span class="o">=</span> <span class="p">[</span><span class="n">r</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">releases</span> <span class="k">if</span> <span class="n">r</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">seen</span><span class="p">]</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nf">summary</span><span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="nf">sorted</span><span class="p">(</span><span class="n">fresh</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">r</span><span class="p">:</span> <span class="n">r</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">published_at</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="sh">""</span><span class="p">)]</span> <span class="k">else</span><span class="p">:</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">baseline</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="nf">len</span><span class="p">(</span><span class="n">releases</span><span class="p">)</span> <span class="c1"># first run: remember everything, alert on nothing </span> <span class="n">current</span> <span class="o">=</span> <span class="p">[</span><span class="n">r</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">releases</span><span class="p">]</span> <span class="n">older</span> <span class="o">=</span> <span class="p">[</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">entry</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">,</span> <span class="p">[])</span> <span class="k">if</span> <span class="n">i</span> <span class="ow">not</span> <span class="ow">in</span> <span class="nf">set</span><span class="p">(</span><span class="n">current</span><span class="p">)]</span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">:</span> <span class="n">headers</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">ETag</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="p">(</span><span class="n">current</span> <span class="o">+</span> <span class="n">older</span><span class="p">)[:</span><span class="n">KEEP_IDS</span><span class="p">]},</span> <span class="n">report</span> <span class="k">def</span> <span class="nf">describe</span><span class="p">(</span><span class="n">report</span><span class="p">):</span> <span class="n">status</span> <span class="o">=</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">]</span> <span class="k">if</span> <span class="n">status</span> <span class="o">==</span> <span class="mi">304</span><span class="p">:</span> <span class="n">what</span> <span class="o">=</span> <span class="sh">"</span><span class="s">not modified</span><span class="sh">"</span> <span class="k">elif</span> <span class="sh">"</span><span class="s">baseline</span><span class="sh">"</span> <span class="ow">in</span> <span class="n">report</span><span class="p">:</span> <span class="n">what</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="s">baseline: </span><span class="si">{</span><span class="n">report</span><span class="p">[</span><span class="sh">'</span><span class="s">baseline</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="s"> releases saved, nothing reported</span><span class="sh">"</span> <span class="k">elif</span> <span class="n">status</span> <span class="o">==</span> <span class="mi">200</span><span class="p">:</span> <span class="n">what</span> <span class="o">=</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">report</span><span class="p">[</span><span class="sh">'</span><span class="s">new</span><span class="sh">'</span><span class="p">])</span><span class="si">}</span><span class="s"> new release(s)</span><span class="sh">"</span> <span class="k">else</span><span class="p">:</span> <span class="n">what</span> <span class="o">=</span> <span class="sh">"</span><span class="s">request failed; state kept</span><span class="sh">"</span> <span class="k">return</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">report</span><span class="p">[</span><span class="sh">'</span><span class="s">repo</span><span class="sh">'</span><span class="p">]</span><span class="si">:</span><span class="o">&lt;</span><span class="mi">20</span><span class="si">}</span><span class="s"> HTTP </span><span class="si">{</span><span class="n">status</span><span class="si">}</span><span class="s"> </span><span class="si">{</span><span class="n">what</span><span class="si">}</span><span class="s"> (rate limit left: </span><span class="si">{</span><span class="n">report</span><span class="p">[</span><span class="sh">'</span><span class="s">remaining</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="s">)</span><span class="sh">"</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">repo_file</span><span class="p">,</span> <span class="n">state_file</span><span class="p">):</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">repo_file</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">repos</span> <span class="o">=</span> <span class="p">[</span><span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">handle</span> <span class="k">if</span> <span class="n">line</span><span class="p">.</span><span class="nf">strip</span><span class="p">()</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">line</span><span class="p">.</span><span class="nf">startswith</span><span class="p">(</span><span class="sh">"</span><span class="s">#</span><span class="sh">"</span><span class="p">)]</span> <span class="k">try</span><span class="p">:</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">state_file</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">state</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">handle</span><span class="p">)</span> <span class="k">except</span> <span class="nb">FileNotFoundError</span><span class="p">:</span> <span class="n">state</span> <span class="o">=</span> <span class="p">{}</span> <span class="nf">print</span><span class="p">(</span><span class="sh">"</span><span class="s">run at</span><span class="sh">"</span><span class="p">,</span> <span class="n">datetime</span><span class="p">.</span><span class="nf">now</span><span class="p">(</span><span class="n">timezone</span><span class="p">.</span><span class="n">utc</span><span class="p">).</span><span class="nf">strftime</span><span class="p">(</span><span class="sh">"</span><span class="s">%Y-%m-%dT%H:%M:%SZ</span><span class="sh">"</span><span class="p">))</span> <span class="k">for</span> <span class="n">number</span><span class="p">,</span> <span class="n">repo</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="n">repos</span><span class="p">):</span> <span class="k">if</span> <span class="n">number</span><span class="p">:</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># serial requests, one per second </span> <span class="n">state</span><span class="p">[</span><span class="n">repo</span><span class="p">],</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="n">repo</span><span class="p">,</span> <span class="n">state</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="n">repo</span><span class="p">,</span> <span class="p">{}))</span> <span class="nf">print</span><span class="p">(</span><span class="nf">describe</span><span class="p">(</span><span class="n">report</span><span class="p">))</span> <span class="k">for</span> <span class="n">release</span> <span class="ow">in</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">]:</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s"> NEW </span><span class="si">{</span><span class="n">release</span><span class="p">[</span><span class="sh">'</span><span class="s">tag_name</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="s"> published </span><span class="si">{</span><span class="n">release</span><span class="p">[</span><span class="sh">'</span><span class="s">published_at</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="sh">"</span> <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="sh">'</span><span class="s"> (pre-release)</span><span class="sh">'</span> <span class="k">if</span> <span class="n">release</span><span class="p">[</span><span class="sh">'</span><span class="s">prerelease</span><span class="sh">'</span><span class="p">]</span> <span class="k">else</span> <span class="sh">''</span><span class="si">}</span><span class="s"> </span><span class="si">{</span><span class="n">release</span><span class="p">[</span><span class="sh">'</span><span class="s">html_url</span><span class="sh">'</span><span class="p">]</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">]</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">403</span><span class="p">,</span> <span class="mi">429</span><span class="p">):</span> <span class="c1"># primary or secondary rate limit: stop, do not push on </span> <span class="nf">print</span><span class="p">(</span><span class="sh">"</span><span class="s">rate limited; stopping this run (state for earlier repos is still saved)</span><span class="sh">"</span><span class="p">)</span> <span class="k">break</span> <span class="n">temporary</span> <span class="o">=</span> <span class="n">state_file</span> <span class="o">+</span> <span class="sh">"</span><span class="s">.tmp</span><span class="sh">"</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">temporary</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">handle</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="nf">dump</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">handle</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="n">os</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="n">temporary</span><span class="p">,</span> <span class="n">state_file</span><span class="p">)</span> <span class="c1"># never leave a half-written state file </span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="sh">"</span><span class="s">__main__</span><span class="sh">"</span><span class="p">:</span> <span class="k">if</span> <span class="nf">len</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">3</span><span class="p">:</span> <span class="n">sys</span><span class="p">.</span><span class="nf">exit</span><span class="p">(</span><span class="n">__doc__</span><span class="p">)</span> <span class="nf">main</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">2</span><span class="p">])</span> </code></pre> </div> <p></p> <p></p> test_release_watch.py (90 lines) <br> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="kn">import</span> <span class="n">os</span> <span class="kn">import</span> <span class="n">unittest</span> <span class="kn">from</span> <span class="n">unittest</span> <span class="kn">import</span> <span class="n">mock</span> <span class="kn">from</span> <span class="n">release_watch</span> <span class="kn">import</span> <span class="n">KEEP_IDS</span><span class="p">,</span> <span class="n">build_request</span><span class="p">,</span> <span class="n">check</span> <span class="k">def</span> <span class="nf">release</span><span class="p">(</span><span class="n">rid</span><span class="p">,</span> <span class="n">tag</span><span class="p">,</span> <span class="n">published</span><span class="p">,</span> <span class="n">draft</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">:</span> <span class="n">rid</span><span class="p">,</span> <span class="sh">"</span><span class="s">tag_name</span><span class="sh">"</span><span class="p">:</span> <span class="n">tag</span><span class="p">,</span> <span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">:</span> <span class="n">tag</span><span class="p">,</span> <span class="sh">"</span><span class="s">published_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">published</span><span class="p">,</span> <span class="sh">"</span><span class="s">prerelease</span><span class="sh">"</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span> <span class="sh">"</span><span class="s">draft</span><span class="sh">"</span><span class="p">:</span> <span class="n">draft</span><span class="p">,</span> <span class="sh">"</span><span class="s">html_url</span><span class="sh">"</span><span class="p">:</span> <span class="sa">f</span><span class="sh">"</span><span class="s">https://example/</span><span class="si">{</span><span class="n">tag</span><span class="si">}</span><span class="sh">"</span><span class="p">}</span> <span class="k">class</span> <span class="nc">FakeApi</span><span class="p">:</span> <span class="sh">"""</span><span class="s">Stands in for fetch(): answers 304 when the caller sends the current ETag.</span><span class="sh">"""</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">releases</span><span class="p">,</span> <span class="n">etag</span><span class="o">=</span><span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">v1</span><span class="sh">"'</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="mi">200</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="n">releases</span><span class="p">,</span> <span class="n">self</span><span class="p">.</span><span class="n">etag</span><span class="p">,</span> <span class="n">self</span><span class="p">.</span><span class="n">status</span><span class="p">,</span> <span class="n">self</span><span class="p">.</span><span class="n">calls</span> <span class="o">=</span> <span class="n">releases</span><span class="p">,</span> <span class="n">etag</span><span class="p">,</span> <span class="n">status</span><span class="p">,</span> <span class="p">[]</span> <span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="n">self</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">etag</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="n">calls</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">etag</span><span class="p">)</span> <span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">ETag</span><span class="sh">"</span><span class="p">:</span> <span class="n">self</span><span class="p">.</span><span class="n">etag</span><span class="p">,</span> <span class="sh">"</span><span class="s">x-ratelimit-remaining</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">59</span><span class="sh">"</span><span class="p">}</span> <span class="k">if</span> <span class="n">self</span><span class="p">.</span><span class="n">status</span> <span class="o">!=</span> <span class="mi">200</span><span class="p">:</span> <span class="k">return</span> <span class="n">self</span><span class="p">.</span><span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="bp">None</span> <span class="k">if</span> <span class="n">etag</span> <span class="o">==</span> <span class="n">self</span><span class="p">.</span><span class="n">etag</span><span class="p">:</span> <span class="k">return</span> <span class="mi">304</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="bp">None</span> <span class="k">return</span> <span class="mi">200</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="nf">list</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">releases</span><span class="p">)</span> <span class="k">class</span> <span class="nc">Check</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_first_run_is_a_silent_baseline</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">api</span> <span class="o">=</span> <span class="nc">FakeApi</span><span class="p">([</span><span class="nf">release</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="sh">"</span><span class="s">v2</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-02-01T00:00:00Z</span><span class="sh">"</span><span class="p">),</span> <span class="nf">release</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="sh">"</span><span class="s">v1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-01-01T00:00:00Z</span><span class="sh">"</span><span class="p">)])</span> <span class="n">entry</span><span class="p">,</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="p">{},</span> <span class="n">fetch</span><span class="o">=</span><span class="n">api</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">baseline</span><span class="sh">"</span><span class="p">],</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="p">[]))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">entry</span><span class="p">,</span> <span class="p">{</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">:</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">v1</span><span class="sh">"'</span><span class="p">,</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">]})</span> <span class="k">def</span> <span class="nf">test_unchanged_repo_returns_304_and_keeps_state</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">api</span> <span class="o">=</span> <span class="nc">FakeApi</span><span class="p">([</span><span class="nf">release</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="sh">"</span><span class="s">v1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-01-01T00:00:00Z</span><span class="sh">"</span><span class="p">)])</span> <span class="n">entry</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="p">{},</span> <span class="n">fetch</span><span class="o">=</span><span class="n">api</span><span class="p">)</span> <span class="n">again</span><span class="p">,</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="n">entry</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="n">api</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">],</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">],</span> <span class="n">again</span><span class="p">),</span> <span class="p">(</span><span class="mi">304</span><span class="p">,</span> <span class="p">[],</span> <span class="n">entry</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">api</span><span class="p">.</span><span class="n">calls</span><span class="p">,</span> <span class="p">[</span><span class="bp">None</span><span class="p">,</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">v1</span><span class="sh">"'</span><span class="p">])</span> <span class="k">def</span> <span class="nf">test_new_releases_reported_oldest_first</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">entry</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">:</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">old</span><span class="sh">"'</span><span class="p">,</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]}</span> <span class="n">api</span> <span class="o">=</span> <span class="nc">FakeApi</span><span class="p">([</span><span class="nf">release</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="sh">"</span><span class="s">v3</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-03-01T00:00:00Z</span><span class="sh">"</span><span class="p">),</span> <span class="nf">release</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="sh">"</span><span class="s">v2</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-02-01T00:00:00Z</span><span class="sh">"</span><span class="p">),</span> <span class="nf">release</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="sh">"</span><span class="s">v1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-01-01T00:00:00Z</span><span class="sh">"</span><span class="p">)],</span> <span class="n">etag</span><span class="o">=</span><span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">new</span><span class="sh">"'</span><span class="p">)</span> <span class="n">entry</span><span class="p">,</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="n">entry</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="n">api</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">([</span><span class="n">r</span><span class="p">[</span><span class="sh">"</span><span class="s">tag_name</span><span class="sh">"</span><span class="p">]</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">]],</span> <span class="p">[</span><span class="sh">"</span><span class="s">v2</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">v3</span><span class="sh">"</span><span class="p">])</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">entry</span><span class="p">,</span> <span class="p">{</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">:</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">new</span><span class="sh">"'</span><span class="p">,</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">]})</span> <span class="k">def</span> <span class="nf">test_drafts_are_ignored</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">api</span> <span class="o">=</span> <span class="nc">FakeApi</span><span class="p">([</span><span class="nf">release</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="sh">"</span><span class="s">v2</span><span class="sh">"</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="n">draft</span><span class="o">=</span><span class="bp">True</span><span class="p">),</span> <span class="nf">release</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="sh">"</span><span class="s">v1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-01-01T00:00:00Z</span><span class="sh">"</span><span class="p">)])</span> <span class="n">_</span><span class="p">,</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="p">{</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]},</span> <span class="n">fetch</span><span class="o">=</span><span class="n">api</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">],</span> <span class="p">[])</span> <span class="k">def</span> <span class="nf">test_repo_with_no_releases_becomes_a_baseline_of_zero</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">entry</span><span class="p">,</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="p">{},</span> <span class="n">fetch</span><span class="o">=</span><span class="nc">FakeApi</span><span class="p">([]))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">baseline</span><span class="sh">"</span><span class="p">],</span> <span class="n">entry</span><span class="p">[</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="p">[]))</span> <span class="k">def</span> <span class="nf">test_errors_keep_previous_state</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">previous</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">etag</span><span class="sh">"</span><span class="p">:</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">x</span><span class="sh">"'</span><span class="p">,</span> <span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]}</span> <span class="k">for</span> <span class="n">status</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">404</span><span class="p">,</span> <span class="mi">403</span><span class="p">,</span> <span class="mi">429</span><span class="p">,</span> <span class="mi">502</span><span class="p">):</span> <span class="n">entry</span><span class="p">,</span> <span class="n">report</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="n">previous</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="nc">FakeApi</span><span class="p">([],</span> <span class="n">status</span><span class="o">=</span><span class="n">status</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">((</span><span class="n">entry</span><span class="p">,</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">new</span><span class="sh">"</span><span class="p">],</span> <span class="n">report</span><span class="p">[</span><span class="sh">"</span><span class="s">status</span><span class="sh">"</span><span class="p">]),</span> <span class="p">(</span><span class="n">previous</span><span class="p">,</span> <span class="p">[],</span> <span class="n">status</span><span class="p">))</span> <span class="k">def</span> <span class="nf">test_seen_ids_are_capped</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="n">entry</span> <span class="o">=</span> <span class="p">{</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">:</span> <span class="nf">list</span><span class="p">(</span><span class="nf">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span> <span class="mi">1000</span> <span class="o">+</span> <span class="n">KEEP_IDS</span><span class="p">))}</span> <span class="n">new_entry</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="nf">check</span><span class="p">(</span><span class="sh">"</span><span class="s">o/r</span><span class="sh">"</span><span class="p">,</span> <span class="n">entry</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="nc">FakeApi</span><span class="p">([</span><span class="nf">release</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="sh">"</span><span class="s">v1</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">2026-01-01T00:00:00Z</span><span class="sh">"</span><span class="p">)]))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">len</span><span class="p">(</span><span class="n">new_entry</span><span class="p">[</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">]),</span> <span class="n">KEEP_IDS</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">new_entry</span><span class="p">[</span><span class="sh">"</span><span class="s">seen</span><span class="sh">"</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span> <span class="mi">1</span><span class="p">)</span> <span class="k">class</span> <span class="nc">Request</span><span class="p">(</span><span class="n">unittest</span><span class="p">.</span><span class="n">TestCase</span><span class="p">):</span> <span class="k">def</span> <span class="nf">test_conditional_header_only_with_etag</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="k">with</span> <span class="n">mock</span><span class="p">.</span><span class="n">patch</span><span class="p">.</span><span class="nf">dict</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">,</span> <span class="p">{},</span> <span class="n">clear</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span> <span class="n">plain</span> <span class="o">=</span> <span class="nf">build_request</span><span class="p">(</span><span class="sh">"</span><span class="s">https://api.github.com/x</span><span class="sh">"</span><span class="p">)</span> <span class="n">conditional</span> <span class="o">=</span> <span class="nf">build_request</span><span class="p">(</span><span class="sh">"</span><span class="s">https://api.github.com/x</span><span class="sh">"</span><span class="p">,</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">abc</span><span class="sh">"'</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertIsNone</span><span class="p">(</span><span class="n">plain</span><span class="p">.</span><span class="nf">get_header</span><span class="p">(</span><span class="sh">"</span><span class="s">If-none-match</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">conditional</span><span class="p">.</span><span class="nf">get_header</span><span class="p">(</span><span class="sh">"</span><span class="s">If-none-match</span><span class="sh">"</span><span class="p">),</span> <span class="sh">'</span><span class="s">W/</span><span class="sh">"</span><span class="s">abc</span><span class="sh">"'</span><span class="p">)</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertIsNone</span><span class="p">(</span><span class="n">plain</span><span class="p">.</span><span class="nf">get_header</span><span class="p">(</span><span class="sh">"</span><span class="s">Authorization</span><span class="sh">"</span><span class="p">))</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="n">plain</span><span class="p">.</span><span class="nf">get_header</span><span class="p">(</span><span class="sh">"</span><span class="s">X-github-api-version</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">2026-03-10</span><span class="sh">"</span><span class="p">)</span> <span class="k">def</span> <span class="nf">test_token_is_sent_only_when_set</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="k">with</span> <span class="n">mock</span><span class="p">.</span><span class="n">patch</span><span class="p">.</span><span class="nf">dict</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">,</span> <span class="p">{</span><span class="sh">"</span><span class="s">GITHUB_TOKEN</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">test-value</span><span class="sh">"</span><span class="p">},</span> <span class="n">clear</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span> <span class="n">self</span><span class="p">.</span><span class="nf">assertEqual</span><span class="p">(</span><span class="nf">build_request</span><span class="p">(</span><span class="sh">"</span><span class="s">https://api.github.com/x</span><span class="sh">"</span><span class="p">).</span><span class="nf">get_header</span><span class="p">(</span><span class="sh">"</span><span class="s">Authorization</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">Bearer test-value</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="sh">"</span><span class="s">__main__</span><span class="sh">"</span><span class="p">:</span> <span class="n">unittest</span><span class="p">.</span><span class="nf">main</span><span class="p">()</span> </code></pre> </div> <p></p> <p>Run the tests with <code>python -m unittest -v test_release_watch</code>.</p> <p><em>This article and its code were drafted by an AI assistant at the account owner's request. The code was run against the live GitHub API without a token on 27 September 2026, from 13:13:50 to 13:14:09 UTC (the earlier ETag comparison is from 13:03:03 UTC), and the tests passed the same day.</em></p> python github api tutorial Abdulwahab Sun, 27 Sep 2026 15:01:03 +0000 https://dev.to/abdulwhab95/normalizing-public-job-board-data-with-python-3pno https://dev.to/abdulwhab95/normalizing-public-job-board-data-with-python-3pno <p>Many companies host their careers page on Greenhouse, Lever or Ashby, and each of these vendors documents a public, read-only JSON API for those boards. You don't need a key or a login. The catch is that the three APIs describe a job in three different ways, so putting several boards in one spreadsheet means mapping three shapes into one.</p> <p>This post covers:</p> <ol> <li>how the three APIs differ,</li> <li>a small standard-library Python script that normalizes title, location and apply link (plus a few optional fields),</li> <li>real output from a run on 27 September 2026 (UTC),</li> <li>JSON and CSV export,</li> <li>missing values, source limits and common mistakes.</li> </ol> <h2> How the three APIs differ </h2> <p>Each board is identified by a slug: the board name in the vendor URL, such as <code>palantir</code> in <code>jobs.lever.co/palantir</code> or <code>ramp</code> in <code>jobs.ashbyhq.com/ramp</code>. Greenhouse's docs call it the <code>board_token</code>.</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th></th> <th>Greenhouse</th> <th>Lever</th> <th>Ashby</th> </tr> </thead> <tbody> <tr> <td>List endpoint</td> <td><code>/v1/boards/{slug}/jobs</code></td> <td><code>/v0/postings/{slug}?mode=json</code></td> <td><code>/posting-api/job-board/{slug}</code></td> </tr> <tr> <td>Response</td> <td><code>{"jobs": [...], "meta": ...}</code></td> <td>bare JSON array</td> <td><code>{"apiVersion": ..., "jobs": [...]}</code></td> </tr> <tr> <td>Title</td> <td><code>title</code></td> <td><code>text</code></td> <td><code>title</code></td> </tr> <tr> <td>Location</td> <td><code>location.name</code></td> <td><code>categories.location</code></td> <td><code>location</code></td> </tr> <tr> <td>Apply link</td> <td> <code>absolute_url</code> (posting page)</td> <td> <code>applyUrl</code> (form), <code>hostedUrl</code> (page)</td> <td> <code>applyUrl</code> (form), <code>jobUrl</code> (page)</td> </tr> <tr> <td>Date</td> <td> <code>first_published</code>, ISO with offset</td> <td> <code>createdAt</code>, epoch milliseconds</td> <td> <code>publishedAt</code>, ISO</td> </tr> <tr> <td>Workplace</td> <td>not provided</td> <td><code>workplaceType</code></td> <td> <code>workplaceType</code>, <code>isRemote</code> </td> </tr> <tr> <td>Department</td> <td>not in the list</td> <td> <code>categories.department</code> or <code>team</code> </td> <td><code>department</code></td> </tr> <tr> <td>Salary</td> <td>not in the list; per job via <code>/jobs/{id}?pay_transparency=true</code> (<code>pay_input_ranges</code>)</td> <td>optional <code>salaryRange</code> </td> <td>with <code>?includeCompensation=true</code> </td> </tr> </tbody> </table></div> <p>A few details decide how you call each one.</p> <p><strong>Greenhouse</strong> keeps the list light: title, a free-text location, a URL and dates. The list has no department names. You can get them two ways: <code>?content=true</code> swaps the light list for a much heavier one that carries every job description, or a separate <code>/departments</code> call nests the jobs under their departments. The script below leaves <code>department</code> empty for Greenhouse; if you need it, call <code>/departments</code> once per board and join on the job id. Also note that <code>absolute_url</code> can point at the company's own careers site rather than greenhouse.io.</p> <p><strong>Lever</strong> returns a bare array, and every posting carries its full description in several forms. In this run, 321 postings came to 6.19 MB. The documented <code>limit</code> and <code>skip</code> parameters let you page or probe a board without downloading all of it. Boards on Lever's EU instance live on <code>api.eu.lever.co</code>, not <code>api.lever.co</code>. The docs list <code>workplaceType</code> values as <code>on-site</code>, <code>remote</code>, <code>hybrid</code> or <code>unspecified</code>, but the board used here returned <code>onsite</code> without the hyphen, so the script accepts both. <code>createdAt</code> isn't in the README's field table, but it is in the responses as epoch milliseconds.</p> <p><strong>Ashby</strong> always includes the HTML and plain-text description. With <code>?includeCompensation=true</code> it adds a <code>compensation</code> object with per-tier details and a <code>summaryComponents</code> list. That list isn't ordered by type: on the first Ramp posting in this run, the equity component came before the salary one, so you need to search for <code>compensationType == "Salary"</code> and not take the first item. Ashby also has <code>isListed</code> (skip <code>false</code>) and two workplace signals that can disagree. The same Ramp posting had <code>workplaceType: "Hybrid"</code> and <code>isRemote: true</code>. The script keeps <code>workplaceType</code>.</p> <h2> The script </h2> <p>Every posting becomes one flat row with the same 13 keys: <code>source</code>, <code>board</code>, <code>job_id</code>, <code>title</code>, <code>location</code>, <code>workplace</code>, <code>department</code>, <code>apply_url</code>, <code>posted_at</code>, and four <code>salary_*</code> fields. Anything a vendor doesn't provide stays <code>None</code>.</p> <p>The endpoints:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="n">ENDPOINTS</span> <span class="o">=</span> <span class="p">{</span> <span class="sh">"</span><span class="s">greenhouse</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://boards-api.greenhouse.io/v1/boards/{}/jobs</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">lever</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://api.lever.co/v0/postings/{}?mode=json</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">ashby</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://api.ashbyhq.com/posting-api/job-board/{}?includeCompensation=true</span><span class="sh">"</span><span class="p">,</span> <span class="p">}</span> </code></pre> </div> <p>The fetch step keeps three outcomes apart. A <strong>404</strong> means no board with that slug exists on that vendor. A <strong>200 with an empty list</strong> means the board exists but has nothing open. A <strong>429, 5xx or network error</strong> (including a connection that drops mid-download) means you don't know yet. Retry a few times with a growing wait, honour <code>Retry-After</code>, then give up and say the status is unknown. Never record that case as "company not found".<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">tries</span><span class="o">=</span><span class="mi">4</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Return (JSON, size in bytes); (None, 0) on 404. Backs off on 429/5xx/network errors.</span><span class="sh">"""</span> <span class="k">for</span> <span class="n">attempt</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">tries</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># never more than one request per second </span> <span class="k">try</span><span class="p">:</span> <span class="n">request</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nc">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="sh">"</span><span class="s">User-Agent</span><span class="sh">"</span><span class="p">:</span> <span class="n">USER_AGENT</span><span class="p">})</span> <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nf">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span> <span class="n">body</span> <span class="o">=</span> <span class="n">response</span><span class="p">.</span><span class="nf">read</span><span class="p">()</span> <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="nf">loads</span><span class="p">(</span><span class="n">body</span><span class="p">),</span> <span class="nf">len</span><span class="p">(</span><span class="n">body</span><span class="p">)</span> <span class="nf">except </span><span class="p">(</span><span class="nb">OSError</span><span class="p">,</span> <span class="n">http</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">HTTPException</span><span class="p">)</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="c1"># HTTP errors, timeouts, dropped reads </span> <span class="n">code</span> <span class="o">=</span> <span class="nf">getattr</span><span class="p">(</span><span class="n">err</span><span class="p">,</span> <span class="sh">"</span><span class="s">code</span><span class="sh">"</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span> <span class="k">if</span> <span class="n">code</span> <span class="o">==</span> <span class="mi">404</span><span class="p">:</span> <span class="k">return</span> <span class="bp">None</span><span class="p">,</span> <span class="mi">0</span> <span class="nf">if </span><span class="p">(</span><span class="n">code</span> <span class="ow">and</span> <span class="n">code</span> <span class="o">!=</span> <span class="mi">429</span> <span class="ow">and</span> <span class="n">code</span> <span class="o">&lt;</span> <span class="mi">500</span><span class="p">)</span> <span class="ow">or</span> <span class="n">attempt</span> <span class="o">==</span> <span class="n">tries</span><span class="p">:</span> <span class="k">raise</span> <span class="n">header</span> <span class="o">=</span> <span class="n">err</span><span class="p">.</span><span class="n">headers</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">Retry-After</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="k">if</span> <span class="n">code</span> <span class="k">else</span> <span class="sh">""</span> <span class="n">wait</span> <span class="o">=</span> <span class="nf">min</span><span class="p">(</span><span class="nf">int</span><span class="p">(</span><span class="n">header</span><span class="p">),</span> <span class="mi">60</span><span class="p">)</span> <span class="k">if</span> <span class="n">header</span><span class="p">.</span><span class="nf">isdigit</span><span class="p">()</span> <span class="k">else</span> <span class="mi">2</span> <span class="o">**</span> <span class="n">attempt</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s"> retry </span><span class="si">{</span><span class="n">attempt</span><span class="si">}</span><span class="s">/</span><span class="si">{</span><span class="n">tries</span> <span class="o">-</span> <span class="mi">1</span><span class="si">}</span><span class="s"> in </span><span class="si">{</span><span class="n">wait</span><span class="si">}</span><span class="s">s: </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="sh">"</span><span class="p">,</span> <span class="nb">file</span><span class="o">=</span><span class="n">sys</span><span class="p">.</span><span class="n">stderr</span><span class="p">)</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="n">wait</span><span class="p">)</span> </code></pre> </div> <p>The mapping itself is one function with a branch per vendor:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="k">def</span> <span class="nf">normalize</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">job</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Map one raw posting to the shared field names. Anything a vendor lacks stays None.</span><span class="sh">"""</span> <span class="k">if</span> <span class="n">source</span> <span class="o">==</span> <span class="sh">"</span><span class="s">greenhouse</span><span class="sh">"</span><span class="p">:</span> <span class="c1"># the /jobs list has no workplace, department or salary </span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">job_id</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">:</span> <span class="p">(</span><span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{}).</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">apply_url</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">absolute_url</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">posted_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">first_published</span><span class="sh">"</span><span class="p">)}</span> <span class="k">if</span> <span class="n">source</span> <span class="o">==</span> <span class="sh">"</span><span class="s">lever</span><span class="sh">"</span><span class="p">:</span> <span class="n">cats</span><span class="p">,</span> <span class="n">pay</span> <span class="o">=</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">categories</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{},</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">salaryRange</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{}</span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">job_id</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">:</span> <span class="n">cats</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">workplaceType</span><span class="sh">"</span><span class="p">),</span> <span class="c1"># docs: on-site/remote/hybrid/unspecified </span> <span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">:</span> <span class="n">cats</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="n">cats</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">team</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">apply_url</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">applyUrl</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">hostedUrl</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">posted_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">createdAt</span><span class="sh">"</span><span class="p">),</span> <span class="c1"># epoch milliseconds </span> <span class="sh">"</span><span class="s">salary_min</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">min</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_max</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">max</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_currency</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_interval</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">interval</span><span class="sh">"</span><span class="p">)}</span> <span class="n">parts</span> <span class="o">=</span> <span class="p">(</span><span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">compensation</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{}).</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">summaryComponents</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]</span> <span class="c1"># Ashby </span> <span class="n">pay</span> <span class="o">=</span> <span class="nf">next</span><span class="p">((</span><span class="n">p</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">parts</span> <span class="k">if</span> <span class="n">p</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">compensationType</span><span class="sh">"</span><span class="p">)</span> <span class="o">==</span> <span class="sh">"</span><span class="s">Salary</span><span class="sh">"</span><span class="p">),</span> <span class="p">{})</span> <span class="c1"># any order </span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">job_id</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">workplaceType</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">apply_url</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">applyUrl</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">jobUrl</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">posted_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">publishedAt</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_min</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">minValue</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_max</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">maxValue</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_currency</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">currencyCode</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_interval</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">interval</span><span class="sh">"</span><span class="p">)}</span> </code></pre> </div> <p>After mapping, each row gets the same cleanup: whitespace is trimmed and collapsed, dates are converted to UTC, and the workplace value is reduced to <code>onsite</code>, <code>hybrid</code>, <code>remote</code> or <code>None</code>. The whitespace step matters because titles arrive with stray spaces. Ramp's board, for example, sends <code>" Security Engineer, Cloud"</code> with a leading space, which would otherwise break exact matching and de-duplication.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code> <span class="n">row</span><span class="p">.</span><span class="nf">update</span><span class="p">(</span><span class="n">job_id</span><span class="o">=</span><span class="nf">str</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">job_id</span><span class="sh">"</span><span class="p">]),</span> <span class="n">posted_at</span><span class="o">=</span><span class="nf">iso_utc</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">posted_at</span><span class="sh">"</span><span class="p">]),</span> <span class="o">**</span><span class="p">{</span><span class="n">key</span><span class="p">:</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">split</span><span class="p">())</span> <span class="ow">or</span> <span class="bp">None</span> <span class="c1"># trim, collapse whitespace </span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="nb">str</span><span class="p">)})</span> <span class="n">kind</span> <span class="o">=</span> <span class="nf">str</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">]).</span><span class="nf">lower</span><span class="p">().</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="s">-</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="c1"># "OnSite", "on-site", "onsite" </span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="n">kind</span> <span class="k">if</span> <span class="n">kind</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">onsite</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">hybrid</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">remote</span><span class="sh">"</span><span class="p">)</span> <span class="k">else</span> <span class="bp">None</span> </code></pre> </div> <p>The full script is 122 lines and needs Python 3.11 or newer, with no third-party packages:</p> <p></p> normalize_jobs.py (full script) <br> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="sh">"""</span><span class="s">Normalize public Greenhouse, Lever and Ashby job boards into one JSON and one CSV. Usage: python normalize_jobs.py greenhouse:stripe lever:palantir ashby:ramp Python 3.11+, standard library only, no API key (these are public job-board APIs). </span><span class="sh">"""</span> <span class="kn">import</span> <span class="n">csv</span> <span class="kn">import</span> <span class="n">http.client</span> <span class="kn">import</span> <span class="n">json</span> <span class="kn">import</span> <span class="n">sys</span> <span class="kn">import</span> <span class="n">time</span> <span class="kn">import</span> <span class="n">urllib.request</span> <span class="kn">from</span> <span class="n">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timezone</span> <span class="n">USER_AGENT</span> <span class="o">=</span> <span class="sh">"</span><span class="s">job-board-normalizer-example/1.0 (tutorial script; one request per board)</span><span class="sh">"</span> <span class="n">ENDPOINTS</span> <span class="o">=</span> <span class="p">{</span> <span class="sh">"</span><span class="s">greenhouse</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://boards-api.greenhouse.io/v1/boards/{}/jobs</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">lever</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://api.lever.co/v0/postings/{}?mode=json</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">ashby</span><span class="sh">"</span><span class="p">:</span> <span class="sh">"</span><span class="s">https://api.ashbyhq.com/posting-api/job-board/{}?includeCompensation=true</span><span class="sh">"</span><span class="p">,</span> <span class="p">}</span> <span class="n">FIELDS</span> <span class="o">=</span> <span class="p">(</span><span class="sh">"</span><span class="s">source board job_id title location workplace department apply_url posted_at </span><span class="sh">"</span> <span class="sh">"</span><span class="s">salary_min salary_max salary_currency salary_interval</span><span class="sh">"</span><span class="p">).</span><span class="nf">split</span><span class="p">()</span> <span class="n">UTC</span> <span class="o">=</span> <span class="sh">"</span><span class="s">%Y-%m-%dT%H:%M:%SZ</span><span class="sh">"</span> <span class="k">def</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">tries</span><span class="o">=</span><span class="mi">4</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Return (JSON, size in bytes); (None, 0) on 404. Backs off on 429/5xx/network errors.</span><span class="sh">"""</span> <span class="k">for</span> <span class="n">attempt</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">tries</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># never more than one request per second </span> <span class="k">try</span><span class="p">:</span> <span class="n">request</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nc">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="sh">"</span><span class="s">User-Agent</span><span class="sh">"</span><span class="p">:</span> <span class="n">USER_AGENT</span><span class="p">})</span> <span class="k">with</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="nf">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span> <span class="n">body</span> <span class="o">=</span> <span class="n">response</span><span class="p">.</span><span class="nf">read</span><span class="p">()</span> <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="nf">loads</span><span class="p">(</span><span class="n">body</span><span class="p">),</span> <span class="nf">len</span><span class="p">(</span><span class="n">body</span><span class="p">)</span> <span class="nf">except </span><span class="p">(</span><span class="nb">OSError</span><span class="p">,</span> <span class="n">http</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">HTTPException</span><span class="p">)</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="c1"># HTTP errors, timeouts, dropped reads </span> <span class="n">code</span> <span class="o">=</span> <span class="nf">getattr</span><span class="p">(</span><span class="n">err</span><span class="p">,</span> <span class="sh">"</span><span class="s">code</span><span class="sh">"</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span> <span class="k">if</span> <span class="n">code</span> <span class="o">==</span> <span class="mi">404</span><span class="p">:</span> <span class="k">return</span> <span class="bp">None</span><span class="p">,</span> <span class="mi">0</span> <span class="nf">if </span><span class="p">(</span><span class="n">code</span> <span class="ow">and</span> <span class="n">code</span> <span class="o">!=</span> <span class="mi">429</span> <span class="ow">and</span> <span class="n">code</span> <span class="o">&lt;</span> <span class="mi">500</span><span class="p">)</span> <span class="ow">or</span> <span class="n">attempt</span> <span class="o">==</span> <span class="n">tries</span><span class="p">:</span> <span class="k">raise</span> <span class="n">header</span> <span class="o">=</span> <span class="n">err</span><span class="p">.</span><span class="n">headers</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">Retry-After</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="k">if</span> <span class="n">code</span> <span class="k">else</span> <span class="sh">""</span> <span class="n">wait</span> <span class="o">=</span> <span class="nf">min</span><span class="p">(</span><span class="nf">int</span><span class="p">(</span><span class="n">header</span><span class="p">),</span> <span class="mi">60</span><span class="p">)</span> <span class="k">if</span> <span class="n">header</span><span class="p">.</span><span class="nf">isdigit</span><span class="p">()</span> <span class="k">else</span> <span class="mi">2</span> <span class="o">**</span> <span class="n">attempt</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s"> retry </span><span class="si">{</span><span class="n">attempt</span><span class="si">}</span><span class="s">/</span><span class="si">{</span><span class="n">tries</span> <span class="o">-</span> <span class="mi">1</span><span class="si">}</span><span class="s"> in </span><span class="si">{</span><span class="n">wait</span><span class="si">}</span><span class="s">s: </span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="sh">"</span><span class="p">,</span> <span class="nb">file</span><span class="o">=</span><span class="n">sys</span><span class="p">.</span><span class="n">stderr</span><span class="p">)</span> <span class="n">time</span><span class="p">.</span><span class="nf">sleep</span><span class="p">(</span><span class="n">wait</span><span class="p">)</span> <span class="k">def</span> <span class="nf">iso_utc</span><span class="p">(</span><span class="n">value</span><span class="p">):</span> <span class="sh">"""</span><span class="s">ISO 8601 with an offset, or epoch milliseconds (Lever) -&gt; </span><span class="sh">'</span><span class="s">YYYY-MM-DDTHH:MM:SSZ</span><span class="sh">'</span><span class="s">.</span><span class="sh">"""</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="nb">float</span><span class="p">)):</span> <span class="k">return</span> <span class="n">datetime</span><span class="p">.</span><span class="nf">fromtimestamp</span><span class="p">(</span><span class="n">value</span> <span class="o">/</span> <span class="mi">1000</span><span class="p">,</span> <span class="n">timezone</span><span class="p">.</span><span class="n">utc</span><span class="p">).</span><span class="nf">strftime</span><span class="p">(</span><span class="n">UTC</span><span class="p">)</span> <span class="k">return</span> <span class="n">datetime</span><span class="p">.</span><span class="nf">fromisoformat</span><span class="p">(</span><span class="n">value</span><span class="p">).</span><span class="nf">astimezone</span><span class="p">(</span><span class="n">timezone</span><span class="p">.</span><span class="n">utc</span><span class="p">).</span><span class="nf">strftime</span><span class="p">(</span><span class="n">UTC</span><span class="p">)</span> <span class="k">if</span> <span class="n">value</span> <span class="k">else</span> <span class="bp">None</span> <span class="k">def</span> <span class="nf">normalize</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">job</span><span class="p">):</span> <span class="sh">"""</span><span class="s">Map one raw posting to the shared field names. Anything a vendor lacks stays None.</span><span class="sh">"""</span> <span class="k">if</span> <span class="n">source</span> <span class="o">==</span> <span class="sh">"</span><span class="s">greenhouse</span><span class="sh">"</span><span class="p">:</span> <span class="c1"># the /jobs list has no workplace, department or salary </span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">job_id</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">:</span> <span class="p">(</span><span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{}).</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">name</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">apply_url</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">absolute_url</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">posted_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">first_published</span><span class="sh">"</span><span class="p">)}</span> <span class="k">if</span> <span class="n">source</span> <span class="o">==</span> <span class="sh">"</span><span class="s">lever</span><span class="sh">"</span><span class="p">:</span> <span class="n">cats</span><span class="p">,</span> <span class="n">pay</span> <span class="o">=</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">categories</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{},</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">salaryRange</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{}</span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">job_id</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">text</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">:</span> <span class="n">cats</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">workplaceType</span><span class="sh">"</span><span class="p">),</span> <span class="c1"># docs: on-site/remote/hybrid/unspecified </span> <span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">:</span> <span class="n">cats</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="n">cats</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">team</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">apply_url</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">applyUrl</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">hostedUrl</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">posted_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">createdAt</span><span class="sh">"</span><span class="p">),</span> <span class="c1"># epoch milliseconds </span> <span class="sh">"</span><span class="s">salary_min</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">min</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_max</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">max</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_currency</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">currency</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_interval</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">interval</span><span class="sh">"</span><span class="p">)}</span> <span class="n">parts</span> <span class="o">=</span> <span class="p">(</span><span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">compensation</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{}).</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">summaryComponents</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]</span> <span class="c1"># Ashby </span> <span class="n">pay</span> <span class="o">=</span> <span class="nf">next</span><span class="p">((</span><span class="n">p</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">parts</span> <span class="k">if</span> <span class="n">p</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">compensationType</span><span class="sh">"</span><span class="p">)</span> <span class="o">==</span> <span class="sh">"</span><span class="s">Salary</span><span class="sh">"</span><span class="p">),</span> <span class="p">{})</span> <span class="c1"># any order </span> <span class="k">return</span> <span class="p">{</span><span class="sh">"</span><span class="s">job_id</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">[</span><span class="sh">"</span><span class="s">id</span><span class="sh">"</span><span class="p">],</span> <span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">workplaceType</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">apply_url</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">applyUrl</span><span class="sh">"</span><span class="p">)</span> <span class="ow">or</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">jobUrl</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">posted_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">publishedAt</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_min</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">minValue</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_max</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">maxValue</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_currency</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">currencyCode</span><span class="sh">"</span><span class="p">),</span> <span class="sh">"</span><span class="s">salary_interval</span><span class="sh">"</span><span class="p">:</span> <span class="n">pay</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">interval</span><span class="sh">"</span><span class="p">)}</span> <span class="k">def</span> <span class="nf">collect</span><span class="p">(</span><span class="n">spec</span><span class="p">):</span> <span class="sh">"""'</span><span class="s">lever:palantir</span><span class="sh">'</span><span class="s"> -&gt; list of normalized rows. Problems print a message and return [].</span><span class="sh">"""</span> <span class="n">source</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">board</span> <span class="o">=</span> <span class="n">spec</span><span class="p">.</span><span class="nf">partition</span><span class="p">(</span><span class="sh">"</span><span class="s">:</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="n">source</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">ENDPOINTS</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">board</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="s">-</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">).</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="s">_</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">).</span><span class="nf">isalnum</span><span class="p">():</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">spec</span><span class="si">}</span><span class="s">: expected greenhouse:&lt;slug&gt;, lever:&lt;slug&gt; or ashby:&lt;slug&gt;</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="p">[]</span> <span class="k">try</span><span class="p">:</span> <span class="n">data</span><span class="p">,</span> <span class="n">size</span> <span class="o">=</span> <span class="nf">fetch</span><span class="p">(</span><span class="n">ENDPOINTS</span><span class="p">[</span><span class="n">source</span><span class="p">].</span><span class="nf">format</span><span class="p">(</span><span class="n">board</span><span class="p">))</span> <span class="nf">except </span><span class="p">(</span><span class="nb">OSError</span><span class="p">,</span> <span class="n">http</span><span class="p">.</span><span class="n">client</span><span class="p">.</span><span class="n">HTTPException</span><span class="p">,</span> <span class="nb">ValueError</span><span class="p">)</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span> <span class="c1"># ValueError: bad JSON </span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">spec</span><span class="si">}</span><span class="s">: request failed (</span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s">) - status unknown, try again later</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="p">[]</span> <span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">spec</span><span class="si">}</span><span class="s">: board not found (HTTP 404) - check the slug and the vendor</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="p">[]</span> <span class="n">jobs</span> <span class="o">=</span> <span class="p">[</span><span class="n">job</span> <span class="k">for</span> <span class="n">job</span> <span class="ow">in</span> <span class="p">(</span><span class="n">data</span> <span class="k">if</span> <span class="n">source</span> <span class="o">==</span> <span class="sh">"</span><span class="s">lever</span><span class="sh">"</span> <span class="k">else</span> <span class="n">data</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">jobs</span><span class="sh">"</span><span class="p">,</span> <span class="p">[]))</span> <span class="k">if</span> <span class="n">job</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="sh">"</span><span class="s">isListed</span><span class="sh">"</span><span class="p">,</span> <span class="bp">True</span><span class="p">)]</span> <span class="c1"># Ashby can mark a job as unlisted </span> <span class="k">if</span> <span class="ow">not</span> <span class="n">jobs</span><span class="p">:</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">spec</span><span class="si">}</span><span class="s">: board exists but has no open jobs right now</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="p">[]</span> <span class="n">rows</span> <span class="o">=</span> <span class="p">[]</span> <span class="k">for</span> <span class="n">job</span> <span class="ow">in</span> <span class="n">jobs</span><span class="p">:</span> <span class="n">row</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">.</span><span class="nf">fromkeys</span><span class="p">(</span><span class="n">FIELDS</span><span class="p">)</span> <span class="c1"># every row gets every column </span> <span class="n">row</span><span class="p">.</span><span class="nf">update</span><span class="p">(</span><span class="nf">normalize</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">job</span><span class="p">),</span> <span class="n">source</span><span class="o">=</span><span class="n">source</span><span class="p">,</span> <span class="n">board</span><span class="o">=</span><span class="n">board</span><span class="p">)</span> <span class="n">row</span><span class="p">.</span><span class="nf">update</span><span class="p">(</span><span class="n">job_id</span><span class="o">=</span><span class="nf">str</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">job_id</span><span class="sh">"</span><span class="p">]),</span> <span class="n">posted_at</span><span class="o">=</span><span class="nf">iso_utc</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">posted_at</span><span class="sh">"</span><span class="p">]),</span> <span class="o">**</span><span class="p">{</span><span class="n">key</span><span class="p">:</span> <span class="sh">"</span><span class="s"> </span><span class="sh">"</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="n">key</span><span class="p">].</span><span class="nf">split</span><span class="p">())</span> <span class="ow">or</span> <span class="bp">None</span> <span class="c1"># trim, collapse whitespace </span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">title</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">)</span> <span class="k">if</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="nb">str</span><span class="p">)})</span> <span class="n">kind</span> <span class="o">=</span> <span class="nf">str</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">]).</span><span class="nf">lower</span><span class="p">().</span><span class="nf">replace</span><span class="p">(</span><span class="sh">"</span><span class="s">-</span><span class="sh">"</span><span class="p">,</span> <span class="sh">""</span><span class="p">)</span> <span class="c1"># "OnSite", "on-site", "onsite" </span> <span class="n">row</span><span class="p">[</span><span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">]</span> <span class="o">=</span> <span class="n">kind</span> <span class="k">if</span> <span class="n">kind</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">onsite</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">hybrid</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">remote</span><span class="sh">"</span><span class="p">)</span> <span class="k">else</span> <span class="bp">None</span> <span class="n">rows</span><span class="p">.</span><span class="nf">append</span><span class="p">(</span><span class="n">row</span><span class="p">)</span> <span class="n">filled</span> <span class="o">=</span> <span class="p">{</span><span class="n">key</span><span class="p">:</span> <span class="nf">sum</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="p">(</span><span class="sh">"</span><span class="s">location</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">workplace</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">department</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">salary_min</span><span class="sh">"</span><span class="p">)}</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">spec</span><span class="si">}</span><span class="s">: </span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span><span class="si">}</span><span class="s"> jobs, </span><span class="si">{</span><span class="n">size</span> <span class="o">/</span> <span class="mf">1e6</span><span class="si">:</span><span class="p">.</span><span class="mi">2</span><span class="n">f</span><span class="si">}</span><span class="s"> MB, non-empty: </span><span class="si">{</span><span class="n">filled</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="k">return</span> <span class="n">rows</span> <span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="sh">"</span><span class="s">__main__</span><span class="sh">"</span><span class="p">:</span> <span class="n">captured_at</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="nf">now</span><span class="p">(</span><span class="n">timezone</span><span class="p">.</span><span class="n">utc</span><span class="p">).</span><span class="nf">strftime</span><span class="p">(</span><span class="n">UTC</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">capture started at </span><span class="si">{</span><span class="n">captured_at</span><span class="si">}</span><span class="sh">"</span><span class="p">)</span> <span class="n">rows</span> <span class="o">=</span> <span class="p">[</span><span class="n">row</span> <span class="k">for</span> <span class="n">spec</span> <span class="ow">in</span> <span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="nf">collect</span><span class="p">(</span><span class="n">spec</span><span class="p">)]</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="sh">"</span><span class="s">jobs.json</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="nf">dump</span><span class="p">({</span><span class="sh">"</span><span class="s">captured_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">captured_at</span><span class="p">,</span> <span class="sh">"</span><span class="s">jobs</span><span class="sh">"</span><span class="p">:</span> <span class="n">rows</span><span class="p">},</span> <span class="n">f</span><span class="p">,</span> <span class="n">ensure_ascii</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="sh">"</span><span class="s">jobs.csv</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">newline</span><span class="o">=</span><span class="sh">""</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8-sig</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="c1"># BOM: Excel sees UTF-8 </span> <span class="n">writer</span> <span class="o">=</span> <span class="n">csv</span><span class="p">.</span><span class="nc">DictWriter</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">fieldnames</span><span class="o">=</span><span class="n">FIELDS</span><span class="p">)</span> <span class="n">writer</span><span class="p">.</span><span class="nf">writeheader</span><span class="p">()</span> <span class="n">writer</span><span class="p">.</span><span class="nf">writerows</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span> <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">wrote </span><span class="si">{</span><span class="nf">len</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span><span class="si">}</span><span class="s"> rows to jobs.json and jobs.csv</span><span class="sh">"</span><span class="p">)</span> </code></pre> </div> <p></p> <h2> Real results </h2> <p>Here is the run against one public board per vendor, plus a made-up Lever slug to show the 404 path:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>$ python normalize_jobs.py greenhouse:stripe lever:palantir ashby:ramp lever:no-such-board-20260927 capture started at 2026-09-27T12:06:53Z greenhouse:stripe: 701 jobs, 0.44 MB, non-empty: {'location': 701, 'workplace': 0, 'department': 0, 'salary_min': 0} lever:palantir: 321 jobs, 6.19 MB, non-empty: {'location': 321, 'workplace': 321, 'department': 321, 'salary_min': 0} ashby:ramp: 158 jobs, 2.86 MB, non-empty: {'location': 158, 'workplace': 158, 'department': 158, 'salary_min': 151} lever:no-such-board-20260927: board not found (HTTP 404) - check the slug and the vendor wrote 1180 rows to jobs.json and jobs.csv </code></pre> </div> <p>This capture ran on 27 September 2026; afterwards the script gained one fix, retrying a connection that drops mid-download, which does not change the output above. None of these boards was empty or rate-limited in this run. The "no open jobs" and back-off paths were checked with offline unit tests that fake the HTTP responses (12 tests, all passing), and so was Lever's <code>salaryRange</code> mapping, since no Palantir posting had one.</p> <p>The table below shows the first three distinct "Engineer" titles per source, in response order.</p> <p><strong>Captured 2026-09-27 12:06:53 UTC.</strong> Postings change constantly, and some of these may already be closed. The companies are examples of public boards that responded on 27 September 2026. This article is not affiliated with or endorsed by them or by Greenhouse, Lever or Ashby.</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Source</th> <th>Board</th> <th>Title</th> <th>Location</th> <th>Apply link</th> </tr> </thead> <tbody> <tr> <td>Greenhouse</td> <td>stripe</td> <td>Abuse Research Engineer</td> <td>Remote from the US</td> <td><a href="https://stripe.com/jobs/search?gh_jid=8172503" rel="noopener noreferrer">stripe.com</a></td> </tr> <tr> <td>Greenhouse</td> <td>stripe</td> <td>AI Engineer</td> <td>Chicago</td> <td><a href="https://stripe.com/jobs/search?gh_jid=8044460" rel="noopener noreferrer">stripe.com</a></td> </tr> <tr> <td>Greenhouse</td> <td>stripe</td> <td>Android BSP Engineer</td> <td>Taipei, Taiwan</td> <td><a href="https://stripe.com/jobs/search?gh_jid=8074125" rel="noopener noreferrer">stripe.com</a></td> </tr> <tr> <td>Lever</td> <td>palantir</td> <td>Backend Software Engineer - Application Development</td> <td>London, United Kingdom</td> <td><a href="https://jobs.lever.co/palantir/10dfc8bc-99ad-4ca2-ab76-853cb90a92c2/apply" rel="noopener noreferrer">jobs.lever.co</a></td> </tr> <tr> <td>Lever</td> <td>palantir</td> <td>Backend Software Engineer - Defense</td> <td>Washington, D.C.</td> <td><a href="https://jobs.lever.co/palantir/1345438c-ebfc-4fa5-b545-30c1414f317c/apply" rel="noopener noreferrer">jobs.lever.co</a></td> </tr> <tr> <td>Lever</td> <td>palantir</td> <td>Backend Software Engineer - Infrastructure</td> <td>New York, NY</td> <td><a href="https://jobs.lever.co/palantir/6fe5515f-f677-4d98-8ac2-1775a425f5e7/apply" rel="noopener noreferrer">jobs.lever.co</a></td> </tr> <tr> <td>Ashby</td> <td>ramp</td> <td>Security Engineer, Cloud</td> <td>New York, NY (HQ)</td> <td><a href="https://jobs.ashbyhq.com/ramp/34413f8d-26bf-4bbc-8ade-eb309a0e2245/application" rel="noopener noreferrer">jobs.ashbyhq.com</a></td> </tr> <tr> <td>Ashby</td> <td>ramp</td> <td>Mobile Engineer, Android</td> <td>New York, NY (HQ)</td> <td><a href="https://jobs.ashbyhq.com/ramp/f564dcf9-9390-4a3f-896f-8047a5086040/application" rel="noopener noreferrer">jobs.ashbyhq.com</a></td> </tr> <tr> <td>Ashby</td> <td>ramp</td> <td>Software Engineer, Frontend</td> <td>New York, NY (HQ)</td> <td><a href="https://jobs.ashbyhq.com/ramp/4e64ab86-4e30-403b-b1b9-41dc052570ce/application" rel="noopener noreferrer">jobs.ashbyhq.com</a></td> </tr> </tbody> </table></div> <h2> Exporting JSON and CSV </h2> <p>The last block of the script writes both files. The JSON keeps the capture time next to the rows, so a saved file always says when it was taken:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="sh">"</span><span class="s">jobs.json</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">json</span><span class="p">.</span><span class="nf">dump</span><span class="p">({</span><span class="sh">"</span><span class="s">captured_at</span><span class="sh">"</span><span class="p">:</span> <span class="n">captured_at</span><span class="p">,</span> <span class="sh">"</span><span class="s">jobs</span><span class="sh">"</span><span class="p">:</span> <span class="n">rows</span><span class="p">},</span> <span class="n">f</span><span class="p">,</span> <span class="n">ensure_ascii</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="sh">"</span><span class="s">jobs.csv</span><span class="sh">"</span><span class="p">,</span> <span class="sh">"</span><span class="s">w</span><span class="sh">"</span><span class="p">,</span> <span class="n">newline</span><span class="o">=</span><span class="sh">""</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">"</span><span class="s">utf-8-sig</span><span class="sh">"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="c1"># BOM: Excel sees UTF-8 </span> <span class="n">writer</span> <span class="o">=</span> <span class="n">csv</span><span class="p">.</span><span class="nc">DictWriter</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">fieldnames</span><span class="o">=</span><span class="n">FIELDS</span><span class="p">)</span> <span class="n">writer</span><span class="p">.</span><span class="nf">writeheader</span><span class="p">()</span> <span class="n">writer</span><span class="p">.</span><span class="nf">writerows</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span> </code></pre> </div> <p>One row from this run's <code>jobs.json</code>:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight json"><code><span class="p">{</span><span class="w"> </span><span class="nl">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ashby"</span><span class="p">,</span><span class="w"> </span><span class="nl">"board"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ramp"</span><span class="p">,</span><span class="w"> </span><span class="nl">"job_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"34413f8d-26bf-4bbc-8ade-eb309a0e2245"</span><span class="p">,</span><span class="w"> </span><span class="nl">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Security Engineer, Cloud"</span><span class="p">,</span><span class="w"> </span><span class="nl">"location"</span><span class="p">:</span><span class="w"> </span><span class="s2">"New York, NY (HQ)"</span><span class="p">,</span><span class="w"> </span><span class="nl">"workplace"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hybrid"</span><span class="p">,</span><span class="w"> </span><span class="nl">"department"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Engineering"</span><span class="p">,</span><span class="w"> </span><span class="nl">"apply_url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://jobs.ashbyhq.com/ramp/34413f8d-26bf-4bbc-8ade-eb309a0e2245/application"</span><span class="p">,</span><span class="w"> </span><span class="nl">"posted_at"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2026-04-07T17:12:35Z"</span><span class="p">,</span><span class="w"> </span><span class="nl">"salary_min"</span><span class="p">:</span><span class="w"> </span><span class="mi">211400</span><span class="p">,</span><span class="w"> </span><span class="nl">"salary_max"</span><span class="p">:</span><span class="w"> </span><span class="mi">290600</span><span class="p">,</span><span class="w"> </span><span class="nl">"salary_currency"</span><span class="p">:</span><span class="w"> </span><span class="s2">"USD"</span><span class="p">,</span><span class="w"> </span><span class="nl">"salary_interval"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1 YEAR"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <p>Three details make the CSV behave:</p> <ul> <li> <strong><code>utf-8-sig</code></strong> writes a byte-order mark so Excel opens the file as UTF-8. Nine of the titles in this run contain an em or en dash (for example "Account Executive, Funded Startups — Hunter, Italy"), and without the BOM Excel can show them as garbage.</li> <li> <strong><code>newline=""</code></strong> is what the <code>csv</code> docs require. Without it you get blank lines between rows on Windows.</li> <li> <strong><code>csv.DictWriter</code></strong> quotes commas and quotes for you, and writes <code>None</code> as an empty cell. In this run's file, 1,029 of 1,180 rows have an empty <code>salary_min</code>, which is honest. A <code>0</code> or "N/A" there would be wrong.</li> </ul> <p>If you run this on a schedule, de-duplicate on <code>(source, board, job_id)</code>. Titles and locations aren't unique, and Palantir alone lists the same title in several cities.</p> <h2> Missing values, source limits and common mistakes </h2> <p><strong>Missing values are information.</strong> Keep them as <code>None</code> or empty. The non-empty counts in the run log show where the gaps are: Greenhouse gave no workplace for any of its 701 rows, but 109 of its location strings contain the word "remote". If you infer a workplace from text, put it in a separate column so the reader can tell reported values from guessed ones. On Ashby, 151 of 158 Ramp postings published a salary. The currencies were USD (140), CAD (6), GBP (4) and SEK (1), and the intervals were <code>1 YEAR</code> (147) and <code>1 MONTH</code> (4).</p> <p><strong>Source limits:</strong></p> <ul> <li>It's a snapshot. Counts and postings change during the day.</li> <li>Only three vendors are covered. Many employers use other systems or their own careers site, so a 404 on all three says nothing about whether a company is hiring.</li> <li>Lever EU boards need <code>api.eu.lever.co</code>, and the script only calls the global host.</li> <li>Location is free text. Greenhouse strings like "Chicago, IL; Atlanta, GA" pack several places into one field (22 of Stripe's rows in this run contain a <code>;</code>). Lever's <code>allLocations</code> and Ashby's <code>secondaryLocations</code> hold extra locations that this script doesn't export.</li> <li>Ashby's summary gives one salary range per posting. When tiers differ by location, keep the raw <code>compensation</code> object if you need per-location pay.</li> <li>All 701 Greenhouse links in this run point to stripe.com, not greenhouse.io. That is the company's choice, and it is still the right link to send people to.</li> </ul> <p><strong>Common mistakes:</strong></p> <ul> <li>Treating a 429 or a timeout as "board not found", or treating a 200 with an empty list as a 404.</li> <li>Taking the first Ashby compensation component as the salary.</li> <li>Reading Lever's <code>createdAt</code> as seconds. <code>datetime.fromtimestamp(1786469891368)</code> fails (with <code>OSError</code> on Windows), so divide by 1000 first.</li> <li>Comparing salaries without checking currency and interval.</li> <li>Assuming a slug belongs to the company you meant. A 200 proves a board exists, not whose it is. Take the slug from the company's own careers page.</li> <li>Fetching too fast. The script waits one second between requests, sends a descriptive <code>User-Agent</code> and fetches each board once per run. These are public boards meant for distribution, so link back to the original posting for applying.</li> </ul> <h2> Official sources </h2> <ul> <li>Greenhouse Job Board API: <a href="https://docs.greenhouse.io/job-board.html" rel="noopener noreferrer">https://docs.greenhouse.io/job-board.html</a> </li> <li>Lever Postings API: <a href="https://github.com/lever/postings-api" rel="noopener noreferrer">https://github.com/lever/postings-api</a> </li> <li>Ashby public job posting API: <a href="https://developers.ashbyhq.com/docs/public-job-posting-api" rel="noopener noreferrer">https://developers.ashbyhq.com/docs/public-job-posting-api</a> </li> <li>Python <code>csv</code> module: <a href="https://docs.python.org/3/library/csv.html" rel="noopener noreferrer">https://docs.python.org/3/library/csv.html</a> </li> <li>Python <code>json</code> module: <a href="https://docs.python.org/3/library/json.html" rel="noopener noreferrer">https://docs.python.org/3/library/json.html</a> </li> </ul> <p><em>This article and its code were drafted by an AI assistant at the account owner's request. The code was run against the live public APIs, and the results and table were captured on 27 September 2026 at 12:06 UTC.</em></p> python api tutorial dataengineering