<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: MarkDonish</title>
    <description>The latest articles on DEV Community by MarkDonish (@markdonish).</description>
    <link>https://dev.to/markdonish</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4255642%2Fe0b67fac-beda-4e9c-b293-c49d212accc4.gif</url>
      <title>DEV Community: MarkDonish</title>
      <link>https://dev.to/markdonish</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/markdonish"/>
    <language>en</language>
    <item>
      <title>How I Replaced Heavy Scraping Stacks with Pure Python Stdlib (+ Live Interactive HTML Reports)</title>
      <dc:creator>MarkDonish</dc:creator>
      <pubDate>Fri, 20 Nov 2026 11:53:25 +0000</pubDate>
      <link>https://dev.to/markdonish/how-i-replaced-heavy-scraping-stacks-with-pure-python-stdlib-live-interactive-html-reports-4g42</link>
      <guid>https://dev.to/markdonish/how-i-replaced-heavy-scraping-stacks-with-pure-python-stdlib-live-interactive-html-reports-4g42</guid>
      <description>&lt;p&gt;Whenever I need to extract HTML tables or JSON-LD from a static directory, clean up currencies and URLs, and send a stakeholder something better than a raw CSV, pulling in Playwright, BeautifulSoup, and Pandas feels like overkill. For static server-rendered HTML, dragging in hundreds of megabytes of dependencies slows down CI and creates version churn.&lt;/p&gt;

&lt;p&gt;So I built &lt;strong&gt;DataPipe Kit&lt;/strong&gt;, a zero-dependency &lt;strong&gt;Python 3.9+&lt;/strong&gt; CLI and library using only the Python standard library (&lt;code&gt;html.parser&lt;/code&gt;, &lt;code&gt;urllib.request&lt;/code&gt;, &lt;code&gt;csv&lt;/code&gt;, &lt;code&gt;json&lt;/code&gt;, &lt;code&gt;re&lt;/code&gt;, and &lt;code&gt;html&lt;/code&gt;).&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GitHub Repo&lt;/strong&gt;: &lt;a href="https://github.com/MarkDonish/datapipe-kit" rel="noopener noreferrer"&gt;https://github.com/MarkDonish/datapipe-kit&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Project Page&lt;/strong&gt;: &lt;a href="https://markdonish.github.io/datapipe-kit/" rel="noopener noreferrer"&gt;https://markdonish.github.io/datapipe-kit/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Live Interactive Demo Report&lt;/strong&gt;: &lt;a href="https://markdonish.github.io/datapipe-kit/demo/report.html" rel="noopener noreferrer"&gt;https://markdonish.github.io/datapipe-kit/demo/report.html&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmmabod9o77957ot5n3n4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmmabod9o77957ot5n3n4.png" alt="DataPipe Kit interactive HTML report preview" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  One-Command Pipeline
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 -m datapipe pipeline examples/sample_catalog.html \
  --out-dir ./out \
  --required Vendor \
  --dedupe-by Vendor \
  --price-fields &lt;span class="s2"&gt;"Monthly Price"&lt;/span&gt; \
  --email-fields &lt;span class="s2"&gt;"Contact Email"&lt;/span&gt; \
  --url-fields Website \
  --drop-invalid-emails
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From a messy 6-row HTML catalog containing duplicate whitespace entries, malformed emails, and currency strings like &lt;code&gt;$1,250.00&lt;/code&gt;, it outputs:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;cleaned.csv&lt;/code&gt; (UTF-8 BOM so Microsoft Excel opens cleanly without encoding issues)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;cleaned.json&lt;/code&gt; (typed numeric floats + email validation flags)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;report.html&lt;/code&gt; (a single-file dark-mode dashboard with live multi-column filtering, numeric/text sorting, KPI cards, and one-click filtered CSV download)&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Three Real-World Edge Cases Worth Handling Explicitly
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Irregular HTML &lt;code&gt;&amp;lt;table&amp;gt;&lt;/code&gt; Rows
&lt;/h3&gt;

&lt;p&gt;Real HTML tables routinely omit trailing &lt;code&gt;&amp;lt;td&amp;gt;&lt;/code&gt; cells or inject spacer &lt;code&gt;&amp;lt;tr&amp;gt;&lt;/code&gt; rows. Padding short rows with &lt;code&gt;""&lt;/code&gt; and skipping blank rows avoids index errors and shifted columns:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_rows_to_records&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;headers&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"col_{idx + 1}"&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;idx&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;header&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  2. Escaping Inline JSON Against &lt;code&gt;&amp;lt;/script&amp;gt;&lt;/code&gt; Breakouts
&lt;/h3&gt;

&lt;p&gt;Embedding serialized JSON directly inside a &lt;code&gt;&amp;lt;script&amp;gt;&lt;/code&gt; tag breaks if any extracted cell contains &lt;code&gt;&amp;lt;/script&amp;gt;&lt;/code&gt;. Escaping &lt;code&gt;&amp;lt;/&lt;/code&gt; to &lt;code&gt;&amp;lt;\/&lt;/code&gt; in Python and escaping cell entities in the DOM helper closes that XSS vector cleanly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"columns"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"rows"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"numericCols"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;numeric_cols&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;ensure_ascii&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"&amp;lt;\\/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  3. Deterministic Currency &amp;amp; URL Normalization
&lt;/h3&gt;

&lt;p&gt;Standardizing strings like &lt;code&gt;"Acme   Cloud"&lt;/code&gt;, &lt;code&gt;"$1,250.00"&lt;/code&gt;, and &lt;code&gt;"acmecloud.io/"&lt;/code&gt; before deduplicating catches duplicates that exact string matching misses, while &lt;code&gt;encoding="utf-8-sig"&lt;/code&gt; ensures Excel renders non-ASCII strings properly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try It Locally
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/MarkDonish/datapipe-kit.git
&lt;span class="nb"&gt;cd&lt;/span&gt; datapipe-kit
python3 -m unittest discover -s tests -v
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All 6 &lt;code&gt;unittest&lt;/code&gt; cases pass in under 0.1s with zero third-party installs (&lt;code&gt;CI&lt;/code&gt; badge green on GitHub Actions). If you build lightweight Python automation scripts or client data handoffs, feel free to fork or test it out!&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Open to custom Python automation, web extraction, and frontend/reporting work — feel free to reach out via &lt;a href="https://github.com/MarkDonish" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt; or email &lt;code&gt;marklodpice@gmail.com&lt;/code&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>webdev</category>
      <category>automation</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
