The latest articles on DEV Community by George R. (@georouv). https://dev.to/georouv https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3823509%2Fd2f957f3-a73f-45e8-a641-2a7ab3a2d157.jpeg https://dev.to/georouv en George R. Sat, 14 Mar 2026 07:07:44 +0000 https://dev.to/georouv/i-built-git-log-for-any-website-track-changes-with-diffs-and-ai-summaries-445g https://dev.to/georouv/i-built-git-log-for-any-website-track-changes-with-diffs-and-ai-summaries-445g <p>Websites change all the time — pricing pages, docs, terms of service. I wanted a simple CLI to tell me <em>what</em> changed, like <code>git log</code> but for any URL.</p> <p>So I built <strong>crawldiff</strong>.</p> <h2> How it works </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>pip <span class="nb">install </span>crawldiff <span class="c"># Snapshot a site</span> crawldiff crawl https://stripe.com/pricing <span class="c"># Come back later — see what changed</span> crawldiff diff https://stripe.com/pricing <span class="nt">--since</span> 7d </code></pre> </div> <p>It uses Cloudflare's new <a href="https://blog.cloudflare.com/crawl-entire-websites-with-a-single-api-call-using-browser-rendering/" rel="noopener noreferrer"><code>/crawl</code> endpoint</a> to snapshot pages, stores everything locally in SQLite, and produces git-style unified diffs — with optional AI summaries via Claude, GPT, or Cloudflare Workers AI.</p> <h2> Features </h2> <ul> <li> <strong>Git-style diffs</strong> — colored unified diffs in your terminal</li> <li> <strong>AI summaries</strong> — "Pricing increased from $25 to $30, new Enterprise tier added"</li> <li> <strong>Watch mode</strong> — <code>crawldiff watch https://competitor.com --every 1h</code> </li> <li> <strong>Multiple outputs</strong> — terminal, JSON (pipe to jq/Slack), Markdown reports</li> <li> <strong>Incremental crawling</strong> — only fetches changed pages via Cloudflare's <code>modifiedSince</code> </li> <li> <strong>Local storage</strong> — everything in SQLite at <code>~/.crawldiff/</code> </li> </ul> <h2> The tech stack </h2> <ul> <li>Python 3.12, typer, rich, httpx</li> <li>Cloudflare Browser Rendering /crawl API</li> <li>difflib for unified diffs</li> <li>SQLite for local snapshot storage</li> <li>96 tests, mypy strict, CI on GitHub Actions</li> </ul> <h2> Why not existing tools? </h2> <p>Most website monitoring tools are SaaS dashboards built for marketing teams. crawldiff is for developers — it's a CLI, it diffs like git, it pipes to anything, and it stores everything locally.</p> <p>Only requirement is a free Cloudflare account.</p> <p><strong>GitHub:</strong> <a href="https://github.com/GeoRouv/crawldiff" rel="noopener noreferrer">github.com/GeoRouv/crawldiff</a><br> <strong>PyPI:</strong> <code>pip install crawldiff</code></p> <p>Happy to answer any questions!</p> python opensource webdev showdev