<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Jeremy Burgos</title>
    <description>The latest articles on DEV Community by Jeremy Burgos (@jeremy-burgos).</description>
    <link>https://dev.to/jeremy-burgos</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3940901%2Fe5254be5-4876-4b2e-a167-06beb4d55018.png</url>
      <title>DEV Community: Jeremy Burgos</title>
      <link>https://dev.to/jeremy-burgos</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jeremy-burgos"/>
    <language>en</language>
    <item>
      <title>Your AI audit tool is a lead generator, not a final verdict</title>
      <dc:creator>Jeremy Burgos</dc:creator>
      <pubDate>Tue, 21 Jul 2026 10:00:00 +0000</pubDate>
      <link>https://dev.to/jeremy-burgos/your-ai-audit-tool-is-a-lead-generator-not-a-final-verdict-20ai</link>
      <guid>https://dev.to/jeremy-burgos/your-ai-audit-tool-is-a-lead-generator-not-a-final-verdict-20ai</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;TL;DR: AI audit output is a lead, not a verdict. Give every recommendation one of five dispositions, verify the survivors against crawl evidence and primary docs, and prove the fix held with a re-crawl. The five-way triage and a seven-point checklist are below.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Before you action a single AI recommendation, sort it into one of these:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ACCEPT   specific, aligned with the page, corroborated by the crawl  -&amp;gt; task as written
REVISE   right direction, wrong specifics                            -&amp;gt; fix the detail, then action
MERGE    several prompts, one root cause                             -&amp;gt; collapse into one task
REJECT   generic, unsupported, or contradicted by the crawl         -&amp;gt; it dies here
BACKLOG  real but low priority                                       -&amp;gt; recorded with its evidence
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That taxonomy is the whole method. Everything else is how you decide which bucket a lead lands in, and how you prove you were right after the fix ships.&lt;/p&gt;

&lt;p&gt;The trap with AI audit output is that it reads the same whether it is correct or not. It names the URL, the issue, and the fix in clean prose, with the same confidence either way. The obviously wrong recommendation is cheap because you ignore it. The plausible, confident, wrong one is expensive because you build a ticket around it: developer time on markup that already validates, internal links pruned because a score flagged them, a redirect removed that was doing its job.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a model can only ever hand you a lead
&lt;/h2&gt;

&lt;p&gt;A lead is a candidate finding. A verdict is a decision to act. A model produces the first and cannot produce the second, and the gap is structural. A model scoring your pages has no access to your live index, no ground truth about your stack, and only the part of the page it can parse. Strong pattern match, narrow view. Good at widening the search for problems, bad at closing the decision on any one. So the rule is absolute: nothing enters the plan on an AI suggestion alone.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the AI gets into the crawl
&lt;/h2&gt;

&lt;p&gt;Screaming Frog connects directly to OpenAI, Gemini, Anthropic, and Ollama, and runs custom prompts at crawl time against each page, returning a score and rationale on the URL row. You can configure up to 100 prompts.&lt;/p&gt;

&lt;p&gt;In the cycle these examples come from, nine prompts ran, each bound to the segment it was written for so it only saw the pages it could judge. Four segments, sixteen pages in the article segment. Binding was also a cost control. Unbound against every URL, the prompts would have made about 312 model calls. Bound, the same analysis took 111, a 64 percent cut, with nothing scored out of scope.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three ways the model is confidently wrong
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm6nq8i661cv3df8licyk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm6nq8i661cv3df8licyk.png" alt="Diagram, three failure modes where AI is confidently wrong, with the verification step for each: read the rendered page, check current vendor docs, use URL Inspection" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It scores what it can see, not what is true.&lt;/strong&gt; The operational-depth prompt in this cycle credited configuration values written in prose and scored them as zero when the same values lived only in a code block or a screenshot. An article with real rule expressions in a fenced block returned zero operational items. An article that stated the same thresholds in sentences scored well. Same values, different visibility. A low score can mean weak content, or content the model could not read.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It restates vendor behavior that has drifted or is plan-gated.&lt;/strong&gt; Four claims in this cycle only survived a check against current vendor docs, each verified against the primary source:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cloudflare's Log action for custom rules is Enterprise-only, so "start in log mode" is wrong for most readers on lower plans.&lt;/li&gt;
&lt;li&gt;Cloud Armor backend buckets accept only edge security policies, not backend security policies, so buckets and backend services are not interchangeable targets.&lt;/li&gt;
&lt;li&gt;HSTS preload needs a max-age of at least 31536000 seconds with includeSubDomains present.&lt;/li&gt;
&lt;li&gt;Google Cloud auto-disables exposed service-account keys by default, on by default since June 16, 2024.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of those are corrections a model makes for you, because it is repeating a confident generalization. Verify vendor claims against the vendor's live docs, by name.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It comments on things it cannot verify.&lt;/strong&gt; A model reading a page has no view of Google's index, so any index-status claim is inference dressed as fact. Confirm it in URL Inspection. The same blindness produces false positives on legitimate business logic: deliberate redirects called chains, product or filter variants called duplicates. The model sees a pattern that looks like a problem and cannot tell it is intentional.&lt;/p&gt;

&lt;h2&gt;
  
  
  Verification, in order
&lt;/h2&gt;

&lt;p&gt;A lead becomes a finding only when it clears three checks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Does the crawl row actually say what the model claims? Most rejects die here.&lt;/li&gt;
&lt;li&gt;If it rests on a vendor feature, plan tier, default, or policy, does the vendor's current documentation confirm it?&lt;/li&gt;
&lt;li&gt;Does the schema pass Google's Rich Results Test, not only the crawler's validator? The two must agree.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Prove it with a re-crawl
&lt;/h2&gt;

&lt;p&gt;A crawl finds problems. A re-crawl proves they were solved. Diff a baseline crawl against a re-crawl in Screaming Frog's Compare mode with identical settings, segments, and prompts. Hold the instrument constant or a later difference could come from the measurement instead of the work. In this cycle, baseline June 11 against re-crawl June 22 showed invalid head HTML 39 to 0, rich result errors 34 to 0, internal and external redirects 9 and 20 to 0, and twelve of sixteen articles gaining 13,703 words of narrated detail.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foby6g4njm8va2g35e3ao.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foby6g4njm8va2g35e3ao.png" alt="Diagram, reverification flow from baseline crawl to re-crawl with identical settings to Compare mode to verified result, with sample deltas: invalid head HTML 39 to 0, schema errors 34 to 0, redirect links 29 to 0, content added 13,703 words" width="800" height="260"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Re-running the same prompts on the edited pages is how you check the edits, not a chance to improve the prompts. The prompt is the ruler. Hold it steady.&lt;/p&gt;

&lt;h2&gt;
  
  
  The checklist
&lt;/h2&gt;

&lt;p&gt;Run every AI lead through this before it becomes a task:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. Specific, or generic best practice it would say about any page?
2. Aligned with what this page is for?
3. Corroborated by the crawl row, rendered content, or response code?
4. If it rests on vendor behavior or a plan tier, do the current docs confirm it?
5. Can the model see what it scored, or is it locked in code, an image, or behind rendering?
6. If it claims indexation, was that checked in Search Console, not inferred?
7. After the fix shipped, was it re-measured against the same baseline?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Clears all seven, it is a finding. Fails one, it is a lead that is not done yet. Anyone can run prompts against a crawl. The skill is the triage, and the re-crawl that proves the survivor held.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Your crawl report averages out the bug: segment it by template</title>
      <dc:creator>Jeremy Burgos</dc:creator>
      <pubDate>Tue, 14 Jul 2026 10:00:00 +0000</pubDate>
      <link>https://dev.to/jeremy-burgos/your-crawl-report-averages-out-the-bug-segment-it-by-template-54ho</link>
      <guid>https://dev.to/jeremy-burgos/your-crawl-report-averages-out-the-bug-segment-it-by-template-54ho</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;TL;DR: Reading a crawl as one flat list averages out the problems that live inside a single template, and it makes every analysis pass run against pages it should ignore. Segment by template and role first. On one audit that cut AI analysis calls from roughly 312 to 111, a 64 percent reduction, with zero out-of-scope pages.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Run this first. One match rule per template, then a count check that proves the rule caught every page it should:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;&lt;span class="c"&gt;# One segment rule per template, matched on the URL path
&lt;/span&gt;&lt;span class="n"&gt;Guides&lt;/span&gt;     &lt;span class="n"&gt;Address&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt; &lt;span class="n"&gt;regex&lt;/span&gt;   ^&lt;span class="n"&gt;https&lt;/span&gt;://&lt;span class="n"&gt;example&lt;/span&gt;\.&lt;span class="n"&gt;com&lt;/span&gt;/&lt;span class="n"&gt;guides&lt;/span&gt;/
&lt;span class="n"&gt;Services&lt;/span&gt;   &lt;span class="n"&gt;Address&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt; &lt;span class="n"&gt;regex&lt;/span&gt;   ^&lt;span class="n"&gt;https&lt;/span&gt;://&lt;span class="n"&gt;example&lt;/span&gt;\.&lt;span class="n"&gt;com&lt;/span&gt;/&lt;span class="n"&gt;services&lt;/span&gt;/
&lt;span class="n"&gt;Articles&lt;/span&gt;   &lt;span class="n"&gt;Address&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt; &lt;span class="n"&gt;regex&lt;/span&gt;   ^&lt;span class="n"&gt;https&lt;/span&gt;://&lt;span class="n"&gt;example&lt;/span&gt;\.&lt;span class="n"&gt;com&lt;/span&gt;/&lt;span class="n"&gt;blog&lt;/span&gt;/

&lt;span class="c"&gt;# Validate the binding before trusting any per-segment number
&lt;/span&gt;&lt;span class="n"&gt;Expected&lt;/span&gt;   ==   &lt;span class="n"&gt;Crawl&lt;/span&gt; &lt;span class="n"&gt;segment&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt;
  &lt;span class="n"&gt;Guides&lt;/span&gt;     &lt;span class="m"&gt;5&lt;/span&gt;  ==   &lt;span class="m"&gt;5&lt;/span&gt;
  &lt;span class="n"&gt;Services&lt;/span&gt;   &lt;span class="m"&gt;5&lt;/span&gt;  ==   &lt;span class="m"&gt;5&lt;/span&gt;
  &lt;span class="n"&gt;Articles&lt;/span&gt;  &lt;span class="m"&gt;16&lt;/span&gt;  ==  &lt;span class="m"&gt;16&lt;/span&gt;
&lt;span class="c"&gt;# A short count means a slug slipped the pattern. Fix the rule first.
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That config is the whole move. One match rule per template, then a validation step so you know the rules caught every page they should. Everything below is why it changes your analysis and how far it scales.&lt;/p&gt;

&lt;p&gt;The problem it solves is the default way most people read a crawl. A crawl of any real site returns one long list of issues, and reading it that way averages out the bugs that live inside a single template. A title-length warning on a deep archive page and the same warning on a core service page collapse into one undifferentiated line. The count went up. Which template caused it stays hidden.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a flat crawl hides your worst problems
&lt;/h2&gt;

&lt;p&gt;When an issue lives inside one template, the duplicates a single template creates, a duplicate H2 from a shared component, schema emitted twice by one block, a flat view buries it among hundreds of unrelated rows. You triage by guessing and fix the loudest symptom instead of the template producing it.&lt;/p&gt;

&lt;p&gt;There is a second cost that only shows up at scale. Every validation pass and every model prompt runs against every URL, including pages the check was never meant to evaluate. On a small crawl that waste is invisible. On a large one it decides whether the analysis is affordable.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp8661i85r7f4lrgy3ov4.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp8661i85r7f4lrgy3ov4.png" alt="A flat crawl issue list on the left versus the same issues grouped by template segment on the right, with one template carrying most of the issues" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Segmenting in Screaming Frog, three ways
&lt;/h2&gt;

&lt;p&gt;Segmentation groups URLs by template or role so every report, issue count, and score reads per template instead of averaged sitewide. It is the same logic as testing by template cohort: stop treating the site as one list, start treating it as a set of templates that each have their own contract.&lt;/p&gt;

&lt;p&gt;In Screaming Frog this is the Segments feature, added in version 19.0. Define segments once and the tool adds a coloured segments column to every tab and a segments bar to the Issues tab. You can set a segment up at the start, during, or at the end of a crawl, so you can segment a crawl you already have without re-running it.&lt;/p&gt;

&lt;p&gt;Three ways to define one:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;URL pattern.&lt;/strong&gt; The common case, templates encoded in the path. One regex match per template against the Address field, as in the block above. This only holds if your URL structure is predictable. If one template is served under several path shapes, a single rule catches one and misses the rest.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Custom extraction.&lt;/strong&gt; When the template is not in the URL, pull an on-page signal that identifies it, a body class or the schema &lt;code&gt;@type&lt;/code&gt;, and segment on that value.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;API data.&lt;/strong&gt; Screaming Frog joins GA4 and Search Console data to the crawl, so you can segment by a metric like sessions or impressions, not just by structure.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Database storage mode for large crawls
&lt;/h2&gt;

&lt;p&gt;Segmentation pays off most on large crawls, and large crawls need database storage mode, which writes the crawl to disk instead of holding it all in memory. That is what lets you save a crawl, reopen it, and compare it against a later one. Screaming Frog's hardware guidance: 8GB of RAM is generally enough under 200,000 URLs, and crawling over one million URLs wants an SSD and 16GB or more, on a 64-bit operating system with at least 4GB as the floor.&lt;/p&gt;

&lt;h2&gt;
  
  
  A real four-segment model
&lt;/h2&gt;

&lt;p&gt;From a sanitized end-to-end review of a production WordPress site. Four segments by role, each judged on different criteria, because the question you ask an article is not the one you ask a service page.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Segment&lt;/th&gt;
&lt;th&gt;URLs&lt;/th&gt;
&lt;th&gt;Judged on&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Articles&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;Technical depth, information gain, claim accuracy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Guides&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Hub purpose, audience, routing to the right pages&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service pages&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Scope clarity, proof, CTA alignment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Trust core&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Evidenced authority, credentials that verify&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fisx6yigtw5c4ef9zqhqv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fisx6yigtw5c4ef9zqhqv.png" alt="Four crawl segments, articles, guides, service pages, and trust core, each labeled with its URL count and the criteria it is judged on" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The counts are not the point. A title warning on an article and a missing proof block on a service page are different problems with different fixes, and a flat crawl flattens them into one list. Per-segment reading also fed the schema work, since schema that breaks at the template level is exactly the kind of issue that hides in a flat view and snaps into focus when you read it per template.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure mode: rules skip slugs silently
&lt;/h2&gt;

&lt;p&gt;URL-pattern rules have a quiet failure mode. A slug that does not follow the dominant naming convention is skipped. No error, no warning. The page falls out of the segment, and every per-segment number after that is wrong by however many slipped through. Validate the binding with the count check in the lead block before you trust anything. On the audit above, the binding was validated against the segment counts before anything ran, so every intended page was scored and none were missed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Segmentation is cost control
&lt;/h2&gt;

&lt;p&gt;Screaming Frog can run language-model prompts and other API-driven checks against page content at crawl time. They bill per call on your own key. Unbound, every prompt runs against every URL. Bound to segments, each runs only against the pages it was written for: the article prompts against articles, the trust prompts against trust pages.&lt;/p&gt;

&lt;p&gt;On the audit above that cut the run from roughly 312 calls to 111, a 64 percent reduction, with zero out-of-scope URLs touched. Same coverage, a third of the calls. Reading per template also makes under-linked templates obvious, so you can route link equity toward priority pages instead of guessing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F766h1eevib8e43dbj88t.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F766h1eevib8e43dbj88t.png" alt="Stat comparison showing 312 unbound AI calls reduced to 111 segment-bound calls, a 64 percent reduction" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Track each template over time
&lt;/h2&gt;

&lt;p&gt;Compare mode loads two saved crawls run with identical settings and reports the change per metric, issue, and segment. Watch one template's counts move between crawls and catch a regression in that template before it shows up as a sitewide number, or before it shows up in rankings. It needs database storage mode and a licensed version, and the second crawl has to use identical settings, or the diff measures your configuration change instead of the site. Per-segment comparison is how a crawl becomes an evidence trail of what changed rather than a snapshot.&lt;/p&gt;

&lt;p&gt;Segment first. Validate the binding. Then every report reads per template, the analysis costs less, and fixes land on the template that produced the problem rather than the page where you happened to notice it.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Compare two crawls to confirm your SEO fixes shipped (and catch what broke)</title>
      <dc:creator>Jeremy Burgos</dc:creator>
      <pubDate>Tue, 07 Jul 2026 21:00:00 +0000</pubDate>
      <link>https://dev.to/jeremy-burgos/compare-two-crawls-to-confirm-your-seo-fixes-shipped-and-catch-what-broke-34e8</link>
      <guid>https://dev.to/jeremy-burgos/compare-two-crawls-to-confirm-your-seo-fixes-shipped-and-catch-what-broke-34e8</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;TL;DR: Making a fix is not proof it shipped. Crawl before you change anything, save that crawl, re-crawl after with identical settings, and compare the two. The comparison reports each issue as resolved, reduced, or unchanged, and surfaces anything your fix broke elsewhere. One real before-and-after below.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;You shipped the fix and closed the ticket. That is not the same as proving the change is live, correct, and free of collateral damage. The proof is a second crawl, compared against the first.&lt;/p&gt;

&lt;p&gt;A crawl comparison loads your re-crawl against a saved baseline and reports every issue as resolved, reduced, or unchanged, URL by URL. Here is the procedure, then a real worked result, then the part most people skip: using the comparison to catch what your fix broke somewhere else.&lt;/p&gt;

&lt;h2&gt;
  
  
  The procedure
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Verify a technical SEO fix with a crawl comparison (Screaming Frog)

1. Set storage mode to Database. Required for saved, comparable crawls.
2. Crawl the site. Save it as your dated baseline.
3. Make the fixes. Deploy. Let the page cache or CDN clear.
4. Re-crawl with the IDENTICAL config and the IDENTICAL segments.
5. Switch to Compare mode and select the baseline and the re-crawl.
6. Read the Issues comparison. Filters: Added, New, Removed, Missing.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Crawl comparison needs database storage mode and a paid license. The free build is capped at 500 URLs with the config restricted, so this is a licensed-tool workflow from step one.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it proved
&lt;/h2&gt;

&lt;p&gt;This is a real cycle on a production WordPress property: a June 11 baseline against a June 22 re-crawl, identical settings, every delta taken from Screaming Frog's Compare mode.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Finding&lt;/th&gt;
&lt;th&gt;Before&lt;/th&gt;
&lt;th&gt;After&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Invalid HTML elements in head&lt;/td&gt;
&lt;td&gt;39&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;Resolved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rich result validation errors&lt;/td&gt;
&lt;td&gt;34&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;Resolved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rich result validation warnings&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Reduced 89%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Internal redirect links&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;Resolved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;External redirect links&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;Resolved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Internal client error (4xx)&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;Resolved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Duplicate H2 across pages&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;Resolved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Non-indexable internal URLs&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Cleaner&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;External links resolving directly to 200&lt;/td&gt;
&lt;td&gt;78&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;td&gt;+22&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two of those rows need reading carefully rather than at face value. The four remaining non-indexable URLs are the XML sitemap files, which are non-indexable by design, not a problem to chase. And the 78 to 100 line is a side effect of the redirect cleanup, because internal links that pointed at redirecting legacy slugs were repointed to their final 200 destination, not a separate win.&lt;/p&gt;

&lt;p&gt;The high-priority cluster, head validation, schema errors, and broken or redirecting links, closed in full. Those are the items that decide whether the rest of a page's signals are read correctly, which is why they go first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule that makes the comparison mean anything
&lt;/h2&gt;

&lt;p&gt;The baseline and the re-crawl have to use the same spider config, the same segments, and the same prompts if you run any. Change the measurement between runs and a later difference could come from the instrument instead of the work.&lt;/p&gt;

&lt;p&gt;Treat the config as a ruler. You are measuring the same site twice and comparing the readings, so the difference is only attributable to your fixes if the ruler did not change. The failure mode to watch for is a fake win: you re-crawl with a tighter include rule, the non-indexable count drops, and it looks like your directive cleanup worked when you simply stopped crawling the URLs that carried the problem. Lock the config before the baseline and do not touch it until after the re-crawl.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading the comparison
&lt;/h2&gt;

&lt;p&gt;The overview gives you current, previous, change, and change percentage on every total. The issues view is where verification happens: each finding from the baseline reads as resolved, reduced, or unchanged, with the high-priority items flagged so you can see at a glance whether the changes that matter actually closed. A number that did not move is telling you a fix did not land, which is exactly what you want to know before you call the work done.&lt;/p&gt;

&lt;p&gt;The change detection runs on four filters: Added, New, Removed, Missing. That is what lets you see which specific URLs moved and in which direction, not just that a count went down.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4qooamili4y4zchons7b.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4qooamili4y4zchons7b.png" alt="Reverification flow: baseline crawl, re-crawl, Compare mode, verified result" width="800" height="260"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The benefit most people skip: regression detection
&lt;/h2&gt;

&lt;p&gt;A single audit snapshot tells you what is broken now. It cannot tell you what your last fix broke. That is the argument for re-crawling even when you are sure the change worked.&lt;/p&gt;

&lt;p&gt;A comparison surfaces new and added issues alongside the resolved ones, and the new ones are where regressions hide:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A redirect fix orphans a page that was only reachable through the old URL.&lt;/li&gt;
&lt;li&gt;A schema template change passes on the template you edited and breaks validation on a different template that shared the component.&lt;/li&gt;
&lt;li&gt;A noindex directive leaks onto pages you meant to keep indexable.&lt;/li&gt;
&lt;li&gt;An internal link repoint creates a fresh redirect chain you did not have before.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these show up if you only look at the issue you set out to fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove it without exposing private data
&lt;/h2&gt;

&lt;p&gt;The worked cycle published zero Search Console figures and zero analytics figures and still proved every structural result. Response codes, the canonical profile, the directive profile, and schema validation all come from the crawl's own data. For client work that means you can show a fix shipped and nothing regressed without disclosing indexation counts or traffic numbers. The crawl-level evidence stands on its own.&lt;/p&gt;

&lt;h2&gt;
  
  
  When a fix does not show up in the re-crawl
&lt;/h2&gt;

&lt;p&gt;Usually one of these:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A page cache or CDN is still serving the old version. Re-crawl after it clears, or verify against a direct origin pull.&lt;/li&gt;
&lt;li&gt;The change deployed to staging and never reached production.&lt;/li&gt;
&lt;li&gt;A segment rule built on a URL pattern silently skipped a slug that did not match the naming convention. Validate your segment counts before trusting per-segment diffs.&lt;/li&gt;
&lt;li&gt;The re-crawl covered a different URL set than the baseline. Back to holding the config constant.&lt;/li&gt;
&lt;li&gt;A rendering difference between the two runs, so the output you are comparing is not the same output.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Work that list before concluding a fix failed. More often the fix is fine and the measurement drifted.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make it recurring
&lt;/h2&gt;

&lt;p&gt;Once you have a saved baseline, a fixed config, and fixed segments, the whole thing becomes a loop: baseline, fix, re-crawl, compare. Because the config and segments are held constant, every new crawl compares cleanly against the last, drift gets caught early, and each fix stays measured rather than assumed to hold. Screaming Frog can even auto-compare the last two crawls on a scheduled run, which turns the loop into close to passive monitoring.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F17lvqxb8b6j8m99s7hlk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F17lvqxb8b6j8m99s7hlk.png" alt="Four-step loop: baseline, implement, re-crawl, compare, returning to baseline" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A crawl finds problems. A re-crawl compared against a saved baseline proves they were solved. The gap between those two is where most audits stop.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>tutorial</category>
      <category>performance</category>
    </item>
    <item>
      <title>JavaScript SEO is a proof problem: capture the evidence before you touch code</title>
      <dc:creator>Jeremy Burgos</dc:creator>
      <pubDate>Tue, 30 Jun 2026 10:30:00 +0000</pubDate>
      <link>https://dev.to/jeremy-burgos/javascript-seo-is-a-proof-problem-capture-the-evidence-before-you-touch-code-2c9j</link>
      <guid>https://dev.to/jeremy-burgos/javascript-seo-is-a-proof-problem-capture-the-evidence-before-you-touch-code-2c9j</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;TL;DR: JS SEO failures are rarely "just JavaScript." Prove three states (response HTML, rendered DOM, indexable state) before changing code. The ten-minute triage below tells you which layer is actually failing.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The rule that saves the most wasted work:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Do not change code until you can show, in captured output,
what is missing or unstable.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The expensive mistake is rewriting templates to fix a rendering problem that was actually a noindex gate in the response, a canonical that flips after load, or links that only exist after a click. The fix was cheap; the misdiagnosis was not. JS SEO issues are a mismatch between what the browser shows and what search systems can reliably fetch, render, and interpret. The work is not to remove JavaScript. It is to prove whether JavaScript is hiding or changing the signals search engines need.&lt;/p&gt;

&lt;h3&gt;
  
  
  Three phases, not one moment
&lt;/h3&gt;

&lt;p&gt;Crawling, rendering, and indexing are separate phases that produce different evidence. Crawling is retrieval: status codes, redirects, robots directives, canonical signals, and links in the response HTML. Rendering is execution: the DOM after scripts run, which may now contain content, links, metadata, or structured data that were not in the response HTML. Indexing is processing and storage of an eligible page. The separation is what lets you rule layers out: a page can be crawlable but not renderable, render fine but be non-indexable, or be indexable and still look thin because the indexed representation never included the rendered content you assumed was there.&lt;/p&gt;

&lt;h3&gt;
  
  
  The three states you prove
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;State 1  Response HTML    raw server HTML, before scripts. Proves app-shell vs real content,
                          fetch-time directives/canonical, links without execution.
State 2  Rendered DOM     after scripts run. Proves what appeared only after load,
                          and whether identity signals changed after load.
State 3  Indexable state  eligibility given status, directives, canonical consistency.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Critical rule: do not rely on JavaScript to fix indexability after load. If a directive is wrong at fetch time, treat it as wrong, because that is the version you cannot count on a search system to repair.&lt;/p&gt;

&lt;h3&gt;
  
  
  Ten-minute triage
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Confirm indexability at fetch time: final status code, robots directives in response HTML, canonical in response HTML. If there is a noindex in the response or the canonical points somewhere unintended, stop and fix the gate. No rendering fix helps a page that is not eligible.&lt;/li&gt;
&lt;li&gt;Compare response HTML to rendered DOM. If primary content, links, or metadata exist only after render, the site depends on execution, which raises the bar for stability and crawl-safe discovery.&lt;/li&gt;
&lt;li&gt;Check interaction dependence. If main content or links appear only after scroll or click, or "Load more" hides crawlable URLs, assume discovery risk until you prove a crawl-safe pattern.&lt;/li&gt;
&lt;li&gt;Validate metadata stability across both states: title, canonical, robots, structured data. A canonical or robots directive that changes after load causes consolidation problems even when content renders fine.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbev171x269anxr6d4x3n.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbev171x269anxr6d4x3n.png" alt="response html vs rendered dom side by side" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For diagnostics, the URL Inspection tool in Search Console shows the crawled versus rendered view, and Screaming Frog with JavaScript rendering enabled surfaces the same discrepancies across a whole site. Both are checks you run on your own property.&lt;/p&gt;

&lt;h3&gt;
  
  
  Rendering strategy sets the risk
&lt;/h3&gt;

&lt;p&gt;Client-side rendering is highest risk when the response is mostly an app shell and the real content only appears after load. SSR lowers risk by delivering meaningful HTML at fetch time, but still needs validation for consistent canonicals and directives. Static generation tends to be most stable. Pre-rendering is a practical bridge when you need stable HTML fast but cannot rewrite the app yet. The evidence tells you which you need: critical elements missing from response HTML, or unstable head signals, is the signal.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>seo</category>
      <category>javascript</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Field vs lab data: why most Core Web Vitals arguments are dataset confusion</title>
      <dc:creator>Jeremy Burgos</dc:creator>
      <pubDate>Tue, 23 Jun 2026 10:30:00 +0000</pubDate>
      <link>https://dev.to/jeremy-burgos/field-vs-lab-data-why-most-core-web-vitals-arguments-are-dataset-confusion-5d6e</link>
      <guid>https://dev.to/jeremy-burgos/field-vs-lab-data-why-most-core-web-vitals-arguments-are-dataset-confusion-5d6e</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;TL;DR: If you cannot name which dataset you are looking at, you are arguing, not diagnosing. Field answers "are real users failing at scale." Lab answers "what mechanism caused it." Sequence: field to scope, lab to diagnose, field to validate.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The sequence that resolves most stuck performance debates:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;field  -&amp;gt; confirm the problem exists and find its scope
lab    -&amp;gt; isolate a cause you can test
field  -&amp;gt; validate the fix against the evidence you started with
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Core Web Vitals work falls apart when two people argue from two datasets. One looks at real-user field results, the other at a lab simulation. Both valid, both useful, answering different questions. Name which one, or you are not diagnosing.&lt;/p&gt;

&lt;h3&gt;
  
  
  What they measure, and the 75th percentile
&lt;/h3&gt;

&lt;p&gt;LCP (good: 2.5s or less) is loading, INP (good: 200ms or less) is responsiveness, CLS (good: 0.1 or less) is visual stability. They are outcome metrics: they tell you what users experienced, not why. The detail that ends most "but it is fast for me" disagreements: pass/fail is the 75th percentile, not an average. A site can feel fast for many users and still fail, because the 75th percentile forces you to care about weaker devices and slower networks.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5p21vv5ct62a6rltpv12.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5p21vv5ct62a6rltpv12.png" alt="page speed insights desktop" width="800" height="380"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Field and lab are not interchangeable
&lt;/h3&gt;

&lt;p&gt;Field data is aggregated real-user experience across devices and networks, including variance you cannot simulate: slow hardware, cellular latency, packet loss, cache states. It answers whether users are actually failing at scale. Lab data is controlled and repeatable, and gives you the traces field cannot: render-blocking resources, long tasks, layout shift events, waterfalls. It answers what mechanism is likely causing the outcome. They diverge because field is a distribution across sessions and lab is a snapshot under one config. Do not average them. Sequence them. If you claim what users experienced, trust field. If you claim why, trust lab traces.&lt;/p&gt;

&lt;h3&gt;
  
  
  An evidence standard that stays falsifiable
&lt;/h3&gt;

&lt;p&gt;Map every claim to an artifact that would look different if the claim were false:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Claim: real users are failing        -&amp;gt; Search Console CWV trend at URL-group level
Claim: it is template-based          -&amp;gt; repeated failures across same-type URLs
Claim: LCP delayed by server/cache   -&amp;gt; lab traces: late doc response, delayed render start
Claim: INP failing, main-thread      -&amp;gt; DevTools recording: long tasks overlapping input
Claim: a release caused a regression -&amp;gt; timing correlation with deploy + change log
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Do not trust a single Lighthouse run as proof of field improvement, desktop-only testing to explain mobile failures, or "it feels faster" as a pass condition.&lt;/p&gt;

&lt;h3&gt;
  
  
  Triage
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Name the failing dataset in one sentence. If you cannot, gather minimum evidence first. Search Console gives field status by URL group; PageSpeed Insights gives lab diagnostics and some field context.&lt;/li&gt;
&lt;li&gt;Confirm scope. Treat it as template-based until proven otherwise. Template scope means high-impact fixes; narrow scope points at conditional triggers.&lt;/li&gt;
&lt;li&gt;Classify the dominant constraint: server/delivery, render path, main thread, layout stability, third-party, or change-driven.&lt;/li&gt;
&lt;li&gt;Pick the smallest test that could prove you wrong. Field failing but lab fine means expand representativeness. LCP failing means confirm the element first. Regressed after deploy means investigate the change before optimizing.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwkeyd1grdkc6ix624enm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwkeyd1grdkc6ix624enm.png" alt="page speed results mobile" width="800" height="366"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  What actually moves the field number
&lt;/h3&gt;

&lt;p&gt;A faster render path for primary content, less main-thread contention, reserved space to stop layout shift, more consistent delivery so cache and routing produce less variance, and removing or conditioning third-party work. Lab scores can improve without any of that moving, if you optimized the test setup instead of the experience. Use lab to prove a bottleneck is gone, then validate with the same field evidence you began with.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fuxijxhmfgysd6imfg85e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fuxijxhmfgysd6imfg85e.png" alt="gtmetrix page speed report" width="800" height="393"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>performance</category>
      <category>seo</category>
      <category>javascript</category>
    </item>
    <item>
      <title>Architecting a Wazuh SIEM with active response: the four layers that matter</title>
      <dc:creator>Jeremy Burgos</dc:creator>
      <pubDate>Tue, 16 Jun 2026 10:30:00 +0000</pubDate>
      <link>https://dev.to/jeremy-burgos/architecting-a-wazuh-siem-with-active-response-the-four-layers-that-matter-4pbg</link>
      <guid>https://dev.to/jeremy-burgos/architecting-a-wazuh-siem-with-active-response-the-four-layers-that-matter-4pbg</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;TL;DR: A Wazuh SIEM and XDR deployment built across four layers (manager and agents, active response, network monitoring, traffic filtering) to move an environment from reactive log review to automated detection and response. Sanitized rules and decoders are public, linked at the end.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The architecture, in the order the layers depend on each other:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Layer 1  Manager + agents      -&amp;gt; broad endpoint coverage, secure agent comms, central dashboard
Layer 2  Active response       -&amp;gt; auto-block/mitigate brute force and unauthorized access
Layer 3  Network monitoring    -&amp;gt; real-time log/network watch, correlation across signals
Layer 4  Traffic filtering     -&amp;gt; block known-bad IPs, domains, signatures; updated on intel
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before this deployment, the environment looked like most: fragmented visibility, log review that depended on someone actually doing it, and no automated response for the threats that show up constantly, brute force, unauthorized access, malicious traffic. That means you find out late and every response starts cold. The goal was not to install a platform. It was to build a monitoring and response layer that acts on its own.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 1: manager, agents, and the dashboard
&lt;/h3&gt;

&lt;p&gt;The foundation is the Wazuh manager with agents deployed across endpoints for broad coverage, a dashboard centralizing alerting and rule management, and secure communication between manager and agents so the monitoring data itself cannot be tampered with in transit. Get this layer wrong and everything above it is built on unreliable signal.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ff6vu75wxj8pugo2ucoir.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ff6vu75wxj8pugo2ucoir.png" alt="wazuh dashboard screenshot"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 2: active response, and where it earns its keep
&lt;/h3&gt;

&lt;p&gt;Active response rules target higher-risk behavior, brute force, unauthorized access, and are designed to block or mitigate automatically. The part that separates a useful deployment from a noisy one is the refinement: the rules get tested and tuned to stay reliable while keeping false positives down. Active response that fires on legitimate activity gets ignored within a week, which is worse than no automation at all, because now nobody trusts the alerts either.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 3: network and log monitoring
&lt;/h3&gt;

&lt;p&gt;Real-time monitoring of system logs and network activity watches for the patterns that matter: unusual logins, unexpected outbound connections, privilege escalations. Existing monitoring signals are integrated so the system can correlate across sources instead of treating each in isolation. Correlation is where you catch the things a single log never shows you.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 4: traffic filtering
&lt;/h3&gt;

&lt;p&gt;Custom filtering rules detect and block unwanted patterns, known malicious IPs, suspicious domains, harmful signatures, and they get updated as threat intelligence evolves so the deployment stays current instead of going stale. Static filtering rules are a snapshot of last quarter's threats.&lt;/p&gt;

&lt;h3&gt;
  
  
  What changed
&lt;/h3&gt;

&lt;p&gt;The environment moved from reactive to structured. Detection improved through real-time visibility into logs, traffic, and endpoint activity. Mitigation got faster because active response shrinks the window between a threat appearing and something being done about it. Network control tightened through filtering. And it integrated without disrupting operations, which is the part that makes it sustainable rather than a one-time project that decays.&lt;/p&gt;

&lt;p&gt;The full deployment, with a video walkthrough of the actual setup and configuration, is at &lt;a href="https://www.jeremyburgos.com/projects/wazuh-siem-and-xdr-deployment/" rel="noopener noreferrer"&gt;https://www.jeremyburgos.com/projects/wazuh-siem-and-xdr-deployment/&lt;/a&gt;. Sanitized detection rules and decoders are public at &lt;a href="https://github.com/Jeremy-Burgos/wazuh-detection-engineering" rel="noopener noreferrer"&gt;https://github.com/Jeremy-Burgos/wazuh-detection-engineering&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjs5vfaxmbsl2wvn73xs5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjs5vfaxmbsl2wvn73xs5.png" alt="github jeremy burgos wazuh overview"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Originally published at &lt;a href="https://www.jeremyburgos.com/projects/wazuh-siem-and-xdr-deployment/" rel="noopener noreferrer"&gt;https://www.jeremyburgos.com/projects/wazuh-siem-and-xdr-deployment/&lt;/a&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>devops</category>
      <category>tutorial</category>
      <category>monitoring</category>
    </item>
    <item>
      <title>Architecting isolated workspaces with Kasm: hardened, egress-controlled, disposable</title>
      <dc:creator>Jeremy Burgos</dc:creator>
      <pubDate>Tue, 09 Jun 2026 10:30:00 +0000</pubDate>
      <link>https://dev.to/jeremy-burgos/architecting-isolated-workspaces-with-kasm-hardened-egress-controlled-disposable-2nmk</link>
      <guid>https://dev.to/jeremy-burgos/architecting-isolated-workspaces-with-kasm-hardened-egress-controlled-disposable-2nmk</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;TL;DR: A hardened Kasm Workspace deployment where every research session runs in an isolated, egress-controlled, throwaway container. Four layers (server hardening, VPN egress, instances and tools, monitoring), built so sensitive workloads never weaken the host.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The architecture:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Layer 1  Server hardening   -&amp;gt; locked-down base, secure headers, minimal mgmt surface
Layer 2  Egress VPN         -&amp;gt; all instance traffic through encrypted tunnels, no raw outbound
Layer 3  Instances + tools  -&amp;gt; AlmaLinux / Parrot / Ubuntu, Brave, SpiderFoot, Forensic OSINT
Layer 4  Monitoring         -&amp;gt; logging, access controls, periodic security review
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Most people do their riskiest browsing in their most valuable environment, the same browser logged into client accounts, email, and banking. This deployment exists to separate the two: a controlled environment that can handle research, OSINT, and testing without weakening the machine or network underneath it. The design had to clear four bars at once, isolation, controlled egress, multi-OS support, and monitoring, which a standard server setup does not.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 1: server hardening
&lt;/h3&gt;

&lt;p&gt;The server is hardened from the ground up, with secure header controls and a locked-down management interface, so the environment can be administered without exposing extra attack surface. This is the base everything else sits on.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 2: egress VPN
&lt;/h3&gt;

&lt;p&gt;Instance traffic is routed through encrypted VPN tunnels rather than direct, unmanaged outbound paths, with authentication and encryption tightened so access and data flows stay protected. The containers are aligned to use secure outbound routing consistently, so nothing leaks out a side door.&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 3: instances and tools
&lt;/h3&gt;

&lt;p&gt;Multiple operating systems, AlmaLinux, Parrot OS, Ubuntu, plus cloud browsers, Brave, Firefox, Chromium, with research tools including SpiderFoot and Forensic OSINT integrated for investigative work. Each session is its own container, so the work is sandboxed and disposable.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fa5xc4yq9h0tc07973nmd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fa5xc4yq9h0tc07973nmd.png" alt="kasm dashboard setup"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 4: monitoring and review
&lt;/h3&gt;

&lt;p&gt;Stricter access controls reduce exposure, and logging plus periodic security review keep the environment stable and trustworthy over time, rather than relying on the initial configuration holding forever.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it changes about the work
&lt;/h3&gt;

&lt;p&gt;The obvious win is privacy: research traffic moves through encrypted tunnels and a closed session leaves nothing behind. The less obvious win is cleaner observation. The moment your research environment is disposable and isolated, you stop confusing your own logged-in, ad-profiled browser with a neutral instrument, and you see closer to what a fresh visitor or a crawler sees. The discipline underneath it, separate sensitive work from disposable work, treat environments as ephemeral, assume new things are untrusted, carries well beyond research.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj0zlroy6lprlcipnp3sk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj0zlroy6lprlcipnp3sk.png" alt="kasm live example using almalinux"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Honest tradeoff: this is not a five-minute setup. It runs on real infrastructure (a hardened DigitalOcean server in this case) and took deliberate work across hardening, VPN, instances, and monitoring. If you do not run your own infrastructure, a clean browser profile with no extensions and no signed-in accounts, on a separate device or VM, gets you a meaningful share of the benefit for far less effort.&lt;/p&gt;

&lt;p&gt;The full build, with a video walkthrough of the setup and configuration, is at &lt;a href="https://www.jeremyburgos.com/projects/secure-kasm-workspace-deployment/" rel="noopener noreferrer"&gt;https://www.jeremyburgos.com/projects/secure-kasm-workspace-deployment/&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Originally published at &lt;a href="https://www.jeremyburgos.com/projects/secure-kasm-workspace-deployment/" rel="noopener noreferrer"&gt;https://www.jeremyburgos.com/projects/secure-kasm-workspace-deployment/&lt;/a&gt;&lt;/p&gt;

</description>
      <category>security</category>
      <category>devops</category>
      <category>tutorial</category>
      <category>privacy</category>
    </item>
    <item>
      <title>Four HTTP security headers every WordPress site should set</title>
      <dc:creator>Jeremy Burgos</dc:creator>
      <pubDate>Tue, 02 Jun 2026 10:30:00 +0000</pubDate>
      <link>https://dev.to/jeremy-burgos/four-http-security-headers-every-wordpress-site-should-set-27pd</link>
      <guid>https://dev.to/jeremy-burgos/four-http-security-headers-every-wordpress-site-should-set-27pd</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;TL;DR: Four response headers, a few minutes of work, most of the header-level security gap closed. Exact values below, plus a one-line curl to check any site.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Run this against your own site first:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-I&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; https://yoursite.com | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="nt"&gt;-E&lt;/span&gt; &lt;span class="s1"&gt;'strict-transport|x-content|x-frame|referrer'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Whatever does not come back is your to-do list. These four headers are public on every request and contain nothing sensitive, so you can check mine, I can check yours, and neither of us has to log into anything. Here is what each one is and the value I actually run in production.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F20jf0z700wmscyls9lm7.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F20jf0z700wmscyls9lm7.png" alt="terminal curl output" width="800" height="358"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Strict-Transport-Security
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;strict-transport-security: max-age=31536000; includeSubDomains; preload
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tells the browser to use HTTPS for your domain, full stop, for the max-age window. Once a browser has seen it, typing &lt;code&gt;http://&lt;/code&gt; does nothing; the browser refuses to send the insecure request. &lt;code&gt;max-age=31536000&lt;/code&gt; is one year. &lt;code&gt;includeSubDomains&lt;/code&gt; pushes the rule to every subdomain, closing the gap where an attacker targets some forgotten staging host. &lt;code&gt;preload&lt;/code&gt; is the part people skip, and it matters: without it, the very first request before the browser has ever seen your header can still go out over HTTP, and that first request is the attack window. Preloaded domains skip it because the browser ships already knowing your domain is HTTPS-only. Submit once at hstspreload.org, it is free, and inclusion rides the Chromium release train so it takes a few weeks.&lt;/p&gt;

&lt;h3&gt;
  
  
  X-Content-Type-Options
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;x-content-type-options: nosniff
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Browsers used to guess at content types when the server was vague, which is exploitable. &lt;code&gt;nosniff&lt;/code&gt; tells the browser to trust the declared Content-Type and stop guessing. There is no other value and nothing to tune. If your production site is missing this, you can fix it before you finish this article.&lt;/p&gt;

&lt;h3&gt;
  
  
  X-Frame-Options
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;x-frame-options: SAMEORIGIN
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your clickjacking defense. It stops someone loading your real, logged-in site in an invisible iframe and floating their own buttons over it. &lt;code&gt;SAMEORIGIN&lt;/code&gt; allows only your own pages to frame your site. &lt;code&gt;DENY&lt;/code&gt; blocks all framing including yours, which is wrong if your WordPress setup uses internal iframes (Elementor previews, some widgets). The modern successor is CSP's &lt;code&gt;frame-ancestors&lt;/code&gt;; run both during the transition, X-Frame-Options for older clients, CSP for the rest.&lt;/p&gt;

&lt;h3&gt;
  
  
  Referrer-Policy
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;referrer-policy: strict-origin-when-cross-origin
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Controls what your site leaks about users when they click away. &lt;code&gt;strict-origin-when-cross-origin&lt;/code&gt; sends the full URL on same-origin requests so your analytics still work, only the origin on cross-origin HTTPS so you are not leaking that someone was on &lt;code&gt;/account/billing&lt;/code&gt;, and nothing on cross-origin HTTP. Set it explicitly so you are not at the mercy of whatever default the next browser release ships.&lt;/p&gt;

&lt;h3&gt;
  
  
  Verifying
&lt;/h3&gt;

&lt;p&gt;The curl above is the fastest check; all four lines should come back. In a browser, DevTools, Network tab, click the document request, read Response Headers. For a letter grade, &lt;a href="https://securityheaders.com" rel="noopener noreferrer"&gt;securityheaders.com&lt;/a&gt; scores you against a known rubric. One quirk: these four alone land a B, and you reach A only once you add Content-Security-Policy.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fmaoaagr9qebye0s7oggm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fmaoaagr9qebye0s7oggm.png" alt="security headers result example" width="800" height="399"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0bn8y2ilcxr46icwhloo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0bn8y2ilcxr46icwhloo.png" alt="dev tools example" width="799" height="388"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;These four are the floor. The next layer is Permissions-Policy and CSP in report-only mode. But if you only ever do these four, you have closed most of the gap, in minutes.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>security</category>
      <category>wordpress</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
