<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: AumCreate</title>
    <description>The latest articles on DEV Community by AumCreate (@aumcreate).</description>
    <link>https://dev.to/aumcreate</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4197587%2F7b2bc397-b01b-4b9d-8bc5-55d08b4e9a31.jpg</url>
      <title>DEV Community: AumCreate</title>
      <link>https://dev.to/aumcreate</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aumcreate"/>
    <language>en</language>
    <item>
      <title>Is your body text too small and too grey? An accessibility pass you can ship this week</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Thu, 05 Nov 2026 04:29:01 +0000</pubDate>
      <link>https://dev.to/aumcreate/is-your-body-text-too-small-and-too-grey-an-accessibility-pass-you-can-ship-this-week-8ih</link>
      <guid>https://dev.to/aumcreate/is-your-body-text-too-small-and-too-grey-an-accessibility-pass-you-can-ship-this-week-8ih</guid>
      <description>&lt;p&gt;A paragraph set at 13px in &lt;code&gt;#999&lt;/code&gt; on a white card looks calm in a mockup and is unreadable on a phone held outdoors. It also fails the contrast rule on its own: &lt;code&gt;#999999&lt;/code&gt; on white is 2.85:1.&lt;/p&gt;

&lt;p&gt;None of what follows needs a redesign. It needs a floor — a minimum type size, a contrast rule, alt text written on purpose, and a focus ring nobody deletes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Body text: pick a floor and keep it
&lt;/h2&gt;

&lt;p&gt;Sixteen pixels is the absolute floor for body copy, and on a marketing page 18px reads better than 16. Secondary text — captions, metadata, form hints — should not go below 14px. Line height of 1.5 to 1.6 for anything longer than a sentence, and a measure of roughly 60 to 75 characters per line.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight css"&gt;&lt;code&gt;&lt;span class="nd"&gt;:root&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="py"&gt;--text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;#1a1a1a&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="py"&gt;--text-muted&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;#5a5a5a&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="py"&gt;--step-0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1.125rem&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c"&gt;/* 18px body */&lt;/span&gt;
  &lt;span class="py"&gt;--step--1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0.875rem&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;  &lt;span class="c"&gt;/* 14px, the floor for secondary text */&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nt"&gt;body&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;font-size&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;var&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="l"&gt;--step-0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nl"&gt;line-height&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1.6&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;color&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;var&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="l"&gt;--text&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nl"&gt;background&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;#ffffff&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nc"&gt;.prose&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;max-width&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;65ch&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nc"&gt;.meta&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;font-size&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;var&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="l"&gt;--step--1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nl"&gt;color&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;var&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="l"&gt;--text-muted&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c"&gt;/* ~6.9:1 on white */&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Set the viewport without blocking zoom. &lt;code&gt;user-scalable=no&lt;/code&gt; and &lt;code&gt;maximum-scale=1&lt;/code&gt; are the two values that stop people from enlarging text on a phone, and both still show up in starter templates.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;meta&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"viewport"&lt;/span&gt; &lt;span class="na"&gt;content=&lt;/span&gt;&lt;span class="s"&gt;"width=device-width, initial-scale=1"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Contrast: the numbers, and where they apply
&lt;/h2&gt;

&lt;p&gt;Normal-size text needs 4.5:1 against its background. Large text — 24px and up, or 18.66px and up when bold — needs 3:1, and so do borders, icons and control outlines that carry meaning.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;#767676&lt;/code&gt; is the lightest grey that still clears 4.5:1 on white, at 4.54:1. One step lighter, &lt;code&gt;#999999&lt;/code&gt;, is 2.85:1 and fails. That single pair of hex values explains most of the grey text on the web.&lt;/p&gt;

&lt;p&gt;Things that get forgotten in the audit:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Placeholder text inside inputs is text. It needs to pass, and it also should not be the only label.&lt;/li&gt;
&lt;li&gt;Text laid over a photo needs a solid or near-solid overlay behind it. A gradient that works on the desktop crop will not work on the mobile crop.&lt;/li&gt;
&lt;li&gt;Links inside body copy need more than a color difference. Underline them, or the link disappears for anyone who cannot separate the two hues.&lt;/li&gt;
&lt;li&gt;Required fields marked only in red are invisible to a chunk of your readers. Add the word.&lt;/li&gt;
&lt;li&gt;Disabled buttons are exempt from the contrast rule, which makes "disable until valid" a bad pattern for a primary action. Let the button work, validate on submit, and show a readable message.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Check every color pair you actually ship, not the ones in the palette file — hover states, footer links on the dark band, and the text inside a filled button included.&lt;/p&gt;

&lt;h2&gt;
  
  
  Alt text is about purpose, not description
&lt;/h2&gt;

&lt;p&gt;The question is not "what is in this picture" but "what is this picture doing here".&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Decorative shapes, background flourishes, spacer images: &lt;code&gt;alt=""&lt;/code&gt;. An empty alt is not laziness; it is the instruction to skip the image.&lt;/li&gt;
&lt;li&gt;A logo at the top of the page: the company name. Nothing else.&lt;/li&gt;
&lt;li&gt;A product photo in a grid: the product and the variant a person would use to tell it apart — material, size, color.&lt;/li&gt;
&lt;li&gt;An icon that is the only content of a link: the destination. Better still, put visible text next to it and drop the image from the accessible name.&lt;/li&gt;
&lt;li&gt;A chart: a short alt naming what it plots, plus a real sentence or table of the numbers nearby. No alt text can carry a chart.&lt;/li&gt;
&lt;li&gt;Captions and alt text should not repeat each other. If the caption says it, the alt does not need to.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Skip "image of" and "photo of" — the screen reader already says it is an image.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keyboard: unplug the mouse and use the page
&lt;/h2&gt;

&lt;p&gt;Tab from the address bar to the last link in the footer. Every stop should show a visible ring, in an order that matches the reading order, with nothing skipped and nothing trapped.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight css"&gt;&lt;code&gt;&lt;span class="nd"&gt;:focus-visible&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;outline&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;3px&lt;/span&gt; &lt;span class="nb"&gt;solid&lt;/span&gt; &lt;span class="nx"&gt;#0b5fff&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c"&gt;/* ~5.1:1 on white, clears the 3:1 non-text rule */&lt;/span&gt;
  &lt;span class="nl"&gt;outline-offset&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2px&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a stylesheet in the project contains &lt;code&gt;outline: none&lt;/code&gt; or &lt;code&gt;outline: 0&lt;/code&gt; on &lt;code&gt;:focus&lt;/code&gt;, that is the bug. Replace it with the rule above rather than deleting the line and hoping.&lt;/p&gt;

&lt;p&gt;A skip link is four lines and saves a keyboard user from tabbing through the whole nav on every page.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;a&lt;/span&gt; &lt;span class="na"&gt;class=&lt;/span&gt;&lt;span class="s"&gt;"skip"&lt;/span&gt; &lt;span class="na"&gt;href=&lt;/span&gt;&lt;span class="s"&gt;"#main"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Skip to content&lt;span class="nt"&gt;&amp;lt;/a&amp;gt;&lt;/span&gt;
&lt;span class="c"&gt;&amp;lt;!-- ... --&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;main&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"main"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight css"&gt;&lt;code&gt;&lt;span class="nc"&gt;.skip&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;position&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;absolute&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;left&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;top&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;transform&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;translateY&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;-120%&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nl"&gt;background&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;#ffffff&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;color&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;#1a1a1a&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;padding&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0.75rem&lt;/span&gt; &lt;span class="m"&gt;1rem&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;z-index&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;100&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nc"&gt;.skip&lt;/span&gt;&lt;span class="nd"&gt;:focus-visible&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;transform&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;translateY&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use &lt;code&gt;&amp;lt;button&amp;gt;&lt;/code&gt; for anything that changes something on the page and &lt;code&gt;&amp;lt;a href&amp;gt;&lt;/code&gt; for anything that goes somewhere. A clickable &lt;code&gt;&amp;lt;div&amp;gt;&lt;/code&gt; is unreachable and unannounced; if you must keep it, add &lt;code&gt;role="button"&lt;/code&gt;, &lt;code&gt;tabindex="0"&lt;/code&gt; and a keydown handler for Enter and Space, then ask why the &lt;code&gt;&amp;lt;button&amp;gt;&lt;/code&gt; was not an option.&lt;/p&gt;

&lt;p&gt;For dialogs, a native &lt;code&gt;&amp;lt;dialog&amp;gt;&lt;/code&gt; opened with &lt;code&gt;showModal()&lt;/code&gt; keeps focus inside and closes on Escape, which removes most of the hand-written focus-trap code. For menus and pickers, native &lt;code&gt;&amp;lt;select&amp;gt;&lt;/code&gt; and &lt;code&gt;&amp;lt;details&amp;gt;&lt;/code&gt; behave correctly with a keyboard by default. Hover-only navigation is unusable on a keyboard and on a touch screen at the same time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Form errors have to be spoken
&lt;/h2&gt;

&lt;p&gt;An error message that only changes color is an error message half your readers never receive. Tie the message to the field and let it be announced when it appears.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Email&lt;span class="nt"&gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;input&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt; &lt;span class="na"&gt;type=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt; &lt;span class="na"&gt;autocomplete=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt;
       &lt;span class="na"&gt;aria-describedby=&lt;/span&gt;&lt;span class="s"&gt;"email-error"&lt;/span&gt; &lt;span class="na"&gt;aria-invalid=&lt;/span&gt;&lt;span class="s"&gt;"true"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;p&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"email-error"&lt;/span&gt; &lt;span class="na"&gt;role=&lt;/span&gt;&lt;span class="s"&gt;"alert"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Enter an email address, for example name@company.com.&lt;span class="nt"&gt;&amp;lt;/p&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Render the container empty on page load and fill it on submit. A message that is inserted into an existing live region is announced; one that arrives already populated inside freshly injected markup is less reliable.&lt;/p&gt;

&lt;p&gt;Motion is the other thing people turn off. A short media query covers the carousels, the parallax and the smooth scrolling in one go.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight css"&gt;&lt;code&gt;&lt;span class="k"&gt;@media&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prefers-reduced-motion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;reduce&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="o"&gt;*,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nd"&gt;::before&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nd"&gt;::after&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;animation-duration&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0.01ms&lt;/span&gt; &lt;span class="cp"&gt;!important&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;animation-iteration-count&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="cp"&gt;!important&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;transition-duration&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;0.01ms&lt;/span&gt; &lt;span class="cp"&gt;!important&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;scroll-behavior&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;auto&lt;/span&gt; &lt;span class="cp"&gt;!important&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The pass to run before you publish
&lt;/h2&gt;

&lt;p&gt;Work through this on the real staging site, not on a component library page.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tab from the browser chrome to the end of the page with the mouse untouched. Visible ring at every stop, order matches the layout, no trap in a modal, no stop on a decorative element.&lt;/li&gt;
&lt;li&gt;Zoom the browser to 200%. Nothing overlaps, nothing is clipped, no content disappears.&lt;/li&gt;
&lt;li&gt;Narrow the window to 320px wide. No sideways page scroll — intentional scroll regions like a data table are fine.&lt;/li&gt;
&lt;li&gt;Run a contrast checker on every text and background pair you ship, including placeholders, captions and hover states. 4.5:1 for body text, 3:1 for large text and for borders that carry meaning.&lt;/li&gt;
&lt;li&gt;Turn on a screen reader for five minutes: NVDA on Windows, or VoiceOver on macOS with Cmd+F5. Listen to the navigation and the contact form. You are checking whether the labels make sense out loud, not whether every element is perfect.&lt;/li&gt;
&lt;li&gt;Submit the contact form empty. The error is announced, the field is described by it, and focus lands somewhere sensible.&lt;/li&gt;
&lt;li&gt;Open the accessibility panel in devtools and read the heading outline. One &lt;code&gt;h1&lt;/code&gt;, no skipped levels, headings used for structure rather than for size.&lt;/li&gt;
&lt;li&gt;Look at every image on the page: it either has alt text that says what it is for, or &lt;code&gt;alt=""&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Check tap targets on a phone. 24×24 CSS pixels is the floor with spacing, and primary actions want more room than that.&lt;/li&gt;
&lt;li&gt;Finish with an automated scan and clear what it flags. It catches the mechanical failures — missing labels, low contrast, bad ARIA — and it cannot tell you whether your alt text is meaningful. That part is still yours.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>a11y</category>
      <category>css</category>
      <category>ux</category>
    </item>
    <item>
      <title>A plugin and theme update routine: snapshot, pin, rehearse, then check what broke</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Wed, 04 Nov 2026 08:13:19 +0000</pubDate>
      <link>https://dev.to/aumcreate/a-plugin-and-theme-update-routine-snapshot-pin-rehearse-then-check-what-broke-3gf4</link>
      <guid>https://dev.to/aumcreate/a-plugin-and-theme-update-routine-snapshot-pin-rehearse-then-check-what-broke-3gf4</guid>
      <description>&lt;p&gt;Most update damage comes from the order of operations. Two plugins updated in the same minute, no snapshot, no record of the version that was running before — that turns a rollback of a few minutes into an evening of guessing which change broke checkout.&lt;/p&gt;

&lt;p&gt;The routine below is what I would run on any WordPress site where an update can cost money. It isn't tied to a host or to a particular plugin.&lt;/p&gt;

&lt;h2&gt;
  
  
  Snapshot the files and the database together
&lt;/h2&gt;

&lt;p&gt;Both, from the same moment. A file backup taken while an order is being written and a database dump taken ten minutes later can disagree about which plugins exist, and restoring that combination produces errors that look like a broken update.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp db &lt;span class="nb"&gt;export &lt;/span&gt;db-&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; +%F-%H%M&lt;span class="si"&gt;)&lt;/span&gt;.sql
&lt;span class="nb"&gt;tar&lt;/span&gt; &lt;span class="nt"&gt;-czf&lt;/span&gt; uploads-&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; +%F-%H%M&lt;span class="si"&gt;)&lt;/span&gt;.tar.gz wp-content/uploads
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few details that matter more than the commands:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;wp db export&lt;/code&gt; writes into the current directory. Move the dump off the server before you start updating.&lt;/li&gt;
&lt;li&gt;Files means more than &lt;code&gt;wp-content&lt;/code&gt;: &lt;code&gt;wp-config.php&lt;/code&gt;, any &lt;code&gt;.htaccess&lt;/code&gt; customisations, and drop-ins like an object cache file are part of what you are restoring.&lt;/li&gt;
&lt;li&gt;A host-level snapshot is convenient, but find the restore button before you need it. Do one restore onto a copy, once, and note how long it took.&lt;/li&gt;
&lt;li&gt;Sanity-check the dump: &lt;code&gt;grep -c 'CREATE TABLE' db-*.sql&lt;/code&gt; should be a plausible number, and the file size shouldn't be a few kilobytes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the site takes orders, switch on maintenance mode for the duration of the dump so nothing writes mid-export, then switch it off.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pin the versions you can't afford to lose
&lt;/h2&gt;

&lt;p&gt;Record what is running before you change it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp plugin list &lt;span class="nt"&gt;--fields&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;name,status,version &lt;span class="nt"&gt;--format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;csv &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; plugins-before.csv
wp theme list  &lt;span class="nt"&gt;--fields&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;name,status,version &lt;span class="nt"&gt;--format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;csv &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; themes-before.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Keep the previous zip for anything that touches checkout, authentication, or page rendering. WP-CLI can put an exact version back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp plugin &lt;span class="nb"&gt;install&lt;/span&gt; &amp;lt;slug&amp;gt; &lt;span class="nt"&gt;--version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&amp;lt;x.y.z&amp;gt; &lt;span class="nt"&gt;--force&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And stop those plugins from updating themselves while you sleep:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nf"&gt;add_filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'auto_update_plugin'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$update&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$item&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nv"&gt;$pinned&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt; &lt;span class="s1"&gt;'your-plugin-slug'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'another-plugin-slug'&lt;/span&gt; &lt;span class="p"&gt;];&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;in_array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$item&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;slug&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$pinned&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;$update&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Themes take the same arguments through &lt;code&gt;auto_update_theme&lt;/code&gt;. Core is controlled separately by the &lt;code&gt;WP_AUTO_UPDATE_CORE&lt;/code&gt; constant, where minor-only is usually the sensible setting.&lt;/p&gt;

&lt;p&gt;Pinning is not the same as never updating. A pinned plugin still gets updated — on your schedule, with the old zip within reach.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rehearse on a copy carrying production data
&lt;/h2&gt;

&lt;p&gt;A staging site with three sample pages won't reproduce the failure you care about. Clone the production database and the plugin set, then:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp plugin update &lt;span class="nt"&gt;--all&lt;/span&gt; &lt;span class="nt"&gt;--dry-run&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The dry run lists what would change and touches nothing. Run it for real on the copy afterwards, and click through the paths that earn money: add to cart, checkout, the contact form, login, search.&lt;/p&gt;

&lt;p&gt;Watch two logs while you click:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;tail&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; wp-content/debug.log
&lt;span class="nb"&gt;tail&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;php &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="s1"&gt;'echo ini_get("error_log");'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;debug.log&lt;/code&gt; only exists when &lt;code&gt;WP_DEBUG&lt;/code&gt; and &lt;code&gt;WP_DEBUG_LOG&lt;/code&gt; are on. Turn them on for the copy, and keep &lt;code&gt;WP_DEBUG_DISPLAY&lt;/code&gt; off so warnings don't land in the rendered page.&lt;/p&gt;

&lt;p&gt;Update in batches — core, then plugins, then the theme — with a check between each. When something breaks you want to know which batch caused it, and "all at once" throws that information away.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check the things that fail quietly
&lt;/h2&gt;

&lt;p&gt;Some breakage is loud: a white screen, a fatal in the log. The rest is subtle, and it surfaces days later as a missing order or a form that never emailed anyone.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Forms: submit one and confirm the entry reached the database or the inbox.&lt;/li&gt;
&lt;li&gt;Checkout: a complete test order, including the confirmation email.&lt;/li&gt;
&lt;li&gt;Login and password reset.&lt;/li&gt;
&lt;li&gt;The REST endpoint a block or theme depends on: &lt;code&gt;curl -s -o /dev/null -w '%{http_code}' "$(wp option get home)/wp-json/"&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Sitemap and feeds: &lt;code&gt;curl -sI "$(wp option get home)/wp-sitemap.xml" | head -n 1&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Cron: &lt;code&gt;wp cron event list&lt;/code&gt;. Updates occasionally reschedule or drop events, and a missed task shows up later as stale imports or no email.&lt;/li&gt;
&lt;li&gt;Caches: page cache, object cache, CDN. Purge everything after the last batch — a stale cached page hides both the fix and the failure.&lt;/li&gt;
&lt;li&gt;Database version: if core asks for an upgrade, snapshot again immediately before &lt;code&gt;wp core update-db&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Image sizes: if a plugin changed its registered sizes, thumbnails may need regenerating.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Diffing the CSVs answers "what actually changed" faster than scrolling the admin:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp plugin list &lt;span class="nt"&gt;--fields&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;name,status,version &lt;span class="nt"&gt;--format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;csv &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; plugins-after.csv
diff plugins-before.csv plugins-after.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Roll back files and database together
&lt;/h2&gt;

&lt;p&gt;Plugin updates run database migrations. Restoring only the plugin folder can leave the database ahead of the code, which shows up as missing tables or a fatal on the next request.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp db import db-YYYY-MM-DD-HHMM.sql
wp plugin &lt;span class="nb"&gt;install&lt;/span&gt; &amp;lt;slug&amp;gt; &lt;span class="nt"&gt;--version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&amp;lt;x.y.z&amp;gt; &lt;span class="nt"&gt;--force&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That restore discards everything written after the dump was taken — which is exactly why the snapshot goes immediately before the update, inside the maintenance window, and not the night before.&lt;/p&gt;

&lt;p&gt;If the site genuinely can't go down, the fallback is to restore files only and then check whether the plugin's migration is safe to re-run. That's a per-plugin judgement call, and a decent argument for preferring plugins that don't migrate on update in the first place.&lt;/p&gt;

&lt;h2&gt;
  
  
  A routine short enough to run every time
&lt;/h2&gt;

&lt;p&gt;A procedure you skip under time pressure is worse than no procedure. The version that fits in ten minutes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Export the database and archive uploads; move both off the server.&lt;/li&gt;
&lt;li&gt;Write plugin and theme versions to CSV.&lt;/li&gt;
&lt;li&gt;Update on the copy first, then click through forms, checkout, login and search.&lt;/li&gt;
&lt;li&gt;Update production in the same order, one batch at a time, in a low-traffic window.&lt;/li&gt;
&lt;li&gt;Purge caches; check the error log, cron, sitemap and REST endpoint.&lt;/li&gt;
&lt;li&gt;Keep the previous zips and the dump until the next round of updates.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nothing here is clever. It's the ordering that saves the evening, not the tooling.&lt;/p&gt;

</description>
      <category>software</category>
      <category>webdev</category>
      <category>wordpress</category>
    </item>
    <item>
      <title>When an extra llms.txt or schema plugin is worth it next to the SEO plugin you already run</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Wed, 04 Nov 2026 08:08:35 +0000</pubDate>
      <link>https://dev.to/aumcreate/when-an-extra-llmstxt-or-schema-plugin-is-worth-it-next-to-the-seo-plugin-you-already-run-13p4</link>
      <guid>https://dev.to/aumcreate/when-an-extra-llmstxt-or-schema-plugin-is-worth-it-next-to-the-seo-plugin-you-already-run-13p4</guid>
      <description>&lt;p&gt;A page that emits two &lt;code&gt;Organization&lt;/code&gt; nodes with different &lt;code&gt;@id&lt;/code&gt; values is the most common symptom of two plugins describing the same business. The graph becomes ambiguous: a consumer has to guess which node is the subject of the page, and any property that appears on only one of them may be read as missing.&lt;/p&gt;

&lt;p&gt;The same test decides whether an &lt;code&gt;llms.txt&lt;/code&gt; plugin is redundant. If nothing on the site serves that path today, a plugin that adds it is the first producer, not a second one. If two things register the route, one file is served and neither admin screen tells you which one answered.&lt;/p&gt;

&lt;p&gt;So: read the published output first, then decide what to install.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find out what the page already publishes
&lt;/h2&gt;

&lt;p&gt;Counting blocks is a start:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-s&lt;/span&gt; https://example.com/about/ | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'application/ld+json'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A count tells you how many blocks exist, not what is inside them. This script walks every block, follows &lt;code&gt;@graph&lt;/code&gt;, and groups nodes by &lt;code&gt;@type&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="cp"&gt;&amp;lt;?php&lt;/span&gt;
&lt;span class="c1"&gt;// scan-jsonld.php — list every JSON-LD node on a page and flag duplicate @type entries.&lt;/span&gt;
&lt;span class="nv"&gt;$url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; &lt;span class="nv"&gt;$url&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nb"&gt;fwrite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="no"&gt;STDERR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'usage: php scan-jsonld.php https://example.com/page/'&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="kc"&gt;PHP_EOL&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nv"&gt;$ch&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;curl_init&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$url&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nb"&gt;curl_setopt_array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$ch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="no"&gt;CURLOPT_RETURNTRANSFER&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="no"&gt;CURLOPT_FOLLOWLOCATION&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="no"&gt;CURLOPT_TIMEOUT&lt;/span&gt;        &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nv"&gt;$html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;curl_exec&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$ch&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nv"&gt;$html&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nb"&gt;fwrite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="no"&gt;STDERR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;curl_error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$ch&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="kc"&gt;PHP_EOL&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="nb"&gt;curl_close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$ch&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nb"&gt;preg_match_all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'#&amp;lt;script[^&amp;gt;]*application/ld\+json[^&amp;gt;]*&amp;gt;(.*?)&amp;lt;/script&amp;gt;#is'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$matches&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nv"&gt;$nodes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
&lt;span class="nv"&gt;$walk&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$src&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;use&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nv"&gt;$walk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nv"&gt;$nodes&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; &lt;span class="nb"&gt;is_array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="k"&gt;isset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'@graph'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;is_array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'@graph'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'@graph'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$child&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nv"&gt;$walk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$child&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$src&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="k"&gt;isset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'@type'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nv"&gt;$nodes&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="s1"&gt;'type'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;implode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'|'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;array&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'@type'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="s1"&gt;'id'&lt;/span&gt;   &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'@id'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="s1"&gt;'(none)'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s1"&gt;'src'&lt;/span&gt;  &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$src&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;];&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$value&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nb"&gt;is_array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$value&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nv"&gt;$walk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$src&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$matches&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$i&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$raw&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nv"&gt;$data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;json_decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nb"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$raw&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; &lt;span class="nb"&gt;is_array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$data&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'block '&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="nv"&gt;$i&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="s1"&gt;': not valid JSON'&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="kc"&gt;PHP_EOL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nv"&gt;$walk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'block '&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="nv"&gt;$i&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nv"&gt;$byType&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
&lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$nodes&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nv"&gt;$byType&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'type'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;][]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$node&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$byType&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$type&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$list&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nv"&gt;$ids&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;array_unique&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nb"&gt;array_column&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$list&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'id'&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nv"&gt;$flag&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$ids&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="s1"&gt;'  &amp;lt;-- same @type, different @id'&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'%-26s %d node(s)  ids: %s%s'&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="kc"&gt;PHP_EOL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$list&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nb"&gt;implode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;', '&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$ids&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nv"&gt;$flag&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it against a handful of representative URLs: home, a service page, a product page, a post, an archive. Duplicates are rarely site-wide. A theme that injects &lt;code&gt;BreadcrumbList&lt;/code&gt; on posts plus a plugin that injects it everywhere produces two nodes on some templates and one on the rest, which is why testing only the home page hides it.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;llms.txt&lt;/code&gt; and sitemaps need the same treatment:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-sI&lt;/span&gt; https://example.com/llms.txt | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; 1        &lt;span class="c"&gt;# 200 means something already serves it&lt;/span&gt;
curl &lt;span class="nt"&gt;-s&lt;/span&gt; https://example.com/robots.txt | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="s1"&gt;'^sitemap'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two &lt;code&gt;Sitemap:&lt;/code&gt; lines pointing at different indexes is noisy but rarely harmful. Two blocks declaring &lt;code&gt;WebSite&lt;/code&gt; with different &lt;code&gt;@id&lt;/code&gt; values is the case worth fixing.&lt;/p&gt;

&lt;h2&gt;
  
  
  What real duplication looks like
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Same &lt;code&gt;@type&lt;/code&gt;, different &lt;code&gt;@id&lt;/code&gt;: two entities where there should be one. Remove one source.&lt;/li&gt;
&lt;li&gt;Same &lt;code&gt;@type&lt;/code&gt;, same &lt;code&gt;@id&lt;/code&gt;, conflicting properties (&lt;code&gt;telephone&lt;/code&gt;, &lt;code&gt;address&lt;/code&gt;, &lt;code&gt;price&lt;/code&gt;): worse, because one of the values is now simply wrong and you can't tell which from the outside.&lt;/li&gt;
&lt;li&gt;A &lt;code&gt;BreadcrumbList&lt;/code&gt; that doesn't match the visible breadcrumb: a rendering problem that also leaks into search results.&lt;/li&gt;
&lt;li&gt;An &lt;code&gt;Article&lt;/code&gt; node whose &lt;code&gt;dateModified&lt;/code&gt; no human edited.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A node that only one source produces is not duplication, even if the plugin you already run is supposed to cover that type. Configuration differs per site; the HTML is the ground truth, not the feature list.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where a second plugin earns its place
&lt;/h2&gt;

&lt;p&gt;The type you need isn't modelled. A full-suite SEO plugin covers the common vocabulary — organization, website, article, breadcrumb, sometimes product — and typically has no field for &lt;code&gt;Service&lt;/code&gt; with &lt;code&gt;areaServed&lt;/code&gt;, no way to express product variants as separate &lt;code&gt;Offer&lt;/code&gt; nodes, and nothing that ties an &lt;code&gt;FAQPage&lt;/code&gt; to the questions actually visible on the page. If your pages describe services by region, that gap is the reason to add something.&lt;/p&gt;

&lt;p&gt;You need per-page control. One graph for every URL of a post type is convenient until two pages need different structures. A plugin that lets you edit the node for a single page solves that without touching the rest.&lt;/p&gt;

&lt;p&gt;Nothing serves &lt;code&gt;llms.txt&lt;/code&gt;. It is a proposed convention rather than a ratified standard, and no major crawler documents that it reads one, so the effect is unproven. Publishing it costs nothing and conflicts with nothing as long as it is the only producer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it just duplicates
&lt;/h2&gt;

&lt;p&gt;If the new plugin emits &lt;code&gt;Organization&lt;/code&gt;, &lt;code&gt;WebSite&lt;/code&gt;, &lt;code&gt;BreadcrumbList&lt;/code&gt; and &lt;code&gt;Article&lt;/code&gt; on the same URLs the suite already covers, you are adding weight and a second set of facts. Meta descriptions and XML sitemaps behave the same way: two plugins writing a &lt;code&gt;description&lt;/code&gt; tag means one wins per page by hook priority, and nobody documented which.&lt;/p&gt;

&lt;p&gt;A rough map — verify against your own output rather than trusting the middle column:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Need&lt;/th&gt;
&lt;th&gt;Full-suite SEO plugin&lt;/th&gt;
&lt;th&gt;Extra plugin justified?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Title, meta description, canonical&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;XML sitemap&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Organization / WebSite&lt;/td&gt;
&lt;td&gt;usually&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BreadcrumbList&lt;/td&gt;
&lt;td&gt;usually&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Article basics&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Service&lt;/code&gt; with &lt;code&gt;areaServed&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;rarely&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Product variants as separate offers&lt;/td&gt;
&lt;td&gt;partial&lt;/td&gt;
&lt;td&gt;maybe&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Per-page node editing&lt;/td&gt;
&lt;td&gt;limited&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;llms.txt&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;usually not&lt;/td&gt;
&lt;td&gt;yes, if nothing else serves it&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  If you add one, remove the overlap first
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Re-run the scanner and save the output.&lt;/li&gt;
&lt;li&gt;Switch off the overlapping module in one of the two. Most full-suite plugins let you disable their schema output; one that doesn't is a reason to prefer a different arrangement.&lt;/li&gt;
&lt;li&gt;Re-run the scanner. The node count should drop and nothing you need should disappear.&lt;/li&gt;
&lt;li&gt;Install the new plugin, re-run the scanner, compare the list of &lt;code&gt;@type&lt;/code&gt; values rather than the counts.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;One plugin that works this shape is &lt;a href="https://aumcreate.com/plugins/aumviso?utm_source=devto&amp;amp;utm_medium=content&amp;amp;utm_campaign=devto-llms-schema-plugin-overlap" rel="noopener noreferrer"&gt;AumViso&lt;/a&gt;: it emits &lt;code&gt;llms.txt&lt;/code&gt;, structured data, a sitemap, meta tags and internal links. If your suite already produces the sitemap and the meta tags, the honest reason to install it is the structured data and the &lt;code&gt;llms.txt&lt;/code&gt;, and you would turn the suite's schema module off in the same session.&lt;/p&gt;

&lt;p&gt;The scanner is the whole test. Same URL, same script, before and after. If the second run shows the same &lt;code&gt;@type&lt;/code&gt; list with more nodes, the plugin added nothing; if it shows a type that wasn't there before, it did its job.&lt;/p&gt;

</description>
      <category>llm</category>
      <category>seo</category>
      <category>wordpress</category>
    </item>
    <item>
      <title>WP-Cron vs system cron: what to check when WordPress scheduled tasks start missing</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Tue, 03 Nov 2026 06:51:10 +0000</pubDate>
      <link>https://dev.to/aumcreate/wp-cron-vs-system-cron-what-to-check-when-wordpress-scheduled-tasks-start-missing-1kfj</link>
      <guid>https://dev.to/aumcreate/wp-cron-vs-system-cron-what-to-check-when-wordpress-scheduled-tasks-start-missing-1kfj</guid>
      <description>&lt;p&gt;A renewal email lands four hours after the invoice goes out. A nightly import stops for six days and nobody notices until the numbers are wrong. Nothing in the logs says cron failed, because nothing failed in a way that logs.&lt;/p&gt;

&lt;p&gt;WP-Cron is a due-check, not a scheduler. On each page request WordPress looks at the &lt;code&gt;cron&lt;/code&gt; option, finds events whose timestamp has passed, and fires a non-blocking loopback request to &lt;code&gt;wp-cron.php&lt;/code&gt; to run them. No traffic, no runs. That is the whole failure mode, and it covers most "the plugin stopped working" reports.&lt;/p&gt;

&lt;h2&gt;
  
  
  Confirm the event is late
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp cron event list &lt;span class="nt"&gt;--fields&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;hook,next_run_gmt,recurrence,args
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;next_run_gmt&lt;/code&gt; is UTC. Compare it with &lt;code&gt;date -u&lt;/code&gt;, not with the clock in your site's timezone settings. An event with a &lt;code&gt;next_run_gmt&lt;/code&gt; hours in the past is overdue. An event that is missing from the list was never scheduled, which is a different bug: plugin inactive, scheduling hooked to admin-only code, or something called &lt;code&gt;wp_clear_scheduled_hook&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Look at the spawn lock
&lt;/h2&gt;

&lt;p&gt;Before spawning, WordPress writes a &lt;code&gt;doing_cron&lt;/code&gt; transient and holds it for &lt;code&gt;WP_CRON_LOCK_TIMEOUT&lt;/code&gt; seconds, 60 by default. That lock is why a dead run doesn't cause every page load to start another one, and it is also why a dead run silences the next minute of attempts.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp transient get doing_cron
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A value that persists across several minutes means something is still holding it. Clearing it is safe once you know no run is in flight:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp transient delete doing_cron
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Check that spawning is enabled
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp config get DISABLE_WP_CRON
wp config get ALTERNATE_WP_CRON
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;DISABLE_WP_CRON&lt;/code&gt; set to true means page loads never trigger anything. That is correct when a system cron exists, and a silent outage when someone copied the setting from a tutorial and the crontab line never got added.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test the loopback request
&lt;/h2&gt;

&lt;p&gt;The spawn is an HTTP request from the server to itself. Basic auth, a maintenance plugin, a WAF rule on &lt;code&gt;wp-cron.php&lt;/code&gt;, or a hosts-file mismatch kills it while the site still looks fine to visitors.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp &lt;span class="nb"&gt;eval&lt;/span&gt; &lt;span class="s1"&gt;'var_dump( wp_remote_retrieve_response_code( wp_remote_get( add_query_arg( "nocache", time(), site_url( "wp-cron.php" ) ), array( "timeout" =&amp;gt; 5 ) ) ) );'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You want &lt;code&gt;200&lt;/code&gt;. This call does run any due events, which is harmless and a little useful. Tools → Site Health → Info has a loopback test that tells you the same thing without typing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find the other scheduler
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;crontab &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;span class="nb"&gt;sudo&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; www-data crontab &lt;span class="nt"&gt;-l&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hosting panels add entries under their own user. Two crons calling &lt;code&gt;wp cron event run --due-now&lt;/code&gt; is survivable, since an event only fires once per due timestamp, but two crons with different paths or one running as root against a different &lt;code&gt;wp-config.php&lt;/code&gt; produces behaviour that looks random.&lt;/p&gt;

&lt;h2&gt;
  
  
  Switch
&lt;/h2&gt;

&lt;p&gt;In &lt;code&gt;wp-config.php&lt;/code&gt;, above the "stop editing" line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nb"&gt;define&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'DISABLE_WP_CRON'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the crontab of the user that owns the files, not root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;*&lt;/span&gt;/5 &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; flock &lt;span class="nt"&gt;-n&lt;/span&gt; /var/lock/example-wp-cron.lock &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'cd /var/www/example.com/current &amp;amp;&amp;amp; /usr/local/bin/wp cron event run --due-now --quiet'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What each part is doing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;cd&lt;/code&gt; into the install so &lt;code&gt;wp&lt;/code&gt; finds &lt;code&gt;wp-config.php&lt;/code&gt; without a &lt;code&gt;--path&lt;/code&gt; flag.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;flock -n&lt;/code&gt; returns immediately when the previous run is still going. Without it a slow run overlaps the next one and two processes write the same options.&lt;/li&gt;
&lt;li&gt;Run as the site user. &lt;code&gt;--allow-root&lt;/code&gt; works and leaves root-owned files in &lt;code&gt;wp-content&lt;/code&gt;, which breaks uploads and plugin updates later.&lt;/li&gt;
&lt;li&gt;Five minutes is a reasonable default. Each invocation boots WordPress, so a small site has a floor of roughly one run per minute.&lt;/li&gt;
&lt;li&gt;A time-critical hook can get its own tighter line (&lt;code&gt;wp cron event run my_hook&lt;/code&gt;) without changing the general interval.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Without shell access, a panel cron or an external cron service hitting a URL works:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://example.com/wp-cron.php?doing_wp_cron=1699999999
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Include a changing value in the query string or a CDN may serve a cached response indefinitely. Test it once and confirm a known event moved in &lt;code&gt;wp cron event list&lt;/code&gt;; whether a direct hit to &lt;code&gt;wp-cron.php&lt;/code&gt; still executes events while &lt;code&gt;DISABLE_WP_CRON&lt;/code&gt; is on is not something to assume across versions and hosts.&lt;/p&gt;

&lt;p&gt;Shared hosting without SSH, a panel cron nobody documented, a site that changed hands with no notes: in any of those the switch above takes longer than ten minutes, because it turns into a conversation with the host. Most support desks will add a crontab line if you hand them the exact command and the path. Moving to a host that gives you shell access is the other clean route, and it is a reasonable moment to have the site rebuilt properly: AumCreate does a fixed-scope WordPress deployment onto hosting you already own — template installed, every page rewritten with your copy, a preview link, one round of changes, then live, with a one-page note on how to edit content — for a one-time $399 (&lt;a href="https://aumcreate.com/setup?utm_source=devto&amp;amp;utm_medium=content&amp;amp;utm_campaign=devto-wp-cron-vs-system-cron" rel="noopener noreferrer"&gt;what the handover covers&lt;/a&gt;). Domain and hosting stay in your name and are billed by those providers, and there is no annual fee from us. It is a build, so if the cron tasks are the actual problem, ask for the scheduled events and who owns them to be written into the handover.&lt;/p&gt;

&lt;h2&gt;
  
  
  What breaks quietly afterwards
&lt;/h2&gt;

&lt;p&gt;Queues that are not WP-Cron. WooCommerce and a long list of plugins put background work into Action Scheduler, which keeps its own tables and its own runner. &lt;code&gt;wp cron event run --due-now&lt;/code&gt; does not drain that queue. Check the scheduled-actions screen after the switch, and add a deterministic trigger if you want one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;*&lt;/span&gt;/5 &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;cd&lt;/span&gt; /var/www/example.com/current &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; /usr/local/bin/wp action-scheduler run &lt;span class="nt"&gt;--quiet&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Long events blocking the batch. &lt;code&gt;wp cron event run --due-now&lt;/code&gt; runs due events in a single process. One event stuck on a remote API eats the run and everything behind it waits for the next tick. Batch heavy work — 50 rows at a time, store a cursor, let the next invocation pick up the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make a missed run visible
&lt;/h2&gt;

&lt;p&gt;Record the last successful run, and take a lock so a slow run cannot double-process:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nf"&gt;add_action&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'myplugin_sync'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;global&lt;/span&gt; &lt;span class="nv"&gt;$wpdb&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="nv"&gt;$got&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$wpdb&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;get_var&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nv"&gt;$wpdb&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;prepare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'SELECT GET_LOCK(%s, %d)'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'myplugin_sync'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'1'&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nv"&gt;$got&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// another run holds the lock&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// work here&lt;/span&gt;
        &lt;span class="nf"&gt;update_option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'myplugin_last_sync'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;finally&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nv"&gt;$wpdb&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nv"&gt;$wpdb&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;prepare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'SELECT RELEASE_LOCK(%s)'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'myplugin_sync'&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;GET_LOCK&lt;/code&gt; is per database connection, so it releases when the process dies, including a hard kill. That makes it a better fit than an option flag that survives a crash and blocks the next ten runs.&lt;/p&gt;

&lt;p&gt;Expose the age of that timestamp to whatever already watches the site:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nf"&gt;add_action&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'rest_api_init'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;register_rest_route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'health/v1'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'/cron'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="s1"&gt;'methods'&lt;/span&gt;             &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'GET'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'permission_callback'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'__return_true'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'callback'&lt;/span&gt;            &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nv"&gt;$last&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;get_option&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'myplugin_last_sync'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="nv"&gt;$age&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$last&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="nb"&gt;time&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nv"&gt;$last&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;PHP_INT_MAX&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;WP_REST_Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="k"&gt;array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'last_sync_age_seconds'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$age&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt;
                &lt;span class="nv"&gt;$age&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;
            &lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A monitor expecting &lt;code&gt;200&lt;/code&gt; from that endpoint every five minutes catches the failure &lt;code&gt;wp cron event list&lt;/code&gt; never shows: a cron entry that exists, runs on schedule, and does nothing useful. Pair it with an attempt counter and a cap, so a task that fails permanently lands somewhere readable instead of retrying forever with no record.&lt;/p&gt;

&lt;p&gt;The switch is ten minutes of work. The part that needs thinking is which events deserve a system cron at all: anything that moves money, sends mail, or expires access belongs there; a weekly transient cleanup does not.&lt;/p&gt;

</description>
      <category>automation</category>
      <category>debugging</category>
      <category>php</category>
      <category>wordpress</category>
    </item>
    <item>
      <title>How many fields does an inquiry form need before sales can quote?</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Tue, 03 Nov 2026 04:55:54 +0000</pubDate>
      <link>https://dev.to/aumcreate/how-many-fields-does-an-inquiry-form-need-before-sales-can-quote-1b2d</link>
      <guid>https://dev.to/aumcreate/how-many-fields-does-an-inquiry-form-need-before-sales-can-quote-1b2d</guid>
      <description>&lt;p&gt;A buyer types "please send price for 500 pcs" into a contact form late at night. The next morning a rep replies with a list of questions: which model, which country, when do you need it. The buyer answers two days later, or doesn't, and somewhere in between a competitor sent a number.&lt;/p&gt;

&lt;p&gt;Making the form longer does not fix that, and neither does making it shorter. The useful question is narrower: which answers does a price depend on? Everything else can move into the reply.&lt;/p&gt;

&lt;h2&gt;
  
  
  Write the quote before you design the form
&lt;/h2&gt;

&lt;p&gt;Get the person who actually quotes prices to produce three quotes for three recent inquiries, from scratch, without reopening the original email thread. Note every question they had to answer in order to reach a number. That list is your field list, and it is usually shorter than the form you already have.&lt;/p&gt;

&lt;p&gt;It tends to come out as five items: what the item is, how many, where it ships, when it is needed, and an address that reaches a human. Everything else is comfort.&lt;/p&gt;

&lt;h2&gt;
  
  
  Give each candidate field a job
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;th&gt;Can the quote be produced without it?&lt;/th&gt;
&lt;th&gt;First form?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model or part number&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Required&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quantity&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Required&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Delivery country and city&lt;/td&gt;
&lt;td&gt;No — freight and tax move with it&lt;/td&gt;
&lt;td&gt;Required&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rough timing&lt;/td&gt;
&lt;td&gt;Sometimes&lt;/td&gt;
&lt;td&gt;Dropdown, with an empty default&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Working email address&lt;/td&gt;
&lt;td&gt;No — there is nowhere to send the quote&lt;/td&gt;
&lt;td&gt;Required&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Company name&lt;/td&gt;
&lt;td&gt;Only if you sell to companies exclusively&lt;/td&gt;
&lt;td&gt;Required in B2B, optional otherwise&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contact name&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Optional&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Phone&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Optional, never required&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Free-text message&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Optional, marked as optional&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Budget&lt;/td&gt;
&lt;td&gt;No — and asking too early costs the inquiry&lt;/td&gt;
&lt;td&gt;Leave out&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;How did you hear about us&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Leave out, or ask on the thank-you page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tax or VAT number&lt;/td&gt;
&lt;td&gt;No, that is an onboarding step&lt;/td&gt;
&lt;td&gt;Leave out&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The test for anything not on this list: if a blank answer would not stop the quote, it belongs in the reply, not on the form. Budget is the one field people insist on and buyers rarely answer honestly at first contact — if the number matters, ask after you have shown you understand the requirement.&lt;/p&gt;

&lt;h2&gt;
  
  
  Friction is rarely the field count
&lt;/h2&gt;

&lt;p&gt;Six clean fields convert better than four bad ones. What kills inquiries:&lt;/p&gt;

&lt;p&gt;A required phone number, on a form used by buyers in another country who do not want a call. A country dropdown with every territory in it, on a phone. A required company name when the buyer is an individual sourcing one sample. A field that only accepts uploads, which fail on mobile data and give no feedback when they do. Validation that clears the form on error. A blank thank-you page that does not say what happens next.&lt;/p&gt;

&lt;p&gt;Every one of those is fixable without removing a question sales needs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The form itself
&lt;/h2&gt;

&lt;p&gt;Plain HTML, labels outside the inputs, input types that bring up the right keyboard on a phone:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;form&lt;/span&gt; &lt;span class="na"&gt;action=&lt;/span&gt;&lt;span class="s"&gt;"/inquiry"&lt;/span&gt; &lt;span class="na"&gt;method=&lt;/span&gt;&lt;span class="s"&gt;"post"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Work email&lt;span class="nt"&gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;input&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt; &lt;span class="na"&gt;type=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt; &lt;span class="na"&gt;autocomplete=&lt;/span&gt;&lt;span class="s"&gt;"email"&lt;/span&gt; &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;

  &lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"company"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Company &lt;span class="nt"&gt;&amp;lt;span&amp;gt;&lt;/span&gt;(optional)&lt;span class="nt"&gt;&amp;lt;/span&amp;gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;input&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"company"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"company"&lt;/span&gt; &lt;span class="na"&gt;type=&lt;/span&gt;&lt;span class="s"&gt;"text"&lt;/span&gt; &lt;span class="na"&gt;autocomplete=&lt;/span&gt;&lt;span class="s"&gt;"organization"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;

  &lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"item"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Model or part number&lt;span class="nt"&gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;input&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"item"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"item"&lt;/span&gt; &lt;span class="na"&gt;type=&lt;/span&gt;&lt;span class="s"&gt;"text"&lt;/span&gt; &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;

  &lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"qty"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Quantity&lt;span class="nt"&gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;input&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"qty"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"qty"&lt;/span&gt; &lt;span class="na"&gt;type=&lt;/span&gt;&lt;span class="s"&gt;"text"&lt;/span&gt; &lt;span class="na"&gt;inputmode=&lt;/span&gt;&lt;span class="s"&gt;"numeric"&lt;/span&gt; &lt;span class="na"&gt;autocomplete=&lt;/span&gt;&lt;span class="s"&gt;"off"&lt;/span&gt; &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;

  &lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"dest"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Delivery country and city&lt;span class="nt"&gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;input&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"dest"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"dest"&lt;/span&gt; &lt;span class="na"&gt;type=&lt;/span&gt;&lt;span class="s"&gt;"text"&lt;/span&gt; &lt;span class="na"&gt;autocomplete=&lt;/span&gt;&lt;span class="s"&gt;"address-level2"&lt;/span&gt; &lt;span class="na"&gt;required&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;

  &lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"when"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;When do you need it?&lt;span class="nt"&gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;select&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"when"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"when"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;option&lt;/span&gt; &lt;span class="na"&gt;value=&lt;/span&gt;&lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Not sure yet&lt;span class="nt"&gt;&amp;lt;/option&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;option&lt;/span&gt; &lt;span class="na"&gt;value=&lt;/span&gt;&lt;span class="s"&gt;"2w"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Within two weeks&lt;span class="nt"&gt;&amp;lt;/option&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;option&lt;/span&gt; &lt;span class="na"&gt;value=&lt;/span&gt;&lt;span class="s"&gt;"1m"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;This month&lt;span class="nt"&gt;&amp;lt;/option&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;option&lt;/span&gt; &lt;span class="na"&gt;value=&lt;/span&gt;&lt;span class="s"&gt;"1q"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;This quarter&lt;span class="nt"&gt;&amp;lt;/option&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;/select&amp;gt;&lt;/span&gt;

  &lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"notes"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Anything else &lt;span class="nt"&gt;&amp;lt;span&amp;gt;&lt;/span&gt;(optional)&lt;span class="nt"&gt;&amp;lt;/span&amp;gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;textarea&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"notes"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"notes"&lt;/span&gt; &lt;span class="na"&gt;rows=&lt;/span&gt;&lt;span class="s"&gt;"4"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&amp;lt;/textarea&amp;gt;&lt;/span&gt;

  &lt;span class="nt"&gt;&amp;lt;p&lt;/span&gt; &lt;span class="na"&gt;class=&lt;/span&gt;&lt;span class="s"&gt;"hp"&lt;/span&gt; &lt;span class="na"&gt;aria-hidden=&lt;/span&gt;&lt;span class="s"&gt;"true"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;label&lt;/span&gt; &lt;span class="na"&gt;for=&lt;/span&gt;&lt;span class="s"&gt;"website"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Website&lt;span class="nt"&gt;&amp;lt;/label&amp;gt;&lt;/span&gt;
    &lt;span class="nt"&gt;&amp;lt;input&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"website"&lt;/span&gt; &lt;span class="na"&gt;name=&lt;/span&gt;&lt;span class="s"&gt;"website"&lt;/span&gt; &lt;span class="na"&gt;type=&lt;/span&gt;&lt;span class="s"&gt;"text"&lt;/span&gt; &lt;span class="na"&gt;tabindex=&lt;/span&gt;&lt;span class="s"&gt;"-1"&lt;/span&gt; &lt;span class="na"&gt;autocomplete=&lt;/span&gt;&lt;span class="s"&gt;"off"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;/p&amp;gt;&lt;/span&gt;

  &lt;span class="nt"&gt;&amp;lt;button&lt;/span&gt; &lt;span class="na"&gt;type=&lt;/span&gt;&lt;span class="s"&gt;"submit"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;Get a quote&lt;span class="nt"&gt;&amp;lt;/button&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/form&amp;gt;&lt;/span&gt;

&lt;span class="nt"&gt;&amp;lt;style&amp;gt;&lt;/span&gt;
  &lt;span class="nc"&gt;.hp&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;position&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;absolute&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nl"&gt;left&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="m"&gt;-9999px&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/style&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notes on the choices. Quantity is &lt;code&gt;type="text"&lt;/code&gt; with &lt;code&gt;inputmode="numeric"&lt;/code&gt; rather than &lt;code&gt;type="number"&lt;/code&gt;, because number inputs change value on a scroll wheel and reject the way people write quantities. The timing dropdown has an empty first option so nobody is pushed into a false answer. The honeypot replaces a captcha: reject the submission server-side if &lt;code&gt;website&lt;/code&gt; is not empty, and skip the whole puzzle that real buyers fail on a train.&lt;/p&gt;

&lt;p&gt;If a file matters — a drawing, a bill of materials, a photo of a damaged part — ask for it in the confirmation, not on the form. By then the buyer has committed, and a link is easier than a failing upload widget.&lt;/p&gt;

&lt;p&gt;Keep the form to one column on every screen size. Preserve what was typed when a validation error fires, and put the error next to the field that caused it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Say what happens after submit
&lt;/h2&gt;

&lt;p&gt;The thank-you page is part of the form. Two things belong on it: what you will send and when, and the one or two details that would speed it up if they have them to hand. "We reply within one working day with a price and lead time. If you have a drawing or a target price, send it to this address and we will include it."&lt;/p&gt;

&lt;p&gt;That single paragraph removes most of the second round trip without adding a field.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read the messages for a fortnight
&lt;/h2&gt;

&lt;p&gt;The free-text box is a research tool. Read every message that arrives for two weeks and count how often the same question appears — "do you ship to X", "is this the same as model Y". Anything asked repeatedly is either a missing field, a missing explanation on the product page, or a shipping restriction nobody wrote down. Fix the page before you fix the form.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to measure afterwards
&lt;/h2&gt;

&lt;p&gt;The share of inquiries where the first reply could be the quote itself, with no clarifying email. And the share that reach your order system without anyone re-keying or chasing a missing detail. If both stay flat after trimming fields, the problem is upstream of the form — usually the product pages do not state what the buyer needs to know to describe their requirement.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hand this to whoever builds it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Required: model or part number, quantity, delivery country and city, email.&lt;/li&gt;
&lt;li&gt;Optional and visibly marked optional: company, name, free text.&lt;/li&gt;
&lt;li&gt;No required phone number, no required budget, no captcha, no required upload.&lt;/li&gt;
&lt;li&gt;Timing as a dropdown with an honest "not sure yet" option.&lt;/li&gt;
&lt;li&gt;Validation that keeps the typed values, with the error beside the field.&lt;/li&gt;
&lt;li&gt;A thank-you page that states the reply time and asks for attachments by return.&lt;/li&gt;
&lt;li&gt;Someone reads the free-text box weekly for the first month.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>product</category>
      <category>productivity</category>
      <category>ux</category>
    </item>
    <item>
      <title>One page, one question: how to structure headings and FAQ blocks so an answer engine can quote a passage</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Mon, 02 Nov 2026 04:38:43 +0000</pubDate>
      <link>https://dev.to/aumcreate/one-page-one-question-how-to-structure-headings-and-faq-blocks-so-an-answer-engine-can-quote-a-3bbd</link>
      <guid>https://dev.to/aumcreate/one-page-one-question-how-to-structure-headings-and-faq-blocks-so-an-answer-engine-can-quote-a-3bbd</guid>
      <description>&lt;p&gt;Someone asks an assistant a question. The assistant fetches a handful of pages, splits them into chunks, and keeps the chunk that looks like an answer. If your page buries the answer under four paragraphs of context, the chunk that survives is the context, and you don't get quoted.&lt;/p&gt;

&lt;p&gt;The unit that gets retrieved and quoted is the passage, not the page. Everything below follows from that: write pages whose passages survive being cut out and read alone.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a chunk looks like by the time it reaches the model
&lt;/h2&gt;

&lt;p&gt;Retrieval systems split HTML into pieces of a few hundred tokens, usually cutting near headings. The piece arrives without your navigation, without the paragraph before it, without the table of contents. A sentence that begins "as mentioned above" or "this approach" arrives with no referent.&lt;/p&gt;

&lt;p&gt;Test any paragraph by asking: if I paste only this block into an empty document, does it still say something true and complete? If not, it is a weak chunk.&lt;/p&gt;

&lt;h2&gt;
  
  
  Should a page answer one question or several?
&lt;/h2&gt;

&lt;p&gt;One. The heading, the opening line and the FAQ should all point at the same question, phrased the way a person would type it. A page that answers "how much does a 3-bedroom move cost in Singapore" will not also answer "how do I pack a kitchen" well enough to be quoted for either.&lt;/p&gt;

&lt;p&gt;Splitting is not a rule about word count. A short page answering two questions is worse than two short pages, because the retrieval system may pull half of the first answer and match it against the second question.&lt;/p&gt;

&lt;p&gt;When two of your pages answer the same question, merge them. Duplicate answers compete with each other and neither becomes the reference.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first sentence under a heading has to be the answer
&lt;/h2&gt;

&lt;p&gt;Readers who scan and extractors that chunk both want the answer in the first sentence, before the justification.&lt;/p&gt;

&lt;p&gt;Which version would you quote?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;## Why is my WordPress site slow?&lt;/span&gt;

Performance is a broad topic and there are many factors to consider. In this
article we will look at several areas that affect how quickly a page loads,
starting with the hosting environment and then moving on to caching...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;## Why is a WordPress site slow?&lt;/span&gt;

A WordPress page is slow when PHP rebuilds the page on every request and the
finished HTML is never stored. The usual causes are no page cache, an expensive
uncached query on each load, and images served at full resolution.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second version answers inside one sentence and stays understandable out of context. The first version only announces that an answer is coming.&lt;/p&gt;

&lt;p&gt;Delete the announcement paragraph entirely rather than shortening it. Openings like "in this article we will cover" have no value as a chunk.&lt;/p&gt;

&lt;h2&gt;
  
  
  Write chunks that name their own subject
&lt;/h2&gt;

&lt;p&gt;A passage that gets pulled out of your page loses its title. Pronouns lose their reference and the chunk becomes useless.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Instead of "it handles retries automatically", write "the queue handles retries automatically".&lt;/li&gt;
&lt;li&gt;Instead of "this approach scales", write "the queue-based import scales".&lt;/li&gt;
&lt;li&gt;Instead of "most people prefer the second option", name what the second option is.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Numbers need units and a date. "5 working days" beats "fast turnaround". "Checked in October 2026" tells the reader the number has an owner and an expiry, and both are things a model can repeat without distorting them.&lt;/p&gt;

&lt;h2&gt;
  
  
  How should FAQ blocks be arranged?
&lt;/h2&gt;

&lt;p&gt;An FAQ block is a set of small, self-contained answer chunks, and it works when you treat each item as its own page in miniature.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One question per item, phrased the way it gets searched. "Do you charge a yearly fee?" beats "Pricing notes".&lt;/li&gt;
&lt;li&gt;The first sentence answers. Detail, caveats, exceptions come after, and they can be dropped without breaking the answer.&lt;/li&gt;
&lt;li&gt;No nested sub-questions. If an item needs a follow-up question, it needs its own page.&lt;/li&gt;
&lt;li&gt;No filler openings. "Great question!" and "That's a common concern" are the first thing a chunk carries and the first thing a reader skips.&lt;/li&gt;
&lt;li&gt;Only questions someone actually asked. Six invented questions that pad a section produce six useless chunks and dilute the ones that matter.&lt;/li&gt;
&lt;li&gt;Keep the answers in the HTML source. Content injected by JavaScript after load is invisible to a fetcher that does not run scripts. A &lt;code&gt;&amp;lt;details&amp;gt;&lt;/code&gt; element is fine because the text is in the markup; a tab widget that loads its panel by request is not.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;h2&amp;gt;&lt;/span&gt;Frequently asked questions&lt;span class="nt"&gt;&amp;lt;/h2&amp;gt;&lt;/span&gt;

&lt;span class="nt"&gt;&amp;lt;details&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;summary&amp;gt;&lt;/span&gt;Does a heading have to be phrased as a question?&lt;span class="nt"&gt;&amp;lt;/summary&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;p&amp;gt;&lt;/span&gt;No, but a question heading matches how people search and gives the
  extractor a clean boundary. "Pricing" is a weaker heading than "How much
  does a delivery cost?"&lt;span class="nt"&gt;&amp;lt;/p&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/details&amp;gt;&lt;/span&gt;

&lt;span class="nt"&gt;&amp;lt;details&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;summary&amp;gt;&lt;/span&gt;Can an extractor read text inside a collapsed element?&lt;span class="nt"&gt;&amp;lt;/summary&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;p&amp;gt;&lt;/span&gt;Yes, when the text is in the HTML. A details element ships its content in
  the markup even while collapsed; a panel fetched by JavaScript after load
  does not.&lt;span class="nt"&gt;&amp;lt;/p&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/details&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Structured data for FAQ is optional. Markup can describe a page, but the text is what gets extracted and repeated. Adding schema to thin answers produces well-labelled thin answers.&lt;/p&gt;

&lt;h2&gt;
  
  
  A page skeleton you can copy
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gh"&gt;# How much does X cost?          &amp;lt;- the question, in the searcher's words&lt;/span&gt;

One or two sentences that answer it, with the number, the currency and
what is included. No preamble.

&lt;span class="gu"&gt;## What is included in the price?&lt;/span&gt;

The first sentence states the inclusion. Then the detail.

&lt;span class="gu"&gt;## What changes the price?&lt;/span&gt;

Named factors, each with a short explanation.

&lt;span class="gu"&gt;## How do I start?&lt;/span&gt;

The concrete next action, one step, not a menu of five.

&lt;span class="gu"&gt;## Frequently asked questions&lt;/span&gt;

Four to six items, each answer-first and self-contained.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notice the heading is not "Pricing". A heading that repeats the question gives the extractor a clean boundary and gives the reader a reason to stop scrolling.&lt;/p&gt;

&lt;h2&gt;
  
  
  Small things that break otherwise good passages
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Comparisons that say "the first option" without naming it, in a table where the column headers were cut off with the chunk.&lt;/li&gt;
&lt;li&gt;Answers that begin with "it depends" and never reach the actual answer. Put the dependency in the second sentence.&lt;/li&gt;
&lt;li&gt;Numbers with no date and no scope. A figure that was true last year, or true only for one project size, gets repeated as if it were general.&lt;/li&gt;
&lt;li&gt;A single page that is really six questions stapled together, each half-answered.&lt;/li&gt;
&lt;li&gt;Repeated boilerplate under every heading, so that three different chunks all start with the same sentence about the company.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Before you publish, read the first sentence of every section on its own
&lt;/h2&gt;

&lt;p&gt;Scroll through the page and read only the first sentence under each heading, with the heading attached. If those sentences form a coherent answer to the page's question, the page will survive chunking. If any of them needs the preceding paragraph to make sense, rewrite it.&lt;/p&gt;

&lt;p&gt;That check takes two minutes and catches most of what makes a page unquotable.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>seo</category>
      <category>writing</category>
    </item>
    <item>
      <title>Cleaning a product CSV before import: encoding, image URLs, variant rows, and finding the line that broke</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Mon, 02 Nov 2026 04:33:32 +0000</pubDate>
      <link>https://dev.to/aumcreate/cleaning-a-product-csv-before-import-encoding-image-urls-variant-rows-and-finding-the-line-that-521l</link>
      <guid>https://dev.to/aumcreate/cleaning-a-product-csv-before-import-encoding-image-urls-variant-rows-and-finding-the-line-that-521l</guid>
      <description>&lt;p&gt;An 800-row product sheet looks like a 20-minute import until the importer stops somewhere past the halfway mark and says the SKU is duplicated. It does not say which line. That is usually the moment people start deleting rows until something passes.&lt;/p&gt;

&lt;p&gt;Most of those failures come from the file, not the store. The encoding, the price column, the image URL column, the variation rows and the SKU column account for nearly all of them. Clean those in a script first and the import turns into a non-event.&lt;/p&gt;

&lt;p&gt;Everything below is plain Python 3 and works on any CSV you are about to feed into an admin importer. Only the column names are WooCommerce-flavoured.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;charset-normalizer requests
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Read the file as bytes and let the library guess
&lt;/h2&gt;

&lt;p&gt;A CSV exported from Excel on a Chinese or Japanese Windows machine is GB18030, not UTF-8. A CSV exported from Excel on macOS often has a UTF-8 BOM. Open a GB18030 file with &lt;code&gt;encoding="utf-8"&lt;/code&gt; and you get either a &lt;code&gt;UnicodeDecodeError&lt;/code&gt; on line 400 or, worse, mojibake that imports fine and renders as &lt;code&gt;Ã¤Â¸Â­&lt;/code&gt; on the product page three days later.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;io&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;unicodedata&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Path&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib.parse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urlparse&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;charset_normalizer&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;from_bytes&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;ZERO_WIDTH&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fromkeys&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;ord&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\u200b\u200c\u200d\ufeff&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;ODD_SPACES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\u00a0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\u2007&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\u202f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\u3000&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;decode_any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read_bytes&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;guess&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;from_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;best&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;guess&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no usable encoding for &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;guess&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\ufeff&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# drop the BOM if there is one
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;
    &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;unicodedata&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;normalize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;NFKC&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;translate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ZERO_WIDTH&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;good&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ODD_SPACES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;good&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[ \t\r\n]+&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;NFKC&lt;/code&gt; folds full-width letters and digits to ASCII, which is what you want in a SKU (&lt;code&gt;ＡＢ－１２&lt;/code&gt; becomes &lt;code&gt;AB-12&lt;/code&gt;). It also rewrites some typographic punctuation, so skip it on the description column if the client cares about curly quotes.&lt;/p&gt;

&lt;p&gt;Non-breaking spaces and zero-width characters are the other silent killer. They survive copy-paste from a chat app, look identical on screen, and make &lt;code&gt;SKU == "AB-12"&lt;/code&gt; false for no visible reason.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prices need one decision: which separator is the decimal one
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;1.299,00&lt;/code&gt;, &lt;code&gt;1,299.00&lt;/code&gt;, &lt;code&gt;1299&lt;/code&gt;, &lt;code&gt;$1299.00&lt;/code&gt;, &lt;code&gt;1299,00 EUR&lt;/code&gt; all arrive in the same file when several people filled in the sheet. The rule that survives contact with real data: if both separators appear, the rightmost one is the decimal separator. If only one appears and it is followed by exactly three digits, it is a thousands separator. Everything else is decimal.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;parse_price&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;empty&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[^\d.,\-]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;        &lt;span class="c1"&gt;# drop currency symbols and letters
&lt;/span&gt;    &lt;span class="n"&gt;neg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isdigit&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no digits in &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="si"&gt;!r}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

    &lt;span class="n"&gt;last_dot&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;last_comma&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rfind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rfind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;last_dot&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;last_comma&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;dec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;last_dot&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;last_comma&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="n"&gt;thou&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;dec&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;last_dot&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;last_comma&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;sep&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;last_dot&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="n"&gt;digits_after&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rfind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sep&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sep&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;digits_after&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;dec&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;thou&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;dec&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;thou&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sep&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;dec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;thou&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;thou&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;thou&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;dec&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;dec&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dec&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unparsable &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="si"&gt;!r}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="nf"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;neg&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run this before the import and eyeball the rows where the input contained a separator. A &lt;code&gt;1.299&lt;/code&gt; that should have been 1.299 euros and got read as 1299 is a refund, not a rounding error.&lt;/p&gt;

&lt;h2&gt;
  
  
  Image URLs fail in ways a browser hides
&lt;/h2&gt;

&lt;p&gt;A URL that opens in your browser can still break the import. The three that bite: a &lt;code&gt;content-type&lt;/code&gt; that is &lt;code&gt;text/html&lt;/code&gt; because the host serves a "hotlinking not allowed" page, an unescaped space from a filename like &lt;code&gt;red shoe.jpg&lt;/code&gt;, and an http URL on an https store (mixed content, image silently dropped).&lt;/p&gt;

&lt;p&gt;Check with a HEAD request, fall back to a streamed GET because plenty of CDNs answer HEAD with 403 or 405, and cache the result so 40 variations sharing one parent image do not make 40 requests.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;SESSION&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;SESSION&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;csv-preflight/1.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check_image&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;problem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urlparse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;scheme&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;problem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;not an absolute http(s) URL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;problem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unescaped space in URL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;problem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;host has no dot&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SESSION&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;head&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;allow_redirects&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;403&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;405&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;501&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
                &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SESSION&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;allow_redirects&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;problem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;HTTP &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;ctype&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content-type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;ctype&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;image/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
                    &lt;span class="n"&gt;problem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content-type &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ctype&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;missing&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RequestException&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;problem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__class__&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;   &lt;span class="c1"&gt;# Timeout, SSLError, TooManyRedirects
&lt;/span&gt;    &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;problem&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;problem&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Split gallery columns on commas before checking, and strip quotes around each part — &lt;code&gt;"img1.jpg, img2.jpg"&lt;/code&gt; pasted out of a spreadsheet carries both.&lt;/p&gt;

&lt;h2&gt;
  
  
  Variation rows are where the SKU logic breaks
&lt;/h2&gt;

&lt;p&gt;A variable product in a CSV is two kinds of row: the parent (&lt;code&gt;Type=variable&lt;/code&gt;) and one row per combination (&lt;code&gt;Type=variation&lt;/code&gt;) whose &lt;code&gt;Parent&lt;/code&gt; column holds the parent SKU. Four things go wrong, in this order of frequency:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a variation points at a parent SKU that exists in the shop but not in the file, or was renamed in one place only;&lt;/li&gt;
&lt;li&gt;two rows share a SKU, usually the parent's price row copied down into the variations;&lt;/li&gt;
&lt;li&gt;the variation has no attribute value, so the importer creates a combination that cannot be selected on the front end;&lt;/li&gt;
&lt;li&gt;the parent has no price and no variation has one either, so the product imports at 0.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Build the parent index first, then validate the variations against it. Do not validate in file order — a variation can legally sit above its parent.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check_skus_and_variants&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;parents&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;sku&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SKU&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;sku&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SKU&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;duplicate, first seen on line &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
            &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;variable&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;parents&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;sku&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;

    &lt;span class="n"&gt;attr_cols&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;header&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;attribute&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;variation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="n"&gt;parent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Parent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;parent&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;parent&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;parents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Parent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;parent SKU not present in this file&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;attr_cols&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;attr_cols&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;variation has no attribute value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Regular price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;parents&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Regular price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;no price on the variation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Finding the line that actually failed
&lt;/h2&gt;

&lt;p&gt;The importer's error message counts data rows, starting at 1 for the first row after the header. &lt;code&gt;csv.reader.line_num&lt;/code&gt; counts physical lines in the file, starting at 1 for the header. If any field contains a line break inside quotes — a description with &lt;code&gt;&amp;lt;br&amp;gt;&lt;/code&gt; typed as a real newline — the two numbers drift apart and you end up staring at the wrong record.&lt;/p&gt;

&lt;p&gt;Report both, and let the reader pick.&lt;/p&gt;

&lt;p&gt;There is a second, sharper signal: the column count. An unquoted comma inside a description pushes every later field one column to the right, so the row arrives with more fields than the header. Catch that before anything else, because every field-level check after it is meaningless.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;load_rows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;decode_any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;io&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;StringIO&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;header&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
    &lt;span class="n"&gt;good&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;rec&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;line_num&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rec&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rec&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;expected &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; columns, got &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rec&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="n"&gt;good&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rec&lt;/span&gt;&lt;span class="p"&gt;)}))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;good&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;report&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;import-report.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;load_rows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nf"&gt;check_skus_and_variants&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;cache&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;image&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'"'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()]:&lt;/span&gt;
                &lt;span class="n"&gt;why&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;check_image&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;why&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                    &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;why&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Regular price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;why&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;parse_price&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Regular price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;why&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Regular price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Regular price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;why&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

    &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;report&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;physical_line&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;column&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;value&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;problem&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; clean rows, &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; problems -&amp;gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;report&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A note on what the report means. &lt;code&gt;physical_line&lt;/code&gt; is the line in the file; the importer's row number is that minus one, minus any multiline rows above it. Sort the report by &lt;code&gt;physical_line&lt;/code&gt;, open the sheet, and look at the columns named — nine times out of ten the value in the &lt;code&gt;value&lt;/code&gt; column is visibly wrong once you are looking at the right cell.&lt;/p&gt;

&lt;p&gt;Fix the sheet, re-export as UTF-8 with all fields quoted, run the script again until the report is empty, then import. A clean file imports in one pass and, more importantly, the data that lands in the database is the same data you reviewed.&lt;/p&gt;

&lt;p&gt;We publish free WordPress plugins (translation, structured data, crawler visibility, digital-goods checkout) at &lt;a href="https://aumcreate.com/plugins?utm_source=devto&amp;amp;utm_medium=content&amp;amp;utm_campaign=devto-csv-import-field-cleaning" rel="noopener noreferrer"&gt;https://aumcreate.com/plugins?utm_source=devto&amp;amp;utm_medium=content&amp;amp;utm_campaign=devto-csv-import-field-cleaning&lt;/a&gt;&lt;/p&gt;

</description>
      <category>automation</category>
      <category>data</category>
      <category>python</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Auditing wp_options autoload, revisions and transients before you clean a WordPress database</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Mon, 02 Nov 2026 04:28:26 +0000</pubDate>
      <link>https://dev.to/aumcreate/auditing-wpoptions-autoload-revisions-and-transients-before-you-clean-a-wordpress-database-51ad</link>
      <guid>https://dev.to/aumcreate/auditing-wpoptions-autoload-revisions-and-transients-before-you-clean-a-wordpress-database-51ad</guid>
      <description>&lt;p&gt;Cleaning a WordPress database is easy to do in the wrong order. The commands that drop revisions and expired transients are one line each, and none of them ask whether the option you just un-autoloaded is read by a plugin on every page load. Audit first, delete second — that order is what lets you undo a mistake without restoring last night's backup.&lt;/p&gt;

&lt;p&gt;Everything below runs against a real database. Substitute your own table prefix wherever &lt;code&gt;wp_&lt;/code&gt; appears.&lt;/p&gt;

&lt;h2&gt;
  
  
  Take a copy you can actually restore from
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp db &lt;span class="nb"&gt;export &lt;/span&gt;pre-cleanup-&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; +%F&lt;span class="si"&gt;)&lt;/span&gt;.sql
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Move that file off the server. A dump sitting in the same web root is not a backup. If the site takes inquiries, do this at the quietest hour you have; a dump of a busy InnoDB database is a long read and it holds locks you don't want during business hours.&lt;/p&gt;

&lt;p&gt;Then look at where the space actually is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp db size &lt;span class="nt"&gt;--tables&lt;/span&gt; &lt;span class="nt"&gt;--human-readable&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;data_length&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;index_length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;mb&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;information_schema&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tables&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;table_schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data_length&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;index_length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;wp_options&lt;/code&gt; is near the top of that list on a site with a few hundred posts, the autoload column is the first thing to open.&lt;/p&gt;

&lt;h2&gt;
  
  
  wp_options: find out what ships with every request
&lt;/h2&gt;

&lt;p&gt;Autoloaded options are read into memory on every request — front end, admin, REST, cron. A large autoloaded payload is a cost you pay whether the visitor needed it or not.&lt;/p&gt;

&lt;p&gt;Newer schemas store autoload as &lt;code&gt;on&lt;/code&gt; / &lt;code&gt;off&lt;/code&gt; / &lt;code&gt;auto-on&lt;/code&gt; / &lt;code&gt;auto-off&lt;/code&gt;; older ones use &lt;code&gt;yes&lt;/code&gt; / &lt;code&gt;no&lt;/code&gt;. Check which you have before writing anything else:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;autoload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_options&lt;/span&gt; &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;autoload&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the total, and the biggest individual rows:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;options&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;LENGTH&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;option_value&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;kb&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_options&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;autoload&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'yes'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'on'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'auto-on'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;option_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;LENGTH&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;option_value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_options&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;autoload&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'yes'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'on'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'auto-on'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;bytes&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For every row near the top you need three answers before changing anything: who owns it, whether it is read on the front end, and whether it regenerates itself.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rn&lt;/span&gt; &lt;span class="s2"&gt;"the_option_name"&lt;/span&gt; wp-content/plugins wp-content/themes &lt;span class="nt"&gt;--include&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;*&lt;/span&gt;.php | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-20&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Crude, but it answers the ownership question faster than reading a settings screen. If nothing matches, the option probably belongs to something uninstalled. Note it down and handle it in a second pass, after you have confirmed the site still works without it.&lt;/p&gt;

&lt;p&gt;Turning autoload off is one update:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;wp_options&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;autoload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'off'&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;option_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'the_option_name'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On a schema that still uses &lt;code&gt;yes&lt;/code&gt; / &lt;code&gt;no&lt;/code&gt;, write &lt;code&gt;'no'&lt;/code&gt;. Then flush:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp cache flush
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Re-run the total query and write the number down. Come back in a week — some plugins call &lt;code&gt;update_option()&lt;/code&gt; on save and autoload can get set back on, so an option you turned off today may be back on after the next settings save. Worth knowing before you tell anyone the database is clean.&lt;/p&gt;

&lt;h2&gt;
  
  
  Revisions: measure by parent, not by total
&lt;/h2&gt;

&lt;p&gt;A revision is a full copy of &lt;code&gt;post_content&lt;/code&gt; in &lt;code&gt;wp_posts&lt;/code&gt;. The count is less interesting than which posts own the copies.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;revisions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;LENGTH&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;post_content&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;mb&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_posts&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;post_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'revision'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_parent&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;post_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;revisions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;LENGTH&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_content&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;mb&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_posts&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;wp_posts&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ID&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_parent&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'revision'&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_parent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_title&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;mb&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The top of that list is usually one or two long pages that get edited on every sales call. Before deleting anything, note two things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Autosaves are stored as revisions too, identified by &lt;code&gt;autosave&lt;/code&gt; in &lt;code&gt;post_name&lt;/code&gt;. Deleting them does not touch published content, but it will surprise someone who has a draft open in the editor right now.&lt;/li&gt;
&lt;li&gt;Custom post types and page-builder content keep their revisions in the same table, and are deleted the same way.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Delete in chunks so a large site does not exhaust memory or hit command-line argument limits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="nv"&gt;ids&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;wp post list &lt;span class="nt"&gt;--post_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;revision &lt;span class="nt"&gt;--format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;ids &lt;span class="nt"&gt;--posts_per_page&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;200&lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$ids&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;wp post delete &lt;span class="nv"&gt;$ids&lt;/span&gt; &lt;span class="nt"&gt;--force&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cap future growth in &lt;code&gt;wp-config.php&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nb"&gt;define&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="s1"&gt;'WP_POST_REVISIONS'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is a ceiling going forward. It does not remove what already exists, and it does not shrink the table — see the rebuild step at the end.&lt;/p&gt;

&lt;h2&gt;
  
  
  Transients: expired is safe, orphaned needs a decision
&lt;/h2&gt;

&lt;p&gt;With no persistent object cache, transients live in &lt;code&gt;wp_options&lt;/code&gt; as a &lt;code&gt;_transient_&amp;lt;key&amp;gt;&lt;/code&gt; row plus a &lt;code&gt;_transient_timeout_&amp;lt;key&amp;gt;&lt;/code&gt; row.&lt;/p&gt;

&lt;p&gt;Count the expired ones:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;expired&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_options&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;option_name&lt;/span&gt; &lt;span class="k"&gt;LIKE&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\_&lt;/span&gt;&lt;span class="s1"&gt;transient&lt;/span&gt;&lt;span class="se"&gt;\_&lt;/span&gt;&lt;span class="s1"&gt;timeout&lt;/span&gt;&lt;span class="se"&gt;\_&lt;/span&gt;&lt;span class="s1"&gt;%'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;option_value&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;UNIX_TIMESTAMP&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are dead. WordPress clears them on its own schedule eventually; doing it now is safe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp transient delete &lt;span class="nt"&gt;--expired&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Orphans are a different case — a value row with no timeout row at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;orphans&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_options&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;option_name&lt;/span&gt; &lt;span class="k"&gt;LIKE&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\_&lt;/span&gt;&lt;span class="s1"&gt;transient&lt;/span&gt;&lt;span class="se"&gt;\_&lt;/span&gt;&lt;span class="s1"&gt;%'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;option_name&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;LIKE&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\_&lt;/span&gt;&lt;span class="s1"&gt;transient&lt;/span&gt;&lt;span class="se"&gt;\_&lt;/span&gt;&lt;span class="s1"&gt;timeout&lt;/span&gt;&lt;span class="se"&gt;\_&lt;/span&gt;&lt;span class="s1"&gt;%'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;EXISTS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_options&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;option_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;CONCAT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'_transient_timeout_'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;SUBSTRING&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;option_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Some were set without an expiry on purpose; some were left behind by a plugin that stopped writing to them. Deleting an orphan costs a cache miss the next time the plugin asks for the value, so the risk is low. If the number is large, find the owning prefix first — a plugin that keeps state in a no-expiry transient may behave differently right after the flush.&lt;/p&gt;

&lt;p&gt;One check worth doing before you count these as waste: if the site runs Redis or Memcached, transients live in the object cache and these option rows are leftovers from before it was enabled. Flush and re-count to find out which world you are in.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp cache flush
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Rebuild the tables and verify
&lt;/h2&gt;

&lt;p&gt;Deleting rows does not shrink the files. On InnoDB tables with a lot of free space, a rebuild reclaims it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp db optimize
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is not free. It rebuilds tables and takes time proportional to their size, so run it in a quiet window rather than at 2pm on a Tuesday.&lt;/p&gt;

&lt;p&gt;Verify in this order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;wp option get siteurl
wp post list &lt;span class="nt"&gt;--post_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;page &lt;span class="nt"&gt;--fields&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;ID,post_title,post_status &lt;span class="nt"&gt;--posts_per_page&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;5
wp plugin list &lt;span class="nt"&gt;--status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;active
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then in a browser, with a cache-busting query string: the home page, the page that had the most revisions, the inquiry or checkout form, and the admin dashboard. Watch the browser console for 404s, and open the settings screen of any plugin whose options you touched — that is where a wrongly deleted option shows up first.&lt;/p&gt;

&lt;p&gt;Re-measure what you measured at the start:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;options&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;LENGTH&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;option_value&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;kb&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_options&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;autoload&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'yes'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'on'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'auto-on'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Compare the two numbers. If the autoloaded total barely moved, the database was not your bottleneck and the next place to look is the slow query log, not another cleanup plugin.&lt;/p&gt;

&lt;p&gt;The order that holds up: export, measure, identify the owner, change one class of rows at a time, verify the front end, rebuild, re-measure. Every step you skip is a step you redo from the backup.&lt;/p&gt;

&lt;p&gt;If you want an outside read on what the site actually serves after the rebuild — broken assets, redirect chains, pages that no longer resolve — a site check at &lt;a href="https://aumcreate.com/tools/site-check?utm_source=devto&amp;amp;utm_medium=content&amp;amp;utm_campaign=devto-wp-db-cleanup-order" rel="noopener noreferrer"&gt;https://aumcreate.com/tools/site-check?utm_source=devto&amp;amp;utm_medium=content&amp;amp;utm_campaign=devto-wp-db-cleanup-order&lt;/a&gt; covers that from the visitor's side.&lt;/p&gt;

</description>
      <category>database</category>
      <category>performance</category>
      <category>sql</category>
      <category>wordpress</category>
    </item>
    <item>
      <title>Which AI crawler is actually loading your server, and which one to rate-limit first</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Sun, 01 Nov 2026 14:28:31 +0000</pubDate>
      <link>https://dev.to/aumcreate/which-ai-crawler-is-actually-loading-your-server-and-which-one-to-rate-limit-first-375a</link>
      <guid>https://dev.to/aumcreate/which-ai-crawler-is-actually-loading-your-server-and-which-one-to-rate-limit-first-375a</guid>
      <description>&lt;p&gt;A site slows down, someone opens the access log, sees a wall of bot requests, and blocks them. Two days later the site is slow again and the list of bots has changed. The step that got skipped is the one that makes the answer stable: pricing what each crawler actually costs before you change anything.&lt;/p&gt;

&lt;p&gt;This is the sequence I'd run on a WordPress box, and it works the same on any LAMP-style stack. It ends with a ranked list and one change to make, instead of a firewall full of guesses.&lt;/p&gt;

&lt;h2&gt;
  
  
  First check whether the web tier is even the bottleneck
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/loadavg
top &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nt"&gt;-n1&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; %CPU | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-15&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the top CPU consumers are &lt;code&gt;php-fpm&lt;/code&gt; and &lt;code&gt;mysqld&lt;/code&gt;, HTTP requests caused this and the access log holds the evidence. If you see heavy &lt;code&gt;kswapd&lt;/code&gt;, &lt;code&gt;kcompactd&lt;/code&gt;, or &lt;code&gt;%wa&lt;/code&gt; near 100, memory or disk is the problem — rate-limiting a crawler buys you a few days and nothing else. Look at the PHP-FPM pool as well. All children busy with a non-zero listen queue means requests are queueing, so continue. Idle children while load is high means the time is going somewhere else: cron, a CLI import, a backup job.&lt;/p&gt;

&lt;h2&gt;
  
  
  Get timing into the log
&lt;/h2&gt;

&lt;p&gt;The default &lt;code&gt;combined&lt;/code&gt; format gives you bytes and user agent but no duration, and duration is what separates a crawler making 200 cheap requests from one making 200 expensive ones.&lt;/p&gt;

&lt;p&gt;nginx:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;log_format&lt;/span&gt; &lt;span class="s"&gt;main_ext&lt;/span&gt; &lt;span class="s"&gt;'&lt;/span&gt;&lt;span class="nv"&gt;$remote_addr&lt;/span&gt; &lt;span class="s"&gt;-&lt;/span&gt; &lt;span class="nv"&gt;$remote_user&lt;/span&gt; &lt;span class="s"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$time_local&lt;/span&gt;&lt;span class="s"&gt;]&lt;/span&gt; &lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$request&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt; &lt;span class="s"&gt;'&lt;/span&gt;
                    &lt;span class="s"&gt;'&lt;/span&gt;&lt;span class="nv"&gt;$status&lt;/span&gt; &lt;span class="nv"&gt;$body_bytes_sent&lt;/span&gt; &lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$http_referer&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt; &lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$http_user_agent&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt; &lt;span class="s"&gt;'&lt;/span&gt;
                    &lt;span class="s"&gt;'rt=&lt;/span&gt;&lt;span class="nv"&gt;$request_time&lt;/span&gt;&lt;span class="s"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;access_log&lt;/span&gt; &lt;span class="n"&gt;/var/log/nginx/access.log&lt;/span&gt; &lt;span class="s"&gt;main_ext&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;nginx &lt;span class="nt"&gt;-t&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; systemctl reload nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Apache: add &lt;code&gt;%D&lt;/code&gt; to a custom &lt;code&gt;LogFormat&lt;/code&gt;; it's the request duration in microseconds. Existing lines keep the old format, so give it an hour before reading.&lt;/p&gt;

&lt;h2&gt;
  
  
  Slice by user agent
&lt;/h2&gt;

&lt;p&gt;Both nginx and Apache &lt;code&gt;combined&lt;/code&gt; quote the request, the referer and the user agent. That makes &lt;code&gt;"&lt;/code&gt; a reliable field separator, and the UA the sixth field:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt;&lt;span class="s1"&gt;'"'&lt;/span&gt; &lt;span class="s1"&gt;'{print $6}'&lt;/span&gt; /var/log/nginx/access.log &lt;span class="se"&gt;\&lt;/span&gt;
  | &lt;span class="nb"&gt;sort&lt;/span&gt; | &lt;span class="nb"&gt;uniq&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-rn&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-20&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You'll get the usual mix: uptime monitors, search engines, your own scripts, and whatever AI crawlers your content attracted. Note the candidates, then move to cost.&lt;/p&gt;

&lt;h2&gt;
  
  
  Price each bot in worker seconds
&lt;/h2&gt;

&lt;p&gt;Hit counts flatter crawlers that are noisy but cheap. Multiply by time instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt;&lt;span class="s1"&gt;'"'&lt;/span&gt; &lt;span class="s1"&gt;'
  {
    split($3, s, " ")        # s[2]=status, s[3]=body bytes
    ua = $6
    rt = $NF; sub(/^rt=/, "", rt)
    hits[ua]++
    bytes[ua] += s[3]
    secs[ua]  += rt
  }
  END {
    for (u in hits)
      printf "%10.1f %8d %13d  %s\n", secs[u], hits[u], bytes[u], u
  }'&lt;/span&gt; /var/log/nginx/access.log | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-rn&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-20&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Columns: seconds of request handling, hits, bytes served, user agent. A crawler at 8,000 hits and 40 seconds is background noise. One at 900 hits and 600 seconds is your problem, and it's probably hitting something expensive.&lt;/p&gt;

&lt;p&gt;If the file is too large to parse in one pass, sample it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;shuf&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; 500000 access.log &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; sample.log
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ratios survive sampling. Absolute counts don't, so don't quote them anywhere.&lt;/p&gt;

&lt;h2&gt;
  
  
  Look at the paths the top bot requests
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt;&lt;span class="s1"&gt;'"'&lt;/span&gt; &lt;span class="s1"&gt;'$6 ~ /BotName/ {split($2, r, " "); print r[2]}'&lt;/span&gt; access.log &lt;span class="se"&gt;\&lt;/span&gt;
  | &lt;span class="nb"&gt;sort&lt;/span&gt; | &lt;span class="nb"&gt;uniq&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; &lt;span class="nt"&gt;-rn&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-20&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Replace &lt;code&gt;BotName&lt;/code&gt; with a token from the previous step. What you're looking for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Search results, faceted filters, cart and checkout URLs. Uncacheable, database-backed, and often the entire cost.&lt;/li&gt;
&lt;li&gt;Pagination deep in the archive, or tag pages that nothing links to.&lt;/li&gt;
&lt;li&gt;A single URL requested thousands of times, which is a crawler bug or a redirect loop more than a load problem.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the expensive paths are the same across several bots, fix the paths first. It's a smaller change and it applies to every visitor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prove causality before you throttle
&lt;/h2&gt;

&lt;p&gt;A user agent is a claim, and a throttle is cheap to test. Short-circuit one UA temporarily:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="c1"&gt;# temporary, inside the server block&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$http_user_agent&lt;/span&gt; &lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="sr"&gt;*&lt;/span&gt; &lt;span class="s"&gt;"BotName")&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kn"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;403&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reload, wait 20–30 minutes, and compare load average, PHP-FPM queue depth and database CPU against the same window before the change. If nothing moved, that bot wasn't the cause and you can drop the rule. Then check whether the UA is genuine rather than a scraper wearing it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dig +short &lt;span class="nt"&gt;-x&lt;/span&gt; 203.0.113.10        &lt;span class="c"&gt;# reverse lookup&lt;/span&gt;
dig +short crawl.example.com      &lt;span class="c"&gt;# forward lookup, should return the same IP&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Forward-confirmed reverse DNS is what separates the real crawler from someone running curl with a copied header. It matters most right before you write a permanent rule.&lt;/p&gt;

&lt;h2&gt;
  
  
  Order to act in
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Order&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;th&gt;Fixes&lt;/th&gt;
&lt;th&gt;How to verify&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Cache or deny the expensive paths&lt;/td&gt;
&lt;td&gt;Every bot at once&lt;/td&gt;
&lt;td&gt;Re-run the path query, seconds column drops&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Per-bot rate limit&lt;/td&gt;
&lt;td&gt;The highest-cost bot only&lt;/td&gt;
&lt;td&gt;Hit count falls, 429s appear for that UA&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;robots.txt&lt;/code&gt; rules&lt;/td&gt;
&lt;td&gt;Crawlers that honour it&lt;/td&gt;
&lt;td&gt;Read the log again after 24–48h&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Firewall block&lt;/td&gt;
&lt;td&gt;One persistent offender&lt;/td&gt;
&lt;td&gt;Requests stop entirely&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A per-UA rate limit looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;map&lt;/span&gt; &lt;span class="nv"&gt;$http_user_agent&lt;/span&gt; &lt;span class="nv"&gt;$bot_key&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kn"&gt;default&lt;/span&gt;          &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;~*GPTBot&lt;/span&gt;         &lt;span class="s"&gt;"gptbot"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;~*ClaudeBot&lt;/span&gt;      &lt;span class="s"&gt;"claudebot"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;limit_req_zone&lt;/span&gt; &lt;span class="nv"&gt;$bot_key&lt;/span&gt; &lt;span class="s"&gt;zone=ai_bots:10m&lt;/span&gt; &lt;span class="s"&gt;rate=30r/m&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;server&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kn"&gt;limit_req&lt;/span&gt; &lt;span class="s"&gt;zone=ai_bots&lt;/span&gt; &lt;span class="s"&gt;burst=20&lt;/span&gt; &lt;span class="s"&gt;nodelay&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kn"&gt;limit_req_status&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="c1"&gt;# ...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Requests with an empty key aren't counted, so ordinary visitors are untouched. Use the tokens you actually saw in your own log, not the two above. If your build exposes &lt;code&gt;$limit_req_status&lt;/code&gt;, add it to the log format and you can confirm the limits are firing instead of assuming.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;robots.txt&lt;/code&gt; sits third for a reason: it's advisory. Some crawlers honour it, some don't, and the only way to know which happened is to read the log a day later. Write the file, then verify it, rather than reporting it as done.&lt;/p&gt;

&lt;h2&gt;
  
  
  Close the loop with the numbers you started with
&lt;/h2&gt;

&lt;p&gt;Run the cost query again over a window after the change. You want the same shape of answer as before: which UA is first, and how many seconds it now costs. If the top entry has changed, you fixed the last problem and this one was already there waiting.&lt;/p&gt;

&lt;p&gt;Wrap the whole sequence in a small script and keep it. One command that prints seconds, hits and bytes per user agent, sorted, is worth more than a firewall rule nobody can explain six months from now.&lt;/p&gt;

&lt;p&gt;If you'd rather not maintain the parser: &lt;a href="https://aumcreate.com/plugins/aumcrawl?utm_source=devto&amp;amp;utm_medium=content&amp;amp;utm_campaign=devto-ai-crawler-load-triage-order" rel="noopener noreferrer"&gt;AumCrawl&lt;/a&gt; is a free plugin that records which AI crawlers are hitting a WordPress site and lets you allow or block them individually from the admin. It sits next to the log work above rather than replacing it — the log is still what tells you what each bot costs.&lt;/p&gt;

</description>
      <category>backend</category>
      <category>linux</category>
      <category>performance</category>
      <category>security</category>
    </item>
    <item>
      <title>Splitting B2B export keyword research by part number, application and industry</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Sun, 01 Nov 2026 14:25:04 +0000</pubDate>
      <link>https://dev.to/aumcreate/splitting-b2b-export-keyword-research-by-part-number-application-and-industry-4od0</link>
      <guid>https://dev.to/aumcreate/splitting-b2b-export-keyword-research-by-part-number-application-and-industry-4od0</guid>
      <description>&lt;p&gt;The output of this process is a spreadsheet: one row per query, one column for the vocabulary the query belongs to, one for where you found it, one for the page it maps to, one for what to do with it. Everything below is about filling those columns honestly. The research part is mostly deciding which rows deserve a page of their own.&lt;/p&gt;

&lt;h2&gt;
  
  
  The vocabulary column decides everything else
&lt;/h2&gt;

&lt;p&gt;B2B buyers search in one of three vocabularies, and which one they use tells you where they are in the buying process and what kind of page can answer them.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Vocabulary&lt;/th&gt;
&lt;th&gt;What the buyer already knows&lt;/th&gt;
&lt;th&gt;Query shape&lt;/th&gt;
&lt;th&gt;Page that answers it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Part number&lt;/td&gt;
&lt;td&gt;the exact part&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ABC-1234&lt;/code&gt;, &lt;code&gt;ABC1234 datasheet&lt;/code&gt;, &lt;code&gt;replacement for XYZ-45&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;family page with parameters, variants, documents&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Application&lt;/td&gt;
&lt;td&gt;the job the part has to do&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;600V mosfet for LED driver&lt;/code&gt;, &lt;code&gt;valve for 180C steam line&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;application page with an app note or a test&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Industry&lt;/td&gt;
&lt;td&gt;the regulated world they sell into&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;automotive grade ...&lt;/code&gt;, &lt;code&gt;medical device ...&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;segment page with certifications and references&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sort every row before you do anything else. A list that mixes the three gets prioritised by whatever has the biggest volume number, which is usually the vaguest query and the hardest page to write.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part numbers: the shortest path from search to enquiry
&lt;/h2&gt;

&lt;p&gt;Start by collecting the strings. They're already written down somewhere:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;your catalogue and order history&lt;/li&gt;
&lt;li&gt;customer emails and RFQs — the exact string is in there, punctuation included&lt;/li&gt;
&lt;li&gt;distributor spreadsheets and price lists&lt;/li&gt;
&lt;li&gt;the cross-reference tables printed in datasheets, both yours and the ones for parts you replace&lt;/li&gt;
&lt;li&gt;drawings, packaging, spare-part lists&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Judging demand works differently here. Volume tools return zero or a dash for most of these strings, and that means "not enough data", not "nobody searches". Better evidence, in rough order of strength: the string appears in an actual enquiry; the string shows impressions in your search console query report; a distributor asked for it by name.&lt;/p&gt;

&lt;p&gt;Normalise before you count. The same part arrives as &lt;code&gt;ABC 1234&lt;/code&gt;, &lt;code&gt;ABC-1234&lt;/code&gt;, &lt;code&gt;abc1234&lt;/code&gt;, &lt;code&gt;ABC/1234&lt;/code&gt;. Counted raw, that's four queries where there is one.&lt;/p&gt;

&lt;p&gt;With a CSV that has a &lt;code&gt;query&lt;/code&gt; column, this groups strings into families and shows you where the spellings cluster:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;

&lt;span class="n"&gt;FAMILY&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;([A-Za-z]{1,5})[\s\-_/]?(\d{2,6})&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;family&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;match&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FAMILY&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="n"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;digits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;match&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;upper&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="k"&gt;match&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;prefix&lt;/span&gt;&lt;span class="si"&gt;}{&lt;/span&gt;&lt;span class="n"&gt;digits&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DictReader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="n"&gt;by_family&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;family&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;query&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;by_family&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;query&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;queries&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;by_family&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])):&lt;/span&gt;
    &lt;span class="n"&gt;sample&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;queries&lt;/span&gt;&lt;span class="p"&gt;)[:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\t&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;queries&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\t&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;sample&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The heuristic is deliberately crude — prefixes like &lt;code&gt;LED&lt;/code&gt; or &lt;code&gt;AC&lt;/code&gt; will pull unrelated strings together, so read the output before you build anything from it. What you're looking for is families with several distinct spellings. Those are the ones where buyers are already searching by part number. A family with exactly one spelling, and that spelling matches your internal code, is usually a string you invented.&lt;/p&gt;

&lt;p&gt;A family earns its own page when you sell it, you can fill the page with parameters, variants and documents, and more than one spelling shows up. A single SKU with no documents belongs in a row of the family table, not on its own URL. Hundreds of near-identical pages end up competing with each other for the same query, which is worse than one complete page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Applications: queries from buyers who don't know your part numbers
&lt;/h2&gt;

&lt;p&gt;These buyers describe the job. The query pattern is a part type plus "for" or "used in" plus a condition: temperature, voltage, IP rating, package, certification.&lt;/p&gt;

&lt;p&gt;Where they show up: support tickets, notes from sales calls, trade-show conversations, distributor application pages, industry forums, and the applications section of your own datasheets. That last one is underrated — it was written by engineers in the buyer's language rather than in your category tree.&lt;/p&gt;

&lt;p&gt;Build a grid rather than a list. Applications down one side, constraints across the other.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Application&lt;/th&gt;
&lt;th&gt;Constraint&lt;/th&gt;
&lt;th&gt;Query example&lt;/th&gt;
&lt;th&gt;Evidence you can show&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;LED driver&lt;/td&gt;
&lt;td&gt;600 V, high switching frequency&lt;/td&gt;
&lt;td&gt;&lt;code&gt;600v mosfet for led driver&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;app note, efficiency curve&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Steam line&lt;/td&gt;
&lt;td&gt;180 °C continuous&lt;/td&gt;
&lt;td&gt;&lt;code&gt;valve for 180c steam line&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;test report&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Battery charger&lt;/td&gt;
&lt;td&gt;reverse polarity protection&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;nothing yet&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Fill a cell only when there's a document, a test result, or a reference customer behind it. Empty cells are information: they tell you what to ask engineering for before you write anything.&lt;/p&gt;

&lt;p&gt;An application phrase without proof belongs in a short FAQ block on the family page. Giving it its own URL produces a page that says the same thing as the family page in slightly different words, which helps nobody.&lt;/p&gt;

&lt;h2&gt;
  
  
  Industry: the vocabulary the buyer's regulator uses
&lt;/h2&gt;

&lt;p&gt;Automotive, medical, lighting, appliance, industrial automation each have their own words for the same component, and those words come from standards and procurement departments, not from your product taxonomy. That's why industry rows often look unfamiliar when you first see them in a query report.&lt;/p&gt;

&lt;p&gt;Industry queries are the vaguest and the most contested. One product page won't hold them. What holds them is a segment page — the parts you ship into that segment, the certifications, the documents, one or two references — with the family and application pages underneath it linking up. If you have no references in a segment, don't build the segment page yet; the page will be a paragraph of adjectives.&lt;/p&gt;

&lt;p&gt;Before a term goes into the sheet, test it out loud: would a buyer say this on a phone call, or is it a literal translation of your internal category name? Translated category names are the most common source of rows that sit for a year with no impressions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ordering the work
&lt;/h2&gt;

&lt;p&gt;Score each row on two things only: how specific the query is, and how much evidence you can put on the page. A row with zero evidence is a dead row regardless of how attractive the keyword looks.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Part-number families first if you sell by part number. The queries are narrow enough that one complete page can answer them, and the person searching already knows what they want.&lt;/li&gt;
&lt;li&gt;Applications second. More reach, more work, and it needs documents you may not have yet.&lt;/li&gt;
&lt;li&gt;Industry last, unless you already have references in that segment.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Monday: the first twenty rows
&lt;/h2&gt;

&lt;p&gt;Take the top twenty rows by that score. For each one, write the single sentence that answers the query, exactly as it would appear on the page. If you can't write the sentence from material you already have, the row isn't a page — it's a request to engineering for a datasheet, a test, or a photo.&lt;/p&gt;

&lt;p&gt;Two checks before anyone opens an editor:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Every row maps to a page, and every page has at least one row whose query a real buyer typed.&lt;/li&gt;
&lt;li&gt;No two rows with the same intent map to different URLs. That's how a site ends up with three pages quietly competing for one search, and it's much cheaper to catch in the sheet than after publishing.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>marketing</category>
      <category>seo</category>
    </item>
    <item>
      <title>Internal linking order: which links to place first, and how to control the anchor text</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Sun, 01 Nov 2026 14:21:09 +0000</pubDate>
      <link>https://dev.to/aumcreate/internal-linking-order-which-links-to-place-first-and-how-to-control-the-anchor-text-3gj6</link>
      <guid>https://dev.to/aumcreate/internal-linking-order-which-links-to-place-first-and-how-to-control-the-anchor-text-3gj6</guid>
      <description>&lt;p&gt;A product page with 46 blog posts behind it and three internal links pointing at it. Two of those sat in the footer, so they appeared on all 46 pages and said the same thing 46 times. The third was in a post about a variant that no longer ships. Every post ended with four "related reading" links chosen by tag, so most of the blog's links pointed at other blog posts.&lt;/p&gt;

&lt;p&gt;Nothing there was broken. The site just never had an order: links were added as pages were published, and nobody asked which page should be receiving them. Decide the order first, then choose the words — editing anchors on a link map you haven't settled on yet wastes the pass.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put the target pages in a file first
&lt;/h2&gt;

&lt;p&gt;Pick the pages that have to be found — the ones that answer a question someone types before they contact you. Somewhere between five and twelve. More than that and you spread the work so thin that nothing moves.&lt;/p&gt;

&lt;p&gt;Write them one path per line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/services/emergency-plumbing
/products/heat-exchanger-120
/guides/sizing-a-pressure-valve
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Save it as &lt;code&gt;priority.txt&lt;/code&gt;. The script at the bottom reads it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Order the link types by what a single edit buys
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Links from pages that already have weight.&lt;/strong&gt; Pages with external backlinks or steady organic clicks have something to pass on. A link from one of those, in the body of the page, does work that a link from an orphaned post nobody visits doesn't. Find them in your search console's links report, or — with no backlink data — in the pages report sorted by clicks. Write the donor list next to each target before you touch anything.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Links between a target page and the pages that support it.&lt;/strong&gt; The target links down to the detail pages: variants, pricing, documents, a case. Each detail page links back up in the body, not just in a breadcrumb. That's what makes a cluster readable — a crawler can see which page is the parent, and a reader can get from a specific question to the main offer without using the menu.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Links between sibling pages that answer the next question.&lt;/strong&gt; Someone reading about a 120 kW unit asks about the 160 kW one. Someone reading about installation asks about maintenance. These cost almost nothing and they pick up the long-tail queries the parent page can't hold.&lt;/p&gt;

&lt;p&gt;Work in that order: a couple of weeks of donor edits, then the cluster, then the siblings.&lt;/p&gt;

&lt;h2&gt;
  
  
  How many in-body links belong on a page
&lt;/h2&gt;

&lt;p&gt;There's no published cap, and anyone quoting one is guessing. The constraint that bites is reader attention: past roughly one link per 100–150 words of body copy, the page reads like a directory and people stop following any of them.&lt;/p&gt;

&lt;p&gt;Navigation and footer links don't belong in that count — they're template, not editorial. Which is also why they do less work than people expect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Find the posts that don't link to the target
&lt;/h2&gt;

&lt;p&gt;On WordPress, this returns published posts whose content never mentions the target path:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_date&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;wp_posts&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'post'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'publish'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_content&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;LIKE&lt;/span&gt; &lt;span class="s1"&gt;'%/services/emergency-plumbing%'&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;post_date&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it in your DB client or with &lt;code&gt;wp db query &amp;lt; missing-links.sql&lt;/code&gt;. Check one row before you trust the result: the editor stores absolute URLs, so the pattern has to match what's actually in &lt;code&gt;post_content&lt;/code&gt;. If your permalinks carry a trailing slash, include it.&lt;/p&gt;

&lt;p&gt;The output is a candidate list, not a to-do list. Pick the three or four posts that already get impressions for something adjacent to the target and edit those. The rest can wait.&lt;/p&gt;

&lt;h2&gt;
  
  
  Control the anchor text instead of the link count
&lt;/h2&gt;

&lt;p&gt;The anchor is the part of a link that tells anything downstream what the target is about. Most sites spend it on "read more" or on the same three words repeated sitewide.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Anchor&lt;/th&gt;
&lt;th&gt;What it carries&lt;/th&gt;
&lt;th&gt;How often&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Exact product or service term&lt;/td&gt;
&lt;td&gt;the target's main query, unambiguously&lt;/td&gt;
&lt;td&gt;once per target, from the strongest donor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Descriptive phrase lifted from the target's own H2&lt;/td&gt;
&lt;td&gt;the specific question the target answers&lt;/td&gt;
&lt;td&gt;the default; vary the phrasing per page&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sentence-natural ("a walkthrough of the sizing steps")&lt;/td&gt;
&lt;td&gt;context from the sentence around it&lt;/td&gt;
&lt;td&gt;wherever the sentence wants it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"click here", "read more", "this page"&lt;/td&gt;
&lt;td&gt;nothing&lt;/td&gt;
&lt;td&gt;replace on sight&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Four rules that hold regardless of how ranking systems change:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One anchor, one URL. If two pages could both take the same phrase, one of them is the wrong target — usually the older, thinner one. Merge or choose.&lt;/li&gt;
&lt;li&gt;Read the sentence with the link text spoken out loud. If the anchor only makes sense because it's blue and underlined, rewrite the sentence.&lt;/li&gt;
&lt;li&gt;Pull anchors from the target page's own headings. If the target has an H2 "Sizing a pressure valve for a 3-inch line", that phrase beats your internal category name, because the page can actually answer it.&lt;/li&gt;
&lt;li&gt;Use the exact term once, from the strongest donor, and vary it elsewhere. Ten identical anchors from ten pages read as one recommendation repeated, and you lose the vocabulary that would have told anything downstream what the page covers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Never reuse one anchor for two different targets across the site. It splits the signal and it annoys readers who click the same words and land somewhere new.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sitewide links: keep the template small
&lt;/h2&gt;

&lt;p&gt;A link in the header appears on every crawlable URL. That's correct for humans — it's how people navigate — but it does very little to differentiate pages: links are processed per page, and the same target with the same anchor from 400 URLs is one anchor repeated.&lt;/p&gt;

&lt;p&gt;Practically: keep the nav to the handful of pages a first-time visitor needs, keep the footer to policy and contact pages, and put the differentiation into body copy where the anchor can change per page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Audit it with a script
&lt;/h2&gt;

&lt;p&gt;This walks your sitemap, counts inbound internal links per path, and separates links inside &lt;code&gt;&amp;lt;main&amp;gt;&lt;/code&gt; or &lt;code&gt;&amp;lt;article&amp;gt;&lt;/code&gt; from nav and footer links. Standard library only, no dependencies.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;#!/usr/bin/env python3
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Internal link audit.
Usage: python3 audit.py https://example.com/sitemap.xml priority.txt &amp;gt; report.csv
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.parse&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.request&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;html.parser&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;HTMLParser&lt;/span&gt;

&lt;span class="n"&gt;SITE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{0.scheme}://{0.netloc}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
&lt;span class="n"&gt;HOST&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SITE&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt;
&lt;span class="n"&gt;UA&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;internal-link-audit/1.0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LinkParser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;HTMLParser&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;MAIN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;main&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;article&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="nf"&gt;super&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;convert_charrefs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;links&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_main&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_starttag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attrs&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MAIN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_main&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attrs&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;href&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;
                &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_href&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_endtag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_href&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_text&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;links&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_href&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_main&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;tag&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MAIN&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_main&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_main&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;UA&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;charset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_content_charset&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;charset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;replace&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;path_of&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;internal_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;#&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;mailto:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tel:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;javascript:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="n"&gt;parts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;HOST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;


&lt;span class="n"&gt;sitemap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;priority&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;
&lt;span class="n"&gt;paths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="nf"&gt;path_of&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;loc&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;loc&amp;gt;\s*(.*?)\s*&amp;lt;/loc&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sitemap&lt;/span&gt;&lt;span class="p"&gt;))})&lt;/span&gt;

&lt;span class="n"&gt;inbound_body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;inbound_all&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;anchors&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SITE&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;skip &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LinkParser&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;feed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;in_main&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;links&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;target&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;internal_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SITE&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;target&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="n"&gt;inbound_all&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;in_main&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;inbound_body&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;anchors&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;priority&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;path&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;priority&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;priority&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;in_body_inbound&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;inbound_body&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;all_inbound&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;inbound_all&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;top_anchors&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; | &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; (&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;anchors&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;most_common&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;

&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;priority&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;in_body_inbound&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
&lt;span class="n"&gt;writer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DictWriter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stdout&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fieldnames&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;keys&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeheader&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it with &lt;code&gt;python3 audit.py https://example.com/sitemap.xml priority.txt &amp;gt; report.csv&lt;/code&gt;. It reads a flat sitemap; if yours is a sitemap index, expand the &lt;code&gt;&amp;lt;loc&amp;gt;&lt;/code&gt; list first or point it at the child sitemap.&lt;/p&gt;

&lt;p&gt;The sort puts priority pages with the lowest in-body inbound count at the top. Those rows are your donor-edit list, in order.&lt;/p&gt;

&lt;p&gt;Two caveats worth knowing before you act on the numbers. Links injected by JavaScript won't show up — the script only sees what's in the HTML, so spot-check one page in the browser. And a path in &lt;code&gt;priority.txt&lt;/code&gt; that isn't in the sitemap still gets a row, usually with zeros; that's often the most useful row on the sheet, because it means the page has no inbound links from anywhere the crawler reaches.&lt;/p&gt;

&lt;h2&gt;
  
  
  After the edits
&lt;/h2&gt;

&lt;p&gt;Re-run the same script three weeks later and diff the report. Then look at whether the priority URL is indexed and whether it's picking up impressions for the phrase you used as its anchor. Give it weeks, not days — an internal link is a hint about structure, not a switch.&lt;/p&gt;

&lt;p&gt;If you're on WordPress and would rather not maintain the script: &lt;a href="https://aumcreate.com/plugins/aumviso?utm_source=devto&amp;amp;utm_medium=content&amp;amp;utm_campaign=devto-internal-link-priority-order" rel="noopener noreferrer"&gt;AumViso&lt;/a&gt; handles sitemap output, schema and internal links, and it's free. I still run the audit above afterwards, because it counts what's actually in the rendered HTML, which is the only version that matters.&lt;/p&gt;

&lt;p&gt;For me it comes down to this: a site with 46 pages and no link order is 46 pages competing to be noticed. A site with 46 pages and one decided order is a hierarchy. The difference is a few hours of editing, not a redesign.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
    </item>
    <item>
      <title>How many pages a company site needs before launch, and what goes on each one</title>
      <dc:creator>AumCreate</dc:creator>
      <pubDate>Sun, 01 Nov 2026 13:53:04 +0000</pubDate>
      <link>https://dev.to/aumcreate/how-many-pages-a-company-site-needs-before-launch-and-what-goes-on-each-one-20cg</link>
      <guid>https://dev.to/aumcreate/how-many-pages-a-company-site-needs-before-launch-and-what-goes-on-each-one-20cg</guid>
      <description>&lt;p&gt;A template sitemap arrives with twelve entries on it, the domain is already bought, and the launch date is three weeks out. The useful question at that point is not how many pages a company site should have. It is which pages this particular sale requires, and what has to be true on each one before someone can act on it.&lt;/p&gt;

&lt;p&gt;Working backwards from a page count gives you pages nobody maintains.&lt;/p&gt;

&lt;h2&gt;
  
  
  Let the purchase path set the floor, not the template
&lt;/h2&gt;

&lt;p&gt;Four inputs decide the minimum: how a customer decides, the size of a typical deal, where trust has to come from, and whether the buying happens in one session or over several weeks.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Business shape&lt;/th&gt;
&lt;th&gt;Pages that must exist at launch&lt;/th&gt;
&lt;th&gt;Reasonable to add later&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Local service, one offer, phone-first&lt;/td&gt;
&lt;td&gt;Home, Contact&lt;/td&gt;
&lt;td&gt;Separate service page, gallery&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quoted B2B, a few distinct offers&lt;/td&gt;
&lt;td&gt;Home, one page per offer, About, Contact, legal&lt;/td&gt;
&lt;td&gt;Process, compliance, case pages&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Self-serve product&lt;/td&gt;
&lt;td&gt;Home, product and pricing, docs or FAQ, Contact, legal&lt;/td&gt;
&lt;td&gt;Comparison pages, changelog&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Long cycle, high value&lt;/td&gt;
&lt;td&gt;Home, offer pages, About, process, Contact, legal&lt;/td&gt;
&lt;td&gt;Case studies, security page, team&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Legal means whatever your jurisdiction requires — privacy notice, terms, and a company imprint where one is mandated. Those are not marketing pages and they are not optional.&lt;/p&gt;

&lt;p&gt;The table is a starting position you argue with, not a rule. If a page sits in the "later" column and nobody can name the sentence that would go on it, it does not go on the list.&lt;/p&gt;

&lt;h2&gt;
  
  
  What has to be on each page before it goes live
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Page&lt;/th&gt;
&lt;th&gt;Its one job&lt;/th&gt;
&lt;th&gt;Must contain before launch&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Home&lt;/td&gt;
&lt;td&gt;Say what this is, for whom, where, and what to do next&lt;/td&gt;
&lt;td&gt;One sentence naming the offer and the customer; the primary action visible without scrolling; one concrete proof point; a link to contact&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Offer page&lt;/td&gt;
&lt;td&gt;Answer "can they do my version of this"&lt;/td&gt;
&lt;td&gt;Who it is for, what is included, what is not, turnaround, what the buyer has to supply, next step&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;About&lt;/td&gt;
&lt;td&gt;Give a reason to trust the people&lt;/td&gt;
&lt;td&gt;Who runs it, where they operate, what they do not do, how to verify the claims&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contact&lt;/td&gt;
&lt;td&gt;Remove the last excuse&lt;/td&gt;
&lt;td&gt;A form that works, an email address, a phone number someone answers, hours, a response time you can keep&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Legal&lt;/td&gt;
&lt;td&gt;Keep you out of trouble&lt;/td&gt;
&lt;td&gt;Privacy notice, terms, required registrations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Thank-you and 404&lt;/td&gt;
&lt;td&gt;Catch people at the edges&lt;/td&gt;
&lt;td&gt;Confirmation of what happens next; links back to the main offers&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  The first screen is not a slogan slot
&lt;/h2&gt;

&lt;p&gt;Many home pages open with a positioning line and a stock photo. A visitor arriving from a search result needs three facts before anything else: what you sell, who it is for, and what the next step costs them in effort. Put those in the first paragraph, as plain text, in the words buyers use when they describe the problem.&lt;/p&gt;

&lt;p&gt;One primary action. Two equally weighted buttons — "Contact us" next to "Download brochure" — split the decision and usually reduce both.&lt;/p&gt;

&lt;p&gt;If geography matters, say where you serve in the first screen, not in the footer. If you work remotely and location does not matter, say that too, because a visitor who cannot tell will assume you are local to someone else.&lt;/p&gt;

&lt;h2&gt;
  
  
  When one offer deserves its own page
&lt;/h2&gt;

&lt;p&gt;The default is one page per offer only if these hold:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A different buyer, or genuinely different questions from the same buyer.&lt;/li&gt;
&lt;li&gt;Different proof is needed — a certificate, a lead time, a spec sheet the others do not have.&lt;/li&gt;
&lt;li&gt;Enough to say to fill a page without padding. A screen and a half of real content is a fair bar.&lt;/li&gt;
&lt;li&gt;Someone will maintain it.&lt;/li&gt;
&lt;li&gt;People search for it under a different phrase than the parent offer.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two or more failing means one page with anchored sections. Splitting because "more pages is better for SEO" produces thin pages that compete with each other for the same query, and they usually cannibalise rather than help. If you split later, use a URL that survives a redesign: &lt;code&gt;/services/name&lt;/code&gt;, not &lt;code&gt;/our-services-2026&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Filling the trust gap with no testimonials, cases or photos
&lt;/h2&gt;

&lt;p&gt;Real options that require inventing nothing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Publish the process step by step, including what you need from the customer and how long each step usually takes.&lt;/li&gt;
&lt;li&gt;Say what you do not do. A page that excludes some work reads as more credible than one claiming everything.&lt;/li&gt;
&lt;li&gt;Show real work with the client's written permission, even if it is one photo and a paragraph.&lt;/li&gt;
&lt;li&gt;Give prices or ranges where the business allows it. "Contact us for pricing" costs you the visitors who only wanted a number.&lt;/li&gt;
&lt;li&gt;Name the person responsible and how to reach them.&lt;/li&gt;
&lt;li&gt;Link to things a reader can check independently — registration, certification, address, a partner's own site.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What you should never do is write your own testimonial, use a stock face captioned as a customer, or publish a case with figures you cannot back up. It reads fine until a buyer asks one question about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pre-publish pass, page by page
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The title contains the offer, in the buyer's words.&lt;/li&gt;
&lt;li&gt;The first paragraph names who the page is for.&lt;/li&gt;
&lt;li&gt;Exactly one primary action, repeated at the end.&lt;/li&gt;
&lt;li&gt;No placeholder text, no "coming soon", no dead items in the menu.&lt;/li&gt;
&lt;li&gt;The form was submitted end to end by a human and the confirmation email arrived.&lt;/li&gt;
&lt;li&gt;Phone numbers are tappable; addresses are text, not baked into an image.&lt;/li&gt;
&lt;li&gt;The page loads on a phone over a slow connection without the layout jumping.&lt;/li&gt;
&lt;li&gt;There is a link to it from at least one other page.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Site-wide: a sitemap, a robots file that blocks nothing you need indexed, and analytics that record which page a form was submitted from — not only that a form was submitted.&lt;/p&gt;

&lt;h2&gt;
  
  
  Leaving room for page seven
&lt;/h2&gt;

&lt;p&gt;Navigation that is a flat list of every page becomes a redesign the moment you add two. Group by what the buyer is trying to do and keep the top level short.&lt;/p&gt;

&lt;p&gt;When you merge, split or rename after launch, redirect the old URL and keep the redirect. Internal links inside other pages are the usual thing that gets missed — the redirect works for search engines and for readers, but paragraphs that reference the old page still point at the old URL.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deciding after launch on something other than taste
&lt;/h2&gt;

&lt;p&gt;Give it a month, then read entrance data per page: which pages people land on, which pages nobody enters, which pages earn internal clicks, and where form submissions start and stop.&lt;/p&gt;

&lt;p&gt;A page with no entrances, no internal links and no form activity is a candidate to merge into a page that does have traffic, not to delete. A page where two different questions keep showing up in the same visit is a candidate to split. New pages should come from a question you have seen more than once, not from a keyword tool's estimate.&lt;/p&gt;

&lt;p&gt;Set a review date when you launch. Otherwise the next decision about the site gets made by whoever is annoyed enough to raise it, and by then the fix is a redesign.&lt;/p&gt;

</description>
      <category>marketing</category>
      <category>product</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
