DEV Community: Stepan Nikonov

SEO-Friendly Web Design Checklist: Architecture Before Aesthetics

Stepan Nikonov — Sat, 12 Sep 2026 19:19:18 +0000

Direct answer: An SEO-friendly design starts with an agreed page inventory, crawlable navigation, stable URL and template rules, equivalent mobile content, responsive media, relevant structured data, and a production-like validation gate. Visual design then works inside that system. It cannot repair important pages that do not exist, links that crawlers cannot follow, or templates that publish the wrong metadata at scale.

This checklist defines the pre-launch design and development contract. Use the static versus dynamic website guide to choose the publishing architecture. Use the site migration checklist when an existing site, domain, or URL set is being replaced.

1. Map page jobs before drawing wireframes

Start with the questions customers need answered and the actions the business needs them to complete. Convert those jobs into a page inventory before deciding how the homepage looks.

A visual sitemap full of generic labels such as Services, Solutions, and Resources is not enough. Each planned URL needs a distinct purpose, owner, content requirement, and next route.

Design input	Decision to record	Acceptance evidence
Page inventory	Which search or customer job each page owns	No two pages compete for the same primary task without a reason
Hierarchy	How pages group into sections and parent routes	A new visitor can identify location and next action from any landing page
Template set	Which page types share fields and behavior	One representative instance exists for every important template
Publishing ownership	Who can create, update, retire, and validate each type	The real owner can complete the workflow without bypassing controls
Launch gate	What must pass before the production switch	Status, rendering, links, metadata, schema, mobile, and performance are recorded

Google's SEO Starter Guide recommends organizing a site logically so users and search engines can understand how pages relate. The practical deliverable is not a promise that every page will rank. It is a maintained map showing why every indexable page exists and how it connects to the rest of the site.

2. Turn the hierarchy into crawlable navigation

Design the global navigation, section navigation, breadcrumbs, contextual links, cards, and footer as one discovery system. Every important page should have at least one durable internal route. High-value pages normally need several routes that match real customer journeys, not one hidden link in an XML sitemap.

Google generally discovers links when they are ordinary <a> elements with an href. A styled button, router event, or clickable card without a resolvable link may work for a person and still be unreliable for crawling. The official crawlable links guidance also recommends concise, descriptive anchor text that tells people and Google what the destination contains.

Use real links for destinations and buttons for actions.
Keep the primary navigation usable with a keyboard and without precise pointer movement.
Do not remove essential section links from mobile merely to make the header smaller.
Show the selected navigation state without making it the only explanation of page location.
Test orphan risk from every template, including campaigns and dynamically created pages.

3. Define URL and page-state rules

Approve URL patterns before content entry and interface development create dozens of exceptions. Decide how service, category, product, article, location, filter, pagination, search, and campaign states behave. Record the final status, indexability, canonical target, sitemap inclusion, and internal-link policy for each family.

State	Required design rule	Failure to prevent
Canonical page	One stable, descriptive URL with a self-canonical	Multiple routes competing for the same content
Moved page	Direct permanent redirect to the closest valid replacement	Redirect chains or blanket homepage redirects
Missing page	Useful error design with the correct HTTP status	A soft 404 that looks live but has no main content
Filtered or sorted state	Explicit crawl and index policy by parameter family	An uncontrolled URL space created by interface controls
Preview or staging page	Access control and a production removal check	Test URLs indexed or the live site left blocked

The SEO-friendly URL guide covers normalization and later URL changes in detail. During design, the important decision is which states are real destinations and which are interface states. Do not let the router make that policy accidentally.

4. Build a template SEO contract

Wireframes should describe required content and metadata, not only rectangles. For every template, define where the title, H1, summary, main content, author or business evidence, media, internal links, CTA, canonical, robots directive, social image, and structured data come from. Mark which fields are required, optional, generated, and editable.

A useful acceptance sample includes normal, long, missing, duplicate, and retired content. Test a long service title, a product without a secondary image, an article with a table, a category with no current inventory, and an unpublished record. This exposes brittle layouts and metadata fallbacks before they affect hundreds of URLs.

If the same mistake can appear on every page of a type, the fix belongs in the template, schema, component, or publishing validation. The template-level technical SEO guide shows why page-by-page cleanup does not hold.

5. Protect mobile content and accessibility

Responsive design may rearrange a page, but it should not remove the information that gives the page its purpose.

Google's mobile-first indexing guidance says primary content, meaningful headings, metadata, structured data, and important images should remain equivalent across mobile and desktop. Content can move into accessible accordions or tabs when space is tight. It should not depend on a swipe, click, or typing action before it exists for crawling.

Accessibility is a design requirement, not a final plugin. W3C organizes WCAG 2.2 around content being perceivable, operable, understandable, and robust. Apply that to navigation order, landmarks, heading structure, color contrast, focus visibility, labels, errors, touch targets, reduced motion, and keyboard operation.

Automated checks help, but a person still needs to complete the main tasks with a keyboard and screen-size changes. Use the mobile SEO checklist for device-level rendering and task validation after the responsive templates exist.

6. Set media and performance budgets before visual sign-off

A fast mockup is not evidence that the production page will be fast. Test the real font files, hero media, navigation, consent controls, forms, embeds, third-party scripts, and content length. Define budgets by template so a later design change has a measurable trade-off instead of a vague instruction to optimize it.

Reserve image and embed dimensions to prevent layout shifts.
Provide responsive image candidates and realistic sizes.
Do not lazy-load the likely LCP image or critical above-the-fold content.
Load nonessential scripts only where the page job requires them.
Test slow phones, constrained networks, long content, and validation errors.

The current Core Web Vitals targets are LCP within 2.5 seconds, INP at 200 milliseconds or less, and CLS at 0.1 or less, evaluated at the 75th percentile separately for mobile and desktop. These are field targets, not a guarantee that a page will rank.

Use the web.dev definitions as the source and the Core Web Vitals diagnosis when a representative template misses them.

7. Map structured data to visible content

Choose structured data from the page's actual job, not from a list of schema types a plugin can emit. Map each supported template to the relevant Google feature documentation, required properties, visible source fields, canonical URL, image ownership, and validation test. Keep the markup synchronized with the page as content changes.

Google's structured data guidelines require markup to represent the main visible content and warn against hidden, irrelevant, or misleading data. Valid markup only creates eligibility. It does not guarantee a rich result.

Template	Possible structured data	Design dependency
Article or guide	Article and BreadcrumbList	Visible author, dates, headline, image, and breadcrumb route
Product	Product and offers when eligible	Visible identity, price, availability, media, and genuine review data
Local business page	Relevant LocalBusiness subtype	Real business details and location-specific visible content
Service page	Often Organization or BreadcrumbList context only	No invented ratings, locations, or unsupported rich-result promises

8. Prove the design on a production-like prototype

Do not approve SEO from a design file alone. Build at least one production-like page for every high-value template using realistic content, routes, media, scripts, and error states. Inspect the initial response and the rendered page. Crawl the prototype, resize it, navigate by keyboard, submit its forms, disable optional scripts, and test its slowest likely conditions.

Request representative URLs. Record response status, redirect destination, headers, canonical, robots directives, and rendered title.
Crawl from the homepage. Confirm that intended pages appear through links and unwanted states do not multiply.
Compare mobile and desktop. Check content, links, metadata, images, structured data, and working tasks.
Test template extremes. Use long headings, missing optional fields, large inventories, no-result states, and validation errors.
Run rich-result, performance, and accessibility checks. Treat automated scores as evidence for investigation, not complete approval.
Assign every defect. Name the template or component, owner, expected rule, retest, and rollback path.

9. Use a launch gate with named owners

The final sign-off should be a recorded comparison against the approved page and template contract.

Design owns responsive behavior and component states. Development owns rendered output, routing, status codes, and performance implementation. Content owners own page purpose and maintained fields. SEO validates crawlability, index signals, internal discovery, and search-facing output. One person may hold several roles, but no requirement should be ownerless.

If the project replaces an existing site, stop here and run the separate migration workflow. Redirect mapping, old-to-new URL reconciliation, baseline crawls, launch monitoring, and rollback are migration work even when the redesign preserves the brand.

What not to do

Do not approve architecture from the homepage mockup. Search visitors can enter through any useful internal page.
Do not hide essential navigation behind JavaScript events. Destinations need resolvable links.
Do not invent pages after components are finished. Page jobs and fields should shape the template.
Do not remove useful mobile content for visual neatness. Rearrange it without changing the page's job.
Do not add every schema type a plugin offers. Mark up visible, supported content only.
Do not call a Lighthouse score the launch gate. Status, links, content, forms, schema, accessibility, and error states need separate tests.
Do not combine a redesign and migration without a baseline. Separate the target design from the changeover plan.

Verification checklist

Every planned indexable page has a distinct customer or search job.
Every important page is reachable through crawlable internal links.
URL families have documented status, canonical, index, and sitemap rules.
Each template defines required content, metadata, media, schema, and error behavior.
Mobile preserves equivalent primary content, headings, links, metadata, and structured data.
Navigation, forms, focus states, labels, errors, contrast, and motion have accessibility acceptance tests.
Representative templates have media, script, and Core Web Vitals budgets.
Structured data represents visible content and passes the relevant feature validation.
Production-like pages pass status, crawl, render, mobile, keyboard, form, and failure-state tests.
Every launch blocker has an owner, retest, rollback condition, and recorded result.

The design is SEO-ready when the team can prove how pages are discovered, what each template publishes, how mobile preserves the page job, which URL states are indexable, and what must pass before launch. A polished screenshot is not that proof.

Evidence basis

Google Search Central, web.dev, and W3C documentation were checked on August 25, 2026. The page-contract table, prototype sequence, launch ownership model, and acceptance checklist are FloxoLab implementation frameworks. They are not Google ranking factors or guarantees.

Technical SEO Audit with Screaming Frog and Search Console

Stepan Nikonov — Tue, 08 Sep 2026 10:12:53 +0000

Export or connect Search Console first, run a scoped Screaming Frog crawl, normalize both datasets to final canonical URLs, and compare the sets. Prioritize pages that already earn impressions or clicks, important pages Google knows but the crawl missed, and crawlable indexable pages with conflicting status, canonical, directive, or sitemap signals. Re-crawl after implementation, then use Search Console to confirm Google's stored view catches up.

The tools answer different questions. Screaming Frog reports what its configured crawler can reach and extract from the live site. Search Console reports Google's processed search and indexing evidence, with reporting limits and delays. Agreement raises confidence. A disagreement is a diagnosis lead, not an automatic error.

This guide starts after you have chosen the crawler and obtained safe viewing access. If you are comparing the free and paid versions, limitations, or whether the software suits you, read the separate Screaming Frog review. If someone else will access the property, define the minimum role using the SEO audit access guide.

Know what each dataset can prove

Evidence	Useful for	Does not prove
Screaming Frog crawl	Live response, internal discovery, directives, canonicals, links, metadata, and crawl depth under the chosen configuration	That Google discovered, rendered, selected, indexed, or ranked the same URL
Performance report	Pages and queries receiving reported Google Search impressions and clicks during the selected period	A complete URL inventory, current live status, or index status for every page
Page indexing report	Google-known URL totals, indexed and non-indexed groups, and representative examples by reason	That every non-indexed URL is an error or that every example can be exported
URL Inspection	Google's stored index view for one URL and a separate live fetch test	A guarantee that the page will appear or rank in search results

Google's Performance report documentation calls this canonical URL aggregation: most performance data is assigned to the page's canonical URL, not to a duplicate URL. That means a row may not use the exact URL variant that received the original request. Preserve the raw URL, but add a normalized join key rather than assuming a missing exact-string match means missing search evidence.

1. Define the scope and protect the baseline

Record the audit boundary before changing any crawler setting. Include the public host, allowed subdomains, staging exclusions, important folders, locale rules, parameter policy, expected canonical host, and the date range for Search Console. Save the raw exports and crawl file. You need the unchanged evidence when a filter or fix later changes the counts.

Property: https://example.com/
Audit date: 2026-08-04
Search Console period: last 90 days
Included hosts: example.com
Excluded areas: /account/, /checkout/, staging.example.com
Rendering: HTML / JavaScript
Canonical policy: HTTPS, non-www, no trailing slash
Expected indexable templates: service, category, product, guide

Do not aim an unrestricted crawl at infrastructure you do not own or have permission to test. Exclude private, transactional, faceted, calendar, and infinite URL spaces before starting. Use an appropriate crawl speed, especially on small or fragile hosting.

2. Export the Google evidence before the crawl

In Search Console, choose the correct property and save these three views:

Performance, Pages. Use a meaningful comparison window, normally 90 days for a stable audit and longer when the business is seasonal. Export clicks, impressions, CTR, and position by page.
Page indexing. Record indexed and non-indexed totals, reasons, trend changes, and examples for the groups that affect intended pages.
Sitemaps. Record submitted files, last read state, and the relationship between submitted and discovered pages.

Search Console's ordinary report export is limited to the rows shown, and Google documents a maximum of 1,000 representative examples for several reports. Performance tables also omit some data for privacy and internal limits. Treat the export as valuable evidence, not a complete census. Larger properties may need the Search Console API or bulk export for fuller performance analysis.

Do not label every "Not indexed" group as a defect. Google explicitly notes that duplicates, alternates, redirects, blocked pages, and intentionally noindexed URLs can be legitimate. The audit question is whether the affected URL was intended to be a separate canonical search result.

3. Configure and run a reproducible crawl

Screaming Frog's default Spider mode is designed as a quick start, but the configuration still determines the evidence. Confirm these settings before pressing Start:

Start URL and host boundary. Use the preferred canonical origin and decide whether subdomains or external CDN hosts belong in scope.
Robots behavior. A normal crawl should respect the intended rules. If you test a second configuration that ignores them, label it clearly because it answers a different question.
Rendering. Start with HTML for a conventional server-rendered site. Use JavaScript rendering when important links, content, canonicals, or directives depend on client rendering.
Includes and excludes. Prevent session URLs, internal search, calendar loops, filter explosions, and private routes from consuming the crawl.
Sitemaps and external URL sources. Add the XML sitemap and Google-derived URLs when you need to find pages that are known elsewhere but not linked by the crawl.
Storage and speed. Choose settings the machine and server can sustain, then preserve them for the validation crawl.

After the crawl finishes, run Crawl Analysis so filters that depend on post-crawl calculations populate. Export the internal HTML inventory and the reports needed for redirect chains, canonicals, directives, response codes, crawl depth, inlinks, and sitemap membership. An issue count on the right side of the interface is not the deliverable. The affected URLs, source URLs, intended behavior, and fix owner are.

4. Connect Search Console or join the exports

The licensed integration can pull Search Analytics and optional URL Inspection data into the crawl. As verified on August 4, 2026, Screaming Frog documents the interface path as Configuration > API Access > Google Search Console. Focus URL Inspection on important templates because the API has a daily property quota and the crawler queries discovered URLs in crawl order unless you narrow the scope.

A manual join is equally valid for a small audit. Keep one row per observed URL, preserve each raw value, and create a consistent normalized key:

raw_url
normalized_url
crawl_status
indexability
declared_canonical
crawl_depth
internal_inlinks
in_sitemap
gsc_clicks
gsc_impressions
gsc_index_group
google_selected_canonical
decision
owner

Normalize only known presentation variants such as the canonical host, protocol, default port, and a documented trailing-slash rule. Do not strip meaningful parameters or merge paths because they look similar. Keep redirects and canonical targets in separate columns so the audit can reveal where variants converge.

5. Reconcile the URL sets

Observed state	Likely question	First check
Crawlable 200 + indexable + GSC performance	Is a valuable page technically stable?	Protect status, canonical, content, links, and search intent before changing it
GSC performance + missing from crawl	Is the page orphaned, redirected, blocked, outside scope, or represented by another variant?	Test the raw GSC URL, final response, inlinks, canonical, and crawl configuration
Crawlable indexable page + no performance row	Is it new, low demand, weakly linked, excluded from the export, or not selected for search?	Check launch date, Page indexing, intent, internal links, and a URL Inspection sample
Non-200 or non-indexable + GSC performance	Did a valuable URL change, or is historical/canonical data still being reported?	Inspect the date trend, redirect target, selected canonical, and replacement page
In sitemap + not internally discovered	Is the sitemap compensating for an orphan?	Find the intended hub, category, or contextual link path
Internally discovered + canonical points elsewhere	Are links promoting a duplicate?	Compare content, canonical intent, sitemap membership, and internal destinations

The most useful bucket is often "GSC evidence but missing from the crawl." It can expose orphan pages, old URLs that still receive search activity, pages reachable only through a sitemap, or a crawler configuration that excluded an important section. The opposite bucket is not automatically bad. A live page with no Performance row may simply have no reported impressions during the selected period.

6. Prioritize impact before issue count

Do not sort the final backlog by the number of warnings. Score each finding across four dimensions:

Business and search value. Does the URL generate leads, sales, impressions, clicks, or support an important user journey?
Technical severity. Does the defect block fetching, indexing, canonical selection, navigation, or a final destination?
Pattern reach. Is this one stale page or a shared template, navigation component, routing rule, or sitemap generator?
Confidence. Do the crawl, Search Console, live page, and intended policy agree on the diagnosis?

Priority = value x severity x pattern reach x confidence

Critical: valuable URL unavailable, blocked, wrongly canonicalized, or removed without a valid replacement
High: repeatable template or navigation defect affecting intended pages
Investigate: evidence conflicts or the intended page job is unclear
Ignore: intentional state with no harmful discovery, index, or user impact

A missing meta description across 2,000 archive pages may be less urgent than one accidental noindex on the primary service page. A large redirect count may be expected history, while five redirecting links in the global navigation deserve a quick component fix.

Turn findings into fixable tickets

Each ticket should name the evidence, rule, shared source, sample URLs, owner, risk, rollback path, and validation method. "Fix 404s" is not enough. "Update the footer component so its four old routes link directly to final 200 destinations, then re-crawl every template using that footer" is testable.

Use the template-level technical SEO fixes when the diagnosis points to shared code. Use the on-page audit when one representative page needs content, heading, media, and internal-link review rather than a sitewide crawl fix.

What not to do

Do not export every issue and call it an audit. Filters report conditions. They do not know the page's job or business value.
Do not assume Screaming Frog is Google. A desktop crawl and Google's stored view differ in timing, discovery sources, rendering, and processing.
Do not treat missing Performance data as zero demand. The selected period, canonical aggregation, export limits, privacy filtering, and a new page can all affect the row set.
Do not force 100% index coverage. Legitimate duplicate, redirected, noindexed, and utility URLs should not all become canonical search results.
Do not change crawl settings between baseline and validation without recording it. Different scope can create a false improvement.
Do not request indexing before the fix is live. First verify the response, rendered output, directives, canonical, and internal links.

Validate the implementation in two stages

Stage one is immediate technical validation. Repeat the same scoped crawl, compare it with the saved baseline, and test representative affected URLs. Confirm the first response, final response, rendered directive, canonical, inlinks, sitemap membership, and template behavior. Make sure a falling issue count was caused by the fix rather than an exclusion or failed crawl.

Stage two is Google validation. Use URL Inspection on a small set of critical representatives. Compare Google's indexed view with the live test, then request indexing only when appropriate. Monitor the relevant Page indexing group and Performance pages over the following crawls. Search Console data can lag, so deployment success and Google processing are separate checkpoints.

Finding:
Affected pattern:
Baseline crawl file:
Search Console evidence:
Expected behavior:
Implementation owner:
Deployment date:
Same-config re-crawl result:
URL Inspection live result:
Google indexed-view follow-up:
Final decision: RESOLVED / MONITOR / REOPEN

Sources checked

Screaming Frog crawl configuration, Search Console integration, crawl analysis, and Google reporting and inspection behavior were verified against official documentation on August 4, 2026.

Mobile SEO Checklist: Test the Page Google and Customers See

Stepan Nikonov — Sun, 06 Sep 2026 07:59:35 +0000

Test the exact canonical URL on a real phone and through Search Console's URL Inspection tool. Compare Google's indexed crawl with the live Google test, then confirm that the mobile page contains the complete primary content, metadata, structured data, images, and crawlable links. Complete the page's real task with touch, a small screen, and the on-screen keyboard.

A responsive layout or green Lighthouse score does not prove those conditions.

Google uses the mobile version of a site's content, crawled with its smartphone agent, for indexing and ranking. There is still one search index, not a separate mobile index. The practical consequence is simple: content or links missing from the mobile result may also be missing from the evidence Google uses to understand the page.

This checklist keeps performance in its own lane. If the issue is LCP, CLS, or INP, use the separate Core Web Vitals diagnosis. Here the question is whether the mobile page is complete, accessible, navigable, and able to finish the customer's task.

Use three views of the same URL

No single test shows the whole mobile experience. Audit one representative URL through these three views, then repeat the relevant checks on each important template.

View	What it can prove	What it cannot prove alone
URL Inspection live test	Google-InspectionTool can fetch the live URL, render it, and retrieve important resources	That Googlebot received the same result, the page is indexed, ranks, or works well for every customer
Browser device mode	Responsive breakpoints, overflow, DOM content, network failures, and common device sizes	Real touch behavior, keyboard effects, weak connectivity, or every mobile browser
Physical phone	Thumb reach, tap accuracy, menus, forms, calls, payments, and the actual business task	Exactly what Googlebot rendered or which resources it could crawl

Google retired the Mobile-Friendly Test, its API, and Search Console's Mobile Usability report on December 1, 2023. Mobile usability still matters. The useful replacement is a small test stack: URL Inspection for Google's fetch and render, browser tools for implementation evidence, and a real phone for the customer journey.

1. Define the page and task before testing

Choose a page that represents a valuable template or action: a service page, product, category, booking flow, local landing page, lead form, or article. Record the exact clean URL and the task a mobile visitor should complete.

URL: https://example.com/service
Template: service detail
Primary mobile task: understand the offer and request a quote
Secondary task: call the business
Test devices: Android Chrome / iPhone Safari
Google view: URL Inspection live test
Decision: PASS / FIX / INVESTIGATE

Test the whole route, not only the first screen. A restaurant page is not successful because its hero fits the viewport if the menu cannot be opened or the order button fails. An article is not complete if its comparison table or references disappear on a phone.

2. Confirm the same indexable page reaches mobile

Responsive design normally serves the same HTML at the same URL and changes the layout with CSS. Google recommends it because it is the easiest mobile pattern to implement and maintain. Dynamic serving and separate mobile URLs can work, but they create more places for content, metadata, canonicals, redirects, and links to diverge.

For the tested URL, confirm:

The phone reaches the intended final URL instead of an irrelevant homepage or mobile error page.
The response is successful and does not depend on a device-specific redirect loop.
The robots rules, canonical, title, and meta description remain equivalent.
The page includes <meta name="viewport" content="width=device-width, initial-scale=1">.
The layout fits the viewport without routine horizontal scrolling or forced zooming.
CSS, JavaScript, images, fonts, and API responses required for the primary content are not blocked.

A desktop-only page can still be indexed if Googlebot Smartphone can access it, but that does not make it usable. Conversely, a polished mobile shell can fail indexing when it carries a different noindex, canonical, empty render, or blocked resource.

3. Compare Google's indexed crawl with the live test

Open the exact URL in Search Console and compare the indexed information with the live test. The indexed view describes Google's last processed version and reports whether it was crawled as a smartphone or desktop. The live test uses Google-InspectionTool to fetch the current page. It is useful after a fix, but it is not the same crawler as Googlebot and its pass does not mean the URL is already indexed.

After the live test completes, inspect the screenshot, rendered HTML, and loaded resources. Search the rendered HTML for the H1, a distinctive sentence from the main content, important internal links, structured data, image alt text, and the primary call to action. The screenshot is a quick visual clue, while the HTML tells you whether the information exists in the render.

Symptom	Likely cause	Next check
Blank or incomplete screenshot	Blocked assets, rendering error, delayed app shell, or server response difference	Resource list, console, server logs, rendered HTML
Main copy absent from HTML	Content requires a click, swipe, typing, or later API request	Disable interaction and reload; inspect the initial and rendered DOM
Images or styles missing	robots.txt, CDN permissions, expiring URLs, mixed content, or incorrect paths	Open each failed resource and compare the mobile request
Wrong canonical or robots rule	Mobile template, edge cache, plugin, or environment drift	Compare response headers and rendered head across devices
Navigation links absent	Menu items injected only after interaction or removed at the breakpoint	Inspect rendered anchors and the mobile navigation source

Do not lazy-load primary content only after user interaction. Google documents that its crawler does not swipe, click, or type to reveal that material. Content can sit inside an accordion for a compact layout, but the information should already be present and available to the rendered page.

4. Compare mobile and desktop content parity

Content parity does not mean identical pixels or identical DOM order. The mobile design can stack columns, collapse supporting sections, or change image crops. It does mean that the same page job and important evidence survive the layout change.

Primary content: the answer, product details, service scope, prices or qualifiers, evidence, and next step remain available.
Headings: clear and meaningful headings still describe the same sections.
Metadata: title and meta description are equivalent across served versions.
Structured data: the same relevant entities and correct URLs are present in the mobile render.
Images and video: useful media, alt text, captions, and supporting context do not disappear.
Internal links: important discovery and next-step links remain in the rendered HTML.

Use a text or DOM comparison when dynamic serving or separate mobile URLs are involved. For a responsive site, resize the page and inspect CSS rules that hide or replace blocks. A visually hidden duplicate can also create confusion, so the goal is one complete, coherent mobile page, not two overlapping copies.

5. Test navigation without hover or guesswork

Open the menu with touch, follow each important route, use the browser Back button, and repeat with the phone rotated. The menu control should have a clear accessible name, visible focus, a correct expanded state, and a close path that does not trap the user.

Important destinations should use normal crawlable links such as <a href="/services">. A styled <div>, a click handler without a resolvable URL, or a desktop link removed from mobile navigation can make a page harder for customers and crawlers to find. Google says limiting the links in a separate mobile version can slow discovery because it indexes the mobile version.

Check the page beyond the global menu:

Breadcrumbs or back paths remain understandable.
Cards have a clear destination and do not require hover to reveal it.
Pagination and load-more states expose crawlable page URLs where discovery matters.
Sticky headers do not cover anchored headings or the first form field.
Phone, email, map, and download actions open the expected handler.

6. Check tap targets, zoom, and overlays

Use a thumb, not a mouse cursor. Test menu icons, close buttons, filters, accordions, pagination, checkboxes, and calls to action near the screen edge. Look for accidental taps, controls that overlap, and actions that appear only after hover.

WCAG 2.2 Level AA sets a 24 by 24 CSS pixel minimum for pointer targets, with documented exceptions including sufficient spacing and inline text links. Its enhanced Level AAA target is 44 by 44 CSS pixels. Google's web.dev guidance recommends aiming around 48 device-independent pixels with space between touch controls. For frequent or consequential actions, the larger practical target is the better design goal.

Do not disable user zoom. A layout that only works by preventing pinch zoom hides an accessibility problem instead of solving it. Let the viewport fit naturally and let users enlarge content when they need to.

Dismiss every banner, consent dialog, chat launcher, and promotion at the tested breakpoint. A close icon must be reachable and remain visible when the on-screen keyboard opens. Google recommends banners that occupy a small portion of the screen instead of promotional interstitials that obscure the primary content.

7. Complete the form and business action

A mobile SEO check is incomplete until the page does the thing it promises. Submit a test lead, book a test appointment, add a product, start checkout, call the number, open directions, or download the file. Confirm the result arrives in the real destination without duplicate submissions.

For forms, verify:

Every input has a persistent visible label and a useful error message.
Email, telephone, number, date, and address fields invoke sensible mobile keyboards or pickers.
Autofill and password managers do not overwrite unrelated fields.
Validation does not clear correct values or jump to an invisible error.
The keyboard does not cover the active field, submit button, or error summary.
Success is visible and the lead, booking, order, or message arrives once.
The action still works with a slow connection or a brief retry.

Record operational failures separately from indexation failures. A form can be fully crawlable and still lose every mobile lead. That makes it a critical business defect even when no SEO tool raises a warning.

8. Check responsive media and layout boundaries

Inspect the page at a narrow phone width, a larger phone, and at least one intermediate width. Breakpoints often fail between the devices a designer chose for screenshots.

Text reflows without a fixed-width container pushing the page sideways.
Tables wrap, scroll inside a labelled container, or transform without losing relationships.
Images use an appropriate source for the viewport and retain meaningful crops.
Video, maps, embeds, code blocks, and long URLs stay within the content column.
Image width and height reserve the correct aspect ratio.
Important text is real HTML, not trapped only inside an image.

These checks overlap with delivery, but this guide does not grade loading performance. If large media or scripts cause slow loading, layout shift, or poor interaction responsiveness, move that evidence into the performance workflow and validate the mobile task again after the fix.

9. Test more than one phone context

One device cannot represent the whole audience. Use analytics or business evidence to choose a small matrix, then include the riskiest context instead of chasing every model.

Context	Why it matters	Practical test
Small viewport	Exposes overflow, cramped controls, and sticky-overlay conflicts	Older or compact phone width
iOS and Android	Browsers, keyboards, date controls, and safe areas differ	Safari on iPhone and Chrome on Android
Slow or unstable network	Late assets and retries can break the task	Browser throttling plus one real cellular test
Text zoom or larger system text	Reveals clipped labels and fixed-height controls	Increase text size and repeat navigation and form tasks
Signed out or fresh session	Consent, location, login, and first-visit overlays change the page	Private window or cleared test profile

Prioritize representative templates and conversion routes. A page-level SEO audit can supply the intended query and page job; this mobile checklist then verifies that the same job survives the smaller screen and smartphone crawl.

What not to do

Do not use a Lighthouse score as the mobile audit result. It is useful evidence, not a content-parity, crawl, navigation, or conversion test.
Do not remove useful content just to shorten the mobile page. Reorganize it with clear sections or accordions while keeping it available.
Do not require a click, swipe, or search to load primary indexable content. Googlebot will not perform those interactions.
Do not block CSS, JavaScript, images, or API resources that Google needs to render the page.
Do not hide navigation behind non-links. Important routes need resolvable <a href> destinations.
Do not rely only on device emulation. It does not reproduce a real finger, keyboard, browser chrome, app handoff, or unreliable connection.
Do not cover the first screen with a promotional interstitial. Use a compact, dismissible banner when the message is necessary.
Do not change URLs during a responsive redesign without a real need. If a move is required, use the site migration SEO checklist.

Final mobile SEO validation record

Finish with observed evidence and an owner:

Canonical URL and response: PASS / FIX / INVESTIGATE
Indexed mobile render and live Google test: PASS / FIX / INVESTIGATE
Primary content parity: PASS / FIX / INVESTIGATE
Metadata and structured data: PASS / FIX / INVESTIGATE
Navigation and crawlable links: PASS / FIX / INVESTIGATE
Tap targets and overlays: PASS / FIX / INVESTIGATE
Form or business task: PASS / FIX / INVESTIGATE
Responsive media and overflow: PASS / FIX / INVESTIGATE

Highest-impact defect:
Evidence and reproduction steps:
Template or page owner:
Release date:
Post-release test:

Re-run the failed task after deployment on the public URL. Then use URL Inspection's live test to confirm the final Google render. For a redesign or platform move, include this record in the broader pre-launch and post-launch comparison so mobile parity does not disappear between environments.

Sources

This checklist is based on current Google Search Central, Search Console, and web.dev guidance.

Programmatic SEO for Small Sites: Scale Only What Deserves a Page

Stepan Nikonov — Tue, 01 Sep 2026 14:22:31 +0000

A small site should use programmatic SEO only when every publishable row represents a distinct entity or situation, answers a repeatable user job, and contains enough verified data to make its page independently useful.

A shared layout is fine. Shared substance with one token changed is not.

Programmatic SEO is a production method, not a promise to publish thousands of pages. Twelve pages generated from one verified dataset and one controlled template still use a programmatic system. The value comes from consistency and maintainability, not URL count.

This workflow covers the dataset, template, eligibility, pilot, and lifecycle decision. Ordinary editorial architecture belongs in a topic cluster plan. Pages that have already lost their purpose belong in an old-page cleanup decision.

Three reasons each page must exist

Reason	Pass condition	Failure signal
Distinct job	A person needs a result for this exact entity, location, compatibility pair, listing, or data slice	The page exists only because a keyword modifier exists
Distinct evidence	The dataset supplies facts, availability, attributes, examples, or constraints specific to the row	The same paragraph works after replacing one noun
Maintainable lifecycle	An owner can verify, update, pause, and retire the row when its source changes	No one knows when the data became wrong or what should happen next

If one reason is missing, keep the information inside a broader page, searchable directory, filter interface, or internal tool until stronger evidence exists.

Good and bad candidates

Candidate	Likely decision	Why
Branch pages with real addresses, hours, services, staff, access details, and local contacts	Possible	Each location can satisfy a distinct visit or contact job
Software integration pages backed by tested capabilities, setup limits, and failure states	Possible	Each pairing can answer a distinct compatibility task
Current listings, events, properties, or jobs with verified attributes and expiration rules	Possible	The entity and availability create a page-level need
Service plus every city where the business has no local proof or operational difference	Reject	Changing the city name does not create useful local evidence
Every color, price, size, and sort combination from a filter interface	Usually reject	Combinations can create duplicate or near-infinite URL inventory
AI-written definitions for every keyword variation	Reject	The process scales wording rather than user value

Google's scaled content abuse policy focuses on many pages created primarily to manipulate rankings rather than help users. It applies whether those pages were produced by AI, conventional code, people, or a mixture.

Automation is not automatically the problem. Scaling content without useful row-level value is.

Step 1: define one repeated page contract

Write the job without mentioning SEO or a keyword:

This page helps [audience] complete [repeatable task]
for [specific entity or situation] using [row-level evidence].

A location page might help a prospective tenant evaluate one building using verified availability, amenities, access, policies, and contact information. The same layout can repeat, but the decision evidence must come from that building's record.

Record the rejection boundary at the same time:

This page will not exist when:
- the entity has no stable unique identity;
- required evidence is missing or expired;
- another page already answers the same job;
- the only difference is a keyword modifier;
- the row cannot be maintained after launch.

This boundary prevents the generation system from treating every database row as an indexable destination.

Step 2: create a data contract before a template

The database decides whether the system can create useful pages. The visual template cannot repair a weak source.

Define required fields, their owner, their source, and what missing data means.

Field	Purpose	Required rule
`entity_id`	Stable identity independent of title or URL	Unique and never reused
`public_name`	Accurate label visible to users	Verified, not keyword-expanded
`page_job`	Repeatable task this row satisfies	Must match the template contract
`unique_evidence`	Facts or utility specific to the row	Minimum completeness threshold must pass
`source`	Origin of every important fact	Named and reviewable
`verified_at`	Freshness control	Within the defined review window
`parent_id`	Directory or category that links to the page	Must resolve to a crawlable parent
`publication_state`	Draft, eligible, published, paused, or retired	Only eligible rows can build indexable pages
`rejection_reason`	Explains why a row did not publish	Required for failed gates

A source timestamp alone does not prove accuracy. Assign a responsible owner and a review method.

For third-party feeds, record what happens when the feed is late, incomplete, or contradictory. The safe default is to pause publication rather than fill gaps with generic copy.

Step 3: separate structure from value

The shared template should control consistent presentation:

navigation, breadcrumbs, headings, and component order;
field labels and units;
disclosures, source notes, and update dates;
related-entity and parent links;
structured data that matches visible content;
empty-state and error behavior.

The row must supply the useful substance:

the direct answer for this entity or situation;
verified attributes, constraints, availability, or compatibility;
specific examples, media, calculations, or next actions where relevant;
known limitations and the last verification date;
a meaningful relationship to parent and related records.

Do not create artificial uniqueness by asking AI to rewrite the same introduction several ways.

AI can help normalize source records, flag missing fields, draft a summary from verified facts, or identify unusual rows for review. It should not invent local knowledge, product behavior, availability, or first-hand experience.

Step 4: make eligibility a hard gate

Keep publication logic explicit and testable:

publishable =
  stable_unique_id
  AND distinct_user_job
  AND required_fields_complete
  AND source_is_current
  AND unique_evidence_threshold_passed
  AND no_existing_page_collision
  AND crawlable_parent_exists
  AND lifecycle_owner_assigned

Failure should produce no indexable page.

Do not build a thin URL and hope a canonical, noindex, or future editorial pass will repair the inventory later. Store the rejected row and its reason outside the public index until it qualifies.

Use a deterministic URL key. Titles and marketing labels change, so they make poor permanent identifiers.

Prevent multiple parameter orders, capitalization variants, tracking parameters, and filter paths from creating alternate public versions. A self-referencing canonical supports a clean page, but canonicalization is not a substitute for controlling duplicate URL generation.

Step 5: plan discovery without infinite crawl paths

Every eligible page needs a normal crawlable link from a useful parent, category, directory, or related page. Search boxes and client-side filter controls alone are weak discovery paths because crawlers do not generally submit searches.

Link important destinations with normal <a href> elements.

Decide which filters are navigation and which deserve indexable landing pages. Most sorting, session, and empty combinations do not.

Google's faceted-navigation guidance warns that parameter combinations can create a near-infinite URL space. When a combination has no results and should not exist, return an appropriate 404 response instead of redirecting every empty state to a generic page.

Include only intended canonical pages in the XML sitemap. A sitemap helps discovery but does not guarantee crawling or indexing. It also does not make an orphaned or low-value page useful.

Crawl budget is rarely the small-site business case

Google's current crawl-budget guide is aimed primarily at sites with roughly one million or more changing pages, sites with ten thousand or more pages changing daily, or sites with a large share of discovered-but-not-indexed URLs. These are rough classifications, not thresholds to chase.

For a small programmatic project, the immediate risks are usually simpler:

weak evidence;
duplicate jobs;
broken or stale data;
orphan pages;
an inventory no one maintains.

Keep the sitemap accurate, review the Page Indexing report, and control unnecessary URLs. Do not justify a bad template with an advanced crawl-budget story.

Step 6: build a deliberately small pilot

Generate a sample that exposes variation instead of selecting only the best records:

one complete, high-confidence row;
one row near the minimum evidence threshold;
one record with optional fields missing;
one entity close to an existing page;
one expired or paused state;
one zero-result or invalid combination.

For many small teams, a first public cohort of roughly 10 to 25 pages is manageable enough for manual review. This is an operational suggestion, not a Google rule. The right number is the number your owner can inspect, support, and reverse safely.

Review layer	What to inspect
Rendered page	Direct answer, headings, missing fields, source note, mobile layout, images, and next action
HTML and status	200 response, unique title and H1, canonical, robots state, crawlable links, and structured data parity
Dataset	Stable ID, source accuracy, duplicate records, dates, and rejection logic
Architecture	Parent path, related links, closest existing page, and collision boundary
Lifecycle	Update, pause, expiration, removal, redirect, and owner behavior

The pilot is not just a smaller launch. It is the test suite for the publishing model.

Step 7: measure before expanding

Track the production system and search outcome separately:

source rows, eligible rows, rejected rows, and rejection reasons;
published pages, build errors, stale records, and manual corrections;
submitted canonical URLs and sampled Google-selected canonicals;
indexed pages and Page Indexing classifications;
queries and landing pages that match the intended job;
qualified actions such as contact, booking, application, or product use;
maintenance time and error rate per update cycle.

Expand only when the sample answers its intended job, the data stays accurate, pages remain distinct, and the team can maintain the lifecycle.

Impressions on unrelated queries, widespread duplicate canonical selection, repeated manual corrections, or no useful action are reasons to stop and repair the model.

Retirement belongs in the template

Define the exit before launch:

Change	Typical response
Temporary data delay	Keep the last verified page only if it remains accurate, label the date, and pause new generation
Required evidence disappears	Remove index eligibility until the record qualifies again
Entity permanently moves to an equivalent replacement	Use a direct permanent redirect when the replacement satisfies the same job
Entity ends with no relevant replacement	Return 404 or 410 and remove links and sitemap entry
Two records are actually one entity	Consolidate the data and URL signals into one maintained page
Users need the page but search does not	Keep the operational page and consider a crawlable `noindex` state

Common failure modes

Starting with a keyword matrix. Multiplying cities by services creates URLs before proving distinct needs or evidence.
Letting the template hide missing data. Generic paragraphs make an incomplete row look finished.
Using AI as the source. Generated prose cannot verify availability, compatibility, location details, or first-hand claims.
Publishing rejected rows with noindex. This leaves avoidable inventory and maintenance debt in production.
Indexing every filter. Valid user controls do not automatically deserve search landing pages.
Relying on the sitemap for architecture. Pages still need meaningful crawlable relationships.
Launching the whole dataset. A template error becomes a sitewide error immediately.
Having no retirement state. Listings, jobs, properties, offers, and integrations change.
Counting URLs as success. The goal is useful outcomes from accurate pages, not inventory growth.

Go or no-go checklist

One repeatable user job exists independently of keyword variations.
Every entity has a stable unique ID and named data source.
Required row-level evidence is defined and testable.
Missing or stale data blocks publication by default.
The template presents verified differences instead of rewritten boilerplate.
Every eligible page has a useful crawlable parent and clean canonical URL.
Filter combinations and alternate URL forms cannot explode the public inventory.
A heterogeneous pilot can be manually inspected and reversed.
Search outcomes, business actions, data errors, and maintenance cost will be measured.
Pause, merge, redirect, noindex, and removal states have owners and rules.

If those conditions feel stricter than writing pages manually, the system is working. Programmatic SEO earns its place when structured evidence makes repeated usefulness cheaper and more reliable. It should never make weak pages cheaper to publish.

Sources checked

Search policy, crawling, indexing, and structured-data guidance were verified from official Google documentation on August 2, 2026.

Do You Need llms.txt? AI Crawler Controls Explained

Stepan Nikonov — Sun, 30 Aug 2026 02:01:13 +0000

Most small-business websites do not need an llms.txt file. It can provide optional context, but it does not grant crawler access, block model training, improve rankings, or guarantee an AI citation.

Publish /llms.txt only if you maintain a documentation-heavy or structured knowledge site and can keep a curated machine-readable map accurate. For an ordinary service business, prioritize crawlable pages, internal links, a clean sitemap, correct structured data, and explicit crawler rules.

The distinction matters because llms.txt and robots.txt solve different problems. The llms.txt proposal describes a Markdown overview for use at inference time. Its own specification says it complements existing standards. It does not define access permission or an opt-out mechanism.

Use the control that matches the job

Control	Actual job	What it cannot promise
`robots.txt`	Tell compliant crawlers which paths they may request	Removal from search, privacy, or universal enforcement
`llms.txt`	Offer a curated Markdown description and links	Crawler access, training opt-out, ranking, or citation
`sitemap.xml`	List preferred indexable URLs for search discovery	Indexing or selection in an AI answer
`noindex` and snippet controls	Control indexing or how much Google may show from a page	A safe result when the crawler is blocked from reading the directive
CDN or WAF rules	Permit or deny requests at the infrastructure layer	Correct intent if rules rely only on a spoofable user-agent string

Do not treat a public text file as a privacy boundary. Never list private dashboards, customer portals, staging hosts, unpublished documents, or sensitive paths in llms.txt. Protect them with authentication and access controls.

What llms.txt is

The proposal places a Markdown file at /llms.txt. It starts with one required H1, may include a short summary and explanatory text, and can group useful links under H2 headings. The specification also proposes optional Markdown versions of detailed pages.

That can be useful for a software library, API, public knowledge base, or complex documentation set. A maintainer can point an agent toward a stable quick start, current reference, examples, and known limitations without asking it to infer the hierarchy from navigation and decorative HTML.

The proposal does not specify how an AI provider must discover, fetch, or use the file. It explicitly leaves processing decisions to each application. Publishing the file is therefore an experiment, not an access-control project.

Why most small-business sites do not need it

A typical service site has a homepage, service pages, a few decision guides, contact information, and perhaps a local profile. Those pages should already explain the business clearly in rendered HTML. Adding another summary file creates a second place where services, locations, prices, policies, and URLs can become stale.

Google's official guidance is especially direct. Pages need normal Search eligibility for AI Overviews and AI Mode, and Google says no new machine-readable file, AI text file, or special markup is required. Googlebot access, indexability, internal discovery, visible text, page experience, and accurate structured data remain the foundation.

Site situation	Recommendation	Reason
Local or service business	Skip it for now	Improve the actual service pages and crawler access first
Small editorial site	Usually skip it	The sitemap, hubs, and contextual links already express the library
Large public documentation set	Consider a test	A curated route can reduce ambiguity for tools that choose to read it
API or developer product	Consider a generated file	Versioned references and examples may benefit from a concise machine-readable index
Site with weak crawlability	Fix access first	An optional file cannot repair blocked pages, bad status codes, or a hostile WAF

Separate search retrieval from model training

There is no single universal “AI bot” switch. Vendors document separate agents for different jobs, and a site owner may want search visibility without allowing potential training use.

Provider	Search or user retrieval	Separate control
Google	`Googlebot` controls Search, including AI Overviews and AI Mode	`Google-Extended` controls certain Gemini training and grounding uses outside Google Search
OpenAI	`OAI-SearchBot` supports ChatGPT search discovery	`GPTBot` is the published control for potential training
Anthropic	`Claude-SearchBot` and `Claude-User` support search and user-directed retrieval	`ClaudeBot` is used for content that could contribute to model training
Perplexity	`PerplexityBot` supports search results; `Perplexity-User` supports user-requested fetches	Perplexity says these two agents are not used to train foundation models

Google states that blocking Google-Extended does not affect inclusion or ranking in Google Search. OpenAI's publisher guidance similarly separates OAI-SearchBot from GPTBot. Anthropic documents three distinct agents so publishers can make separate choices.

Build the policy before editing robots.txt

Write the business decision first. Otherwise a copied rule can quietly block the exact search surface you intended to keep.

List the public content. Decide which pages may be retrieved by search and user-directed agents.
Separate model development. Decide whether to block training-specific agents where a vendor provides that control.
Protect private content properly. Use authentication, authorization, network restrictions, or removal. Do not rely on crawler courtesy.
Inspect the whole delivery path. Robots.txt, CDN, WAF, hosting firewall, rate limits, JavaScript challenges, and origin responses can disagree.
Record an owner and review date. Crawler names, IP ranges, and product behavior change.

The AI search citation guide covers content eligibility and evidence after access is working. The technical SEO fixes guide covers infrastructure problems that ordinary content edits cannot solve.

A cautious robots.txt example

This example expresses one specific policy: allow documented search agents, block documented training controls, and leave normal search crawling open. It is not a universal recommendation. Read every vendor's current documentation and your existing robots file before adapting it.

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

This does not remove a page from search. A robots disallow can prevent a crawler from seeing a noindex directive on the blocked page. If removal is the goal, follow the search engine's documented removal or indexing controls instead of combining random directives.

It also does not create a hard security block. Robots rules are instructions for compliant agents. A scraper can ignore them, and a legitimate agent can still be stopped by an overly broad firewall rule.

Check the CDN and WAF separately

Allowing a crawler in robots.txt is only the first gate. OpenAI and Perplexity both tell publishers to check CDN, WAF, bot mitigation, response codes, and published IP ranges when access fails. Perplexity recommends combining the expected user agent with its current IP list rather than trusting the header alone.

A safe validation sequence is:

Request the live /robots.txt and confirm the deployed file matches the policy.
Request representative public URLs with the documented user-agent token.
Confirm a successful response, useful HTML, and no challenge page.
Inspect CDN and origin logs for 401, 403, 429, redirect loops, and unusually large responses.
Where the vendor publishes IP ranges, verify both the header and source range before creating an allow rule.
Recheck after deployment because caches and crawler processing can delay the observed change.

curl -I -A "OAI-SearchBot" https://example.com/important-page
curl -I -A "Claude-SearchBot" https://example.com/important-page
curl -I -A "PerplexityBot" https://example.com/important-page

A forged user-agent can pass this simple request test. Use it to find obvious blocks, not to authenticate crawler identity.

When an llms.txt test is worth maintaining

Consider a limited test when all of these conditions are true:

the public site contains a large, stable documentation or knowledge corpus;
the preferred entry points and current versions are difficult to infer from navigation alone;
the file can be generated from the same source of truth as the documentation;
every linked URL is public, canonical, useful, and maintained;
the team can test named tools or agents instead of treating the file as a universal standard;
removing the experiment would not damage ordinary crawling, indexing, or user navigation.

If you publish it, keep the file concise. Name the project, explain the scope, link only to preferred public resources, and record how it is generated. Do not paste your sitemap into Markdown or duplicate full pages into a second unmanaged content layer.

What not to do

Do not use llms.txt as an opt-out file. Use each provider's documented crawler token and access controls.
Do not block every AI-related token. A training control and a search bot may have different jobs.
Do not list sensitive paths. Public crawler files are not a substitute for authentication.
Do not create a stale second website map. Generate the file from maintained data or do not publish it.
Do not expect a citation lift. Access and context create eligibility, not selection.
Do not trust a user agent by itself. Use official IP data and logs when infrastructure policy depends on identity.
Do not let the experiment delay basic SEO. Fix status codes, internal links, canonicals, sitemaps, rendered text, and page quality first.

FloxoLab's current decision

FloxoLab does not need llms.txt to make its guides eligible for Google AI features or ChatGPT Search. The current priorities are a crawlable public library, one canonical URL per task, accurate metadata, responsive media, contextual links, and explicit crawler access.

A generated file may become worth testing after the SEO guide library expands and the filterable hub is stable. If that happens, the file should be derived from the content manifest, not maintained as another hand-written index. The topic-cluster guide explains why one source of truth matters when a library grows.

Decision checklist

The business goal is written before crawler rules are changed.
Search retrieval and potential training are treated as separate decisions.
Private content is protected with real access controls.
Robots.txt, CDN, WAF, and origin behavior agree.
Official crawler documentation and IP sources were checked recently.
Important public pages return useful HTML to allowed agents.
An llms.txt experiment has a narrow purpose, owner, generator, and removal plan.
No ranking or citation outcome is promised.

For most small businesses, the right answer is simple: skip llms.txt for now, make the real pages easy to crawl and understand, and control each crawler according to the job it performs.

Sources checked

Behavior and crawler names were verified from primary or official documentation on August 2, 2026.

SEO Automation with n8n: What to Automate Safely

Stepan Nikonov — Thu, 27 Aug 2026 17:09:01 +0000

SEO automation is safest when it collects evidence, applies repeatable rules, and asks a person to approve changes that could affect indexing, rankings, or published copy.

n8n is useful for that middle layer. It can pull Search Console data, compare periods, filter noisy rows, log findings, and route a review queue without being allowed to rewrite the site.

A workflow that says “these five pages lost impressions after a release” saves time. A workflow that changes five canonical tags because a metric moved can damage the site before anyone checks the reason.

Start with the native option

For an occasional check, use Search Console directly. Its Performance report already lets you filter by query, page, country, device, search appearance, and date. You can compare periods and export a table without building or maintaining a workflow.

n8n becomes useful when the work is recurring or needs more than one system:

run the same report every week;
compare a current window with a previous equivalent window;
ignore changes below a minimum impression threshold;
join SEO data with a release log or content inventory;
deduplicate alerts that have already been reviewed;
send different issues to different owners;
preserve an audit trail of what was detected and decided.

The automation should reduce repetitive preparation. It should not conceal how the decision was made.

Use a three-level safety model

Level	Automation role	Examples	Default control
Observe	Read data and record facts	Search Console exports, sitemap checks, status monitoring	Can run unattended
Recommend	Apply rules and prepare a review queue	flag falling pages, find sitemap mismatches, draft a title option	A person reviews the evidence
Change	Modify a public or indexable resource	edit copy, redirects, canonicals, robots rules, internal links	Explicit approval plus rollback

Most small sites should keep their first SEO workflow in the first two levels. The potential gain from automatically changing a title is small compared with the cost of changing the wrong page or losing the original context.

What is generally safe to automate

Scheduled Search Console exports

Google provides a Search Analytics API for querying performance data. A request can group results by dimensions such as page, query, country, and device, and can apply filters before returning rows.

The API is not a complete event log. Google states that the service does not guarantee every row and returns top rows rather than all rows. Search Console also omits anonymized queries for privacy. Treat the export as strong first-party evidence with known limits, not an exhaustive ledger of every search.

Period comparisons with noise controls

A useful workflow can calculate:

click change = current clicks - previous clicks
impression change = current impressions - previous impressions
CTR change = current CTR - previous CTR

Do not alert on percentage change alone. A fall from two impressions to one is a 50 percent decline but is rarely a useful incident. Require a minimum volume and a meaningful absolute change.

review when:
current impressions >= 100
and absolute impression change >= 50
and percentage change <= -30%

Those values are starting rules, not universal SEO standards. Tune them to the size and volatility of the site.

Technical monitoring

n8n can schedule narrow checks for:

important URLs returning a non-200 status;
sitemap URLs that redirect or fail;
canonical targets that no longer resolve;
a robots.txt or sitemap fetch that suddenly changes;
a critical page becoming unavailable;
a release list that does not match the expected public URLs.

Do not turn a small n8n instance into an uncontrolled crawler. A broad crawl can overload the site, duplicate a proper crawler, or create false alarms from temporary rate limits.

Evidence packaging

Instead of sending “traffic down,” build a review packet with:

affected URL;
metric and comparison windows;
absolute and percentage change;
minimum sample threshold;
recent release or content-change date;
current status code and canonical;
a link to Search Console for manual investigation;
the previous decision if the same issue was reviewed before.

That makes the alert actionable and keeps the workflow from pretending it knows the cause.

What should not run without review

Do not let a first SEO automation publish or change these items by itself:

titles, descriptions, headings, or body copy;
canonical tags;
redirects;
noindex, robots.txt, or crawl directives;
sitemap membership;
internal links at scale;
deletion or consolidation of pages;
repeated indexing requests;
AI-generated articles or local pages.

The same metric can have several causes. Falling clicks may reflect lower demand, a ranking change, a different result layout, a tracking issue, a migration, or a page that is intentionally targeting a narrower audience.

A workflow can identify the symptom. It cannot infer the business decision from one number.

A safe weekly n8n workflow

Schedule Trigger
  -> HTTP Request: Search Analytics API
  -> normalize rows
  -> compare with previous period
  -> apply volume and change thresholds
  -> check the alert ledger
  -> write a review record
  -> notify the owner

Step 1: choose equivalent periods

Compare like with like. A seven-day period can be compared with the previous seven days, but weekday patterns and holidays can still distort a small sample. For a low-volume site, 28-day windows are usually more stable.

Record the exact dates. “Last month” becomes ambiguous when somebody opens the alert later.

Step 2: query only the data you need

The Search Analytics API accepts a rowLimit from 1 to 25,000 and uses startRow for pagination. Google also warns that queries grouped or filtered by page and query can be more expensive against API load limits.

Start with the smallest useful query:

one search type;
one property;
only the required dimensions;
a bounded date range;
a row limit appropriate to the site.

Do not repeatedly query long historical ranges on every execution. Store completed period summaries and fetch only the window that needs updating.

Step 3: label preliminary data

Recent Search Console data can be incomplete. The API can return metadata showing that a result contains incomplete or still-processing data.

A weekly report should either avoid the newest days or label them so a reviewer does not compare a final period with a preliminary one.

Step 4: normalize before comparing

Keep the comparison key explicit. If the workflow groups by page, normalize the public URL rules first but do not silently merge distinct pages.

Store raw input separately from calculated fields. That makes the result inspectable when a threshold behaves unexpectedly.

Step 5: prevent duplicate alerts

Create a stable key such as:

property + report_type + page + current_period_end

Before sending a notification, check whether that key is already in the alert ledger. Update an existing record when the evidence changes instead of creating a new task every time the workflow runs.

Step 6: notify, then wait for a decision

Send the review packet to email, Slack, Google Chat, Telegram, or the team’s normal task system.

For a workflow that must continue after a person responds, n8n can use a Wait step or an app operation that waits for approval. Record the reviewer, timestamp, and decision before any downstream action.

Example review logic

Finding	Workflow can do	Person must decide
Important URL returns 404	capture response, timestamp, expected URL, and recent release	restore, redirect, or intentionally retire
Sitemap URL redirects	list source and destination and check destination status	update sitemap, redirect map, or page plan
Impressions fall 40 percent	verify sample, compare periods, attach page and query rows	whether demand, rankings, intent, or a site change caused it
Two pages appear for one query	package page-level Search Console data	whether intents differ or pages should be clarified or merged
CTR falls while impressions rise	flag query and page combinations	whether the snippet, ranking mix, or result layout explains it

Add failure handling before trusting the report

OAuth tokens expire, APIs return rate limits, schemas change, and a partial execution can send a misleading “all clear.”

Fail the report if the source request fails.
Record the number of source rows and filtered rows.
Distinguish “zero findings” from “no data received.”
Retry only temporary failures with a bounded policy.
Send exhausted failures to an Error Workflow.
Store the last successful run time.
Test with a known sample before scheduling production runs.

Use the failed n8n workflow debugging guide when a node fails. Add an n8n Error Workflow so a broken SEO report cannot fail silently.

Cloud or self-hosted n8n?

n8n Cloud is the simpler starting point for a small team because hosting, updates, and the workflow service are managed. Self-hosting can offer more infrastructure control, but it also makes backups, upgrades, security, uptime, and incident recovery your responsibility.

Choose based on operational responsibility, not on the SEO logic. The workflow should use least-privilege credentials either way. A read-only Search Console scope is preferable when the workflow only reads performance data.

A practical launch checklist

Begin with one property and one report.
Use read-only credentials.
Keep recent incomplete data out of final comparisons or label it.
Set minimum volume and absolute-change thresholds.
Store raw evidence and calculation inputs.
Deduplicate alerts.
Require a human decision for public site changes.
Add a rollback plan before automating any approved write.
Monitor the automation itself.
Review the rules after several real runs.

SEO automation is useful when it gives a person a shorter, better evidence queue. It becomes risky when the workflow converts an uncertain signal directly into a site change.

Sources

Originally published on FloxoLab.

How to Deduplicate Leads in n8n Before They Reach Your CRM

Stepan Nikonov — Tue, 18 Aug 2026 16:07:37 +0000

Normalize each lead, compare the right identity keys, update an existing contact when the match is safe, and send ambiguous records to review instead of polluting the CRM.

Lead deduplication is not one filter. A reliable n8n workflow separates repeated submissions from repeated people, normalizes the values used for comparison, checks persistent records, and lets the CRM enforce uniqueness whenever the destination supports it.

Short answer: use a stable source ID to stop the same event twice, normalized email or phone to find an existing person, and a CRM-native create/update or upsert operation for the final write. Never auto-merge two records on name alone.

First identify which duplicate you are stopping

Duplicate type	Example	Best first control
Repeated item in one execution	A spreadsheet import contains the same row twice	Remove Duplicates on the current input
Repeated source event	A webhook or retry delivers submission `lead_8472` again	Persistent check on the stable source ID
Same person from another source	The same customer submits a website form after a Facebook form	Normalized identity lookup in the CRM
Conflicting identity	Email matches one contact while phone matches another	Human review, not an automatic merge

The Facebook Lead Ads workflow uses a Meta lead ID to stop one submission from creating several sheet rows. This guide goes further by handling several sources and an existing CRM full of contacts.

Choose identity keys before building nodes

Key	What it proves	Rule
Source record or submission ID	The same source event or object	Strongest key for replay protection, but not for matching a person across systems
Normalized email	Usually the same inbox	Good exact contact key when present and valid
Normalized phone	Usually the same reachable number	Useful with country context; shared business or family numbers need caution
Name	Only that the text looks similar	Use as review context, never as the only automatic merge key

Decide field ownership too. A new form may fill a missing phone number, but it should not silently replace a verified CRM email, owner, lifecycle stage, or consent value. Deduplication decides which record to use. A separate update policy decides which fields may change.

A practical n8n workflow shape

Lead trigger
  → Edit Fields: normalize keys
  → Remove Duplicates: current batch or repeated event
  → Data Table or CRM: persistent lookup
  → Switch: new / exact match / conflict
  → CRM: create or update
  → Data Table: record source key and CRM ID
  → Alert only when review or failure is needed

Start with one lead source and one CRM. The same pattern can later sit behind Facebook forms, website forms, imports, chat, and email without giving each source a different duplicate policy.

Step 1: normalize the fields once

Add an Edit Fields node immediately after the trigger. Preserve the original values for audit and display, then create separate comparison fields:

source_key
email_original
email_key
phone_original
phone_key
full_name
source
received_at

For an email comparison key, an n8n expression can trim whitespace and use lowercase:

{{ ($json.email || '').trim().toLowerCase() }}

Build source_key from the provider and its stable ID, for example facebook:lead_8472 or website:submission_193. Do not concatenate name and timestamp and call that stable.

For phone numbers, remove presentation punctuation and normalize with the correct country context. Simply stripping every non-digit character can make two different international numbers look deceptively similar.

Step 2: use Remove Duplicates for the narrow job it does well

For duplicate rows arriving together, configure Remove Duplicates → Remove Items Repeated Within Current Input, choose Selected Fields, and compare source_key. This is ideal for imports and batched records.

For the same source event arriving in later executions, the node also supports Remove Items Processed in Previous Executions. Use Value Is New and deduplicate on source_key. Scope can be the individual node or the workflow. Current n8n documentation says the default history size is 10,000 items.

Do not treat that history as your customer database. The history has a configured size, can be cleared, and only knows the values you gave that node or workflow. It cannot decide that two different source IDs belong to the same person.

Step 3: add a persistent lookup

Option A: n8n Data Table for a small lead ledger

A Data Table is a practical first version for a small business that does not yet have strong CRM lookup rules. Create a table such as lead_identity with these columns:

source_key
email_key
phone_key
crm_id
status
first_seen_at
last_seen_at

The current Data Table node can check whether a row exists, get rows, insert, update, and upsert using conditions. Search the strongest available key before the CRM write, then upsert the final crm_id only after the CRM succeeds.

n8n describes Data Tables as light to moderate storage. The default total limit across all tables in an instance is 50 MB, although self-hosted instances can change it. That makes a table useful as a compact identity ledger, not a replacement for a mature CRM or warehouse.

Option B: search the CRM directly

If the CRM already owns the customer record, query it first. The current n8n HubSpot node, for example, supports searching contacts and a Create/Update a contact operation. Other CRM nodes expose different operations, so check the destination's exact contract before copying a pattern.

Prefer a CRM record ID or custom unique source ID once you have one. HubSpot's current Contacts API can retrieve by email and batch upsert by email or a custom unique identifier. It also notes that partial upserts are not supported when email is the identifier, which is one reason not to send an incomplete payload blindly.

Step 4: route new, matched, and conflicting leads

Use a Switch node with three explicit outcomes:

No match: create the contact, then save the returned CRM ID with the source key.
One safe match: update only allowed fields and attach the new source or submission as an activity.
Conflict: do not create or merge automatically. Send the candidate record IDs and incoming values to an owner for review.

A conflict includes more than two search results. It also includes email matching contact A while phone matches contact B, or a source ID pointing to a record whose verified identity now disagrees with the payload. Preserve the lead and notify someone. Silent deletion is not deduplication.

Step 5: make the CRM write duplicate-safe

Use the destination's create/update or upsert operation when it can enforce a unique identifier. This is stronger than a separate search followed by create because two n8n executions can run at nearly the same time, both see no result, and both create a contact before either writes the ledger.

If the CRM cannot upsert atomically, use its unique-field constraint, serialize the critical section, or put the identity key in a database with a unique index. Catch the destination's duplicate response and retrieve the existing record instead of treating every conflict as an unrecoverable failure.

After a successful write, upsert the source key and returned CRM ID into the Data Table. Then send notifications or start follow-up. This ordering prevents the team from receiving a success alert for a contact that never reached the CRM.

Test with a small duplicate matrix

Test	Expected result
Same source ID delivered twice	One CRM record; the replay is stopped or attached without another create
Different source ID, same normalized email	Existing contact updated according to field rules
Same phone with different formatting	One candidate after country-aware normalization
Same name, different email and phone	New record or manual review, never an automatic merge by name
Email matches A, phone matches B	Conflict branch with both record IDs visible
Two identical executions started together	One destination record because the final write enforces uniqueness
CRM write fails	No success alert and no ledger row marked complete

Common mistakes

Comparing raw email or phone values without normalization.
Using name as a unique key.
Assuming a source submission ID identifies the same person across every channel.
Dropping duplicates without recording the new campaign, form, message, or timestamp.
Overwriting verified CRM fields with blank or lower-confidence form values.
Running search and create concurrently without a destination uniqueness control.
Using a temporary in-workflow list as the only long-term duplicate barrier.

Useful questions before publishing

Can Remove Duplicates replace a CRM lookup?

No. It is useful for repeated items and remembered values, but it does not understand an existing customer record, alternate email, shared phone, merge history, or field ownership.

Should email or phone be the main key?

Use the strongest verified identifier your process reliably collects. Email is often the simplest contact key. Phone can be useful after country-aware normalization. Keep a stable source ID for replay protection even when email is the CRM match key.

Should a duplicate lead be discarded?

Usually not. The person may have submitted a new service request or responded to a different campaign. Reuse the contact record, then preserve the new submission as an activity, note, deal, or source event.

Privacy and operating rules

Store only the identity fields needed for matching and follow-up.
Restrict Data Tables, execution data, and CRM credentials to the responsible project members.
Set retention rules for rejected and ambiguous records.
Do not place full lead payloads in chat alerts when a secure CRM link is enough.
Document who resolves conflicts and how quickly the review queue should be checked.

For retries around state-changing API calls, use the separate duplicate-safe n8n API retry pattern. That guide protects one operation from being repeated. This guide protects CRM identity across leads and sources.

Sources checked

Originally published on FloxoLab.

How to Back Up and Restore Self-Hosted n8n

Stepan Nikonov — Sat, 15 Aug 2026 21:21:07 +0000

A workflow export is not a full instance backup. Protect the database, encryption key, deployment configuration, and binary storage, then prove that they work together on an isolated restore.

A recoverable self-hosted n8n backup has four core parts: the database, the original encryption key, the deployment configuration and secrets, and any binary-data storage your executions still need.

Back up the persistent .n8n volume even when PostgreSQL stores the main database. n8n documents that this directory can still contain the encryption key, logs, and source-control assets.

What must be in the backup

Asset	Why it matters	Typical location
Database	Workflows, encrypted credentials, users, projects, settings, and retained execution data	`~/.n8n/database.sqlite` or PostgreSQL
Encryption key	Decrypts credentials stored in the database	`~/.n8n` settings or `N8N_ENCRYPTION_KEY`
Deployment config	Restores database connection, public URL, proxy, timezone, pruning, and execution behavior	Compose file, environment settings, secret references, proxy config
Binary data	Restores retained files handled by executions	Persistent filesystem, database, or configured external store
Custom and community nodes	Allows restored workflows to load the same node types and versions	Package list, custom-node directory, container image, or build manifest

Do not put the raw encryption key, database password, OAuth secrets, or an unredacted environment file in a public repository. Keep a sanitized deployment definition in version control and protect the actual secrets in a restricted password manager, secret store, or encrypted backup.

Choose the database path first

Self-hosted n8n uses SQLite by default. The database file is ~/.n8n/database.sqlite. PostgreSQL is the other supported database option. The backup and restore commands must match the database your instance actually uses.

SQLite: capture a consistent persistent volume

The simplest small-instance method is a short maintenance window: stop the n8n application, archive its persistent volume, then start it again. Stopping writes avoids treating an arbitrary live file copy as a consistent database backup.

docker compose stop n8n
docker run --rm \
  -v n8n_data:/data:ro \
  -v "$PWD/backups:/backup" \
  alpine tar czf /backup/n8n-data-2026-07-18.tgz -C /data .
docker compose start n8n

Replace the service and volume names with the names in your deployment. If downtime is not acceptable, use a storage snapshot or SQLite-aware backup process that guarantees a consistent result. Do not assume that copying database.sqlite while executions are writing to it is safe.

PostgreSQL: use a database dump

PostgreSQL's official documentation recommends pg_dump for logical backups. A custom-format dump works with pg_restore and is practical for restoring into a fresh database:

pg_dump --format=custom --file=n8n-2026-07-18.dump n8n

The dump does not replace the n8n persistent volume or explicit encryption key. It also does not capture the Compose file, environment settings, proxy configuration, or filesystem binary data. Store those as separate parts of the same dated backup set.

Add native n8n exports as a second recovery layer

n8n's Server CLI can export all database entity types. Its documentation positions this tooling for backups and migrations, including moves between SQLite and PostgreSQL. Execution-history data tables are excluded by default because they can be large.

docker exec -u node n8n \
  n8n export:entities \
  --outputDir=/home/node/.n8n/cli-backup

Workflow and credential exports are also useful for selective recovery or versioned copies:

n8n export:workflow --backup --output=backups/workflows/
n8n export:credentials --backup --output=backups/credentials/

Avoid decrypted credential exports for routine backups. n8n supports a --decrypted flag for migrations to a different secret key, but the resulting files expose every sensitive value in plain text.

A safe restore runbook

First restore into an isolated instance with no production webhooks, schedules, email sends, payment calls, or CRM writes. The first recovery test should never overwrite the only production copy.

Record the target. Use the same n8n version first, the same database type, and compatible custom-node versions.
Create fresh infrastructure. Prepare a new Docker volume or empty PostgreSQL database rather than clearing production.
Restore the original encryption key. Put the protected N8N_ENCRYPTION_KEY or original n8n settings in place before n8n reads the restored credentials.
Restore the database. Extract the stopped SQLite volume archive into the new volume, or restore the PostgreSQL dump into the new database.
Restore the surrounding state. Reapply deployment settings, binary storage, custom nodes, public URL settings, and reverse-proxy configuration.
Start without production traffic. Keep DNS, proxy routing, and outbound side effects isolated while checking startup and migration logs.
Verify the application. Confirm login, projects, workflows, credentials, node availability, required execution history, and retained binary files.
Run one safe test. Use test credentials or a non-destructive workflow and confirm that a credential can decrypt and authenticate.
Publish deliberately. Review which workflows should be published before switching traffic.

Restore SQLite into a new volume

docker volume create n8n_restore_data
docker run --rm \
  -v n8n_restore_data:/data \
  -v "$PWD/backups:/backup:ro" \
  alpine tar xzf /backup/n8n-data-2026-07-18.tgz -C /data

Point a separate restore Compose file at n8n_restore_data. Do not bind it to the production hostname or webhook route until the checklist passes.

Restore PostgreSQL into a fresh database

createdb n8n_restore
pg_restore --dbname=n8n_restore n8n-2026-07-18.dump

Real deployments may need explicit host, user, role, ownership, TLS, and schema options. Test the exact command your operator will use and document it without embedding the password.

Restore-test checklist

The backup job has a timestamp, size, success state, and failure alert.
The database and encryption key come from the same recoverable setup.
The restore starts on a separate hostname, volume, and database.
Credential nodes open without decryption errors and one test authentication succeeds.
Required custom or community nodes load at the expected versions.
Published workflows, schedules, and webhook paths are reviewed before traffic moves.
Required retained binary files can be opened.
The restore time and missing manual steps are recorded for the next test.

Common backup failures

Mistake	What breaks during recovery
Only exporting workflow JSON	The complete instance state is not restored
Database without the original key	Credentials remain encrypted but cannot be used
Copying a live SQLite file casually	The backup may not represent one consistent database state
Skipping binary storage	Retained documents or images may be missing
Restoring straight over production	A bad archive or wrong config can remove the working recovery path
Never testing the restore	Missing keys, permissions, packages, and manual steps appear during the incident

Sources

Originally published on FloxoLab.

Retry Failed n8n API Requests Without Creating Duplicates

Stepan Nikonov — Thu, 13 Aug 2026 04:15:40 +0000

A timeout can hide a successful order, lead, payment, or message. The safe retry pattern uses one stable operation key, retries only temporary failures, and records the confirmed result.

Turning on Retry On Fail in n8n does not prevent duplicates by itself. It repeats the failed node. If an API completed the first request but its response never reached n8n, repeating a POST with a new key can create the same object twice.

Decide whether the request is safe to repeat

HTTP defines GET, HEAD, OPTIONS, and TRACE as safe methods. PUT, DELETE, and the safe methods are idempotent by their intended semantics. POST is not automatically idempotent.

Real APIs add their own rules, so the provider's documentation remains the contract. A DELETE endpoint may be harmless to repeat, while a POST that sends an email, creates a payment, or reserves stock needs explicit duplicate protection.

Failure	Default action	Duplicate risk
Network timeout or reset	Retry the identical request with the same operation key	High, because the remote service may already have committed the action
429 Too Many Requests	Wait for the provider's limit window, then retry	Controlled only if the operation remains idempotent
500, 502, 503, or 504	Use a small bounded retry policy when the provider permits it	Potentially high for state-changing requests
400, 401, 403, or most 404s	Fix the request, credentials, permission, or resource first	Blind retries rarely fix the cause
409 Conflict	Inspect the API-specific error before deciding	May indicate an existing operation or key conflict

Step 1: create one key for one business operation

Build the key before the HTTP Request node from an ID that already belongs to the event: an order ID, checkout ID, lead ID, invoice ID, or source event ID. Include the action name so different operations on the same record cannot collide.

={{ 'create-order:' + $json.orderId }}

Keep this value unchanged across automatic node retries, manual execution retries, and later recovery runs. Do not use $now, the execution ID, or a freshly generated random value on each attempt. Those identify the attempt, not the operation, so the destination sees every retry as new work.

The key must also stay paired with the same endpoint and payload. If you correct the amount, recipient, or other business data, treat that as a deliberate new operation according to the provider's rules.

Step 2: use the API's idempotency feature first

If the destination API supports idempotency, send the stable key in the exact header or field its documentation specifies. Stripe, for example, accepts an Idempotency-Key header for POST requests and returns the stored result when the same request is repeated with the same key.

That header name is not universal. Some APIs use a request ID, client reference, deduplication ID, or provider-specific field.

Open the HTTP Request node and add the provider's documented idempotency header.
Map the stable operation key as its value.
Keep the method, URL, and request body identical on every retry.
Save the external object ID from the confirmed response for later reconciliation.

Adding an Idempotency-Key header only works when the destination implements that contract. An unknown header may simply be ignored.

Step 3: configure a bounded n8n retry

Every n8n node has a Retry On Fail setting. When enabled, n8n reruns the node after a failure until it succeeds or reaches the configured attempt limit. For a known transient endpoint, start with a small policy such as three tries and a short wait between tries. Increase it only when the provider's limit and incident behavior justify the extra traffic.

This built-in setting is best when every error produced by that node is safe to retry. If you need status-aware behavior, enable Include Response Headers and Status and Never Error in the HTTP Request node. The node can then pass the status code and headers to an IF or Switch branch instead of stopping on every non-2xx response.

Send 2xx responses to the success branch.
Send 429 and approved temporary 5xx responses to a Wait step and bounded retry path.
Send authentication, validation, permission, and unexpected responses to a review or alert branch.
Stop after the maximum attempt count. Never create an unbounded loop.

Step 4: slow down 429 retries

A 429 response means the client sent too many requests in the provider's current limit window. Some APIs include Retry-After as seconds or an HTTP date. Read and honor it when the provider documents that behavior. If the header is absent, use the API's published limit and increase the delay between attempts rather than retrying immediately.

The HTTP Request node also has Items per Batch and Batch Interval options. They control how many input items are sent in each batch and how many milliseconds n8n waits between batches. Use them to prevent a known burst from creating the 429 in the first place.

Step 5: add a persistent duplicate check when needed

When the API has no idempotency feature, keep a small request ledger. n8n's Data Table node can check whether a row exists, insert rows, update them, and upsert by conditions.

Column	Purpose
`request_key`	Stable ID for the business operation
`status`	`pending`, `succeeded`, `failed`, or `unknown`
`external_id`	ID returned by the destination when available
`updated_at`	Timestamp for recovery and stale-pending review

Use this sequence:

Check for a row with the request key.
If it already says succeeded, return the stored result and skip the API call.
If no row exists, record pending, then call the API.
On a confirmed 2xx response, update the row to succeeded and store the external ID.
On a timeout, keep the state as unknown until you query the destination or retry safely.

This ledger is useful for sequential or low-concurrency workflows, but a check followed by an insert is not an atomic lock. Two executions can check at the same moment and both see no row. For parallel production traffic, prefer destination-side idempotency or a database table with a unique constraint on request_key and an atomic claim operation.

Do not use workflow static data as a production lock

The $getWorkflowStaticData() helper looks convenient for storing processed IDs, but n8n labels it experimental, says the data should be small, and warns that it may behave unreliably under high-frequency executions. That makes it unsuitable as the main duplicate barrier for valuable orders, payments, leads, or outbound messages.

A production-safe request sequence

Validate that the event contains a stable source ID.
Create the operation key once.
Check any local success record.
Send the request with the provider's idempotency mechanism.
Retry only network failures, 429s, and provider-approved temporary errors.
Reuse the same key and payload for every attempt.
Store the external ID after confirmed success.
Route exhausted or uncertain outcomes to an Error Workflow or manual review queue.

If you do not yet know why the node failed, use the failed n8n workflow debugging guide. Once the retry budget is exhausted, use an n8n Error Workflow to send the failure to Telegram, Slack, Google Chat, email, or another incident channel.

Sources

Originally published on FloxoLab.

Product Photo to WooCommerce Listing with n8n and AI Vision

Stepan Nikonov — Tue, 11 Aug 2026 05:50:47 +0000

A private admin chat workflow for ecommerce: send a product photo, analyze the image, create a WooCommerce listing, attach the product image, and get the product URL back for review.

This is a small product-intake workflow, not a promise that AI can run an ecommerce store by itself. The useful part is narrower: a product photo becomes a reviewable WooCommerce product page without a person opening WordPress, uploading the image, writing the first description, and sending the review link by hand.

Photo in, review link out.

The workflow handles the repetitive first pass. A person can still check the title, price, category, and claims before using it in a real store.

Input	AI step	Output
Product photo in a private admin chat	Vision analysis creates listing fields	WooCommerce product URL for review

The generated product page

The result is a real WooCommerce product page in a temporary test store. For this demo, the workflow creates the product with the uploaded image, a short title, a short description, and a demo price. The screenshot still shows the store's default category because category mapping was not part of this prototype.

The generated WooCommerce page includes the uploaded image, product title, demo price, and short description.

The loop closes in the same chat

The operator sends a product photo into a private admin chat. After n8n creates the WooCommerce listing, the bot replies with the product URL so the person can inspect it immediately. Telegram is just the internal intake tool here, not a public FloxoLab contact channel.

The same private admin chat closes the loop: the product photo comes in, and the WooCommerce product URL comes back.

The workflow map

The n8n canvas stays simple: receive the photo, fetch the Telegram file, upload it to WordPress Media, analyze the image, create the WooCommerce product, and send a message back to the chat.

The workflow connects a private admin chat to WordPress Media, AI vision, WooCommerce product creation, and a return message.

The AI and tool setup

The image analysis step describes the visible product details. The AI agent then uses that analysis to prepare the product fields and call the WooCommerce tool. In this demo, the price is fixed and the text fields are intentionally short.

The AI agent turns image analysis into concise WooCommerce product fields.

Why this works for WooCommerce teams

This approach works well for WooCommerce because the store already has the pieces the workflow needs: media uploads, product creation, product status, categories, and product URLs. n8n can talk to WooCommerce through native nodes, so a small team can test a product-intake workflow without building a custom admin panel first.

For this demo I used a temporary WordPress and WooCommerce test store. That made it easy to validate the workflow quickly before thinking about a real client store.

What the AI is allowed to do

Use image facts — Listing text should come from the image analysis, not invented product claims.
Keep fields short — The title and description stay compact so a person can review them quickly.
Use demo defaults — This prototype uses a fixed demo price. A real store should pull price, stock, SKU, and category from a trusted source.
Avoid fake details — No sizes, materials, shipping promises, inventory, or brand claims should be added unless they are provided by the store.

Good fits

WooCommerce stores — Small shops that still create product pages manually from product photos or supplier images.
Ecommerce agencies — Teams onboarding new product lines that need a faster first draft before final merchandising review.
Catalog operations — Teams that repeatedly turn image inputs into product records, review queues, or marketplace drafts.
Admin chat intake — The intake chat could be Telegram, Messenger, or Viber depending on the team's daily tools. The important part is private operator intake, not a public support chatbot.

What I would improve next

Draft first — Create products as draft by default, then publish after approval.
Approval button — Add a review step so the operator can approve, edit, or reject a generated product before it goes live.
Better product data — Map SKU, category, price, stock, size, color, and supplier source from trusted tables instead of asking AI to infer them.
Duplicate checks — Search existing products before creating a new page, especially when supplier photos repeat across batches.
Clear failures — If a message has no photo, the file cannot be downloaded, or the image analysis is weak, send a clear error back to the chat.

Related guide

This build leans on an AI agent node rather than a fixed script. For the difference between chatbots, automation workflows, and agents, read AI Agents for Business: What They Actually Do (and 3 Ways to Use Them).

Bottom line

This workflow is useful because it turns a product photo into a reviewable WooCommerce listing without building a custom admin interface. A person should still check the title, category, price, and product claims before using it in a real store, but the blank-page work is already done: image upload, first draft, product page, and review link are prepared automatically.

Even with human review, the workflow can save the repetitive first pass: uploading the image, opening WooCommerce, creating the product, writing a short description, and sending the page link to the person responsible for approval.

Originally published on FloxoLab.

Form + AI Chat Intake with n8n

Stepan Nikonov — Sun, 09 Aug 2026 15:05:30 +0000

A practical workflow that turns a vague website message into something a human can actually act on: a lead record, a private alert, and a workflow map email.

This could have been a dramatic automation story about one workflow saving a business 150,000 hours. It is not. This is a practical demo of a smaller, more common problem: turning a vague website message into something a human can actually act on.

The workflow uses the same intake surfaces FloxoLab already has on the site: a contact form for structured requests and an AI chat for people who start with a messy question.

The form collects fields. The chat collects context.

The useful workflow is not the chatbot by itself. It is the handoff: what gets saved, who gets notified, and what the human can do next.

Inputs	AI step	Output
Contact form and AI chat	Groq returns JSON	Lead brief and map email

Real tools, test data, and why that matters

This workflow shows the working intake pattern: validation, AI response, lead record, private alert, and follow-up email.

Production versions usually need more testing, cleaner edge-case handling, more careful copy, and fields that match the team's real sales or support process. A demo proves the path. Production makes it boring enough to trust.

The workflow has two entry points: a structured contact form path and a conversational AI chat path.

1. Start with the boring form path

The contact form is the clean path. It already has the fields a human needs: name, email, current tools, budget range, and a short message. The workflow checks whether the email already exists, creates a Notion lead if it is new, sends a private alert, and returns a simple OK response.

The form gives the workflow structured fields before any AI is involved.

The form path is intentionally simple: check duplicate, create lead, notify, respond.

2. Let the chat handle messy first messages

The AI chat is for the person who does not know what to put in a form yet. It validates the message, keeps a short safe history, sends a compact instruction set to Groq, and expects a JSON response with reply text, email-offer state, and optional plan data.

The AI prompt is not magic. It is a set of instructions that can be rewritten when the first version does not work.

The chat asks for useful context, then offers to send a workflow map by email.

The AI path is narrow: validate, build messages, call Groq, extract a safe reply.

3. Decide when the workflow should act

The decision point is deliberately plain. If the model says the email is ready and the user has provided enough context, the workflow checks for duplicates, creates a lead, builds an email, and sends it. If not, it simply returns the chat reply.

The workflow should not create records or send emails just because a model replied. It needs an explicit state.

4. Give the human something useful

The useful handoff is not "a lead arrived." It is a lead record with enough context, a private alert that tells the builder what happened, and a first-pass workflow map the user can reply to.

The Notion table can stay tiny or grow into a fuller CRM with source, status, budget, urgency, owner, and next action fields.

The alert can go to Slack, email, or a private internal channel. The important part is that the right person sees it.

The email is a first-pass map, not a final quote: outcome, tools, steps, build range, monthly cost notes, and what to correct.

Demo vs production

A demo can prove the path in half a day. A production workflow needs better copy, fallback paths, duplicate handling, error alerts, cleaner logs, and privacy-safe fields.

Sometimes the hardest production bug is remembering to remove "This message was sent automatically with n8n." That sounds small, but it is exactly the kind of polish that separates a working demo from a workflow a business can comfortably use.

Fields are flexible. The Notion database can have five fields or twenty-five: source, budget, tool stack, urgency, owner, status, next action, or whatever the handoff needs.

Alerts are flexible. The notification can go to Slack, email, Telegram privately, a CRM task, or the channel the team actually checks.

The AI behavior is flexible. It can ask one question, collect missing fields, draft the first reply, or stop and ask a human to review. The prompt is editable.

What can be customized

I can spend 30 days trying to design the perfect intake workflow on paper, or build several working versions in half a day and learn from real messages. For small automations, the second path is usually more useful.

The CRM can be Notion, Airtable, HubSpot, Google Sheets, or something else. The email can be plain text or formatted. The AI model can be Groq, OpenAI, Claude, or no AI at all if the form fields are enough.

Bottom line

This is the kind of workflow automation Philippines teams can inspect: visible inputs, validation before action, AI that returns structured data, a human-readable lead record, and a clear next step.

It is not a giant AI sales machine. It is a small intake workflow that makes the first human response easier.

Originally published on FloxoLab.

AI Email Router with n8n

Stepan Nikonov — Fri, 07 Aug 2026 16:20:35 +0000

A practical n8n workflow that reads Gmail, normalizes the fields, adds basic rule signals, classifies the email with Groq, validates the result, routes a lead into a CRM sheet, alerts Slack, and logs the operation.

This workflow shows the practical parts of AI email routing in the Philippines: the input, the rules, the classifier, the validation step, the route, and the log.

The lead path is walked through in detail because it is enough to show the pattern. Support, invoice, spam, and review branches use the same structure: validate first, route second, act third, log last.

The slow part is not connecting nodes.

It is deciding what should happen, testing real emails, tuning labels, and making the workflow understandable enough for someone else to own.

Trigger	AI step	Detailed path
Unread Gmail message	Groq classifier returns JSON	Lead -> CRM -> Slack -> log

The full workflow map

The canvas has five possible routes. This page follows the lead route end to end, while keeping the other branches visible on the map.

The workflow uses rules, AI classification, validation, routing, action nodes, and one master Google Sheets log.

1. Start with a controlled test email

Before building the rest of the workflow, send yourself one test email for the scenario you want to prove. In this case, the test email is a lead inquiry asking about n8n automation services and pricing.

The Gmail Trigger is then tested and pinned. That gives every next node a stable input while the workflow is being built, instead of waiting for Gmail to trigger again.

The first checkpoint is simple: confirm that Gmail exposes the message id, subject, sender, snippet, labels, and payload.

2. Normalize fields once

Gmail gives useful data, but it is not shaped for the rest of the workflow. The Extract Fields node turns raw Gmail output into clean top-level fields: sender, sender name, domain, subject, body snippet, attachment names, MIME types, and a PDF flag.

Normalize once, then every later node can use clean expressions like sender, domain, subject, and has_pdf.

3. Add basic rule signals before AI

Basic Rules does not replace the classifier in this demo. It adds cheap, inspectable signals before the AI step: suspicious domains, no-reply senders, newsletter patterns, aggressive spam words, trusted domains, and attachment flags.

That makes the workflow easier to review. If the model later calls something spam, validation can check whether the rule-based signals agree before archiving anything.

The rules are intentionally editable. A real team would tune known spam domains, newsletter senders, and trusted domains over time.

4. Keep the AI classifier small

The classifier is called through an HTTP Request to Groq. Its job is narrow: return one label, one confidence score, and one short reason.

{
  "label": "lead",
  "confidence": 0.95,
  "reason": "The email is an inquiry about services and pricing from a potential client."
}

The AI does not create tickets, archive email, or decide the final business action. It only returns structured classification data.

5. Validate before routing

AI output is not safe to trust directly. It can return markdown, broken JSON, a low confidence value, or a label that is semantically correct but operationally unsafe.

Invalid JSON goes to review. The workflow should not fall over because the model wrapped JSON in markdown.
Confidence below 0.75 goes to review. If the model is unsure, a human should see it.
Spam needs rule confirmation. A spam label alone is not enough to archive a message.
Invoice without PDF goes to review. The action depends on attachments, not only on the AI label.

6. Route by validated route, not raw AI label

The router uses the validated route field. This keeps the Switch node simple: the safety decisions already happened in Validation.

Routes include support, lead, invoice, spam archive, and human review.

7. Lead path: CRM, Slack, and master log

The tested path for this demo is lead routing. The workflow writes a CRM row, posts a Slack alert, and then appends the operation to the master log.

Google Sheets receives the sender name, email, domain, subject, source, status, and created timestamp.

Slack gets a readable lead alert with the classification reason and confidence.

The CRM sheet is intentionally plain. For many small workflows, a clean table is enough for the first version.

The master log records completed routes: message id, timestamp, sender, label, confidence, reason, route, and action.

This demo log records completed workflow routes. A production version should add a separate Error Trigger workflow for failed executions.

Other branches use the same pattern

Fully testing every branch means preparing separate tables, statuses, dates, files, review rows, and sometimes real attachments. That setup work matters more than simply drawing lines between nodes.

support — Create a Notion or Airtable ticket, draft a reply, notify Slack, then log the completed route.
invoice — Save the PDF to Drive, add a row to an invoice sheet, notify admin, then log the route. If there is no PDF, send it to review.
spam — Archive only when rule signals and confidence agree. Otherwise send it to review.
review — Add uncertain messages to a human review queue instead of forcing the workflow to guess.

What this demo does not cover

This is a useful first version, not the final production checklist. A production build would add an Error Trigger workflow, retry rules, stricter credential ownership, more test messages, and clearer handoff notes for the person who owns the inbox.

It would also test each branch with real examples: a support question, a real invoice PDF, a marketing email, a spam candidate, and at least one messy message that should go to review.

The setup pieces can be scaffolded too, such as a Drive folder, Sheets logs, a CRM sheet, and a Notion ticket database. I covered that pattern in the Claude connectors guide.

Bottom line

This is the kind of workflow automation Philippines teams can inspect: one trigger, clean fields, editable rules, a small AI classifier, validation before action, visible branches, Slack alerts, and a master log.

It is not AI magic. It is a small operations workflow that becomes useful when the labels, thresholds, and ownership are tuned against real email.

Originally published on FloxoLab.