Ninad Pathak

Posted on Aug 9 • Originally published at ninadpathak.com

Technical SEO Checklist for Documentation Sites

#documentationseo #technicalwriting #devrel

Originally published at https://ninadpathak.com/articles/seo-for-technical-documentation/.

I built the audit for this guide around two pages: a working Cloudflare documentation guide and a deliberately broken local fixture. The contrast was useful because both pages had visible content, yet only one gave a crawler and a developer a dependable path through the task.

A page succeeds when search engines can discover and interpret it, the right developer recognizes the task from the result, and the instructions carry that person to a working state. That is what this audit checks through search intent, discovery, crawling, indexing, canonicalization, page quality, performance, and measurement.

Technical SEO audit checklist: what to check

Area	Question	Blocking evidence
Search intent	Does this page own one developer task?	Another page already answers the same query better
Discovery	Can a crawler reach the URL through links?	No crawlable internal link from an indexable page
Crawling	Can the server return the page and its required resources?	Blocked URL, redirect loop, repeated `5xx`, inaccessible HTML
Indexing	Is the page eligible for search?	`noindex`, soft 404, duplicate without a clear canonical
Canonicalization	Do all URL signals identify the same page?	Canonical, sitemap, redirect, and internal links disagree
Page quality	Can the reader identify and complete the task?	Generic title, stale version, missing prerequisites, broken example
Measurement	Can the team see what happened after release?	No baseline, change annotation, or page owner

Step-by-step SEO audit for documentation sites

Work through these eight checks in order. Fix the first blocking failure before moving to the next check.

1. Search intent and page ownership

Make page ownership the first thing in your audit because developer searches usually contain a task, command, error, product, parameter, or version. The page that owns the query should name that task and make the successful state easy to recognize.

Search	Page that should own it	Successful outcome
`cloudflare workers deploy`	Deployment guide	A Worker reaches a live URL
`stripe 401 invalid api key`	Authentication troubleshooting page	The request succeeds after the cause is fixed
`kubectl logs flags`	Command reference	The reader chooses and runs the correct flag
`oauth refresh token expiry`	Concept plus implementation guide	The application renews access safely
`migrate sdk v2 to v3`	Versioned migration guide	The application runs on the supported version

If two pages complete the same task, compare the documentation, marketing site, support center, changelog, and older versions before writing anything new. One page should become the canonical owner, with useful material from the weaker page moved into it before the duplicate is redirected or demoted.

The documentation organization guide covers that consolidation work in detail.

2. Internal links and XML sitemaps

Google's SEO Starter Guide identifies links as a primary way crawlers discover pages. Trace every important documentation URL back to a relevant hub, guide, or sibling page instead of treating its XML sitemap entry as the entire discovery strategy.

The internal route has to make sense for a person too. A useful link carries the reader toward a prerequisite, a deeper explanation, or the next task rather than existing only because an audit wanted another inbound link.

Crawlable internal links

Google's crawlable-link guidance recommends an <a> element with an href that resolves to a web address. Look for that ordinary HTML before trusting a navigation path, since JavaScript click handlers and empty anchors are less dependable for crawling and keyboard navigation.

<a href="/docs/webhooks/verify-signatures/">
  Verify webhook signatures
</a>

The anchor should still identify the destination when someone reads it without the surrounding sentence. “Verify webhook signatures” carries the task on its own, while “Read more” asks both the reader and the crawler to infer it.

XML sitemap checks

Treat the sitemap as a record of the URLs the site actually wants search engines to consider. Google's sitemap guidance recommends fully qualified canonical URLs, which makes the following signals straightforward to compare:

The sitemap returns 200 and valid XML.
Every URL uses the preferred host and protocol.
Redirects, 404 pages, parameter variants, and noncanonical duplicates are absent.
<lastmod> changes only after a meaningful page update.
The sitemap is declared in robots.txt or submitted through Search Console.

Google ignores <priority> and <changefreq>, so those fields do not need release time.

3. Crawling, rendering, and indexing

The browser can make a broken page look healthy after JavaScript has run. Read the server response before reviewing the layout because it shows whether the content, links, and canonical metadata exist without asking a renderer to repair the page.

HTTP response and rendered HTML

For the Cloudflare audit, I started with the final URL and headers:

curl --silent --show-error --location \
  --dump-header headers.txt \
  --output page.html \
  https://docs.example.com/api/authentication/

The response reveals the status code, content type, redirect chain, X-Robots-Tag, and cache behavior before presentation enters the picture. It also exposes branded error pages that return 200 and risk being treated as soft 404s.

Open the page in a clean browser context and compare the rendered result with the source. The stage is complete when the task content, title, headings, navigation, code, links, and canonical appear without a logged-in state or extra interaction.

Google's JavaScript SEO guidance explains how rendering affects indexing. Search Console URL Inspection shows Google's indexed view and live-test result.

`robots.txt` and `noindex`

robots.txt controls crawling, while a page-level or header-level noindex controls indexing. The distinction matters because Google still has to crawl a page before it can see the indexing directive.

Resolve duplicates through redirects or canonical signals rather than hiding them in robots.txt. A noindex directive fits pages that should remain accessible but should not appear in search.

4. Canonical URLs and documentation versions

Redirects and rel="canonical" are strong canonicalization signals. Sitemap inclusion is weaker, so compare the signals together because disagreement usually reveals that templates, navigation, and deployment rules are describing different preferred URLs.

Include:

Final response URL
Source rel="canonical"
XML sitemap URL
Internal-link destinations
hreflang URLs when present
Structured-data URL
Open Graph URL

The preferred page needs an absolute self-referencing canonical, with every supporting signal pointing to it. Google's canonicalization guidance also recommends linking internally to that preferred URL.

<link
  rel="canonical"
  href="https://docs.example.com/api/authentication/"
>

Versioned documentation is where policy becomes more important than a universal rule. Choose among three patterns based on whether the old instructions can still produce a valid result:

Current version only: Redirect retired task pages when the old instructions are no longer useful.
Multiple supported versions: Give each version a distinct URL, visible label, navigation path, and self-canonical.
Historical versions: Keep them accessible, then decide whether they should remain indexable based on support and search demand.

If the instructions materially differ, keep the older page distinct rather than canonicalizing it to the current version. The pages are not interchangeable when following the wrong one can break an integration, and the URL policy should preserve that distinction.

5. Page titles, headings, and task content

Labels such as “Overview,” “Configuration,” and “Usage” lose meaning when they leave their section. A useful label should still identify the task in a search result, browser tab, documentation tree, or copied URL.

The Google title-link guidance recommends a distinct, concise, accurate <title>. Google may also use the H1, prominent text, og:title, and anchor text when generating a result title.

Strong task pages repeat the same task across the URL, title, H1, description, and opening without repeating the same sentence:

URL:         /docs/webhooks/verify-signatures/
<title>:     Verify webhook signatures | Orbit Docs
H1:          Verify webhook signatures
Description: Validate Orbit webhook signatures and reject replayed requests.
Opening:     Use the signing secret and timestamp header to verify each payload.

Reading the headings without the sidebar shows whether the procedure still makes sense. “Create an API key” and “Recover from an expired key” survive that test, while “Setup” and “Errors” depend on navigation context that search visitors may never see.

The task is complete only when the page carries a reader through the required access and versions, a working command or request, expected output, recognizable failure symptoms, recovery, and cleanup. The technical tutorial guide shows how to test that path from a clean environment.

Return version-sensitive pages to the review queue when an SDK release, renamed field, changed permission, UI move, deprecation, support pattern, or ranking loss makes the instructions suspect. The updated date should change when the page itself changes meaningfully, not merely because the review took place.

6. Performance, mobile, and structured data

Documentation sites reuse a small number of templates across many URLs, which makes template choice part of the evidence. Sample a task guide, API reference, search-results page, and versioned page because a fast homepage says little about how those heavier layouts behave.

Core Web Vitals

Field data shows whether visitors experienced a problem, while lab tests reproduce it on the affected template. Documentation regressions often come from client-side search, syntax highlighting, large navigation trees, embedded consoles, chat widgets, and layout shifts when code or fonts load.

Those three measurements become useful when they sit beside the element or script responsible. Largest Contentful Paint, Interaction to Next Paint, and Cumulative Layout Shift should lead to a specific template change and another test of the same page, not another generic score on a release checklist.

Mobile documentation pages

Test a narrow viewport and browser zoom because code, tables, and navigation often fail there before the prose does. The page passes when code blocks and tables scroll inside their container, while drawers, copy controls, search, feedback, and chat overlays leave the task visible.

The same pass follows a few heading anchors and looks at long endpoint names near sticky navigation. These details matter even when most developers first discover the page on desktop because the template and indexing signals still serve every device.

HTTPS and structured data

A clean documentation path loads the canonical page and its required resources over HTTPS without mixed content. Route HTTP and alternate-host requests to the preferred URL before those variants spread through internal links or the sitemap.

Structured data should follow the visible page rather than inventing a richer result. When Article or TechArticle and BreadcrumbList fit, compare the URL, headline, dates, author, and breadcrumb path with the canonical page before validating the markup.

7. Run the documentation SEO audit script

I built a small standard-library Python auditor for this guide. It checks one page's response, robots.txt access, index directives, title, description, canonical, H1, language, internal links, anchor text, image alt attributes, and sitemap membership.

The documentation SEO audit script runs as a standalone file:

python3 docs-seo-audit.py \
  https://developers.cloudflare.com/workers/get-started/guide/ \
  --json cloudflare-docs-audit.json

The command makes read-only requests and returns a human-readable report plus a JSON receipt.

Cloudflare Workers audit result

I ran the auditor against Cloudflare's Workers CLI getting-started guide. It returned 12 passes, zero warnings, and zero errors.

The receipt records the URL and evidence for each source-HTML check.

The rendered page also exposes its place in the wider documentation system through the global directory, Workers sidebar, breadcrumbs, page title, outline, prerequisites, and first task.

The page keeps the current task visible inside the wider product documentation.

Audit failure test

I ran the same script against a deliberately broken local fixture. The fixture had noindex, an empty title, no canonical, two H1 elements, no crawlable internal links, and an image without an alt attribute.

The script exits unsuccessfully when it finds blocking index or title problems.

Audit script limitations

The script inspects one page's source HTML. It does not crawl the entire site, execute JavaScript, test Core Web Vitals, confirm Google's selected canonical, or prove that the page deserves to rank.

Use the script as the first gate because it turns obvious source-level failures into an inspectable receipt. The wider audit still needs a rendered-browser review, a link crawl, Search Console URL Inspection, field performance data, and a human attempt to complete the task.

8. Measure documentation search performance

A baseline gives the team a point of comparison before a title, canonical, internal-link path, template, or version policy changes. Save the page and query data with the release date, then compare equivalent windows after search engines have had time to recrawl the page.

Clicks, impressions, CTR, average position, index state, and the URL selected for the query each describe a different part of the result. The process has worked when the team can explain what moved, test the likely cause against the page and competing results, and choose the next change without guessing.