DEV Community: Vishal Chauhan

Turning an SEO audit into tasks your AI coding agent can actually fix (Claude Code, Cursor, Codex)

Vishal Chauhan — Fri, 09 Oct 2026 08:07:48 +0000

A typical SEO audit export is a spreadsheet with a few hundred rows: "Missing canonical (18 pages)," "Duplicate title (11 pages)," "Orphan page," "Redirect chain." Pasting that into Claude Code, Cursor, or Codex with "fix these" usually goes badly. The agent edits the wrong template, "fixes" a warning that didn't matter, or changes robots.txt in a way nobody reviews.

Coding agents are good at SEO fixes because most technical SEO problems are template problems, and a template is code. The trick is the translation step between the audit and the agent. This post is the process I use.

Step 1: Sort findings into three buckets

Not every SEO finding is a coding task. Before anything goes to an agent, I sort the list.

Delegate to the agent (deterministic, lives in code, easy to verify):

Missing, duplicate, or self-contradicting rel="canonical" tags
Duplicate or empty <title> and meta descriptions produced by a template
Leftover noindex (meta tag or X-Robots-Tag header) on production routes
Sitemap problems: wrong host, non-canonical URLs, URLs that 404 or redirect
Internal links pointing at redirects or 404s
Redirect chains (A→B→C) that should be a single hop
Missing or invalid JSON-LD (Organization, SoftwareApplication, BlogPosting, BreadcrumbList)
Missing alt attributes on content images, and heading-level order in components

Do it with the agent, but decide yourself (code changes that encode a business decision):

robots.txt rules, especially for AI crawlers
Which URL is canonical when two pages overlap
Redirect maps after a URL restructure
hreflang sets

Keep away from the agent (not code problems):

Content quality, search intent, and topic gaps
Backlinks and authority
Anything that's really a keyword-strategy decision

The first bucket usually covers most of what a technical audit flags. The second bucket is where an agent can do real damage if you let it decide on its own.

Step 2: Group by cause, not by URL

Audits report symptoms per URL. Agents fix causes. "Missing canonical on 18 pages" isn't 18 tasks. It's usually one layout or one generateMetadata-style function that never sets a canonical. Before writing tasks, look at the affected URLs and ask what they share: a route pattern (/blog/[slug]), a layout, a CMS content type.

One task per cause keeps diffs small and reviewable, and it stops the agent from patching 18 pages one by one, which you'd then have to maintain.

Step 3: Write each task so it can be verified

This template has worked well for me with all three agents:

## Task: Add self-referencing canonical to blog posts

**Finding:** 18 URLs under /blog/* have no <link rel="canonical">.
**Evidence:** `curl -s https://example.com/blog/hello-world | grep -i canonical` returns nothing.
**Likely cause:** app/blog/[slug]/page.tsx builds metadata without `alternates.canonical`.
**Change:** Set the canonical to the absolute production URL of the post
(https://example.com/blog/<slug>), built from the site's configured base URL.
**Scope:** Only files under app/blog/. Do not touch robots.txt, redirects, or other routes.
**Acceptance criteria:**
- Every /blog/<slug> page renders exactly one canonical tag.
- The canonical is absolute, uses https, the production host, and no query string.
- No other route's <head> output changes.
**Verify:** `npm run build`, then run scripts/check_seo.py against urls-blog.txt.

What each part does:

Evidence stops the agent from "fixing" something that isn't broken. If it can't reproduce the evidence, it should say so.
Likely cause is a hint, not an order. Agents are good at confirming or correcting it once they open the file.
Scope is the most important line. Without it, agents tidy up nearby code, and that's how a canonical fix turns into a 40-file diff.
Acceptance criteria are what you'll review against. Make them observable in rendered HTML, not in source code.
Verify gives the agent a way to check its own work before handing it back.

Step 4: Give the repo standing instructions

Rules that apply to every SEO task belong in the repository, not in each prompt. Codex and Cursor read an AGENTS.md file in the repo. Claude Code reads CLAUDE.md, and you can pull in a shared file from there with an @AGENTS.md import line. I keep the SEO rules in one place:

## SEO rules
- Production base URL: https://example.com (never localhost, preview, or staging hosts).
- Canonicals: absolute, https, production host, no query strings, self-referencing by default.
- Never add `noindex` or change robots.txt without an explicit instruction in the task.
- Titles come from the page's own data; never hardcode the site name as the full title.
- JSON-LD must only state facts visible on the page (no prices or ratings that aren't shown).
- After any <head> change, run: python3 scripts/check_seo.py urls.txt

That last rule matters most. It turns "looks right" into a command with an exit code.

Step 5: Verify against rendered HTML, not the diff

SEO bugs live in the HTML a crawler receives. A diff can look perfect while a parent layout overrides the canonical, or a client component injects a second <title>. So the check should fetch real pages. Here's a small standard-library Python script I give agents (and use myself). It takes a file of URLs, fails on missing or duplicate canonicals, canonicals pointing elsewhere, noindex, redirects, error status codes, and titles duplicated across pages:

#!/usr/bin/env python3
"""check_seo.py: verify basic SEO invariants for a list of URLs (one per line)."""
import sys
import urllib.error
import urllib.request
from urllib.parse import urlsplit, urlunsplit
from collections import defaultdict
from html.parser import HTMLParser


class HeadParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.titles, self.canonicals, self.robots = [], [], []
        self.in_title = self.in_body = False

    def handle_starttag(self, tag, attrs):
        a = {k: (v or "") for k, v in attrs}
        if tag == "body":
            self.in_body = True
        elif tag == "title" and not self.in_body:  # ignore <title> inside inline SVGs
            self.in_title = True
            self.titles.append("")
        elif tag == "link" and "canonical" in a.get("rel", "").lower().split():
            self.canonicals.append(a.get("href", ""))
        elif tag == "meta" and a.get("name", "").lower() == "robots":
            self.robots.append(a.get("content", ""))

    def handle_endtag(self, tag):
        if tag == "title":
            self.in_title = False

    def handle_data(self, data):
        if self.in_title:
            self.titles[-1] += data


def norm(u):
    """Treat https://a.com and https://a.com/ as the same URL."""
    parts = urlsplit(u)
    return urlunsplit(parts._replace(path=parts.path or "/"))


def check(url):
    req = urllib.request.Request(url, headers={"User-Agent": "seo-check/1.0"})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            final, x_robots = r.geturl(), r.headers.get("X-Robots-Tag", "")
            html = r.read().decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        return None, [f"HTTP {e.code}"]
    p = HeadParser()
    p.feed(html)
    problems = []
    if norm(final) != norm(url):
        problems.append(f"redirects to {final}")
    if len(p.titles) != 1 or not p.titles[0].strip():
        problems.append(f"{len(p.titles)} <title> tags")
    if len(p.canonicals) != 1:
        problems.append(f"{len(p.canonicals)} canonical tags")
    elif norm(p.canonicals[0]) != norm(url):
        problems.append(f"canonical -> {p.canonicals[0]}")
    if any("noindex" in v.lower() for v in p.robots + [x_robots]):
        problems.append("noindex")
    title = p.titles[0].strip() if p.titles else None
    return title, problems


if __name__ == "__main__":
    urls = [line.strip() for line in open(sys.argv[1]) if line.strip()]
    by_title, failed = defaultdict(list), False
    for url in urls:
        title, problems = check(url)
        if title:
            by_title[title].append(url)
        print(("FAIL " if problems else "ok   ") + url + ("  " + "; ".join(problems) if problems else ""))
        failed |= bool(problems)
    for title, pages in by_title.items():
        if len(pages) > 1:
            print(f"FAIL duplicate title {title!r} on {len(pages)} URLs")
            failed = True
    sys.exit(1 if failed else 0)

Run it against a local production build (npm run build && npm start) with a URL list that covers one page per template, then again against production after deploy. List URLs in their canonical form. A trailing-slash mismatch counts as a failure on purpose, because it's a real inconsistency.

It's deliberately narrow. It doesn't judge title wording or content. It checks the invariants an agent is most likely to break.

Step 6: Review like it's a migration

Even with good tasks, review SEO pull requests more strictly than ordinary UI changes, because mistakes are quiet. A wrong canonical or a stray noindex doesn't throw an error. It slowly removes pages from search. What I look for:

The diff stays inside the stated scope.
No changes to robots.txt, redirects, middleware, or sitemap generation unless the task asked for them.
The verification output is pasted into the PR description.
Structured data passes Google's Rich Results Test for one example URL.

After deploying, re-crawl the affected URLs and use URL Inspection in Search Console on one or two of them. Search engines take days to weeks to reprocess changes, so judge the fix by the HTML now and the rankings later.

Where an audit tool fits

Most of the work above is translation: turning "18 pages missing canonical" into one scoped, verifiable task. That's the part I got tired of doing by hand, and it's why I built Siteory. It audits a site for SEO, AI-search readiness, and security, and every finding comes with evidence and a fix prompt you can paste into Claude, Codex, Cursor, or OpenCode. Paid reports also export Markdown files like prioritized-fixes.md. It doesn't edit your code. You review the change, ship it, and re-scan.

Whether you use a tool or a spreadsheet, the rules are the same. One cause per task, a hard scope, acceptance criteria you can see in the HTML, and a command that proves it. Agents are fast. Give them something they can be checked against.

A 20-minute website security audit: CSP, HSTS, SPF/DMARC, and CAA explained for developers

Vishal Chauhan — Fri, 09 Oct 2026 08:02:08 +0000

Most of a site's basic security posture is visible from outside: HTTP response headers and a few DNS records. You don't need access to the codebase to check them, and you don't need a scanner to read them. You need curl, dig, and twenty minutes.

This isn't a penetration test. It won't find an injection bug in your checkout. It tells you whether the cheap, standard protections are switched on and configured sensibly, and those are the protections most often missing.

Set two variables and keep a notes file open:

SITE=https://example.com
DOMAIN=example.com
curl -sI "$SITE" > headers.txt

Minutes 0–3: HTTPS and HSTS

First check that plain HTTP redirects to HTTPS:

curl -sI http://$DOMAIN | grep -iE "^(HTTP|location)"

You want a 301 or 308 to the https:// URL. Then look for HSTS:

grep -i strict-transport-security headers.txt

Strict-Transport-Security tells browsers to use HTTPS for your host for max-age seconds, even if someone types http:// or clicks an old link. Browsers ignore it when it arrives over plain HTTP, so it only counts on HTTPS responses.

A reasonable target is:

Strict-Transport-Security: max-age=31536000; includeSubDomains

Be careful with two parts of it:

includeSubDomains applies to every subdomain. If a legacy status. or mail. host only serves HTTP, browsers will refuse to reach it. Check your subdomains before adding it.
preload asks to be included in the HSTS preload list built into browsers. hstspreload.org requires a valid certificate, an HTTP→HTTPS redirect on the same host, HTTPS on all subdomains, and a header with max-age of at least 31536000 plus both includeSubDomains and preload. Getting removed from the list is slow, so treat preload as a one-way door.

Minutes 3–9: Content-Security-Policy

grep -iE "content-security-policy" headers.txt

CSP is the most valuable header and the hardest to get right. It tells the browser which sources of script, style, frames, and so on are allowed, and that limits the damage when an XSS bug lets an attacker inject markup.

What to look for when a policy exists:

'unsafe-inline' in script-src without a nonce or hash. That undoes most of CSP's protection against XSS.
Wildcards like script-src https: or *, which allow script from anywhere.
No object-src 'none' and no base-uri. <base> tag injection can redirect relative script URLs.
No frame-ancestors. This is the modern way to stop other sites framing yours (clickjacking).

If there's no policy at all, don't write a strict one and ship it in one go. Something will break. Roll it out in report-only mode:

Content-Security-Policy-Report-Only: default-src 'self'; script-src 'self' 'nonce-{RANDOM}' 'strict-dynamic'; object-src 'none'; base-uri 'none'; frame-ancestors 'self'; report-to csp

Collect the violation reports for a week, fix the legitimate sources, then switch the header name to Content-Security-Policy. Nonce-based policies need a fresh random nonce on every response, so pages served entirely from a static cache need hashes instead.

Two gotchas: frame-ancestors and reporting directives don't work in a <meta http-equiv> CSP, only in a real header. And a policy that's only report-only gives no protection at all.

Minutes 9–12: the small headers

These take a line each and are easy to check:

grep -iE "x-content-type-options|referrer-policy|permissions-policy|x-frame-options|x-xss-protection" headers.txt

X-Content-Type-Options: nosniff stops browsers guessing a response's MIME type. Turn it on everywhere.
Referrer-Policy: strict-origin-when-cross-origin sends only your origin, not full URLs with paths and query strings, to other sites. Modern browsers already default to this, but setting it explicitly protects you from older clients and accidental overrides.
Permissions-Policy turns off browser features you don't use, for example camera=(), microphone=(), geolocation=().
X-Frame-Options: DENY or SAMEORIGIN is the legacy clickjacking control. Keep it for older browsers. When CSP frame-ancestors is present, modern browsers use that instead.
X-XSS-Protection is deprecated. The old browser XSS filters it controlled have been removed, and in some cases they caused leaks of their own. Omit it or set it to 0.

Minutes 12–14: cookies

Look at every Set-Cookie header, especially after logging in:

curl -sI "$SITE/login" | grep -i set-cookie

Session cookies should have:

Secure, so they're only sent over HTTPS.
HttpOnly, so page JavaScript can't read them. This is your second line of defence if XSS happens.
SameSite=Lax or Strict, which helps against cross-site request forgery.

The __Host- prefix is worth knowing: a cookie named __Host-session is only accepted if it's Secure, has Path=/, and has no Domain attribute. That stops a compromised subdomain from setting it.

Minutes 14–18: email authentication (SPF, DKIM, DMARC)

These are DNS records, not HTTP headers, but they belong in a website audit. If your domain can be spoofed, attackers can send convincing phishing "from" your brand.

dig +short TXT $DOMAIN | grep -i spf
dig +short TXT _dmarc.$DOMAIN

SPF lists the servers allowed to send mail for the domain:

v=spf1 include:_spf.google.com include:sendgrid.net ~all

Things that go wrong: two SPF records (you're only allowed one, and two makes SPF fail), and going over the 10 DNS-lookup limit that SPF sets. Every include, a, mx, exists, and redirect counts, including nested ones. Too many includes cause a permerror.

DKIM signs outgoing mail. Your email provider gives you a selector and a public key to publish at selector._domainkey.example.com. You can't find the selector by guessing, so check your provider's settings or the DKIM-Signature header (s=) of a message you sent.

DMARC tells receivers what to do when SPF or DKIM fail alignment, and where to send reports:

v=DMARC1; p=none; rua=mailto:dmarc-reports@example.com

Start at p=none to collect reports, then move to p=quarantine and finally p=reject once every legitimate sender passes. Since February 2024, Gmail and Yahoo require bulk senders to have DMARC in place, so this isn't optional if you send marketing email at volume.

Domains that never send mail should say so explicitly, because parked domains get spoofed too:

example.com.         TXT  "v=spf1 -all"
_dmarc.example.com.  TXT  "v=DMARC1; p=reject;"

Minutes 18–19: CAA

dig +short CAA $DOMAIN

A CAA record lists which certificate authorities may issue certificates for your domain. CAs have been required to check it before issuing since 2017. Without it, any publicly trusted CA may issue for you. With it:

example.com.  CAA 0 issue "letsencrypt.org"
example.com.  CAA 0 issuewild ";"
example.com.  CAA 0 iodef "mailto:security@example.com"

That allows Let's Encrypt, forbids wildcard certificates, and says where to report violations. CAs look up CAA records starting at the exact hostname and climb the DNS tree until they find one, so a record on the apex covers subdomains that don't have their own.

The common mistake is forgetting a CA you actually use. If your CDN or hosting provider issues certificates for you, include its CA(s), or the next automatic renewal will fail. Check the provider's docs before you publish the record.

Minute 19–20: security.txt and writing it up

curl -s "$SITE/.well-known/security.txt"

RFC 9116 defines security.txt as the place researchers look for how to report a vulnerability. Contact and Expires are required:

Contact: mailto:security@example.com
Expires: 2027-10-01T00:00:00.000Z
Policy: https://example.com/security-policy

Then write the findings down in priority order, and separate vulnerabilities from observations. A missing CAA record isn't the same as a session cookie without HttpOnly. I'd order the fixes like this:

No HTTPS redirect or HSTS on login and app pages.
Session cookies missing Secure or HttpOnly.
No DMARC, or SPF broken by duplicates or the lookup limit.
No CSP, or a CSP with 'unsafe-inline' scripts. Start report-only.
Missing small headers.
CAA and security.txt.

For each finding, record the exact header or record you saw, the line you want, and how you'll verify it (the curl or dig command above). That last part is what turns an audit into a pull request.

If you'd rather not run the HTTP side of this by hand on every site you look after, Siteory's security audit checks the header, CSP, and cookie rules automatically. It labels each result as a finding or an observation and gives you the fix and a re-check. The DNS checks (SPF, DMARC, CAA) are still a dig away, and doing the whole thing by hand once is the best way to learn what each control actually does.

How to check if your website shows up in AI search: an AEO/GEO checklist for ChatGPT, Perplexity, and Google AI Overviews

Vishal Chauhan — Fri, 09 Oct 2026 08:00:43 +0000

When someone asks me why ChatGPT or Perplexity never mentions their site, the cause is usually boring. The crawler was blocked, the page was empty until JavaScript ran, or a stray noindex was sitting in a template.

You can't force an AI engine to cite you. You can make sure nothing technical stops it from fetching, reading, and quoting your page. That part is checkable. Work through this list in order, because each step depends on the one before it.

1. Know which bots you're dealing with

"AI search" isn't one crawler. Each vendor runs separate user agents for search, training, and live user requests, and robots.txt treats each one separately.

Engine	Bot for search/answers	Bot for model training	User-triggered fetches
ChatGPT (OpenAI)	`OAI-SearchBot`	`GPTBot`	`ChatGPT-User`
Perplexity	`PerplexityBot`	(not used for training, per Perplexity)	`Perplexity-User`
Claude (Anthropic)	`Claude-SearchBot`	`ClaudeBot`	`Claude-User`
Google AI Overviews / AI Mode	`Googlebot` (same as Search)	`Google-Extended` (a robots.txt token, not a crawler)	—

A few details matter here:

OpenAI's docs say a site that blocks OAI-SearchBot won't be shown in ChatGPT search answers, apart from navigational links. You can block GPTBot for training and still allow OAI-SearchBot for search. The two settings are independent.
Google says AI Overviews and AI Mode are part of Search, so Googlebot is the control. Google-Extended covers Gemini training and grounding in some of Google's other systems. It doesn't decide whether you appear in AI Overviews.
User-triggered fetchers (ChatGPT-User, Perplexity-User) act for a person asking a question. Both vendors note that robots.txt may not apply to them the way it applies to crawlers.

Decide your policy first. A common one is to allow the search bots and decide separately about training bots.

2. Read your robots.txt the way a crawler does

Open https://yoursite.com/robots.txt and read it carefully. The rule that catches people is that a crawler follows only the group that matches its user agent most specifically. If no group names it, it falls back to User-agent: *. It doesn't merge the two.

So this:

User-agent: *
Disallow: /admin
Disallow: /checkout

User-agent: GPTBot
Allow: /

gives GPTBot access to /admin and /checkout, because the GPTBot group replaces the * group for that bot. If you add named groups for AI bots, repeat your Disallow lines in each of them.

The reverse mistake is just as common: a blanket Disallow: / under a named bot, copied from a "block AI" snippet years ago, which now also blocks the search bot you want. Check for:

No Disallow for OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, or Bingbot.
A Sitemap: line pointing at the production host, not staging.

3. Check the layer robots.txt doesn't control: your CDN and firewall

robots.txt is a request. Your CDN or WAF actually enforces access. A site can allow OAI-SearchBot in robots.txt and still serve it a 403 or a challenge page.

Cloudflare, for example, announced in July 2025 that new domains block AI crawlers by default unless the owner allows them, and other CDNs have similar toggles. Check your CDN's bot settings, your WAF's user-agent and country rules, and any rate limits a burst of crawler requests could trip.

A quick sanity test from your terminal:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://yoursite.com/

This only tests rules keyed on the user-agent string. Real crawlers also come from published IP ranges (OpenAI and Perplexity both publish JSON files of them), and some WAFs verify those ranges. Your server logs are the real proof. Search them for OAI-SearchBot, PerplexityBot, and Claude-SearchBot and look at the status codes they got.

4. Make sure the content exists without JavaScript

Googlebot renders JavaScript. Many AI fetchers don't. In December 2024 Vercel published an analysis of crawler traffic and found that the crawlers from OpenAI and Anthropic downloaded JavaScript files but didn't execute them. If your key copy, pricing, or docs only show up after client-side rendering, those bots may see an empty shell.

The test takes ten seconds:

curl -s https://yoursite.com/pricing | grep -i "your exact pricing sentence"

If the grep finds nothing but you can see the sentence in a browser, that content is rendered on the client. Fixes, in rough order of effort:

Use server-side rendering or static generation for marketing pages, docs, and pricing. In Next.js, Nuxt, Astro, or SvelteKit this is often a configuration change rather than a rewrite.
Put headings, the first paragraph, and any JSON-LD in the initial HTML.
Don't hide answer content behind tabs or accordions that only load their content on click.

5. Confirm the page is indexable and snippet-eligible

For Google, the requirement is stated plainly. To show up as a supporting link in AI Overviews or AI Mode, a page has to be indexed and eligible to appear in Search with a snippet. Google also says there are no extra technical requirements beyond that.

So look for the things that quietly remove eligibility:

<meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header left over from staging.
nosnippet, or a very small max-snippet, which limits what can be shown or quoted.
A rel="canonical" pointing to a different URL, so the page you care about isn't the one that gets indexed.
Redirect chains, soft 404s, or pages that return 200 with an error message.

Use the URL Inspection tool in Google Search Console to see what Googlebot actually got. Do the same in Bing Webmaster Tools. Bing's index sits behind Bing search and Microsoft Copilot, and you can import your verified sites from Search Console in a couple of clicks.

6. Write passages that can be lifted out and quoted

Once the plumbing works, the question is whether there's a sentence worth quoting. Answer engines pull passages, not whole pages. So:

Answer the question in the first sentence or two under a heading, then give detail.
Phrase headings the way people ask ("How long does X take?").
Keep paragraphs self-contained, with specific units, versions, and dates.
Show a "last updated" date and a named author, and don't keep the only copy of a fact inside an image.

None of this is AI-specific. It's what won featured snippets for years, which is why I treat AEO and GEO as SEO done carefully rather than as a separate channel.

7. Add structured data that matches the page

Google says you don't need any special schema to appear in AI features. Structured data still helps machines tell what a page is about, as long as it matches the visible text. For most product sites that means:

Organization and WebSite on the homepage.
SoftwareApplication or Product on the product page, with a price only if the price is shown on the page.
Article or BlogPosting (with author and dateModified) on posts.
BreadcrumbList on deeper pages.

Validate with Google's Rich Results Test. Markup that claims things the page doesn't show is worse than none.

8. Treat llms.txt as optional

llms.txt is a proposed convention, a Markdown file at your site root that maps your important pages for language models. It's cheap to add, but don't expect it to change rankings or citations. Google says you don't need new machine-readable or AI text files to appear in its AI features. If you add one, keep it short and accurate.

9. Measure what you can

Search Console: clicks and impressions from AI Overviews and AI Mode count toward the normal Performance report under the "Web" search type.
Server logs: count requests and status codes for OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User, and Perplexity-User. Steady 200s mean you're reachable. 403s mean step 3 needs work.
Analytics: watch referral traffic from chatgpt.com, perplexity.ai, and claude.ai.
Spot checks: once a month, ask the engines the questions your pages answer and record which URLs they cite. It's crude but honest.

The checklist

Policy decided: which search bots, training bots, and user fetchers you allow.
robots.txt allows the search bots, with no accidental Disallow: / and named groups that repeat your private paths.
The CDN and WAF don't block or challenge those bots, which the logs show getting 200s.
Key content is in the server HTML, so a curl and grep finds it.
No noindex or nosnippet and no wrong canonical. URL Inspection is clean in Google and Bing.
Answer-first headings and self-contained paragraphs, with dates and author shown.
Structured data matches the visible text and passes validation.
llms.txt is optional, short, and accurate.
Measurement set up in Search Console, logs, and referrals.

I built Siteory to run a lot of these checks in one pass. It crawls a bounded sample of your public pages, checks robots rules, sitemaps, canonicals, JSON-LD, thin copy, and llms.txt, and explains each finding along with how to fix it. It doesn't promise citations, and nothing honest can. Everything above also works by hand with curl, Search Console, and an hour of attention. Either way, fix the plumbing before you spend money on prompt-tracking tools. A monitoring dashboard won't help if the crawler is getting a 403.