DEV Community: Reza Pratama

Search bots vs training bots: a robots.txt map for Google, ChatGPT and Perplexity

Reza Pratama — Wed, 04 Nov 2026 00:37:27 +0000

Here is a robots.txt pattern worth checking for. A long block of AI bot names, copied from a forum thread, all set to Disallow: /. The goal is usually to stop content being used to train models. But the same founder often also wants ChatGPT and Perplexity to recommend their product.

Those two goals can live in the same file. A copied block like that often doesn't do it, because it blocks both kinds of bot.

Google's John Mueller touched on this in a recent interview reported by Search Engine Journal. He said AI systems "crawl slightly differently" because "they want to do it for training versus just like getting the freshest content." Then he added: "that's just is like another flavor of technical SEO."

So here is that flavor, written down. I went through the crawler docs from Google, OpenAI and Perplexity and sorted each bot by what it is for.

The two kinds of AI bot

Most AI companies now separate their bots by job:

Search or answer bots fetch pages so the product can show and link them in answers. Block these and you can drop out of those answers.
Training bots or tokens decide whether your content may be used to train models. Block these and, according to the docs below, your search visibility in that product is not the thing you are switching off.

There is also a third group: fetchers that run when a user asks a question. Their docs say robots.txt may not apply to them, so they are not a lever you control in the same way.

Google

Google's AI features guide says robots.txt rules for Googlebot are the control for how your site is crawled for Search, and AI Overviews and AI Mode are part of Search. To limit what is shown from a page, it points to nosnippet, data-nosnippet, max-snippet and noindex.

Training and Gemini grounding are a separate control. Google's list of common crawlers describes Google-Extended as a robots.txt token that manages whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. The same page says it does not affect inclusion in Google Search and is not a ranking signal.

Token	What it controls
`Googlebot`	Google Search, including AI Overviews and AI Mode
`Google-Extended`	Gemini training and grounding, not Search inclusion

OpenAI (ChatGPT)

OpenAI's crawler docs say each setting is independent:

User agent	What it is for
`OAI-SearchBot`	Surfacing sites in ChatGPT search. Opted out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links.
`GPTBot`	Crawling content that may be used to train OpenAI's foundation models
`ChatGPT-User`	Visits triggered by a user's request. OpenAI says robots.txt rules may not apply, and it is not used to decide what appears in search.

OpenAI also notes it can take about 24 hours for its search systems to pick up a robots.txt change.

Perplexity

Perplexity's bot docs list two:

User agent	What it is for
`PerplexityBot`	Surfacing and linking sites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.
`Perplexity-User`	Fetches when a user asks a question. Perplexity says it generally ignores robots.txt because a user asked for it.

A robots.txt that matches both goals

If your goal is "show up in AI answers, but don't train on my content", a file built from those docs could look like this:

# Search and answer bots: allowed
User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training controls: opted out
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Be aware of one tradeoff. Per Google's docs, blocking Google-Extended also opts you out of grounding in the Gemini app, not only training. If showing up in Gemini answers matters to you, you might choose to leave it allowed. That is the point of this whole post: make each line a decision, not a copy and paste.

robots.txt is not the whole story

Your firewall or CDN can block a bot that robots.txt allows. Google's AI features guide lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" as an SEO basic. OpenAI and Perplexity both publish IP ranges and recommend allowing requests from them. Perplexity's docs even include Cloudflare and AWS WAF steps, and warn that Cloudflare's Bot Fight Mode can still challenge its bot.

Three quick checks:

# 1. What does your live robots.txt actually say?
curl -s https://yourdomain.com/robots.txt

# 2. Are the search bots in your server or CDN logs at all?
grep -E "OAI-SearchBot|PerplexityBot|Googlebot" access.log | tail -20

# 3. Are they getting 200s, or 403s from a bot rule?
grep -E "OAI-SearchBot|PerplexityBot" access.log | awk '{print $9}' | sort | uniq -c

One caution from Google's docs: user agent strings can be spoofed. If you are writing allow rules, match the published IP ranges too, not only the name.

Then watch what happens

Changing robots.txt is easy. Knowing whether it helped is harder, because AI answers vary from one run to the next and changes take time to show up. I'd write down the date of the change, keep a fixed list of buyer questions, and check them again over the following weeks in ChatGPT, Perplexity and Google.

I work at Serpair, which tracks Google rankings and AI mentions over time so you can line them up against what you changed. If you want a starting point, our free SEO and AI visibility check shows where your domain stands today, no sign-up needed.

What does your robots.txt say about AI bots right now? I'm curious how many people have an AI block in there they don't remember adding.

Google says there is no special trick for AI Overviews. Here is what that means for your site.

Reza Pratama — Sat, 31 Oct 2026 03:25:58 +0000

A lot of founders I talk to want a special recipe for getting into Google AI Overviews. A new file, a new tag, a new kind of schema. So I went back and read what Google itself says. The answer is less exciting, but it is more useful.

Google's page on AI features and your website says it directly. There are no additional requirements to appear in AI Overviews or AI Mode, and no special optimizations are needed. You don't need new machine readable files, AI text files or special schema.org markup.

That doesn't mean there is nothing to do. It means the work is normal Search work, plus a couple of settings that can quietly take you out. Here is what Google's docs say, turned into checks you can do this weekend.

1. The one hard rule: indexed, with a snippet

Google gives one technical requirement for being shown as a supporting link in AI Overviews or AI Mode. The page must be indexed and eligible to be shown in Google Search with a snippet.

Both halves matter.

Indexed. Open Google Search Console, paste the URL into the URL Inspection tool, and check that the page is on Google. If your most important page (pricing, the main feature page, the "what is X" page) isn't indexed, nothing else in this post will help yet.

Eligible for a snippet. This is the part people miss. Google lists nosnippet, data-nosnippet, max-snippet and noindex as the controls that limit what is shown from your pages in Search, and that includes AI features. Sometimes they were added on purpose and forgotten, and sometimes a theme or plugin added them.

A quick way to check one page from your terminal:

# Look for snippet controls in the HTML meta tags
curl -s https://yourdomain.com/pricing | grep -iE 'nosnippet|max-snippet|noindex'

# Look for the same controls sent as an HTTP header
curl -sI https://yourdomain.com/pricing | grep -i 'x-robots-tag'

If nothing comes back, good. If you see nosnippet or a very small max-snippet value, ask why it is there. A data-nosnippet attribute on a block of text hides only that block, so check it isn't wrapped around the paragraph that actually explains your product.

Remember that Googlebot may see different HTML than your browser. The URL Inspection tool shows the HTML Google received when it crawled the page, which is the version that counts.

2. Google-Extended is not what many people think

This is the setting I would check first, because it is easy to get wrong in both directions.

Google's list of common crawlers describes Google-Extended as a robots.txt token. It doesn't crawl on its own. It controls whether content Google crawls from your site may be used for two things:

training future Gemini models
grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Grounding means giving the model content from the Google Search index at the moment someone asks a question.

The same page also says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

So here is the trade-off in plain words:

Blocking Google-Extended does not remove you from Google Search, and Google's AI features doc says the control for AI Overviews and AI Mode is Googlebot plus the snippet controls above.
Blocking Google-Extended does opt your content out of grounding in the Gemini app.

Some sites added a block like this when AI training became a hot topic:

User-agent: Google-Extended
Disallow: /

That is a fair choice if you don't want your content used for training. But if you want to show up when someone asks Gemini about your category, you have turned that off too. Make the choice on purpose.

Open https://yourdomain.com/robots.txt now and look for Google-Extended. Also check that you aren't blocking Googlebot from pages you care about. Google says robots.txt rules for Googlebot are the control for how your site is crawled for Search, and its list of SEO basics includes allowing crawling "by any CDN or hosting infrastructure". Some bot protection settings block crawlers even when robots.txt allows them, so a clean robots.txt is not the whole story.

3. Write for the follow-up questions, not only the main one

Google's doc explains that AI Overviews and AI Mode may use a "query fan-out" technique. That means issuing several related searches across subtopics to build one answer. It also says AI Mode is meant for questions that need more exploration or comparison.

So the answer is built from the smaller questions around the main one. For a typical SaaS product, those are things like:

Who is it for, and who is it not for?
How does it compare with the usual alternative?
What does it cost, and is there a free trial?
How long does setup take?
What does it not do?

If those answers only live in sales calls or support chats, there is nothing on the open web for Google to link to. You don't need a new page for each. A clear section on an existing page, with the answer in plain text near the top, is often enough. Google's own list of best practices includes "making sure that important content is available in textual form". So an answer that only exists in an image, a video, or a tab that needs a click to load is harder to use.

4. Measure in the place Google points you to

People often ask how to see AI Overviews traffic on its own. Google's answer, on the same page, is that sites showing in AI Overviews and AI Mode are counted in the overall search traffic in Search Console. It sits in the Performance report under the "Web" search type. It isn't split out as its own line.

That changes how I'd measure. You can't open one report that says "AI Overviews: X clicks". What you can do:

Pick a short list of real buyer questions in your category, plus the pages you want to be linked for.
In Search Console, watch impressions and clicks for those queries and pages over weeks, not days.
Every so often, search those questions yourself in a logged out browser. Note whether an AI Overview shows up, which sites it links to, and whether you are one of them.
Ask the same questions in the Gemini app and note who it names.

Google also says AI Overviews often don't show up at all, because they only appear when Google decides they add something beyond the normal results. So "no AI Overview for this query" is a normal result. It is not a sign you did something wrong.

One check tells you very little. Checks spread over a couple of months start to show a pattern.

A 20-minute checklist

[ ] URL Inspection: your top 3 pages are indexed.
[ ] No unwanted nosnippet, max-snippet or noindex in the HTML or X-Robots-Tag header.
[ ] data-nosnippet isn't hiding your main product description.
[ ] robots.txt: Googlebot can reach important pages, and any Google-Extended rule is a choice you made on purpose.
[ ] Your CDN or bot protection isn't blocking Googlebot.
[ ] The common follow-up questions about your product are answered in plain text somewhere on the site.
[ ] You have a short list of buyer questions to check again in a few weeks.

Where I'd start

None of this is a trick, and Google is clear that meeting every requirement still doesn't guarantee a page will be crawled, indexed or shown. But the snippet controls and the Google-Extended rule are two settings that can quietly take you out, and they take minutes to check.

Disclosure: I work at Serpair. We build a growth-journey tracker that keeps a history of your Google rankings, AI mentions and competitors, so you can see which changes actually worked. If you want a quick first snapshot before you start changing things, there is a free Insights check that runs from your public domain, with no signup. The checklist above works fine without it, though.

If you find a forgotten nosnippet or Google-Extended rule on your own site, I'd like to hear what it was doing there.

PHP2Shell (CVE-2026-109618, CVE-2026-109619): how to check and patch your PHP version

Reza Pratama — Thu, 22 Oct 2026 20:58:16 +0000

On 21 Oct 2026 the PHP project shipped fixes for two critical vulnerabilities in the PHP engine, known together as PHP2Shell (php.net):

CVE-2026-109618, remote code execution via out-of-memory corruption.
CVE-2026-109619, remote code execution via stale global state.

Both can allow unauthenticated remote code execution and both are rated CVSS v4.0 9.2 (CVE-2026-109618, CVE-2026-109619). The flaws are in PHP itself, so updating your framework, theme or plugins does not fix them. As Cloudflare puts it, administrators "need to update the PHP software running their applications" (Cloudflare).

What's in scope

The PHP project says "all PHP installations serving web requests" are at risk, including PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed. Wiz says every PHP version released in the last decade, back to 5.4, is affected by at least one of the bugs (Wiz). Wiz also says frontier AI models helped its researchers find the bugs and turn them into working exploits.

Fixed versions

PHP branch	Fixed version
8.5	8.5.12
8.4	8.4.27
8.3	8.3.36
8.2	8.2.35

There is no official PHP2Shell release below 8.2; patch via a vendor backport or upgrade. The PHP security team published minimal source backports for 7.4, 8.0 and 8.1 for people who build from source or have no long-term support vendor.

Check the runtime that actually serves requests

Hosting dashboard or your host. Most panels show the PHP version per site. Ask your host in writing whether the PHP2Shell fix is installed.
WordPress: Tools > Site Health > Info > Server.
Your own site's response headers. Some servers send X-Powered-By: PHP/8.x.y. Many hide it, and a missing header proves nothing either way.

Two catches from Qualys. First, the version that php -v shows on the command line "may belong to a different installation from the one serving web requests". Second, distributions and support vendors that ship their own patched packages may keep an older version number, so check your vendor's advisory rather than the number alone (Qualys).

Don't leave a phpinfo() page online to check. It shows your server details to anyone.

Mitigations are not a fix

The PHP project says its temporary mitigations reduce some attack paths but "do not make an installation safe". Cloudflare says its PHP2Shell protections are "deployed automatically on all plans with WAF deployed, including Free users", and still tells customers to patch. A WAF buys time while you update. It doesn't replace the update.

Exploitation status

Amazon Threat Intelligence saw a PHP2Shell exploit attempt dating to 17 Oct 2026, four days before disclosure, and after the patch said "PHP2Shell is now being broadly exploited in the wild" (AWS Security Blog). CISA's alert the same day said "CISA has not observed exploitation of these vulnerabilities as of the time of publication" (CISA). The AWS statement is the more recent picture. AWS also says to start investigations from 17 Oct.

On 8.1 or older

PHP 8.1 reached end of life on 31 Dec 2025, PHP 8.0 on 26 Nov 2023 and PHP 7.4 on 28 Nov 2022. CISA says organizations that cannot update immediately, or that continue to use PHP 8.1 or earlier, "should take PHP-based systems offline until they can be patched or updated". PHP 8.2 security support ends on 31 Dec 2026, and 8.3 is supported until 31 Dec 2027, so aim for 8.3 or newer if you're upgrading anyway.

The 5 steps

Find every PHP app and site you run, including staging and anything that bundles PHP (NCSC-NL keeps a list of such products).
Check the version of the runtime serving requests, not only the CLI.
Update to 8.5.12, 8.4.27, 8.3.36 or 8.2.35, or get your host or vendor to confirm a patched build in writing.
Firewall on while you patch.
Watch. Review from 17 Oct, check Search Console's Security issues report, and rotate passwords, tokens and API keys if anything looks off. Wiz showed the flaws can leak "passwords, tokens, and API keys" from server memory.

The SEO part devs tend to skip

Does your SEO tool check your PHP version? Search Console only tells you after you're hacked: its Security issues report shows findings once Google determines a site "was hacked, or that it exhibits behavior that could potentially harm a visitor or their computer". Google's hacked site guide names "an older, insecure version of Drupal, Joomla! or WordPress" as a way in, and a hacked site can get a "This site may be hacked" label, drop out of results and out of AI Overviews.

I work at Serpair. Our free check checks whether your site advertises a PHP/CMS version affected by PHP2Shell and known CVEs. It's passive and only reads the publicly advertised version, so it sends no attack traffic. "Version hidden / not exposed" is a possible result, and hidden doesn't mean patched. A passive check isn't a full security audit. You get a partial report by work email, then register on Serpair for the full report. Free, no credit card required. Sources and the check are in the original post: PHP2Shell vulnerability: check if your site is affected

How did patching go on your stack? Did your distro or host ship a backport, and did php -v match what your web server was actually running?

Google's new AI agent terms: a Google Workspace hardening checklist for AI agents

Reza Pratama — Thu, 22 Oct 2026 07:11:33 +0000

Google sent us a legal notice this month. We're not in trouble: every Google Workspace and Cloud admin got the notice. It covers Google's new AI agent terms, effective 12 Oct 2026, which update the Google Workspace Service Specific Terms and the Google Cloud Terms of Service. The headline is in the Workspace terms: the customer is "solely responsible" for any Agentic AI Action. This is the developer version: what the terms say, then what to actually change.

The three clauses that matter

Cloud terms 3.3 now restricts using "the Services or any AI Agent ... in violation of the AUP" (the Acceptable Use Policy).
Cloud terms 13.2: you indemnify Google for "Customer's or an End User's use of the Services or AI Agents in breach of the AUP or Section 3.3 (Restrictions)".
Workspace terms 13.3: "Customer is solely responsible for: (i) any Agentic AI Action", and 13.3(b) adds "Customer is solely responsible for any liability arising from disabling or bypassing any of these safeguards" (the human confirmation safeguards).

Combine 3.3 with Cloud terms 4.2, which says Google "may immediately Suspend" if "Customer is in breach of Section 3.3 (Restrictions)", and the practical risk is clear: a misbehaving agent can get the account suspended. Google's account help page says it directly: "Violations may result from instructions you give to an AI agent."

The 5 guardrails

List your agents. Inventory OAuth grants. Personal accounts: Google Account > Security > third-party connections. Workspace: Admin console > Security > Access and data control > API controls, then review and restrict third-party app access.
Read-only access. Use read-only Gmail and Drive scopes for agents that summarize or search. Grant send or delete only to a flow that has a human approval step.
A human approves sends. If a tool asks for human confirmation, keep it on. SST 13.3(b) puts bypass liability on you.
A separate account. Give agents their own properly licensed account or service account, not the founder's admin account. The agent guidelines prohibit "obfuscating or misrepresenting agent identity". Don't use it to share files around Workspace limits.
A spending cap. Budget caps and billing alerts. The agent guidelines: "you remain liable for any charges accrued by your agents."

Also worth doing

Log every external action: timestamp, tool, target, approver. An append-only file or table is enough.
Rate-limit outbound email and posting in code, not in the prompt. "Spam, phishing, or malware" is on the Generative AI Prohibited Use Policy list.
Treat fetched web content as untrusted input. Prompt injection is a real risk for any agent that browses.
Don't use agents to multiply accounts. "Creating or managing numerous accounts to evade detection" is on the agent guidelines' prohibited list.

Not legal advice. Read the terms for your own setup.

The flip side: AI is also how people find your product now. Developers and buyers ask ChatGPT or read Google AI Overviews, so AI visibility in AI search is worth tracking too. That's what I work on at Serpair.

Originally published on Serpair: Google's new AI agent terms, explained.

How are you using AI in your business right now? Did Google notify you too? Have you changed anything since, and what guardrails do you have in place? Drop your setup in the comments and I'll fold the best ideas into this checklist.

Your Friends Have All Signed Up. Now What? A Findability Checklist for Builders

Reza Pratama — Sat, 17 Oct 2026 09:50:11 +0000

Once your friends have all signed up, the next users have to find you on their own, and that mostly happens through Google and AI assistants. This checklist covers the technical side of being findable, in the order I'd do it. I'm Reza from the Serpair team, and this is the list we'd have wanted after our own launch.

Some context first. Serpair started because the team launched a startup and Google didn't rank it. Most founders hit some version of that wall, and YC's Ankit Gupta describes the fix well in How To Get Your First Users: "Finding your first users is more of a search problem than a persuasion problem." So treat findability like any other bug and work through it.

1. Rank for your own name

If someone hears about your product and searches its name, you should come up first. Plenty of new products don't, especially when the name is a common word or another company already uses it.

Check it:

Search your product name in a private window. Then search the name plus your category, like "acme invoicing".
Verify your site in Google Search Console and use URL Inspection on your home page to confirm it's indexed.
Search your name in ChatGPT, Perplexity and Gemini and see what they think you are.

Fix it:

Put the name and the category together in your home page title and first heading. "Acme: invoicing for freelancers" beats "Acme" on its own.
Use exactly the same name everywhere: site, social profiles, launch listings and GitHub.
Tell Google your preferred site name with WebSite structured data on the home page. Google says this markup is "most important" if you want to state a preference, and you can add an alternateName as a fallback (Google docs).

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "Acme",
  "alternateName": ["Acme Invoicing", "acme.com"],
  "url": "https://acme.com/"
}
</script>

2. Add Organization structured data, and keep expectations honest

Organization markup helps Google "disambiguate your organization in search results," which matters when other things share your name (Google docs). The sameAs property links your official profiles so the pieces connect.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Acme",
  "url": "https://acme.com",
  "logo": "https://acme.com/logo.png",
  "description": "Invoicing for freelancers.",
  "sameAs": [
    "https://x.com/acmehq",
    "https://github.com/acmehq",
    "https://www.linkedin.com/company/acmehq"
  ]
}
</script>

Check every sameAs URL by hand. A typo in a handle points Google at someone else.

Now the honest part. Schema is not a shortcut to AI citations. Ahrefs tracked 1,885 pages that added JSON-LD and found no meaningful lift in citations from Google AI Mode or ChatGPT. Those pages were already heavily cited, so the result says less about brand-new sites. Google also says there's "no special schema.org structured data" needed for its AI features, and that structured data should match the visible text (Google docs). Add schema so Google understands who you are, then put the facts that matter in plain visible text.

3. Let the AI crawlers in

AI assistants can only cite pages their crawlers can reach. Each company documents its own bots:

OAI-SearchBot powers ChatGPT search. OpenAI says sites that opt out "will not be shown in ChatGPT search answers" (OpenAI docs).
GPTBot is OpenAI's training crawler. It's a separate setting, so you can block training and still allow search.
PerplexityBot surfaces and links sites in Perplexity results (Perplexity docs).
Googlebot covers Google Search, including AI Overviews and AI Mode. Google says robots.txt rules for Googlebot are the control for those features (Google docs).

A minimal robots.txt that welcomes search crawlers:

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://acme.com/sitemap.xml

Then check your CDN or firewall, because robots.txt isn't the only gate. Google lists CDN and hosting settings as something to check, and Perplexity's docs include rules for allowing its bots through Cloudflare and AWS WAF. A quick smoke test from your terminal, using a shortened version of the user agent OpenAI documents:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
  https://acme.com/

A 403 or a challenge page means something is blocking the bot. Some firewalls also check the real IP ranges, which OpenAI and Perplexity publish, so treat a 200 here as a good sign rather than proof.

4. Write one page that answers the problem

Crawlable isn't the same as useful. AI search tools often run several related searches for one question. Google calls this "query fan-out." Then they pull passages from pages that answer a specific part of the question clearly.

Pick the exact problem your product solves, in your users' words. Look at how they describe it on Reddit, Hacker News or in support emails.
Write one page that answers it in the first two or three sentences, then explains the details.
Keep the important content in HTML text, not only in images, videos or client-side widgets.
Add a short FAQ with the follow-up questions people actually ask.

5. Get mentioned where answers come from

AI answers lean on sources the web already trusts. Pew found that Wikipedia, YouTube and Reddit were the three most cited sources in Google's AI summaries. You can't buy your way into those, but you can show up honestly:

List your product on relevant directories and launch sites with the same name and one-line description.
Answer real questions in communities where your users ask them, and mention your product only when it actually helps.
Write comparison or "how to" posts on platforms like this one, with a link back to your site.

6. Measure, then repeat

In GA4 or your analytics tool, look for referrers like chatgpt.com, perplexity.ai and gemini.google.com. Some AI visits arrive with no referrer and land in direct traffic, so the count is a floor.
In Search Console, watch impressions for your brand name and your problem queries.
Re-ask the same prompts every couple of weeks. SparkToro found that AI tools give the same list of brands less than 1 in 100 times. Track how often you appear across many runs, not where you rank in a single answer.

If you'd rather not do step 6 by hand, Serpair tracks your Google rankings and whether ChatGPT, Perplexity and Gemini mention you, then turns it into a short list of fixes. It's free to start.

Your friends got you to launch. This checklist helps the next people find you.

How to check whether AI assistants cite your site (a developer's workflow)

Reza Pratama — Fri, 16 Oct 2026 13:16:22 +0000

Search traffic used to mean one thing: Google results. Now a growing share of questions go to AI assistants like ChatGPT, Perplexity and Gemini. If you ship a product or docs site, it's worth knowing whether those assistants mention you.

This post walks through a simple, repeatable way to check. It needs a spreadsheet, a bit of discipline and, optionally, a small script.

1. Write prompts the way users ask

Don't test with your brand name. Test with the problems your users have.

Good sources for prompts:

Questions from support tickets and onboarding calls.
Queries in Google Search Console's Performance report.
"People also ask" boxes on Google for your main topics.
Threads on Reddit, Stack Overflow or GitHub Discussions in your niche.

Aim for 10 to 20 prompts. Mix broad ones ("best tool to monitor cron jobs") with specific ones ("how do I alert on a missed cron run in Node").

2. Set up a tracking sheet

Use one row per prompt per assistant per check. These columns are enough:

date | assistant | prompt | brand_mentioned (y/n) | our_url_cited (y/n) | cited_urls | competitors_named | notes

Record the cited URLs, not only whether you appeared. The list of cited domains is the most useful part. It shows which pages assistants trust for your topic.

3. Run the checks consistently

AI answers vary between runs, accounts and locations. You can't remove that noise, but you can keep it steady.

Use a fresh or logged-out session where possible.
Use the same wording every time.
Run each prompt twice and note if the answers differ.
Repeat the full set every two weeks, not every day.

Assistants that browse the web, like Perplexity, show sources directly. For others, ask a follow-up such as "which sources did you use?" and note what you get.

4. Optional: script part of it

If an assistant offers an API, you can loop over your prompts and save the raw answers. API answers can differ from the consumer chat apps, so treat them as a signal, not ground truth.

import csv, datetime, re

def extract_urls(text):
    return re.findall(r"https?://[^\s)\]]+", text)

def log_answer(writer, assistant, prompt, answer, brand, domain):
    urls = extract_urls(answer)
    writer.writerow([
        datetime.date.today().isoformat(),
        assistant,
        prompt,
        "y" if brand.lower() in answer.lower() else "n",
        "y" if any(domain in u for u in urls) else "n",
        " ".join(urls),
    ])

# answers = {prompt: call_your_assistant_api(prompt) for prompt in prompts}
# with open("ai-checks.csv", "a", newline="") as f:
#     w = csv.writer(f)
#     for p, a in answers.items():
#         log_answer(w, "assistant-name", p, a, "YourBrand", "yourdomain.com")

5. Read the results like a developer

Look for patterns, not single answers.

Cited, not named: your page is used but your brand isn't stated. Put your product name near the answer on the page.
Named, not cited: assistants know you from elsewhere, like directories or reviews. Make sure those third-party pages are accurate.
Competitors cited instead: open their cited pages. Note how they structure the answer, and where it sits on the page.
Nobody cited: the topic may be open. A clear, focused page could fill it.

6. Fix what you can control

Most fixes are ordinary web hygiene:

Put a direct one or two sentence answer under each question-style heading.
Make sure pages render meaningful HTML without needing client-side JavaScript.
Don't block search and AI crawlers in robots.txt by accident.
Add structured data that matches visible content, such as Organization, SoftwareApplication or FAQPage.
Keep pricing and feature facts consistent across your site and listings.

Wrapping up

A small, steady checking habit beats guessing. Twenty prompts, a sheet and a fortnightly rerun will show you more than any single screenshot.

I work on Serpair, which runs this kind of Google and AI-answer tracking for you. But the manual method above works fine on its own, and it's a good place to start.