DEV Community: Andres Rusic

I asked Google AI Mode and Perplexity 232 local questions. Here's what they actually cite.

Andres Rusic — Fri, 30 Oct 2026 20:52:07 +0000

Last month I crawled 3,014 Houston business websites to see what AI crawlers can actually read. That left a question the crawl data couldn't answer on its own: does any of it matter? Do AI search engines cite the sites that are easy to read, or something else entirely?

So on October 30 I asked Google AI Mode and Perplexity the kind of questions customers type every day, like "Who is the best plumber in Katy, TX?" and "How much does AC repair cost in Sugar Land, TX?". That came to 232 questions across 15 industries and 8 Houston-area cities, with no brand names and no system prompt. Then I traced every source each answer cited and every business it recommended, and matched the cited websites back to the October crawl.

The full report, charts and method are on our site: Houston AI Citation Study. The data is free under CC BY 4.0 on Hugging Face and Kaggle.

Two engines, two very different playbooks

Here's where the citations came from when I asked who the best business in a city was:

Source	Google AI Mode	Perplexity
Google Business Profiles	79%	0%
Business websites	5%	49%
Directories, review and ranking sites	9%	45%
Reddit and social media	0.2%	0.8%

AI Mode leans almost entirely on Google Business Profiles. 95% of the businesses it recommended came with a profile card, and in 101 of 108 answers every single pick did. Those businesses had a median of 4.9 stars and 323 reviews.

Perplexity works more like a researcher. It attaches sources to each pick, and the business's own website was one of them 38% of the time. The rest came from Yelp, Zocdoc, Best Lawyers, HAR.com and similar directories.

The two rarely agree. They named the same top business for only 9 of 108 questions, and shared even one cited website on just a quarter of them. Reddit, for all the attention it gets in AI search conversations, showed up in 9 of 464 answers.

Do AI-ready websites actually get cited more?

This was the question I cared about most. The October crawl checked four basics: AI search crawlers allowed in robots.txt, at least 120 words readable without JavaScript, business schema, and a single H1.

I looked at it two ways. First, in time order. Of the 1,580 sites from the crawl in the industries I asked about, the ones that passed all four checks on October 2 were cited by October 30 at 5.9%, against 1.4% for everyone else. After controlling for industry and link authority, that's an odds ratio of 4.6 (95% CI 2.3 to 9.2).

Second, cited versus not cited. The 603 local business homepages the engines cited passed all four checks 65% of the time. The 1,531 sites that weren't cited passed 32% of the time.

Check	Cited	Not cited	Adjusted odds ratio
All four basics	65%	32%	3.6
120+ words without JavaScript	97%	82%	5.4
Business schema	81%	56%	3.1
FAQ schema	46%	17%	3.4
Question-style subheadings	68%	35%	2.8

What surprised me is that link authority doesn't explain it. Using Common Crawl's web graph, the cited sites were, if anything, less linked-to than the rest, and the gap held at every authority level.

I want to be careful here: this is a strong association, not proof of cause. Ahrefs ran a controlled test this year and found that adding schema to pages that were already being cited didn't raise their citations. My data covers a different situation, local sites the engines may never have seen before, but businesses that fix the basics usually fix other things too.

A few other findings worth knowing

City pages matter. For "best ___ in [city]" questions, 80% of the business pages that got cited had the city in the URL or page title.

Booking pages count. On price questions, 22% of AI Mode's citations went to booking pages on Square, Tekmetric or Vetstoria that list public prices.

Ranking yourself doesn't help. When a business's own "Best plumbers in Katy" post was cited, Perplexity recommended that business 47% of the time, about the same as businesses whose other pages were cited (45%).

The answers move. I asked Perplexity every question a second time, about 10 minutes later. The sources overlapped 98.5% and the top pick held 82% of the time, but only 69% of the businesses it named stayed the same. A single check isn't a measurement.

City names trip it up. 9 of 58 answers about Pasadena, Texas cited businesses in Pasadena, California.

How I built it

I collected the answers with a short Python script: Perplexity's Sonar API at default settings, and Google AI Mode through SerpApi with the search located in Houston. All 696 answers came in within about 30 minutes.

Every cited domain was classified as a business site, directory, Google profile, booking page and so on. I used hand-coded lists for roughly 640 domains, plus the crawl data and the address and phone details in each site's schema for the rest. None were left unclassified.

Recommendations came from AI Mode's profile cards and from the business names in Perplexity's answers, matched to the numbered citations next to them. I checked 55 at random by hand. 53 names were right, and I fixed the matching bugs that turned up.

The website checks ran on the same crawler as the October study, so cited and non-cited sites were measured the same way. The statistics are logistic regressions with industry fixed effects, with bootstrapped confidence intervals.

The limits are real: one day, one metro, two engines, and API results rather than what a signed-in person sees. ChatGPT and Google's AI Overviews weren't part of this round.

What I'd do with this

If you run a local business, the takeaways are unglamorous but clear. Keep your Google Business Profile complete and keep reviews coming in, because that's what AI Mode reads. Then make sure your website gives Perplexity something to cite: real text in the HTML, LocalBusiness schema, one H1, question-style headings backed by FAQ schema, a page for each city you serve, and prices people can see.

Disclosure: I run Marketing & AI, a Houston SEO and AEO agency. SEO agencies were one of the 15 industries, and we weren't cited in a single answer either.

I crawled 3,014 Houston business websites to see what AI crawlers actually see

Andres Rusic — Sun, 04 Oct 2026 01:47:54 +0000

Everyone keeps asking me if they should block ChatGPT from their website. So I went and looked at what businesses in Houston are actually doing, and the answer surprised me: almost nobody is blocking AI. Their sites just aren't built in a way AI can read.

Here's what I did and what I found. The full dataset is public if you want to poke at it [links at the bottom].

The setup

I pulled every business in the Houston metro that lists a website in OpenStreetMap, deduped by domain, and ended up with 3,014 sites. Anything sharing a domain across 3 or more locations got treated as a chain, which left 2,474 independents.

For each site the crawler fetched three things, once: robots.txt, llms.txt, and the homepage. It identified itself with its own user agent and skipped any site whose robots.txt told it to stay out. It runs on a Cloudflare Worker with HTMLRewriter, which streams the HTML so attribute order doesn't matter and a heavy page doesn't blow up memory [I cap it at 1.5 MB].

Four checks made up what I call the "AI-ready basics":

robots.txt lets the AI search crawlers in (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot since they feed AI Overviews and Copilot)
at least 120 words of readable text in the raw HTML, before any JavaScript runs
some kind of business schema in JSON-LD (LocalBusiness, Organization, etc)
exactly one H1

The results (independents)

31% pass all four
45% have no business schema at all
27% have no H1
19% show under 120 words before JavaScript runs (restaurants: 33%)
30% already serve an llms.txt, and several are clearly plugin-generated [one literally says "Generated by Rank Math SEO"]

Chains weren't any better: 29% pass all four.

The robots.txt part, and where I was wrong

Only 1.6% of independents block an AI search crawler in robots.txt. I evaluated rules per crawler token for the homepage path using RFC 9309 longest-match, so a site that blocks /search but not / doesn't count as blocked. Training-only tokens (GPTBot, ClaudeBot, Google-Extended, CCBot) are reported separately, since blocking those opts you out of training, not out of answers.

A commenter on LinkedIn pointed out that robots.txt isn't the whole story: a site can allow a bot in the file and still block it at the firewall. Fair point, so I tested it. I took 400 random sites that allow AI crawlers in robots.txt and fetched each homepage with a normal browser user agent and then with each crawler's user agent:

User agent	Blocked anyway
GPTBot	9.8%
ClaudeBot	10.1%
OAI-SearchBot	3.0%
PerplexityBot	3.3%
Claude-SearchBot	2.7%

Most of those are hard 403s from bot protection, which lines up with the "block AI bots" toggles CDNs ship. Counting both layers, effective AI search blocking is about 5%, not 1.6%. Caveat: I sent the user agents from my own network, not the crawlers' IP ranges, so a firewall that only blocks impostors would make these numbers a bit high. Server logs are the only way to know for sure on a given site.

The JavaScript part is the one that actually matters

GPTBot, ClaudeBot and PerplexityBot mostly read raw HTML and don't execute JavaScript. So a homepage that renders its content client-side looks close to empty to them. I counted words inside p, li, h1 to h6, td and dd in the raw HTML; under 120 means there's basically nothing for an AI answer to quote.

If you build sites, the cheap fixes are boring and that's the point:

render the main copy on the server or pre-render it (Astro, Next SSG, whatever you like)
ship a LocalBusiness JSON-LD block with the exact name, address, phone and hours
check robots.txt and your CDN's bot settings, separately
one H1, and subheads written as the questions customers actually ask

Data

Everything's CC BY 4.0, aggregate only, no individual business named:

Report and method: https://marketingandai.com/research/houston-ai-visibility-index/
Hugging Face: https://huggingface.co/datasets/Marketingandai/houston-ai-visibility-index
Kaggle: https://www.kaggle.com/datasets/marketingandai/houston-ai-visibility-index-october-2026

If you want to check a single site the same way, the scanner is free: https://marketingandai.com/free-tools/ai-search-visibility/

Curious if anyone's seen different numbers in their own city. I'd bet the JavaScript one is worse in places with more agency-built SPAs.