DEV Community: bpmcginley

133 of 10,099 Shopify stores block an AI crawler. Six block the one ChatGPT shops with.

bpmcginley — Sun, 06 Sep 2026 00:33:59 +0000

Merchants are told two opposite things about AI crawlers: block them, because
they take your content and give nothing back; and admit them, because that is
how a store gets into an AI shopping answer. Both assume a decision is being
made. We wanted to know how many stores have made one, and which way.

Method

Every store in a corpus of 10,099 known Shopify storefronts has its
/robots.txt read as part of a scan. The file is parsed the way the major
crawlers document parsing it: most specific User-agent group wins, * is
the fallback, longest matching path rule wins, Allow beats Disallow on a
tie. Each of twelve crawler names is asked one question: may it fetch
/products/? A store counts as blocking a crawler when the answer is no.

A second reading looks at the product page for a meta name="robots" tag
carrying noai or noimageai.

Readings were taken between 29 August and 2 September 2026. Every store that
blocked at least one crawler, or carried the tag, is one row in the CSV at
the end. The rest of the corpus blocked nothing and is the denominator.

What this cannot see. A robots.txt is a request. A store can also block a
crawler at the edge, with a bot-management rule or a firewall, and that block
is invisible here because the scanner is not the crawler being blocked. Every
count below is a floor.

Result

133 of 10,099 stores block at least one crawler: 1.32%.

Crawler	What it feeds	Fetches at answer time	Stores blocking
CCBot	Common Crawl	no	81
GPTBot	OpenAI training & retrieval	no	77
Bytespider	TikTok / Doubao	no	72
Amazonbot	Alexa+ / Rufus	no	60
Google-Extended	AI Overviews & AI Mode grounding	no	58
ClaudeBot	Claude retrieval & citations	no	53
Applebot-Extended	Apple Intelligence	no	48
meta-externalagent	Meta AI	no	47
ChatGPT-User	Live fetches during a ChatGPT chat	yes	11
PerplexityBot	Perplexity search & shopping	yes	10
OAI-SearchBot	ChatGPT search & shopping results	yes	6
Perplexity-User	Live fetches when a Perplexity user asks	yes	3

"Fetches at answer time" marks the four crawlers that read a page, or index
for a search result, at the moment a person is asking. The other eight crawl
ahead of time, to train or ground a model. The split follows each operator's
published description of the name. The twelve are the names merchants'
block lists actually carry, not every answer-time agent that exists;
Anthropic's and Google's live-fetch agents are not among them.

How they block

130 of 133 block with Disallow: /. The whole site, not the product pages. 3 block /products/ alone.
54 block exactly one crawler.
28 block the same eight, verbatim: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, GPTBot, Google-Extended, meta-externalagent. 4 more block those eight and others. Identical lists do not arise independently; this one reads like a copied snippet, and it contains none of the four answer-time crawlers.
1 store blocks all twelve.

Training or answering

Of the 133 stores that block anything, 120 block only training and grounding
crawlers. 13 block at least one crawler that fetches at answer time; 1 of
those blocks only answer-time crawlers, 12 block both kinds.

The clearest pair is OpenAI's. GPTBot trains; OAI-SearchBot is what
ChatGPT's search and shopping results are built from. 77 stores block the
first. 6 block the second, and every one of those 6 also blocks the first.
The reverse, keep the shopping crawler out and let the training crawler in,
happens on 0 stores. No store in the corpus has decided to stay out of AI shopping answers.
The stores that are out of them by robots.txt are there because a copied
training opt-out happened to include the name.

The noai tag

8 stores carry a noai or noimageai meta robots tag on their product page.
6 of the 8 block nothing at all in robots.txt. By their names, six of the eight are
musicians' merchandise stores, which again looks like one template rather
than eight decisions. The tag is a separate mechanism and this study makes no
claim about which crawlers honour it.

What it does and does not mean

Blocking AI crawlers is rare on Shopify. 98.7% of stores block none of
the twelve by robots.txt. Where it happens it is mostly a whole-site rule
aimed at training crawlers, and mostly one list.

The shopping crawlers are almost never blocked. 13 stores in 10,099 block
any of the four answer-time crawlers we checked. For the rest, whatever stands between
the store and an AI shopping answer, it is not robots.txt.

It is not a recommendation. A merchant who does not want their catalogue
in a training set has a reason, and a copied list that does the job is a
fine way to act on it. The finding is narrower: the list most of them copied
was written before the answer-time crawlers existed, so a store using it has
made a decision about training and, without noticing, none about shopping.

It is a floor. Edge blocks are real and this cannot count them.

Reproduce it

curl -s https://EXAMPLE.com/robots.txt

Find the User-agent: group that names the crawler; if none does, the *
group applies. Under that group, the longest rule matching /products/
decides, and Disallow: / matches everything.

All 139 rows, one per store, are in
ai-crawler-blocking.csv:
host, grade, date read, one column per crawler, whether the noai tag was
present, and whether the rules were whole-site or product pages only. Every
row names a domain whose robots.txt you can read yourself.

Originally published at
shelfglance.com/research/ai-crawler-blocking,
where the corpus of 10,099 scanned storefronts lives.

Shopify's agent-commerce category filter doesn't filter. We checked 190 stores.

bpmcginley — Wed, 02 Sep 2026 15:15:18 +0000

Since 2026 every Shopify storefront answers an agent-commerce endpoint at
POST /api/ucp/mcp, advertised at GET /.well-known/ucp. Merchants did not
turn it on and it is not in their admin. It speaks the Universal Commerce
Protocol over JSON-RPC, and the tool that matters is search_catalog: an AI
shopping agent asks a store for its catalogue and gets structured product data
back - integer prices in minor units with a currency, variants, SKUs, canonical
URLs, and a Shopify taxonomy category per product.

Fetch the tool list from any store and search_catalog declares
catalog.filters.categories, an array of strings documented as "category
filters combined with OR logic", next to catalog.filters.price.{min,max}.

So an agent should be able to ask for running shoes and get running shoes. We
were about to write a paragraph about what it costs a merchant to leave the
category field blank. Then we tried it.

Method

200 stores, drawn deterministically from a corpus of 10,099 known Shopify
storefronts: sort the hostnames, take every Nth. Reproducible, so nobody has
to take "we picked 200 stores" on trust. Run on 2 September 2026. Every store
got the same five calls, 10 products requested each time:

#	Call	Filter sent	A working filter would
1	Control	none	return products
2	Impossible category	`gid://shopify/TaxonomyCategory/zz-99-99-99`	return nothing
3	The store's own category	a category the control's products carry	return at least that product
4	Same, unwrapped	the bare id without `gid://...`	the other form an agent would try
5	Price control	`price.max = 1`	return nothing - nothing costs a cent

Calls 2 and 3 only mean something together. Call 2 alone cannot distinguish
"the filter is ignored" from "the filter rejects everything". Those are
opposite findings, and both happen.

The query matters more than it looks. Generic words ("gift", "set", "new")
surface a catalogue's odd corners rather than its catalogue, and produce
numbers that are measured honestly and still wrong. Every query here is three
words from one of the store's own product titles, read from its public
/products.json.

Result

190 stores answered.

What the store did with `filters.categories`	Stores
Ignored it - the impossible category returned the full unfiltered set	186
Rejected everything - zero for every value, including its own products' category	4
Filtered correctly - impossible category empty, own category not	0

Of the 200 sampled, 10 could not be read: eight served no product feed to
build a query from, two matched nothing for their own product title.

177 of the 190 returned at least one product carrying a taxonomy category, so
this is not a missing-data story. The category is there. The filter does not
use it.

The control is the whole reason this is publishable

A null result is worth nothing unless you can show the request worked. The
control is price.max, sent in the same envelope to the same store with one
field changed.

On 150 of 190 stores, price.max of one cent correctly returned nothing.
On 148 of those, the very same request that honoured price ignored the category.

One field in filters moves the result and the other does not, in the same
call, on the same store. If nothing had moved, the right conclusion would have
been that we were sending it wrong.

The bug we shipped first

The first full run reported 177 stores refusing the request outright, and it
was about to be written up as a discovery. It was our bug.

A product comes back with
categories: [{"value": "gid://shopify/TaxonomyCategory/hb-3-2-1-1", "taxonomy": "shopify"}]

objects. filters.categories is declared as an array of strings. We were passing the object straight back in, violating the endpoint's own schema, and calling its entirely correct refusal a finding about Shopify.

Taking .value fixed it, and call 4 exists because of it: if neither the
wrapped identifier nor the bare one narrows anything, "you sent it wrong"
stops being available as an explanation. This is the exact failure this kind
of post usually ships with, and the only defence is publishing the rows.

What it does and does not mean

It does not mean an uncategorised product is invisible to agents. Nothing
is being filtered out of anything, so a blank category excludes you from
nothing. That was the sentence we expected to write, and it is false.

It does mean an agent cannot narrow a catalogue search by category today,
whatever the schema says, on any of the 190 stores we could read.

It is not a complaint about Shopify. UCP shipped in 2026; a schema arriving
before every part of its behaviour is ordinary for a young protocol doing a
hard thing. The lesson is older than agentic commerce: test what an endpoint
does rather than reading what it declares.

Reproduce it in one call

Pick any Shopify domain:

curl -s https://EXAMPLE.com/api/ucp/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{
    "name":"search_catalog",
    "arguments":{"catalog":{"query":"YOUR QUERY","filters":{
      "categories":["gid://shopify/TaxonomyCategory/zz-99-99-99"]}}}}}'

Run it with and without the filters block and compare the counts. Calling
the endpoint requires serving a UCP platform profile the store can fetch;
ours declares empty payment_handlers, because a scanner takes no payments.

All 190 readings, one row per store - domain, query, products returned for
each of the five calls, and the verdict - are in
filter-survey.csv.
Every row names a domain you can re-run yourself, which is the point of
publishing it.

Originally published at
shelfglance.com/research/ucp-category-filter,
where the corpus of 10,099 scanned storefronts lives.

How to Generate Link Previews Like Slack (Without the Edge-Case Hell)

bpmcginley — Sat, 30 May 2026 14:31:38 +0000

You've seen it a thousand times: you paste a URL into Slack, Discord, or iMessage and it blooms into a tidy card with a title, an image, and a description. It's one of the highest-trust little UI elements you can add to a chat app, a comment box, a CMS, or a bookmarking tool. A bare URL looks like spam. An unfurled link looks legit — and gets clicked.

So you decide to build it. How hard can it be? You fetch the page, grab a few meta tags, done by lunch.

Then reality shows up.

What a link preview actually is

The card is built from Open Graph tags in the page's <head>:

<meta property="og:title"       content="The Verge" />
<meta property="og:description" content="Technology, science and culture." />
<meta property="og:image"       content="https://.../cover.png" />
```



Read those, render a card. Simple — in the demo. The pain is everything *between* "fetch the page" and "read the tags."

## The five things that break your weekend scraper

1. **Redirects.** `t.co`, `bit.ly`, and `http → https` mean the URL you were handed isn't the page you parse. You have to follow them.
2. **Missing tags.** Tons of sites have no `og:*` at all. You need a fallback chain: Open Graph → Twitter Card → `<title>`/`meta description` → first `<h1>`/`<img>`.
3. **Relative image URLs.** `og:image` is often `/img/cover.png`, not a full URL. Resolve it against the final page URL or the image just won't load.
4. **Timeouts and giant pages.** A slow or multi-megabyte page will hang your request or eat your memory. You need a hard timeout and a byte cap.
5. **SSRF — the dangerous one.** If users submit the URLs, an attacker can point you at `http://169.254.169.254` (cloud metadata) or `http://localhost` to reach internal services. You must block private/loopback/link-local IPs — on **every** redirect hop, not just the first.

That last one is why "just scrape it yourself" quietly becomes a security review. Link-preview features are a classic SSRF vector in real apps.

## The shortcut: one request, clean JSON

If you'd rather not own all of that, hand the URL to a service that already has, and get structured data back:



```javascript
const res = await fetch(
  "https://link-preview14.p.rapidapi.com/preview?url=" +
    encodeURIComponent(targetUrl),
  { headers: {
      "X-RapidAPI-Key": "YOUR_KEY",
      "X-RapidAPI-Host": "link-preview14.p.rapidapi.com",
  } },
);
const preview = await res.json();
```



You get back:



```json
{
  "resolvedUrl": "https://www.theverge.com/",
  "title": "The Verge",
  "description": "The Verge is about technology and how it makes us feel.",
  "image": "https://www.theverge.com/static-assets/og-image.png",
  "favicon": "https://www.theverge.com/favicon.ico",
  "siteName": "The Verge",
  "themeColor": "#5200ff"
}
```



Redirects followed, relative URLs resolved, fallbacks applied, private IPs blocked, responses cached. Every missing field comes back as `null`, so the shape never changes and your render code stays boring (the good kind).

## When to build vs. when to buy

Building it yourself is totally reasonable if you control the input — a fixed list of trusted URLs, internal pages, that kind of thing. The moment URLs are **user-submitted**, the edge cases and the SSRF surface make a purpose-built endpoint the faster, safer call.

If you want to skip the weekend, the API above is on RapidAPI with a free tier (no card): **[Link Preview API](https://rapidapi.com/bpmcginley/api/link-preview14)**. Full write-up with the field reference is [here](https://linkpreviewapi.onrender.com/blog/how-to-generate-link-previews.html).

What edge case bit you hardest building link previews? I'll add the bad ones to the article.