DEV Community

Cover image for I sent an AI agent to buy from real stores. Here's what actually breaks.
Omkar Palika
Omkar Palika

Posted on

I sent an AI agent to buy from real stores. Here's what actually breaks.

#ai #opensource #ecommerce #webdev

AI agents are starting to shop for people. ChatGPT has an operator that clicks around the web. Perplexity ships a "buy" button. Anthropic and others are wiring agents into real checkout flows. If you run an online store, some fraction of your future customers won't be humans clicking — they'll be agents acting on a human's behalf.

Which raised a question I couldn't answer for my own projects: can an AI agent actually complete a purchase on this store?

Every "agent-readiness" tool I found just lints your markup — does your robots.txt allow agent crawlers, do you have structured data, that kind of thing. Useful, but it answers a different question. Clean markup doesn't mean an agent can navigate your funnel any more than a valid HTML resume means you can do the job. The only way to know is to send an agent through and watch where it dies.

So I built AgentiQA. It's open source (MIT). Here's how it works and — more interestingly — what I found when I pointed it at real stores.

Two layers

Static checks (stdlib only, no API key): does robots.txt allow agent user-agents (Claude-User, ChatGPT-User, OAI-SearchBot, PerplexityBot), is there JSON-LD Product/Offer data, OpenGraph tags, a reachable sitemap. This is the cheap "is the door unlocked" pass.

The live shopper agent: Claude driving a real headless Chromium browser through the actual funnel — find a product, add to cart, reach checkout — recording milestones as it goes. This is the part that answers the real question.

How the agent "sees" the page

No screenshots, no vision model. The agent reads a text snapshot of the DOM: the page URL and title, every interactive element indexed by a number, and the visible text. Roughly:

URL: https://store.example/
TITLE: All Products

INTERACTIVE ELEMENTS:
[0] <a href=/product/blue-top> Blue Top
[1] <button> Add to cart
[2] <input type=text name=search> Search
...

VISIBLE TEXT (truncated):
Blue Top  Rs. 500  In Stock ...
Enter fullscreen mode Exit fullscreen mode

The agent gets a tiny toolset — read_page, click(element_id), type_text(element_id, text), goto(url), record_milestone(stage) — and loops: read the page, decide, act, read again. Text-only is cheaper than vision, deterministic, and it's roughly what an agent shopper actually operates on anyway.

The safety part that matters: the payment guard is code, not a prompt

Any tool that drives a browser through checkout has to never touch payment data. You cannot rely on a system prompt for that — prompts are suggestions, and a model under pressure to "complete the task" can rationalize its way past one.

So the refusal lives in the executor, not the instructions. Before any keystroke, the field is checked:

PAYMENT_FIELD_RE = re.compile(r"card|cvv|cvc|cc-|expir|security.?code|pan\b", re.IGNORECASE)

def is_payment_field(el) -> bool:
    for attr in ("name", "id", "autocomplete", "placeholder", "aria-label"):
        v = el.get_attribute(attr)
        if v and PAYMENT_FIELD_RE.search(v):
            return True
    return False
Enter fullscreen mode Exit fullscreen mode

If the agent tries to type into anything that looks like a payment field, the tool returns a hard REFUSED and tells it to stop and summarize. It also never creates accounts and never submits a final order — it stops at the checkout page. The model's judgment is a second layer, not the only one.

Zero-cost driver: the Claude CLI, not just an API key

The obvious way to drive this is the Anthropic API, which costs ~$1–2 per full audit. But for a solo dev validating an idea, I didn't want a per-run bill. So there's a second driver that speaks to the claude CLI (the Claude Code subscription) over a small JSON protocol — one action per turn, resumed across steps:

python -m agentiqa https://your-store.com --driver cli   # subscription, no API key
python -m agentiqa https://your-store.com --driver api   # API key, faster
# default: api if ANTHROPIC_API_KEY is set, else cli
Enter fullscreen mode Exit fullscreen mode

Same browser loop, same safety guards — just a different way to get the model's next move. It means you can run the whole thing on a subscription you already pay for.

What I actually found

Here's the honest part, and it wasn't what I expected going in.

I ran the benchmark against a set of real, public demo stores — the kind built to be tested against, with genuine cart and checkout flows. My assumption was that agents would faceplant everywhere. They didn't.

On clean, functional stores, the agent checked out fine. Product → cart → checkout, no drama. One "failure" in my batch turned out to be a dead server throwing a Cloudflare SSL error — not an agent problem at all.

The failures that are real are dumber and quieter than "the agent is dumb":

  • An "add to cart" button that works for a human click but produces no state an agent can detect — no cart-count change, no confirmation, no visible signal. A human sees the cart bump and moves on. The agent has no way to know it worked, so it retries or stalls.
  • No cart link anywhere in the nav. Humans know to look top-right. An agent needs a discoverable path.
  • Product data an agent can't parse — price and availability rendered in a way that's obvious visually but not in the DOM text.

None of these show up in a markup linter. And here's the kicker I found: a store scored 1/4 on static readiness and the agent still checked out perfectly. Static score did not predict agent success. The behavioral test and the markup test measure different things — you need the behavioral one.

Try it on your store 

pip install -r requirements.txt
playwright install chromium

# free static check, no API key:
python -m agentiqa https://your-store.com --no-agent

# full run with the live agent:
python -m agentiqa https://your-store.com
Enter fullscreen mode Exit fullscreen mode

You get an HTML report: a verdict banner (did the agent complete checkout?), a prioritized fix list, the funnel, and the full step-by-step transcript of what the agent did. There's also a batch mode that runs a list of stores and emits an aggregate leaderboard.

Only run it against stores you own or have permission to test.

Where this goes

Agentic commerce is early. Right now the interesting finding is that the gap between "works for humans" and "works for agents" is made of small, invisible things — and nobody's funnel-testing for them yet. The tool is open source; I'd genuinely like to know what breaks on your store, because my sample is small and demo stores are easier than messy production ones.

Repo, demo GIF, and the full report format: https://github.com/OmkarPalika/agentiqa

If you run an e-commerce store and want to know whether an AI agent can buy from you — or you just want to see an agent narrate its way through a checkout and give up at a broken button — point it at your site and tell me what you find.

Top comments (0)