I built an AI API that turns any online store URL into a structured product catalog

CRJ — Sat, 05 Sep 2026 04:03:08 +0000

I watched a store owner retype 600 products by hand while migrating platforms. It took her team two weeks. That pain became this API.

What it does

You POST one URL — the homepage of any online store. The API crawls it the way a person would (categories, pagination, product pages) and returns the entire catalog as structured JSON: name, brand, category, price, SKU, photos, and an AI-written description for every product.

curl -X POST https://.../v1/extract \
  -H "Authorization: Bearer KEY" \
  -d '{"url": "https://any-store.com"}'
# → {"job_id": "...", "poll": "/v1/jobs/<id>"}

No CSS selectors to configure. No per-site templates to maintain. The LLM reads each page and understands it.

The hard part was not the crawling

Everyone assumes the crawler is the hard part. It isn't — fetch plus a BFS queue gets you there in an afternoon.

The hard part is that e-commerce pages lie to you:

A category page looks exactly like a product page ("Steel Drill 4-12mm — from $29").
Listing cards carry a name, a price and a photo — everything a product has.
The same product appears at /drills/steel-drill and /promo/steel-drill with different descriptions.

We shipped three "cleanup guards" that only exist because each one fixed a real production incident: a listing-card detector (if a page links to N similar slugs, it's a category — return null), a dedup pass with slug-digit tie-breaking, and a parent-category pruner. The prompt itself instructs the model to answer null when the page isn't a single product — teaching the AI to say "this isn't a product" mattered more than teaching it to extract.

Numbers from a real run

A Brazilian industrial-tools store, cold start: 160 pages crawled → 126 candidates → 50 clean products in 72 seconds, ~113k input tokens. Zero duplicates, zero category pages leaking into the results.

Honest limitations

JavaScript-rendered storefronts (SPA) aren't supported yet — that's the next tier.
One page = one product is assumed; stores where SKUs only exist inside filterable listings won't extract.
It respects robots.txt by default (there's an owner-override flag).

Try it

There's a free tier (50 calls/month, no card): AI Product Catalog Extractor on RapidAPI.

Live demo of a real extraction (50 products, with photos): demo page.