DEV Community

CRJ
CRJ

Posted on

I built an AI API that turns any online store URL into a structured product catalog

#ai #api #llm #saas

I watched a store owner retype 600 products by hand while migrating platforms. It took her team two weeks. That pain became this API.

What it does

You POST one URL — the homepage of any online store. The API crawls it the way a person would (categories, pagination, product pages) and returns the entire catalog as structured JSON: name, brand, category, price, SKU, photos, and an AI-written description for every product.

curl -X POST https://.../v1/extract \
  -H "Authorization: Bearer KEY" \
  -d '{"url": "https://any-store.com"}'
# → {"job_id": "...", "poll": "/v1/jobs/<id>"}
Enter fullscreen mode Exit fullscreen mode

No CSS selectors to configure. No per-site templates to maintain. The LLM reads each page and understands it.

The hard part was not the crawling

Everyone assumes the crawler is the hard part. It isn't — fetch plus a BFS queue gets you there in an afternoon.

The hard part is that e-commerce pages lie to you:

  • A category page looks exactly like a product page ("Steel Drill 4-12mm — from $29").
  • Listing cards carry a name, a price and a photo — everything a product has.
  • The same product appears at /drills/steel-drill and /promo/steel-drill with different descriptions.

We shipped three "cleanup guards" that only exist because each one fixed a real production incident: a listing-card detector (if a page links to N similar slugs, it's a category — return null), a dedup pass with slug-digit tie-breaking, and a parent-category pruner. The prompt itself instructs the model to answer null when the page isn't a single product — teaching the AI to say "this isn't a product" mattered more than teaching it to extract.

Numbers from a real run

A Brazilian industrial-tools store, cold start: 160 pages crawled → 126 candidates → 50 clean products in 72 seconds, ~113k input tokens. Zero duplicates, zero category pages leaking into the results.

Honest limitations

  • JavaScript-rendered storefronts (SPA) aren't supported yet — that's the next tier.
  • One page = one product is assumed; stores where SKUs only exist inside filterable listings won't extract.
  • It respects robots.txt by default (there's an owner-override flag).

Try it

There's a free tier (50 calls/month, no card): AI Product Catalog Extractor on RapidAPI.

Live demo of a real extraction (50 products, with photos): demo page.

I'd genuinely love to hear about stores that break it — edge cases are the roadmap.

Top comments (0)