How to migrate an online store's product catalog in minutes (not weeks)

CRJ — Sat, 05 Sep 2026 22:08:02 +0000

Migrating a store to a new platform is mostly one boring, expensive problem: getting the product catalog out of the old site. The checkout, the theme, the domain — those are afternoons. The catalog is weeks.

Here is how to do it in minutes instead, and why the three obvious approaches usually don't get you there.

Why catalog migration eats weeks

1. Retyping by hand

The default. Someone opens the old store in one tab and the new admin in another, and copies 600 products across. At a realistic 3-4 minutes per product — name, description, price, SKU, category, download the photo, re-upload it — that's about 35 hours of work. It is also the version that silently loses data: whoever gets tired at product 400 starts skipping the specs table.

2. The CSV export that "already exists"

Every platform has an export button, so this feels solved. It isn't:

The export gives you their column names, not the new platform's.
Photos come as URLs pointing at the old CDN, which dies with the old plan.
Variants, bundles and specs get flattened into one unparseable text blob.
Legacy or custom-built stores (a huge share of small B2B sites) have no export at all.

You end up writing a transform script anyway — and you can only start once you have someone's admin credentials, which for an agency migrating a client is often the long pole.

3. A scraper with CSS selectors

The engineer's answer, and it works — for exactly one site. You open DevTools, find that the price lives in .product-info__price > span, write the selector, and it holds until the store's theme updates. Multiply by every client you migrate and you are maintaining a per-site template library. Selector-based scrapers don't generalize; that's their whole problem.

The AI approach: no selectors at all

Instead of teaching a script where the price is, you let a model read the page and tell you what's on it. The crawler is generic — follow links, find category pages, follow pagination, collect candidate product URLs — and the extraction is a prompt.

The non-obvious part is not extraction. It's rejection. E-commerce pages lie: a category page carries a name, a price and a photo, exactly like a product page. So the single most important instruction in the prompt is that the model must answer null when the page is a listing, not a product. Teaching the AI to say "this isn't a product" mattered more than teaching it to extract.

Step by step, with curl

I packaged the engine as an API. Subscribe on RapidAPI (free tier, no card), grab your key, and it's two calls: submit a job, poll it.

1. Submit the store URL

curl -X POST 'https://ai-product-catalog-extractor.p.rapidapi.com/v1/extract' \
  -H 'x-rapidapi-key: YOUR_KEY' \
  -H 'x-rapidapi-host: ai-product-catalog-extractor.p.rapidapi.com' \
  -H 'content-type: application/json' \
  -d '{
    "url": "https://the-old-store.com/",
    "limites": { "max_paginas": 200 }
  }'

You get a 202 back:

{ "job_id": "9f2c...", "poll": "/v1/jobs/9f2c..." }

2. Poll until it's done

curl 'https://ai-product-catalog-extractor.p.rapidapi.com/v1/jobs/9f2c...' \
  -H 'x-rapidapi-key: YOUR_KEY' \
  -H 'x-rapidapi-host: ai-product-catalog-extractor.p.rapidapi.com'

While it runs you get live progress — fase (phase), paginas_visitadas, itens_extraidos — so you can render a progress bar instead of a spinner.

3. Read the catalog

{
  "status": "done",
  "result": {
    "itens": [
      {
        "nome": "Rebite Repuxo Alumínio 4,8 x 12mm",
        "marca": "CRV",
        "categoria": "Rebites",
        "sku": "RA-4812",
        "preco": 42.9,
        "descricao": "Rebite de repuxo em alumínio com haste de aço...",
        "foto": "https://old-store.com/img/ra-4812.jpg",
        "_url": "https://old-store.com/produtos/rebite-repuxo-4812"
      }
    ],
    "total_candidatos": 126,
    "truncado": false,
    "usage": { "tokens_entrada": 113559, "chamadas_ia": 60 }
  }
}

About those Portuguese field names. The engine was born inside a Brazilian ERP, so the default preset ships nome, preco, foto, descricao. That's cosmetic, not structural — you can send your own schema and get whatever keys your target platform wants:

"schema": [
  { "key": "title", "tipo": "texto", "label": "product name" },
  { "key": "price", "tipo": "number" },
  { "key": "image", "tipo": "foto" }
]

Same engine, keys ready to POST straight into Shopify or WooCommerce.

Real numbers

A cold run against a Brazilian industrial-tools store, no configuration, no selectors:

160 pages crawled → 126 candidates → 50 clean products in 72 seconds. Zero duplicates, zero category pages leaking into the results, ~113k input tokens (a few cents of model cost).

Compare that with 35 hours of retyping.

Honest limitations

SPA storefronts don't work yet. If the products only exist after JavaScript runs (VTEX, headless Shopify, client-side Next.js), you'll get zero. Static HTML only, for now.
One page = one product is assumed. Stores where SKUs exist only inside a filterable listing, with no individual URL, return nothing.
robots.txt is respected by default. There's an owner-authorization flag for when it's your own store.
Crawling is rate-limited per host on purpose, so a large catalog takes minutes, not seconds. Pointing at the listing page instead of the homepage speeds things up a lot.

Try it on your own migration

API listing (free tier, 50 calls/month): AI Product Catalog Extractor on RapidAPI
Live demo of a real extraction: 72.60.244.148:3024
How it was built, and the three cleanup guards that came from production bugs: I built an AI API that turns any online store URL into a structured product catalog

If you point it at a store and it breaks, tell me which one — the edge cases are the roadmap.

I built an AI API that turns any online store URL into a structured product catalog

CRJ — Sat, 05 Sep 2026 04:03:08 +0000

I watched a store owner retype 600 products by hand while migrating platforms. It took her team two weeks. That pain became this API.

What it does

You POST one URL — the homepage of any online store. The API crawls it the way a person would (categories, pagination, product pages) and returns the entire catalog as structured JSON: name, brand, category, price, SKU, photos, and an AI-written description for every product.

curl -X POST https://.../v1/extract \
  -H "Authorization: Bearer KEY" \
  -d '{"url": "https://any-store.com"}'
# → {"job_id": "...", "poll": "/v1/jobs/<id>"}

No CSS selectors to configure. No per-site templates to maintain. The LLM reads each page and understands it.

The hard part was not the crawling

Everyone assumes the crawler is the hard part. It isn't — fetch plus a BFS queue gets you there in an afternoon.

The hard part is that e-commerce pages lie to you:

A category page looks exactly like a product page ("Steel Drill 4-12mm — from $29").
Listing cards carry a name, a price and a photo — everything a product has.
The same product appears at /drills/steel-drill and /promo/steel-drill with different descriptions.

We shipped three "cleanup guards" that only exist because each one fixed a real production incident: a listing-card detector (if a page links to N similar slugs, it's a category — return null), a dedup pass with slug-digit tie-breaking, and a parent-category pruner. The prompt itself instructs the model to answer null when the page isn't a single product — teaching the AI to say "this isn't a product" mattered more than teaching it to extract.

Numbers from a real run

A Brazilian industrial-tools store, cold start: 160 pages crawled → 126 candidates → 50 clean products in 72 seconds, ~113k input tokens. Zero duplicates, zero category pages leaking into the results.

Honest limitations

JavaScript-rendered storefronts (SPA) aren't supported yet — that's the next tier.
One page = one product is assumed; stores where SKUs only exist inside filterable listings won't extract.
It respects robots.txt by default (there's an owner-override flag).

Try it

There's a free tier (50 calls/month, no card): AI Product Catalog Extractor on RapidAPI.

Live demo of a real extraction (50 products, with photos): demo page.

I'd genuinely love to hear about stores that break it — edge cases are the roadmap.

DEV Community: CRJ

How to migrate an online store's product catalog in minutes (not weeks)

Why catalog migration eats weeks

1. Retyping by hand

2. The CSV export that "already exists"

3. A scraper with CSS selectors

The AI approach: no selectors at all

Step by step, with curl

Real numbers

Honest limitations

Try it on your own migration

I built an AI API that turns any online store URL into a structured product catalog

What it does

The hard part was not the crawling

Numbers from a real run

Honest limitations

Try it