DEV Community: Viktor

Vinted Scraper in Go: Search, Prices, and the Walls You'll Hit (2026)

Viktor — Wed, 14 Oct 2026 08:03:42 +0000

Vinted is one of the biggest second-hand fashion marketplaces in Europe, and its public catalog is useful for price tracking, brand research, and spotting what actually sells in a niche. There is no official public product API for third-party tools, so most people end up scraping the HTML or reverse-engineering the same JSON the website uses.

This guide builds a small Go scraper that:

Searches Vinted for a keyword on a chosen country domain
Pulls out item id, title, brand, size, price, and seller
Pages through results politely
Stores snapshots in SQLite so you can track prices over time

I'll also be honest about where the DIY approach breaks.

Legal note: Only scrape publicly visible data, keep your request rate low, and read Vinted's terms for the country you target. If you need this data for a commercial product at scale, assume you'll need proxies and ongoing maintenance — or an API that already handles them.

How Vinted search is structured

A search URL looks like this:

https://www.vinted.fr/catalog?search_text=nike+zoom&page=1

Country matters. vinted.fr, vinted.de, vinted.co.uk, vinted.pl, and vinted.com are separate marketplaces with different inventories and currencies. Don't assume a listing on .fr exists on .com.

The page is heavily client-rendered. The HTML shell is mostly empty; the useful data arrives as JSON from Vinted's internal catalog API. You can either:

Parse that internal JSON (fragile: paths and auth headers change)
Or render the page in a browser and scrape the DOM (slow, still fingerprintable)

For a small DIY tool, listening to the catalog JSON from a headless browser is more reliable than hand-rolling their signed requests. For a pure-Go path without a browser, people often replay the catalog endpoint with careful headers — until it breaks. We'll start with the pure-Go approach against the public catalog URL and fall back honestly when it fails.

Setup

mkdir vinted-scraper && cd vinted-scraper
go mod init vinted-scraper
# no third-party deps for the basic version

Step 1: Fetch a search page without looking like a bot on day one

At minimum, send a real browser User-Agent, an Accept-Language that matches the domain, and reuse one http.Client so cookies persist:

package main

import (
    "fmt"
    "io"
    "math/rand"
    "net/http"
    "net/url"
    "strings"
    "time"
)

var client = &http.Client{Timeout: 25 * time.Second}

func headers(domain string) http.Header {
    h := make(http.Header)
    h.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36")
    h.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
    h.Set("Accept-Language", langFor(domain))
    h.Set("Connection", "keep-alive")
    return h
}

func langFor(domain string) string {
    switch domain {
    case "fr":
        return "fr-FR,fr;q=0.9,en;q=0.8"
    case "de":
        return "de-DE,de;q=0.9,en;q=0.8"
    case "co.uk":
        return "en-GB,en;q=0.9"
    case "pl":
        return "pl-PL,pl;q=0.9,en;q=0.8"
    default:
        return "en-US,en;q=0.9"
    }
}

func fetch(rawURL string, domain string) ([]byte, error) {
    var last error
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequest(http.MethodGet, rawURL, nil)
        if err != nil {
            return nil, err
        }
        req.Header = headers(domain)
        resp, err := client.Do(req)
        if err != nil {
            last = err
        } else {
            body, readErr := io.ReadAll(resp.Body)
            resp.Body.Close()
            if readErr != nil {
                last = readErr
            } else if resp.StatusCode == 403 || resp.StatusCode == 429 || resp.StatusCode == 503 {
                last = fmt.Errorf("blocked: HTTP %d", resp.StatusCode)
            } else if resp.StatusCode >= 400 {
                last = fmt.Errorf("HTTP %d", resp.StatusCode)
            } else if looksLikeChallenge(body) {
                last = fmt.Errorf("challenge / captcha page")
            } else {
                return body, nil
            }
        }
        wait := time.Duration(5*(1<<attempt))*time.Second + time.Duration(rand.Intn(3000))*time.Millisecond
        time.Sleep(wait)
    }
    return nil, last
}

func looksLikeChallenge(body []byte) bool {
    s := strings.ToLower(string(body))
    return strings.Contains(s, "captcha") || strings.Contains(s, "cf-challenge") || strings.Contains(s, "access denied")
}

A 200 with a Cloudflare challenge page is still a failure. Check the body, not only the status code.

Step 2: Prefer the JSON the site already loads

When you open a catalog page in DevTools, you'll see requests to something like /api/v2/catalog/items. That payload is cleaner than the DOM. The hard part is that those calls often need cookies and anti-bot tokens minted by the first HTML load.

A practical DIY pattern in Go:

Fetch the HTML once to get cookies into your client jar (use net/http/cookiejar).
Replay the catalog API URL with the same jar and headers.
Parse the JSON into your own structs.

type CatalogResponse struct {
    Items []CatalogItem `json:"items"`
    Pagination struct {
        CurrentPage int  `json:"current_page"`
        TotalPages  int  `json:"total_pages"`
    } `json:"pagination"`
}

type CatalogItem struct {
    ID    int64  `json:"id"`
    Title string `json:"title"`
    URL   string `json:"url"`
    Price string `json:"price"` // sometimes a string; sometimes nested — normalize below
    Brand struct {
        Title string `json:"title"`
    } `json:"brand"`
    SizeTitle string `json:"size_title"`
    User      struct {
        ID    int64  `json:"id"`
        Login string `json:"login"`
    } `json:"user"`
    Photo struct {
        URL string `json:"url"`
    } `json:"photo"`
}

type Item struct {
    ID     int64
    Title  string
    Brand  string
    Size   string
    Price  float64
    URL    string
    Seller string
}

Field names and nesting change. Keep a saved fixture from a successful response and a unit test that fails loudly when Vinted renames size_title again.

Price parsing is locale-dependent ("12,50 €" vs "£12.50"). Don't ParseFloat the raw string — strip currency symbols and normalize commas first.

Step 3: Paginate politely across country domains

func search(domain, query string, maxPages int) ([]Item, error) {
    var out []Item
    seen := map[int64]bool{}
    for page := 1; page <= maxPages; page++ {
        u := url.URL{
            Scheme: "https",
            Host:   "www.vinted." + domain,
            Path:   "/catalog",
        }
        q := u.Query()
        q.Set("search_text", query)
        q.Set("page", fmt.Sprintf("%d", page))
        u.RawQuery = q.Encode()

        body, err := fetch(u.String(), domain)
        if err != nil {
            return out, fmt.Errorf("page %d: %w", page, err)
        }
        items, err := parseCatalog(body, domain) // your JSON or HTML parser
        if err != nil {
            return out, err
        }
        if len(items) == 0 {
            break
        }
        for _, it := range items {
            if !seen[it.ID] {
                seen[it.ID] = true
                out = append(out, it)
            }
        }
        fmt.Printf("page %d: %d items\n", page, len(items))
        time.Sleep(time.Duration(3+rand.Intn(5)) * time.Second)
    }
    return out, nil
}

Random delays of a few seconds matter more than clever headers. Burst traffic from one datacenter IP is the fastest way onto a challenge wall.

Step 4: Store snapshots for price history

A single scrape is rarely useful. SQLite is enough:

import (
    "database/sql"
    "time"
    _ "modernc.org/sqlite" // or github.com/mattn/go-sqlite3
)

func save(db *sql.DB, keyword string, items []Item) error {
    _, err := db.Exec(`CREATE TABLE IF NOT EXISTS snapshots (
        id INTEGER, keyword TEXT, title TEXT, brand TEXT, size TEXT,
        price REAL, url TEXT, seller TEXT, scraped_at TEXT
    )`)
    if err != nil {
        return err
    }
    now := time.Now().UTC().Format(time.RFC3339)
    tx, err := db.Begin()
    if err != nil {
        return err
    }
    stmt, err := tx.Prepare(`INSERT INTO snapshots VALUES (?,?,?,?,?,?,?,?,?)`)
    if err != nil {
        return err
    }
    defer stmt.Close()
    for _, it := range items {
        if _, err := stmt.Exec(it.ID, keyword, it.Title, it.Brand, it.Size, it.Price, it.URL, it.Seller, now); err != nil {
            tx.Rollback()
            return err
        }
    }
    return tx.Commit()
}

Run it on a cron and you can ask:

SELECT id, date(scraped_at) AS day, MIN(price) AS price
FROM snapshots
WHERE keyword = 'nike zoom' AND brand = 'Nike'
GROUP BY id, day
ORDER BY id, day;

Where DIY falls apart

The script above is fine for a few pages a day from a home connection. Past that you'll hit:

Cloudflare / bot challenges at volume. From AWS, GCP, or a cheap VPS you often get blocked on the first request. Residential proxies cost real money and still need care.
Country domains. Inventory, currency, and anti-bot behavior differ per host. A scraper tuned on vinted.fr will mis-parse prices on vinted.co.uk.
Internal API drift. Catalog JSON field names and required cookies change. Budget time for fixtures and alerts when a page returns zero items.
Sold / reserved items. Listings disappear or flip to reserved quickly in popular niches. If you only store the latest scrape, you lose the price history that made the project worth doing.
Headless browsers aren't free wins. Playwright helps with client-rendered pages but is slower, heavier, and still fingerprinted.

Option: use an API for the hard part

If you'd rather not maintain proxies and selectors, a scraping API can return the same catalog as JSON. For example, ScrapeAtlas* has a Vinted search endpoint that takes the query, domain, and page:

func searchAPI(apiKey, query, domain string, page int) ([]Item, error) {
    u, _ := url.Parse("https://api.scrapeatlas.com/v1/vinted/search")
    q := u.Query()
    q.Set("query", query)
    q.Set("domain", domain) // fr, de, co.uk, com, ...
    q.Set("page", fmt.Sprintf("%d", page))
    q.Set("per_page", "24")
    u.RawQuery = q.Encode()

    req, _ := http.NewRequest(http.MethodGet, u.String(), nil)
    req.Header.Set("x-api-key", apiKey)
    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    // Decode their JSON into your Item slice; check status for
    // "failed" / "challenged" / "partial" the same way you would
    // for any upstream that can be rate-limited.
    return decodeScrapeAtlas(resp.Body)
}

There are also endpoints for a single item by id/URL, a member profile, and a member's wardrobe. domain selects which marketplace the data was read from. That output can feed the same save() function, so you can swap fetchers without changing storage.

Wrapping up

For small, occasional scrapes: Go net/http + careful headers + slow delays can work from a residential IP.
For daily tracking of a few keywords: add a cookie jar, response fixtures, zero-result alerts, and per-domain locale parsing.
For anything at scale or from cloud servers: plan for residential proxies and ongoing breakage, or an API that absorbs that cost.

Whichever route you take, store snapshots instead of overwriting rows. On Vinted the listing that was €40 yesterday may be gone tomorrow — the history is the product.

Questions or a selector that broke? Leave a comment and I'll update the guide.

*I build it.

Amazon Scraper in Python: An Honest Developer Guide (2026)

Viktor — Mon, 12 Oct 2026 10:12:49 +0000

Amazon is probably the most-scraped e-commerce site there is, and it's also one of the best defended. People scrape it to track prices, monitor competitors, research product niches, or check what's actually ranking for a keyword.

This guide covers what works in 2026, what breaks, and why. We'll build a small Python scraper that:

Searches Amazon for a keyword
Pulls out ASIN, title, price, rating, and review count
Pages through the results
Stores snapshots in SQLite so you can track prices over time

I'll also be upfront about where the DIY approach stops working.

Legal note: Only scrape publicly visible data, keep your request rate low, and read Amazon's Conditions of Use. If you need product data at scale for a commercial product, look at the official Product Advertising API first (it requires an Associates account with qualifying sales).

How Amazon search pages are structured

A search URL looks like this:

https://www.amazon.com/s?k=wireless+earbuds&page=2

k is the keyword
page is the 1-based page number (Amazon usually caps search at around 7–20 pages)

Every organic result is a div with data-component-type="s-search-result" and a data-asin attribute. The ASIN (Amazon Standard Identification Number) is the stable ID you want: https://www.amazon.com/dp/<ASIN> always resolves to the product.

Inside each result card:

Field	Selector (2026, subject to change)
Title	`h2 span`
Price	`span.a-price > span.a-offscreen` (first one is the current price)
List price	`span.a-price.a-text-price > span.a-offscreen`
Rating	`span.a-icon-alt` (text like "4.4 out of 5 stars")
Review count	`span.s-underline-text` or `a[href*="customerReviews"] span`

Amazon changes class names regularly and A/B tests layouts, so write your parser defensively: every field may be missing.

Setup

pip install requests beautifulsoup4 lxml

Step 1: Fetch a search page without getting blocked immediately

The bare requests.get() with the default python-requests/2.x user agent usually gets a 503 or a CAPTCHA page on the first try. At minimum, send realistic browser headers and reuse a session so cookies persist:

import random
import time
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/17.6 Safari/605.1.15",
]

session = requests.Session()

def headers():
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    }

class Blocked(Exception):
    pass

def fetch(url, params=None, retries=3):
    for attempt in range(retries):
        resp = session.get(url, params=params, headers=headers(), timeout=20)
        blocked = (
            resp.status_code in (503, 429)
            or "validateCaptcha" in resp.text
            or "Robot Check" in resp.text
        )
        if not blocked and resp.ok:
            return resp.text
        wait = 2 ** attempt * 5 + random.uniform(0, 3)
        print(f"Blocked or error ({resp.status_code}), sleeping {wait:.0f}s")
        time.sleep(wait)
    raise Blocked(f"Gave up on {resp.url}")

Note that a CAPTCHA page often comes back with HTTP 200, so checking only the status code isn't enough. Look for validateCaptcha in the body.

Step 2: Parse the results

import re
from bs4 import BeautifulSoup

def to_float(text):
    if not text:
        return None
    m = re.search(r"[\d,]+(?:\.\d+)?", text)
    return float(m.group().replace(",", "")) if m else None

def text_of(el):
    return el.get_text(strip=True) if el else None

def parse_search(html):
    soup = BeautifulSoup(html, "lxml")
    products = []
    for card in soup.select('div[data-component-type="s-search-result"]'):
        asin = card.get("data-asin")
        if not asin:
            continue
        # Skip sponsored placements if you only want organic results
        sponsored = card.select_one('span.puis-label-popover-default, span:-soup-contains("Sponsored")')

        rating_text = text_of(card.select_one("span.a-icon-alt"))  # "4.4 out of 5 stars"
        products.append({
            "asin": asin,
            "title": text_of(card.select_one("h2 span")),
            "price": to_float(text_of(card.select_one("span.a-price > span.a-offscreen"))),
            "list_price": to_float(text_of(card.select_one("span.a-price.a-text-price > span.a-offscreen"))),
            "rating": to_float(rating_text.split(" ")[0]) if rating_text else None,
            "reviews": to_float(text_of(card.select_one("span.s-underline-text"))),
            "sponsored": sponsored is not None,
            "url": f"https://www.amazon.com/dp/{asin}",
        })
    return products

A few gotchas I hit:

Prices are locale-formatted. amazon.de gives you 12,99 €. If you scrape other marketplaces, parse per locale instead of stripping commas.
Review counts can be abbreviated ("1.2K") in some layouts. Handle that if you see it.
Sponsored results are mixed in. Decide up front whether you want them; for rank tracking you usually don't.
Duplicate ASINs show up across pages. Dedupe by ASIN.

Step 3: Paginate politely

SEARCH_URL = "https://www.amazon.com/s"

def search(keyword, max_pages=5):
    seen, results = set(), []
    for page in range(1, max_pages + 1):
        html = fetch(SEARCH_URL, params={"k": keyword, "page": page})
        items = parse_search(html)
        if not items:
            break  # end of results, or a layout we don't recognize
        for item in items:
            if item["asin"] not in seen:
                seen.add(item["asin"])
                results.append(item)
        print(f"page {page}: {len(items)} items")
        time.sleep(random.uniform(3, 7))  # don't hammer it
    return results

Random delays of a few seconds matter more than any header trick. Burst traffic from one IP is the fastest way to get a CAPTCHA wall.

Step 4: Store snapshots for price tracking

The useful thing is usually the history, not a single scrape. SQLite is all you need:

import sqlite3
from datetime import datetime, timezone

def save(products, keyword, db="amazon.db"):
    con = sqlite3.connect(db)
    con.execute("""
        CREATE TABLE IF NOT EXISTS snapshots (
            asin TEXT, keyword TEXT, title TEXT, price REAL, list_price REAL,
            rating REAL, reviews REAL, sponsored INTEGER, position INTEGER,
            scraped_at TEXT
        )""")
    now = datetime.now(timezone.utc).isoformat()
    con.executemany(
        "INSERT INTO snapshots VALUES (?,?,?,?,?,?,?,?,?,?)",
        [(p["asin"], keyword, p["title"], p["price"], p["list_price"], p["rating"],
          p["reviews"], int(p["sponsored"]), i, now)
         for i, p in enumerate(products, 1)],
    )
    con.commit()
    con.close()

if __name__ == "__main__":
    kw = "wireless earbuds"
    data = search(kw, max_pages=3)
    save(data, kw)
    print(f"saved {len(data)} products")

Run it daily with cron and you have price and rank history:

SELECT asin, date(scraped_at) AS day, MIN(price) AS price, MIN(position) AS rank
FROM snapshots
WHERE keyword = 'wireless earbuds'
GROUP BY asin, day
ORDER BY asin, day;

Where DIY falls apart

Here's the honest part. The script above works for a few pages a day from a home connection. Past that, you'll hit:

CAPTCHAs at volume. From datacenter IPs (AWS, GCP, a VPS), you'll often get blocked on the first request. You need residential proxies, which cost real money.
TLS / browser fingerprinting. Amazon looks at more than headers. Libraries like curl_cffi (which impersonates Chrome's TLS fingerprint) help a lot over plain requests.
Layout drift. Selectors break every few weeks. Budget time for maintenance and add an alert when a page returns zero items.
Location-dependent prices. Prices and availability depend on the delivery ZIP code (stored in a cookie). Your numbers may not match what your users see.
Headless browsers aren't a silver bullet. Playwright gets you past some checks but is 10–50x slower and still gets fingerprinted.

Option: use an API for the hard part

If you'd rather not maintain proxies and selectors, a scraping API can return the same data as JSON. For example, ScrapeAtlas* has an Amazon product search endpoint that takes the keyword, marketplace, and a pagination cursor:

import os
import requests

API = "https://api.scrapeatlas.com/v1/amazon/product-search"

def search_api(keyword, country="US", max_pages=3):
    results, cursor = [], None
    for _ in range(max_pages):
        params = {"query": keyword, "country": country}
        if cursor:
            params["cursor"] = cursor
        resp = requests.get(
            API,
            headers={"x-api-key": os.environ["SCRAPEATLAS_API_KEY"]},
            params=params,
            timeout=60,
        )
        data = resp.json()
        if data["status"] in ("failed", "challenged"):
            raise RuntimeError(f"request failed: {data['requestId']}")
        for item in data["items"]:
            p = item["product"]
            results.append({
                "asin": p["id"],
                "title": p["title"],
                "price": (p.get("price") or {}).get("current"),
                "list_price": (p.get("price") or {}).get("original"),
                "rating": (p.get("rating") or {}).get("average"),
                "reviews": (p.get("rating") or {}).get("count"),
                "sponsored": False,
                "url": p["url"],
            })
        if not data["pagination"]["has_more"]:
            break
        cursor = data["pagination"]["next_cursor"]
    return results

The output fits the same save() function, so you can swap one fetcher for the other. Check status: partial means you got usable data but some fields weren't available. There are also endpoints for single products by ASIN, reviews, offers/sellers, and best-seller lists, and country covers marketplaces like GB, DE, and JP.

Wrapping up

For small, occasional scrapes: requests + BeautifulSoup + slow, random delays works fine.
For daily tracking of a few keywords: add curl_cffi, retry logic, and a zero-results alert.
For anything at scale or from cloud servers: you'll need residential proxies and ongoing maintenance, or an API that handles them for you.

Whichever route you take, store snapshots rather than overwriting rows. The price history is where the value is.

Questions or a selector that broke? Drop a comment and I'll update the guide.

*I build it.