Scraping Amazon prices in 8 countries: 6 ways it broke (none of them were the proxy)

We asked an AI agent a simple question: what does the same pair of headphones cost on Amazon around the world? Seven minutes and $0.019 of residential traffic later we had a price table, plus six failure modes that every price-monitoring pipeline will meet in the wild.

TL;DR. The Sony WH-1000XM5 showed a 52% spread across seven markets and a 30% spread inside the eurozone on the exact same ASIN: 207,04 € in Germany, 269,99 € in France. Getting traffic out of eight countries took two minutes. Getting correct numbers out of eight Amazons was the real work: blocks that return HTTP 200, ASINs that don't exist across marketplaces, an exit Amazon geolocated to Turkey, and a market that doesn't render prices in HTML at all. Validator code below.

The question

Anyone who sets prices, monitors MAP compliance, or runs a comparison service needs to know what a product costs as seen from each market, not from a datacenter in Virginia. Amazon is the canonical target: same catalog backbone, genuinely different prices per marketplace, and an anti-bot system that makes naive collection pointless.

The setup

The agent got the endpoint of our 2extract MCP server, an API key, and this brief:

"I need to compare the price of the same Amazon SKU across the US, Germany, UK, Japan, France, Italy, Spain and Poland. Set up residential proxies, keep the spend under $1.50, and give me connection strings per country."

It made five kinds of tool calls:

1. getAccountBalance → $1.89 available.
2. listGeoCountries → IDs for the eight markets.
3. listProxyPlans → one residential pay-as-you-go plan, $3.99/GB.
4. createProxyResource → one resource for all eight countries, with a $1.50/month spend cap attached at creation.
5. buildConnectionString ×8 → one username per country.

Two details are worth noting if you're designing tools for agents. First, the server's initialize response includes an instructions field telling the agent to resolve IDs before creating anything, check the balance, check for duplicates, and never hand-assemble a connection string; the agent followed it. Second, geo targeting is a parameter in the username, not a separate proxy per country:

2xt-customer-...-proxy-amazon_sku_pricing-country-de:[e-mail]:5555
2xt-customer-...-proxy-amazon_sku_pricing-country-jp:[e-mail]:5555

One resource, one spend cap, eight exits. Setup including recon: about two minutes.

The measurement

• Target: Sony WH-1000XM5, ASIN B09XS7JWHH.
• Method: fetch amazon.<tld>/dp/<ASIN> through each country exit with a desktop Chrome UA and matching Accept-Language, up to four attempts with a fresh exit each time. Take the first buy-box price (a-offscreen in document order), cross-check it against the embedded "displayPrice" JSON, and convert to USD with a same-hour FX snapshot (1 EUR = $1.116, 1 GBP = $1.335, $1 = ¥157.9).
• Control: the same fetch, direct from a datacenter box.

The results

Market	ASIN	Local price	≈ USD	vs cheapest
United States	B09XS7JWHH	$198.00 (sold by Amazon; before sales tax)	$198.00	baseline
Japan	B09Z2S9FSX*	¥34,800	$220.42	+11%
Italy	B09Y2MYL5C*	199,00 €	$222.11	+12%
Germany	B09XS7JWHH	207,04 €	$231.09	+17%
United Kingdom	B09Y2MYL5C*	£199.00	$265.63	+34%
Spain	B09XS7JWHH	260,97 €	$291.28	+47%
France	B09XS7JWHH	269,99 €	$301.35	+52%
Poland	B09XS7JWHH	no server-rendered price (F5)	n/a	n/a

* Local ASIN, resolved through that marketplace's own search (F2).

The 52% headline is real but partly tax: the US price excludes sales tax, every other row includes VAT or consumption tax (19 to 22% in the eurozone, 20% in the UK, 10% in Japan).

The eurozone number has no such excuse. Same ASIN, same currency: 207,04 € in Germany, 269,99 € in France. That's about 63 € more for a French buyer, and VAT doesn't explain it (19% vs 20%). Italy is lower still at 199,00 €, but under a different ASIN, so it stays out of the like-for-like comparison.

This is a one-day snapshot, and $198 in the US is a deal price on a previous-generation model. A real monitor tracks the same ASIN set over time.

What broke

F1. Blocks fail politely

The datacenter control fetch returned HTTP 200 with a 3.7 KB body: Amazon's robot-check page. A pipeline that trusts status codes archives a captcha as a price page, every day, forever. Validate content, not codes.

F2. ASINs are not global

B09XS7JWHH exists on .com, .de, .fr, .es and .pl, and returns an honest 404 on .co.uk, .it and .co.jp. One product, three ASINs across eight markets. We resolved the local ones by searching each marketplace through its own exit.

The lesson isn't the workaround. It's that "same product" is a per-marketplace identity problem, and a 404 can mean "wrong ASIN", not "blocked". If your pipeline retries 404s as blocks, it will burn traffic forever on a product that isn't there.

F3. The Turkey incident

The first German page came back with a perfectly German title and no euro prices anywhere. Buried in the markup:

<input id="currencyOfPreference" value="TRY">

Amazon's geo-IP had placed our exit, which both ipinfo and the proxy placed in Germany, in Turkey, and served a Turkey-localized amazon.de: lira prices, no buy box. A fresh exit fixed it on the first retry; it was one IP, not the pool. But the general point holds for every geo-targeted pipeline:

The proxy's claimed country, the IP database's verdict, and the target site's verdict are three different things. Check the one that matters: the site's.

F4. Residential is not a magic pass

amazon.com robot-checked 3 of the first 4 US residential exits and served the full page on the fourth. Some residential IPs are already burned on Amazon. Rotation per connection turns that into a short retry loop that converges in seconds.

Separately, roughly one in four fresh CONNECTs in this run failed with a 500 when a peer dropped between assignment and connect. Small sample, one run: read it as "handle it", not as a rate. Both are normal. Budget four retries.

F5. Not every market speaks HTML

amazon.pl returned a correct Polish page with no server-rendered price; the buy box is hydrated by JavaScript. The only "zł" on the page was a WEEE recycling note ("9,00 zł to średni koszt..."), which a lazy regex would have happily reported as the price. Poland needs the AJAX price endpoint or a headless browser. "No price" has to be a first-class result, not an exception.

F6. Watch the balance, not the counters

Minutes after collection, the per-resource and per-account traffic counters still read zero while the account balance had already moved. If an agent enforces its own budget, have it read the balance.

The validator

Here is the classification logic, condensed. Every fetch ends in exactly one named outcome, and only two of them (BLOCKED, WRONG_GEO) trigger a retry.

import re
import requests

EXPECTED_CURRENCY = {
    "com": "USD", "de": "EUR", "fr": "EUR", "it": "EUR",
    "es": "EUR", "co.uk": "GBP", "co.jp": "JPY", "pl": "PLN",
}
ROBOT_MARKERS = (
    "validateCaptcha",
    "Type the characters you see",
    "[e-mail]",
)


def classify(resp, tld):
    html = resp.text
    if resp.status_code == 404:
        return "NOT_FOUND", None          # F2: wrong ASIN here, not a block
    if len(html) < 20_000 or any(m in html for m in ROBOT_MARKERS):
        return "BLOCKED", None            # F1: usually arrives as HTTP 200
    cur = re.search(r'id="currencyOfPreference"[^>]*value="([A-Z]{3})"', html)
    if cur and cur.group(1) != EXPECTED_CURRENCY[tld]:
        return "WRONG_GEO", cur.group(1)  # F3: trust the site, not the label
    price = re.search(r'class="a-offscreen">([^<]+)<', html)
    if not price:
        return "NO_SSR_PRICE", None       # F5: route to a renderer
    return "OK", price.group(1).strip()

And the retry loop. A new Session per attempt means a new CONNECT, which means a fresh residential exit:

UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36")
RETRYABLE = {"BLOCKED", "WRONG_GEO"}


def fetch_price(tld, asin, conn, accept_language, attempts=4):
    # conn: the string buildConnectionString returned, used as-is
    proxy = f"http://{conn}"
    proxies = {"http": proxy, "https": proxy}
    headers = {"User-Agent": UA, "Accept-Language": accept_language}
    for attempt in range(1, attempts + 1):
        try:
            with requests.Session() as s:
                r = s.get(f"https://www.amazon.{tld}/dp/{asin}",
                          headers=headers, proxies=proxies, timeout=30)
        except requests.exceptions.ProxyError:
            continue                      # F4: dead peer on CONNECT, reroll
        status, value = classify(r, tld)
        if status not in RETRYABLE:
            return status, value, attempt
    return "GAVE_UP", None, attempts

Parser debt

We hit three extraction bugs in an hour: an "other offers" box mistaken for the buy box, an i18n string mistaken for an availability flag, and ¥34,800 parsed as $0.22 because the thousands separator was read as a decimal point. The fix for the last one is to stop guessing separators globally:

def parse_price(text, currency):
    digits = re.sub(r"[^\d,.]", "", text)
    if currency == "JPY":                   # no minor units
        return float(re.sub(r"[,.]", "", digits))
    m = re.match(r"^(.*)[,.](\d{2})$", digits)  # last separator + 2 digits = decimals
    if m:
        whole = re.sub(r"[,.]", "", m.group(1))
        return float(f"{whole}.{m.group(2)}")
    return float(re.sub(r"[,.]", "", digits))

assert parse_price("¥34,800", "JPY") == 34800
assert parse_price("207,04 €", "EUR") == 207.04
assert parse_price("$1,299.00", "USD") == 1299.00

Every number in the table survived a cross-check against displayPrice. Budget the same skepticism for yours.

What it cost

Item	Value
Balance before	$1.88935
Balance after	$1.87017
Measured spend	$0.019
Spend cap set at creation	$1.50/month (untouched)
Wall-clock, first tool call to final table	~7 minutes

The spend cap matters more than the spend. An agent with API access and a credit balance can loop; an agent whose every resource is created with a hard ceiling can't run up a bill. We set $1.50 and used 1.3% of it.

Limitations

• Snapshot, not series. One day, one product, one deal price in the US.
• Tax basis differs. US excludes sales tax; every other market includes VAT or consumption tax.
• Local ASINs come from search. GB, IT and JP use ASINs found through each marketplace's search. A production monitor should pin these in a verified ASIN map.
• Small samples on failures. The F4 numbers come from one short run.
• Poland has no plain-HTML price. See F5.

Takeaways

1. Classify every response into a named outcome. Status codes don't tell you whether you got the page.
2. Check geo against the target site's own signal (currency, locale fields), not the proxy label or an IP database.
3. Treat "no price" and "wrong ASIN" as results, not retries.
4. Put a hard spend cap on anything an agent can create.

---

Setup used: 2extract MCP server (endpoint https://mcp.2extract.com/mcp), residential pay-as-you-go plan, scopes proxies:read, proxies:write, balance:read, stats:read, geo:read.

This post was drafted with AI assistance from our lab notebook and raw run data, then edited by the 2extract team. The experiment itself was run by an AI agent through the MCP server described above.