Scraping 20,000 physical stores (most of it didn't work)

Rubén Jiménez — Sat, 15 Aug 2026 18:39:55 +0000

Last year I needed a specific door lock, a FAC 946RP, for a project at home. I knew which part I
wanted. I could not find out which shop near me had one on the shelf, and that was the only thing I
wanted to know.

I got an Amazon listing with two-day delivery and a couple of dead forum threads. No shop. Finding
the answer took forty minutes. The lock cost fifteen euros.

Partle is my attempt at the missing answer. It is a search engine
over the inventory of physical shops, and it currently holds over 1.3 million product listings
across more than 19,000 store locations in 67 countries. Most of that data came from scraping.
Most of what I learned came from the attempts that failed.

Finding the shops is a separate problem from scraping them

Before you can scrape a shop you have to know it exists, and shops are not in any database somebody
publishes. They are in OpenStreetMap, so I query the Overpass API for hardware, DIY and garden
retail, then filter out the chains to keep the independents.

Querying a whole country does not scale, and it fails in a way that wastes a lot of time. The
Netherlands returns about 1,800 elements in under a minute. Germany raised
Overpass query failed via both area and bbox after 1,093 seconds. France failed after 943. Italy,
Spain and Poland were queued behind them, ready to do the same. While it is running you cannot tell
it is failing, so that is forty-five minutes that looks like progress.

The fix was querying subdivisions instead of countries, keyed by ISO 3166-2, the standard that gives
each region inside a country a code like DE-BY for Bavaria:

area["ISO3166-2"="DE-BY"][admin_level=4]

query	result
whole Germany	failed after 1,093 s
DE-HB (Bremen)	26 elements in 6 s
DE-BY (Bavaria, the largest state)	814 elements in 43 s

I had been using bounding boxes before that, and switching also fixed a bug I had not connected to
it. A bounding box is a rectangle in latitude and longitude, so it cannot express a border and
quietly clips neighbouring countries into your results. It also cannot express Alaska, which crosses
the antimeridian and so spans the full 360 degrees of longitude. An administrative area is the real
polygon, so neither problem happens.

Five whole-country entries became 89 subdivisions, and the work queue is now 144 regions. I pulled
the codes from OpenStreetMap itself (rel["ISO3166-2"~"^DE-"][admin_level=4]) rather than typing
them out, because a mistyped subdivision code and a bounding box aimed at the wrong country produce
equally plausible output, and the only thing that reveals either is reading the store names.

Most shops cannot be scraped at all

Here is the number I wish I had measured in the first week. I took 205 real US stores from Illinois
and Pennsylvania and asked how many had a website with a machine-readable catalogue on it. Three
did. That is 1.5%.

Anti-bot defences were not the obstacle. A neighbourhood hardware shop's website is five pages of
contact details and opening hours, so there is nothing to extract, because the shop never put a
catalogue online.

What moved the number was writing extractors per e-commerce platform instead of per site. Shops do
not build their own storefronts, they buy Shopify or Lightspeed or WooCommerce, and each of those
has a predictable product endpoint. Same 205 stores, platform extractors instead of generic
scraping: 21 stores, 10.2%. Seven times the yield.

I wish I had measured that hit rate on a small sample before building for the whole set. A 1.5%
yield and a 10.2% yield justify completely different amounts of engineering, and I spent weeks on
the assumption rather than the measurement.

Anti-bot systems, ranked by how much they cost me

The chains are the opposite problem. They have real catalogues with tens of thousands of products
each, and varying appetite for being read by a robot. The five biggest catalogues on Partle are the
Swiss sites of Conrad (electronics, 80,047 listings) and Hornbach (DIY, 33,839), the Swiss
department store Manor (5,623), Decathlon (sports) and the Spanish online supplier Rationalstock
(2,437).

No protection. Several large retailers serve clean server-rendered HTML, where a plain Scrapy spider works and keeps working. More of them do this than I expected.
JavaScript rendering plus fingerprinting. Handled with Camoufox, an anti-detect Firefox build. Two gotchas cost me an afternoon each: it advertises brotli in Accept-Encoding but mishandles the response, so I strip br with a route intercept, and launch_server() mangles a null proxy config, so I build the launch options and call Node directly.
Akamai Bot Manager. Carrefour returns 403 to everything I have tried, Camoufox with randomised fingerprints included. I stopped. Beating it means residential proxies, which means paying per request to take data from someone who has made it very clear they do not want me to.

Every spider is also a maintenance liability. You write one, it works until the store redesigns its
frontend, then you write it again. A shop that hands you a feed is worth ten that you scrape, and I
should have started asking for feeds earlier.

The crawler does not run on the server

The site runs on a Hetzner VPS with two cores and 3.7 GB of RAM, for about five euros a month, and
it serves the frontend, the API and PostgreSQL. A parallel crawl on the same box spikes load and
risks the kernel's out-of-memory killer taking production down.

So the crawler runs on a Lenovo Legion Y540 in my garage, twenty workers, writing to the Hetzner
database over an SSH tunnel. It is a residential connection with no uptime guarantee. That is fine
for a batch job that can restart, and would be bad for anything serving live requests.

Two crontab lines keep it going without me:

20 3    * * *        cron_refresh.sh    # re-import every known store registry
40 1,7,13,19 * * *   cron_discover.sh   # drain whatever discovery work is pending

The six-hourly discovery run is not really a schedule. Each run drains everything pending and exits,
and a second copy exits immediately on the runner's own lock file (flock). It is there so that a
crashed run, a rebooted box, or a region I added to the queue last week gets picked up without
anybody remembering it exists.

I considered driving this with an LLM agent workflow and decided against it. An agent session ends
when the terminal closes and it costs tokens per step, so it is not a scheduler, and pointing one at
a multi-day crawl means paying a model to watch a shell loop. cron on an always-on box survives
reboots and runs for years for free.

Server rendering, and why it was not enough

Partle started as a Vite single-page app. It worked, but nothing could index it: someone searching
"donde comprar cerrojo FAC Madrid" gets nothing from a client-rendered app, because there is no
content in the HTML for a crawler to read.

Migrating to Next.js App Router made every product (/p/{id}-{slug}), store (/s/{id}-{slug}) and
profile page server-rendered, with JSON-LD structured data in the initial HTML: Product, Offer,
LocalBusiness, GeoCoordinates.

At the time of writing, Google has indexed zero of those pages. A new domain with no inbound links
does not get crawled just because its markup is correct. Structured data did not get me traffic by
itself, and I do not think it does for anyone. Server rendering is the first thing I would fix if I
started again, and on its own it changed nothing.

The MCP server is the part I care about

Partle exposes a remote server for the Model Context Protocol (MCP), the standard assistants use to
call external tools, over its Streamable HTTP transport at https://partle.rubenayla.xyz/mcp/. It
is published on the official registry as xyz.rubenayla.partle/marketplace. Any MCP-capable
assistant can connect and search products and stores. Reads need no auth; the public API is
rate-limited to 100 requests per hour per IP, which is plenty for conversational use.

{
  "mcpServers": {
    "partle": {
      "url": "https://partle.rubenayla.xyz/mcp/"
    }
  }
}

Publishing to the registry uses domain verification: generate an Ed25519 key pair, serve the public
key at /.well-known/mcp-registry-auth, publish a server.json describing the server. It is
straightforward and thinly documented. One warning from my own error log, since it cost me the
ability to publish updates: keep the private key. I generated mine in /tmp, authenticated, and
deleted it as cleanup. The public half is still deployed and the private half is gone.

I care about this more than about the website because an assistant talking to someone renovating a
bathroom already has the context of the job. What is missing is a structured way to ask what is on a
shelf nearby, which is a protocol problem, not a web design one. Cross-language search already works
that way through a semantic=true flag, so asking for a "drill" also matches "taladro" and
"Bohrmaschine" — which matters when the catalogue spans 67 countries and every listing is in the
language of the shop that wrote it.

What is not there yet

Coverage is uneven, and I would rather say so than have you find out by searching. A handful of
large chains and online catalogues account for most of the product listings. Most of those
store locations are independent shops mapped in OpenStreetMap with no inventory listed, for the
reason above: there is no catalogue online to read.

The rest of the stack, briefly: FastAPI and SQLAlchemy behind nginx, packaged with uv (a Python
package manager), a Next.js frontend in TypeScript, and one PostgreSQL instance doing both storage
and full-text search, which has been enough at this size.

The site is partle.rubenayla.xyz, the API needs no key
(docs), and if you have an MCP-capable assistant you can
point it at https://partle.rubenayla.xyz/mcp/ and search the same way I originally wanted to.

DEV Community: Rubén Jiménez