The 200-result ceiling that forced a tiled crawler

One Google Maps query stops at roughly 200 results. Every other decision in this pipeline falls out of that single number.

I live in Dubai. Google Maps knows every plumber, nursery and blood-test lab in the city. No tool lets me browse that, filter it, or ask a structural question of it.

So I built one. Business Directory Toolkit is MIT-licensed and turns a city into a browsable business directory. directory.pooyagolchian.com is a live Dubai deployment standing on 1,400 requests to SearchApi's Google Maps engine. I am writing it up as part of their developer ambassador programme. That programme backs developers who ship something real on their APIs and leaves the work in your name.

1,400 SearchApi requests returned 15,246 unique businesses. That is 10.9 new businesses per request.

Most articles about scraping Google Maps stop at one API call and a console.log. That part already works. Everything expensive happens after it.

The ceiling that shapes everything

A single Google Maps query cannot enumerate a city. Page 11 does not come back as an empty list. It comes back with no local_results key at all, so the results are absent rather than empty. I committed that probe to the repository as a test fixture so nobody has to spend a credit rediscovering it.

If you were expecting 60, that is the ceiling on the Places API Text Search, which returns 20 results across three pages and stops. The Maps engine goes considerably deeper before it gives out. It still gives out, and that ceiling decides how you build.

Tiling forces a budget model

Tiling is mandatory, not an optimisation. Dubai is 44 geographic squares crossed with 40 categories, which makes 1,760 tile-and-category pairs. Pagination depth is the only dial that moves the bill.

SearchApi fits that shape for structural reasons rather than promotional ones. It takes Google's own location format straight through, as ll=@lat,lng,zoom. Tiling a city means expressing a coordinate and a zoom level, so no geocoding service sits in the middle.

One SearchApi request costs one credit. Because the unit of billing matches the unit of work, the whole cost model collapses into a 3x3 lookup table. Budget guards only work because the arithmetic stays that simple.

The response also carries fields that replace other services. Roughly 99.8% of results arrive with country_code and city, which drives the in-city filter with no geocoder. Every single result carries gps_coordinates.

The pipeline reassigns each business to the tile it actually sits in rather than the one whose query found it. About half of them sat somewhere other than where I looked, because Google answers from a radius.

place_id arrives on 100% of results and stays stable. It is the dedup key, the database partition key, and the only value the takedown suppression list may hold.

Failure is predictable enough that the retry policy fits in one predicate.

/**
 * Rate limits and server errors are worth retrying; a 400 or a 401 will fail
 * identically forever and retrying only burns time and credits.
 */
export function isRetryable(status: number): boolean {
  return status === 429 || status >= 500;
}

A 400 throws immediately, which is correct, because a malformed query does not improve on the fourth attempt.

Price it before you buy

Five commands are the required spine, and only one of them spends money on the corpus.

Plan the crawl. It costs nothing, issues nothing and writes nothing.

pnpm plan --city dubai

It prints 1,250 first-page jobs against a 3,170 worst case and a 2,000 default budget. It also warns that full depth overruns by 1,170. The planner drops 510 of the 1,760 pairs before committing a single credit. Crawling law firms in the desert spends money to find nothing.

The only irreversible spend

Crawl. This is the only irreversible spend in the project, so it refuses to run without --yes.

pnpm crawl --dry-run
pnpm crawl --city dubai --yes --budget 200

Four guards stack here. The mandatory flag, the free dry run, a hard budget stop, and the plan-time drops. A fifth brake sits inside the crawl loop, and it is four lines of arithmetic.

export function shouldFetchNextPage(
  outcome: PageOutcome,
  minNewUniqueRatio = DEFAULT_MIN_NEW_RATIO, // 0.3
): boolean {
  if (outcome.page >= outcome.maxPages) return false;
  if (outcome.resultCount === 0) return false;
  if (outcome.resultCount < FULL_PAGE) return false;   // FULL_PAGE = 20
  return outcome.newUnique / outcome.resultCount >= minNewUniqueRatio;
}

The crawler buys a page only when the previous page came back full and at least 30% of it was new. Every constant in there came from a measurement rather than a guess.

The crawler also writes every raw response to disk before anything parses it. That habit is the single best decision in the project, and I will come back to why.

Classify the vocabulary, not businesses

The taxonomy pass turns a scrape into a directory, and it costs almost nothing. This is the step that makes the unit economics work.

Seed the taxonomy. No model call, no network, no credits.

pnpm seed-taxonomy

132 ordered keyword rules classify the head of the category vocabulary.

Classify the tail. This spends Anthropic tokens and never SearchApi credits.

pnpm classify --yes

The idea worth stealing comes next. The obvious approach sends every business to a model, which scales with your dataset forever. Category vocabulary saturates while business count does not.

My 15,246 businesses contained only 1,787 distinct category strings, which is 8.5 times fewer items. The keyword rules handle the head, so the model only ever sees a tail of 537 strings. It sees them once, across every crawl this project will ever run. A re-crawl that introduces no new strings prints a marginal cost of zero and exits.

Handle the head deterministically and the model never sees 99.5% of your corpus. The committed map wins over both the rules and the model, so the taxonomy is permanent. The next crawl re-applies it for free, and so does the next city.

Thin pages stay out of it. A category needs at least three businesses in a neighbourhood before that page exists at all.

Four gates measure, one does not

Load. Dedupe, drop suppressed listings, assign each business to its nearest tile by coordinates, then normalise.

pnpm load --dry-run
pnpm load --yes

Five acceptance gates print on every run.

• at least 10,000 unique businesses
• 90% phone coverage
• 99% taxonomy coverage
• unique slugs
• zero rows outside the country

My crawl passes at 92.2% phone coverage and 99.5% taxonomy coverage. Four of those five gates measure something. The fifth prints a pass because normalise already rejects out-of-country rows, so it restates an invariant rather than testing one.

Two of those thresholds started higher and came down on evidence. I wanted 95% phone coverage. A probe of the detail endpoint across 15 businesses recovered exactly one extra number. The parser rejected it correctly as an Indian number. So 92.2% is a ceiling, not a shortfall, and the gate now says so.

One crawl produced 14,981 business pages, 782 neighbourhood-by-category landing pages, 81 category pages and 40 neighbourhood hubs. That is 15,887 URLs in the sitemap, all from a single dataset.

Optional passes sit after the spine. pnpm demand asks google_autocomplete what people actually search, at one credit per category, ordered by real query popularity. That turns "which pages should I build" from a guess into a measurement. pnpm leads scores prospects. pnpm export writes CSV, JSON or NDJSON for a CRM.

20,226 review snippets nearly shipped

Because the crawler stores every response untouched before parsing, everything downstream is free to re-run.

pnpm load --from-archive

No network, no credits, full rebuild. I changed the taxonomy, the ranking constants and the normalisation repeatedly over weeks, all against one paid crawl. The test suite runs against committed fixtures and spends nothing.

That archive also produced the ugliest moment in the project. A single unresolvable path in the data loader made Next's file tracer give up and wildcard the directory. That put 1,400 raw crawl files inside the deployment bundle, carrying 20,226 verbatim Google review snippets. Some of them named individual employees. Nothing read them and all of them would have shipped.

The related bug is worse and more interesting. Review theme extraction rewards terms frequent for one business and rare everywhere else, which is precisely the shape of a staff member's name. The first live run produced a Sofitel listing tagged with three employees' first names. A blocklist would have been endless. The fix is a property instead, since a real theme recurs across many businesses while a person's name belongs to one.

Theme extraction reruns against the archive, so the fix never needed a fresh crawl.

My tests sat green throughout. Every headline defect in this repository turned up the same way, by reading real output rather than by failing an assertion. That is the argument for building against a city you already know. Business Directory Toolkit had a real target from day one. When the crawler came back with a couple of hundred nurseries, I could tell whether that smelled right.

What the live demo does

directory.pooyagolchian.com runs on CloudFront as prerendered pages.

A query for dentist returns 512 matches. Typing a +971 number returns the business that owns it. Typeahead answers from a JSON endpoint in one round trip. Business pages carry LocalBusiness structured data, a canonical, and both phone formats. An honest label says the rating came from Google rather than from me.

Some things genuinely do not exist yet. The site has no pagination, so large categories cap at 120 rendered rows and disclose it in plain text. Sorting filters what is already on the page rather than querying the server. I wrote the DynamoDB path and nothing reads it. I would rather say that than let you find out by clicking.

The lead score ranked ATMs

One crawl produces two products. The 1,400 requests that built a public website also built four prospect lists, offline, at zero marginal cost. That is one acquisition cost against two outputs, and it is the only version of the growth-engine claim I will defend.

The second product nearly shipped broken, and the failure was mine. The scoring took two attempts.

The directory carries a chart I did not expect to build. Median review count climbs with rating exactly as you would hope. The bands run 12, 18, 48, 93 and 139. Then it collapses to 11 at exactly 5.0. 2,283 Dubai businesses hold a perfect score, and the typical one has eleven reviews behind it. A perfect score is the bottom of the evidence, not the top of the scale. Listings therefore rank on a credibility-weighted mean instead of a raw average.

My first lead score reused that same credibility-weighted rating, which sounds sensible and inverts itself in practice. Across 641 real leads the score correlated with review count at negative 0.28. The more trade a business had, the further down my call sheet it went. Bank ATMs topped the list. A 3.6-rated hospital with 5,562 reviews sat at number 522 of 641.

The fix generalises into one rule worth stealing. The health term must never be a function of the quantity the signal measures. Score on establishment instead, and the correlation flips to positive 0.08.

pnpm leads scores against four signals, which are no-website, no-hours, weak-reputation and low-visibility. Each signal maps to a different sales conversation, so the tool enforces one signal per run. On my Dubai corpus, 3,820 reachable prospects have no website and 892 have never published hours. Compose the filters and you get 321 restaurants with 20 or more reviews and no website, which is a web agency's afternoon.

Two caveats I will not bury. Reachable means the business has a phone number, so those counts run smaller than the raw count of businesses with the gap. And a lead list is research, not permission to contact anyone. The CLI prints that on every run.

Porting the crawl to Lisbon

The crawl is genuinely city-agnostic. Tiles, categories, bounding boxes, country code and phone region all live in one JSON file. Adding a city requires no code change.

pnpm cities generate --name "Lisbon"

That reads OpenStreetMap and writes the config at zero SearchApi cost. It refuses to fabricate, so a city with too few candidate centres throws rather than falling back to an even grid. Every generated config carries an unverified stamp until somebody actually crawls it. Verification stores evidence rather than a boolean, because a bare verified: true invites a flip, and producing a false one costs a real crawl.

The web layer still says Dubai in a few places, and so do a couple of CLI report labels. Porting the crawl is a data file. Porting the site is still a small patch, and I would rather write that down than imply otherwise.

What I am not claiming

I have proven the toolkit on exactly one city, and that run stopped after 18 of the 44 configured tiles. Even Dubai is not finished. I have taken no latency, Core Web Vitals, ranking or traffic numbers, so I am quoting none.

The repository ships the pipeline and never the corpus. Git ignores both the raw archive and the built dataset, and CI fails the build if either one lands in a commit. Running this requires your own key and your own crawl.

That last constraint is deliberate, and it is the part I find most interesting commercially. A toolkit that does nothing without a SearchApi key means every serious user opens an account. Shipping a hosted API or a paid dataset would have inverted that, because then I would hold the only key that mattered.

What to copy

• Probe the ceiling first, then commit the probe. Page 11 of a Google Maps search returns no local_results key, and that fixture now costs nobody a credit.
• Make the unit of billing the unit of work. One request equals one credit, so a 3x3 lookup table prints the invoice before you spend it.
• Classify the vocabulary, not the rows. My 15,246 businesses held 1,787 distinct category strings, so the model saw a tail of 537 once.
• Never let a health term depend on the quantity its signal measures. That one rule moved the lead score from negative 0.28 to positive 0.08.
• Write every raw response to disk before you parse it. You pay for the call once, and every rebuild after that costs nothing. Then check what your bundler copies.

The code is on GitHub. Take the ranking, the phone parsing or the taxonomy pass on its own, since packages/core is pure and has no idea what a city is.

---

I originally published this at pooyagolchian.com.