A $1.50 search API where the URL is the interface

#api #webdev #google #ai

I use a hosted Google search API to find company websites from a company name, and I switched to Bright Data's SERP endpoint because the billing model fits intermittent use: $1.50 per 1,000 successful requests, a 5,000 request monthly free tier, no monthly minimum, and failed requests are not billed.

Then my first accuracy test confirmed 15 of 25. The API was not slow and it was not erroring. Two parameters were missing from the URL I sent, and one of them changed the search results from Japan to India.

The call

The endpoint takes a search URL and returns the engine's response. For Google with organic results in a light JSON shape:

curl -X POST https://api.brightdata.com/request \
  -H "Authorization: Bearer $BRIGHTDATA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "zone": "<serp zone>",
    "format": "raw",
    "url": "https://www.google.com/search?q=ACME+%E6%A0%AA%E5%BC%8F%E4%BC%9A%E7%A4%BE&hl=ja&gl=jp&brd_json=parsed_light",
    "data_format": "parsed_light"
  }'

format: "raw" returns the body as text. brd_json=parsed_light asks for the parsed view, and data_format tells the API to hand it back as JSON rather than a string.

The five things that broke

Symptom	Cause	Fix
Japanese companies missing from results	no `gl` parameter, so the geolocation defaulted elsewhere	set `gl=jp` (and `hl=ja`)
`400` with no useful message	`num` is rejected by the SERP layer	ask for the default page size, paginate with `start` if needed
request timed out	`brd_json=1` (full JSON) on a heavy result set	use `brd_json=parsed_light`
took 6 to 12 seconds per call	it is a real browser fetch, not a cached index	budget for it, run calls concurrently
occasional `502`	upstream fetch failure	retry once, then treat as no result

The geolocation one is the expensive mistake. With no gl, the API returned results with a country field of India or Portugal for Japanese company names. Nothing in the response says the search was performed from the wrong place; the results just look plausible and are wrong. If your pipeline extracts the first organic link, you get a confident wrong answer.

What the numbers looked like

I test any search provider against a hard set: 25 companies whose websites I have already confirmed by hand, chosen because name-based guessing had failed on them before. For each company I count three things separately:

retrieved: the correct domain appears anywhere in the organic results
confirmed: my verification step accepts it as the company's site
false confirmation: the pipeline accepted a wrong domain

Path	Retrieved	Confirmed	Wrong domain accepted
guessed URL from the company name, old rules	not applicable	15 of 25	yes, before a Japan check was added
plain JSON, no geolocation	16 of 25	15 of 25	0
parsed light JSON with `gl=jp`	19 of 25	16 of 25	0

Retrieval went from 64 percent to 76 percent just by telling the API where I was searching from. The confirmed count moved more, because correct results that were previously outranked by same-name foreign companies became reachable.

Zero wrong domains in the two API configurations is the number I care about most, and it is zero because the verification step now requires evidence that the page belongs to a Japanese entity: Japanese text, a Japan address, or a location. Before that check existed, the guessing path nearly accepted a Slovenian domain for a Japanese company whose name matched. For this use case a miss costs one manual lookup, and a wrong domain corrupts the record silently.

What comes back

parsed_light gives you the organic block as structured JSON: position, title, link, description, plus the engine's own metadata. It is around ten results, which is what the first page holds. The full JSON variant contains every module Google renders on that page, and on a rich result set it is large enough that the whole request occasionally fails. If you need a knowledge panel or related questions, pay for the full parse deliberately rather than by default.

One billing detail that surprised me, from the docs: the pay-per-success model applies to the standard configuration. If you set custom headers or cookies on the zone, every request is billed including failures, and retries are still not extra. If you need a specific Accept-Language or a consent cookie, model the cost as "all attempts" rather than "successes".

Where this tool is the wrong choice

For anything that can wait, a queued batch API is cheaper per request. This endpoint exists for the case where you want the answer in the same second, and you pay for that: a real browser, a proxy that is not your datacenter IP, and a parser on the other side.

It is also not a search index. Asking it the same question repeatedly costs the same every time, so caching the query and the result is not optional at any volume.

What it costs at different volumes, and where the cheaper option is not the one you would guess, is in the pricing model post.

If you have moved a scraping job from a self-hosted headless browser to a hosted SERP API: what did you lose that you did not expect?

Originally published at agentjournal.dev.

DEV Community