DEV Community

Christo
Christo

Posted on

Google kills the Custom Search JSON API on 2027-01-01. Here is a self-hosted drop-in.

#opensource #api #javascript #python

Google's Custom Search JSON API is closed to new customers, and existing customers have until January 1, 2027 to move off it. Straight from Google's own overview page:

The Custom Search JSON API is closed to new customers. Existing Custom Search JSON API customers have until January 1, 2027 to transition to an alternative solution.

The suggested replacement is Vertex AI Search. It's a different API, a different response shape, and a paid product. Whatever you wrote against customsearch/v1 — the client library, the parsing, the pagination loop — gets rewritten.

I didn't want to rewrite mine, so I built the other option: a small self-hosted service that speaks Google's customsearch/v1 wire format on top of a SearXNG instance you run yourself.

- customsearch({version: 'v1'})
+ customsearch({version: 'v1', rootUrl: 'http://localhost:8080/'})
Enter fullscreen mode Exit fullscreen mode

It's called cse-bridge. MIT, zero runtime dependencies, docker compose up -d.

The gap it fills

If you search around, you'll find people correctly pointing out that SearXNG already returns JSON. That's true, and it isn't enough. SearXNG's payload is its own shape — verified in get_json_response:

data = {
    'query': sq.query,
    'results': [_.as_dict() for _ in rc.get_ordered_results()],
    'answers': [...],
    'corrections': list(rc.corrections),
    'infoboxes': rc.infoboxes,
    'suggestions': list(rc.suggestions),
    'unresponsive_engines': get_translated_errors(rc.unresponsive_engines),
}
Enter fullscreen mode Exit fullscreen mode

Each result is roughly {url, title, content, engine}. Your existing code wants items[].link, items[].displayLink, items[].htmlSnippet, queries.nextPage[0].startIndex, searchInformation.totalResults. Nobody had written the adapter, so I did.

Two things that were harder than expected

1. There is no result count. At all.

Look at that payload again. There is no number_of_results, no total, no estimate. So totalResults has to be synthesized, and how you synthesize it decides whether your callers break.

There is one abandoned prototype of this same idea floating around. It does this:

total_results = searxng_data.get('number_of_results', len(searxng_results) * 100)
Enter fullscreen mode Exit fullscreen mode

That key does not exist, so the fallback always fires: ten results become a reported total of 1000. Any client looping while start < totalResults then pages into empty space for ninety results.

cse-bridge reports a lower bound instead — what has actually been retrieved, plus one page's worth only when a next page genuinely exists. It grows monotonically as you page (20, then 30, then 40), so the loop still terminates correctly, and it is never "0" while items exist. It is not a real total and it does not pretend to be. If your UI prints "about 1,240,000 results", you will now see an honest, much smaller number.

2. A live SearXNG reshuffles between identical calls

This one only shows up against a real instance. SearXNG merges several engines per request, those engines have varying latency, and some drop out entirely (unresponsive_engines will show you things like ["brave", "Suspended: too many requests"]). Run the same query twice, seconds apart, and the ordering differs.

Page straight through that and start=11 re-serves links start=1 already showed. Measured on a live instance:

start= 1  items=10
start=11  items=10
start=21  items=10
LINKS: 30  UNIQUE: 28   <-- two duplicates
Enter fullscreen mode Exit fullscreen mode

Google resolves a query to a stable result set and pages within it. So cse-bridge does the same: a query resolves to one de-duplicated ordered set held for a TTL (default 5 minutes), and pages are slices of that set. Re-measured:

LINKS: 30  UNIQUE: 30
Enter fullscreen mode Exit fullscreen mode

As a bonus, a cached deep page costs zero backend calls instead of re-walking pages 1..N.

Client recipes

All three of these are verified end to end against a live stack, with unmodified client libraries.

Node, @googleapis/customsearch

import { customsearch } from '@googleapis/customsearch';

const client = customsearch({ version: 'v1', rootUrl: 'http://localhost:8080/' });
const res = await client.cse.list({ q: 'test', cx: 'default', auth: 'k' });
console.log(res.data.items.length); // 10
Enter fullscreen mode Exit fullscreen mode

Python, google-api-python-client

from google.api_core.client_options import ClientOptions
from googleapiclient.discovery import build

service = build("customsearch", "v1", developerKey="k",
                client_options=ClientOptions(api_endpoint="http://localhost:8080"))
res = service.cse().list(q="test", cx="default").execute()
Enter fullscreen mode Exit fullscreen mode

LangChain, GoogleSearchAPIWrapper

This one builds its client inside a validator and sets extra="forbid", so you cannot pass client_options in. Swap the built service afterwards — one line, wrapper class untouched:

search = GoogleSearchAPIWrapper(google_api_key="k", google_cse_id="default")
search.search_engine = build("customsearch", "v1", developerKey="k",
                             client_options=ClientOptions(api_endpoint="http://localhost:8080"))
search.run("test")
Enter fullscreen mode Exit fullscreen mode

Everything downstream (GoogleSearchRun, agent toolkits) works unchanged, because it all goes through search_engine.

What your cx becomes

Your cx used to identify a Programmable Search Engine in Google's control panel. Here it selects a block in profiles.yml, so the client keeps sending the same cx and you decide server-side what it means:

default:
  categories: [general]

"012345678901234567890:abcdefghij":
  description: Was the Docs PSE in the Google control panel
  categories: [general]
  site: docs.example.com
Enter fullscreen mode Exit fullscreen mode

An unknown cx falls back to default rather than erroring — a client you are migrating cannot change the cx it sends.

One gotcha that cost me an hour

If you are on Python behind a corporate proxy, google-api-python-client will die before it sends a byte:

httplib2.error.ProxiesUnavailableError: Proxy support missing but proxy use was requested!
Enter fullscreen mode Exit fullscreen mode

httplib2 raises whenever HTTP_PROXY / HTTPS_PROXY are set and the optional PySocks package is missing — and that check runs before it evaluates host bypass, so adding localhost to NO_PROXY does not help. Install PySocks, or clear the proxy vars for the process. Your bridge is on localhost; that traffic should not be proxied anyway.

Honest limitations

  • totalResults is a lower bound, not a web-wide estimate.
  • 100 results max per query (start no higher than 91), same as Google.
  • No pagemap, no structured data, no rich snippets.
  • No image search.
  • Result quality is your SearXNG's engine configuration, not Google's.
  • Rate limits are now yours to own.

Try it 

git clone https://github.com/Booyaka101/cse-bridge.git
cd cse-bridge
docker compose up -d
curl 'http://localhost:8080/customsearch/v1?key=k&cx=default&q=rust%20async%20runtime&num=3'
Enter fullscreen mode Exit fullscreen mode

Or npm i -g cse-bridge, or docker pull ghcr.io/booyaka101/cse-bridge.

Source: github.com/Booyaka101/cse-bridge

If you are one of the roughly 65,000 monthly @googleapis/customsearch downloads, you have a deadline. Might as well find out now whether a swap works for you.

Top comments (0)