<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Alexandr Kazmin</title>
    <description>The latest articles on DEV Community by Alexandr Kazmin (@alkaznodemaven).</description>
    <link>https://dev.to/alkaznodemaven</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4166105%2F83835fb5-dbb7-4de8-928f-57b9d45b4393.jpg</url>
      <title>DEV Community: Alexandr Kazmin</title>
      <link>https://dev.to/alkaznodemaven</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/alkaznodemaven"/>
    <language>en</language>
    <item>
      <title>Give your AI agent real Google rankings with a self-hosted MCP server</title>
      <dc:creator>Alexandr Kazmin</dc:creator>
      <pubDate>Mon, 16 Nov 2026 12:30:45 +0000</pubDate>
      <link>https://dev.to/alkaznodemaven/give-your-ai-agent-real-google-rankings-with-a-self-hosted-mcp-server-mi6</link>
      <guid>https://dev.to/alkaznodemaven/give-your-ai-agent-real-google-rankings-with-a-self-hosted-mcp-server-mi6</guid>
      <description>&lt;p&gt;Most agents can already "search the web". What they get back is a handful of sources picked by&lt;br&gt;
their own search provider, not what Google shows for a query: no positions, no AI Overview, no&lt;br&gt;
"People also ask", and no say in which country the results are for. For SEO work, rank checks or&lt;br&gt;
competitor research, that is the part you need.&lt;/p&gt;

&lt;p&gt;This walkthrough plugs Google's actual results page into Claude Code, Cursor, Gemini CLI or&lt;br&gt;
Claude Desktop as a tool, using &lt;a href="https://github.com/nodemaven/serpmaven" rel="noopener noreferrer"&gt;SERP Maven&lt;/a&gt;, an&lt;br&gt;
open-source Google scraper with a built-in MCP server. It runs on your machine through your&lt;br&gt;
residential proxy, so there is no per-search bill. Setup takes about five minutes.&lt;/p&gt;
&lt;h2&gt;
  
  
  What you need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Python 3.10+, or &lt;a href="https://docs.astral.sh/uv/" rel="noopener noreferrer"&gt;uv&lt;/a&gt; (it brings its own Python).&lt;/li&gt;
&lt;li&gt;A residential proxy with &lt;strong&gt;sticky sessions&lt;/strong&gt;. Any provider works. We build SERP Maven at
NodeMaven, so NodeMaven proxies have a one-command login; others take a proxy URL.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  1. Install
&lt;/h2&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;uv tool install serpmaven
serpmaven install
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The second command downloads the browser SERP Maven drives, about 150 MB, once per machine.&lt;br&gt;
Do it now: an MCP server cannot stop and ask you about a download.&lt;/p&gt;
&lt;h2&gt;
  
  
  2. Save your proxy login
&lt;/h2&gt;

&lt;p&gt;For NodeMaven, enter the proxy username and password from the dashboard:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;serpmaven login
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For any other provider, give the proxy URL with &lt;code&gt;{session}&lt;/code&gt; where the provider expects a&lt;br&gt;
session id in the username:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;serpmaven login --proxy &lt;span class="s2"&gt;"http://USER-session-{session}:PASS@gate.example.com:7000"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The login is stored in a file only your user can read. The MCP server picks it up from there,&lt;br&gt;
so no password goes into your editor's config.&lt;/p&gt;
&lt;h2&gt;
  
  
  3. Try one search from the terminal
&lt;/h2&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;serpmaven search &lt;span class="s2"&gt;"best running shoes"&lt;/span&gt; --country us
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The first search takes a few minutes. SERP Maven opens a real Chrome window and browses a few&lt;br&gt;
ordinary pages on the first proxy IP before it asks Google anything, because a fresh browser&lt;br&gt;
on a fresh IP is the thing Google refuses most. Later searches take seconds, and new proxy IPs&lt;br&gt;
start from the cookies Google already accepted. The run ends with a summary of pages served&lt;br&gt;
and traffic used.&lt;/p&gt;
&lt;h2&gt;
  
  
  4. Connect it to your agent
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Claude Code.&lt;/strong&gt; One command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;claude mcp add google-search -- serpmaven mcp --country us --prewarm
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Cursor.&lt;/strong&gt; Add this to &lt;code&gt;~/.cursor/mcp.json&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"mcpServers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"google-search"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"serpmaven"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"args"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"mcp"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"--country"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"--prewarm"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Gemini CLI.&lt;/strong&gt; The repository is also a Gemini CLI extension:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;gemini extensions install https://github.com/nodemaven/serpmaven
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Claude Desktop.&lt;/strong&gt; The same JSON block goes into Settings → Developer → Edit Config. Restart&lt;br&gt;
Claude Desktop and the &lt;code&gt;google_search&lt;/code&gt; tool appears in the tools menu:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9n09o13pnlcapr4e28ma.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9n09o13pnlcapr4e28ma.png" alt="Claude Desktop asking to run google_search with the query, country, language and links" width="772" height="485"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If a client says it cannot find &lt;code&gt;serpmaven&lt;/code&gt;, it was started without your shell's PATH. Put the&lt;br&gt;
full path from &lt;code&gt;which serpmaven&lt;/code&gt; (&lt;code&gt;where serpmaven&lt;/code&gt; on Windows) into &lt;code&gt;command&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;--prewarm&lt;/code&gt; warms the first proxy IP as soon as the server starts. That takes three to five&lt;br&gt;
minutes, longer than some clients wait for a tool, so give it those minutes after a restart&lt;br&gt;
before the first question. Later searches take under ten seconds. With NodeMaven, &lt;code&gt;--country&lt;/code&gt;&lt;br&gt;
picks the country of the proxy IP; the tool's &lt;code&gt;gl&lt;/code&gt; argument sets the country Google answers for&lt;br&gt;
on each call.&lt;/p&gt;
&lt;h2&gt;
  
  
  What the agent gets back
&lt;/h2&gt;

&lt;p&gt;The server exposes one tool, &lt;code&gt;google_search(query, pages, hl, gl, links)&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;query&lt;/code&gt; - the search;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pages&lt;/code&gt; - result pages to fetch, about 10 results each (default 1);&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;hl&lt;/code&gt;, &lt;code&gt;gl&lt;/code&gt; - interface language and country, e.g. &lt;code&gt;de&lt;/code&gt; and &lt;code&gt;de&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;links&lt;/code&gt; - how many top results get their full URL (default 5); fewer answers sooner.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Ask your agent something like &lt;em&gt;"Who ranks on Google in the US for 'best python web scraping&lt;br&gt;
library'?"&lt;/em&gt; and the tool returns compact JSON with Google's own order. A real answer, trimmed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"results"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"position"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Scrapy, a fast high-level web crawling &amp;amp; scraping ..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://github.com/scrapy/scrapy"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"snippet"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Scrapy is a web scraping framework to extract structured data from websites..."&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"position"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Advanced Web Scraping Tutorial! (w/ Python Beautiful Soup ..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.youtube.com/watch?v=DcI_AZqfZVc"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"position"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"High Level Web Scraping Library for Python"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.reddit.com/r/Python/comments/1iy78rw/high_level_web_scraping_library_for_python/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"snippet"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Hi, I started working on an open source Python library that handles common web scraping tasks..."&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"ai_overview"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"There is no single best Python web scraping library because the right choice depends on your target site and scale. For small static pages, Beautiful Soup paired with Requests or HTTPX is the easiest starting point..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"related_searches"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"Best python web scraping library free"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Best python web scraping library github"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Web scraping tools"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Results past the first five carry the site instead of the full link (raise &lt;code&gt;links&lt;/code&gt; if the agent&lt;br&gt;
needs them). Organic results and related searches come with every page; "People also ask",&lt;br&gt;
Google's AI Overview and the map pack (name, rating, address, hours) come when Google shows them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;The proxy is the only cost. The first IP's warm-up uses about 60 MB once. After that a search&lt;br&gt;
is about 0.4 MB, so 1000 searches are roughly 0.4 GB of proxy traffic. Keep &lt;code&gt;pages&lt;/code&gt; at 1 unless&lt;br&gt;
you need more: every page is traffic on your plan.&lt;/p&gt;

&lt;p&gt;New to NodeMaven? &lt;a href="https://dashboard.nodemaven.com/accounts/signup/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=serpmaven_mcp" rel="noopener noreferrer"&gt;Sign up&lt;/a&gt;&lt;br&gt;
and enter &lt;strong&gt;GOOGLENM&lt;/strong&gt; for 2 GB of free traffic.&lt;/p&gt;

&lt;h2&gt;
  
  
  When something goes wrong
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The server shows as failed or disconnected&lt;/strong&gt; in your client: it stops at start when there
is no saved proxy login or no browser. Run &lt;code&gt;serpmaven login&lt;/code&gt; and &lt;code&gt;serpmaven install&lt;/code&gt; once,
then restart the client.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The first search times out&lt;/strong&gt; right after the client starts: the proxy IP is still warming
up. Wait a few minutes and ask again; the warm-up keeps going in the background.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A captcha or a block is never read as "no results".&lt;/strong&gt; SERP Maven drops the proxy IP Google
refused and asks again on another one; a search that still fails comes back with an error,
so the agent can tell "Google refused" from "nothing found".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Results for a different query&lt;/strong&gt; are Google's quiet way of turning a scraper away. When a
proxy IP starts answering like that, SERP Maven drops it and asks the query again on a fresh
one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On a server without a screen,&lt;/strong&gt; run it under &lt;code&gt;xvfb-run -a&lt;/code&gt;, or use the Docker image
&lt;code&gt;ghcr.io/nodemaven/serpmaven&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;serpmaven doctor&lt;/code&gt;&lt;/strong&gt; starts the browser on a blank page and reports what a website would
see. Run it first if searches keep failing.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The same thing without an agent
&lt;/h2&gt;

&lt;p&gt;The MCP server is one of four ways to run SERP Maven. The others:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;serpmaven search -f queries.txt -o results.jsonl&lt;/code&gt; for a batch;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;serpmaven serve&lt;/code&gt; for a local HTTP API with SERP-API-style JSON;&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;from serpmaven import Scraper&lt;/code&gt; in Python.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Code, options and issues: &lt;a href="https://github.com/nodemaven/serpmaven" rel="noopener noreferrer"&gt;github.com/nodemaven/serpmaven&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>mcp</category>
      <category>agentskills</category>
    </item>
    <item>
      <title>We ran 10k Google searches through residential proxies. Here is what poisoned results actually look like</title>
      <dc:creator>Alexandr Kazmin</dc:creator>
      <pubDate>Thu, 29 Oct 2026 13:55:37 +0000</pubDate>
      <link>https://dev.to/alkaznodemaven/we-ran-10k-google-searches-through-residential-proxies-here-is-what-poisoned-results-actually-look-405l</link>
      <guid>https://dev.to/alkaznodemaven/we-ran-10k-google-searches-through-residential-proxies-here-is-what-poisoned-results-actually-look-405l</guid>
      <description>&lt;p&gt;"Poisoned results" have become one of the most talked-about problems in Google scraping. The idea: instead of blocking an automated client, Google quietly serves it a normal-looking results page for a different query. No captcha, no error code - just wrong data that slides into your dataset.&lt;/p&gt;

&lt;p&gt;We had been collecting Google results for a while for our own benchmarks, so we went back through about 10,000 results pages, gathered through real Chrome browsers and around 160 residential proxy IPs, to see what poisoning looks like in practice. Three things stood out.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Poisoning is real, but it is not random noise
&lt;/h2&gt;

&lt;p&gt;It happened. On one proxy IP, after a few normal searches, Google started answering with pages that had nothing to do with the query:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;em&gt;"how to set up a fish tank"&lt;/em&gt; returned Oracle, Mozilla and Wikipedia pages&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;"best coffee makers under 100"&lt;/em&gt; returned Under Armour's store and a dictionary entry for "under"&lt;/li&gt;
&lt;li&gt;
&lt;em&gt;"mortgage calculator with taxes"&lt;/em&gt; returned generic calculator sites&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two things about it are worth knowing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It is rare.&lt;/strong&gt; About 0.3% of all the pages we looked at were poisoned, and all of them came from a single proxy IP. On US proxy IPs we did not see it at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It sticks to an IP.&lt;/strong&gt; Once that proxy IP started getting junk, it kept getting junk for dozens of searches in a row. Poisoning looked like a state the IP was put into, not a coin flip on each request.&lt;/p&gt;

&lt;p&gt;That second point is what makes it catchable.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. How to tell a poisoned page from a normal one
&lt;/h2&gt;

&lt;p&gt;The tell is simple: on a poisoned page, almost no result title contains the words you searched for. On a normal page, most of them do.&lt;/p&gt;

&lt;p&gt;We measured it as the share of result titles that carry at least half of the query's meaningful words (after dropping words like "how", "to", "best"). On normal pages this was usually 80-100%. On poisoned pages it was mostly 0-10%.&lt;/p&gt;

&lt;p&gt;The catch is that about 1% of perfectly normal pages also score low. Brand queries are the typical case: titles say "Costco" and "Costco Wholesale" but not "what time does Costco open". Local queries answered with business names do the same.&lt;/p&gt;

&lt;p&gt;So a single low page proves nothing. Three low pages in a row on the same proxy IP do. Across the whole dataset, that rule flagged exactly one proxy IP, the poisoned one, and nothing else. When it fires, the fix is cheap: drop the IP and ask the same query again on a fresh one. And since the check runs on a page you already downloaded, it costs no extra requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. There is no such thing as "the" Google results page
&lt;/h2&gt;

&lt;p&gt;While checking for poisoning, we compared the same queries across several sources: our own browser, five commercial SERP APIs, and repeated requests to each.&lt;/p&gt;

&lt;p&gt;Take the top 10 result domains for a query and compare two sources. The overlap between any two of them averaged only 26-56%. Even a single API asked the same question twice, minutes apart, agreed with itself only 38-77% of the time.&lt;/p&gt;

&lt;p&gt;Two browser searches from the same proxy IP at the same moment, on the other hand, matched 97%.&lt;/p&gt;

&lt;p&gt;The difference is location. Google's US results depend heavily on which city the request appears to come from and when it is made. Two sources that use different IPs are, in effect, asking two different Googles.&lt;/p&gt;

&lt;p&gt;The practical takeaway: if you compare rankings across tools or days, keep the proxy location fixed, or you will be measuring geography, not ranking changes. And "my results don't match what a SERP API returns" is not by itself a sign that either one is wrong.&lt;/p&gt;

&lt;p&gt;Two smaller observations from the same comparison:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;AI Overviews are now the norm.&lt;/strong&gt; For everyday queries, Google showed an AI Overview on most pages. Two of the five APIs we tried did not return it at all.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Map places show up on a small share of queries&lt;/strong&gt;, mostly ones with "near me" or a city in them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  4. Do you need a browser for every page?
&lt;/h2&gt;

&lt;p&gt;A real browser is the most reliable way past Google's checks, and also the most expensive: every page pulls scripts, images and fonts, and on residential proxies you pay for every byte.&lt;/p&gt;

&lt;p&gt;We tested a hybrid: let the browser pass Google's checks, then fetch results pages directly over HTTPS with the browser's own cookies and headers, falling back to the browser whenever Google asks again. We ran 300 side-by-side pairs, the same query through the browser and through HTTPS on the same IP. The results were not identical - the top 10 overlapped 86%, against 97% for two browser searches - but they were just as good: the same number of results, equally relevant titles, and the same AI Overview every time.&lt;/p&gt;

&lt;p&gt;What it saved:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Browser for every page&lt;/th&gt;
&lt;th&gt;Hybrid&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Traffic per search&lt;/td&gt;
&lt;td&gt;0.8 MB&lt;/td&gt;
&lt;td&gt;0.4 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Time per search&lt;/td&gt;
&lt;td&gt;~9 s&lt;/td&gt;
&lt;td&gt;~1.4 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  The tool behind the numbers
&lt;/h2&gt;

&lt;p&gt;All of this was collected with &lt;a href="https://github.com/nodemaven/google-browser-scraper" rel="noopener noreferrer"&gt;Google Browser Scraper&lt;/a&gt;, an open-source tool we maintain. Version 0.1 ships both findings: hybrid mode is on by default, and the three-in-a-row poisoning check drops affected proxy IPs automatically. It runs as a CLI, a Python library, a local SERP-style API, or an MCP server for AI agents, and returns JSON with organic results, "People also ask", AI Overview and map places.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;google-browser-scraper
patchright &lt;span class="nb"&gt;install &lt;/span&gt;chromium
google-browser-scraper search &lt;span class="s2"&gt;"best running shoes"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--proxy&lt;/span&gt; &lt;span class="s2"&gt;"http://USER-session-{session}:PASS@gate.example.com:7000"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It works with any residential proxy that supports sticky sessions. It has built-in support for &lt;a href="https://nodemaven.com/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=google_browser_scraper&amp;amp;utm_content=research" rel="noopener noreferrer"&gt;NodeMaven&lt;/a&gt;, and the code &lt;strong&gt;GOOGLENM&lt;/strong&gt; gives new accounts 2 GB of free traffic.&lt;/p&gt;

&lt;p&gt;If you have seen poisoned results yourself, especially on non-US IPs, we would like to hear what they looked like.&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>seo</category>
      <category>datascience</category>
    </item>
    <item>
      <title>A cold proxy exit gets Google results 20% of the time. A warmed one gets 87%</title>
      <dc:creator>Alexandr Kazmin</dc:creator>
      <pubDate>Tue, 06 Oct 2026 10:54:43 +0000</pubDate>
      <link>https://dev.to/alkaznodemaven/a-cold-proxy-exit-gets-google-results-20-of-the-time-a-warmed-one-gets-87-3el8</link>
      <guid>https://dev.to/alkaznodemaven/a-cold-proxy-exit-gets-google-results-20-of-the-time-a-warmed-one-gets-87-3el8</guid>
      <description>&lt;p&gt;You rotate to a fresh residential IP, open Google in a real browser, type a query - and get a captcha. Again. We assumed that was the proxy. We measured it, and it mostly wasn't: what mattered most was whether that exit had done anything before the search.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers
&lt;/h2&gt;

&lt;p&gt;Every attempt below opens google.com in a real, headful browser (Chromium 151&lt;br&gt;
through Patchright), types a query into the search box and checks whether Google&lt;br&gt;
served results or a captcha. Each attempt gets a fresh sticky session on a&lt;br&gt;
residential pool, so each one starts from a new exit.&lt;/p&gt;

&lt;p&gt;The cold arm searches straight away. The warm arm first browses six ordinary&lt;br&gt;
pages on the same session, then searches.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;served&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;cold&lt;/td&gt;
&lt;td&gt;20% (46 of 232)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;warm, six pages&lt;/td&gt;
&lt;td&gt;87% (409 of 471)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That is four runs on two machines. Inside every run the cold and warm arms are&lt;br&gt;
interleaved, because the hour of the day is one of the biggest effects we have&lt;br&gt;
measured, and running the two arms an hour apart would compare the hours.&lt;/p&gt;

&lt;p&gt;One run tried more depths side by side:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;warm-up depth&lt;/th&gt;
&lt;th&gt;0&lt;/th&gt;
&lt;th&gt;2&lt;/th&gt;
&lt;th&gt;4&lt;/th&gt;
&lt;th&gt;7&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;served&lt;/td&gt;
&lt;td&gt;11%&lt;/td&gt;
&lt;td&gt;24%&lt;/td&gt;
&lt;td&gt;33%&lt;/td&gt;
&lt;td&gt;82%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Depth 7 is the six-page warm-up above. A separate test of a single page of&lt;br&gt;
warm-up moved nothing (30% against 32%). The effect builds with depth, and the&lt;br&gt;
big jump is at the deepest rung.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we don't know
&lt;/h2&gt;

&lt;p&gt;Four of the six warm-up pages are Google's own. So two explanations fit every&lt;br&gt;
row: Google's infrastructure has already seen this exit behave normally, or the&lt;br&gt;
browser has simply lived through several navigations. The run that separates&lt;br&gt;
them - the same depth with third-party pages only - has not been done yet. This&lt;br&gt;
is an effect without a mechanism, and I'd rather say so than guess.&lt;/p&gt;

&lt;h2&gt;
  
  
  The machine matters too
&lt;/h2&gt;

&lt;p&gt;The same code, the same gateway and the same parameters, in overlapping hours:&lt;br&gt;
a Windows workstation was served 39% (24 of 61), a Linux VPS 0% (0 of 84). We&lt;br&gt;
haven't isolated which property of the Linux host Google reads. If your scraper&lt;br&gt;
works on your laptop and dies on a server, this is a candidate.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;Warm-up is not free. In a smoke run of six queries, warming one exit took about&lt;br&gt;
three minutes and 34 MB of proxy traffic. After that, each results page took&lt;br&gt;
17-21 seconds and 0.2-1.6 MB. So the strategy pays only if you keep a warm exit&lt;br&gt;
for several queries instead of burning a new one each time.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do with this.
&lt;/h2&gt;

&lt;p&gt;Don't send a search from a cold exit. Give it a few ordinary pages first, keep the sticky session, and reuse a warm exit for as many queries as Google keeps serving it. Rotating to a new IP on every request is the most expensive way to get captchas.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproduce it
&lt;/h2&gt;

&lt;p&gt;Everything is public: the harness, the run files and the scripts that turn rows&lt;br&gt;
into the tables above.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Harness and raw rows: &lt;a href="https://github.com/nodemaven/proxy-benchmark" rel="noopener noreferrer"&gt;https://github.com/nodemaven/proxy-benchmark&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The scraper we built on it: &lt;a href="https://github.com/nodemaven/google-browser-scraper" rel="noopener noreferrer"&gt;https://github.com/nodemaven/google-browser-scraper&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;google-browser-scraper
patchright &lt;span class="nb"&gt;install &lt;/span&gt;chromium
google-browser-scraper search &lt;span class="s2"&gt;"best running shoes"&lt;/span&gt; &lt;span class="nt"&gt;--proxy&lt;/span&gt; &lt;span class="s2"&gt;"http://USER-session-{session}:PASS@your-gateway:port"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;{session}&lt;/code&gt; is where your provider expects a session id. It works with any&lt;br&gt;
provider; I work at one (NodeMaven), which is why we had the pool to measure&lt;br&gt;
this on. If you re-run it and get different numbers, I'd happy to see them.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>playwright</category>
      <category>proxies</category>
    </item>
  </channel>
</rss>
