<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: SerpScraper.dev</title>
    <description>The latest articles on DEV Community by SerpScraper.dev (@serpscraperdev).</description>
    <link>https://dev.to/serpscraperdev</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2109705%2F47d319df-6491-498e-b9b7-b6072856803e.png</url>
      <title>DEV Community: SerpScraper.dev</title>
      <link>https://dev.to/serpscraperdev</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/serpscraperdev"/>
    <language>en</language>
    <item>
      <title>How to Build a Google Search API Integration with PHP and Laravel</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Wed, 28 Oct 2026 06:57:35 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/how-to-build-a-google-search-api-integration-with-php-and-laravel-38nn</link>
      <guid>https://dev.to/serpscraperdev/how-to-build-a-google-search-api-integration-with-php-and-laravel-38nn</guid>
      <description>&lt;p&gt;If you're building an SEO platform, a keyword rank tracker, a competitor research tool, or an AI application that needs real-time web data, Google search results can be a valuable data source.&lt;/p&gt;

&lt;p&gt;The challenge is collecting that data reliably.&lt;/p&gt;

&lt;p&gt;Writing your own scraper means dealing with changing HTML structures, proxy management, CAPTCHA challenges, retries, and data normalization. These tasks can quickly become more complicated than the application you're actually trying to build.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;Google Search API&lt;/strong&gt; offers a simpler approach: send an HTTP request with your search query and receive structured JSON that your application can process.&lt;/p&gt;

&lt;p&gt;In this tutorial, we'll explore how to retrieve Google SERP data, configure country and language parameters, paginate search results, and integrate a search API into PHP and Laravel applications using &lt;a href="https://serpscraper.dev/" rel="noopener noreferrer"&gt;SerpScraper.dev&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. What Is a Google Search API?
&lt;/h2&gt;

&lt;p&gt;A Google Search API provides programmatic access to search result data through an HTTP interface.&lt;/p&gt;

&lt;p&gt;Instead of downloading raw HTML and writing custom parsers, your application sends a query and receives structured data containing fields such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Result titles and URLs&lt;/li&gt;
&lt;li&gt;Organic ranking positions&lt;/li&gt;
&lt;li&gt;Search snippets&lt;/li&gt;
&lt;li&gt;Related keywords and questions, where available&lt;/li&gt;
&lt;li&gt;Additional SERP features supported by the endpoint&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This data can power many applications:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;SEO dashboards and keyword rank trackers&lt;/li&gt;
&lt;li&gt;Competitor analysis platforms&lt;/li&gt;
&lt;li&gt;Search monitoring and market research tools&lt;/li&gt;
&lt;li&gt;Content research automation&lt;/li&gt;
&lt;li&gt;AI agents and retrieval-augmented generation (RAG) pipelines&lt;/li&gt;
&lt;li&gt;Data enrichment and search intelligence services&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It's important to distinguish a third-party SERP data API from Google's official Programmable Search products. Their coverage, response formats, and use cases can differ significantly.&lt;/p&gt;

&lt;p&gt;For applications that need organic search results rather than a search experience restricted to configured websites, evaluate the available endpoints and their actual response data before choosing a provider.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Getting Started with SerpScraper.dev
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://serpscraper.dev/web-search-api" rel="noopener noreferrer"&gt;SerpScraper.dev&lt;/a&gt; provides a REST API for retrieving structured search data from Google and other supported search sources.&lt;/p&gt;

&lt;p&gt;The API handles the search data collection process, so you can focus on building your application instead of maintaining scraping infrastructure.&lt;/p&gt;

&lt;p&gt;Getting started is straightforward:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Create an account at &lt;a href="https://serpscraper.dev/" rel="noopener noreferrer"&gt;SerpScraper.dev&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Obtain your API token from the dashboard.&lt;/li&gt;
&lt;li&gt;Review the &lt;a href="https://serpscraper.dev/docs" rel="noopener noreferrer"&gt;API documentation&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Send your first search request.&lt;/li&gt;
&lt;li&gt;Process the returned JSON in your application.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;New accounts receive 5,000 free credits at signup, with no credit card required. Check the current &lt;a href="https://serpscraper.dev/pricing" rel="noopener noreferrer"&gt;pricing plans&lt;/a&gt; for credit usage and plan limits.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Make Your First Google Search API Request
&lt;/h2&gt;

&lt;p&gt;The Web Search API endpoint is:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;GET https://serpscraper.dev/api/v1/webs-search&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Let's start with a simple cURL request.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-G&lt;/span&gt; &lt;span class="s2"&gt;"https://serpscraper.dev/api/v1/webs-search"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"keyword=laravel seo"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"gl=US"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"hl=en"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"page=1"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"size=10"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--data-urlencode&lt;/span&gt; &lt;span class="s2"&gt;"token=YOUR_API_KEY"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Replace &lt;code&gt;YOUR_API_KEY&lt;/code&gt; with your actual API token.&lt;/p&gt;

&lt;p&gt;The request uses several parameters:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Parameter&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;keyword&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Search query&lt;/td&gt;
&lt;td&gt;&lt;code&gt;laravel seo&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gl&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Target country&lt;/td&gt;
&lt;td&gt;&lt;code&gt;US&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;hl&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Preferred result language&lt;/td&gt;
&lt;td&gt;&lt;code&gt;en&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;page&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Results page&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;size&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Requested results per page&lt;/td&gt;
&lt;td&gt;&lt;code&gt;10&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;token&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;API authentication token&lt;/td&gt;
&lt;td&gt;Your API key&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The country and language parameters are useful when your application monitors keywords across multiple markets.&lt;/p&gt;

&lt;p&gt;For example, the same query may return different websites when targeting the United States, the United Kingdom, or Vietnam.&lt;/p&gt;

&lt;p&gt;For the complete parameter list and supported response fields, see the &lt;a href="https://serpscraper.dev/docs" rel="noopener noreferrer"&gt;SerpScraper.dev documentation&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Working with JSON Search Results
&lt;/h2&gt;

&lt;p&gt;The API returns a structured response with a request summary, endpoint-specific data, and timing information.&lt;/p&gt;

&lt;p&gt;A simplified example of the response structure looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"request"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"keyword"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"laravel seo"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"page"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"size"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"gl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"hl"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"en"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"data"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"items"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"position"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Example SEO Guide"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"link"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://example.com/seo-guide"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"An example search result description."&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"in_seconds"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.31&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is an illustrative response, not a live API result. The exact fields depend on the endpoint and returned data.&lt;/p&gt;

&lt;p&gt;Once the JSON is available, your application can save the results to a database, calculate ranking changes, generate reports, or pass selected fields to an AI model.&lt;/p&gt;

&lt;p&gt;For a rank tracker, for example, you might store the keyword, search date, country, domain, URL, and ranking position. Comparing snapshots over time lets you identify changes that a single search cannot reveal.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Retrieving More Than 10 Search Results
&lt;/h2&gt;

&lt;p&gt;One common challenge in SERP data collection is pagination.&lt;/p&gt;

&lt;p&gt;Suppose you're building a rank-tracking tool and need to collect up to 100 organic search results for a keyword.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;size&lt;/code&gt; parameter controls how many results you request per page. SerpScraper.dev's documented Web Search API supports a &lt;code&gt;size&lt;/code&gt; range of 1 to 100, while &lt;code&gt;page&lt;/code&gt; determines which results page to retrieve.&lt;/p&gt;

&lt;p&gt;For example, you can request 100 results using:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;keyword=laravel seo
page=1
size=100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;However, requesting 100 results doesn't guarantee that every query will return 100 organic results. The number actually returned depends on the available search data and endpoint behavior.&lt;/p&gt;

&lt;p&gt;If you choose to retrieve 10 results per request, you can paginate through successive pages:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Page 1: Results 1–10
Page 2: Results 11–20
Page 3: Results 21–30
...
Page 10: Results 91–100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If every page returns 10 results, this approach requires approximately 10 requests to collect 100 results.&lt;/p&gt;

&lt;p&gt;The API documentation specifies pagination and a maximum retrieval depth of 500 results per keyword. Check the current documentation for endpoint-specific details before designing a large-scale collection pipeline.&lt;/p&gt;

&lt;h3&gt;
  
  
  A Practical Pagination Strategy
&lt;/h3&gt;

&lt;p&gt;Rather than assuming every request returns the requested number of results, build your pagination loop around the actual response:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Request the first page.&lt;/li&gt;
&lt;li&gt;Extract the returned organic results.&lt;/li&gt;
&lt;li&gt;Store or process the results.&lt;/li&gt;
&lt;li&gt;Continue to the next page while more results are needed.&lt;/li&gt;
&lt;li&gt;Stop when the target is reached, the response contains no more results, or a configured limit is reached.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Also consider deduplicating results by URL. Search results can overlap or change between requests, particularly when collection takes place over an extended period.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Integrating Google Search API with Laravel
&lt;/h2&gt;

&lt;p&gt;If you're developing with Laravel, its HTTP client provides a convenient way to call external APIs.&lt;/p&gt;

&lt;p&gt;First, add your token to the &lt;code&gt;.env&lt;/code&gt; file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SERPSCRAPER_TOKEN=your_api_token
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then expose the value through &lt;code&gt;config/services.php&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="s1"&gt;'serpscraper'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="s1"&gt;'token'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;env&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'SERPSCRAPER_TOKEN'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;],&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can now make a request from your Laravel service:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="kn"&gt;use&lt;/span&gt; &lt;span class="nc"&gt;Illuminate\Support\Facades\Http&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nv"&gt;$response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Http&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'https://serpscraper.dev/api/v1/webs-search'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s1"&gt;'keyword'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'laravel seo'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'gl'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'US'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'hl'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'en'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'page'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'size'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'token'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;config&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'services.serpscraper.token'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;]);&lt;/span&gt;

&lt;span class="nv"&gt;$response&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="k"&gt;throw&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="nv"&gt;$payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$response&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="nv"&gt;$items&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;data_get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'data.items'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[]);&lt;/span&gt;

&lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$items&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$item&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nv"&gt;$position&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;data_get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$item&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'position'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nv"&gt;$title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;data_get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$item&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'title'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nv"&gt;$url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;data_get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$item&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'link'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

    &lt;span class="c1"&gt;// Save or process the result.&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This example demonstrates the integration pattern using the documented response structure. Verify the returned fields against the endpoint response when implementing your application.&lt;/p&gt;

&lt;p&gt;A few production considerations are worth keeping in mind:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Store API credentials in environment configuration or a secret manager.&lt;/li&gt;
&lt;li&gt;Never expose your API token in public frontend code.&lt;/li&gt;
&lt;li&gt;Set reasonable HTTP timeouts.&lt;/li&gt;
&lt;li&gt;Handle non-success HTTP responses.&lt;/li&gt;
&lt;li&gt;Retry transient failures with exponential backoff.&lt;/li&gt;
&lt;li&gt;Log request metadata without logging secret tokens.&lt;/li&gt;
&lt;li&gt;Respect your account's quota and rate limits.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you prefer a client library, SerpScraper.dev also documents a PHP SDK.&lt;/p&gt;

&lt;p&gt;Install it with Composer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;composer require tienphat/serpapi-php
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The library provides a PHP-oriented interface for making search requests. Refer to the &lt;a href="https://serpscraper.dev/docs" rel="noopener noreferrer"&gt;official documentation&lt;/a&gt; for the current SDK usage examples.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Estimating API Usage and Costs
&lt;/h2&gt;

&lt;p&gt;Before launching a rank tracker or a large search data pipeline, estimate your expected request volume.&lt;/p&gt;

&lt;p&gt;Consider an application that tracks 1,000 keywords daily.&lt;/p&gt;

&lt;p&gt;If you request one page per keyword, that means 1,000 API requests per day. If you collect 10 pages for every keyword, the volume increases to 10,000 requests per day.&lt;/p&gt;

&lt;p&gt;Your actual credit consumption also depends on how many results each request returns and the applicable credit rules.&lt;/p&gt;

&lt;p&gt;According to the current Web Search API documentation, standard web search consumes one credit per 10 results returned, rounded up. For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Results returned&lt;/th&gt;
&lt;th&gt;Credit usage&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;1 credit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;5 credits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;td&gt;10 credits&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Empty pages do not consume credits under the documented rules. Review the &lt;a href="https://serpscraper.dev/pricing" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; for current plan allowances and any endpoint-specific differences.&lt;/p&gt;

&lt;p&gt;For production workloads, monitor:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Total credits consumed&lt;/li&gt;
&lt;li&gt;Successful and failed requests&lt;/li&gt;
&lt;li&gt;Results returned per query&lt;/li&gt;
&lt;li&gt;Average response time&lt;/li&gt;
&lt;li&gt;Retry frequency&lt;/li&gt;
&lt;li&gt;Cost per keyword or collection batch&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These metrics help you estimate the actual cost of your application instead of relying on request counts alone.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Beyond Google Web Search
&lt;/h2&gt;

&lt;p&gt;Organic search results are only one part of the search data ecosystem.&lt;/p&gt;

&lt;p&gt;Depending on your application, you may also need:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Image Search API&lt;/strong&gt; for image discovery and metadata&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;News Search API&lt;/strong&gt; for news monitoring&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shopping Search API&lt;/strong&gt; for product research&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scholar Search API&lt;/strong&gt; for academic research workflows&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Maps Search API&lt;/strong&gt; for local business data&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Autocomplete&lt;/strong&gt; for keyword suggestions&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google Trends data&lt;/strong&gt; for trend monitoring&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Webpage extraction&lt;/strong&gt; for structured page content&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;SerpScraper.dev provides multiple search and data intelligence endpoints through its API platform.&lt;/p&gt;

&lt;p&gt;Explore the &lt;a href="https://serpscraper.dev/" rel="noopener noreferrer"&gt;available APIs&lt;/a&gt; to see which endpoints match your application's requirements.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Final Thoughts
&lt;/h2&gt;

&lt;p&gt;Building a search-powered application is much easier when data collection is separated from your core business logic.&lt;/p&gt;

&lt;p&gt;Instead of maintaining a custom scraper, managing proxies, and parsing raw HTML yourself, you can use a SERP API to retrieve structured search data and focus on the features that matter to your users.&lt;/p&gt;

&lt;p&gt;For PHP and Laravel developers, the integration can be as simple as an HTTP request, a JSON response, and a service class that handles pagination, errors, and storage.&lt;/p&gt;

&lt;p&gt;If you're building an SEO tool, rank tracker, market research platform, or AI application, you can explore the &lt;a href="https://serpscraper.dev/web-search-api" rel="noopener noreferrer"&gt;Google Search API on SerpScraper.dev&lt;/a&gt;, review the &lt;a href="https://serpscraper.dev/docs" rel="noopener noreferrer"&gt;API documentation&lt;/a&gt;, and start testing with the available free credits.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What are you building with search data?&lt;/strong&gt; If you're working on a rank tracker, an SEO automation workflow, or an AI-powered research tool, share your use case and the challenges you're trying to solve.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>api</category>
      <category>php</category>
      <category>seo</category>
    </item>
    <item>
      <title>Build a Keyword and Backlink Report in Python With an SEO API</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Thu, 15 Oct 2026 08:46:33 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/build-a-keyword-and-backlink-report-in-python-with-an-seo-api-5cjg</link>
      <guid>https://dev.to/serpscraperdev/build-a-keyword-and-backlink-report-in-python-with-an-seo-api-5cjg</guid>
      <description>&lt;p&gt;If you build marketing tools, you eventually need the same three numbers over and over: &lt;strong&gt;how many people search a keyword, what it is worth (CPC), and how strong a competitor's backlink profile is&lt;/strong&gt;. Keyword research tools show them in a dashboard, but copying numbers out of a dashboard does not scale.&lt;/p&gt;

&lt;p&gt;An &lt;strong&gt;SEO API&lt;/strong&gt; returns the same data as JSON, so a script can fetch it on a schedule. In this post you will build a small report that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Takes a list of keywords and returns search volume and CPC for each.&lt;/li&gt;
&lt;li&gt;Takes a list of competitor domains and returns a backlink summary for each.&lt;/li&gt;
&lt;li&gt;Writes everything to CSV files you can open in a spreadsheet or feed to a dashboard.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What you need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Python 3 and the &lt;code&gt;requests&lt;/code&gt; package&lt;/li&gt;
&lt;li&gt;An API key, stored in the &lt;code&gt;AXIOAPI_KEY&lt;/code&gt; environment variable (sign up from the &lt;a href="https://axioapi.com/apis/seo" rel="noopener noreferrer"&gt;SEO API page&lt;/a&gt;; new accounts receive free credits after verification)
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;API&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://axioapi.com/api/v1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;HEADERS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Authorization&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Bearer &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;AXIOAPI_KEY&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Accept&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;application/json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 1: a keyword data API call
&lt;/h2&gt;

&lt;p&gt;The keyword endpoint accepts up to 10 keywords and a country code, and bills &lt;strong&gt;once per request&lt;/strong&gt;, so always send full batches of 10 instead of one keyword at a time.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;keyword_metrics&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;keywords&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;country&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;us&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/seo/keywords/metrics&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;HEADERS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;keywords&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;keywords&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;country&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;country&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;batches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;keyword_report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;keywords&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;keywords.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;country&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;us&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;batches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;keywords&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;extend&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;keyword_metrics&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;country&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

    &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;search_volume&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;keyword&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;search_volume&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cpc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;keyword&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;search_volume&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cpc&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details worth knowing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Volume depends on the country.&lt;/strong&gt; Always pass the market you actually sell in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Missing metrics come back empty, not guessed.&lt;/strong&gt; That is why the sort key uses &lt;code&gt;or 0&lt;/code&gt;. A blank volume means the data source had no figure, not that nobody searches the keyword.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A high CPC is a quick signal of commercial intent, which helps when you pick between two keywords with similar volume.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: a backlink API call for competitors
&lt;/h2&gt;

&lt;p&gt;Next, a backlink summary for each competitor. The response lists the figures of &lt;strong&gt;every data source that answered&lt;/strong&gt; and says which ones did not, so you can tell a complete result from a partial one.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;backlink_summary&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/seo/domains/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/backlinks&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;HEADERS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;backlink_report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domains&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;backlinks.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;domain&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;metrics&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partial&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;missing_sources&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;domain&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;domains&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;backlink_summary&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;missing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;missing&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sources&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
                &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partial&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;missing&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Writing one row per source keeps the disagreement between sources visible. Backlink counts are &lt;strong&gt;estimates&lt;/strong&gt; and different crawlers will not agree, so treat them as a way to compare sites, not as exact totals.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: run it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;keyword_report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;seo api&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;backlink checker&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;keyword research tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;domain authority&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="n"&gt;country&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;us&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;backlink_report&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;example.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;competitor.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Wrote keywords.csv and backlinks.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it with &lt;code&gt;python report.py&lt;/code&gt;, and put it in a cron job or a scheduled CI workflow to get a fresh CSV every week.&lt;/p&gt;

&lt;h2&gt;
  
  
  Make it a recurring domain report
&lt;/h2&gt;

&lt;p&gt;A one-off CSV answers "where do we stand today". To answer "are we growing", add the domain overview and 12 months of history for the same domain:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}{&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;HEADERS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;domain_report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;overview&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/seo/domains/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;country&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;US&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;backlinks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/seo/domains/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/backlinks&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;history&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/seo/domains/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/history&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;months&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;domain&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;history&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;incomplete&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;overview&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partial&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;backlinks&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;partial&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;missing_sources&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;overview&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;missing&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;backlinks&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;missing&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])),&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Store each run (a JSON file, SQLite, or a spreadsheet tab) and compare it with the previous one. Flag any run where &lt;code&gt;incomplete&lt;/code&gt; is true instead of silently charting partial data. There is a longer walkthrough in &lt;a href="https://axioapi.com/guides/seo-report-api-domain-ranking" rel="noopener noreferrer"&gt;the SEO report API guide&lt;/a&gt; and a focused one on &lt;a href="https://axioapi.com/guides/keyword-data-api-volume-cpc" rel="noopener noreferrer"&gt;pulling search volume and CPC in code&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep it reliable and cheap
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Batch keywords.&lt;/strong&gt; 10 per call is the maximum, and it is billed once per call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cache results.&lt;/strong&gt; Search volume moves slowly. Re-fetching the same keyword daily wastes credits, weekly is plenty.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Handle partial data.&lt;/strong&gt; If &lt;code&gt;partial&lt;/code&gt; is true, retry later or note the gap in your report.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Respect the data.&lt;/strong&gt; These are third-party estimates. Do not promise a client an exact number.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  When this approach is a good fit, and when it is not
&lt;/h2&gt;

&lt;p&gt;It fits when you want keyword and link data &lt;strong&gt;inside your own product or script&lt;/strong&gt;: an internal dashboard, a client report generator, a content planning tool, or a quick competitor check.&lt;/p&gt;

&lt;p&gt;It is not a replacement for everything. There are no live SERP, rank tracking or local pack endpoints, so if you need those, a SERP-focused provider is the better choice. I wrote a plain comparison here: &lt;a href="https://axioapi.com/guides/dataforseo-api-alternative" rel="noopener noreferrer"&gt;an SEO API alternative to DataForSEO and Semrush&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If you only need to check one domain by hand, the &lt;a href="https://axioapi.com/tools/backlink-checker" rel="noopener noreferrer"&gt;free backlink checker&lt;/a&gt; needs no code and no account for a few runs per day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wrap-up
&lt;/h2&gt;

&lt;p&gt;An SEO API turns keyword research and backlink checks into something you can schedule, version and test like any other code. The script above is about 60 lines and covers the three numbers most marketing tools need.&lt;/p&gt;

&lt;p&gt;If you build something with it, tell me in the comments what you would add: rank history, content gap analysis, or alerts when a competitor gains links. Full endpoint list and prices are on the &lt;a href="https://axioapi.com/apis/seo" rel="noopener noreferrer"&gt;SEO API page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>python</category>
      <category>seo</category>
      <category>api</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>SERP API solutions for data scraping in 2026</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Mon, 21 Sep 2026 04:21:18 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/serp-api-solutions-for-data-scraping-in-2026-20dp</link>
      <guid>https://dev.to/serpscraperdev/serp-api-solutions-for-data-scraping-in-2026-20dp</guid>
      <description>&lt;p&gt;Managing a custom search engine scraper in 2026 is often a case of diminishing returns. After a decade of building data pipelines, I’ve seen countless engineering teams waste hundreds of hours fighting CAPTCHAs, managing proxy rotations, and patching broken regex parsers whenever a search engine tweaks its DOM. If you’re still building your own infrastructure, you’re likely spending 80% of your time on maintenance rather than data extraction.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why DIY Scrapers Fail at Scale
&lt;/h3&gt;

&lt;p&gt;Modern anti-bot systems have evolved beyond simple rate-limiting. They now analyze TCP/IP fingerprints, TLS handshakes, and behavioral signals to detect headless environments like Puppeteer or Playwright. When you attempt to scrape at scale, you hit three walls:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;The CAPTCHA Barrier:&lt;/strong&gt; Automated pattern detection triggers visual challenges that are notoriously difficult to bypass at scale.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Fingerprinting:&lt;/strong&gt; Search engines cross-reference your header, cookie, and canvas settings to identify synthetic traffic.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;IP Reputation:&lt;/strong&gt; Datacenter IPs are effectively blacklisted, forcing you to pay premium prices for reliable residential proxy pools.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  The Shift to Managed Extraction
&lt;/h3&gt;

&lt;p&gt;Modern infrastructure providers abstract the request lifecycle entirely. They handle the browser emulation, rotation, and parsing, returning a clean JSON payload. This keeps your pipeline stable, as the API provider absorbs the "breakage" whenever search engine layouts change.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Economics of Scale
&lt;/h3&gt;

&lt;p&gt;When scaling to millions of queries, your choice of pricing model is critical.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Pay-as-you-go:&lt;/strong&gt; Ideal for fluctuating volume. You avoid the "subscription trap" where you lose 40% of your prepaid credits because your traffic didn't hit the projected monthly threshold.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Subscription:&lt;/strong&gt; Generally lower unit costs, but requires predictable traffic to ensure ROI. &lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Be wary of hidden costs. Many providers charge a multiplier if a query requires JavaScript rendering to display interactive maps or dynamic SERP features. If you are budget-conscious, always calculate the "cost per successful request" rather than just the base price per query.&lt;/p&gt;

&lt;h3&gt;
  
  
  Performance Benchmarks (2026)
&lt;/h3&gt;

&lt;p&gt;Choosing a provider often comes down to the trade-off between latency and data depth. Here is how the top-tier solutions typically perform under load:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Provider&lt;/th&gt;
&lt;th&gt;Latency&lt;/th&gt;
&lt;th&gt;Focus&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;DataForSEO&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~1200ms&lt;/td&gt;
&lt;td&gt;High-volume, cost-effective batch processing.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Nimble&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~800ms&lt;/td&gt;
&lt;td&gt;Real-time speed and premium residential targeting.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;SerpApi&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~1500ms&lt;/td&gt;
&lt;td&gt;Complex visual parsing and structured extraction.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Zenserp&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~950ms&lt;/td&gt;
&lt;td&gt;Rapid integration and low-friction setup.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Localization and Compliance
&lt;/h3&gt;

&lt;p&gt;If you need city or zip-code level accuracy, ensure your provider uses legitimate residential IPs. Simply "simulating" coordinates is no longer enough; modern search engines verify the request origin via actual geo-located nodes. &lt;/p&gt;

&lt;p&gt;From a compliance standpoint, managed services are safer. They naturally space out requests across thousands of residential IPs, mimicking human behavior, which reduces server strain and keeps your integration within standard terms of service.&lt;/p&gt;

&lt;h3&gt;
  
  
  Final Recommendation
&lt;/h3&gt;

&lt;p&gt;If your project requires high-frequency data, stop maintaining your own headless browsers. The engineering hours saved by offloading proxy management and parser maintenance will almost always outweigh the cost of an API subscription. Focus your development team on what happens &lt;em&gt;after&lt;/em&gt; the data is received—the actual analysis—rather than the tedious work of keeping a scraper alive.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/serp-api-solutions-for-data-scraping-in-2026" rel="noopener noreferrer"&gt;SERP API solutions for data scraping in 2026&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Google serp api: Technical infrastructure guide for 2026</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Sun, 20 Sep 2026 06:20:21 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/google-serp-api-technical-infrastructure-guide-for-2026-5aof</link>
      <guid>https://dev.to/serpscraperdev/google-serp-api-technical-infrastructure-guide-for-2026-5aof</guid>
      <description>&lt;p&gt;After a decade of managing data pipelines, I have learned one hard lesson: building your own scrapers for search data is a trap. If you are still rotating proxies and fixing broken CSS selectors, you aren't an engineer—you are a full-time maintainer of a legacy system that will inevitably fail.&lt;/p&gt;

&lt;p&gt;By 2026, the complexity of search result pages—especially with the rise of AI-generated summaries—has made DIY scraping economically and technically unviable. Here is how you should handle search data at scale.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Problem with DIY Parsing
&lt;/h3&gt;

&lt;p&gt;Google’s DOM is fluid. Elements shift, new ad formats appear, and class names change daily. When you write custom XPath or CSS selectors, you are building on sand. A professional approach involves moving to managed services that decouple the frontend from your application. Instead of raw HTML, you should be consuming normalized JSON schemas. This shift alone eliminates the need for constant maintenance and ensures your downstream analytics remain consistent.&lt;/p&gt;

&lt;h3&gt;
  
  
  Infrastructure: Proxies and Captchas
&lt;/h3&gt;

&lt;p&gt;The primary barrier to reliable data extraction is IP reputation. If you are managing your own pool, you are likely battling blacklists and CAPTCHAs. Managed providers solve this by:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Residential Proxy Networks:&lt;/strong&gt; Routing traffic to mimic genuine human behavior.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Automated Fingerprinting:&lt;/strong&gt; Spoofing device headers to bypass anti-bot systems.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Geolocation:&lt;/strong&gt; Simulating localized search contexts, which is critical for accurate rank tracking.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Offloading this reputational risk is the single best way to stabilize your request success rate.&lt;/p&gt;

&lt;h3&gt;
  
  
  Extracting Modern Search Features
&lt;/h3&gt;

&lt;p&gt;Standard scrapers often miss the "new" web—AI Overviews, local packs, and rich snippets. These components require specialized parsing logic that most internal libraries cannot handle. Today’s professional APIs are engineered to turn these complex, non-linear blocks into structured JSON. If your architecture isn't built to capture these features, your data is incomplete.&lt;/p&gt;

&lt;h3&gt;
  
  
  Integrating with LLM Pipelines
&lt;/h3&gt;

&lt;p&gt;If you are building RAG (Retrieval-Augmented Generation) applications, search data is your ground truth. The most efficient workflow I have implemented involves:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; Fetching structured JSON via API.&lt;/li&gt;
&lt;li&gt; Filtering and condensing the content to save tokens.&lt;/li&gt;
&lt;li&gt; Injecting the clean data into your LLM context window.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Modern providers even offer webhooks, allowing your AI agents to trigger updates based on real-time search trends rather than relying on manual polling.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Hidden Cost of "Free"
&lt;/h3&gt;

&lt;p&gt;When calculating the TCO (Total Cost of Ownership), most teams forget the engineering hours required to maintain in-house scrapers. Once you exceed 5,000 requests per month, a managed API is almost always cheaper than the salary of an engineer dedicated to keeping a custom scraper alive. Look for providers that charge only for successful results—this aligns their incentives with your success.&lt;/p&gt;

&lt;h3&gt;
  
  
  Final Thoughts
&lt;/h3&gt;

&lt;p&gt;If you are still managing your own infrastructure for search data, you are likely accruing massive technical debt. Moving to a professional extraction layer isn't just about saving time; it’s about ensuring your systems don't collapse when the search interface changes. For those looking to modernize, check out the documentation at &lt;a href="https://serpscraper.dev" rel="noopener noreferrer"&gt;serpscraper.dev&lt;/a&gt; to see how clean, structured search data can streamline your development workflow. Stop building the infrastructure and start building the product.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/google-serp-api-technical-infrastructure-guide-for-2026" rel="noopener noreferrer"&gt;Google serp api: Technical infrastructure guide for 2026&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Best news api for sentiment analysis: a 2026 data pipeline guide</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Sat, 19 Sep 2026 11:21:52 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/best-news-api-for-sentiment-analysis-a-2026-data-pipeline-guide-9im</link>
      <guid>https://dev.to/serpscraperdev/best-news-api-for-sentiment-analysis-a-2026-data-pipeline-guide-9im</guid>
      <description>&lt;p&gt;Over the last few years of building financial and news data pipelines, I've seen many engineering teams make the same expensive mistake: purchasing a premium, pre-scored sentiment API assuming it will solve their NLP needs out of the box, only to realize the "off-the-shelf" scores fail miserably on industry-specific jargon. &lt;/p&gt;

&lt;p&gt;When you outsource scoring, you inherit a "black box" problem. For example, the word &lt;em&gt;correction&lt;/em&gt; might be neutral in a general news feed, but in a market-facing pipeline, it signals a bearish trend. &lt;/p&gt;

&lt;p&gt;Here is my engineering guide to navigating these architecture trade-offs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Raw Text vs. Pre-Scored APIs
&lt;/h2&gt;

&lt;p&gt;Your choice boils down to development speed versus domain control:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Pre-scored APIs:&lt;/strong&gt; Best for rapid prototyping. You get instant polarity labels but are locked into the provider's default vocabulary and weighting.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Raw Text APIs:&lt;/strong&gt; Best for high-accuracy production systems. You ingest raw text and run it through your own NLP stack (e.g., custom fine-tuned RoBERTa models on Hugging Face). While this increases operational overhead, the latency of running local inference on a GPU is often lower than hitting a remote API for every single news item.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Handling High-Volume Ingestion in Python
&lt;/h2&gt;

&lt;p&gt;When building real-time ingestion workers, blocking your main event loop is a production killer. I always use asynchronous execution paired with strict schema validation.&lt;/p&gt;

&lt;p&gt;Here is the boilerplate pattern I rely on using &lt;code&gt;asyncio&lt;/code&gt;, &lt;code&gt;httpx&lt;/code&gt;, and &lt;code&gt;Pydantic&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;httpx&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pydantic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ArticleSchema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;published_at&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;alias&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;publishedAt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch_news&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;httpx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AsyncClient&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="c1"&gt;# Validate immediately at the ingestion boundary
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;ArticleSchema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;article&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;article&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;articles&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# Implement exponential backoff here
&lt;/span&gt;        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Error fetching data: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Using Pydantic at your pipeline's entry point ensures that any upstream API schema changes don't corrupt your database or downstream ML models.&lt;/p&gt;

&lt;h2&gt;
  
  
  Defeating the "Dictionary Trap"
&lt;/h2&gt;

&lt;p&gt;Many legacy sentiment APIs rely on dictionary-based lookup tables. This fails on three core linguistic fronts:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Negation Detection:&lt;/strong&gt; Failing to understand that "not bad" is positive.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Contextual Polarity:&lt;/strong&gt; Misinterpreting words like "volatile," which could be highly positive for market makers but negative for long-term investors.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Multilingual Nuance:&lt;/strong&gt; Translating literally instead of using native Transformer models.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Before committing to an API, run a manual validation set of 1,000 domain-specific headlines to baseline their accuracy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Eliminating Latency and Backtesting Bias
&lt;/h2&gt;

&lt;p&gt;For real-time pipelines, standard REST polling is too slow. Look for providers offering dedicated WebSocket feeds. The delta between "time-of-publish" and "time-of-access" must be minimized; if your system processes a sentiment signal minutes after publication, the alpha is already gone.&lt;/p&gt;

&lt;p&gt;Furthermore, if you are backtesting historical strategies, ensure your provider offers &lt;strong&gt;point-in-time snapshots&lt;/strong&gt;. Using historical data that has been retroactively updated or corrected introduces &lt;em&gt;look-ahead bias&lt;/em&gt;, rendering your backtesting results completely unreliable.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/best-news-api-for-sentiment-analysis-a-2026-data-pipeline-guide" rel="noopener noreferrer"&gt;Best news api for sentiment analysis: a 2026 data pipeline guide&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Finding a cheap search volume api that actually works</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Fri, 18 Sep 2026 02:30:24 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/finding-a-cheap-search-volume-api-that-actually-works-2ehl</link>
      <guid>https://dev.to/serpscraperdev/finding-a-cheap-search-volume-api-that-actually-works-2ehl</guid>
      <description>&lt;p&gt;When I started building my first SEO tool, the biggest bottleneck wasn't the UI or the stack—it was the recurring cost of retrieving accurate search metrics. Many developers fall into the trap of overpaying for enterprise-grade solutions when lighter, more efficient alternatives are available.&lt;/p&gt;

&lt;h3&gt;
  
  
  Selecting the Right Billing Model
&lt;/h3&gt;

&lt;p&gt;For startups or side projects, I always lean toward &lt;strong&gt;pay-as-you-go&lt;/strong&gt; models. Tiered subscriptions are seductive but often lead to wasted capital during low-traffic months. The goal is to align your overhead directly with user activity. &lt;/p&gt;

&lt;p&gt;Before signing up, verify if the provider offers &lt;strong&gt;custom hard limits&lt;/strong&gt;. This is crucial to prevent "billing shocks" where an unexpected traffic spike—or a rogue script—consumes your entire monthly budget in minutes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data Granularity: Why You Should Avoid Defaults
&lt;/h3&gt;

&lt;p&gt;The official Google Ads API is often surprisingly restrictive. It tends to bundle similar queries into broad "buckets," effectively hiding the long-tail keywords that actually convert. &lt;/p&gt;

&lt;p&gt;In my experience, third-party providers that leverage &lt;strong&gt;clickstream data&lt;/strong&gt; offer a distinct competitive advantage. They de-cluster this data, allowing you to present "hidden" long-tail phrases to your users. If you want your tool to stand out, stop serving the same generic, aggregate metrics everyone else is using. Give your users the granular, non-clustered insights they can’t find in standard reports.&lt;/p&gt;

&lt;h3&gt;
  
  
  Evaluating Technical Integration
&lt;/h3&gt;

&lt;p&gt;If a provider’s documentation is a maze, run away. I look for three non-negotiables:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Bulk Endpoint Support:&lt;/strong&gt; If you’re making single requests for thousands of keywords, your latency will be unusable and your costs will explode.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Environments:&lt;/strong&gt; Don't pay to test your integration. A developer-friendly provider will always offer a free sandbox mode.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Standardized Status Codes:&lt;/strong&gt; The presence of proper HTTP codes (429 for rate limits, 400 for bad payloads) tells me the engineering team behind the API actually knows how to build for developers.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Strategies for Scaling
&lt;/h3&gt;

&lt;p&gt;I’ve found that the best way to maintain a sustainable margin is to treat the API as a commodity. Don't build your core business logic around one provider's specific JSON structure. Instead, implement a &lt;strong&gt;proxy pattern&lt;/strong&gt; in your codebase.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Middleware Abstraction:&lt;/strong&gt; Create an internal layer that translates your application's needs into the provider's format. This makes swapping providers effortless if your current partner raises prices or their data quality dips.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Aggressive Caching:&lt;/strong&gt; Implement a Time-To-Live (TTL) cache for your keyword data. There is no reason to ping the API twice for the same search term in a 30-day window. Serving results from your own database is the fastest way to slash costs and improve response times.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Summary Comparison Table
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Feature&lt;/th&gt;
&lt;th&gt;Tiered Subscription&lt;/th&gt;
&lt;th&gt;Pay-As-You-Go&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Best For&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Predictable, high volume&lt;/td&gt;
&lt;td&gt;Startups, variable traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Main Risk&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Paying for idle credits&lt;/td&gt;
&lt;td&gt;Uncapped costs during spikes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Efficiency&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Stable monthly budget&lt;/td&gt;
&lt;td&gt;Direct cost-per-user correlation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Building a high-quality tool requires balancing data accuracy with infrastructure costs. By abstracting your data layer and prioritizing bulk-processing capabilities, you can keep your overhead low while delivering a premium experience. Always test with a small volume first to ensure the data matches your expectations before scaling up your production load.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/finding-a-cheap-search-volume-api-that-actually-works" rel="noopener noreferrer"&gt;Finding a cheap search volume api that actually works&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>How to scrape Google search results without getting blocked 2026</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Thu, 17 Sep 2026 05:01:02 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/how-to-scrape-google-search-results-without-getting-blocked-2026-1loh</link>
      <guid>https://dev.to/serpscraperdev/how-to-scrape-google-search-results-without-getting-blocked-2026-1loh</guid>
      <description>&lt;p&gt;I’ve spent years building and maintaining data collection pipelines, and the most common pitfall I see developers fall into is assuming bot-detection is just about IP reputation. It isn't. Modern security infrastructure performs a deep packet inspection on your network handshake, HTTP headers, and client footprint simultaneously. If you point standard Selenium or a vanilla HTTP client at a search engine's results page, you are practically begging to be flagged.&lt;/p&gt;

&lt;p&gt;To bypass these blocks reliably, you have to transition from brute-force request flooding to precise client emulation. Here is how to configure a highly resilient scraper from scratch.&lt;/p&gt;

&lt;h3&gt;
  
  
  The TLS Fingerprint Bottleneck
&lt;/h3&gt;

&lt;p&gt;Even if you rotate your User-Agent header perfectly, standard client libraries like Python's &lt;code&gt;requests&lt;/code&gt; or Node's &lt;code&gt;axios&lt;/code&gt; will be blocked. Why? Because of &lt;strong&gt;TLS Fingerprinting&lt;/strong&gt;. &lt;/p&gt;

&lt;p&gt;During the TLS handshake, your client advertises its supported cipher suites, extension lists, and SSL configurations. Modern anti-bot systems keep a database of these signatures. If your headers say "Chrome" but your TLS signature says "Python requests", the server terminates the session immediately.&lt;/p&gt;

&lt;p&gt;To solve this, use client libraries capable of &lt;strong&gt;TLS Impersonation&lt;/strong&gt;. For Python, I highly recommend using &lt;code&gt;curl_cffi&lt;/code&gt; instead of traditional HTTP libraries. It mimics the exact TLS handshakes of modern browsers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Mimicking a legitimate Chrome client using curl_cffi
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;curl_cffi&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.google.com/search?q=web+scraping+best+practices&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;impersonate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chrome110&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Status Code: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Routing Traffic via Residential Proxies
&lt;/h3&gt;

&lt;p&gt;Using datacenter IPs for high-frequency scraping is a waste of resources. Datacenter subnets are easily flagged and blacklisted in bulk. &lt;/p&gt;

&lt;p&gt;You must route your requests through &lt;strong&gt;residential proxies&lt;/strong&gt;. These proxies route traffic through residential Internet Service Providers (ISPs), assigning your traffic the same reputation score as an actual home internet user.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Static/Datacenter Proxies:&lt;/strong&gt; Highly detectable. Best used only for internal testing.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Rotating Residential Proxies:&lt;/strong&gt; Rotates your IP address with each request or session. Mandatory for production-scale data mining.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Simulating Human Behavior (Pacing and Jitter)
&lt;/h3&gt;

&lt;p&gt;Sending requests at exact 1.0-second intervals is a dead giveaway for behavioral anomaly detection. To bypass detection, your scraper's traffic patterns must look organic:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Introduce Jitter:&lt;/strong&gt; Add randomized delays between requests. Instead of a fixed pause, use a random interval:&lt;br&gt;
&lt;/p&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;

&lt;span class="c1"&gt;# Pause between 2 to 7 seconds
&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;uniform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;2.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;7.0&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Set Concurrency Limits:&lt;/strong&gt; Keep your concurrent connections low (ideally under 5 simultaneous requests per proxy IP) to avoid triggering rate-limits.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Custom Scrapers vs. Managed APIs
&lt;/h3&gt;

&lt;p&gt;Before building out your own infrastructure, consider the maintenance overhead. Anti-bot heuristics change constantly, meaning your custom scripts will require weekly or monthly maintenance.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Custom Infrastructure&lt;/th&gt;
&lt;th&gt;Managed SERP APIs&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Maintenance&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;High (constantly fixing blocks)&lt;/td&gt;
&lt;td&gt;Zero (handled by provider)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Reliability&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Variable&lt;/td&gt;
&lt;td&gt;&amp;gt;99% success rate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Setup Cost&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;High (engineering hours)&lt;/td&gt;
&lt;td&gt;Predictable (pay-per-request)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For small or experimental projects, a custom &lt;code&gt;curl_cffi&lt;/code&gt; stack with a basic residential proxy pool works perfectly. However, if your production pipeline relies on consistent, high-volume search engine data, offloading the browser emulation, proxy management, and CAPTCHA solving to a dedicated API will save you substantial development hours.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/how-to-scrape-google-search-results-without-getting-blocked-2026" rel="noopener noreferrer"&gt;How to scrape Google search results without getting blocked 2026&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>How to get search volume from Google API without paying</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Wed, 16 Sep 2026 02:16:31 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/how-to-get-search-volume-from-google-api-without-paying-31d2</link>
      <guid>https://dev.to/serpscraperdev/how-to-get-search-volume-from-google-api-without-paying-31d2</guid>
      <description>&lt;p&gt;After years of architecting internal search marketing engines, I’ve repeatedly encountered the same engineering challenge: developers trying to fetch raw keyword volume programmatically without racking up massive enterprise API bills. Let's address the technical reality of accessing this data stream, what it actually costs, and how to build a stable integration pipeline.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Technical Reality of Google's Data Pipelines
&lt;/h3&gt;

&lt;p&gt;Many teams look for undocumented endpoints or try scraping the Keyword Planner (GKP) web UI. From my experience, scraping GKP is a high-maintenance trap. Google constantly rotates CSS selectors, implements sophisticated bot detection, and will quickly rate-limit your proxies.&lt;/p&gt;

&lt;p&gt;The only reliable access is via the &lt;strong&gt;Google Ads API&lt;/strong&gt; (specifically the &lt;code&gt;KeywordPlanIdeaService&lt;/code&gt;). While Google does not charge per API request, there is a catch: to retrieve exact, granular search volumes instead of broad, useless ranges (e.g., "10K - 100K"), your linked Google Ads developer account must have an active billing profile and historical campaign spend. If you try to query it on a completely cold, zero-spend account, you will only receive bucketed approximations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Setting Up the Infrastructure
&lt;/h3&gt;

&lt;p&gt;To communicate with the API, you need to establish a secure authorization chain. Here is the blueprint I use:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;GCP Project:&lt;/strong&gt; Create a project in the Google Cloud Console and enable the &lt;strong&gt;Google Ads API&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;OAuth2 Credentials:&lt;/strong&gt; Configure your consent screen and generate a &lt;code&gt;Client ID&lt;/code&gt; and &lt;code&gt;Client Secret&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Developer Token:&lt;/strong&gt; Apply for a token via your Google Ads Manager Account (MCC). Note that basic access is typically approved quickly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Billing Link:&lt;/strong&gt; Ensure your Google Ads account has an active billing method and a history of small ad runs.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Programmatic Execution with Python
&lt;/h3&gt;

&lt;p&gt;Do not try to write raw REST requests. The Google Ads API relies heavily on gRPC and Protobuf serialization. Always use the official client library.&lt;/p&gt;

&lt;p&gt;Here is a conceptual architecture of how I structure our keyword fetch services:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google.ads.googleads.client&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;GoogleAdsClient&lt;/span&gt;

&lt;span class="c1"&gt;# Load credentials securely from configuration
&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;GoogleAdsClient&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_from_storage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;google-ads.yaml&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;keyword_service&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_service&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;KeywordPlanIdeaService&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Define your request payload
&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;GenerateKeywordIdeasRequest&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;YOUR_ADS_CUSTOMER_ID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keyword_plan_network&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;enums&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;KeywordPlanNetworkEnum&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GOOGLE_SEARCH&lt;/span&gt;

&lt;span class="c1"&gt;# Add seed keywords for volume retrieval
&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keyword_seed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keywords&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;extend&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;api integration&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;backend performance&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="c1"&gt;# Execute request (simplified)
# response = keyword_service.generate_keyword_ideas(request=request)
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Scaling &amp;amp; Rate Limit Mitigation
&lt;/h3&gt;

&lt;p&gt;When processing keywords at scale, you will quickly hit API quotas. To avoid &lt;code&gt;429 Too Many Requests&lt;/code&gt; exceptions, implement these two strategies in your code:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Batching:&lt;/strong&gt; Don't query keywords one by one. Group up to 1,000 keywords into a single request. This drastically reduces network overhead and token usage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exponential Backoff:&lt;/strong&gt; Wrap your API calls in a retry loop using a backoff algorithm to handle temporary throttling gracefully.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Understanding the Data Output
&lt;/h3&gt;

&lt;p&gt;Finally, remember that Google does not return precise, real-time counters. The metrics are 12-month averages mapped to roughly 80 logarithmic bands. Treat this data as relative search intent rather than exact transactional logs. Ensure your data pipeline caches these results locally to minimize redundant API calls and stay within your developer token's rate limits.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/how-to-get-search-volume-from-google-api-without-paying" rel="noopener noreferrer"&gt;How to get search volume from Google API without paying&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Cheapest Google search API alternatives for developers</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Tue, 15 Sep 2026 07:21:28 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/cheapest-google-search-api-alternatives-for-developers-36je</link>
      <guid>https://dev.to/serpscraperdev/cheapest-google-search-api-alternatives-for-developers-36je</guid>
      <description>&lt;p&gt;If you've been relying on Google's Custom Search JSON API for your projects, it's time to map out your migration strategy. Google has closed signups and set a hard deprecation date for January 1, 2027. Even worse, their recommended replacement, Vertex AI Search, restricts search queries to a maximum of 50 specified domains. If your application requires broad, open-web search data, Vertex simply will not work.&lt;/p&gt;

&lt;p&gt;In my experience building data aggregation pipelines, migrating search infrastructure under a tight deadline is a massive headache. To save you the trial and error, here is how I evaluate budget-friendly web search APIs and what the current landscape looks like for developers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why Google’s native tools fall short
&lt;/h3&gt;

&lt;p&gt;For years, many of us tolerated Google's $5 per 1,000 queries pricing because it was the default. But Google Programmable Search is fundamentally different from a true SERP API. Programmable Search is designed as an internal site-search tool; it cannot scrape organic listings, ads, or rich snippets from the wider web. For competitive intelligence, SEO tools, or LLM retrieval-augmented generation (RAG) pipelines, you need raw, unrestricted SERP data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Key Technical Benchmarks to Evaluate
&lt;/h3&gt;

&lt;p&gt;When auditing alternative providers, I prioritize five core metrics to ensure application stability:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Cost per 1,000 queries:&lt;/strong&gt; Look for scalable, tiered structures. Modern third-party alternatives usually charge a fraction of Google's legacy rate.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Response Latency:&lt;/strong&gt; Consistent sub-500ms latency is the benchmark for real-time applications.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Rate Limits:&lt;/strong&gt; Look for generous concurrent request limits to avoid building complex queuing systems on your backend.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Payload Cleanliness:&lt;/strong&gt; Structured, flat JSON outputs save countless hours of writing and maintaining custom parsers.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Proxy and CAPTCHA Management:&lt;/strong&gt; Choose APIs that handle proxy rotation and CAPTCHAs automatically so you don't have to build scraping infrastructure yourself.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The Modern Alternative Landscape
&lt;/h3&gt;

&lt;p&gt;The market has evolved, and specialized SERP APIs now offer much better pricing and deeper data access. Here is a quick comparison of what you can expect:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Legacy Google JSON API&lt;/th&gt;
&lt;th&gt;Modern SERP APIs (e.g., serpscraper.dev)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cost / 1k Queries&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$5.00&lt;/td&gt;
&lt;td&gt;$0.30 - $3.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Search Scope&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Restricted/Domain-specific&lt;/td&gt;
&lt;td&gt;Full, open-web SERPs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Typical Latency&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&amp;lt; 300ms (site-restricted)&lt;/td&gt;
&lt;td&gt;&amp;lt; 500ms (full web)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Proxy/CAPTCHA Handling&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;N/A&lt;/td&gt;
&lt;td&gt;Built-in&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Developer Optimization Tips
&lt;/h3&gt;

&lt;p&gt;To maximize your ROI with any budget-friendly API, I highly recommend two strategies:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Smart Caching:&lt;/strong&gt; Do not query the API for duplicate terms in tight loops. Implement a Redis caching layer for search queries that do not require real-time updates.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Verify Parsing Quality:&lt;/strong&gt; Ensure your chosen provider accurately parses localized search results and map packs, which are often fragile when scraped manually.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Transitioning away from Google’s ecosystem is actually a massive opportunity to lower your monthly API bill while unlocking richer, unrestricted search data for your applications.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/cheapest-google-search-api-alternatives-for-developers" rel="noopener noreferrer"&gt;Cheapest Google search API alternatives for developers&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>How to scrape Zillow listings without getting blocked</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Mon, 14 Sep 2026 12:14:38 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/how-to-scrape-zillow-listings-without-getting-blocked-aeh</link>
      <guid>https://dev.to/serpscraperdev/how-to-scrape-zillow-listings-without-getting-blocked-aeh</guid>
      <description>&lt;p&gt;Modern real estate data acquisition is a game of cat-and-mouse. When I first started building data pipelines for property analysis, I assumed the official routes would be straightforward. I was wrong. The public API has been effectively dead for years, and the replacement, Bridge Interactive, is gated behind strict MLS credentials and prohibitive costs that lock out most indie developers.&lt;/p&gt;

&lt;p&gt;This leaves web harvesting as the only viable path. However, the platform’s security stack is sophisticated, employing HUMAN Security (formerly PerimeterX) and JA4 TLS fingerprinting to intercept automated traffic before it even loads the DOM.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Security Hurdle
&lt;/h3&gt;

&lt;p&gt;If you are still using standard Python &lt;code&gt;requests&lt;/code&gt; or basic headless browsers without customization, you’re hitting a wall. The server inspects your "Client Hello" packet; if your cipher suite order or TLS extensions don't match a legitimate browser profile, you get blocked immediately.&lt;/p&gt;

&lt;p&gt;The fix isn't just switching &lt;code&gt;User-Agent&lt;/code&gt; strings. You need to align your HTTP/2 frames and TLS signatures to mimic a real Chrome instance. Once I started using a TLS-specialized adapter that aligns with modern browser handshakes, my block rate plummeted from nearly 100% to under 5%.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data Extraction Strategy
&lt;/h3&gt;

&lt;p&gt;Avoid the trap of scraping HTML elements by class name. Zillow frequently obfuscates its DOM, meaning your selectors will break after almost every deployment. Instead, look for the &lt;code&gt;__NEXT_DATA__&lt;/code&gt; script tag.&lt;/p&gt;

&lt;p&gt;This tag contains the entire page state as a structured JSON object. Parsing this is significantly more stable. Your workflow should look like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Fetch the page via a stealth-enabled browser client.&lt;/li&gt;
&lt;li&gt;Locate the &lt;code&gt;__NEXT_DATA__&lt;/code&gt; JSON string.&lt;/li&gt;
&lt;li&gt;Access the data at &lt;code&gt;props.pageProps.searchPageState.cat1.searchResults.mapResults&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Bypassing Pagination Limits
&lt;/h3&gt;

&lt;p&gt;You will notice the platform caps search results at 820 listings per query. To harvest an entire city, I use a quadtree algorithm. By splitting your target geographic coordinates into smaller bounding boxes and recursing until each "tile" contains fewer than 800 items, you can effectively map an entire metro area without missing data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Build vs. Buy
&lt;/h3&gt;

&lt;p&gt;Maintaining your own proxy pool is an expensive and time-consuming endeavor. Residential proxies are pricey, and the engineering hours spent debugging "cat-and-mouse" security updates are significant. &lt;/p&gt;

&lt;p&gt;If you are just starting out, building a custom scraper is a great learning exercise in networking and reverse engineering. However, for production-grade pipelines, relying on a managed scraping API is often cheaper in the long run. By offloading proxy rotation and CAPTCHA handling to specialized services, you keep your data flow consistent while focusing on the actual analysis rather than infrastructure maintenance.&lt;/p&gt;

&lt;h3&gt;
  
  
  Legal Considerations
&lt;/h3&gt;

&lt;p&gt;While scraping public data is generally protected under the CFAA, be careful with how you use the output. Raw facts like prices and addresses are typically fair game, but proprietary imagery and the "Zestimate" algorithm are protected intellectual property. Always consult your legal counsel before redistributing scraped data in a commercial product.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/how-to-scrape-zillow-listings-without-getting-blocked" rel="noopener noreferrer"&gt;How to scrape Zillow listings without getting blocked&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Best rotating residential proxy API for scraping in 2026</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Mon, 14 Sep 2026 07:27:06 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/best-rotating-residential-proxy-api-for-scraping-in-2026-4ffj</link>
      <guid>https://dev.to/serpscraperdev/best-rotating-residential-proxy-api-for-scraping-in-2026-4ffj</guid>
      <description>&lt;p&gt;As developers, we often fall into the trap of thinking our scraper needs a bigger proxy pool to survive. In reality, throwing more IPs at a bot-protected target is like trying to fix a leaky pipe by increasing the water pressure. The bottleneck isn't usually your pool size; it’s your lack of header management, weak TLS fingerprinting, and poor handling of TCP connection timeouts.&lt;/p&gt;

&lt;p&gt;After years of managing data pipelines, I’ve learned that the secret isn’t just having IPs—it’s how you manage the lifecycle of your sessions.&lt;/p&gt;

&lt;h3&gt;
  
  
  Understanding the Infrastructure Layers
&lt;/h3&gt;

&lt;p&gt;When you use a gateway for residential connections, you aren't just rotating IPs; you’re managing a handshake between your client and a peer-to-peer network. &lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Raw Residential Proxies:&lt;/strong&gt; These act as a backconnect gateway. You connect to a single endpoint, and the provider handles the rotation. You still need to deal with browser fingerprinting, user-agents, and managing cookies.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Managed Scraping APIs:&lt;/strong&gt; These are the "abstraction layer." You hit a single endpoint, and the provider handles the headless browser execution, CAPTCHA solving, and header normalization internally.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Why Pool Size is a Vanity Metric
&lt;/h3&gt;

&lt;p&gt;A pool of 100 million IPs is useless if 90% of them are burned or have poor reputation scores. I’ve shifted my focus to &lt;strong&gt;Success Rate&lt;/strong&gt; and &lt;strong&gt;Response Latency&lt;/strong&gt;. If your provider has bad "pool hygiene," your application spends more time waiting for TCP timeouts than parsing actual DOM nodes.&lt;/p&gt;

&lt;p&gt;When choosing a provider, evaluate them on these three technical pillars:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;ASN and Geographic Precision:&lt;/strong&gt; Can you route through a specific ISP or city-level gateway? This is vital for localized anti-bot walls.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;TLS Fingerprinting:&lt;/strong&gt; If your Python &lt;code&gt;requests&lt;/code&gt; script claims to be Chrome but your TLS handshake exposes the standard Python SSL library signature, a modern WAF will block you instantly. Look for providers that normalize these signatures.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Concurrency limits:&lt;/strong&gt; Ensure your provider doesn't throttle connections if you're scaling across Kubernetes worker nodes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The Trade-off: Build vs. Buy
&lt;/h3&gt;

&lt;p&gt;The choice between raw pools and managed APIs usually comes down to your team’s engineering bandwidth.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Go with a raw proxy pool (e.g., Decodo or NetNut)&lt;/strong&gt; if you have a dedicated security team. You need full control over the session persistence and custom headers. NetNut, for instance, offers high speed by sourcing directly from ISPs, which is fantastic for low-latency requirements.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Use a managed API (e.g., ScraperAPI)&lt;/strong&gt; if you want to ship faster. It abstracts away the "cat and mouse" game of browser spoofing. You don't need to write custom retry logic; the API handles it for you.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Pro-Tips for Optimization
&lt;/h3&gt;

&lt;p&gt;If you're paying by the gigabyte (which most premium services do), your scrapers are probably wasting money.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; &lt;strong&gt;Drop non-essential assets:&lt;/strong&gt; Block requests for images, fonts, and stylesheets. This can reduce your bandwidth bill by up to 60%.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Set aggressive timeouts:&lt;/strong&gt; Never let a script hang. Define a strict &lt;code&gt;connect&lt;/code&gt; and &lt;code&gt;read&lt;/code&gt; timeout. A dead node in a residential network should be killed immediately, not allowed to block your thread.&lt;/li&gt;
&lt;li&gt; &lt;strong&gt;Optimize for your target:&lt;/strong&gt; Don’t use a high-cost residential proxy to scrape a public, unprotected domain. Use cheap datacenter proxies for open targets and save your premium budget for the high-security walls.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Ultimately, your goal is to mimic a legitimate user journey. If your infrastructure isn't handling TLS, headers, and JS execution as a cohesive unit, no amount of IP rotation will save your scraper from the next wave of security updates.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/best-rotating-residential-proxy-api-for-scraping-in-2026" rel="noopener noreferrer"&gt;Best rotating residential proxy API for scraping in 2026&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Puppeteer headless browser scraping api: build vs buy guide</title>
      <dc:creator>SerpScraper.dev</dc:creator>
      <pubDate>Sun, 13 Sep 2026 04:12:33 +0000</pubDate>
      <link>https://dev.to/serpscraperdev/puppeteer-headless-browser-scraping-api-build-vs-buy-guide-3dfn</link>
      <guid>https://dev.to/serpscraperdev/puppeteer-headless-browser-scraping-api-build-vs-buy-guide-3dfn</guid>
      <description>&lt;p&gt;I’ve spent the last few years managing high-throughput data ingestion pipelines. If there is one thing that will systematically destroy your engineering budget and wake you up at 3 AM, it is managing a self-hosted cluster of headless Chrome instances. &lt;/p&gt;

&lt;p&gt;A basic Puppeteer script takes ten lines of code, but scaling it to millions of pages introduces massive infrastructure bottlenecks. Let’s break down the technical trade-offs of building your own browser cluster versus offloading the heavy lifting to a managed API.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why Headless Chrome Eats Your Budget
&lt;/h3&gt;

&lt;p&gt;Chromium is designed for rendering client-side pages in consumer environments, not low-resource headless server nodes. In my production builds, I’ve logged these average metrics per active tab:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;RAM Consumption:&lt;/strong&gt; 1.2GB to 2.0GB per active browser context under standard application loads.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;CPU Overhead:&lt;/strong&gt; ~0.5 dedicated CPU cores per concurrent page rendering.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Zombie Processes:&lt;/strong&gt; Ghost Chromium binaries that fail to terminate on script exit, locking up server memory permanently.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Bandwidth Leakage:&lt;/strong&gt; Headless browsers load CSS, images, and tracking scripts, inflating residential proxy bandwidth costs by up to 10x compared to simple raw HTTP requests.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Architectural Choice: Puppeteer vs. Puppeteer-Core vs. Playwright
&lt;/h3&gt;

&lt;p&gt;If you decide to build, do not deploy standard &lt;code&gt;puppeteer&lt;/code&gt; to production. It downloads a bundled Chromium binary (&amp;gt;150MB), inflating your Docker images past 1GB and slowing down deployment pipelines. &lt;/p&gt;

&lt;p&gt;Instead, use &lt;code&gt;puppeteer-core&lt;/code&gt; and connect it to a remote, separately managed browser pool using WebSockets:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;puppeteer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;puppeteer-core&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;puppeteer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;browserWSEndpoint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`ws://your-browser-cluster-endpoint`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For enterprise pipelines, also evaluate Playwright. While Puppeteer is the standard for Chrome-focused tasks, Playwright offers native &lt;strong&gt;Browser Contexts&lt;/strong&gt;. These are highly isolated, lightweight environments that function like separate browser profiles but share a single underlying process, vastly reducing memory overhead and startup latency during parallel runs.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Anti-Bot Arms Race
&lt;/h3&gt;

&lt;p&gt;Deploying your code is only half the battle. Modern anti-bot firewalls detect default Puppeteer instances almost instantly. They analyze:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;  &lt;strong&gt;Automation Flags:&lt;/strong&gt; Specifically checking if &lt;code&gt;navigator.webdriver&lt;/code&gt; evaluates to &lt;code&gt;true&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Fingerprint Signatures:&lt;/strong&gt; Inspecting Canvas rendering, WebGL variations, and available media codecs.&lt;/li&gt;
&lt;li&gt;  &lt;strong&gt;Network Fingerprints:&lt;/strong&gt; Matching TLS/JA4 handshake signatures with standard consumer browsers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;While plugins like &lt;code&gt;puppeteer-extra-plugin-stealth&lt;/code&gt; help, security platforms constantly evolve. To maintain high success rates, you must dynamically inject realistic human-like behaviors (non-linear mouse paths, randomized delays) and route all traffic through high-quality residential proxy pools.&lt;/p&gt;

&lt;h3&gt;
  
  
  Build vs. Buy Decision Matrix
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Architectural Metric&lt;/th&gt;
&lt;th&gt;Self-Hosted Cluster&lt;/th&gt;
&lt;th&gt;Managed API (e.g., serpscraper.dev)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Engineering Overhead&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;High (continuous proxy rotation, memory-leak fixes)&lt;/td&gt;
&lt;td&gt;Minimal (simple HTTP API integration)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Infrastructure Cost&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;High (demands massive multi-core, high-RAM VMs)&lt;/td&gt;
&lt;td&gt;Variable (success-based billing)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Proxy Management&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Complex (finding vendors, rotating IPs, CIDR bans)&lt;/td&gt;
&lt;td&gt;Fully automated inside the gateway&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Bypass Capabilities&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Manual patches (frequent maintenance loops)&lt;/td&gt;
&lt;td&gt;Dynamic fingerprinting &amp;amp; CAPTCHA solving&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Making the Call
&lt;/h3&gt;

&lt;p&gt;If your operations run locally under 5,000 pages per month, a simple self-hosted script is all you need. &lt;/p&gt;

&lt;p&gt;However, if you are scaling past 100,000 dynamic pages per day, managing your own infrastructure quickly becomes a full-time engineering drain. Migrating to a managed solution like &lt;code&gt;serpscraper.dev&lt;/code&gt; shifts this operational complexity to specialized cloud pools, allowing your team to focus on processing data rather than debugging zombie Chrome processes.&lt;/p&gt;




&lt;p&gt;Originally published at &lt;a href="https://serpscraper.dev/posts/puppeteer-headless-browser-scraping-api-build-vs-buy-guide" rel="noopener noreferrer"&gt;Puppeteer headless browser scraping api: build vs buy guide&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
