DEV Community: Darshan Khandelwal

Best Rank Tracker APIs in 2026: Features, Pricing & Performance

Darshan Khandelwal — Wed, 05 Aug 2026 10:16:51 +0000

A rank tracker API is essential for anyone serious about SEO, digital marketing, or competitive intelligence. Whether you’re building an in-house SEO tool or automating rank monitoring at scale, the right rank tracking API can make or break your workflow. Manually checking keyword positions on Google is slow and not scalable; a good API lets you programmatically monitor thousands of keywords across locations and devices.

If you’ve ever tried building an SEO tool or automating rank monitoring at scale, you already know this: not all Rank Tracking APIs are created equal. Some are fast, some are affordable, and some are not developer-friendly. So, instead of blindly trusting marketing claims, I decided to test them all head-to-head.

In this article, I’ll walk you through six of the most popular rank-tracking APIs

What Is a Rank Tracker API?

A rank tracker API is a programmatic interface that lets you query Google (or other search engines) and retrieve keyword ranking data in structured JSON format. Unlike manual rank-checking tools, a rank tracker API can be integrated directly into your own dashboard, spreadsheet, or SEO platform, enabling automated, real-time position monitoring at scale.

Who Needs a Rank Tracker API?

SEO agencies track hundreds of client keywords daily
SaaS founders building SEO tools or dashboards
Developers automating competitor monitoring
Content teams tracking blog post rankings over time
Best Rank Tracking APIs Compared: Speed, Success Rate & Support

Best Rank Tracking APIs Compared: Speed, Success Rate & Support

How We Will Test Each Rank Tracker API

Analysis will be made based on five attributes.

Scalability– Number of keywords that can be tracked at a time.

Pricing– Per request cost.

Developer-friendly– How easy is it for developers to integrate the API?

Speed– How fast an API responds.

Stability– Uptime of APIs.

I will be using this Python code to test the APIs. We have also created a rank tracking tool with Google Sheets and Serp API, so check it out.

1import requests2import time3import random4import urllib.parse5 6# List of search terms7rank_tracking_terms = [8    "best running shoes",9    "digital marketing agency",10    "best cafe in Chicago",11    "seo tools for small business",12    "best rank tracking api"13]14# Replace with your actual API endpoint15# Make sure it includes {query} where the search term should be inserted16base_url = "https://api.example.com/rank-tracker?query={query}"17total_requests = 1018success_count = 019total_time = 020for i in range(total_requests):21    try:22        search_term = random.choice(rank_tracking_terms)23        encoded_query = urllib.parse.quote(search_term)  # URL encode the query24        url = base_url.format(query=encoded_query)25        start_time = time.time()26        response = requests.get(url)27        end_time = time.time()28        request_time = end_time - start_time29        total_time += request_time30        if response.status_code == 200:31            success_count += 132        print(f"Request {i+1}: '{search_term}' took {request_time:.2f}s | Status: {response.status_code}")33    except Exception as e:34        print(f"Request {i+1} with '{search_term}' failed due to: {str(e)}")35# Final Stats36average_time = total_time / total_requests37success_rate = (success_count / total_requests) * 10038print(f"\nTotal Requests: {total_requests}")39print(f"Successful: {success_count}")40print(f"Average Time: {average_time:.2f} seconds")41print(f"Success Rate: {success_rate:.2f}%")

Scrapingdog’s Rank Tracker API

It offers the Google SERP API, which can be used to track keywords.

The API will provide you with a JSON response, which will have a key by the name of position , which can be used to track the ranks of keywords in the Google search results.

Details

Once you sign up, you will receive 1000 free credits. You can use them to test this API.

Per keyword tracking, the cost would start at $0.001 and go below $0.00029 with a higher volume.

Scrapingdog offers clear documentation, video tutorials, and blogs to help developers easily integrate APIs in their working environment. Our YouTube channel has multiple tutorials on Google SERP APIs.

Customer support is available 24*7 to help you resolve any query related to the services offered.

Testing the API

Scrapingdog is blazing fast, with an average speed of 1.31 seconds, and makes keyword tracking effortless.

On the flip side, using a sluggish SERP API can cripple your dashboard — sometimes to the point of freezing it. Just take a look at Neil Patel’s SEO tool; it struggles quite a bit, possibly because a turtle-speed API powers it.

Note: Recently, the num=100 parameter was removed. Now, to scrape 100 results from Google, you need to make 10 API calls (each to retrieve 10 results). Due to this depreciation, the rank tracking tools & their pipeline broke.

At Scrapingdog, we created our own rank tracking system using our Google Search API & n8n. You can check out the blog here.

SerpAPI

Serpapi also offers Google Scraping APIs that can scrape data from Google Search results and can be used as a rank tracker.

Details

First-time users will get 100 credits for free for testing the APIs.

Per keyword tracking, the cost would start at $0.015 and go below $0.0075 with a higher volume.

They have great documentation, and their API can be easily integrated within any working environment.

Support is great and replies within minutes. You can contact them through chat support from the website or through emails.

Testing the API

With an average response time of just 2.49 seconds, SerpAPI proves to be a good solution.

Ideal for building responsive and reliable rank tracking tools with their Google Search API.

This kind of use case is becoming common across many SaaS platforms. You can see more AI SaaS product ideas that startups are building today.

Brightdata

Brightdata is another great choice for scraping search results and tracking keyword ranks.

Details

They do offer a few free credits for first-time users.

The default pricing is $2.55/CPM, and if you have higher requirements, you can contact them.

Documentation is clear and concise. Any developer can integrate their proxies and APIs easily.

They have one of the best support systems in this scraping industry. Sometimes, it feels like they are just waiting for your query to drop. They will answer your query with lightning-fast speed.

Testing the API

With an average speed of around 5.81 seconds, this API may not be ideal for building a responsive rank-tracking tool.

However, its scalability makes it suitable for general SERP scraping tasks. The close is something on the higher end. So before committing to a paid plan its worth checking its alternatives.

SearchAPI

SearchAPI provides a complete solution around Google. Offers a variety of scrapers, including a rank tracking API.

Details

Just like every other product, they also offer free credits for new signups.

Per keyword tracking, the cost would start at $0.004 and go below $0.002 with a higher volume.

The documentation is well-organized, and users can simply copy the code and run it directly in their development environment.

You can contact them through chat or email.

Testing the API

Clocking in at an average of 8.87 seconds per request, this API isn’t winning any speed awards.

Still, if you’re more focused on scale than speed, it can get the job done for basic SERP scraping.

Apyhub

Apyhub is another rank-tracking API that offers great tools around SEO.

Details

On a new signup, you get very few free credits to test the API. I mean, you cannot even test the API completely. Around 5 API calls are free for a day.

Finding the right API on the website is a challenge in itself. Documentation is not great.

Pricing is not clear. So, cannot comment on that.

Support does not work. Nobody responds.

Testing the API

With an average speed of 7.13 seconds and a success rate of 80%, this API might test your patience if you’re building a real-time rank tracker.

That said, if you’re scraping SERP data in bulk, this API is a NO NO!

ScraperAPI

ScraperAPI is an old player in the web scraping industry, and they also offer a SERP API for tracking the ranks of keywords.

Details

Just like its competitors, it also offers free credits for testing the API.

Per keyword tracking, the cost would start at $0.01225 and go below $0.002075 with a higher volume.

Documentation is clear enough and can be easily integrated.

Support is available, but you might have to wait a whole day to get your query resolved. They also do not offer instant chat support.

Testing the API

Testing this API was a rollercoaster — and not the fun kind. At one point, I was just hoping the program would finish. The fastest response I got was 8.32 seconds. One request even took 170 seconds (yes, you read that right).

With a success rate of 80%, I’m honestly not sure whether to laugh, cry, or just close the terminal and walk away.

Which Rank Tracker API Should You Use?

Every API might look the same from the top, but once you test them, you see the real strength of the product, and you can determine which one best fits your rank tracking mechanism.

Scrapingdog and Serpapi are clear winners when it comes to response time.

Even if you look at the success rate offered by these APIs, you will reach the same conclusion. There are only two choices if you ignore the pricing. But if you consider the pricing as well, then Scrapingdog is worth every penny.

You can integrate each of these tools or APIs into your digital marketing strategy to check keyword rankings, and each will work effectively.

But if response time is something that you care about, pick APIs that have faster response times.

Frequently Asked Questions (FAQs)

How does a rank tracking API work?
A rank tracking API sends a query to Google, fetches the search results, and returns ranking data in JSON format. You can then use fields like position to see where a website ranks for a target keyword.
Which rank tracking API was the fastest in this comparison?
In this test, Scrapingdog was the fastest with an average response time of 1.31 seconds. SerpAPI came next at 2.49 seconds, while the other APIs were noticeably slower.
What factors were used to compare these rank tracking APIs?
The APIs were compared on scalability, pricing, developer-friendliness, speed, and stability. These factors help show not just raw performance, but also how practical each API is for real SEO workflows.
Which rank tracking APIs performed best overall?
Based on speed and success rate, Scrapingdog and SerpAPI were the top performers. If pricing is also considered, the blog concludes that Scrapingdog offers the best overall value.

10 Best Serper Alternatives for Fast & Reliable SERP APIs (2026)

Darshan Khandelwal — Thu, 30 Jul 2026 07:46:49 +0000

Back in 2023, when Serper.dev was launched, it soon became a major hotspot in the SERP API market. It is dead cheap, fast and flexible compared to any other options available in the market. However, if we look beyond organic search results, the API takes a complete U-turn and gets back behind its competitors.

It roughly covers 12 Google APIs, and if your project needs richer SERP features like AI Overviews, Immersive Products, Short videos, Ads or anything beyond basic SERP, the API fails silently.

We tested seven of the most popular Serper.dev alternatives in the market across speed, pricing, output richness, and AI readiness. Whether you’re building an AI monitoring tool, a rank tracker, or an SEO tool, this guide will tell you exactly which one deserves your API credits.

Why Developers are Moving Away From Serper.dev

Serper.dev is indeed a great product, and there is a lot of positivity around it on X and LinkedIn by the developer community. However, some concerning points around it are making developers drift away from the product:

Limited Engines and Scrapers Support — As we discussed above, Serper only covers 12 API Endpoints, including Google Search, Shopping, News, Images, Videos, Maps and a few others. If you need Baidu, DuckDuckGo, Bing, Amazon, and other search engines' support, you’ll have to look for another provider, which becomes a tedious task.

Data Richness Gap — There is a huge gap between the data points offered by Serper and other enterprise data collectors. Various fields like AI Overviews, rich Knowledge Graph data, Inline and Short videos, and ads are missing from it.

What to look for in a Serper.dev Alternative

Before discussing the various providers, here are the points based on which we will be evaluating them:

Pricing model: Monthly plans. Rollovers and credit expiry windows.

Response time: Measured at p50 and p95.

Output richness: Does it return PAA boxes, AI Overviews, sitelinks, discussions, shopping carousels, local packs, or just 10 blue links?

AI-readiness: Can the output feed directly into an LLM pipeline? Structured JSON matters here.

Multi-engine support: Google-only vs. Bing, Baidu, YouTube, and others.

Reliability at scale: Uptime, rate limits, and how the API behaves under concurrency.

Support: Response time, documentation quality, and whether there’s a human on the other end.

Note: We will be comparing the products based on Google Advanced SERP UI, not the basic one, which can be accessed via the gbv=1 parameter.

Scrapingdog

Starting price: ~$40/month for 20,000 Google SERP API calls.
Free tier: 200 credits on signup
Best for: Teams that need value at scale + a wide API surface area

Scrapingdog started as a dedicated general web scraping company that later added Google SERP APIs on top of its robust infrastructure.

So, basically Scrapingdog does not just acts as a Google SERP provider but also supports a two-step workflow in which you can seamlessly extract content inside those links appearing in the search results through its Web Scraping API providing it a great leverage to offer all the data under one hood avoiding developers to use multiple providers for search and scraping services which can result in waste of time and resources.

Scrapingdog’s multi-engine capability is not only backed by its large portfolio of Google Search APIs, which counts to twenty-seven (a huge number), but also supports other search engines like Baidu, Bing, Amazon, Walmart, and YouTube. If you’re running a search engine product or an SEO platform, the length and breadth of APIs alone save you from managing multiple accounts.

Pricing Breakdown:

No credit expiry on PAYG top-ups. If your volume fluctuates, you’re not burning credits you didn’t use.

Performance (Tested on 250 concurrency):

Avg response time: 2.75s(mean), 2.71s (p50), ~3.23s (p95)

Success rate: 100%

Code example, migrating from Serper in a minute:

1# Serper.dev (old)2import requests3 4response = requests.post(5    "https://google.serper.dev/search",6    headers={"X-API-KEY": "YOUR_SERPER_KEY"},7    json={"q": "best web scraping API", "num": 10}8)9results = response.json()10 11# Scrapingdog (drop-in replacement)12response = requests.get(13    "https://api.scrapingdog.com/google",14    params={15        "api_key": "YOUR_SCRAPINGDOG_KEY",16        "query": "best web scraping API",17        "results": 10,18        "country": "us"19    }20)21results = response.json()

Dead Simple! Isn’t it? Plus, the response schemas are the same, so no need to check for different property names in the JSON.

Why choose Scrapingdog over Serper.dev?

As Serper.dev only supports the basic Google SERP, we will be comparing both the LITE and Advanced Google SERP API pricing of Scrapingdog with it.

Serper vs Scrapingdog Advance Search Pricing Comparison

Scrapingdog not only becomes a super economical alternative to Serper at scale in LITE Search, but also in advanced search, which is expensive to scrape.

17+ Google endpoints plus multi search engine support vs. Serper’s ~12

No credit expiry on pay-as-you-go credits

Scrapingdog supports multiple featured snippets, including AI Overviews, Ads, and Knowledge Graph.

Built on the same infrastructure that handles tens of millions of web scraping requests, so reliability at scale is proven.

Human support response time under 5 minutes, clear and vast documentation make it a better choice.

Note: You should also use the Google News Scraping API to monitor your brand's mentions on the Internet.

SerpAPI

Starting price: $75/month for 5,000 searches ($15/1K)
Free tier: 100 searches/month
Best for: Enterprise teams that need legal coverage, multi-engine support, and maximum reliability

SerpAPI is the most established player in this list. The product is consistently maintained by its dedicated engineers, documentation is clear and concise, and APIs deliver the most complete JSON output in the market.

What makes them unique:

Legal US Shield: For plans above the “Developer” tier, SerpAPI provides a legal cover for scraping search results as long as you use it for ethical purposes.

Huge Portfolio: SerpAPI definitely holds the title for having the most search engine APIs in the industry. From Google to Yandex, from Amazon to Walmart, it literally covers every major search engine on the planet.

Open Status Pages: SerpAPI has status pages that consist of average speed and success rate for every API it supports on its platform.

Pricing Breakdown:

Pricing can be brutal not only for developers but also for medium-sized enterprises. However, if you have an enterprise-level volume, then SerpAPI can be the best option for you.

Performance:

Avg response time: 4.83s(mean)

Success rate: 100%

When to choose SerpAPI over Serper.dev:

You need advanced featured snippets from search engines.

You need all of the major search engines under one roof.

You’re building an enterprise SaaS where support SLA matters.

You need clear and concise documentation with dedicated engineers' support that can reply within 5 minutes.

SearchAPI

Starting price: ~$40/month for 10,000 searches
Free tier: 100 searches

A relative newcomer, but it has a strong reputation backed by its great performance and consistent maintenance of its data pipeline. It is positioned somewhere between Scrapingdog and SerpAPI in the enterprise pricing level set.

What stands out:

Covers structured JSON APIs across all Google endpoints.

Solid documentation with code examples in Python, Node.js, Ruby, PHP, and Go.

Cleaner logs consisting of all the needed information about the API call.

Pricing Breakdown:

Performance:

Avg response time: 2.9s(mean)

Success rate: 100%

When to choose SearchAPI over Serper.dev:

You need dedicated APIs related to Google AI Pages.

You want to track ads, citations, sponsored products and various other snippets, which Serper doesn’t yet support.

You need fast support as you can’t tolerate inconsistency with real users in production.

DataForSEO

Starting price: $600 per 1M searches (async queue)
Best for: SEO platforms, rank trackers, agencies doing bulk keyword research

DataForSEO is infrastructurally different from the others on this list. It not only offers a simple synchronous API that returns results in 2–5 seconds, but it also operates a task-based async model in which you just have to submit a batch of queries, and results will be ready in a queue, sometimes in minutes, sometimes longer, depending on load.

That may sound like a downside if you are running a live application, but for SEO teams running bulk rank tracking jobs or keyword research, it’s actually a feature. You don’t need real-time results for a 50,000 keyword crawl. You need low cost and high throughput.

Key differentiators:

Cheapest cost per query at scale: ~$0.60/1K on Standard Queue, as low as $0.006/1K on Sandbox. For bulk workflows, nothing beats this.

SERP + keyword + backlink data: DataForSEO isn’t just a SERP API; it has endpoints for keyword difficulty, search volume, backlink indexes, and more. If you’re building an SEO platform, this is a one-stop shop.

White-label friendly: Clean data model that’s easy to pipe into your own dashboard or product.

Pricing Breakdown:

The pricing changes on the basis of the API Mode:

For the Standard queue, it costs $0.0006 per SERP, and the cost per 1M is $600.

For the Priority queue, it costs $0.0012 per SERP, and the cost per 1M is 1200$.

For LIVE mode, it costs $0.002 per SERP, and the cost per 1M is 2000$.

Performance:

We couldn’t test the API due to too much complex documentation.

When to choose DataForSEO over Serper.dev:
You run bulk/batch SEO workflows, not real-time queries.

You’re building an SEO product and need keyword and backlink data alongside SERP.

Volume is high (1M+ searches/month), and cost is the primary constraint.

Real-time response time is not critical.

Exa & Tavily — Best for AI-Native Workflows

Exa starting price: ~$5/1K queries (content search)
Tavily starting price: ~$3/1K queries
Best for: LLM agents, RAG pipelines, AI research tools

These two products deserve a different section altogether because they’re solving a different problem than the rest of this list fundamentally. They’re not really SERP APIs. They’re AI-native search APIs built specifically for LLM workflows.

Exa uses neural search trained on link prediction to find content by meaning, not just keywords. When you query Exa, you get not only 10–100 links just like other search engines, but also full page content of these links extracted in clean markdown, ready to drop into an LLM prompt without any additional scraping step.

Tavily is similar but focused on citation-ready results. It’s deeply integrated with LangChain and LlamaIndex, which is why it’s become the default search tool for many AI agent frameworks.

The critical difference from SERP APIs:

A traditional SERP API (Serper, Scrapingdog, SerpAPI) returns metadata about search results, titles, URLs, snippets, and knowledge graph data. To get the actual page content, you need a separate scraping step.

Exa and Tavily return the content itself, full text, structured for LLM consumption. That’s why they’re in a different category.

When to choose Exa/Tavily over Serper.dev:

You’re building LLM agents or RAG pipelines and need page content, not just metadata

You’re using LangChain, LlamaIndex, or similar AI frameworks

Search freshness is secondary to content quality for your use case

Head-to-Head: Scrapingdog vs. Serper.dev

The pricing gap widens significantly at scale. At 1M searches/month, Serper charges ~$700 per month. Scrapingdog’s equivalent cost for the same job is ~330$.

Response schema comparison:

Both APIs return JSON. Here’s a simplified side-by-side of a Google SERP response:

1// Serper.dev response (simplified)2{3  "organic": [4    { "title": "...", "link": "...", "snippet": "...", "position": 1 }5  ],6  "peopleAlsoAsk": [...],7  "relatedSearches": [...]8}9 10// Scrapingdog response (simplified)11{12  "organic_data": [13    { "title": "...", "link": "...", "snippet": "...", "displayed_link": "...", "position": 1 }14  ],15  "peopleAlsoAsk": [...],16  "related_searches": [...],17  "ai_overview": { "text_blocks": "...", "sources": [...] },18  "knowledge_graph": {...}19}

The structure is similar enough that migration is minimal. Scrapingdog’s response includes additional fields like displayed_link, ai_overview, and knowledge_graph that gives your pipeline more data to work with.

How to Migrate from Serper.dev to Scrapingdog (Step by Step)

If you’re ready to switch, here’s the complete migration path:

Step 1: Get your Scrapingdog API key

Step 2: Update your endpoint and parameters

1# Python example2 3import requests4 5# Old Serper.dev code6def search_serper(query):7    response = requests.post(8        "https://google.serper.dev/search",9        headers={"X-API-KEY": "SERPER_KEY"},10        json={"q": query, "num": 10, "gl": "us", "hl": "en"}11    )12    return response.json()13 14# New Scrapingdog code15def search_scrapingdog(query):16    response = requests.get(17        "https://api.scrapingdog.com/google",18        params={19            "api_key": "YOUR_API_KEY",20            "query": query,21            "results": 10,22            "country": "us",23            "language": "en"24        }25    )26    return response.json()

1// Node.js example2 3// Old: Serper.dev4const searchSerper = async (query) => {5  const res = await fetch("https://google.serper.dev/search", {6    method: "POST",7    headers: {8      "X-API-KEY": "SERPER_KEY",9      "Content-Type": "application/json"10    },11    body: JSON.stringify({ q: query })12  });13  return res.json();14};15 16// New: Scrapingdog17const searchScrapingdog = async (query) => {18  const params = new URLSearchParams({19    api_key: "YOUR_API_KEY",20    query: query,21    results: 10,22    country: "us"23  });24  const res = await fetch(`https://api.scrapingdog.com/google?${params}`);25  return res.json();26};

Step 3: Update your response parser

Map the field names. Most are identical or near-identical:

Step 4: Run parallel for one week

Keep both integrations live. Run the same queries through both and compare outputs. Once you’re satisfied, cut over completely.

Frequently Asked Question

Is there a free alternative to Serper.dev?
Yes. Scrapingdog offers 200 free credits on signup. SerpAPI gives 100 searches/month. Serper itself offers 2,500 free searches/month, which is the most generous free tier in the list. For completely free options, you’re looking at open-source DIY approaches (Scrapy + proxies), which work but require significant maintenance.

What is the cheapest alternative to Serper.dev at scale?
At 100K+ searches/month, Scrapingdog and DataForSEO are the cheapest options. Scrapingdog costs around $0.25–0.30 per 1K searches at high volume. DataForSEO’s async Standard Queue can go even lower for bulk non-realtime workloads.

Does Scrapingdog support the same endpoints as Serper.dev?
Yes, and it also supports other endpoints as well. Everything Serper covers (web, news, images, shopping, maps, scholar) is available in Scrapingdog, plus Bing, Baidu, Google AI Overview, Google AI Mode, Google Trends, Google Finance, Google Patents, and more.

Is Serper.dev good for AI agents?
It works, but it’s not built for it. Serper returns search metadata titles, snippets, URLs. For AI agents that need actual page content in LLM-ready format, you still need a separate scraping step. If your use case is LLM-native, look at Exa or Tavily. If you want SERP data + the option to also scrape full pages under one API, Scrapingdog’s Data Extraction API handles both.

Does Serper.dev have an uptime SLA?
Serper doesn’t publish a formal uptime SLA. SerpAPI and SearchAPI both publish uptime stats. Scrapingdog maintains a status page at status.scrapingdog.com.

Conclusion

Serper.dev is a solid tool; we’re not here to trash it. But the limited endpoint coverage and no multi-engine support make it a poor fit for teams that are scaling beyond a prototype.

If you’re evaluating alternatives in 2026, the decision mostly comes down to three questions:

How much volume do you need? Low volume → Serper. High volume → Scrapingdog, DataForSEO, SearchAPI and SerpAPI.

Do you need fast results? Yes → Scrapingdog, Serper.

Are you building for LLMs specifically? SERP metadata → Scrapingdog. Full content extraction → Exa or Tavily.

For the majority of developers who are building LLMs, rank trackers and SEO tools, Scrapingdog delivers the best combination of price, endpoint breadth, and reliability at any volume tier.

You can test it for free with 200 credits at scrapingdog.com — no credit card required.

5 Best LinkedIn Scraping Tools (Ranked) by Scalability, Data Quality, and Pricing

Darshan Khandelwal — Wed, 29 Jul 2026 13:33:01 +0000

LinkedIn Scrapers are quite in demand in 2026. LinkedIn has the largest pool of corporates available today.

So obviously, you can find many people on this platform who could have the same interests as yours. But websites like LinkedIn can serve you with data that can be used for selling goods and products.

LinkedIn has more than 800 million active users where people share their work experience, skills, and achievements daily. If you scrape and use this data wisely then you can generate a lot of leads for your business.

Of course, this data can also be used for other purposes like finding the right candidate for the job or maybe enriching your own CRM.

In this article, we will talk about the best LinkedIn scrapers in the market. Using these scrapers you can scrape Linkedin at scale.

Quick Comparison of Best LinkedIn Scraping Tools

Advantages of using LinkedIn Scrapers Instead of Collecting Data Manually

You will always stay anonymous. On each request, a new IP will be used to scrape a page. Your IP will always be hidden.

The pricing will be less as compared to the official API.

You can get parsed JSON as the output.

3rd Party APIs can be customized according to demands.

24*7 support is available with many of them.

Some Challenges of Scraping LinkedIn at Scale

If you have researched enough, you must by now know that there are tools with which you can scrape a few hundred profiles/jobs without any problem.

These tools either use your LinkedIn Profile/LinkedIn Sales Navigator which makes them limited in terms of their abilities to extract data.

The main problem arises when you scrape LinkedIn at scale. Some of them are ⬇️

Legal and Ethical Issues:

User Agreement Violation: LinkedIn’s terms of service prohibit the scraping of their website. Engaging in this activity can result in legal action against the perpetrators.

Privacy Concerns: Extracting user data without consent violates privacy norms and can lead to severe repercussions. I have discussed in detail about the legality of scraping LinkedIn in this blog.

Technical Challenges:

Rate Limits: LinkedIn monitors and restricts frequent and massive data requests. An IP address can be temporarily banned if it makes too many requests in a short period.

Dynamic Content Loading: LinkedIn uses AJAX and infinite scrolling to load content dynamically. Traditional scraping methods often fail to capture this kind of content.

Complex Website Structure: LinkedIn’s DOM structure is intricate, and elements might not have consistent class or ID names. This can make the scraping process unstable.

Captchas: LinkedIn employs captchas to deter automated bots, making scraping even more challenging.

Cookies and Sessions: Managing sessions and cookies is necessary to mimic a real user browsing pattern and avoid detection.

Maintenance Issues:

Frequent Changes: LinkedIn, like other modern web platforms, frequently changes its user interface and underlying code. This means scrapers need constant updating to remain functional.

Data Quality: Ensuring the scraped data’s accuracy, relevancy, and completeness can be challenging, especially at scale.

Infrastructure and Costs:

Large-scale Scraping: Scraping at scale requires a distributed system, proxy networks, and cloud infrastructure, increasing the complexity and costs.

Data Storage: Storing vast amounts of scraped data efficiently and securely is another challenge.

Anti-Scraping Mechanisms:

Sophisticated Detection: LinkedIn employs sophisticated bot detection mechanisms. Mimicking human-like behavior becomes essential to avoid detection.

Continuous Monitoring: LinkedIn monitors for suspicious activities and can block accounts or IP addresses even if you successfully scrape data.

Advantages of using LinkedIn Scrapers Instead of Collecting Data Manually

- You will always stay anonymous. A new IP will be used on each request to scrape a page. Your IP will always be hidden.
- The pricing will be lower than the official API.
- You can get parsed JSON as the output.
- 3rd Party APIs can be customized according to demands.
- 24*7 support is available.

Best LinkedIn Scrapers [In 2026]

We will be judging these LinkedIn lead Scraper APIs based on 5 attributes.

Scalability means how many pages you can scrape in a day.
Pricing of the API. What is the cost of one API call?
Developer-friendly refers to the ease with which a software engineer can use the service.
Stability refers to how much load a service can handle or for how long the service is in the market.
Data Quality refers to how old the data is.

EnrichmentAPI

EnrichmentAPI offers a scalable solution for collecting public profile data. They also offer a reverse contact API, a company enrichment API, an email finder API, an email verification API, etc.

Scalability

You can pull data of millions of profiles in a day with this API. We have tested this, and we pulled around 1.8M profiles in a day with all the details like experience, skills, education, etc.

Pricing

They offer a free plan to test the API. On signup, you get 20 credits. Other than that, their smallest plan starts from $49/month, in which you can easily enrich around 1500 profiles.

Developer Friendly
Documentation is very clear, and any developer can integrate the APIs easily. You get instant chat support too through a chat widget or email.

Stability
EnrichmentAPI has been in the market for like 3 years. They have been providing the enrichment data consistently helping many CRM and AI companies to collect people data at scale.

Data Quality
You will get fresh data on every API call.

Scrapingdog’s LinkedIn Scraper API

Scrapingdog offers a simple and easy-to-use LinkedIn Scraper API.

This API can be used to scrape either a person’s profile or a company profile. Other than this Scrapingdog offers LinkedIn Jobs scraper to scrape job data from this platform.

Scalability

Well, you can scrape around 1 million profiles from Scrapingdog’s LinkedIn Scraping API be it job data or profile data.

Pricing

The enterprise pack will cost $1k per month and you can scrape 110k profiles. Each profile will cost $0.009.

Developer Friendly

The documentation is self-explanatory and the user can test the API directly from the dashboard without setting up any coding environment.

Stability

Scrapingdog has been in the market for 5 years now and has more than 200 users which proves its stability in scraping LinkedIn at scale.

Data Quality

We always scrape fresh data.

Brightdata LinkedIn Scraper

Brightdata, along with its large proxy network also provides LinkedIn Scrapers.

Scalability

Their scalability is great but the error rate might go up due to disturbances in their proxy pool. Bright data goes down frequently.

Pricing

Their solution could be a little expensive. The per-profile cost is around $0.05.

Developer Friendly

The whole documentation is quite easy to read and there are request builders which can help you get started quickly.

Stability

No doubt Brightdata has the biggest proxy pool in the market but they frequently go down sometimes. But overall their infrastructure is quite solid.

Data Quality

They do scrape fresh data like Scrapingdog.

Read More: 5 Best Bright Data Alternatives for Web Scraping

People Data Labs

PDL provides data enrichment technology. You can use their APIs to find valuable insights for any prospect.

Scalability

PDL is designed for small projects. You cannot scrape thousands of profiles with it in a short time frame. So, the scalability is not great.

Pricing

It will cost you $0.28 per profile. This makes them very very costly.

Developer Friendly

The documentation is nice which makes them developer-friendly.

Stability
If you are ok with old data then it is stable.

Data Quality

The data you get will be from an old database because they don’t scrape it fresh. They have data sources that they renew at regular intervals.

Apify

Apify is another great LinkedIn scraper. You can scrape person’s profile as well as the company profile.

Scalability

Their API only works if you pass cookies from your logged-in LinkedIn account. So, if you pass cookies from a normal account, you will be able to scrape a maximum of 50 profiles, and if you are passing cookies from a premium account, you will scrape a maximum of 500 profiles. Once you reach these limits your account will be banned by Linkedin and logged out of your account.

So, Apify is for those who just want to scrape a very small number of profiles.

Pricing

They provide a 3-day free trial after that they charge $25 per month.

Developer Friendly

Documentation is very clear and whether you are a developer or a non-developer you will be able to scrape profiles using Apify. It provides a hook through which you can scrape profiles using make.com.

Stability

Service is completely stable and you can scrape a small number of profiles very easily without getting blocked.

Data Quality

You will always get fresh data from their API. Since you are passing the cookies then you will get data like skills, history, etc which are absent in almost all the other scrapers.

Skrapp.io’s LinkedIn Extension

Wanted to mention this particular product as a special mention in this article. They aren't a dedicated LinkedIn Scraper, but a service that can be used on top of it.

Skrapp.io focuses on email finding and data enrichment from LinkedIn profiles. They specialize in extracting verified business emails rather than comprehensive profile scraping.

Scalability

Skrapp.io is designed for moderate-scale email extraction. With their Professional plan, you get 10,000 email credits per month, and their Enterprise plan offers 25,000 credits monthly. Each credit allows you to find one verified email address, making it suitable for targeted email campaigns rather than mass profile scraping.

Pricing

Professional plan costs $37/month (billed annually) for 10,000 email credits, making each verified email cost approximately $0.0037. Enterprise plan costs $75/month (billed annually) for 25,000 credits, bringing the cost down to $0.003 per email. They also offer a free plan with limited credits.

Developer Friendly

Skrapp.io offers browser extensions for LinkedIn, Sales Navigator, and LinkedIn Recruiter, making it accessible for non-developers. They also provide API integrations for Enterprise users. The platform is user-friendly with clear documentation and multiple export options.

Stability

Skrapp.io has been serving over 2 million professionals and has established partnerships with major companies like Adobe, IBM, and Microsoft. Their service focuses on compliance, and they only charge for valid and catch-all emails, ensuring fair usage. The service is stable for email extraction purposes.

Data Quality

They provide real-time email verification with their integrated email verifier. The platform ensures high accuracy by only charging for deliverable emails marked as 'Valid' or 'Catch-all.' However, their focus is primarily on email data rather than comprehensive profile information.

Final Verdict

We have compiled a report considering all the factors above. This report is the result of a comprehensive analysis of each API.

Although they may appear similar at first glance, further testing reveals that only a small number of APIs (one or two) are stable and suitable for production purposes.

Therefore, it is essential to evaluate the options based on your specific requirements and choose the most appropriate one from the given list.

I hope this list and my testing will help you pick one of them. If you liked the article do consider sharing it.

FAQ (Frequently Asked Questions)

(1) What is the best LinkedIn data extractor tool?
The best LinkedIn data extractor tool is one that can reliably collect profile, company, and job data at scale without requiring a logged-in account. API-based LinkedIn scrapers such as Scrapingdog, Bright Data are commonly preferred because they return structured data, handle dynamic LinkedIn pages, and help reduce blocking risks compared to browser extensions or manual scraping scripts.

(2) What is the best LinkedIn scraper for lead generation?
For lead generation, the best LinkedIn scraper is one that can extract key contact and professional details such as name, job title, company, industry, and profile URL. Tools that support bulk scraping, filters, and clean data output are ideal for sales teams and outreach campaigns.

(3) Is it safe to scrape data from LinkedIn?
Scraping LinkedIn is safer when only publicly available data is collected and scraping activity is kept within reasonable limits. Using tools with proxy rotation, rate limiting, and proper request handling helps reduce the risk of IP blocks while supporting more stable data extraction.

3 Google Scholar APIs Worth Checking Out in 2026

Darshan Khandelwal — Wed, 22 Jul 2026 07:18:49 +0000

Best Google Scholar API Comparison (2026)

Below is a quick comparison of the best Google Scholar APIs tested in this article, based on success rate, speed, and reliability.

Google Scholar data can unlock a goldmine of academic insights.

By analyzing this data, you can identify trending research topics, track citation growth over time, evaluate the impact of a paper or author, discover new publications in a specific field, or even map collaboration networks between researchers and institutions. For universities, it helps benchmark academic performance.

For developers, it powers tools like reference managers, citation analyzers, and literature review platforms.

Simply put, Scholar data helps you stay ahead in the research game, whether you’re publishing, studying, or building.

A Google Scholar API allows you to programmatically access academic research data such as article titles, author names, abstracts, citation counts, publication dates, journals, and even related works — all without manually browsing through the website.

It’s a game-changer for researchers, developers, and anyone who needs large volumes of scholarly information, including students managing assignments more efficiently. For those dealing with tight deadlines, some also explore options to do my homework more efficiently. Whether you’re building an academic search tool, analyzing trends in research papers, or simply pulling citation metrics for a report, a Google Scholar API helps you automate that process efficiently and at scale.

In this article, we will test 3 of the best Google Scholar APIs and rank them based on their performances. This study will help you select the best option for your next project.

Criteria To Test These Google Scholar APIs

Analysis will be made based on these five attributes.

Scalability– Number of researches that can be scraped.

Pricing– Per request cost.

Developer-friendly– How easy is it for developers to integrate the API?

Speed– How fast an API responds.

Stability– Uptime of APIs.

I will be using this Python code to test the APIs.

1import requests2import time3import random4 5 6# List of search terms7research_terms = [8    "cancer",9    "biology",10    "NLP Papers 2024",11    "machine learning in healthcare",12    "quantum computing applications"13]14 15# Replace with your actual API endpoint16# Make sure it includes {query} where the search term should be inserted17api_key="your-api-key"18base_url = "https://api.example.com/google_scholar"19#20 21 22 23 24 25total_requests = 1026success_count = 027total_time = 028 29for i in range(total_requests):30    try:31        search_term=random.choice(research_terms)32 33 34        params = {35  "engine": "google_scholar",36  "q": search_term,37  "api_key": api_key38}39 40 41 42        start_time = time.time()43        response = requests.get(base_url, params=params)44        end_time = time.time()45 46        request_time = end_time - start_time47        total_time += request_time48 49        if response.status_code == 200:50            success_count += 151        print(f"Request {i+1}: '{search_term}' took {request_time:.2f}s | Status: {response.status_code}")52 53    except Exception as e:54        print(f"Request {i+1} with '{search_term}' failed due to: {str(e)}")55 56# Final Stats57average_time = total_time / total_requests58success_rate = (success_count / total_requests) * 10059 60print(f"\n🔍 Total Requests: {total_requests}")61print(f"✅ Successful: {success_count}")62print(f"⏱️ Average Time: {average_time:.2f} seconds")63print(f"📊 Success Rate: {success_rate:.2f}%")

Scrapingdog Google Scholar API

Google Scholar API offered by Scrapingdog can be used for scraping Google Scholar data at scale. API will return structured JSON data, which will have all the basic details from the title to the link of the research.

Details

You will get 1000 free credits on sign-up. You can use these credits to test the API completely before upgrading.

The cost per API call starts at $0.001 and decreases to below $0.00029 as usage volume increases.

The documentation is very clear and provides ready-made code snippets. One can easily integrate the API by directly copying and pasting the code from the dashboard. We regularly post new articles and videos on YouTube.

You can contact them through email or on-site chat support. They are available 24*7 and will reply to your query within seconds.

Testing the API

We got amazing results.

With a 100% success rate and an average response time of just 1.26 seconds, Scrapingdog's Google Scholar API proves to be incredibly fast and reliable for scraping Scholar at scale.

SerpAPI

SerpAPI also offers Google Scholar API, and I am sure you already know about them. API will return a beautiful, structured JSON response.

Details

When you sign up, you get 100 credits for free. You can test any API before going for a bigger pack.

Pricing for one API call starts from $0.015(15x of Scrapingdog😲) and drops below $0.0075 with higher volume.

The documentation is clear and concise. Any developer can integrate the API in no time.

You can reach out to them through on-site chat support or through email. They have a great support team and will help you out with any query.

Testing the API

We got a 100% success rate with an average response time of 2.51 seconds.

SearchAPI

SearchAPI is another choice if you are looking to scrape Google Scholar.

Details

They provide 100 free credits for testing the API.

Per API, call cost will start from $0.004(4x of Scrapingdog😲) and drop below $0.002 with a higher volume.

Any developer can easily integrate their APIs into their working environment.

You can contact them through chat or email.

Testing the API

SearchAPI also provided a 100% success rate with an average speed of 3.59 seconds.

Conclusion

If you compare each API based on speed, the results will look like this.

Here, it looks like Scrapingdog and SerpAPI can both be used for scraping Scholar data at scale. Now, let’s compare them based on pricing.

Here, things look very different. If you consider the price, then Scrapingdog becomes a clear choice.

FAQ (Frequently Asked Questions)

1. What is a Google Scholar API, and what is it used for?

A Google Scholar API allows you to extract research data like article titles, authors, abstracts, and citation counts. This data is commonly used for academic research, citation analysis, literature reviews, and building research tools.

2. Does Google Scholar have an official API?

No, Google Scholar does not provide an official public API. This is why third-party Google Scholar APIs are commonly used to extract research data reliably.

3. What factors should I consider when choosing a Google Scholar API?

Key factors include data accuracy, success rate, ease of integration, pricing, and whether the API handles proxies and blocks automatically.

What’s the best way to scrape Google Scholar search results? The most reliable way is to use a dedicated Google Scholar API that manages proxies, rate limits, and anti-bot systems for you.

Best Oxylabs Alternatives in 2026: Features, Pricing & Comparison

Darshan Khandelwal — Wed, 15 Jul 2026 10:37:41 +0000

TL;DR

The best Oxylabs alternatives in 2026, compared by pricing, ease of use, and scalability, include:
Scrapingdog – Best overall alternative with predictable credit-based pricing, dedicated scraping APIs, no KYC, and plans starting at $40/month.
Bright Data – Enterprise-grade platform with the largest proxy network and the highest success rates, best suited for large-scale scraping.
Decodo – Affordable residential proxy provider with no KYC, ideal for developers who want Oxylabs-quality proxies at a lower cost.
ScraperAPI – Easy-to-use scraping API with transparent credit-based pricing, built-in proxy rotation, and JavaScript rendering.
Zyte – Strong choice for Scrapy users with flexible pay-as-you-go pricing and managed anti-bot infrastructure.

Looking for an alternative to Oxylabs for web scraping? In this article, I’ll show you 5 more affordable options, just as effective, and backed by support teams that actually reply.

Before we compare the alternatives in detail, here's a quick overview of their pricing, billing models, KYC requirements, and ideal use cases.

Overall Comparison

How We Evaluated These Oxylabs Alternatives

To keep this comparison fair, we evaluated every provider using the same criteria: pricing, billing model, onboarding process, KYC requirements, dedicated scraping APIs, documentation quality, customer support, and overall value for developers and businesses. Pricing information was collected from each provider's publicly available plans as of mid-2026.

Why Look for an Oxylabs Alternative

A few recurring complaints show up across independent reviews and buyer forums:

KYC friction — Oxylabs requires identity verification for trials and several products, which can delay onboarding for individual developers.

High minimums — Residential proxy traffic starts at $8/GB. That sounds fine until you do the math. A single Google page can run around 1MB, so 1GB burns through in just a few hundred calls.

Enterprise-first support model — Great if you’re a mid-market or enterprise buyer with an account manager, less great if you just want a quick answer in a chat window. The support team is extremely unhelpful and takes several hours to respond. Many users have complained about the same issue in the reviews as well.

Nine separate product lines — residential, datacenter, ISP, mobile, Web Scraper API, SERP API, Web Unblocker, and more, each with its own pricing model, which makes it genuinely hard to estimate a monthly bill upfront.

What Makes a Good Oxylabs Alternative?

Choosing a web scraping platform isn't just about finding the lowest price. A good Oxylabs alternative should offer reliable proxy infrastructure, transparent pricing, automatic CAPTCHA handling, JavaScript rendering, responsive customer support, and enough flexibility to scale from small projects to enterprise workloads. Depending on your use case, features like dedicated Google Search APIs, pay-as-you-go pricing, or residential proxies may matter more than having the largest proxy pool.

Scrapingdog

Scrapingdog is a web scraping API that handles proxy rotation, headless browser rendering, and CAPTCHA solving behind a single endpoint, plus dedicated APIs for Google Search, Google Maps, Amazon, Walmart, LinkedIn, and more that return clean, parsed JSON instead of raw HTML.

Pros

No KYC and no credit card required to try it. The free plan ships with 200 free credits.

Paid plans start at $40/month for 200,000 credits, with a pay-as-you-go option (25,000 credits for $10, credits never expire) for irregular workloads.

Every plan unlocks every API, there’s no feature-gating between tiers, only credits and concurrency.

Failed or blocked requests are never charged.

With 70+ dedicated APIs — Google SERP, Google Shopping, Amazon Product Scraper, Walmart, and more. You get purpose-built endpoints for nearly every major target, starting around $0.001 per request and dropping even lower at scale.

Documentation includes code samples across most major languages, and support is available via chat without needing an account manager.

Cons

Some heavily protected or less common websites might not get scraped.

No option to pick your own proxy location down to the city level.

Best for: developers, small-to-mid-size and enterprise teams who want predictable credit-based pricing, dedicated scraper endpoints for popular targets, and zero onboarding friction.

Brightdata

Bright Data is the largest commercial proxy network on the market, with a residential pool reported at around 400 million IPs across 195 countries. Alongside that, it offers a general Web Scraper API, SERP API, Web Unlocker, and ready-made datasets.

Pros

Usage-based pricing across the board — no single flat subscription. Residential proxies run roughly $8/GB pay-as-you-go, dropping toward $2.50–$3/GB on committed volume.

Web Unlocker and SERP API are typically priced per 1,000 successful requests, starting around $0.75–$1.50/1K depending on the tier.

Independent benchmarks consistently place Bright Data at or near the top on success rate against heavily protected targets like Google, Indeed, and Zillow.

No meaningful free tier, but trial credits are available, and the platform becomes genuinely cost-competitive once monthly spend passes roughly $200.

Cons

It’s genuinely expensive at low volume, with no real free plan, small projects end up paying enterprise-level rates for a fraction of the usage.

Pricing is spread across nine-plus separate products (residential, datacenter, SERP API, Web Unlocker, etc.), so figuring out what your actual monthly bill will look like takes real effort.

Best for: enterprise teams that need the largest possible IP pool and the highest success rate on the hardest anti-bot targets, and have the budget to match.

Decodo

Decodo rebranded from Smartproxy in 2025 but kept the same infrastructure: a 115M+ IP residential network, a Site Unblocker product, and an all-in-one Scraping API covering search engines and e-commerce sites.

Pros

Fully self-serve. You do not need to pass any KYC resistance.

Residential proxies start around $3.75/GB on the entry tier and scale down toward $2/GB at the 1TB tier; pay-as-you-go sits around $4/GB with zero commitment.

Site Unblocker is billed per 1,000 successful requests, at roughly $1.20/1K, and covers CAPTCHA solving and JavaScript rendering automatically.

ISO/IEC 27001 certified, which matters if your team has a compliance checklist.

Cons

The free trial is tiny with just 3 days and 100MB, which is barely enough to test one scraper run, let alone a real workload.

On sites with tougher anti-bot protection like Cloudflare or Akamai, success rates drop off unless you’re on the pricier Site Unblocker tier.

Best for: teams that want Oxylabs-style residential proxy quality at roughly half the entry price, without going through KYC.

ScraperAPI

ScraperAPI wraps proxy rotation, JavaScript rendering, and CAPTCHA solving into a single flat-rate credit system, with dedicated structured-data endpoints for Amazon and Google.

Pros
Plans start around $49/month, with a free-credit trial and no credit card required to sign up.

Pricing is credit-based but transparent: a fixed number of credits per request regardless of how “hard” the target site turns out to be, which makes budgeting easier than Oxylabs’ per-product pricing.

Includes an async endpoint, structured data endpoints, and a no-code scheduling tool (DataPipeline) on all plans.

Cons
Google APIs are extremely slow and come with super low success rates.

Like most credit-based tools, the advertised price looks cheap upfront, but JavaScript rendering and premium proxies can multiply your effective cost well beyond the headline rate.

Best for: developers who already have their own parsing logic and just want a reliable, predictably-priced proxy and rendering layer.

Zyte

Zyte (formerly Scrapinghub) is the company behind the open-source Scrapy framework, and its Zyte API automatically tiers pricing by how difficult a target site is to scrape.

Pros

Pay-as-you-go HTTP requests run roughly $0.13–$1.27 per 1,000, and browser-rendered requests run $1.01–$16.08 per 1,000, depending on site difficulty.

Monthly commitments ($100 / $200 / $500) unlock meaningfully lower per-request rates.

Scrapy Cloud is sold separately for teams hosting their own Python/Scrapy spiders, starting around $9/month per unit.

$5 free trial credit is available to test before committing.

Cons

Because pricing is tier-based and automatic, it can be hard to predict your exact monthly bill in advance.

Response times and support have both drawn complaints. Many users are reporting that they got a reply after 10+ days.

Best for: teams already built on Scrapy who want managed anti-bot handling without leaving the Python ecosystem.

> Pricing is approximate and pulled from publicly listed rates as of mid-2026 — always confirm current numbers on each provider’s pricing page before budgeting, since these change often.

Which Oxylabs Alternative Should You Actually Pick?

Tight budget, need dedicated APIs (Google, Amazon, Walmart): Scrapingdog.

Need the biggest IP pool and highest success rate on the hardest targets, budget isn’t the constraint: Bright Data.

Want cheap self-serve residential bandwidth without KYC: Decodo.

Building on Scrapy or need Python-native tooling: Zyte.

Oxylabs is a genuinely strong platform, and it’s still the right call if you’re an enterprise buyer who needs a dedicated account manager and can absorb its pricing. But for most developers and small teams, one of the five alternatives above will get you the same data with less friction and a smaller bill.

How to Find All URLs on a Domain's Website: 4 Methods Explained

Darshan Khandelwal — Wed, 24 Jun 2026 13:00:14 +0000

Finding all the URLs on a domain sounds simple until you actually try it. Google’s site: search misses pages. Sitemaps skip no-indexed content. Manual crawling breaks on large sites. And if the website renders pages via JavaScript, most tools won’t even see them.

Whether you’re auditing your own site before a migration, mapping a competitor’s content structure, hunting for broken links, or building a scraper that needs every page URL, this is one of the most common (and most misunderstood) tasks in web scraping and SEO.

In this guide, we’ll walk through multiple methods to find all URLs on a domain, from zero-code approaches you can run in 30 seconds to Python scripts that scale to thousands of pages. For each method, we’ll cover what it finds, what it misses, and when to use it.

Let’s get into it.

Find All URLs On A Domain’s Website

Before jumping into the methods, it’s worth knowing the most common reasons people do this, because your goal will determine which method makes the most sense.

SEO auditing — find orphan pages, check for duplicate content, spot pages that shouldn’t be indexed, or verify that all your important URLs are actually crawlable.

Broken link detection — crawl every URL on your domain and check their status codes. A full URL list is the starting point for any broken link audit.

Site migration — before moving to a new domain or CMS, you need a complete inventory of every page so nothing gets left behind or loses its redirect.

Competitive research — map out a competitor’s content structure to find topics they’re covering that you aren’t.

Web scraping — if you want to scrape data from every page on a site, you first need a list of every page to scrape.

Content audits — identify thin pages, outdated posts, or content gaps across your entire site.

Now, let’s discuss all the techniques in brief.

Google search technique

Google Search is the fastest zero-setup method; just enter a search query to find indexed pages on any domain. The catch: Google excludes pages for reasons like duplicate content or noindex tags, so results are never a complete picture.

Sitemaps and robots.txt

Sitemaps and robots.txt give you a more structured view of a site’s pages than Google search, but they come with their own blind spots. A domain can have multiple sitemaps, some pages may be 302 redirecting at the time you check, and SEO plugins like RankMath or Yoast automatically drop non-canonical pages from the sitemap entirely. So if Page A has a canonical pointing to Page B, Page A simply won’t appear.

SEO crawling tools

SEO crawling tools are the most straightforward option — no code, detailed output, and easy to use. The tradeoff is cost; most tools limit free usage to 500 pages, which works fine for smaller sites but requires a paid plan once you scale.

Scraping

If you’re comfortable with coding, a custom scraper gives you the most control: filter by URL pattern, exclude directories, handle pagination, or integrate directly into your pipeline. It takes more setup, but for large or complex sites, it’s often the only method that gets you everything you need.

Through Google Site Search

One of the quickest ways to find URLs on a website is with Google’s site search feature. Here’s how to use it:

Go to Google.com

In the search bar, type site:example.com (replace example.com with the website you want to search).

Hit enter to see the list of indexed pages for that website.

Let’s find all the pages on scrapingdog.com. Search for site:scrapingdog.com on Google and hit enter.

Google will return a list of indexed pages for a specific domain.

However, as discussed earlier, Google may exclude certain pages for reasons such as duplicate content, low-quality, or inaccessible pages.

So, the “site:” search query is best for getting rough estimates, but it may not be the most accurate measure.

If you want to do it programatically then you can take help of the Google Search API and Python to extract all the links without manually checking each page on google.

Through Sitemaps and robots.txt

Sitemaps and robots.txt are two files that almost every website exposes publicly, and together they can give you a near-complete picture of a site’s URL structure. Unlike Google search, you’re not dependent on what Google has chosen to index; you’re looking directly at what the site owner has declared. Here’s how to use each one.

Using sitemaps

A sitemap is an XML file listing all important website pages for search engine indexing. Webmasters use it to help search engines understand the website’s structure and content for better indexing.

Every decent website has a sitemap as it improves Google rankings and is considered a good SEO practice. To learn how to create and optimize one effectively, read the article for practical tips.

Here’s what a standard sitemap looks like:

The element specifies the page URL, indicates the last modification time, and signifies the relative importance for search engines (higher priority means more frequent crawling).

Now, where to find a sitemap? Check for /sitemap.xml on the website (e.g., https://scrapingdog.com/sitemap.xml)

Websites can have multiple sitemaps in various locations, including: /sitemap.xml.gz, /sitemap_index.xml, /sitemap_index.xml.gz, /sitemap.php, /sitemapindex.xml, /sitemap.gz.

Most websites mention the number of sitemaps they have under the domain in the robots.txt file, which we are going to discuss next.

Using robots.txt

The robots.txt file instructs search engine crawlers on which pages to index and which ones to exclude from indexing. It can also specify the location of the website’s sitemap. The file is often located at the /robots.txt path (e.g., https://scrapingdog.com/robots.txt).

Here’s an example of a robots.txt file. Some routes are disallowed for indexing. The sitemap location is also present.

You need to visit both sitemaps and find all the URLs within the website. Note that, for smaller sitemaps, you can manually copy the URLs from each tag. But for larger sitemaps, consider using an online tool to convert the XML format to a more manageable format, such as CSV. There are free tools available, like the one Seowl sitemap extractor.

Through SEO Crawling Tools

Now let’s see how SEO crawling tools help us find all website pages. There are various SEO crawlers in the market, we’ll explore the free tool XML-Sitemaps.com. Enter your URL and click “START” to create a sitemap. This tool is suitable when you need to quickly create a sitemap for a small website (up to 500) pages.

The process will start and you will see the number of pages scanned (167 in this case) and the number of pages indexed (127 in this case). This indicates that only around 127 of the scanned pages are currently indexed in Google Search.

Once the crawling process is complete, the sitemap preview will display all the website’s indexed URLs, including the last modification date and time, as well as the priority of each URL.

You can download the XML sitemap file or receive it via email and put it on your website afterward.

Screamingfrog is another free tool that can help you with exporting URLs from a domain. Keep in mind that the free version allows you to get 500 URLs from a single domain. To use it, you have to download the software on your machine & unlike SaaS, it will crawl the domain from your setup only.

Through Web Scraping

If you’re a developer, you can build your crawler script to find all URLs on a website. Here, you can take advantage of web scraping APIs like Scrapingdog.

This method offers more flexibility and control over crawling compared to previous methods. It allows you to customize behavior, handle dynamic content, and extract URLs based on specific patterns or criteria.

You can use any language for web crawling, such as Python, JavaScript, or Golang. In this example, we will use Python. Also, sign up for the trial Scrapingdog pack.

You can export the list and keep it in CSV inside a folder.

Using Python and Scrapingdog

Before writing the code, make sure you have installed the necessary libraries.

Install the libraries using pip:

1pip install beautifulsoup4 requests

Next, let’s create a Python file (I am naming the file as main.py) and write the code.

If you paste your target website in the Scrapingdog scraper, you will get a ready to use python code.

Just copy this code and paste it into the main.py file. I have used format as links, this ensures you get output as links only and not the whole HTML data.

1import requests2 3url = "https://api.scrapingdog.com/scrape"4params = {5    "api_key": "your-api-key",6    "url": "https://www.scrapingdog.com",7    "dynamic": "false",8    "formats": "links"9}10 11response = requests.get(url, params=params)12print(response.text)

Once you run this code, you will get this JSON response with all the links available on this page.

Now, you can continue making these GET requests to all these URLs using Scrapingdog to extract all the links on that particular domain.

Now, if you want to store all these URLs in a CSV file, you can use Python’s pandas library.

1import requests2import pandas as pd3 4url = "https://api.scrapingdog.com/scrape"5params = {6    "api_key": "your-api-key",7    "url": "https://www.scrapingdog.com",8    "dynamic": "false",9    "formats": "links"10}11 12response = requests.get(url, params=params)13 14if response.ok:15    data = response.json()16    links = data.get("links", [])17    18    df = pd.DataFrame(links, columns=["URL"])19    df.to_csv("scrapingdog_links.csv", index=False)20    print(f"Saved {len(df)} links to scrapingdog_links.csv")21else:22    print(f"Request failed with status code: {response.status_code}")

The crawled URLs will be stored in a CSV file, as shown below:

You can read our guide web scraping with python to get more idea on how a scraper can be built using Python.

Which Method Should You Use?

With four methods on the table, the right choice comes down to your goal, technical comfort, and the size of the site you’re working with. Here’s a quick reference:

Key Takeaways

No single method finds every URL on a domain. For a complete picture, always combine at least two methods.

Google’s site: operator is the fastest starting point but should only be used for rough estimates — it regularly misses pages due to noindex tags, duplicate content filters, and indexing delays.

Screaming Frog is the best free option for a full SEO crawl, but caps at 500 URLs on the free plan. For larger sites, use a Python crawler or Scrapingdog’s API.

JavaScript-heavy sites require a rendering-capable tool. Standard HTTP requests won’t execute JavaScript, so use Scrapingdog’s API to get fully rendered HTML before parsing links.

Conclusion

Finding all the URLs on a domain is one of those tasks that looks straightforward on the surface but has more nuance than most people expect. Different methods find different pages, and no single approach gives you the full picture every time.

The good news is you don’t need to pick just one. Start with a sitemap for the declared URLs, layer in a crawler for anything that was missed, and fall back to Scrapingdog’s API when JavaScript rendering gets in the way.

Whether you’re running an SEO audit, prepping for a site migration, or building a scraper, a clean, complete URL list is always the starting point. Now you have six ways to build one.

If you want to skip the setup entirely and start extracting URLs right away, try Scrapingdog for free with 1000 credits, no credit card required.

Scrape Baidu Search Results using Python: A Step by Step Guide

Darshan Khandelwal — Tue, 23 Jun 2026 13:16:51 +0000

When it comes to search engine data, most tutorials focus on Google, but what if your target audience is in China? That’s where Baidu, the country’s leading search engine, comes in. Whether you’re tracking keyword rankings, monitoring competitors, or gathering localized insights, scraping Baidu search results can give you access to valuable, real-time data that isn’t available elsewhere.

In this guide, we’ll walk you through how to scrape Baidu search results using Python, step by step no prior scraping experience required.

Why scrape Baidu Search Results?

Baidu dominates the Chinese search market, holding over 75% share, making it the go-to source for online information in China. Scraping Baidu search results allows developers, marketers, and researchers to gain deep insights into the Chinese web ecosystem. You can:

🔍 Track keyword rankings for SEO in Chinese markets.

🏢 Monitor competitors and analyze their visibility in Baidu’s SERPs.

📰 Collect news and content trends relevant to specific industries.

📊 Build datasets for training AI and NLP models with localized content.

🌏 Understand market behavior and consumer interests unique to China.

In short, if your business or research involves Chinese audiences, scraping Baidu is one of the most direct and effective ways to access that data.

Why use Scrapingdog to Scrape Baidu search results?

Scraping Baidu directly can be challenging due to its strict anti-bot systems, IP rate limits, and frequent layout changes. That’s where Scrapingdog comes in, and it takes care of all the heavy lifting like proxy rotation, CAPTCHA handling, and response parsing. Instead of worrying about blocked requests or HTML changes, you can simply send a query and receive structured, ready-to-use data.

With Scrapingdog’s Baidu Search API, you get clean JSON output containing titles, links, and snippets, and all accessible in one simple API call. It’s fast, reliable, and built to scale, whether you’re scraping a few keywords for SEO or running large-scale data collection for AI and analytics. In short, it saves developers hours of setup time while maintaining accuracy and consistency across requests.

Prerequisite

I hope you have already installed Python 3.x on your machine; if not, you can download it from here.

Now, create a folder by any name you like. I am naming the folder as webpython. Inside this folder, create a Python file and name it whatever you like. Now, install requests library inside this folder. We will use this to make an HTTP connection with the host website.

The final step is to sign up for Scrapingdog’s trial pack. You’ll receive 1,000 free credits upon signup, which you can use to test any of Scrapingdog’s APIs.

Scrape Baidu Search page with Python

After signup on Scrapingdog you will be redirected to your dashboard. You will find a Baidu Search API over there.

Once you click on it, you will see a scraper where you just need to pass a sample query, and it will pull the data and return it to you in neat JSON format.

For this tutorial, we will pass the query as web scraping. After passing the query you will get a ready Python code; you just have to copy that and paste it in your Python file.

1import requests2 3api_key = "your-api-key"4url = "https://api.scrapingdog.com/baidu/search"5 6params = {7    "api_key": api_key,8    "query": "web scraping"9}10 11response = requests.get(url, params=params)12 13if response.status_code == 200:14    data = response.json()15    print(data)16else:17    print(f"Request failed with status code: {response.status_code}")

Let me explain this code step-by-step.

First we have imported the requests library. This will be used for making the HTTP connection with the host website.

Then we are storing the API key and the Scrapingdog’s Baidu Search API URL.

Then we are making the GET request with the help of requests library.

If the request is successful (status 200), it prints the search results.

If it fails, it prints the error status code.

Once you run the code you will get this parsed JSON data.

In this JSON response you will find the title, link, snippet and rank of the result.

Key Takeaways:

You can programmatically extract Baidu search results using Scrapingdog’s Baidu Search API.

The API returns clean, structured SERP data without manual HTML parsing.

Using an API solves common challenges like proxy rotation and anti-bot measures.

You can customize search parameters like language, pagination, and filters.

This approach scales well for data collection and analytics.

Conclusion

Scraping Baidu data doesn’t have to be complex or time-consuming. With Python and Scrapingdog’s Baidu Search API, you can easily extract clean, structured data from Baidu’s search results without dealing with IP bans, CAPTCHAs, or changing HTML layouts. Whether you’re monitoring keyword trends, gathering localized content, or powering AI applications, this API makes it effortless to access reliable Baidu search data at scale.

In short, instead of spending hours maintaining scrapers, you can focus on what really matters, analyzing data to make smarter decisions in one of the country’s leading markets.

Start your free trial on Scrapingdog today and automate Baidu scraping in just a few lines of Python.

FAQ (Frequently Asked Questions)

1: Is it legal to scrape Baidu search results?

Yes, scraping publicly available Baidu search results is generally allowed. However, you should always follow Baidu’s terms of service and local data regulations when using the data commercially.

2: Can I scrape Baidu without getting blocked?

Direct scraping often leads to IP bans and CAPTCHAs. Scrapingdog handles proxies and CAPTCHAs automatically, so you can scrape Baidu reliably.

3: What data can I extract from Baidu using Scrapingdog?

You can extract the title, URL, snippet, and ranking position in structured JSON format.

4: Can Scrapingdog handle large-scale Baidu scraping?

Yes, Scrapingdog is built for scale and supports both small and high-volume Baidu scraping use cases.

How to Scrape Home Depot Product Data [Step-by-Step Guide]

Darshan Khandelwal — Tue, 09 Jun 2026 13:37:58 +0000

Scraping Home Depot isn’t straightforward. It uses JavaScript rendering, rate limiting, CAPTCHA challenges, and IP bans to block automated requests. You need the right tools and approach to collect data reliably at scale.

In this guide, you’ll learn how to scrape Home Depot using Python, from listing pages and product details to reviews and pagination, and export everything into a clean CSV file. You can also refer our guide on web scraping with python to learn scraping with python.

TL;DR
If you know the complete logic begind the scraping and parsing then you can just copy this code and use it.

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.scrapingdog.com/scrape",
    params={
        "api_key": "your-api-key",
        "url": "https://www.homedepot.com/p/RYOBI-ONE-18V-Lithium-Ion-2-0Ah-4-0Ah-Starter-Kit-with-1-2Ah-HP-Battery-1-4Ah-HP-Battery-and-Charger-PSK1212SB/339206615",
        "dynamic": "false"
    }
)
print(response.status_code)

obj={}
soup = BeautifulSoup(response.text, "html.parser")

# Title
title = soup.find("h1", class_="sui-line-clamp-unset")
obj["Title"] = title.get_text(strip=True)

# Price - dollar, cents in separate spans
price_parts = soup.find_all("span", class_="sui-font-display")[:3]
price = "".join([p.get_text(strip=True) for p in price_parts])
obj["Price"]=price

# Image - actual class in this page
img = soup.find("img", attrs={"data-testid": "small-image"})
obj["image"]=img["src"]

# Features
features_ul = soup.find("ul", class_="sui-text-base")
if features_ul:
    features = [li.get_text(strip=True) for li in features_ul.find_all("li")]
    obj["features"]=features

# Rating
rating_tag = soup.find("a", class_="sui-flex-row")
obj["Rating"]=rating_tag["title"]

print(obj)

Why Scrape Home Depot?

Home Depot has over 1 million products listed online, making it a valuable source for e-commerce data. Here’s what you can do with it:

Price monitoring — Track price changes and adjust your pricing strategy.
Competitor analysis — See how competing brands position and price their products.
Inventory tracking — Monitor stock availability to spot supply gaps or demand surges.
Review analysis — Collect customer feedback at scale to understand sentiment.
Market research — Identify trending products and top-rated brands.

Requirements

To start with, you need Python installed on your machine. If you have not installed it, then you can download it from here. For this tutorial, we will require three libraries:

requests — We will use this to make an HTTP connection with the Host website.
BS4 — This will be used for parsing the data downloaded using requests.
Pandas — This will be used to store data in a CSV file.
The final step would be to signup for a trial account of Scrapingdog. You will get 1000 free credits, which are enough for this tutorial.

Scraping Home Depot Product Page

For this tutorial, we are going to scrape this page from Home Depot. Create a Python file by any name you like. I am naming the file as home.py.

From the product page, we will scrape:

Title
Price
Images
Features
Rating

Scraping Home Depot with Scrapingdog

import requests

url = "https://api.scrapingdog.com/scrape"
params = {
    "api_key": "your-api-key",
    "url": "https://www.homedepot.com/p/RYOBI-ONE-18V-Lithium-Ion-2-0Ah-4-0Ah-Starter-Kit-with-1-2Ah-HP-Battery-1-4Ah-HP-Battery-and-Charger-PSK1212SB/339206615?MERCH=REC-_-rv_homepage_rr-_-n/a-_-0-_-n/a-_-n/a-_-n/a-_-n/a-_-n/a",
    "dynamic": "false"
}

response = requests.get(url, params=params)
print(response.status_code)
print(response.text)

Here we are making the GET request using requests to the scrape endpoint provided by Scrapingdog. Do not forget to pass your own private Scrapingdog’s key.

Let’s run this code.

After running the code, you will get a 200 status code along with the HTML of the target page.

Now, we can use beautifulSoup to parse this data.

Parsing the required data with BeautifulSoup

Let’s start with the title. We have to find the DOM location of the title.

The title is wrapped inside a H1 tag with a class sui-line-clamp-unset.

The pricing is hidden inside the first three span tag with class sui-font-display.

The product image is wrapped inside an img tag with class sui-w-full.

All the features of the product are located inside the ul tag with class sui-text-base.

The rating of the product is wrapped inside a tag with class sui-flex-row.

Now, we have the DOM location of every data point we want to parse from the raw HTML. Let’s implement this logic with BS4.

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.scrapingdog.com/scrape",
    params={
        "api_key": "your-api-key",
        "url": "https://www.homedepot.com/p/RYOBI-ONE-18V-Lithium-Ion-2-0Ah-4-0Ah-Starter-Kit-with-1-2Ah-HP-Battery-1-4Ah-HP-Battery-and-Charger-PSK1212SB/339206615",
        "dynamic": "false"
    }
)
print(response.status_code)

obj={}
soup = BeautifulSoup(response.text, "html.parser")

# Title
title = soup.find("h1", class_="sui-line-clamp-unset")
obj["Title"] = title.get_text(strip=True)

# Price — dollar, cents in separate spans
price_parts = soup.find_all("span", class_="sui-font-display")[:3]
price = "".join([p.get_text(strip=True) for p in price_parts])
obj["Price"]=price

# Image — actual class in this page
img = soup.find("img", attrs={"data-testid": "small-image"})
obj["image"]=img["src"]

# Features
features_ul = soup.find("ul", class_="sui-text-base")
if features_ul:
    features = [li.get_text(strip=True) for li in features_ul.find_all("li")]
    obj["features"]=features

# Rating
rating_tag = soup.find("a", class_="sui-flex-row")
obj["Rating"]=rating_tag["title"]

print(obj)

Once you run this code you will get this beautiful parsed JSON data with all the details we were looking for.

{'Title': 'ONE+ 18V Lithium-Ion 2.0Ah, 4.0Ah Starter Kit with (1) 2Ah HP Battery, (1) 4Ah HP Battery and Charger', 'Price': '$9900', 'image': 'https://images.thdstatic.com/productImages/3f4be679-3a35-40e5-8935-f6876f66f03a/svn/ryobi-power-tool-batteries-psk1212sb-64_600.jpg', 'features': ['Up to 4X More Runtime', '4Ah & 2Ah Lithium HIGH PERFORMANCE Batteries & Charger', 'Power All 300 ONE+ Products With Any RYOBI 18V ONE+ Battery', 'View More Details'], 'Rating': '4.5 out of 5'}

Storing the data in a CSV file

extracted_data.append(obj)

df = pd.DataFrame(extracted_data)
df.to_csv("home_depot.csv", index=False)
print(" Data saved to csv successfully.")

Appends each scraped product as a dict into a list, then converts that list into a pandas DataFrame and exports it as home_depot.csv. This will create a csv file inside your folder with the scraped data.

You can repeat this process in loop with the help of a for loop for various different products on homedepot.com.

1. Is it legal to scrape Home Depot?

Scraping publicly available data is generally legal.

2. Does Home Depot block web scrapers?

Yes. It uses rate limiting, CAPTCHAs, IP bans, and JavaScript rendering to detect and block bots. That’s why you are recommended to use web scraping APIs like Scrapingdog.

3. Do I need a proxy to scrape Home Depot?

Yes. Rotating residential or ISP proxies are recommended to avoid IP bans at scale.

4. What data can I scrape from Home Depot?

Product names, prices, availability, ratings, reviews, specs, model numbers, and images.

5. Can I scrape Home Depot without JavaScript rendering?

No. Home Depot heavily relies on JavaScript to load product data, so you’ll need a headless browser or a scraping API.

6. How do I handle pagination when scraping Home Depot?

Increment the page offset parameter in the URL and loop through pages until no more results are returned.

Conclusion

Scraping Home Depot with Python gives you access to valuable product data at scale. In this guide, we covered everything from setting up your environment to extracting listings, product details, reviews, and handling pagination.

The biggest challenge is avoiding blocks, use rotating proxies and handle JavaScript rendering to stay undetected. If you want to skip the infrastructure hassle, Scrapingdog’s web scraping api handles all of that out of the box.

Happy scraping!

SearchAPI Alternatives in 2026

Darshan Khandelwal — Tue, 19 May 2026 12:33:40 +0000

SearchAPI.io, a developer focused SERP APIs infrastricture platform provides structured data to developers from majorly Google and other sources. It covers 50+ APIs, mainly focused on Google SERPs, and includes a $2M legal coverage clause, ZeroTrace privacy mode, and clean documentation. For a certain category of developers who need multi-search-engine coverage and legal protection for mass scraping, this can be a legitimate choice.

But still, there are specific conditions where SearchAPI falls short and can become a major issue when you move it into production:

The free tier is only available till 100 searches, and there is no pay-as-you-go model, causing credits to run out before even completing testing. The rate limit caps you at 20% of your monthly credits per hour, which means on burst workloads, you’ll hit throttling even when you have credits left. And at $2–4 per 1,000 searches, depending on your plan, the cost math gets uncomfortable once you’re running 100K+ queries a month. The cost decreases at scale, but it can still be higher than that of competitors.

We tested 7 of the strongest SearchAPI.io alternatives on price, speed, output richness, endpoint coverage, and flexibility. Here’s the complete breakdown.

Quick Comparison Table

What SearchAPI.io Actually Does Well

Before getting into why developers switch, let’s be clear about where SearchAPI.io genuinely earns its place:

2M$ Legal Coverage: SearchAPI offers a legal shield for collecting data from its APIs under US law applicable to customers worldwide. For enterprise teams, who scrape at scale and have concerns regarding the legality, you are already covered.

ZeroTrace mode: Setting zero_trace=true in any request prevents SearchAPI from logging or storing your request parameters, queries, or responses. For teams that don’t want someone to store their sensitive data, it is quite helpful.

50+ endpoints under one API key: Google Search, Google Maps, Google News, Google Shopping, Google Scholar, Google Jobs, Google Trends, Google Lens, Google Finance, Google Patents, YouTube, Amazon, Bing, Baidu, and more are all accessible through a consistent engine parameter in a single API schema. If you’re managing data from multiple search surfaces in one product, the unified schema reduces integration overhead.

99.9% SLA: SearchAPI guarantees uptime. For production systems where downtime has a direct business cost, this commitment matters.

Why Developers Look for SearchAPI.io Alternatives

Despite those strengths, three friction points come up consistently once teams move from evaluation to production:

The 100-Search Free Tier Is Not Enough to Evaluate
Most competing SERP APIs offer 1,000 (Scrapingdog) + PAYG, 2,500 (Serper), or 100/month ongoing (SerpAPI) free searches. SearchAPI only gives you 100 searches once. That’s not enough to test against your real query set, benchmark response quality across different geographies, or validate the output schema against your parsing logic. You’re essentially being pushed to pay $40 before you’ve properly evaluated the product.
The 20% Per-Hour Rate Limit Is a Real Constraint
The 20% cap on monthly credits per hour usage can cause a constraint in your workflow. There can be some moments where, for a short period of time, you may need a large burst to handle the API requests, but you will hit a ceiling before you exhaust your credits for a month.
Cost Gap Widens Dramatically at Scale
At 100K searches/month, SearchAPI.io costs approximately $200–400, depending on your plan tier. Scrapingdog at the same volume costs ~$25–35. That’s a 6–10x pricing gap for comparable Google SERP data quality. For projects where volume scales quickly, the compounding cost difference is significant.

What to Look for in a SearchAPI.io Alternative

Use this framework when evaluating your options:

Pricing model: Subscription vs. PAYG. Check the credit rollover policy and overage fees.
Rate limits: Per-hour, per-day, or concurrent request limits.
Endpoint coverage: Does it cover the search surfaces you actually need?
Response time: Measure at p50 and p95, not just median. Burst latency matters for real-time workflows.
Output richness: Does it cover featured snippets on Google like AI Overviews, People Also Ask, Knowledge Graph and much more?
Free tier: Can you properly evaluate it before paying?

1. Scrapingdog — Best Overall Alternative to SearchAPI.io

Starting price: 40$ for 40k Searches
Free tier: 1,000 credits on signup, no credit card required
Best for: Teams that want lower cost, broader Google coverage, and other search engine coverage.

Scrapingdog’s Google SERP API is itself a powerful engine, powered by 25+ Google APIs, and is the most direct challenger to SearchAPI. The subscription can become 6–10x more economical at scale, with a credit rollover feature and no hourly rate caps, only a concurrency limit. Moreover, Scrapingdog also offers a pay-as-you-go system, allowing you to test the APIs thoroughly.

This doesn’t close the story. SERP is just one step in completing your data workflow, what if you need the content inside those links? That’s where the real game begins. Not all vendors in the market offer complete data coverage. With Scrapingdog’s powerful Web Scraping API, customers can directly scrape the content inside those links listed on SERP and complete their data workflow for research purposes.

Pricing comparison head-to-head:

Performance (our test results):
For LITE Search:

Avg response time: 1.6sec, 2.4sec (p95)
Success rate on Google SERP: 100%
For normal advanced search:

Avg response time: 2.7sec, 3.5sec (p95)
Success rate on Google SERP: 100%
Code example — migrating from SearchAPI.io to Scrapingdog:

import requests

# SearchAPI.io (current code)
def search_searchapi(query, location="United States"):
    response = requests.get(
        "https://www.searchapi.io/api/v1/search",
        params={
            "engine": "google",
            "q": query,
            "location": location,
            "api_key": "YOUR_SEARCHAPI_KEY"
        }
    )
    return response.json()


# Scrapingdog (drop-in replacement)
def search_scrapingdog(query, country="us"):
    response = requests.get(
        "https://api.scrapingdog.com/google",
        params={
            "api_key": "YOUR_SCRAPINGDOG_KEY",
            "query": query,
            "results": 10,
            "country": country,
            "language": "en"
        }
    )
    return response.json()

// Node.js - SearchAPI.io to Scrapingdog migration

// SearchAPI.io (old)
const searchSearchAPI = async (query) => {
  const params = new URLSearchParams({
    engine: "google",
    q: query,
    api_key: process.env.SEARCHAPI_KEY
  });
  const res = await fetch(`https://www.searchapi.io/api/v1/search?${params}`);
  return res.json();
};
// Scrapingdog (new)
const searchScrapingdog = async (query) => {
  const params = new URLSearchParams({
    api_key: process.env.SCRAPINGDOG_KEY,
    query: query,
    results: 10,
    country: "us"
  });
  const res = await fetch(`https://api.scrapingdog.com/google?${params}`);
  return res.json();
};

Response schema mapping:

Both APIs return structured JSON. The field names differ slightly:

SearchAPI.io field Scrapingdog field organic_results[] organic_data[] organic_results[].link organic_data[].link organic_results[].snippet organic_data[].snippet related_questions[] peopleAlsoAsk[] related_searches[] related_searches[] knowledge_graph knowledge_graph ai_overview ai_overview (dedicated endpoint)

Migration typically takes a minute; the schema is close enough that it’s a field rename exercise, not a rewrite.

Why choose Scrapingdog over SearchAPI.io:

5x cheaper at comparable volumes
True PAYG, in which credits don’t expire — pay for what you use.
10x more generous free tier, which covers web scraping and SERP APIs(1,000 credits vs. 100)
Faster responses from the API
A detailed output response coverin every inch of information from HTML.
Higher concurrency limit, allowing you to scrape even large number of results in a short period of time.

2. SerpAPI — Best for Maximum Engine Coverage

Starting price: $75/month for 5,000 searches ($15/1K)
Free tier: 100 searches/month (ongoing)
Best for: Teams that need 80+ search engines, the deepest feature set, and enterprise SLA

SerpAPI is the most established player in the SERP industry, having operated since 2017. It is backed by a strong portfolio of 100+ APIs, double the number offered by SearchAPI on its platform. If your project requires an enterprise-level of scraping, maximum output richness, SerpAPI can be the best implementation.

Where SerpAPI is clearly ahead of SearchAPI.io:

80+ search engine APIs vs. SearchAPI.io’s ~50 — including Naver, DuckDuckGo, Yahoo, and niche engines SearchAPI.io doesn’t cover
Ludicrous Speed mode — a premium low-latency option for real-time applications
More comprehensive output parsing — sitelinks, dates, video carousels, discussion threads, event results, and inline images are all covered by SearchAPI, but SerpAPI tries to get you the most out of the page. They consider every piece of information as an important data point.

The honest tradeoff:

SerpAPI is the most expensive option on this list by a significant margin, $15/1K at the Developer tier, $9/1K at the Big Data tier. For most teams, the 100+ engine coverage is more than they need, and they’re paying a premium for features they never use. SearchAPI.io already covers the search engines most projects actually need at roughly a third of the cost.

Pricing Breakdown:

When to choose SerpAPI over SearchAPI.io:

You need multi-engine APIs.
You’re already integrated on SerpAPI’s SDK, and the migration cost exceeds the price premium
Budget is not a primary constraint

3. Serper.dev — Best for Speed + Budget at Google-Only Scale

Starting price: $50 for 50,000 searches (~$1/1K)
Free tier: 2,500 searches (most generous free tier on this list)
Best for: Google-only workflows at high volume where response time is critical

Serper.dev is the fastest option on this list, with an average response time of nearly 1 second, driven by a lightweight architecture optimised for a single search engine. If your use case is Google-only and latency is the primary metric, Serper is hard to beat.

However, it is worth noting that Serper only covers the Google LITE page, not the current normal one. It has only 12 APIs in its portfolio, which all target different pages of Google and not any other search engine. But the product is still worth its price.

Where Serper wins over SearchAPI.io:

Speed — consistently the fastest real-time SERP API in independent testing
Price — $1/1K is 2–4x cheaper than SearchAPI.io at equivalent tiers
Free tier — 2,500 searches vs. SearchAPI.io’s 100. You can properly evaluate Serper before paying
Simple pricing — flat rate, easy to predict

When to choose Serper.dev over SearchAPI.io:

You only need Google (no Bing, YouTube, or Amazon data)
Response time is the most important metric for your use case
You want the most generous free tier to evaluate before paying
Volume is high, and you need the best cost per search

4. DataForSEO — Best for Bulk SEO Workflows

Starting price: ~$60 per 100K searches (async queue)
Free tier: $1 credit
Best for: SEO agencies, rank trackers, and keyword intelligence platforms running bulk queries.

DataForSEO operates on a fundamentally different model from SearchAPI.io and the other providers on this list. Rather than a synchronous request-response API (submit a query, get results in 2–5 seconds), DataForSEO uses a task queue system: you submit queries in batches and retrieve results when they’re ready, which can take sometime depending on the number of queries submitted.

The setup may sound like a downside for real-time applications. But it is not, for SEO firms running large keyword tracking, weekly site audits, async processing is what they need instead of real-time data, which helps them significantly drive the cost lower at scale.

Why DataForSEO costs less:
The async model lets DataForSEO optimise server utilisation across time zones, batching similar queries and avoiding redundant requests. The Standard Queue is the cheapest tier; the Live API (real-time) costs more but is still significantly cheaper than SearchAPI.io.

Pricing at scale comparison:

Beyond SERP — DataForSEO’s broader data platform:

DataForSEO isn’t just a SERP API. It also provides keyword difficulty scores, search volume data, backlink indexes, on-page SEO data, and SERP feature tracking. If you’re building an SEO product and need SERP data alongside keyword intelligence, DataForSEO is a one-stop shop that SearchAPI.io isn’t.

When to choose DataForSEO over SearchAPI.io:

Your workload is batch/async — not real-time
Volume is high (100K+ queries/month), and cost is the primary constraint
You need keyword research and backlink data alongside SERP results
You’re building an SEO platform, not a real-time consumer application

5. Bright Data — Best Enterprise Data Platform

Starting price: Custom (contact sales)
Free tier: Trial only
Best for: Enterprise organisations with compliance requirements and global geo-targeting needs

Bright Data is the biggest player in this whole web scraping market. They have a huge network of 150M+ IPs across 195 countries, which is the largest in the industry. And they have not sourced their IPs from any third-party provider, but they completely own their IPs, which gives them supreme flexibility in organising across their platform.

Bright Data can be expensive for small-to-medium scale enterprises, and the platform complexity makes the onboarding difficult. However, the super-large infrastructure makes their API super scalable in the market. It has various capabilities wrt to captcha solving technologies, browser fingerprinting, which most others can’t even match in the market.

What makes Bright Data genuinely different:

150M+ IP coverage — residential, datacenter, ISP, and mobile IPs across 195 countries. For geo-specific SERP data (local SEO, country-specific search results), this is unmatched.
Compliance documentation — SOC 2, GDPR, and CCPA paperwork that enterprise procurement requires.
Dedicated account management — a real human who knows your workload.

When to choose Bright Data over SearchAPI.io:

Enterprise scale with compliance requirements
Geo-targeting at a granularity (city, ISP, carrier-level) that smaller providers can’t support
You need SERP as part of a broader web data strategy

Head-to-Head: Scrapingdog vs. SearchAPI.io

Since this is the comparison most readers are here for, here’s the detailed breakdown:

The hourly rate limit in practice:

SearchAPI.io limits you to 20% of your monthly credits per hour. On the $40/month entry plan, that works out to roughly 800–1,000 searches per hour maximum. If your pipeline is bursty, a batch job that runs at 2am across 5,000 URLs before the business day starts — you’ll be throttled mid-run even with credits remaining. Scrapingdog has no hourly cap. You can burn 10,000 credits in an hour if your workload demands it.

Which Alternative Should You Choose?

All the providers mentioned in the list have one of the best web scraping services in the market. Scrapingdog and DataForSEO are known for a scalable API at low market prices, SerpAPI and BrightData are known for enterprise-scale volume, and Serper is known for its cheap and fast API in the market.
You should thoroughly test out all the vendors to be sure about the service, to know if the API would be a good value at scale. Factors such as concurrency, pricing, support and data coverage should be considered before taking a long-term decision.

Frequently Asked Question

Is there a free alternative to SearchAPI.io?

Yes, Scrapingdog provides you 100 free credits on registering on its platform without requiring a credit card. Serper and SerpAPI also offer a free tier with 2500 and 100 API calls, respectively. For genuinely free at higher volume, DataForSEO provides $1 in credit with no card required.

What is the cheapest SearchAPI.io alternative?

At 100K+ or 1M+ search volumes, Scrapingdog stands out as the most economical option among the competitors, providing services at a cost roughly 5x cheaper than SearchAPI at comparable volumes. DataForSEO’s Standard Queue is cheaper still for async/batch workloads that don’t need real-time results.

Does Scrapingdog cover the same endpoints as SearchAPI.io?

Scrapingdog covers 25+ APIs of Google, and has a total portfolio of 70+ APIs on its platform. It covers all the APIs offered by SearchAPI, including Google, Bing, YouTube, and Amazon, with a powerful General web scraping API that allows you to complete your data workflow and get all the information from a single platform.

Conclusion

SearchAPI is a great product and a highly scalable SERP API, backed by genuine strengths such as a 99.9% SLA, ZeroTrace mode, and 35+ APIs, which earn it a strong place in the market.

However, for the majority of developers, it is quite expensive compared to competitors offering the same or even better quality services. Plus, the 20% per-hour cap adds operational friction that other providers do not have.

Scrapingdog offers the best combination of pricing, Google endpoint breadth, PAYG flexibility, and a free tier large enough to properly evaluate the service before spending anything.

For developers and enterprises reading this article, Scrapingdog provides great flexibility while also being far more economical at scale, maintaining a 100% success rate with unmatched speed in the market. Additionally, the wide range of APIs covered by Scrapingdog gives customers the advantage of accessing all their data through a single platform.

Test it free with 1,000 credits at scrapingdog.com. No credit card required!

How to Scrape Google Search Results with Node.js

Darshan Khandelwal — Tue, 05 May 2026 13:20:02 +0000

Google Search is one of the most valuable sources of publicly available data, from SERP rankings and competitor analysis to keyword research and lead generation. But scraping it is not straightforward. Google actively blocks bots, changes its HTML structure, and heavily relies on JavaScript rendering.

In this guide, you’ll learn how to scrape Google Search results using Node.js and later with the Google SERP API. If you prefer Python, check out our guide on scraping Google Search with Python.

TL;DR

If you’re short on time or already understand the scraping logic, you can simply copy this code and use it in your environment.

const axios = require('axios');

const params = {
  api_key: 'your-api-key',
  query: 'what is scraping?',
  country: 'us',
  advance_search: 'true',
  domain: 'google.com'
};

axios.get('https://api.scrapingdog.com/google', { params })
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.error('Error:', error.message);
  });

Don’t forget to use your own API key.

Challenges of Scraping Google Search

Bot Detection — Google fingerprints requests using IP reputation, request headers, and behavioral patterns.
JavaScript Rendering — Results are JS-rendered, so raw HTTP libraries like Axios return incomplete HTML. A full browser environment is required.
CAPTCHA — Suspicious traffic triggers CAPTCHA, which becomes a major blocker at scale.
Frequent DOM Changes — Google updates its HTML structure without notice, breaking selectors regularly.
IP Blocking — High request volumes from a single IP get it banned fast. Proxy rotation is almost mandatory.
Geo-Restricted Results — SERPs vary by location, so region-specific scraping requires geo-targeted proxies.

Prerequisites

Before getting started, make sure you have the following:

Node.js (v18 or above) and npm installed on your machine. If you are new to JavaScript scraping, you can first read our guide on web scraping with Node.js.
Plawright for rendering the Google search page. You can install it with the command npm install playwright.
Axios for making an HTTP connection with the host website. You can install it with the command npm install axios.
Cheerio for parsing the raw HTML. Cheerio can be installed via npm install cheerio.
A Scrapingdog API key. You can get this by signing up for the trial pack.

Method 1: Scraping Google Search with Playwright and Cheerio

First, we will fetch the raw HTML using Playwright. Then, we will parse that HTML using Cheerio. If you want to learn the basics of Playwright first, check out our guide on web scraping with Playwright.

For this tutorial, we will scrape this page from Google.

const { chromium } = require('playwright');
const cheerio = require('cheerio');

async function scrapeGoogleSearch(query) {
  const browser = await chromium.launch({
    headless: false,
    channel: 'chrome'
  });

  const page = await browser.newPage();

  await page.setViewportSize({ width: 1920, height: 1080 });

   // Navigate to search results
  const searchUrl = `https://www.google.com/search?q=${query}&gl=us&hl=en`;
  await page.goto(searchUrl, { waitUntil: 'domcontentloaded' });

  const html = await page.content();
  await browser.close();
  return html;

}

// Run
scrapeGoogleSearch('spaceX').then(results => {
  console.log(JSON.stringify(results, null, 2));
}).catch(console.error);

Here is what the code is doing:

Launch a real Chrome browser- Playwright launches an actual Chrome instance (channel: 'chrome') with headless: false, meaning the browser runs visibly. This avoids the bot detection that headless mode often triggers.
Open a new page and set viewport- A new browser tab is created and the viewport is set to 1920×1080, a standard desktop resolution that mimics a real user.
Build the search URL and navigate- The query is embedded into a standard Google Search URL with gl=us&hl=en to lock results to English (US). Playwright navigates to it and waits until the DOM is loaded.
Extract the full page HTML- Once the page is rendered, page.content() grabs the complete HTML including all JS-rendered content — something raw HTTP libraries cannot do.
Close the browser and return HTML- The browser is closed cleanly and the raw HTML is returned.

If you see this after running the code, then it means your Playwright setup is working.

Parsing with Cheerio
We will parse three things:

Title
Description
Link

The title is wrapped inside the h3 element with the class LC20lb.

The description is wrapped inside the div tag with the class VwiC3b.

The link is wrapped inside a tag with the class zReHs.

The entire page is wrapped inside a div container with id search.

According to the information gathered above, the Cheerio logic will be:

const $ = cheerio.load(html);
const results = [];

$('#search .tF2Cxc').each((i, el) => {
  const title = $(el).find('h3.LC20lb').first().text();
  const link = $(el).find('a.zReHs').first().attr('href');
  const description = $(el).find('div.VwiC3b').first().text();

  if (title && link) {
    results.push({ title, link, description });
  }
});

return results;

After running the complete code, you get this output:

[
  {
    "title": "SpaceX",
    "link": "https://spacex.com/",
    "description": "SpaceX's Starship spacecraft and Super Heavy rocket is a fully reusable transportation system designed to carry both crew and cargo to Earth orbit, the Moon, ..."
  },
  {
    "title": "SpaceX",
    "link": "https://en.wikipedia.org/wiki/SpaceX",
    "description": "SpaceX was founded by Elon Musk in 2002 with a vision of decreasing the costs of space launches, paving the way to a self-sustaining colony on Mars. In 2008, ...Read more"
  },
  {
    "title": "SpaceX (@spacex) • Instagram photos and videos",
    "link": "https://www.instagram.com/spacex/",
    "description": "17M followers · 3 following · 996 posts · @spacex: “SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft”"
  }
]

Complete Code

const { chromium } = require('playwright');
const cheerio = require('cheerio');

async function scrapeGoogleSearch(query) {
  const browser = await chromium.launch({
    headless: false,
    channel: 'chrome'
  });

  const page = await browser.newPage();

  await page.setViewportSize({ width: 1920, height: 1080 });


  // Navigate to search results
  const searchUrl = `https://www.google.com/search?q=spaceX&gl=us&hl=en`;
  await page.goto(searchUrl, { waitUntil: 'domcontentloaded' });


  const html = await page.content();
  await browser.close();

  // Parse results with Cheerio
  const $ = cheerio.load(html);
  const results = [];

  $('div#search .tF2Cxc').each((i, el) => {
    const title = $(el).find('h3.LC20lb').first().text();
    const link = $(el).find('a.zReHs').first().attr('href');
    const description = $(el).find('div.VwiC3b').first().text();

    if (title && link) {
      results.push({ title, link, description });
    }
  });

  return results;
}

// Run
scrapeGoogleSearch('spaceX').then(results => {
  console.log(JSON.stringify(results, null, 2));
}).catch(console.error);

Method 2: Scraping Google Search with Scrapingdog

The approach above works well if you’re scraping a few hundred pages. But once you scale to thousands or millions of pages, a Playwright + Cheerio setup starts to break down. Here, we’ll use a Google SERP API that handles the hard parts for us, including proxies, headers, and browser management.

Here, we will use Axios to make the GET request to the Scrapingdog’s Google SERP API.

Want to see the full workflow in action? Watch this video tutorial to learn how to Scrape Google Search Results with Node.js & Scrapingdog’s Google SERP API

const axios = require('axios');

async function scrapeGoogleSearch(query) {
  const response = await axios.get('https://api.scrapingdog.com/google', {
    params: {
      api_key: 'YOUR_API_KEY',
      query: query,
      results: 10,
      country: 'us',
    }
  });

  const results = response.data.organic_results.map(result => ({
    title: result.title,
    link: result.link,
    description: result.snippet
  }));

  return results;
}

scrapeGoogleSearch('web scraping tools').then(results => {
  console.log(JSON.stringify(results, null, 2));
}).catch(console.error);

As you can see, with Scrapingdog, we only had to make a simple GET request. There was no need to manage a browser or write custom parsing logic. Running this code will produce a beautiful JSON response that contains every little detail, like Google Ads, AI overview, Video links, people also ask, etc.

{
  "search_information": {
    "time_taken": 0.3,
    "total_results": 2110,
    "query_displayed": "spaceX",
    "organic_results_state": "Results for exact spelling",
    "original_query_yields_zero_results": false,
    "url": "https://www.google.com/search?q=spaceX&oq=spaceX&gl=us&hl=en&sourceid=chrome&ie=UTF-8"
  },
  "peopleAlsoAskedFor": [
    {
      "question": "Is SpaceX fully owned by Elon Musk?",
      "rank": 1
    },
    {
      "question": "Can I buy stock in SpaceX?",
      "rank": 2
    },
    {
      "question": "What is the real purpose of SpaceX?",
      "rank": 3
    },
    {
      "question": "What's the average salary of a SpaceX employee?",
      "rank": 4
    }
  ],
  "organic_results": [
    {
      "title": "SpaceX",
      "link": "https://spacex.com/",
      "displayed_link": "https://spacex.com",
      "favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA10lEQVR4Ad2QIQiEQBBFfweDxXrNds1qsJoslu1mk71tEOxBMG41W8yb7WzcaBeEvRkXRLy7zt1LH97O/mHwO6TK4AvVsjkHhPXEdNEpgrgjs+uUc7c6QhdeRUVPatUlPJVma2VMWSh6us01ThLFdq2QK0thkSGuSOuMTHrDb4YMN0SbimXjbhHgnYHV3uMDpXGEzVKZ40Yodx5b0Fhnm9up/LYK2fH1EF+X0YdrAzx6y3EWp6t8XRP4ozrCSHjq9ZgT50G4fhufR984E9Nlx3qaiS7GH/ECeAedlKpi2RoAAAAASUVORK5CYII=",
      "source": "SpaceX",
      "snippet": "SpaceX's Starship spacecraft and Super Heavy rocket is a fully reusable transportation system designed to carry both crew and cargo to Earth orbit, the Moon, ...",
      "highlighted_keywords": [
        "SpaceX's"
      ],
      "extended_sitelinks": [
        {
          "title": "Launches",
          "link": "https://www.spacex.com/launches",
          "snippet": "Home to SpaceX headquarters and one of the world's first ..."
        },
        {
          "title": "Careers",
          "link": "https://www.spacex.com/careers",
          "snippet": "SpaceX designs, manufactures and launches advanced rockets ..."
        },
        {
          "title": "Mission",
          "link": "https://www.spacex.com/mission",
          "snippet": "SpaceX designs, manufactures and launches advanced rockets ..."
        },
        {
          "title": "Starship",
          "link": "https://www.spacex.com/vehicles/starship",
          "snippet": "Starship is the world's most powerful launch vehicle ever ..."
        },
        {
          "title": "Updates",
          "link": "https://www.spacex.com/updates",
          "snippet": "SpaceX designs, manufactures and launches advanced rockets ..."
        }
      ],
      "rank": 1,
      "page_rank": 1
    },
    {
      "title": "SpaceX",
      "link": "https://en.wikipedia.org/wiki/SpaceX",
      "displayed_link": "https://en.wikipedia.org › wiki › SpaceX",
      "favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAnklEQVR4AeTNIQiDQABG4b+u17X1aF6PK3YEO9iMJqPVau82y4FgMezS0oVLhqsHtrcqeqzDXv3CEz/6L4yTtZM3dnHmPTtjzXZAXKYVo4agkU2GI2Lloc6JDez1+flswMu1EQZ3xlE7lK8eKDkjtwE+crBMV+wesKmCiisGGepZIfQJpMj9SNb2MYWrChjVkULuCyCfRvsdmBieyQQAsoDk/9ryhFMAAAAASUVORK5CYII=",
      "source": "Wikipedia",
      "snippet": "SpaceX was founded by Elon Musk in 2002 with a vision of decreasing the costs of space launches, paving the way to a self-sustaining colony on Mars. In 2008, ...Read more",
      "highlighted_keywords": [
        "SpaceX"
      ],
      "rank": 2,
      "page_rank": 2
    },
    {
      "title": "SpaceX (@spacex) • Instagram photos and videos",
      "link": "https://www.instagram.com/spacex/",
      "displayed_link": "17.1M+ followers",
      "favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAMAAABF0y+mAAABcVBMVEVHcEx3GPlwGPx1FvyDC/OWBumqA+K+AeDQANvVAs7rDa7jAtfoAMflAriKGP2ZCvunAvLEG/TgV9/oZt72G8nxB8nxALCKHPymGf7Pg/H86vf////zds79AcD9ALT3DZ30t+j63fP1st31quT8AKf8AJn98/jaO973Lr38AIvAAOz8Drn4V7370PHiGe34E6n7HZv9AHv5wOP1Wpr8D5H9AGz9GYX6q8r2YKL5dZP3tMT8D2/8KW31Y5f9AV/9Inn+G17+KFf3v879J0f+MUf5g2v3wbr9Nyz6RUP9Ml/9Qxr+SjX7zcr6Gkb8AEL+UQ/9Wiv98uT70rD8XQT8ZxD9ZCr40sP4Tn/9cAz+egj9ciX1km/1X1L9O0j4nD7+hAT9hRb8fh34unb9jQT3jZX73a78dwX+mwL6ymPzoZD8YSf9qAD+sQD7wBL8zEn5tV79jwb+uAD+vgH+wwD+LHP8owX+ygD6Olj7rQz+twf8UjMSMf5jAAAAe3RSTlMAW9L////////FW///2P///////////8b/////////xf///////////////////////////v/////////////////////////////////////////////////////////////////F////W////f//2f///9fY/9hbxcXA5ZvtAAACBUlEQVR4AUTPRaLkNhSF4f9IV3Kp/JjfCzMzM8xCG8lWMs9SsoIkTdNmZubiKtvqduMnZl0BkgCBEI+zhiBwD5f0ZA3wQj1Eu1YiMZQADyAA9SRXgpCEaGXcODFKoq/NEXRHXYmnshMak0xdGC/PiO1Mi1rTbA0LyM/BOOWyuISrHyLPTRoKZDaBndvoQrezQA/IWlBe6DXCsOdgong+bdCLyzW5F3VzpY7TckLH4ESRlvsbNlj2V4DNu3klk/Ed8HUvYlVKwa4NSmedWzNX1dbUcxlzsJMgxNppsywKv6l5M9YJZtizxOfObknxVOkVaAblle1BB0JNssXaJhA05tlKMOrcIXcMSOBc8IBvqzbQZC8SjIfMwxxz0iLnn0vQS+cx7ye0nPcPT7qPBmk2my7b4A0XlmiNzEMDQfD53vLF6sigLHLGMyzBCMzaawXf/HcAPqpmvqbDq5dGlQVjlexMYvqD0C3P+iVOccGBM0L1+d6uYozzRc6TbvA3/cI0r7g4dgoKy/SCee+akH32ltf38muG0kzR+PUfPl+DHq1in3jnSp8ZtjAxgz+u7OUJfb51cmk0Tqf158QwgcQUmM5zy9+tqnr9+nCf+3seU4hFCKHsptStxoEQg4fPhp7dP1dB8u6+LFzjqOq6afof/IWH/38O40irXXZ1g2+q99K9TWBgAACS27uoe/IOhgAAAABJRU5ErkJggg==",
      "source": "Instagram · spacex",
      "snippet": "17M followers · 3 following · 996 posts · @spacex: “SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft”",
      "highlighted_keywords": [
        "spacex",
        "SpaceX"
      ],
      "rank": 3,
      "page_rank": 3
    },
    {
      "title": "SpaceX",
      "link": "https://www.youtube.com/spacex",
      "displayed_link": "6.8M+ followers",
      "favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAcElEQVR4AWP4//8/RZh6BgCZAkDsAMUNWDFCXgDFACCV8J/B+D8pGKwHRAKRAUyQDEMMQAYEUGBAAsiABpwKHjz4/9/BAZ8BDXgNgIMNGyg04MABkg1AeCEgAK8XKA5EiqORooSELykXEJuUBz43AgAIA1ZhBoG9vwAAAABJRU5ErkJggg==",
      "source": "YouTube · SpaceX",
      "snippet": "SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft. ...more SpaceX designs, manufactures and launches the world's ...Read more",
      "highlighted_keywords": [
        "SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft"
      ],
      "rank": 4,
      "page_rank": 4
    },
    {
      "title": "SpaceX (@SpaceX) / Posts / X",
      "link": "https://x.com/SpaceX",
      "displayed_link": "41.6M+ followers",
      "favicon": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA/0lEQVR4AbXPIazCMACE4d+L2qoZFEGSIGcRc/gJJB5XMzGJmK9EN0HMi+qaibkKVF1txdQe4g0YzPK5yyWXHL9TaPNQ89LojH87N1rbJcXkMF4Fk31UMrf34hm14KUeoQxGArALHTMuQD2cAWQfJXOpgTbksGr9ng8qluShJTPhyCdx63POg7rEim95ZyR68I1ggQpnCEGwyPicw6hZtPEGmnhkycqOio1zm6XuFtyw5XDXfGvuau0dXHzJp8pfBPuhIXO9ZK5ILUCdSvLYMpc6ASBtl3EaC97I4KaFaOCaBE9Zn5jUsVqR2vcTJZO1DdbGoZryVp94Ka/mQfE7f2T3df0WBhLDAAAAAElFTkSuQmCC",
      "source": "X · SpaceX",
      "snippet": "SpaceX (@SpaceX) - Posts - SpaceX designs, manufactures and launches the world's most advanced rockets and spacecraft | X (formerly Twitter)",
      "highlighted_keywords": [
        "SpaceX",
        "SpaceX",
        "SpaceX"
      ],
      "rank": 5,
      "page_rank": 5
    }
  ],
  "relatedSearches": [
    {
      "query": "Spacex news",
      "link": "https://www.google.com/search?gl=us&q=Spacex+news"
    },
    {
      "query": "Spacex falcon heavy launch",
      "link": "https://www.google.com/search?gl=us&q=Spacex+falcon+heavy+launch"
    },
    {
      "query": "SpaceX owner",
      "link": "https://www.google.com/search?gl=us&q=SpaceX+owner"
    },
    {
      "query": "SpaceX launch",
      "link": "https://www.google.com/search?gl=us&q=SpaceX+launch"
    },
    {
      "query": "SpaceX Starship",
      "link": "https://www.google.com/search?gl=us&q=SpaceX+Starship"
    },
    {
      "query": "SpaceX stock",
      "link": "https://www.google.com/search?gl=us&q=SpaceX+stock"
    },
    {
      "query": "SpaceX careers",
      "link": "https://www.google.com/search?gl=us&q=SpaceX+careers"
    },
    {
      "query": "SpaceX IPO",
      "link": "https://www.google.com/search?gl=us&q=SpaceX+IPO"
    }
  ]

}

How to Handle Pagination?

To scrape all the pages from Google, you can use the page parameter from Scrapingdog. You can read more about it in the documentation.

async function scrapeMultiplePages(query, pages = 3) {
  const allResults = [];

  for (let i = 0; i < pages; i++) {
    const response = await axios.get('https://api.scrapingdog.com/google', {
      params: {
        api_key: 'YOUR_API_KEY',
        query: query,
        results: 10,
        country: 'us',
        page: i
      }
    });

    const results = response.data.organic_results.map(result => ({
      title: result.title,
      link: result.link,
      description: result.snippet
    }));

    allResults.push(...results);
  }

  return allResults;
}

scrapeMultiplePages('web scraping tools', 3).then(results => {
  console.log(`Total results: ${results.length}`);
  console.log(JSON.stringify(results, null, 2));
}).catch(console.error);

With this approach, pagination is just incrementing the page parameter; no browser delays, no block risk, no extra complexity.

Parsing Different Result Types

The Scrapingdog API response contains much more than just organic results. Here’s how to parse each result type from the JSON:

Organic Result

const organic = response.data.organic_results.map(result => ({
  rank: result.rank,
  title: result.title,
  link: result.link,
  snippet: result.snippet
}));

console.log('Organic Results:', organic);

People Also Ask

const paa = response.data.peopleAlsoAskedFor.map(item => ({
  rank: item.rank,
  question: item.question
}));

console.log('People Also Ask:', paa);

Related Searches

const related = response.data.relatedSearches.map(item => ({
  query: item.query,
  link: item.link
}));

console.log('Related Searches:', related);

Sitelinks (only on top branded results)

const topResult = response.data.organic_results[0];

if (topResult.extended_sitelinks) {
  const sitelinks = topResult.extended_sitelinks.map(s => ({
    title: s.title,
    link: s.link,
    snippet: s.snippet
  }));
  console.log('Sitelinks:', sitelinks);
}

Search Metadata

const meta = response.data.search_information;

console.log(`Query: ${meta.query_displayed}`);
console.log(`Total Results: ${meta.total_results}`);
console.log(`Time Taken: ${meta.time_taken}s`);

The advantage of the Scrapingdog API is that all these result types come back in a single structured response, no need to write separate CSS selectors for each one or worry about Google changing its DOM structure.

Export the scraped data to CSV
Node.js has a built-in fs module so no extra dependencies are needed for this step.

const fs = require('fs');
const axios = require('axios');

function convertToCSV(results) {
  const headers = ['Rank', 'Title', 'Link', 'Snippet'];

  const rows = results.map(r => [
    r.rank,
    `"${r.title.replace(/"/g, '""')}"`,
    r.link,
    `"${r.snippet.replace(/"/g, '""')}"`
  ]);

  return [headers.join(','), ...rows.map(row => row.join(','))].join('\n');
}

async function scrapeAndExport(query) {
  const response = await axios.get('https://api.scrapingdog.com/google', {
    params: {
      api_key: 'YOUR_API_KEY',
      query: query,
      results: 10,
      country: 'us'
    }
  });

  const results = response.data.organic_results.map(r => ({
    rank: r.rank,
    title: r.title,
    link: r.link,
    snippet: r.snippet
  }));

  const csv = convertToCSV(results);
  const filename = `${query.replace(/\s+/g, '_')}_results.csv`;

  fs.writeFileSync(filename, csv, 'utf8');
  console.log(`Saved ${results.length} results to ${filename}`);
}

scrapeAndExport('spaceX').catch(console.error);

Once you run this code, you will find a CSV file inside the folder.

Conclusion

In this guide, we covered two ways to scrape Google Search results with Node.js: Playwright and the Scrapingdog API. Playwright is good for low-volume testing and learning, but breaks easily at scale due to Google’s aggressive bot detection and frequent DOM changes.

For production use, the Scrapingdog’s Google SERP API is the clear winner. It handles proxies, CAPTCHA, and browser fingerprinting for you, returning clean, structured JSON with organic results, People Also Ask, sitelinks, and pagination in a single call, so you can focus on using the data rather than fighting to get it.

Frequently Asked Questions

Is scraping Google Search results legal?

Scraping public Google data is legal. For commercial use at scale, using an API like Scrapingdog is the safer route.

Why does my scraper get blocked so quickly?

Google detects bots through IP reputation, request frequency, and browser fingerprinting. Without proxy rotation and human-like behavior, even a few requests will trigger a block.

Can I scrape Google Search results for free?

Yes, using Playwright or Puppeteer, but you’ll hit blocks fast without proxies. Scrapingdog offers a free tier to get started without any infrastructure setup.

How many results can I scrape per page?

Google returns 10 results per page by default. You can paginate using the page parameter page=0 for page 1, page=1 for page 2, and so on.

Do I need to parse HTML when using the Scrapingdog API?

No. The API returns structured JSON directly, so there’s no need to write CSS selectors or worry about Google changing its DOM structure.

How to Scrape ChatGPT with Python

Darshan Khandelwal — Wed, 22 Apr 2026 11:46:53 +0000

ChatGPT has become one of the most used AI platforms on the internet. Developers, researchers, and marketers are increasingly interested in extracting ChatGPT responses at scale. Whether to track what ChatGPT says about their brand, build AI monitoring pipelines, or power LLM-based research workflows.

The problem? Scraping ChatGPT manually is painful. It requires browser automation, anti-bot bypass logic, session management, and handling dynamic content, all of which break frequently as OpenAI updates its platform.

In this guide, you will learn how to scrape ChatGPT with Python and Scrapingdog’s ChatGPT Scraper API, a dedicated solution that handles all the complexity for you, so you can extract ChatGPT responses with a single API call.

Why Scrape ChatGPT?

Most people use ChatGPT interactively, but querying it programmatically at scale unlocks real business value. Here’s why developers and companies do it:

Brand & AI Visibility Monitoring — Many users ask ChatGPT for recommendations instead of Googling. Scraping lets you track whether ChatGPT mentions your brand or your competitors.

AI Response Research — Study LLM behavior, bias, or hallucinations by collecting responses to hundreds of prompts systematically.

Competitor Intelligence — You can query ChatGPT about competitor products, pricing, and features, and combine it with structured product data from APIs like Amazon Scraper API to validate what ChatGPT recommends against real-time pricing and listings.

Training Data & Benchmarking — Use ChatGPT responses as reference data when fine-tuning or benchmarking your own models.

Automated Content Pipelines — Feed ChatGPT responses into enrichment or summarization workflows without manual copy-pasting.

What Is Scrapingdog’s ChatGPT Scraper API?

Scrapingdog has recently launched a dedicated ChatGPT Scraper API . You just have to pass a query/prompt to the API and in return you will get clean, structured JSON, without dealing with any of the underlying browser automation or anti-bot complexity.

Key features:
Submit any prompt and get ChatGPT’s response back as structured data
No browser automation required .
Handles Cloudflare restrictions.
Supports concurrent requests for high-volume workflows
Returns clean JSON output ready for downstream processing
Python-friendly with simple requests integration
This means you can focus entirely on your data pipeline, not on fighting OpenAI’s anti-scraping stack.

Prerequisites
Before you start, make sure you have:

Python 3.8+ installed
A Scrapingdog account with an API key.
The requests library installed:

pip install requests

That’s it. No Playwright, no Selenium, no proxy configuration required.

Scraping ChatGPT with Python

Let’s start with the simplest possible example by sending a single prompt to ChatGPT and printing the response.

We will start from the dashboard.

Once you pass the query/prompt in the scraper you will get a ready to use python code.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/chatgpt"

params = {
    "api_key": api_key,
    "prompt": "What are the best web scraping tools in 2026?"
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

This code sends a prompt to Scrapingdog’s ChatGPT Scraper API and prints the response.

Here’s the flow:

Sets up credentials — your API key and the endpoint URL.
Defines params — the API key and the prompt you want to send to ChatGPT.
Makes a GET request — sends the prompt to the API via requests.get().
Handles the response — if successful (status 200), parses and prints the JSON response; otherwise prints the error code.
Once you run this code you get this JSON response.

The response has two main fields:

conversation A structured array containing the full exchange — your prompt (user) and ChatGPT’s reply (assistant). The assistant’s response is broken down into typed content blocks:

paragraph — introductory or descriptive text
heading — section titles (e.g., “AI-Powered Scraping Tools”)
bullet_list — tools listed under each category with their strengths and best-use cases

markdown The exact same response, but as a raw markdown string — useful if you want to render it directly in a UI or blog without parsing the structured JSON.

Parsing the API Response

The response contains structured content blocks inside conversation[-1]["response"]. Here’s how to handle each block type — paragraph, heading, and bullet_list and extract clean text from them.

import requests

API_KEY = "your_scrapingdog_api_key"
ENDPOINT = "https://api.scrapingdog.com/chatgpt"


def parse_response_blocks(blocks: list) -> str:
    """
    Parse the assistant's structured response blocks into plain text.
    Handles: paragraph, heading, bullet_list
    """
    lines = []

    for block in blocks:
        block_type = block.get("type")

        if block_type == "paragraph":
            lines.append(block.get("text", ""))
            lines.append("")  # blank line after paragraph

        elif block_type == "heading":
            level = block.get("level", 2)
            prefix = "#" * level
            lines.append(f"{prefix} {block.get('text', '')}")
            lines.append("")

        elif block_type == "bullet_list":
            for item in block.get("items", []):
                heading = item.get("heading", "")
                text = item.get("text", "")
                # Skip duplicate sub-items (heading-only entries like "Strengths:", "Best for:")
                if text and heading not in ("Strengths:", "Best for:"):
                    lines.append(f"- **{heading}**: {text}" if heading else f"- {text}")
            lines.append("")

    return "\n".join(lines)


def query_chatgpt(prompt: str) -> dict:
    """
    Query ChatGPT via Scrapingdog and return a parsed result dict.
    """
    response = requests.get(
        ENDPOINT,
        params={"api_key": API_KEY, "prompt": prompt},
        timeout=90
    )
    response.raise_for_status()
    data = response.json()

    # Extract the assistant turn (always the last in the conversation)
    assistant_turn = data["conversation"][-1]
    response_blocks = assistant_turn.get("response", [])

    return {
        "prompt": data["conversation"][0]["message"],
        "markdown": data.get("markdown", ""),         # pre-rendered markdown string
        "parsed_text": parse_response_blocks(response_blocks),  # custom parsed text
        "raw_blocks": response_blocks                 # original structured blocks
    }


if __name__ == "__main__":
    result = query_chatgpt("What are the best web scraping tools in 2026?")

    print("Prompt:", result["prompt"])
    print("\n--- Parsed Text ---")
    print(result["parsed_text"])

    print("\n--- Markdown (pre-rendered) ---")
    print(result["markdown"][:500], "...")  # preview first 500 chars

Extracting Specific Block Types

If you only need specific content, for example just the headings to build a table of contents or just the bullet points, you can filter the blocks directly:

def extract_headings(blocks: list) -> list[str]:
    """Return all heading texts from the response."""
    return [b["text"] for b in blocks if b.get("type") == "heading"]


def extract_bullet_items(blocks: list) -> list[dict]:
    """Return all bullet list items across all bullet_list blocks."""
    items = []
    for block in blocks:
        if block.get("type") == "bullet_list":
            for item in block.get("items", []):
                # Filter out sub-label duplicates
                if item.get("text") and item.get("heading") not in ("Strengths:", "Best for:"):
                    items.append(item)
    return items


# Usage
result = query_chatgpt("What are the best web scraping tools in 2026?")
blocks = result["raw_blocks"]

print("Sections:", extract_headings(blocks))
print("Tools mentioned:", [i["heading"] for i in extract_bullet_items(blocks)])

Real-World Use Cases

Brand Monitoring on ChatGPT- Track what ChatGPT says about your company across different question angles:

brand_prompts = [
    "What is Scrapingdog?",
    "What are the best web scraping APIs?",
    "Is Scrapingdog a good tool for web scraping?",
    "What alternatives are there to Scrapingdog?",
    "What do people say about Scrapingdog?"
]

results = batch_query_chatgpt(brand_prompts)

Schedule this to run weekly or daily and compare the markdown responses over time to see how your ChatGPT visibility changes.

Competitor Research

competitor_prompts = [
    "Compare Scrapingdog vs Brightdata for web scraping",
    "What is the cheapest web scraping API in 2025?",
    "Which scraping API has the best Google SERP support?",
    "What are developers saying about Apify vs Scrapingdog?"
]

competitor_data = batch_query_chatgpt(competitor_prompts)

SEO & AI Visibility Research– As ChatGPT increasingly influences user decisions, knowing whether your content surfaces in its responses is critical. Pair this with Google Search Result Scraper to compare your rankings across both Google and ChatGPT simultaneously:

seo_prompts = [
    "Best Python tutorial sites in 2025",
    "Where can I learn web scraping online?",
    "Top blogs about Python and data engineering",
]

seo_visibility = batch_query_chatgpt(seo_prompts)

# Check if your brand appears in the markdown response
your_brand = "scrapingdog"
for r in seo_visibility:
    mentioned = your_brand.lower() in r["markdown"].lower()
    print(f"Prompt: {r['prompt'][:50]}")
    print(f"Mentioned: {' YES' if mentioned else ' NO'}\n")

Note: You should also use the Google News Scraping API to monitor your brand's mentions on the Internet.

Key Takeaways:

DIY scraping of ChatGPT is fragile — Cloudflare protection, streaming responses, and frequent UI changes make browser automation an unreliable approach for production use.
The API returns structured blocks, not plain text — responses come as typed content blocks (paragraph, heading, bullet_list) plus a pre-rendered markdown field; parse accordingly.
Use markdown for simplicity, raw_blocks for control — if you just need the text, grab data["markdown"]; if you need to extract headings or bullet items individually, iterate over conversation[-1]["response"].
Brand and AI visibility monitoring is the killer use case — as users shift from Google to ChatGPT for recommendations, tracking whether ChatGPT mentions your product is becoming as important as tracking your Google rankings.

Conclusion

Scraping ChatGPT manually is possible but fragile, browser automation breaks with every OpenAI update, session management is tedious, and handling streaming responses adds significant complexity.

Scrapingdog’s ChatGPT Scraper API eliminates all of that. With a single API call, you get ChatGPT’s response in clean JSON, ready to process in Python. Whether you’re building a brand monitoring pipeline, running AI research at scale, or tracking how ChatGPT talks about your product, this approach is production-ready from day one.

Frequently Asked Questions

Is scraping ChatGPT legal?
In this article, whatever we scraped was available publicly. Scrapingdog does not scrape anything that is behind a login screen.

Why not use the OpenAI API directly?
The OpenAI API generates responses; it doesn’t reflect what ChatGPT’s web interface says with web search enabled. Plus, scraping ChatGPT with Scrapingdog is almost 15x cheaper than using the API.

Will the API break when OpenAI updates its UI?
No. Scrapingdog maintains the underlying infrastructure, so UI changes on OpenAI’s end are handled on their side; your code stays untouched.

Can I use this with async Python?
Yes. Wrap the requests calls with httpx and asyncio for concurrent scraping.

What does the API return?
A conversation array (structured content blocks per turn) and a markdown field (full response as a ready-to-use Markdown string). See Scrapingdog’s docs for the full schema.

How to scrape Google AI Overviews using Python

Darshan Khandelwal — Fri, 13 Feb 2026 09:44:18 +0000

Google’s AI Overviews are changing how search results are displayed, delivering AI-generated summaries at the very top of the SERP. While these responses are helpful for users, they’re tricky to capture programmatically.

Whether you’re analyzing your brand presence, tracking how Google summarizes answers, or just experimenting with AI-generated content, scraping these overviews can unlock powerful insights.

In this tutorial, I will show you how you can scrape AI overviews from Google search using Python and Scrapingdog’s Google Search API.

What are AI Overviews?

AI Overviews are Google’s experimental feature that uses generative AI to answer search queries directly at the top of the search results page.

Instead of just showing blue links, Google summarizes information from various websites into a concise, natural-language response, often citing sources underneath.

Google introduced this feature in May 2024, initially for users in the United States. It was later rolled out to other countries as well. Currently, AI Overviews are available only in English, though Google may expand language support in the future.

Why Scrape Google AI Overviews?

Google’s AI Overviews are transforming how people consume information. Instead of relying completely on blue links, users now get AI-generated answers at the very top of search results, often without clicking anything.

This shift presents both a challenge and an opportunity.

Here’s why scraping AI Overviews matters:

📢 Track Brand Mentions
Know when your brand (or your competitor’s) is referenced in Google’s AI responses, even if you’re not ranking #1 organically.

🧠 Understand Search Intent Better
AI Overviews often reflect Google’s best guess at user intent. Scraping them gives you a window into how Google “thinks”.

But in the past, many answers from AI overviews have raised eyebrows too, and that made people question the reliability of AI-generated summaries and highlighted the need for better accuracy verification before deploying these features at scale.

✍️ Content & SEO Research
Identify what types of answers Google prefers, what sources it cites, and how it summarizes complex topics. Since AI overviews reduce organic traffic, it becomes necessary for any brand to appear in AI overviews.

🔍 Competitive Intelligence
Learn which companies or products consistently show up in AI-generated summaries and why. AI overviews have changed how this intelligence is gathered and how it impacts the overall SEO industry.

Why use Scrapingdog for scraping AI overviews?

Scraping AI Overviews isn’t easy, and they don’t appear on every search. They’re dynamically rendered, and they’re often wrapped in JavaScript-heavy containers. Traditional scrapers break. Headless browsers are slow and expensive.

That’s where Scrapingdog’s Google Search API stands out:

⚡ Fast & Scalable: Avoid setting up your own browsers and proxies. Scrapingdog handles that useless stuff so that you keep collecting the data at scale without hassle.

📦 Includes Source Attribution: Extracts citations, reference links, and summary text from the AI Overview box.

🔁 Works with All Search Parameters: Supports pagination, country targeting, device type, and more.

🧪 Great for Experiments & Monitoring: Track when Overviews appear, how they change, and which sites are cited.

Whether you’re building a dashboard to monitor brand mentions in Overviews or analyzing how AI rewrites search content, Scrapingdog gives you a reliable, high-speed way to access this data that too without browser automation and proxy setup nightmares.

Prerequisite
Python 3.x should be available on your machine. If not, then you can download it from here.
Install requests library for making HTTP connections with the API.
Account on Scrapingdog. You will get 1000 free credits on signup.

How to Scrape AI Overviews

To scrape Google AI overviews, you have to pass a sample query to Scrapingdog’s Google Search API. For this tutorial, we are going to use the query what is AI overview. The Google search page will look like this for this query.

Now, pass the query to the Google Search Scraper Playground.

As you can see in the above image, after typing the query in the input field, you will get a ready-to-use Python code. You can just copy this code and paste it into your Python file.

import requests

api_key = "your-api-key"
url = "https://api.scrapingdog.com/google"

params = {
    "api_key": api_key,
    "query": "what is AI overview",
    "country": "us",
    "advance_search": "true",
    "domain": "google.com"
}

response = requests.get(url, params=params)

if response.status_code == 200:
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

Do not forget to use your own API key in the above code.

Here’s a brief explanation of the code in points:

API Configuration:

Sets up the ScrapingDog API key for authentication
Defines the endpoint URL (https://api.scrapingdog.com/google) for Google SERP scraping
api_key: Your authentication credential from ScrapingDog
query: The search term to scrape (“what is AI overview”)
country: Target location for search results (US in this case)
advance_search: Enables the extraction of AI Overviews and other advanced features.
domain: Specifies which Google domain to scrape (google.com)
Once you run this code, you will get this neat JSON response.

AI overview response from Scrapingdog

Within the JSON response, you will get an object ai_overview which will have the data from the AI overview section. Look pretty aesthetic, right?🔥

Now, in some cases, you might not get this data. For example, check this JSON response from the Google Search API.

Handling the ScrapingDog AI overview Extension Link:

When AI Overviews aren’t immediately available in the main response, ScrapingDog provides a fallback mechanism through the scrapingdog_link field.

How It Works:

The initial response may contain a url (original Google URL) and a scrapingdog_link (extension API endpoint)
If the AI Overview is missing from the primary JSON response, use the scrapingdog_link to retrieve it
Make a simple GET request to this link with no additional parameters needed
The response will contain the AI Overview data in JSON format
Critical Timing:

⚠️ 60-Second Expiration: You must make the GET request to scrapingdog_link within 60 seconds of receiving it. After this window, the link expires and becomes invalid.

Usage Example:

# If AI overview is not in main response
if 'scrapingdog_link' in data:
    extension_response = requests.get(data['scrapingdog_link'])
    ai_overview_data = extension_response.json()

This two-step approach ensures you can always capture AI Overviews, even when they require additional rendering time from Google’s servers.

Key Takeaways

ScrapingDog’s SERP API simplifies the complex process of extracting AI Overviews by handling proxy management, anti-bot detection, and browser fingerprinting automatically
The advance_search parameter unlocks rich SERP features, including AI Overviews, featured snippets, Ads and knowledge panels
The two-step approach with scrapingdog_link ensures you never miss AI Overview data, even when it requires extended rendering time
With just a few lines of Python code, you can monitor AI Overviews at scale for competitive analysis, content strategy, and SEO optimization

Conclusion

Scraping Google AI Overviews has become essential for staying competitive in today’s SEO landscape. As Google continues to prioritize AI-generated content at the top of search results, understanding how your content appears or doesn’t appear in these overviews is crucial for visibility and traffic. With the help of Python and powerful SERP APIs from Scrapingdog, we were able to achieve our goal without the hassle of setting up a browser or proxy.

FAQs (Frequently Asked Questions)

What are Google AI Overviews?
Google AI Overviews are AI-generated summaries displayed at the top of Google search results. They combine information from multiple websites and include source links.
Why are AI Overviews hard to scrape?
AI Overviews are dynamically rendered using JavaScript and don’t appear for every query, making traditional HTML scrapers unreliable.
How can I scrape Google AI Overviews using Python?
You can scrape AI Overviews using Scrapingdog’s Google Search API by sending a search query and enabling advanced search features.
What can I use scraped AI Overviews for?
Scraped AI Overviews can be used for SEO research, brand monitoring, and competitor analysis.