DEV Community: Darshan Khandelwal

Top Ruby HTTP Clients to Use in 2026

Darshan Khandelwal — Wed, 16 Sep 2026 12:27:41 +0000

When building a web scraper with Ruby, one of the most important tools at your disposal is the HTTP client.

An HTTP client is a software library or framework that enables you to send and receive HTTP requests (GET, POST, PUT, etc) and responses to communicate with web servers.

With so many HTTP clients available in the Ruby ecosystem, it can be challenging to choose the best one for your project.

In this article, we’ll take a look at some of the best Ruby HTTP clients available and compare their features, performance, and ease of use.

Whether you’re building a simple web scraper or a complex RESTful API, choosing the right HTTP client can make all the difference in your project’s success. So, without further ado, let’s dive in!

Factors on which rank will be decided for Ruby HTTP Clients

Let me just define what set of factors I am going to consider in order to rank Ruby HTTP clients in decreasing order.

Performance– The library should be fast and lightweight. It should be able to handle a large number of concurrent requests without delaying the response.

Documentation– Clear and price documentation is another factor to benchmark any library. It should be well written so that developers can jump-start their work asap.

Community– The community should be large enough to cater to all the problems one might face while coding.

Github Star– At last, we will also look at the number of stars a library has. The number will help us understand its quality and perceived utility.

For testing the speed we are going to make GET and POST requests with libraries and then test the timing.

For the GET request, we are going to use this API and for the POST request, we are going to use this API.

You have to create a dedicated folder in which we will keep our ruby file. I am naming the file as check.rb. You can pick any name you like. To run the file you just have to open the folder in your terminal and type ruby check.rb and then hit enter.

Our setup is complete let’s start testing the libraries.

HTTParty

It is a ruby gem that is made on top of Net::HTTP library. It is super simple to use and comes with features like query parameters, request headers, and basic authentication. Let’s see how we can make a GET and a POST request with httparty and measure the time taken by the library to implement the task.

For measuring the time taken we will use the Benchmark library.

GET Request

require 'httparty'
require 'benchmark'

time = Benchmark.realtime do
  response = HTTParty.get('https://httpbin.org/get')
  puts response.body
end

puts "Request took #{time} seconds"

For this example, we have used realtime() method provided by the Benchmark library to measure the time taken by the request. It will return the number of seconds it took to complete the request.

Once I run this code I get Request took 0.398039 seconds on the terminal. That means this library took 0.398 seconds to complete the task. Let’s make a POST request now.

POST request

require 'httparty'
require 'benchmark'

time = Benchmark.realtime do
  response = HTTParty.get('https://httpbin.org/get')
  puts response.body
end

puts "Request took #{time} seconds"

Once I run this code I get Request took 0.435745 seconds on the terminal. So, this means that the library took around 0.436 seconds to complete the request.

The documentation of the library is very well written and it explains each step with an example. Other than that you can find great tutorials from other websites on httparty. This indicates the library has great community support.

HTTParty can automatically parse response bodies in various formats, including JSON, XML, and YAML, and return them as Ruby objects or hashes. Plus it can handle error messages by returning appropriate messages.

Overall any developer can kick-start his/her journey with this gem comfortably.

Faraday

This is another HTTP client that provides simple APIs for making HTTP connections with any web server. It has the capability to handle connection- timeout, errors, and it can even retry the request for you if the first connection could not go through successfully. The retry function is very helpful when it comes to web scraping. You can keep trying until a request status is 200.

It also provides adaptors for Typhoeus, Excon and Net::HTTP, it opens options for developers to choose an adaptor according to their own requirements.

Now let’s benchmark this library by making GET and POST requests.

require 'faraday'
require 'benchmark'

time = Benchmark.realtime do
  connection = Faraday.new('https://httpbin.org')
  response = connection.get('/get')
  puts response.body
end

puts "Request took #{time} seconds"

Once I run this code I get Request took 0.054039 seconds on the terminal. That means this library took 0.054 seconds to complete the task. Let’s make a POST request now.

require 'faraday'
require 'benchmark'

time = Benchmark.realtime do
  connection = Faraday.new('https://httpbin.org')
  response = connection.post('/post', {foo: 'bar'})
  puts response.body
end

puts "Request took #{time} seconds"

POST request with faraday took around 0.081 seconds. Well, the speed is just fantastic!

Apart from the speed the documentation of faraday is very well written.

It explains every method it has to offer with an example. Faraday also uses a middleware architecture that allows you to modify requests and responses in a flexible and composable way. You can add or remove middleware to customize the behavior of your requests.

While scraping any website at scale you have to modify the headers on every new request for that faraday provides a simple way to set custom headers and options for your requests, such as authentication credentials, timeouts, and SSL settings.

When you search for faraday on google, you will find many tutorials. This means that community support is also great for this library.

Overall, Faraday is a powerful and flexible library that can simplify the process of making HTTP requests and handling responses in your Ruby applications.

RestClient

It is another popular HTTP client library. With this library too you can make GET, POST, DELETE, etc requests to any http or https API endpoint.

RestClient also allows you to set a timeout for your requests, ensuring that your application doesn’t hang or become unresponsive if a request takes too long to complete.

Let’s see how this library performs with GET and POST requests.

require 'rest-client'
require 'benchmark'

time = Benchmark.realtime do
  response = RestClient.get 'https://httpbin.org/get'
  puts "Response code: #{response.code}"
end

puts "Request took #{time} seconds"

After running this code I am getting 0.173 seconds. Now, let’s see how this library performs with a POST request.

require 'rest-client'
require 'benchmark'

time = Benchmark.realtime do
  response = RestClient.post 'https://httpbin.org/post', { :param1 => 'value1', :param2 => 'value2' }
  puts "Response code: #{response.code}"
end

puts "Request took #{time} seconds"

It took around 0.1898 seconds to make the POST request.

Just like Faraday, RestClient also allows developers to set custom headers and parameters for HTTP requests, which makes it flexible and customizable for different use cases.

I did not find any major tutorials on RestClient and the documentation is not so well written.

Typhoeus

Typhoeus is a Ruby gem that can make parallel HTTP requests with ease. Since it is built on top of libcurl library you can make asynchronous calls. It means you can make multiple API calls and then handle the response as they arrive.

Let’s check its performance with a GET request.

require 'typhoeus'
require 'benchmark'

time = Benchmark.realtime do
  response = Typhoeus.get('https://httpbin.org/get')
  puts "Response code: #{response.code}"
  puts "Response body: #{response.body}"
end

puts "Request took #{time} seconds"

So, it took around 0.1282 seconds to implement the request. Let’s check how it performs with a POST request.

require 'typhoeus'
require 'benchmark'

response_time = Benchmark.realtime do
  response = Typhoeus.post('https://httpbin.org/post', body: {foo: 'bar'})
  puts "Response code: #{response.code}"
  puts "Response body: #{response.body}"
end

puts "Response time: #{(response_time * 1000).round(2)} ms"

The POST request took around 0.1153 seconds.

You will find the documentation of this library quite helpful. It explains everything right from installation to advanced methods with an example. You can even set the maximum concurrency of the request with it. By the way, the built-in limit of concurrency is 200.

If you are looking for a high-performance HTTP client then Typoeus could be one of the choices. Overall it’s a great library.

Excon

It is a pure Ruby HTTP client library that is built on top of the Ruby standard library Net::HTTP. It can provide SSL/TLS encryptions, and streaming responses and you can make asynchronous parallel requests. Many famous Ruby frameworks like Fog and Chef also use this library.

Let’s check the performance of this library with a simple GET request.

require 'excon'
require 'benchmark'

url = 'https://httpbin.org/get'

time = Benchmark.realtime do
  Excon.get(url)
end

puts "Time taken: #{time.round(2)} seconds"

So, it took around 0.23 seconds to make the GET request. Let’s perform a test with a POST request.

require 'excon'
require 'benchmark'

url = 'https://httpbin.org/post'
payload = {key1: 'value1', key2: 'value2'}

time = Benchmark.realtime do
  Excon.post(url, body: payload.to_json, headers: {'Content-Type' => 'application/json'})
end

puts "Time taken: #{time.round(2)} seconds"

POST request took around 0.28 seconds.

The documentation is quite detailed which is great news for beginners. Excon is backed by a large community that keeps this library updated. Regular new updates are released to minimize any errors.

On the other hand, Excon does not come with built-in middleware for common tasks such as JSON parsing or logging. While this allows for greater flexibility, it may require more setup time. Excon has some advanced features that make the learning curve a bit steeper.

Results!!

Let’s compare all the stats and see who is the clear winner.

As you can see Faraday is a clear winner in terms of speed. But it is in close competition with HttParty in terms of Stars marked on their GitHub repository. But overall Faraday is the winner due to its speed and great community support.

In terms of speed, HTTParty is very slow in comparison with other libraries. But since it has great community support you can consider this library for smaller projects. You will find great tutorials on this library on the internet.

Here are Some Key Takeaways:

Ruby provides multiple HTTP client libraries such as Net::HTTP, HTTParty, Faraday, and RestClient.

Different clients offer varying levels of simplicity, flexibility, and middleware support.

Choosing the right HTTP client depends on project size, performance needs, and ease of use.

Proper header configuration and timeout handling improve request reliability.

HTTP clients form the foundation for APIs, web scraping, and backend integrations in Ruby applications.

Conclusion

In this article, we examined the top five popular libraries in terms of their speed of execution and community support.

And Faraday came out as the clear winner. However, it does not mean that other libraries are not capable of building apps and scrapers. However, it is advisable to use Faraday while building any web scraper as this library can speed up web scraping. Faraday is 87% faster than HTTParty which is just tremendous. Regular updates are made to this library to make this library even more powerful.

You can of course test them all at your end with the code snippets shared above. Speed will depend on the network but overall faraday will come out as the clear winner.

I hope you like this little tutorial and if you do then please do not forget to share it with your friends and on your social media.

How to Scrape Websites with Rust: A Step by Step Tutorial

Darshan Khandelwal — Thu, 10 Sep 2026 13:37:26 +0000

In this article, we will learn web scraping through Rust. This programming language isn’t really popular and is not in much use.

This tutorial will focus on extracting data using this programming language and then I will talk about the advantages and disadvantages of using Rust.

We will scrape this http://books.toscrape.com/ using two popular libraries of Rust reqwest and scraper. We will talk about these libraries in a bit.

At the end of this tutorial, you will have a basic idea of how Rust works and how it can be used for web scraping.

What is Rust?

Rust is a high-level programming language designed by Mozilla. It is built with a main focus on software building. It works great when it comes to low-level memory manipulation like pointers in C and C++.

Concurrent connections are also quite stable in Rust. Multiple software components can run independently and simultaneously without putting too much stress on the server.

Error handling is also top-notch because concurrency errors are compile-time errors instead of run-time errors. This saves time, and a proper message about the error is shown.

This language is also used in game development and blockchain technology. Many big companies, such as AWS and Microsoft, already use it in their architecture.

Setting up The Prerequisites to Web Scrape with Rust

I am assuming that you have already installed Rust and Cargo (package manager of Rust) on your machine and if not then you can refer to this guide for further instructions on installing Rust. First, we have to create a rust project.

cargo new rust_tutorial

Then we have to install two Rust libraries which will be used in the course of this tutorial.

reqwest: It will be used for making an HTTP connection with the host website.

scraper: It will be used for selecting DOM elements and parsing HTML.

Both of these libraries can be installed by adding them to your cargo.toml file.

[dependencies]

reqwest = "0.10.8"
scraper = "0.12.0"

Both 0.10.8 and 0.12.0 are the latest versions of the libraries. Now finally you can access them in your main project file src/main.rs.

What Are We Scraping Using Rust?

It is always better to decide what you want to scrape. We will scrape titles and the prices of the individual books from this page.

The process will be pretty straightforward. First, we will inspect chrome to identify the exact location of these elements in the DOM, and then we will use scraper library to parse them out.

Scraping Individual Book Data

Let’s scrape book titles and prices in a step-by-step manner. First, you have to identify the DOM element location.

As you can see in the above book title is stored inside the title attribute of a the tag. Now let’s see where is the price stored.

Price is stored under the p tag with class price_color. Now, let’s code it in rust and extract this data.

The first step would be to import all the relevant libraries in the main file src/main.rs.

use reqwest::Client;
use scraper::{Html, Selector};

Using reqwest we are going to make an HTTP connection to the host website and using scraper library we are going to parse the HTML content that we are going to receive by making the GET request through reqwest library.

Now, we have to create a client which can be used for sending connection requests using reqwest.

let client = Client::new();

Then finally we are going to send the GET request to our target URL using the client we just created above.

let mut res = client.get("http://books.toscrape.com/")
    .send()
    .unwrap();

Here we have used mut modifier to bind the value to the variable. This improves code readability and once you change this value in the future you might have to change other parts of the code as well.

So, once the request is sent you will get a response in HTML format. But you have to extract that HTML string from res variable using .text().unwrap(). .unwrap() is like a try-catch thing where it asks the program to deliver the results and if there is any error it asks the program to stop the execution asap.

let body = res.text().unwrap();

Here res.text().unwrap() will return an HTML string and we are storing that string in the body variable.

Now, we have a string through which we can extract all the data we want. Before we use the scraper library we have to convert this string into an scraper::Html object using Html::parse_document.

let document = Html::parse_document(&body);

Now, this object can be used for selecting elements and navigating to the desired element.

First, let’s create a selector for the book title. We are going to use the Selector::parse function to create a scraper::Selector object.

let book_title_selector = Selector::parse("h3 > a").unwrap();

Now this object can be used for selecting elements from the HTML document. We have passed h3 > a as arguments to the parse function. That is a CSS selector for the elements we are interested in. h3 > a means it is going to select all the a tags which are the children of h3 tags.

As you can see in the image the target a tag is the child of h3 tag. Due to this, we have used h3 > a in the above code.

Since there are so many books we are going to iterate over all of them using the for loop.

for book_title in document.select(&book_title_selector) {
    let title = book_title.text().collect::<Vec<_>>();
    println!("Title: {}", title[0]);
}

select method will provide us with a list of elements that matches the selector book_title_selector. Then we are iterating over that list to find the title attribute and finally print it.

Here Vec<_>> represents a dynamically sized array. It is a vector where you can access any element by its position in the vector.

The next and final step is to extract the price.

let book_price_selector = Selector::parse(".price_color").unwrap();

Again we have used Selector::parse function to create the scraper::Selector object. As discussed above price is stored under the price_color class. So, we have passed this as a CSS selector to the parse function.

Then again we are going to use for loop like we did above to iterate over all the price elements.

for book_price in document.select(&book_price_selector) {
    let price = book_price.text().collect::<Vec<_>>();
    println!("Price: {}", price[0]);
}

Once you find the match of the selector it will get the text and print it on the console.

Finally, we have completed the code which can extract the title and the price from the target URL. Now, once you save this and run the code using cargo run you will get output that looks something like this.

Title: A Light in the Attic
Price: £51.77
Title: Tipping the Velvet
Price: £53.74
Title: Soumission
Price: £50.10
Title: Sharp Objects
Price: £47.82
Title: Sapiens: A Brief History of Humankind
Price: £54.23
Title: The Requiem Red
Price: £22.65
Title: The Dirty Little Secrets of Getting Your Dream Job
Price: £33.34
Title: The Coming Woman: A Novel Based on the Life of the Infamous Feminist, Victoria Woodhull
Price: £17.93
Title: The Boys in the Boat: Nine Americans and Their Epic Quest for Gold at the 1936 Berlin Olympics
Price: £22.60
Title: The Black Maria
Price: £52.15

Complete Code

You can make more changes to the code to extract other information like star ratings of books etc. You can use the same technique of first inspecting and finding the location of the element and then extracting them using the Selector function.

But for now, the code will look like this.

use reqwest::Client;
use scraper::{Html, Selector};

// Create a new client
let client = Client::new();

// Send a GET request to the website
let mut res = client.get("http://books.toscrape.com/")
    .send()
    .unwrap();

// Extract the HTML from the response
let body = res.text().unwrap();

// Parse the HTML into a document
let document = Html::parse_document(&body);

// Create a selector for the book titles
let book_title_selector = Selector::parse("h3 > a").unwrap();

// Iterate over the book titles
for book_title in document.select(&book_title_selector) {
    let title = book_title.text().collect::<Vec<_>>();
    println!("Title: {}", title[0]);
}

// Create a selector for the book prices
let book_price_selector = Selector::parse(".price_color").unwrap();

// Iterate over the book prices
for book_price in document.select(&book_price_selector) {
    let price = book_price.text().collect::<Vec<_>>();
    println!("Price: {}", price[0]);
}

Advantages of using Rust

Rust is an efficient programming language like C++. You can build heavy-duty games and Software using it.

It can handle a high volume of concurrent calls, unlike Python.

Rust can even interact with languages like C and Python.

Disadvantages of using Rust

Rust is a new language if we compare it to Nodejs and Python. Due to the small community, it becomes very difficult for a beginner to resolve even a small error.

Rust syntax is not that easy to understand as compared to Python or Nodejs. So, it becomes very difficult to read and understand the code.

Key Takeaways:

The blog provides a beginner-friendly introduction to building web scrapers using Rust.

Rust libraries like reqwest and scraper allow you to fetch HTML content and parse data efficiently.

Rust offers strong performance and memory safety, making it well-suited for high-performance scraping tasks.

Static websites can be scraped directly in Rust, but JavaScript-heavy sites may require additional tools or a scraping API.

The tutorial walks through step-by-step code examples to help you build and run a basic Rust scraper.

Conclusion

We learned how Rust can be used for web scraping purposes. Using Rust you can scrape other dynamic websites as well. Even in the above code, you can make a few more changes to scrape images and ratings. This will surely improve your web scraping skills with Rust.

I hope you like this little tutorial and if you do then please do not forget to share it with your friends and on your social media.

How to Scrape YouTube Transcripts Using Python

Darshan Khandelwal — Wed, 02 Sep 2026 13:08:19 +0000

YouTube is more than just a video platform — it’s one of the largest searchable knowledge bases in the world. From lectures and podcasts to tutorials and interviews, there’s valuable content locked inside videos. The challenge? Extracting it.

In this guide, we’ll show you how to scrape YouTube transcripts programmatically using Python and the Scrapingdog YouTube Transcript API. Instead of battling YouTube’s HTML or unreliable third-party scrapers, you’ll fetch clean, structured JSON in a single API call, ready to analyze, export, or integrate into your own applications.

Why Scrape YouTube transcripts?

Here are a few use cases where this data is useful:

Summarize videos without watching them end to end
Build searchable archives of content
Perform sentiment or keyword analysis
Repurpose quotes for blogs, social posts, or research

Prerequisites

Create a folder by any name you like. I am naming the folder as youtube.

Now, create a Python file inside this folder. I hope you have Python installed on your computer. I am naming the file as trans.py.

Now, install the requests library inside this folder. This will help us make an HTTP connection with the host. You can install it with the command pip install requests.

Scrape YouTube Transcript with Python

Before creating a scraper, it’s recommended to go through the API documentation. This API will help us pull a complete transcript for any given video.

In this tutorial, we’ll pull data from one of Scrapingdog’s YouTube videos. The ID of this video is deXSHXI8HuU.

1import requests2 3api_key = "your-api-key"4url = "https://api.scrapingdog.com/youtube/transcripts/"5 6params = {7    "api_key": api_key,8    "v": "deXSHXI8HuU"9}10 11response = requests.get(url, params=params)12 13if response.status_code == 200:14    data = response.json()15    print(data)16else:17    print(f"Request failed with status code: {response.status_code}")

In this code, we’re passing our personal API key along with the video ID. Let’s run it and check the output. You can execute it by running python trans.py in your terminal (cmd or bash).

I can even change the language of this data by using the language parameter.

We’ve also built a free tool that lets you extract transcripts from any YouTube video, powered by the same API behind this tutorial.

Recently, we built a tutorial in Make.com (a no-code automation tool), using the YouTube Transcript API. We built a workflow that uses the transcript from a video & converts it into a LinkedIn post. 🎥

Key Takeaways:

Shows how to programmatically fetch YouTube video transcripts using Python.

Demonstrates using an API or library to retrieve clean transcript text.

Covers handling cases where transcripts are auto-generated vs manually provided.

Provides example Python code to automate the scraping process.

Useful for content analysis, summarization, and AI training workflows.

Conclusion

YouTube transcripts are a goldmine of information, from making videos searchable to enabling sentiment analysis, content repurposing, or accessibility improvements. Instead of manually copying transcripts, you can automate the entire process using Python and a transcript scraping API.

With just a few lines of code, you can turn raw video pages into structured, usable text data. Whether you’re analyzing competitors, doing market research, or simply archiving your own content, scraping transcripts gives you cleaner insights and saves a ton of time.

How to Feed Data to an LLM Using Python: A Practical Guide

Darshan Khandelwal — Mon, 17 Aug 2026 11:54:35 +0000

LLMs do not have access to real-time information. They do not know what happened in the last hour. Sometimes you want to analyze data patterns in a file, which might take hours of manual job but if you feed that data to an LLM, you can generate a conclusion within minutes.

In this article, we will learn how you can feed live data scraped from a website using Scrapingdog to any LLM.

Why use Scrapingdog for feeding the data to an LLM?

Most LLM APIs can’t browse the web on their own. Even when they can, feeding them raw HTML is far from ideal. A typical webpage may contain 200KB of navigation menus, ads, JavaScript, and other boilerplate, while only 2KB contains the actual content you need. Passing that directly to an LLM wastes valuable context window space and makes it harder for the model to focus on relevant information.

On top of that, most LLMs simply make a standard HTTP request when fetching a webpage. They don’t handle JavaScript rendering, proxy rotation, CAPTCHAs, or other anti-bot protections, so many requests fail or return incomplete content. That’s where a web scraping API like Scrapingdog becomes essential.

Best Scraping API for Training LLMs

Prerequisite

Before we start writing our code, make sure we have these things in place.

You have to create an account on Scrapingdog. On signup, you will get 200 free credits.

Access to the Anthropic API.

I hope you already have Python 3.x on your machine. If not, then you can download it from here.

Install the requests library for making HTTP connections with Scrapingdog and Anthropic APIs.

Scraping the Data with Scrapingdog

For this article, we will feed live Google News data using the Google News API provided by Scrapingdog. You can learn more about the API by reading the documentation. You can even try the API directly from the dashboard.

You will get ready to use JSON data, which prevents extra token consumption. Click the Get Code button to copy the ready-to-use Python code.

1import requests2 3api_key = "your-api-key"4url = "https://api.scrapingdog.com/google_news"5 6params = {7    "api_key": api_key,8    "query": "usa vs iran",9    "country": "us",10    "advance_search": "false",11    "domain": "google.com"12}13 14response = requests.get(url, params=params)15 16if response.status_code == 200:17    data = response.json()18    print(data)19else:20    print(f"Request failed with status code: {response.status_code}")

Now, we can feed this JSON data to any LLM model.

Feeding scraped data to an LLM

Before we feed this JSON to an LLM, we can trim it down even further. A query returns a news_results array where each item looks like this:

1{2  "title": "US and Iran pause strikes for third night to make space for talks",3  "snippet": "President Donald Trump has paused attacks on Iran for the third night in a row...",4  "source": "BBC",5  "lastUpdated": "2 days ago",6  "url": "https://www.bbc.com/news/articles/c5y45kdkynpo",7  "scrapingdog_link": "https://api.scrapingdog.com/scrape?api_key=...&url=...",8  "imgSrc": "http://t0.gstatic.com/images?q=..."9}

Fields like scrapingdog_link and imgSrc are useful for your app's UI but add nothing for the model; they just burn tokens. Strip the response down to what the LLM actually needs (title, snippet, source, lastUpdated, url) and flatten it into plain text:

1def format_for_llm(news_results):2    entries = [3        f"[{i + 1}] {item['title']}\n{item['snippet']}\n"4        f"Source: {item['source']} ({item['lastUpdated']})\nURL: {item['url']}"5        for i, item in enumerate(news_results)6    ]7    return "\n\n".join(entries)

This keeps token usage low and gives the model something readable instead of a nested JSON blob it has to mentally parse. Now, we can finally feed this data.

1def get_latest_news(query):2    response = requests.get(3        "https://api.scrapingdog.com/google_news/",4        params={5            "api_key": os.environ["SCRAPINGDOG_API_KEY"],6            "query": query,7        },8    )9    return response.json()["news_results"]10 11context = format_for_llm(get_latest_news("Iran US ceasefire talks"))12 13response = requests.post(14    "https://api.anthropic.com/v1/messages",15    headers={16        "Content-Type": "application/json",17        "x-api-key": os.environ["ANTHROPIC_API_KEY"],18        "anthropic-version": "2023-06-01",19    },20    json={21        "model": "claude-sonnet-4-6",22        "max_tokens": 1000,23        "messages": [24            {25                "role": "user",26                "content": (27                    "Using the following recent news articles, answer the question. "28                    f"Cite the source for each fact.\n\n{context}\n\n"29                    "Question: What's the current status of the Iran-US talks?"30                ),31            }32        ],33    },34)

Let me explain this code in brief:

Fetches recent news from Scrapingdog’s Google News API based on a search query.

Formats the news articles into a clean text context suitable for an LLM.

Sends the context to Claude Sonnet 4.6 via Anthropic’s Messages API.

Prompts Claude to answer a specific question using only the provided news.

Requests source citations so each factual statement is backed by the original news articles.

Returns an AI-generated, up-to-date answer grounded in the latest news rather than the model’s training data.

This way, you can create a live data feed for any LLM.

You can use any Scrapingdog API as the data source, not just the Google News API. For example, you can use the Google Search API, Amazon Search API, or any of the other APIs too.

Conclusion

Feeding an LLM live data isn’t complicated once you have a clean source to pull from. Scrape it, trim it down to what the model actually needs, and pass it along with a clear question; that’s the whole loop. What used to take hours of manually reading through articles or spreadsheets now takes a few seconds and a well-formed prompt. And this pattern isn’t tied to news; the same three steps work with the Google Search API, Amazon Search API, or any other Scrapingdog endpoint, so you can build the same kind of live-data feed for whatever your LLM needs to know about.

Best Rank Tracker APIs in 2026: Features, Pricing & Performance

Darshan Khandelwal — Wed, 05 Aug 2026 10:16:51 +0000

A rank tracker API is essential for anyone serious about SEO, digital marketing, or competitive intelligence. Whether you’re building an in-house SEO tool or automating rank monitoring at scale, the right rank tracking API can make or break your workflow. Manually checking keyword positions on Google is slow and not scalable; a good API lets you programmatically monitor thousands of keywords across locations and devices.

If you’ve ever tried building an SEO tool or automating rank monitoring at scale, you already know this: not all Rank Tracking APIs are created equal. Some are fast, some are affordable, and some are not developer-friendly. So, instead of blindly trusting marketing claims, I decided to test them all head-to-head.

In this article, I’ll walk you through six of the most popular rank-tracking APIs

What Is a Rank Tracker API?

A rank tracker API is a programmatic interface that lets you query Google (or other search engines) and retrieve keyword ranking data in structured JSON format. Unlike manual rank-checking tools, a rank tracker API can be integrated directly into your own dashboard, spreadsheet, or SEO platform, enabling automated, real-time position monitoring at scale.

Who Needs a Rank Tracker API?

SEO agencies track hundreds of client keywords daily
SaaS founders building SEO tools or dashboards
Developers automating competitor monitoring
Content teams tracking blog post rankings over time
Best Rank Tracking APIs Compared: Speed, Success Rate & Support

Best Rank Tracking APIs Compared: Speed, Success Rate & Support

How We Will Test Each Rank Tracker API

Analysis will be made based on five attributes.

Scalability– Number of keywords that can be tracked at a time.

Pricing– Per request cost.

Developer-friendly– How easy is it for developers to integrate the API?

Speed– How fast an API responds.

Stability– Uptime of APIs.

I will be using this Python code to test the APIs. We have also created a rank tracking tool with Google Sheets and Serp API, so check it out.

1import requests2import time3import random4import urllib.parse5 6# List of search terms7rank_tracking_terms = [8    "best running shoes",9    "digital marketing agency",10    "best cafe in Chicago",11    "seo tools for small business",12    "best rank tracking api"13]14# Replace with your actual API endpoint15# Make sure it includes {query} where the search term should be inserted16base_url = "https://api.example.com/rank-tracker?query={query}"17total_requests = 1018success_count = 019total_time = 020for i in range(total_requests):21    try:22        search_term = random.choice(rank_tracking_terms)23        encoded_query = urllib.parse.quote(search_term)  # URL encode the query24        url = base_url.format(query=encoded_query)25        start_time = time.time()26        response = requests.get(url)27        end_time = time.time()28        request_time = end_time - start_time29        total_time += request_time30        if response.status_code == 200:31            success_count += 132        print(f"Request {i+1}: '{search_term}' took {request_time:.2f}s | Status: {response.status_code}")33    except Exception as e:34        print(f"Request {i+1} with '{search_term}' failed due to: {str(e)}")35# Final Stats36average_time = total_time / total_requests37success_rate = (success_count / total_requests) * 10038print(f"\nTotal Requests: {total_requests}")39print(f"Successful: {success_count}")40print(f"Average Time: {average_time:.2f} seconds")41print(f"Success Rate: {success_rate:.2f}%")

Scrapingdog’s Rank Tracker API

It offers the Google SERP API, which can be used to track keywords.

The API will provide you with a JSON response, which will have a key by the name of position , which can be used to track the ranks of keywords in the Google search results.

Details

Once you sign up, you will receive 1000 free credits. You can use them to test this API.

Per keyword tracking, the cost would start at $0.001 and go below $0.00029 with a higher volume.

Scrapingdog offers clear documentation, video tutorials, and blogs to help developers easily integrate APIs in their working environment. Our YouTube channel has multiple tutorials on Google SERP APIs.

Customer support is available 24*7 to help you resolve any query related to the services offered.

Testing the API

Scrapingdog is blazing fast, with an average speed of 1.31 seconds, and makes keyword tracking effortless.

On the flip side, using a sluggish SERP API can cripple your dashboard — sometimes to the point of freezing it. Just take a look at Neil Patel’s SEO tool; it struggles quite a bit, possibly because a turtle-speed API powers it.

Note: Recently, the num=100 parameter was removed. Now, to scrape 100 results from Google, you need to make 10 API calls (each to retrieve 10 results). Due to this depreciation, the rank tracking tools & their pipeline broke.

At Scrapingdog, we created our own rank tracking system using our Google Search API & n8n. You can check out the blog here.

SerpAPI

Serpapi also offers Google Scraping APIs that can scrape data from Google Search results and can be used as a rank tracker.

Details

First-time users will get 100 credits for free for testing the APIs.

Per keyword tracking, the cost would start at $0.015 and go below $0.0075 with a higher volume.

They have great documentation, and their API can be easily integrated within any working environment.

Support is great and replies within minutes. You can contact them through chat support from the website or through emails.

Testing the API

With an average response time of just 2.49 seconds, SerpAPI proves to be a good solution.

Ideal for building responsive and reliable rank tracking tools with their Google Search API.

This kind of use case is becoming common across many SaaS platforms. You can see more AI SaaS product ideas that startups are building today.

Brightdata

Brightdata is another great choice for scraping search results and tracking keyword ranks.

Details

They do offer a few free credits for first-time users.

The default pricing is $2.55/CPM, and if you have higher requirements, you can contact them.

Documentation is clear and concise. Any developer can integrate their proxies and APIs easily.

They have one of the best support systems in this scraping industry. Sometimes, it feels like they are just waiting for your query to drop. They will answer your query with lightning-fast speed.

Testing the API

With an average speed of around 5.81 seconds, this API may not be ideal for building a responsive rank-tracking tool.

However, its scalability makes it suitable for general SERP scraping tasks. The close is something on the higher end. So before committing to a paid plan its worth checking its alternatives.

SearchAPI

SearchAPI provides a complete solution around Google. Offers a variety of scrapers, including a rank tracking API.

Details

Just like every other product, they also offer free credits for new signups.

Per keyword tracking, the cost would start at $0.004 and go below $0.002 with a higher volume.

The documentation is well-organized, and users can simply copy the code and run it directly in their development environment.

You can contact them through chat or email.

Testing the API

Clocking in at an average of 8.87 seconds per request, this API isn’t winning any speed awards.

Still, if you’re more focused on scale than speed, it can get the job done for basic SERP scraping.

Apyhub

Apyhub is another rank-tracking API that offers great tools around SEO.

Details

On a new signup, you get very few free credits to test the API. I mean, you cannot even test the API completely. Around 5 API calls are free for a day.

Finding the right API on the website is a challenge in itself. Documentation is not great.

Pricing is not clear. So, cannot comment on that.

Support does not work. Nobody responds.

Testing the API

With an average speed of 7.13 seconds and a success rate of 80%, this API might test your patience if you’re building a real-time rank tracker.

That said, if you’re scraping SERP data in bulk, this API is a NO NO!

ScraperAPI

ScraperAPI is an old player in the web scraping industry, and they also offer a SERP API for tracking the ranks of keywords.

Details

Just like its competitors, it also offers free credits for testing the API.

Per keyword tracking, the cost would start at $0.01225 and go below $0.002075 with a higher volume.

Documentation is clear enough and can be easily integrated.

Support is available, but you might have to wait a whole day to get your query resolved. They also do not offer instant chat support.

Testing the API

Testing this API was a rollercoaster — and not the fun kind. At one point, I was just hoping the program would finish. The fastest response I got was 8.32 seconds. One request even took 170 seconds (yes, you read that right).

With a success rate of 80%, I’m honestly not sure whether to laugh, cry, or just close the terminal and walk away.

Which Rank Tracker API Should You Use?

Every API might look the same from the top, but once you test them, you see the real strength of the product, and you can determine which one best fits your rank tracking mechanism.

Scrapingdog and Serpapi are clear winners when it comes to response time.

Even if you look at the success rate offered by these APIs, you will reach the same conclusion. There are only two choices if you ignore the pricing. But if you consider the pricing as well, then Scrapingdog is worth every penny.

You can integrate each of these tools or APIs into your digital marketing strategy to check keyword rankings, and each will work effectively.

But if response time is something that you care about, pick APIs that have faster response times.

Frequently Asked Questions (FAQs)

How does a rank tracking API work?
A rank tracking API sends a query to Google, fetches the search results, and returns ranking data in JSON format. You can then use fields like position to see where a website ranks for a target keyword.
Which rank tracking API was the fastest in this comparison?
In this test, Scrapingdog was the fastest with an average response time of 1.31 seconds. SerpAPI came next at 2.49 seconds, while the other APIs were noticeably slower.
What factors were used to compare these rank tracking APIs?
The APIs were compared on scalability, pricing, developer-friendliness, speed, and stability. These factors help show not just raw performance, but also how practical each API is for real SEO workflows.
Which rank tracking APIs performed best overall?
Based on speed and success rate, Scrapingdog and SerpAPI were the top performers. If pricing is also considered, the blog concludes that Scrapingdog offers the best overall value.

10 Best Serper Alternatives for Fast & Reliable SERP APIs (2026)

Darshan Khandelwal — Thu, 30 Jul 2026 07:46:49 +0000

Back in 2023, when Serper.dev was launched, it soon became a major hotspot in the SERP API market. It is dead cheap, fast and flexible compared to any other options available in the market. However, if we look beyond organic search results, the API takes a complete U-turn and gets back behind its competitors.

It roughly covers 12 Google APIs, and if your project needs richer SERP features like AI Overviews, Immersive Products, Short videos, Ads or anything beyond basic SERP, the API fails silently.

We tested seven of the most popular Serper.dev alternatives in the market across speed, pricing, output richness, and AI readiness. Whether you’re building an AI monitoring tool, a rank tracker, or an SEO tool, this guide will tell you exactly which one deserves your API credits.

Why Developers are Moving Away From Serper.dev

Serper.dev is indeed a great product, and there is a lot of positivity around it on X and LinkedIn by the developer community. However, some concerning points around it are making developers drift away from the product:

Limited Engines and Scrapers Support — As we discussed above, Serper only covers 12 API Endpoints, including Google Search, Shopping, News, Images, Videos, Maps and a few others. If you need Baidu, DuckDuckGo, Bing, Amazon, and other search engines' support, you’ll have to look for another provider, which becomes a tedious task.

Data Richness Gap — There is a huge gap between the data points offered by Serper and other enterprise data collectors. Various fields like AI Overviews, rich Knowledge Graph data, Inline and Short videos, and ads are missing from it.

What to look for in a Serper.dev Alternative

Before discussing the various providers, here are the points based on which we will be evaluating them:

Pricing model: Monthly plans. Rollovers and credit expiry windows.

Response time: Measured at p50 and p95.

Output richness: Does it return PAA boxes, AI Overviews, sitelinks, discussions, shopping carousels, local packs, or just 10 blue links?

AI-readiness: Can the output feed directly into an LLM pipeline? Structured JSON matters here.

Multi-engine support: Google-only vs. Bing, Baidu, YouTube, and others.

Reliability at scale: Uptime, rate limits, and how the API behaves under concurrency.

Support: Response time, documentation quality, and whether there’s a human on the other end.

Note: We will be comparing the products based on Google Advanced SERP UI, not the basic one, which can be accessed via the gbv=1 parameter.

Scrapingdog

Starting price: ~$40/month for 20,000 Google SERP API calls.
Free tier: 200 credits on signup
Best for: Teams that need value at scale + a wide API surface area

Scrapingdog started as a dedicated general web scraping company that later added Google SERP APIs on top of its robust infrastructure.

So, basically Scrapingdog does not just acts as a Google SERP provider but also supports a two-step workflow in which you can seamlessly extract content inside those links appearing in the search results through its Web Scraping API providing it a great leverage to offer all the data under one hood avoiding developers to use multiple providers for search and scraping services which can result in waste of time and resources.

Scrapingdog’s multi-engine capability is not only backed by its large portfolio of Google Search APIs, which counts to twenty-seven (a huge number), but also supports other search engines like Baidu, Bing, Amazon, Walmart, and YouTube. If you’re running a search engine product or an SEO platform, the length and breadth of APIs alone save you from managing multiple accounts.

Pricing Breakdown:

No credit expiry on PAYG top-ups. If your volume fluctuates, you’re not burning credits you didn’t use.

Performance (Tested on 250 concurrency):

Avg response time: 2.75s(mean), 2.71s (p50), ~3.23s (p95)

Success rate: 100%

Code example, migrating from Serper in a minute:

1# Serper.dev (old)2import requests3 4response = requests.post(5    "https://google.serper.dev/search",6    headers={"X-API-KEY": "YOUR_SERPER_KEY"},7    json={"q": "best web scraping API", "num": 10}8)9results = response.json()10 11# Scrapingdog (drop-in replacement)12response = requests.get(13    "https://api.scrapingdog.com/google",14    params={15        "api_key": "YOUR_SCRAPINGDOG_KEY",16        "query": "best web scraping API",17        "results": 10,18        "country": "us"19    }20)21results = response.json()

Dead Simple! Isn’t it? Plus, the response schemas are the same, so no need to check for different property names in the JSON.

Why choose Scrapingdog over Serper.dev?

As Serper.dev only supports the basic Google SERP, we will be comparing both the LITE and Advanced Google SERP API pricing of Scrapingdog with it.

Serper vs Scrapingdog Advance Search Pricing Comparison

Scrapingdog not only becomes a super economical alternative to Serper at scale in LITE Search, but also in advanced search, which is expensive to scrape.

17+ Google endpoints plus multi search engine support vs. Serper’s ~12

No credit expiry on pay-as-you-go credits

Scrapingdog supports multiple featured snippets, including AI Overviews, Ads, and Knowledge Graph.

Built on the same infrastructure that handles tens of millions of web scraping requests, so reliability at scale is proven.

Human support response time under 5 minutes, clear and vast documentation make it a better choice.

Note: You should also use the Google News Scraping API to monitor your brand's mentions on the Internet.

SerpAPI

Starting price: $75/month for 5,000 searches ($15/1K)
Free tier: 100 searches/month
Best for: Enterprise teams that need legal coverage, multi-engine support, and maximum reliability

SerpAPI is the most established player in this list. The product is consistently maintained by its dedicated engineers, documentation is clear and concise, and APIs deliver the most complete JSON output in the market.

What makes them unique:

Legal US Shield: For plans above the “Developer” tier, SerpAPI provides a legal cover for scraping search results as long as you use it for ethical purposes.

Huge Portfolio: SerpAPI definitely holds the title for having the most search engine APIs in the industry. From Google to Yandex, from Amazon to Walmart, it literally covers every major search engine on the planet.

Open Status Pages: SerpAPI has status pages that consist of average speed and success rate for every API it supports on its platform.

Pricing Breakdown:

Pricing can be brutal not only for developers but also for medium-sized enterprises. However, if you have an enterprise-level volume, then SerpAPI can be the best option for you.

Performance:

Avg response time: 4.83s(mean)

Success rate: 100%

When to choose SerpAPI over Serper.dev:

You need advanced featured snippets from search engines.

You need all of the major search engines under one roof.

You’re building an enterprise SaaS where support SLA matters.

You need clear and concise documentation with dedicated engineers' support that can reply within 5 minutes.

SearchAPI

Starting price: ~$40/month for 10,000 searches
Free tier: 100 searches

A relative newcomer, but it has a strong reputation backed by its great performance and consistent maintenance of its data pipeline. It is positioned somewhere between Scrapingdog and SerpAPI in the enterprise pricing level set.

What stands out:

Covers structured JSON APIs across all Google endpoints.

Solid documentation with code examples in Python, Node.js, Ruby, PHP, and Go.

Cleaner logs consisting of all the needed information about the API call.

Pricing Breakdown:

Performance:

Avg response time: 2.9s(mean)

Success rate: 100%

When to choose SearchAPI over Serper.dev:

You need dedicated APIs related to Google AI Pages.

You want to track ads, citations, sponsored products and various other snippets, which Serper doesn’t yet support.

You need fast support as you can’t tolerate inconsistency with real users in production.

DataForSEO

Starting price: $600 per 1M searches (async queue)
Best for: SEO platforms, rank trackers, agencies doing bulk keyword research

DataForSEO is infrastructurally different from the others on this list. It not only offers a simple synchronous API that returns results in 2–5 seconds, but it also operates a task-based async model in which you just have to submit a batch of queries, and results will be ready in a queue, sometimes in minutes, sometimes longer, depending on load.

That may sound like a downside if you are running a live application, but for SEO teams running bulk rank tracking jobs or keyword research, it’s actually a feature. You don’t need real-time results for a 50,000 keyword crawl. You need low cost and high throughput.

Key differentiators:

Cheapest cost per query at scale: ~$0.60/1K on Standard Queue, as low as $0.006/1K on Sandbox. For bulk workflows, nothing beats this.

SERP + keyword + backlink data: DataForSEO isn’t just a SERP API; it has endpoints for keyword difficulty, search volume, backlink indexes, and more. If you’re building an SEO platform, this is a one-stop shop.

White-label friendly: Clean data model that’s easy to pipe into your own dashboard or product.

Pricing Breakdown:

The pricing changes on the basis of the API Mode:

For the Standard queue, it costs $0.0006 per SERP, and the cost per 1M is $600.

For the Priority queue, it costs $0.0012 per SERP, and the cost per 1M is 1200$.

For LIVE mode, it costs $0.002 per SERP, and the cost per 1M is 2000$.

Performance:

We couldn’t test the API due to too much complex documentation.

When to choose DataForSEO over Serper.dev:
You run bulk/batch SEO workflows, not real-time queries.

You’re building an SEO product and need keyword and backlink data alongside SERP.

Volume is high (1M+ searches/month), and cost is the primary constraint.

Real-time response time is not critical.

Exa & Tavily — Best for AI-Native Workflows

Exa starting price: ~$5/1K queries (content search)
Tavily starting price: ~$3/1K queries
Best for: LLM agents, RAG pipelines, AI research tools

These two products deserve a different section altogether because they’re solving a different problem than the rest of this list fundamentally. They’re not really SERP APIs. They’re AI-native search APIs built specifically for LLM workflows.

Exa uses neural search trained on link prediction to find content by meaning, not just keywords. When you query Exa, you get not only 10–100 links just like other search engines, but also full page content of these links extracted in clean markdown, ready to drop into an LLM prompt without any additional scraping step.

Tavily is similar but focused on citation-ready results. It’s deeply integrated with LangChain and LlamaIndex, which is why it’s become the default search tool for many AI agent frameworks.

The critical difference from SERP APIs:

A traditional SERP API (Serper, Scrapingdog, SerpAPI) returns metadata about search results, titles, URLs, snippets, and knowledge graph data. To get the actual page content, you need a separate scraping step.

Exa and Tavily return the content itself, full text, structured for LLM consumption. That’s why they’re in a different category.

When to choose Exa/Tavily over Serper.dev:

You’re building LLM agents or RAG pipelines and need page content, not just metadata

You’re using LangChain, LlamaIndex, or similar AI frameworks

Search freshness is secondary to content quality for your use case

Head-to-Head: Scrapingdog vs. Serper.dev

The pricing gap widens significantly at scale. At 1M searches/month, Serper charges ~$700 per month. Scrapingdog’s equivalent cost for the same job is ~330$.

Response schema comparison:

Both APIs return JSON. Here’s a simplified side-by-side of a Google SERP response:

1// Serper.dev response (simplified)2{3  "organic": [4    { "title": "...", "link": "...", "snippet": "...", "position": 1 }5  ],6  "peopleAlsoAsk": [...],7  "relatedSearches": [...]8}9 10// Scrapingdog response (simplified)11{12  "organic_data": [13    { "title": "...", "link": "...", "snippet": "...", "displayed_link": "...", "position": 1 }14  ],15  "peopleAlsoAsk": [...],16  "related_searches": [...],17  "ai_overview": { "text_blocks": "...", "sources": [...] },18  "knowledge_graph": {...}19}

The structure is similar enough that migration is minimal. Scrapingdog’s response includes additional fields like displayed_link, ai_overview, and knowledge_graph that gives your pipeline more data to work with.

How to Migrate from Serper.dev to Scrapingdog (Step by Step)

If you’re ready to switch, here’s the complete migration path:

Step 1: Get your Scrapingdog API key

Step 2: Update your endpoint and parameters

1# Python example2 3import requests4 5# Old Serper.dev code6def search_serper(query):7    response = requests.post(8        "https://google.serper.dev/search",9        headers={"X-API-KEY": "SERPER_KEY"},10        json={"q": query, "num": 10, "gl": "us", "hl": "en"}11    )12    return response.json()13 14# New Scrapingdog code15def search_scrapingdog(query):16    response = requests.get(17        "https://api.scrapingdog.com/google",18        params={19            "api_key": "YOUR_API_KEY",20            "query": query,21            "results": 10,22            "country": "us",23            "language": "en"24        }25    )26    return response.json()

1// Node.js example2 3// Old: Serper.dev4const searchSerper = async (query) => {5  const res = await fetch("https://google.serper.dev/search", {6    method: "POST",7    headers: {8      "X-API-KEY": "SERPER_KEY",9      "Content-Type": "application/json"10    },11    body: JSON.stringify({ q: query })12  });13  return res.json();14};15 16// New: Scrapingdog17const searchScrapingdog = async (query) => {18  const params = new URLSearchParams({19    api_key: "YOUR_API_KEY",20    query: query,21    results: 10,22    country: "us"23  });24  const res = await fetch(`https://api.scrapingdog.com/google?${params}`);25  return res.json();26};

Step 3: Update your response parser

Map the field names. Most are identical or near-identical:

Step 4: Run parallel for one week

Keep both integrations live. Run the same queries through both and compare outputs. Once you’re satisfied, cut over completely.

Frequently Asked Question

Is there a free alternative to Serper.dev?
Yes. Scrapingdog offers 200 free credits on signup. SerpAPI gives 100 searches/month. Serper itself offers 2,500 free searches/month, which is the most generous free tier in the list. For completely free options, you’re looking at open-source DIY approaches (Scrapy + proxies), which work but require significant maintenance.

What is the cheapest alternative to Serper.dev at scale?
At 100K+ searches/month, Scrapingdog and DataForSEO are the cheapest options. Scrapingdog costs around $0.25–0.30 per 1K searches at high volume. DataForSEO’s async Standard Queue can go even lower for bulk non-realtime workloads.

Does Scrapingdog support the same endpoints as Serper.dev?
Yes, and it also supports other endpoints as well. Everything Serper covers (web, news, images, shopping, maps, scholar) is available in Scrapingdog, plus Bing, Baidu, Google AI Overview, Google AI Mode, Google Trends, Google Finance, Google Patents, and more.

Is Serper.dev good for AI agents?
It works, but it’s not built for it. Serper returns search metadata titles, snippets, URLs. For AI agents that need actual page content in LLM-ready format, you still need a separate scraping step. If your use case is LLM-native, look at Exa or Tavily. If you want SERP data + the option to also scrape full pages under one API, Scrapingdog’s Data Extraction API handles both.

Does Serper.dev have an uptime SLA?
Serper doesn’t publish a formal uptime SLA. SerpAPI and SearchAPI both publish uptime stats. Scrapingdog maintains a status page at status.scrapingdog.com.

Conclusion

Serper.dev is a solid tool; we’re not here to trash it. But the limited endpoint coverage and no multi-engine support make it a poor fit for teams that are scaling beyond a prototype.

If you’re evaluating alternatives in 2026, the decision mostly comes down to three questions:

How much volume do you need? Low volume → Serper. High volume → Scrapingdog, DataForSEO, SearchAPI and SerpAPI.

Do you need fast results? Yes → Scrapingdog, Serper.

Are you building for LLMs specifically? SERP metadata → Scrapingdog. Full content extraction → Exa or Tavily.

For the majority of developers who are building LLMs, rank trackers and SEO tools, Scrapingdog delivers the best combination of price, endpoint breadth, and reliability at any volume tier.

You can test it for free with 200 credits at scrapingdog.com — no credit card required.

5 Best LinkedIn Scraping Tools (Ranked) by Scalability, Data Quality, and Pricing

Darshan Khandelwal — Wed, 29 Jul 2026 13:33:01 +0000

LinkedIn Scrapers are quite in demand in 2026. LinkedIn has the largest pool of corporates available today.

So obviously, you can find many people on this platform who could have the same interests as yours. But websites like LinkedIn can serve you with data that can be used for selling goods and products.

LinkedIn has more than 800 million active users where people share their work experience, skills, and achievements daily. If you scrape and use this data wisely then you can generate a lot of leads for your business.

Of course, this data can also be used for other purposes like finding the right candidate for the job or maybe enriching your own CRM.

In this article, we will talk about the best LinkedIn scrapers in the market. Using these scrapers you can scrape Linkedin at scale.

Quick Comparison of Best LinkedIn Scraping Tools

Advantages of using LinkedIn Scrapers Instead of Collecting Data Manually

You will always stay anonymous. On each request, a new IP will be used to scrape a page. Your IP will always be hidden.

The pricing will be less as compared to the official API.

You can get parsed JSON as the output.

3rd Party APIs can be customized according to demands.

24*7 support is available with many of them.

Some Challenges of Scraping LinkedIn at Scale

If you have researched enough, you must by now know that there are tools with which you can scrape a few hundred profiles/jobs without any problem.

These tools either use your LinkedIn Profile/LinkedIn Sales Navigator which makes them limited in terms of their abilities to extract data.

The main problem arises when you scrape LinkedIn at scale. Some of them are ⬇️

Legal and Ethical Issues:

User Agreement Violation: LinkedIn’s terms of service prohibit the scraping of their website. Engaging in this activity can result in legal action against the perpetrators.

Privacy Concerns: Extracting user data without consent violates privacy norms and can lead to severe repercussions. I have discussed in detail about the legality of scraping LinkedIn in this blog.

Technical Challenges:

Rate Limits: LinkedIn monitors and restricts frequent and massive data requests. An IP address can be temporarily banned if it makes too many requests in a short period.

Dynamic Content Loading: LinkedIn uses AJAX and infinite scrolling to load content dynamically. Traditional scraping methods often fail to capture this kind of content.

Complex Website Structure: LinkedIn’s DOM structure is intricate, and elements might not have consistent class or ID names. This can make the scraping process unstable.

Captchas: LinkedIn employs captchas to deter automated bots, making scraping even more challenging.

Cookies and Sessions: Managing sessions and cookies is necessary to mimic a real user browsing pattern and avoid detection.

Maintenance Issues:

Frequent Changes: LinkedIn, like other modern web platforms, frequently changes its user interface and underlying code. This means scrapers need constant updating to remain functional.

Data Quality: Ensuring the scraped data’s accuracy, relevancy, and completeness can be challenging, especially at scale.

Infrastructure and Costs:

Large-scale Scraping: Scraping at scale requires a distributed system, proxy networks, and cloud infrastructure, increasing the complexity and costs.

Data Storage: Storing vast amounts of scraped data efficiently and securely is another challenge.

Anti-Scraping Mechanisms:

Sophisticated Detection: LinkedIn employs sophisticated bot detection mechanisms. Mimicking human-like behavior becomes essential to avoid detection.

Continuous Monitoring: LinkedIn monitors for suspicious activities and can block accounts or IP addresses even if you successfully scrape data.

Advantages of using LinkedIn Scrapers Instead of Collecting Data Manually

- You will always stay anonymous. A new IP will be used on each request to scrape a page. Your IP will always be hidden.
- The pricing will be lower than the official API.
- You can get parsed JSON as the output.
- 3rd Party APIs can be customized according to demands.
- 24*7 support is available.

Best LinkedIn Scrapers [In 2026]

We will be judging these LinkedIn lead Scraper APIs based on 5 attributes.

Scalability means how many pages you can scrape in a day.
Pricing of the API. What is the cost of one API call?
Developer-friendly refers to the ease with which a software engineer can use the service.
Stability refers to how much load a service can handle or for how long the service is in the market.
Data Quality refers to how old the data is.

EnrichmentAPI

EnrichmentAPI offers a scalable solution for collecting public profile data. They also offer a reverse contact API, a company enrichment API, an email finder API, an email verification API, etc.

Scalability

You can pull data of millions of profiles in a day with this API. We have tested this, and we pulled around 1.8M profiles in a day with all the details like experience, skills, education, etc.

Pricing

They offer a free plan to test the API. On signup, you get 20 credits. Other than that, their smallest plan starts from $49/month, in which you can easily enrich around 1500 profiles.

Developer Friendly
Documentation is very clear, and any developer can integrate the APIs easily. You get instant chat support too through a chat widget or email.

Stability
EnrichmentAPI has been in the market for like 3 years. They have been providing the enrichment data consistently helping many CRM and AI companies to collect people data at scale.

Data Quality
You will get fresh data on every API call.

Scrapingdog’s LinkedIn Scraper API

Scrapingdog offers a simple and easy-to-use LinkedIn Scraper API.

This API can be used to scrape either a person’s profile or a company profile. Other than this Scrapingdog offers LinkedIn Jobs scraper to scrape job data from this platform.

Scalability

Well, you can scrape around 1 million profiles from Scrapingdog’s LinkedIn Scraping API be it job data or profile data.

Pricing

The enterprise pack will cost $1k per month and you can scrape 110k profiles. Each profile will cost $0.009.

Developer Friendly

The documentation is self-explanatory and the user can test the API directly from the dashboard without setting up any coding environment.

Stability

Scrapingdog has been in the market for 5 years now and has more than 200 users which proves its stability in scraping LinkedIn at scale.

Data Quality

We always scrape fresh data.

Brightdata LinkedIn Scraper

Brightdata, along with its large proxy network also provides LinkedIn Scrapers.

Scalability

Their scalability is great but the error rate might go up due to disturbances in their proxy pool. Bright data goes down frequently.

Pricing

Their solution could be a little expensive. The per-profile cost is around $0.05.

Developer Friendly

The whole documentation is quite easy to read and there are request builders which can help you get started quickly.

Stability

No doubt Brightdata has the biggest proxy pool in the market but they frequently go down sometimes. But overall their infrastructure is quite solid.

Data Quality

They do scrape fresh data like Scrapingdog.

Read More: 5 Best Bright Data Alternatives for Web Scraping

People Data Labs

PDL provides data enrichment technology. You can use their APIs to find valuable insights for any prospect.

Scalability

PDL is designed for small projects. You cannot scrape thousands of profiles with it in a short time frame. So, the scalability is not great.

Pricing

It will cost you $0.28 per profile. This makes them very very costly.

Developer Friendly

The documentation is nice which makes them developer-friendly.

Stability
If you are ok with old data then it is stable.

Data Quality

The data you get will be from an old database because they don’t scrape it fresh. They have data sources that they renew at regular intervals.

Apify

Apify is another great LinkedIn scraper. You can scrape person’s profile as well as the company profile.

Scalability

Their API only works if you pass cookies from your logged-in LinkedIn account. So, if you pass cookies from a normal account, you will be able to scrape a maximum of 50 profiles, and if you are passing cookies from a premium account, you will scrape a maximum of 500 profiles. Once you reach these limits your account will be banned by Linkedin and logged out of your account.

So, Apify is for those who just want to scrape a very small number of profiles.

Pricing

They provide a 3-day free trial after that they charge $25 per month.

Developer Friendly

Documentation is very clear and whether you are a developer or a non-developer you will be able to scrape profiles using Apify. It provides a hook through which you can scrape profiles using make.com.

Stability

Service is completely stable and you can scrape a small number of profiles very easily without getting blocked.

Data Quality

You will always get fresh data from their API. Since you are passing the cookies then you will get data like skills, history, etc which are absent in almost all the other scrapers.

Skrapp.io’s LinkedIn Extension

Wanted to mention this particular product as a special mention in this article. They aren't a dedicated LinkedIn Scraper, but a service that can be used on top of it.

Skrapp.io focuses on email finding and data enrichment from LinkedIn profiles. They specialize in extracting verified business emails rather than comprehensive profile scraping.

Scalability

Skrapp.io is designed for moderate-scale email extraction. With their Professional plan, you get 10,000 email credits per month, and their Enterprise plan offers 25,000 credits monthly. Each credit allows you to find one verified email address, making it suitable for targeted email campaigns rather than mass profile scraping.

Pricing

Professional plan costs $37/month (billed annually) for 10,000 email credits, making each verified email cost approximately $0.0037. Enterprise plan costs $75/month (billed annually) for 25,000 credits, bringing the cost down to $0.003 per email. They also offer a free plan with limited credits.

Developer Friendly

Skrapp.io offers browser extensions for LinkedIn, Sales Navigator, and LinkedIn Recruiter, making it accessible for non-developers. They also provide API integrations for Enterprise users. The platform is user-friendly with clear documentation and multiple export options.

Stability

Skrapp.io has been serving over 2 million professionals and has established partnerships with major companies like Adobe, IBM, and Microsoft. Their service focuses on compliance, and they only charge for valid and catch-all emails, ensuring fair usage. The service is stable for email extraction purposes.

Data Quality

They provide real-time email verification with their integrated email verifier. The platform ensures high accuracy by only charging for deliverable emails marked as 'Valid' or 'Catch-all.' However, their focus is primarily on email data rather than comprehensive profile information.

Final Verdict

We have compiled a report considering all the factors above. This report is the result of a comprehensive analysis of each API.

Although they may appear similar at first glance, further testing reveals that only a small number of APIs (one or two) are stable and suitable for production purposes.

Therefore, it is essential to evaluate the options based on your specific requirements and choose the most appropriate one from the given list.

I hope this list and my testing will help you pick one of them. If you liked the article do consider sharing it.

FAQ (Frequently Asked Questions)

(1) What is the best LinkedIn data extractor tool?
The best LinkedIn data extractor tool is one that can reliably collect profile, company, and job data at scale without requiring a logged-in account. API-based LinkedIn scrapers such as Scrapingdog, Bright Data are commonly preferred because they return structured data, handle dynamic LinkedIn pages, and help reduce blocking risks compared to browser extensions or manual scraping scripts.

(2) What is the best LinkedIn scraper for lead generation?
For lead generation, the best LinkedIn scraper is one that can extract key contact and professional details such as name, job title, company, industry, and profile URL. Tools that support bulk scraping, filters, and clean data output are ideal for sales teams and outreach campaigns.

(3) Is it safe to scrape data from LinkedIn?
Scraping LinkedIn is safer when only publicly available data is collected and scraping activity is kept within reasonable limits. Using tools with proxy rotation, rate limiting, and proper request handling helps reduce the risk of IP blocks while supporting more stable data extraction.

3 Google Scholar APIs Worth Checking Out in 2026

Darshan Khandelwal — Wed, 22 Jul 2026 07:18:49 +0000

Best Google Scholar API Comparison (2026)

Below is a quick comparison of the best Google Scholar APIs tested in this article, based on success rate, speed, and reliability.

Google Scholar data can unlock a goldmine of academic insights.

By analyzing this data, you can identify trending research topics, track citation growth over time, evaluate the impact of a paper or author, discover new publications in a specific field, or even map collaboration networks between researchers and institutions. For universities, it helps benchmark academic performance.

For developers, it powers tools like reference managers, citation analyzers, and literature review platforms.

Simply put, Scholar data helps you stay ahead in the research game, whether you’re publishing, studying, or building.

A Google Scholar API allows you to programmatically access academic research data such as article titles, author names, abstracts, citation counts, publication dates, journals, and even related works — all without manually browsing through the website.

It’s a game-changer for researchers, developers, and anyone who needs large volumes of scholarly information, including students managing assignments more efficiently. For those dealing with tight deadlines, some also explore options to do my homework more efficiently. Whether you’re building an academic search tool, analyzing trends in research papers, or simply pulling citation metrics for a report, a Google Scholar API helps you automate that process efficiently and at scale.

In this article, we will test 3 of the best Google Scholar APIs and rank them based on their performances. This study will help you select the best option for your next project.

Criteria To Test These Google Scholar APIs

Analysis will be made based on these five attributes.

Scalability– Number of researches that can be scraped.

Pricing– Per request cost.

Developer-friendly– How easy is it for developers to integrate the API?

Speed– How fast an API responds.

Stability– Uptime of APIs.

I will be using this Python code to test the APIs.

1import requests2import time3import random4 5 6# List of search terms7research_terms = [8    "cancer",9    "biology",10    "NLP Papers 2024",11    "machine learning in healthcare",12    "quantum computing applications"13]14 15# Replace with your actual API endpoint16# Make sure it includes {query} where the search term should be inserted17api_key="your-api-key"18base_url = "https://api.example.com/google_scholar"19#20 21 22 23 24 25total_requests = 1026success_count = 027total_time = 028 29for i in range(total_requests):30    try:31        search_term=random.choice(research_terms)32 33 34        params = {35  "engine": "google_scholar",36  "q": search_term,37  "api_key": api_key38}39 40 41 42        start_time = time.time()43        response = requests.get(base_url, params=params)44        end_time = time.time()45 46        request_time = end_time - start_time47        total_time += request_time48 49        if response.status_code == 200:50            success_count += 151        print(f"Request {i+1}: '{search_term}' took {request_time:.2f}s | Status: {response.status_code}")52 53    except Exception as e:54        print(f"Request {i+1} with '{search_term}' failed due to: {str(e)}")55 56# Final Stats57average_time = total_time / total_requests58success_rate = (success_count / total_requests) * 10059 60print(f"\n🔍 Total Requests: {total_requests}")61print(f"✅ Successful: {success_count}")62print(f"⏱️ Average Time: {average_time:.2f} seconds")63print(f"📊 Success Rate: {success_rate:.2f}%")

Scrapingdog Google Scholar API

Google Scholar API offered by Scrapingdog can be used for scraping Google Scholar data at scale. API will return structured JSON data, which will have all the basic details from the title to the link of the research.

Details

You will get 1000 free credits on sign-up. You can use these credits to test the API completely before upgrading.

The cost per API call starts at $0.001 and decreases to below $0.00029 as usage volume increases.

The documentation is very clear and provides ready-made code snippets. One can easily integrate the API by directly copying and pasting the code from the dashboard. We regularly post new articles and videos on YouTube.

You can contact them through email or on-site chat support. They are available 24*7 and will reply to your query within seconds.

Testing the API

We got amazing results.

With a 100% success rate and an average response time of just 1.26 seconds, Scrapingdog's Google Scholar API proves to be incredibly fast and reliable for scraping Scholar at scale.

SerpAPI

SerpAPI also offers Google Scholar API, and I am sure you already know about them. API will return a beautiful, structured JSON response.

Details

When you sign up, you get 100 credits for free. You can test any API before going for a bigger pack.

Pricing for one API call starts from $0.015(15x of Scrapingdog😲) and drops below $0.0075 with higher volume.

The documentation is clear and concise. Any developer can integrate the API in no time.

You can reach out to them through on-site chat support or through email. They have a great support team and will help you out with any query.

Testing the API

We got a 100% success rate with an average response time of 2.51 seconds.

SearchAPI

SearchAPI is another choice if you are looking to scrape Google Scholar.

Details

They provide 100 free credits for testing the API.

Per API, call cost will start from $0.004(4x of Scrapingdog😲) and drop below $0.002 with a higher volume.

Any developer can easily integrate their APIs into their working environment.

You can contact them through chat or email.

Testing the API

SearchAPI also provided a 100% success rate with an average speed of 3.59 seconds.

Conclusion

If you compare each API based on speed, the results will look like this.

Here, it looks like Scrapingdog and SerpAPI can both be used for scraping Scholar data at scale. Now, let’s compare them based on pricing.

Here, things look very different. If you consider the price, then Scrapingdog becomes a clear choice.

FAQ (Frequently Asked Questions)

1. What is a Google Scholar API, and what is it used for?

A Google Scholar API allows you to extract research data like article titles, authors, abstracts, and citation counts. This data is commonly used for academic research, citation analysis, literature reviews, and building research tools.

2. Does Google Scholar have an official API?

No, Google Scholar does not provide an official public API. This is why third-party Google Scholar APIs are commonly used to extract research data reliably.

3. What factors should I consider when choosing a Google Scholar API?

Key factors include data accuracy, success rate, ease of integration, pricing, and whether the API handles proxies and blocks automatically.

What’s the best way to scrape Google Scholar search results? The most reliable way is to use a dedicated Google Scholar API that manages proxies, rate limits, and anti-bot systems for you.

Best Oxylabs Alternatives in 2026: Features, Pricing & Comparison

Darshan Khandelwal — Wed, 15 Jul 2026 10:37:41 +0000

TL;DR

The best Oxylabs alternatives in 2026, compared by pricing, ease of use, and scalability, include:
Scrapingdog – Best overall alternative with predictable credit-based pricing, dedicated scraping APIs, no KYC, and plans starting at $40/month.
Bright Data – Enterprise-grade platform with the largest proxy network and the highest success rates, best suited for large-scale scraping.
Decodo – Affordable residential proxy provider with no KYC, ideal for developers who want Oxylabs-quality proxies at a lower cost.
ScraperAPI – Easy-to-use scraping API with transparent credit-based pricing, built-in proxy rotation, and JavaScript rendering.
Zyte – Strong choice for Scrapy users with flexible pay-as-you-go pricing and managed anti-bot infrastructure.

Looking for an alternative to Oxylabs for web scraping? In this article, I’ll show you 5 more affordable options, just as effective, and backed by support teams that actually reply.

Before we compare the alternatives in detail, here's a quick overview of their pricing, billing models, KYC requirements, and ideal use cases.

Overall Comparison

How We Evaluated These Oxylabs Alternatives

To keep this comparison fair, we evaluated every provider using the same criteria: pricing, billing model, onboarding process, KYC requirements, dedicated scraping APIs, documentation quality, customer support, and overall value for developers and businesses. Pricing information was collected from each provider's publicly available plans as of mid-2026.

Why Look for an Oxylabs Alternative

A few recurring complaints show up across independent reviews and buyer forums:

KYC friction — Oxylabs requires identity verification for trials and several products, which can delay onboarding for individual developers.

High minimums — Residential proxy traffic starts at $8/GB. That sounds fine until you do the math. A single Google page can run around 1MB, so 1GB burns through in just a few hundred calls.

Enterprise-first support model — Great if you’re a mid-market or enterprise buyer with an account manager, less great if you just want a quick answer in a chat window. The support team is extremely unhelpful and takes several hours to respond. Many users have complained about the same issue in the reviews as well.

Nine separate product lines — residential, datacenter, ISP, mobile, Web Scraper API, SERP API, Web Unblocker, and more, each with its own pricing model, which makes it genuinely hard to estimate a monthly bill upfront.

What Makes a Good Oxylabs Alternative?

Choosing a web scraping platform isn't just about finding the lowest price. A good Oxylabs alternative should offer reliable proxy infrastructure, transparent pricing, automatic CAPTCHA handling, JavaScript rendering, responsive customer support, and enough flexibility to scale from small projects to enterprise workloads. Depending on your use case, features like dedicated Google Search APIs, pay-as-you-go pricing, or residential proxies may matter more than having the largest proxy pool.

Scrapingdog

Scrapingdog is a web scraping API that handles proxy rotation, headless browser rendering, and CAPTCHA solving behind a single endpoint, plus dedicated APIs for Google Search, Google Maps, Amazon, Walmart, LinkedIn, and more that return clean, parsed JSON instead of raw HTML.

Pros

No KYC and no credit card required to try it. The free plan ships with 200 free credits.

Paid plans start at $40/month for 200,000 credits, with a pay-as-you-go option (25,000 credits for $10, credits never expire) for irregular workloads.

Every plan unlocks every API, there’s no feature-gating between tiers, only credits and concurrency.

Failed or blocked requests are never charged.

With 70+ dedicated APIs — Google SERP, Google Shopping, Amazon Product Scraper, Walmart, and more. You get purpose-built endpoints for nearly every major target, starting around $0.001 per request and dropping even lower at scale.

Documentation includes code samples across most major languages, and support is available via chat without needing an account manager.

Cons

Some heavily protected or less common websites might not get scraped.

No option to pick your own proxy location down to the city level.

Best for: developers, small-to-mid-size and enterprise teams who want predictable credit-based pricing, dedicated scraper endpoints for popular targets, and zero onboarding friction.

Brightdata

Bright Data is the largest commercial proxy network on the market, with a residential pool reported at around 400 million IPs across 195 countries. Alongside that, it offers a general Web Scraper API, SERP API, Web Unlocker, and ready-made datasets.

Pros

Usage-based pricing across the board — no single flat subscription. Residential proxies run roughly $8/GB pay-as-you-go, dropping toward $2.50–$3/GB on committed volume.

Web Unlocker and SERP API are typically priced per 1,000 successful requests, starting around $0.75–$1.50/1K depending on the tier.

Independent benchmarks consistently place Bright Data at or near the top on success rate against heavily protected targets like Google, Indeed, and Zillow.

No meaningful free tier, but trial credits are available, and the platform becomes genuinely cost-competitive once monthly spend passes roughly $200.

Cons

It’s genuinely expensive at low volume, with no real free plan, small projects end up paying enterprise-level rates for a fraction of the usage.

Pricing is spread across nine-plus separate products (residential, datacenter, SERP API, Web Unlocker, etc.), so figuring out what your actual monthly bill will look like takes real effort.

Best for: enterprise teams that need the largest possible IP pool and the highest success rate on the hardest anti-bot targets, and have the budget to match.

Decodo

Decodo rebranded from Smartproxy in 2025 but kept the same infrastructure: a 115M+ IP residential network, a Site Unblocker product, and an all-in-one Scraping API covering search engines and e-commerce sites.

Pros

Fully self-serve. You do not need to pass any KYC resistance.

Residential proxies start around $3.75/GB on the entry tier and scale down toward $2/GB at the 1TB tier; pay-as-you-go sits around $4/GB with zero commitment.

Site Unblocker is billed per 1,000 successful requests, at roughly $1.20/1K, and covers CAPTCHA solving and JavaScript rendering automatically.

ISO/IEC 27001 certified, which matters if your team has a compliance checklist.

Cons

The free trial is tiny with just 3 days and 100MB, which is barely enough to test one scraper run, let alone a real workload.

On sites with tougher anti-bot protection like Cloudflare or Akamai, success rates drop off unless you’re on the pricier Site Unblocker tier.

Best for: teams that want Oxylabs-style residential proxy quality at roughly half the entry price, without going through KYC.

ScraperAPI

ScraperAPI wraps proxy rotation, JavaScript rendering, and CAPTCHA solving into a single flat-rate credit system, with dedicated structured-data endpoints for Amazon and Google.

Pros
Plans start around $49/month, with a free-credit trial and no credit card required to sign up.

Pricing is credit-based but transparent: a fixed number of credits per request regardless of how “hard” the target site turns out to be, which makes budgeting easier than Oxylabs’ per-product pricing.

Includes an async endpoint, structured data endpoints, and a no-code scheduling tool (DataPipeline) on all plans.

Cons
Google APIs are extremely slow and come with super low success rates.

Like most credit-based tools, the advertised price looks cheap upfront, but JavaScript rendering and premium proxies can multiply your effective cost well beyond the headline rate.

Best for: developers who already have their own parsing logic and just want a reliable, predictably-priced proxy and rendering layer.

Zyte

Zyte (formerly Scrapinghub) is the company behind the open-source Scrapy framework, and its Zyte API automatically tiers pricing by how difficult a target site is to scrape.

Pros

Pay-as-you-go HTTP requests run roughly $0.13–$1.27 per 1,000, and browser-rendered requests run $1.01–$16.08 per 1,000, depending on site difficulty.

Monthly commitments ($100 / $200 / $500) unlock meaningfully lower per-request rates.

Scrapy Cloud is sold separately for teams hosting their own Python/Scrapy spiders, starting around $9/month per unit.

$5 free trial credit is available to test before committing.

Cons

Because pricing is tier-based and automatic, it can be hard to predict your exact monthly bill in advance.

Response times and support have both drawn complaints. Many users are reporting that they got a reply after 10+ days.

Best for: teams already built on Scrapy who want managed anti-bot handling without leaving the Python ecosystem.

> Pricing is approximate and pulled from publicly listed rates as of mid-2026 — always confirm current numbers on each provider’s pricing page before budgeting, since these change often.

Which Oxylabs Alternative Should You Actually Pick?

Tight budget, need dedicated APIs (Google, Amazon, Walmart): Scrapingdog.

Need the biggest IP pool and highest success rate on the hardest targets, budget isn’t the constraint: Bright Data.

Want cheap self-serve residential bandwidth without KYC: Decodo.

Building on Scrapy or need Python-native tooling: Zyte.

Oxylabs is a genuinely strong platform, and it’s still the right call if you’re an enterprise buyer who needs a dedicated account manager and can absorb its pricing. But for most developers and small teams, one of the five alternatives above will get you the same data with less friction and a smaller bill.

How to Find All URLs on a Domain's Website: 4 Methods Explained

Darshan Khandelwal — Wed, 24 Jun 2026 13:00:14 +0000

Finding all the URLs on a domain sounds simple until you actually try it. Google’s site: search misses pages. Sitemaps skip no-indexed content. Manual crawling breaks on large sites. And if the website renders pages via JavaScript, most tools won’t even see them.

Whether you’re auditing your own site before a migration, mapping a competitor’s content structure, hunting for broken links, or building a scraper that needs every page URL, this is one of the most common (and most misunderstood) tasks in web scraping and SEO.

In this guide, we’ll walk through multiple methods to find all URLs on a domain, from zero-code approaches you can run in 30 seconds to Python scripts that scale to thousands of pages. For each method, we’ll cover what it finds, what it misses, and when to use it.

Let’s get into it.

Find All URLs On A Domain’s Website

Before jumping into the methods, it’s worth knowing the most common reasons people do this, because your goal will determine which method makes the most sense.

SEO auditing — find orphan pages, check for duplicate content, spot pages that shouldn’t be indexed, or verify that all your important URLs are actually crawlable.

Broken link detection — crawl every URL on your domain and check their status codes. A full URL list is the starting point for any broken link audit.

Site migration — before moving to a new domain or CMS, you need a complete inventory of every page so nothing gets left behind or loses its redirect.

Competitive research — map out a competitor’s content structure to find topics they’re covering that you aren’t.

Web scraping — if you want to scrape data from every page on a site, you first need a list of every page to scrape.

Content audits — identify thin pages, outdated posts, or content gaps across your entire site.

Now, let’s discuss all the techniques in brief.

Google search technique

Google Search is the fastest zero-setup method; just enter a search query to find indexed pages on any domain. The catch: Google excludes pages for reasons like duplicate content or noindex tags, so results are never a complete picture.

Sitemaps and robots.txt

Sitemaps and robots.txt give you a more structured view of a site’s pages than Google search, but they come with their own blind spots. A domain can have multiple sitemaps, some pages may be 302 redirecting at the time you check, and SEO plugins like RankMath or Yoast automatically drop non-canonical pages from the sitemap entirely. So if Page A has a canonical pointing to Page B, Page A simply won’t appear.

SEO crawling tools

SEO crawling tools are the most straightforward option — no code, detailed output, and easy to use. The tradeoff is cost; most tools limit free usage to 500 pages, which works fine for smaller sites but requires a paid plan once you scale.

Scraping

If you’re comfortable with coding, a custom scraper gives you the most control: filter by URL pattern, exclude directories, handle pagination, or integrate directly into your pipeline. It takes more setup, but for large or complex sites, it’s often the only method that gets you everything you need.

Through Google Site Search

One of the quickest ways to find URLs on a website is with Google’s site search feature. Here’s how to use it:

Go to Google.com

In the search bar, type site:example.com (replace example.com with the website you want to search).

Hit enter to see the list of indexed pages for that website.

Let’s find all the pages on scrapingdog.com. Search for site:scrapingdog.com on Google and hit enter.

Google will return a list of indexed pages for a specific domain.

However, as discussed earlier, Google may exclude certain pages for reasons such as duplicate content, low-quality, or inaccessible pages.

So, the “site:” search query is best for getting rough estimates, but it may not be the most accurate measure.

If you want to do it programatically then you can take help of the Google Search API and Python to extract all the links without manually checking each page on google.

Through Sitemaps and robots.txt

Sitemaps and robots.txt are two files that almost every website exposes publicly, and together they can give you a near-complete picture of a site’s URL structure. Unlike Google search, you’re not dependent on what Google has chosen to index; you’re looking directly at what the site owner has declared. Here’s how to use each one.

Using sitemaps

A sitemap is an XML file listing all important website pages for search engine indexing. Webmasters use it to help search engines understand the website’s structure and content for better indexing.

Every decent website has a sitemap as it improves Google rankings and is considered a good SEO practice. To learn how to create and optimize one effectively, read the article for practical tips.

Here’s what a standard sitemap looks like:

The element specifies the page URL, indicates the last modification time, and signifies the relative importance for search engines (higher priority means more frequent crawling).

Now, where to find a sitemap? Check for /sitemap.xml on the website (e.g., https://scrapingdog.com/sitemap.xml)

Websites can have multiple sitemaps in various locations, including: /sitemap.xml.gz, /sitemap_index.xml, /sitemap_index.xml.gz, /sitemap.php, /sitemapindex.xml, /sitemap.gz.

Most websites mention the number of sitemaps they have under the domain in the robots.txt file, which we are going to discuss next.

Using robots.txt

The robots.txt file instructs search engine crawlers on which pages to index and which ones to exclude from indexing. It can also specify the location of the website’s sitemap. The file is often located at the /robots.txt path (e.g., https://scrapingdog.com/robots.txt).

Here’s an example of a robots.txt file. Some routes are disallowed for indexing. The sitemap location is also present.

You need to visit both sitemaps and find all the URLs within the website. Note that, for smaller sitemaps, you can manually copy the URLs from each tag. But for larger sitemaps, consider using an online tool to convert the XML format to a more manageable format, such as CSV. There are free tools available, like the one Seowl sitemap extractor.

Through SEO Crawling Tools

Now let’s see how SEO crawling tools help us find all website pages. There are various SEO crawlers in the market, we’ll explore the free tool XML-Sitemaps.com. Enter your URL and click “START” to create a sitemap. This tool is suitable when you need to quickly create a sitemap for a small website (up to 500) pages.

The process will start and you will see the number of pages scanned (167 in this case) and the number of pages indexed (127 in this case). This indicates that only around 127 of the scanned pages are currently indexed in Google Search.

Once the crawling process is complete, the sitemap preview will display all the website’s indexed URLs, including the last modification date and time, as well as the priority of each URL.

You can download the XML sitemap file or receive it via email and put it on your website afterward.

Screamingfrog is another free tool that can help you with exporting URLs from a domain. Keep in mind that the free version allows you to get 500 URLs from a single domain. To use it, you have to download the software on your machine & unlike SaaS, it will crawl the domain from your setup only.

Through Web Scraping

If you’re a developer, you can build your crawler script to find all URLs on a website. Here, you can take advantage of web scraping APIs like Scrapingdog.

This method offers more flexibility and control over crawling compared to previous methods. It allows you to customize behavior, handle dynamic content, and extract URLs based on specific patterns or criteria.

You can use any language for web crawling, such as Python, JavaScript, or Golang. In this example, we will use Python. Also, sign up for the trial Scrapingdog pack.

You can export the list and keep it in CSV inside a folder.

Using Python and Scrapingdog

Before writing the code, make sure you have installed the necessary libraries.

Install the libraries using pip:

1pip install beautifulsoup4 requests

Next, let’s create a Python file (I am naming the file as main.py) and write the code.

If you paste your target website in the Scrapingdog scraper, you will get a ready to use python code.

Just copy this code and paste it into the main.py file. I have used format as links, this ensures you get output as links only and not the whole HTML data.

1import requests2 3url = "https://api.scrapingdog.com/scrape"4params = {5    "api_key": "your-api-key",6    "url": "https://www.scrapingdog.com",7    "dynamic": "false",8    "formats": "links"9}10 11response = requests.get(url, params=params)12print(response.text)

Once you run this code, you will get this JSON response with all the links available on this page.

Now, you can continue making these GET requests to all these URLs using Scrapingdog to extract all the links on that particular domain.

Now, if you want to store all these URLs in a CSV file, you can use Python’s pandas library.

1import requests2import pandas as pd3 4url = "https://api.scrapingdog.com/scrape"5params = {6    "api_key": "your-api-key",7    "url": "https://www.scrapingdog.com",8    "dynamic": "false",9    "formats": "links"10}11 12response = requests.get(url, params=params)13 14if response.ok:15    data = response.json()16    links = data.get("links", [])17    18    df = pd.DataFrame(links, columns=["URL"])19    df.to_csv("scrapingdog_links.csv", index=False)20    print(f"Saved {len(df)} links to scrapingdog_links.csv")21else:22    print(f"Request failed with status code: {response.status_code}")

The crawled URLs will be stored in a CSV file, as shown below:

You can read our guide web scraping with python to get more idea on how a scraper can be built using Python.

Which Method Should You Use?

With four methods on the table, the right choice comes down to your goal, technical comfort, and the size of the site you’re working with. Here’s a quick reference:

Key Takeaways

No single method finds every URL on a domain. For a complete picture, always combine at least two methods.

Google’s site: operator is the fastest starting point but should only be used for rough estimates — it regularly misses pages due to noindex tags, duplicate content filters, and indexing delays.

Screaming Frog is the best free option for a full SEO crawl, but caps at 500 URLs on the free plan. For larger sites, use a Python crawler or Scrapingdog’s API.

JavaScript-heavy sites require a rendering-capable tool. Standard HTTP requests won’t execute JavaScript, so use Scrapingdog’s API to get fully rendered HTML before parsing links.

Conclusion

Finding all the URLs on a domain is one of those tasks that looks straightforward on the surface but has more nuance than most people expect. Different methods find different pages, and no single approach gives you the full picture every time.

The good news is you don’t need to pick just one. Start with a sitemap for the declared URLs, layer in a crawler for anything that was missed, and fall back to Scrapingdog’s API when JavaScript rendering gets in the way.

Whether you’re running an SEO audit, prepping for a site migration, or building a scraper, a clean, complete URL list is always the starting point. Now you have six ways to build one.

If you want to skip the setup entirely and start extracting URLs right away, try Scrapingdog for free with 1000 credits, no credit card required.

Scrape Baidu Search Results using Python: A Step by Step Guide

Darshan Khandelwal — Tue, 23 Jun 2026 13:16:51 +0000

When it comes to search engine data, most tutorials focus on Google, but what if your target audience is in China? That’s where Baidu, the country’s leading search engine, comes in. Whether you’re tracking keyword rankings, monitoring competitors, or gathering localized insights, scraping Baidu search results can give you access to valuable, real-time data that isn’t available elsewhere.

In this guide, we’ll walk you through how to scrape Baidu search results using Python, step by step no prior scraping experience required.

Why scrape Baidu Search Results?

Baidu dominates the Chinese search market, holding over 75% share, making it the go-to source for online information in China. Scraping Baidu search results allows developers, marketers, and researchers to gain deep insights into the Chinese web ecosystem. You can:

🔍 Track keyword rankings for SEO in Chinese markets.

🏢 Monitor competitors and analyze their visibility in Baidu’s SERPs.

📰 Collect news and content trends relevant to specific industries.

📊 Build datasets for training AI and NLP models with localized content.

🌏 Understand market behavior and consumer interests unique to China.

In short, if your business or research involves Chinese audiences, scraping Baidu is one of the most direct and effective ways to access that data.

Why use Scrapingdog to Scrape Baidu search results?

Scraping Baidu directly can be challenging due to its strict anti-bot systems, IP rate limits, and frequent layout changes. That’s where Scrapingdog comes in, and it takes care of all the heavy lifting like proxy rotation, CAPTCHA handling, and response parsing. Instead of worrying about blocked requests or HTML changes, you can simply send a query and receive structured, ready-to-use data.

With Scrapingdog’s Baidu Search API, you get clean JSON output containing titles, links, and snippets, and all accessible in one simple API call. It’s fast, reliable, and built to scale, whether you’re scraping a few keywords for SEO or running large-scale data collection for AI and analytics. In short, it saves developers hours of setup time while maintaining accuracy and consistency across requests.

Prerequisite

I hope you have already installed Python 3.x on your machine; if not, you can download it from here.

Now, create a folder by any name you like. I am naming the folder as webpython. Inside this folder, create a Python file and name it whatever you like. Now, install requests library inside this folder. We will use this to make an HTTP connection with the host website.

The final step is to sign up for Scrapingdog’s trial pack. You’ll receive 1,000 free credits upon signup, which you can use to test any of Scrapingdog’s APIs.

Scrape Baidu Search page with Python

After signup on Scrapingdog you will be redirected to your dashboard. You will find a Baidu Search API over there.

Once you click on it, you will see a scraper where you just need to pass a sample query, and it will pull the data and return it to you in neat JSON format.

For this tutorial, we will pass the query as web scraping. After passing the query you will get a ready Python code; you just have to copy that and paste it in your Python file.

1import requests2 3api_key = "your-api-key"4url = "https://api.scrapingdog.com/baidu/search"5 6params = {7    "api_key": api_key,8    "query": "web scraping"9}10 11response = requests.get(url, params=params)12 13if response.status_code == 200:14    data = response.json()15    print(data)16else:17    print(f"Request failed with status code: {response.status_code}")

Let me explain this code step-by-step.

First we have imported the requests library. This will be used for making the HTTP connection with the host website.

Then we are storing the API key and the Scrapingdog’s Baidu Search API URL.

Then we are making the GET request with the help of requests library.

If the request is successful (status 200), it prints the search results.

If it fails, it prints the error status code.

Once you run the code you will get this parsed JSON data.

In this JSON response you will find the title, link, snippet and rank of the result.

Key Takeaways:

You can programmatically extract Baidu search results using Scrapingdog’s Baidu Search API.

The API returns clean, structured SERP data without manual HTML parsing.

Using an API solves common challenges like proxy rotation and anti-bot measures.

You can customize search parameters like language, pagination, and filters.

This approach scales well for data collection and analytics.

Conclusion

Scraping Baidu data doesn’t have to be complex or time-consuming. With Python and Scrapingdog’s Baidu Search API, you can easily extract clean, structured data from Baidu’s search results without dealing with IP bans, CAPTCHAs, or changing HTML layouts. Whether you’re monitoring keyword trends, gathering localized content, or powering AI applications, this API makes it effortless to access reliable Baidu search data at scale.

In short, instead of spending hours maintaining scrapers, you can focus on what really matters, analyzing data to make smarter decisions in one of the country’s leading markets.

Start your free trial on Scrapingdog today and automate Baidu scraping in just a few lines of Python.

FAQ (Frequently Asked Questions)

1: Is it legal to scrape Baidu search results?

Yes, scraping publicly available Baidu search results is generally allowed. However, you should always follow Baidu’s terms of service and local data regulations when using the data commercially.

2: Can I scrape Baidu without getting blocked?

Direct scraping often leads to IP bans and CAPTCHAs. Scrapingdog handles proxies and CAPTCHAs automatically, so you can scrape Baidu reliably.

3: What data can I extract from Baidu using Scrapingdog?

You can extract the title, URL, snippet, and ranking position in structured JSON format.

4: Can Scrapingdog handle large-scale Baidu scraping?

Yes, Scrapingdog is built for scale and supports both small and high-volume Baidu scraping use cases.

How to Scrape Home Depot Product Data [Step-by-Step Guide]

Darshan Khandelwal — Tue, 09 Jun 2026 13:37:58 +0000

Scraping Home Depot isn’t straightforward. It uses JavaScript rendering, rate limiting, CAPTCHA challenges, and IP bans to block automated requests. You need the right tools and approach to collect data reliably at scale.

In this guide, you’ll learn how to scrape Home Depot using Python, from listing pages and product details to reviews and pagination, and export everything into a clean CSV file. You can also refer our guide on web scraping with python to learn scraping with python.

TL;DR
If you know the complete logic begind the scraping and parsing then you can just copy this code and use it.

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.scrapingdog.com/scrape",
    params={
        "api_key": "your-api-key",
        "url": "https://www.homedepot.com/p/RYOBI-ONE-18V-Lithium-Ion-2-0Ah-4-0Ah-Starter-Kit-with-1-2Ah-HP-Battery-1-4Ah-HP-Battery-and-Charger-PSK1212SB/339206615",
        "dynamic": "false"
    }
)
print(response.status_code)

obj={}
soup = BeautifulSoup(response.text, "html.parser")

# Title
title = soup.find("h1", class_="sui-line-clamp-unset")
obj["Title"] = title.get_text(strip=True)

# Price - dollar, cents in separate spans
price_parts = soup.find_all("span", class_="sui-font-display")[:3]
price = "".join([p.get_text(strip=True) for p in price_parts])
obj["Price"]=price

# Image - actual class in this page
img = soup.find("img", attrs={"data-testid": "small-image"})
obj["image"]=img["src"]

# Features
features_ul = soup.find("ul", class_="sui-text-base")
if features_ul:
    features = [li.get_text(strip=True) for li in features_ul.find_all("li")]
    obj["features"]=features

# Rating
rating_tag = soup.find("a", class_="sui-flex-row")
obj["Rating"]=rating_tag["title"]

print(obj)

Why Scrape Home Depot?

Home Depot has over 1 million products listed online, making it a valuable source for e-commerce data. Here’s what you can do with it:

Price monitoring — Track price changes and adjust your pricing strategy.
Competitor analysis — See how competing brands position and price their products.
Inventory tracking — Monitor stock availability to spot supply gaps or demand surges.
Review analysis — Collect customer feedback at scale to understand sentiment.
Market research — Identify trending products and top-rated brands.

Requirements

To start with, you need Python installed on your machine. If you have not installed it, then you can download it from here. For this tutorial, we will require three libraries:

requests — We will use this to make an HTTP connection with the Host website.
BS4 — This will be used for parsing the data downloaded using requests.
Pandas — This will be used to store data in a CSV file.
The final step would be to signup for a trial account of Scrapingdog. You will get 1000 free credits, which are enough for this tutorial.

Scraping Home Depot Product Page

For this tutorial, we are going to scrape this page from Home Depot. Create a Python file by any name you like. I am naming the file as home.py.

From the product page, we will scrape:

Title
Price
Images
Features
Rating

Scraping Home Depot with Scrapingdog

import requests

url = "https://api.scrapingdog.com/scrape"
params = {
    "api_key": "your-api-key",
    "url": "https://www.homedepot.com/p/RYOBI-ONE-18V-Lithium-Ion-2-0Ah-4-0Ah-Starter-Kit-with-1-2Ah-HP-Battery-1-4Ah-HP-Battery-and-Charger-PSK1212SB/339206615?MERCH=REC-_-rv_homepage_rr-_-n/a-_-0-_-n/a-_-n/a-_-n/a-_-n/a-_-n/a",
    "dynamic": "false"
}

response = requests.get(url, params=params)
print(response.status_code)
print(response.text)

Here we are making the GET request using requests to the scrape endpoint provided by Scrapingdog. Do not forget to pass your own private Scrapingdog’s key.

Let’s run this code.

After running the code, you will get a 200 status code along with the HTML of the target page.

Now, we can use beautifulSoup to parse this data.

Parsing the required data with BeautifulSoup

Let’s start with the title. We have to find the DOM location of the title.

The title is wrapped inside a H1 tag with a class sui-line-clamp-unset.

The pricing is hidden inside the first three span tag with class sui-font-display.

The product image is wrapped inside an img tag with class sui-w-full.

All the features of the product are located inside the ul tag with class sui-text-base.

The rating of the product is wrapped inside a tag with class sui-flex-row.

Now, we have the DOM location of every data point we want to parse from the raw HTML. Let’s implement this logic with BS4.

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://api.scrapingdog.com/scrape",
    params={
        "api_key": "your-api-key",
        "url": "https://www.homedepot.com/p/RYOBI-ONE-18V-Lithium-Ion-2-0Ah-4-0Ah-Starter-Kit-with-1-2Ah-HP-Battery-1-4Ah-HP-Battery-and-Charger-PSK1212SB/339206615",
        "dynamic": "false"
    }
)
print(response.status_code)

obj={}
soup = BeautifulSoup(response.text, "html.parser")

# Title
title = soup.find("h1", class_="sui-line-clamp-unset")
obj["Title"] = title.get_text(strip=True)

# Price — dollar, cents in separate spans
price_parts = soup.find_all("span", class_="sui-font-display")[:3]
price = "".join([p.get_text(strip=True) for p in price_parts])
obj["Price"]=price

# Image — actual class in this page
img = soup.find("img", attrs={"data-testid": "small-image"})
obj["image"]=img["src"]

# Features
features_ul = soup.find("ul", class_="sui-text-base")
if features_ul:
    features = [li.get_text(strip=True) for li in features_ul.find_all("li")]
    obj["features"]=features

# Rating
rating_tag = soup.find("a", class_="sui-flex-row")
obj["Rating"]=rating_tag["title"]

print(obj)

Once you run this code you will get this beautiful parsed JSON data with all the details we were looking for.

{'Title': 'ONE+ 18V Lithium-Ion 2.0Ah, 4.0Ah Starter Kit with (1) 2Ah HP Battery, (1) 4Ah HP Battery and Charger', 'Price': '$9900', 'image': 'https://images.thdstatic.com/productImages/3f4be679-3a35-40e5-8935-f6876f66f03a/svn/ryobi-power-tool-batteries-psk1212sb-64_600.jpg', 'features': ['Up to 4X More Runtime', '4Ah & 2Ah Lithium HIGH PERFORMANCE Batteries & Charger', 'Power All 300 ONE+ Products With Any RYOBI 18V ONE+ Battery', 'View More Details'], 'Rating': '4.5 out of 5'}

Storing the data in a CSV file

extracted_data.append(obj)

df = pd.DataFrame(extracted_data)
df.to_csv("home_depot.csv", index=False)
print(" Data saved to csv successfully.")

Appends each scraped product as a dict into a list, then converts that list into a pandas DataFrame and exports it as home_depot.csv. This will create a csv file inside your folder with the scraped data.

You can repeat this process in loop with the help of a for loop for various different products on homedepot.com.

1. Is it legal to scrape Home Depot?

Scraping publicly available data is generally legal.

2. Does Home Depot block web scrapers?

Yes. It uses rate limiting, CAPTCHAs, IP bans, and JavaScript rendering to detect and block bots. That’s why you are recommended to use web scraping APIs like Scrapingdog.

3. Do I need a proxy to scrape Home Depot?

Yes. Rotating residential or ISP proxies are recommended to avoid IP bans at scale.

4. What data can I scrape from Home Depot?

Product names, prices, availability, ratings, reviews, specs, model numbers, and images.

5. Can I scrape Home Depot without JavaScript rendering?

No. Home Depot heavily relies on JavaScript to load product data, so you’ll need a headless browser or a scraping API.

6. How do I handle pagination when scraping Home Depot?

Increment the page offset parameter in the URL and loop through pages until no more results are returned.

Conclusion

Scraping Home Depot with Python gives you access to valuable product data at scale. In this guide, we covered everything from setting up your environment to extracting listings, product details, reviews, and handling pagination.

The biggest challenge is avoiding blocks, use rotating proxies and handle JavaScript rendering to stay undetected. If you want to skip the infrastructure hassle, Scrapingdog’s web scraping api handles all of that out of the box.

Happy scraping!