DEV Community: Scrapfly

Comprehensive Guide to OkHttp for Java and Kotlin

Scrapfly — Thu, 05 Dec 2024 14:20:37 +0000

When building applications that communicate over the web, a reliable and efficient HTTP client is essential. OkHttp, one of the most popular HTTP clients for Java and Android development, stands out as a powerful tool for handling network operations.

This article will explore OkHttp’s key features, setup instructions, basic usage, advanced functionalities, and common issues you might encounter.

What is OkHttp?

OkHttp is an HTTP client designed for both Java and Kotlin, widely recognized for its reliability and performance. It simplifies network communication by providing a consistent interface for making HTTP requests, handling responses, and optimizing network traffic.

Whether you're developing a mobile app or a backend service, OkHttp offers a robust solution for dealing with HTTP requests and responses efficiently.

In Android and Java development, OkHttp acts as a modern alternative to other HTTP clients, thanks to its lightweight design and support for features like connection pooling and timeouts. It’s a go-to choice for developers aiming to implement complex network operations with minimal configuration.

Main OkHttp Features

OkHttp offers an impressive range of features designed to enhance network performance and simplify development. Below is a feature table showcasing some of its capabilities:

Feature	Description
HTTP/2 Support	Enables multiplexed requests for better connection pooling.
Async Requests	Facilitates non-blocking, parallel network requests.
Interceptors	Customizes requests and responses (logging, modifying headers).
Built-in Caching	Reduces network usage by caching responses.
Timeouts	Configurable connection, read, and write timeouts.
HTTPS Support	Out-of-the-box HTTPS support with certificate pinning.
Compression	GZIP and deflate for response compression.
Redirect Handling	Automatic handling of 3xx HTTP redirects.
WebSocket Support	Full support for WebSocket connections.
Authentication Support	Basic, Digest, and custom authentication mechanisms.

While OkHttp covers a wide array of features, there are a few limitations to be aware of. Notably, it lacks built-in retry strategies, and modern conveniences like auto .json() methods are not natively included. However, these shortcomings can often be mitigated with custom implementations or third-party libraries.

Setting Up OkHttp

Setting up OkHttp is straightforward, whether you’re developing a Java application or an Android project. For Java, you can include OkHttp by adding it to your Maven or Gradle build configuration. Below are installation instructions:

To install okhttp using Maven :

<dependency>
  <groupId>com.squareup.okhttp3</groupId>
  <artifactId>okhttp</artifactId>
  <version>4.12.0</version>
</dependency>

To install okhttp using Java Gradle :

implementation 'com.squareup.okhttp3:okhttp:4.12.0'

To install okhttp using Kotlin Gradle :

implementation("com.squareup.okhttp3:okhttp:4.12.0")

Basic Usage

To get started with OkHttp, let’s explore some basic usage patterns.

GET Requests and Response Details

Making a simple GET request with OkHttp is simple. Here’s how you can retrieve the response body and status code:

OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
    .url("https://web-scraping.dev/api/products")
    .build();

try (Response response = client.newCall(request).execute()) {
    if (response.isSuccessful()) {
        System.out.println("Response Body: " + response.body().string());
        System.out.println("Status Code: " + response.code());
    }
}

Handling Errors

Error handling in OkHttp is critical to ensure smooth user experiences. Common issues such as connection timeouts or 4xx/5xx HTTP errors can be caught and managed using try-catch blocks. You can also inspect the status code to handle specific HTTP errors.

try (Response response = client.newCall(request).execute()) {
    if (!response.isSuccessful()) {
        throw new IOException("Unexpected code " + response);
    }
}

Synchronous and Asynchronous Requests

OkHttp supports both synchronous and asynchronous requests, providing flexibility based on your application’s needs. The synchronous request waits for the response before proceeding, while the asynchronous request runs in the background.

// Synchronous
Response response = client.newCall(request).execute();

// Asynchronous
client.newCall(request).enqueue(new Callback() {
    // Add callback for success:
    @Override
    public void onResponse(Call call, Response response) throws IOException {
        System.out.println(response.body().string());
    }
    // Add callback for failure:
    @Override
    public void onFailure(Call call, IOException e) {
        e.printStackTrace();
    }
});

Advanced Usage

OkHttp also supports a variety of advanced use cases, including different HTTP methods, handling headers and cookies, and configuring timeouts.

POST, PUT, HEAD, PATCH Requests

OkHttp allows you to make POST, PUT, HEAD, and PATCH requests with ease. Here’s are basic examples of each:

POST

RequestBody body = RequestBody.create(MediaType.parse("application/json; charset=utf-8"), "{\"key\":\"value\"}");
Request request = new Request.Builder()
    .url("https://httpbin.dev/post")
    .post(body)
    .build();

HEAD

Request request = new Request.Builder()
    .url("https://httpbin.dev/head")
    .head() // HEAD request does not require a body
    .build();

PUT

RequestBody body = RequestBody.create(MediaType.parse("application/json; charset=utf-8"), "{\"key\":\"updatedValue\"}");
Request request = new Request.Builder()
    .url("https://httpbin.dev/put")
    .put(body) // PUT request with a body
    .build();

PATCH

RequestBody body = RequestBody.create(MediaType.parse("application/json; charset=utf-8"), "{\"key\":\"patchedValue\"}");
Request request = new Request.Builder()
    .url("https://httpbin.dev/patch")
    .patch(body) // PATCH request with a body
    .build();

Adding Query Parameters

OkHttp provides a straightforward way to add query parameters to your requests without manually building the URL string. Using the HttpUrl.Builder class, you can easily add key-value pairs as query parameters to a URL before making the request.

OkHttpClient client = new OkHttpClient();

HttpUrl.Builder urlBuilder = HttpUrl.parse("https://web-scraping.dev/api/products")
        .newBuilder();
urlBuilder.addQueryParameter("page", "2");
urlBuilder.addQueryParameter("order", "desc");
urlBuilder.addQueryParameter("category", "apparel");

String url = urlBuilder.build().toString();

// Building the request with the URL containing query parameters
Request request = new Request.Builder()
        .url(url)
        .build();

Adding Headers

You can add custom headers to your requests, like authentication tokens or user-agent strings:

Request request = new Request.Builder()
    .url("https://web-scraping.dev/api/products")
    .addHeader("Authorization", "Bearer token")
    .build();

Adding Cookies

OkHttp also allows you to manage cookies by using its CookieJar interface. This enables you to persist cookies between requests or manage them manually.

OkHttpClient client = new OkHttpClient.Builder()
        .cookieJar(new CookieJar() {
            @Override
            public void saveFromResponse(HttpUrl url, List<Cookie> cookies) {
                for (Cookie cookie : cookies) {
                    System.out.println("Saving cookie: " + cookie);
                }
            }

            @Override
            public List<Cookie> loadForRequest(HttpUrl url) {
                return List.of(); // Empty for this example
            }
        })
        .build();

Request request = new Request.Builder()
        .url("https://web-scraping.dev/blocked")
        .build();

Adding Timeout

OkHttp gives you the ability to configure connection, read, and write timeouts for each HTTP request:

OkHttpClient client = new OkHttpClient.Builder()
        .connectTimeout(10, TimeUnit.SECONDS) // Set connection timeout
        .writeTimeout(10, TimeUnit.SECONDS) // Set write timeout
        .readTimeout(30, TimeUnit.SECONDS) // Set read timeout
        .build();

Request request = new Request.Builder()
        .url("https://web-scraping.dev/api/products")
        .build();

Adding Retries

Although OkHttp doesn’t have built-in retry strategies, you can easily add them yourself by using interceptors or implementing custom logic for retries.

To implement it using interceptors, we can create a new okhttp interceptor that has the retry logic.

public class RetryInterceptor implements Interceptor {

    private int maxRetries;

    public RetryInterceptor(int maxRetries) {
        this.maxRetries = maxRetries;
    }

    @Override
    public Response intercept(Chain chain) throws IOException {
        Request request = chain.request();
        Response response = null;
        IOException exception = null;

        int attempt = 0;

        // Attempt the request up to maxRetries times
        while (attempt < maxRetries) {
            try {
                response = chain.proceed(request);

                // If response is successful, return it
                if (response.isSuccessful()) {
                    return response;
                }
            } catch (IOException e) {
                exception = e;
            } finally {
                attempt++;
            }

            // Wait before retrying the request (optional delay)
            if (attempt < maxRetries) {
                try {
                    Thread.sleep(2000); // 2-second delay between retries
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
        }

        // If all attempts failed, throw the exception
        if (exception != null) {
            throw exception;
        }

        // If the response is null and maxRetries are exhausted, throw an IOException
        throw new IOException("Maximum retry attempts (" + maxRetries + ") exhausted");
    }
}

Now we can use this interceptor within our okhttp client builder.

OkHttpClient client = new OkHttpClient.Builder()
                .addInterceptor(new RetryInterceptor(3)) // Add custom retry interceptor
                .build();

Request request = new Request.Builder()
        .url("https://web-scraping.dev/api/products")
        .build();

Disabling HTTPS

In certain scenarios, you might need to disable HTTPS validation. While this is generally not recommended, it can be done for testing purposes:

OkHttpClient client = new OkHttpClient.Builder()
    .sslSocketFactory(createUnsafeSslSocketFactory(), new X509TrustManager() {
        // Implementation here
    })
    .build();

WebSocket

OkHttp makes it simple to work with WebSockets, allowing you to easily connect, send, and receive messages in real-time. In the example below, you will see how to establish a WebSocket connection using OkHttp, send a message to the server, and handle incoming messages asynchronously using WebSocketListener.

OkHttpClient client = new OkHttpClient();

Request request = new Request.Builder()
        .url("wss://example.com/socket")
        .build();

WebSocketListener listener = new WebSocketListener() {
    @Override
    public void onOpen(WebSocket webSocket, Response response) {
        System.out.println("WebSocket Opened");
        webSocket.send("Hello Server!");
    }

    @Override
    public void onMessage(WebSocket webSocket, String text) {
        System.out.println("Message from server: " + text);
    }

    @Override
    public void onFailure(WebSocket webSocket, Throwable t, Response response) {
        t.printStackTrace();
    }
};

client.newWebSocket(request, listener);
client.dispatcher().executorService().shutdown();

Common OkHttp Errors and Issues

While OkHttp is highly reliable, you may encounter errors from time to time. Some common issues include:

Issue/Exception	Description	Solution
java.net.SocketTimeoutException	Occurs when the connection, read, or write operation exceeds the set timeout.	Increase the timeout duration with `.connectTimeout()`, `.readTimeout()`, or `.writeTimeout()` on `OkHttpClient`.
java.net.UnknownHostException	Indicates that the domain name cannot be resolved by DNS.	Check network connection, ensure DNS is reachable, or configure a custom DNS with OkHttp.
java.io.IOException: unexpected end of stream	Often caused by network disconnection or improper server response.	Retry the request or ensure server responses are properly formatted.
SSLHandshakeException	SSL/TLS handshake fails, commonly due to incorrect certificates or unsupported protocols.	Ensure correct SSL certificates, or set `hostnameVerifier` and `certificatePinner` if needed.
java.net.ProtocolException	Indicates a protocol error, such as an invalid HTTP response or request method.	Verify request setup (e.g., method type, headers) or handle specific protocols.
java.io.EOFException	Happens when OkHttp reads an incomplete response body from the server.	Use interceptors to handle partial responses or consider increasing the read timeout.
java.net.ConnectException	Failed to connect to the server, often due to unreachable server or network issues.	Check server availability, network connection, or retry the connection.
HTTP 413 Payload Too Large	Server rejects the request due to a large payload.	Reduce payload size or consult the server for acceptable payload limits.
HTTP 429 Too Many Requests	Server rate-limiting response due to too many requests in a short period.	Implement retry logic with backoff or adhere to rate limits.
ResponseBody is null	Occurs when a request is successful but the server returns no content.	Check if the endpoint is expected to return data, or handle null `ResponseBody` safely.
TimeoutException: A connection timeout occurred	Appears when establishing the connection takes too long.	Increase connection timeout or use exponential backoff for retries.
IOException: Cleartext HTTP traffic not permitted	Occurs on Android when accessing a plain HTTP URL instead of HTTPS.	Add `cleartextTrafficPermitted` in the network security config or switch to HTTPS if possible.
IllegalArgumentException: unexpected url	Raised when an invalid or malformed URL is used in the request.	Ensure the URL is correctly formatted and valid.

Power Up With Scrapfly

While OkHTTP can be used for web scraping purposes, you will often find yourself facing blocks and rate-limits.

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

Here is a simple example of how you can use okhttp with Scrapfly's Scraping API.

import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;

import java.io.IOException;

public class OkHttpExample {
    public static void main(String[] args) {
        OkHttpClient client = new OkHttpClient();
        HttpUrl.Builder urlBuilder = HttpUrl.parse("https://api.scrapfly.io/scrape")
                .newBuilder();
        // Required parameters: your API key and URL to scrape
        urlBuilder.addQueryParameter("key", "YOUR_API_KEY");
        urlBuilder.addQueryParameter("url", "https://web-scraping.dev/product/1");
        // Optional parameters:
        // enable anti scraping protection bypass
        urlBuilder.addQueryParameter("asp", "true");
        // use proxies of a specific countries
        urlBuilder.addQueryParameter("country", "US,CA,DE");
        // enable headless browser
        urlBuilder.addQueryParameter("render_js", "true");
        // see more on scrapfly docs: https://scrapfly.io/docs/scrape-api/getting-started#spec

        // Building and send request
        String url = urlBuilder.build().toString();
        Request request = new Request.Builder()
                .url(url)
                .build();
        try (Response response = client.newCall(request).execute()) {
            if (response.isSuccessful()) {
                System.out.println("Response Body: " + response.body().string());
                System.out.println("Status Code: " + response.code());
            } else {
                System.out.println("Request Failed: " + response.code());
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

FAQ

To wrap our intro to okhttp let's take a look at some frequently asked questions that we might have missed in our guide:

Can OkHttp be used for Crawling?

Yes, OkHttp can be used for web scraping and crawling tasks. It handles multiple concurrent HTTP requests, making it a solid choice for crawlers. However, for specialized crawling purposes, you may want to look into dedicated crawl APIs for better efficiency.

Can OkHttp handle large file uploads or downloads?

Yes, OkHttp is capable of handling large file uploads and downloads efficiently. It allows you to stream data via RequestBody for uploads and ResponseBody for downloads, ensuring that large files can be processed in chunks without consuming excessive memory. You can implement custom logic to track progress, manage buffers, and avoid memory issues during these operations.

Is OkHttp thread-safe?

Yes, OkHttp is designed to be thread-safe. The OkHttpClient instance itself can be shared across multiple threads, making it ideal for concurrent requests. However, individual Request and Response objects are not thread-safe, so they should be used in a single thread or context. To maximize efficiency, it’s recommended to reuse the same OkHttpClient instance across your application.

Summary

OkHttp is a versatile and powerful HTTP client that provides developers with the tools they need for smooth, efficient network communication in Java and Android development. Here's a quick recap of the key points we covered:

Supports HTTP/2 , asynchronous requests , interceptors , built-in caching , timeouts , and HTTPS.
Offers features like compression , redirect handling , WebSocket , and authentication support.
Simple to set up using Maven or Gradle.
Provides an easy way to handle GET , POST , and other HTTP methods, both synchronously and asynchronously.
Add custom headers , cookies , and configure timeouts easily.
Handle advanced cases like retry logic and HTTPS disabling.
Typical errors include timeouts , connection issues , and SSL/TLS errors , all of which can be managed with proper handling and configuration.

OkHttp is a reliable choice for developers seeking control, efficiency, and a rich set of features for network communication. Its strengths lie in its simplicity combined with the flexibility needed for more advanced use cases.

Instant Data Scraper Guide - Web Scraping with No Code

Scrapfly — Wed, 04 Dec 2024 15:43:44 +0000

Web scraping has become an essential tool for accessing and organizing data from the web, but coding skills are often seen as a barrier. Thankfully, no-code automation tools allow anyone to collect data effortlessly—without writing any code.

By leveraging accessible automation tools like Zapier and Make.com, you can automate the scraping process in minutes. In this guide, we'll explore how these no-code solutions can be combined with tools like Google Sheets integrations and browser extensions to make data scraping easy and accessible.

Whether you're a beginner or just looking for an efficient way to gather web data, this guide will show you how to instantly scrape data without any technical headaches.

What is Instant Data Scraping?

Instant Data Scraping is a concept that simplifies the process of collecting data from websites by using accessible, no-code tools.

Instead of traditional coding methods, this approach focuses on leveraging user-friendly interfaces and automated workflows to perform web scraping. Think of it as a combination of tools like Zapier or Make.com that enable you to extract data as effortlessly as setting up a simple workflow.

With this concept, changes in documents (such as new entries in a Google Sheet or interactions through browser extensions) can trigger data scraping actions.

This makes data collection as easy as updating a spreadsheet or setting up an automation, allowing anyone to harness the power of web scraping without needing to learn how to code.

Instant Scraping with Google Sheets

One of the most accessible ways to start instant data scraping is by integrating it with Google Sheets. This method takes advantage of change detection, such as:

Adding new rows
Deleting rows
Updating existing rows
Creating new sheets

All of these actions can trigger web scraping actions automatically. Imagine updating a Google Sheet with a website URL and instantly scraping in relevant data without lifting a finger.

To streamline the scraping and data extraction process, we’ll also be using Scrapfly's no-code integrations.

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

While Scrapfly's API's require coding knowledge to be used, Scrapfly offers no-code integrations with automation tools like Zapier and Make.com. Those integrations allow everyone to leverage Scrapfly's cutting-edge features through simple and accessible UIs.

Connecting Google Sheets and Scrapfly with Make.com

Make.com is an automation platform that allows users to build complex workflows without writing code. It’s similar to Zapier, but with added flexibility for connecting apps and services in more detailed ways.

In this guide, we’ll use Make.com to automate the entire scraping process—from detecting changes in Google Sheets to extracting data via Scrapfly.

Create an new spreadsheet in google sheets. The spreadsheet should have 2 sheets, one to add new URLs to scrape and the other is to store the data extracted from the URLs.

Create a new scenario from you make.com dashboard using the "Create a new scenario button":

In the scenario builder, we will start with the input data node, which is the Google Sheets action "Watch New Rows". The action will prompt you to connect your google account to get access to the your sheets.

Once the account is connected, fill the action options as in the image below, note that the spreadsheet id and the sheet name vary according to the sheet you have created on google sheets.

Now that we set all the action options, the action returns a list of new rows every time it runs.

The action will automatically have a trigger on it that triggers the action every 15 minutes. Meaning that once we are done with the scenario and it is activated, it will check for new changes in the google sheet every 15 minutes. Feel free to change this time interval to suit your needs.

Since the "Watch New Rows" action returns a list of new rows, we will need to add an "Iterator" node that iterates over this list and allows us to scrape each URL one by one.

The URLs will be located in column A of the sheet, so will add this column as the iterartor array.

Now we will use the Scrapfly's "Scrape a webpage" action.

The action will prompt you to connect you Scrapfly account using the API key which you can find a the top of your Scrapfly dashbaord.

The URL value fed to the "Scrape a webpage" action will be the value coming from the "Iterator".

The "Scrape a webpage" action returns a collection with a "Result" object that contains the "Content" of the the scraped webpage.

Now connect the "Scrape a webpage" action to a new "Extract Data" action provided by Scrapfly. This action allows you to extract structured JSON data from unstructured data like HTML, raw text, or markdown using AI.

Inside the action options, set the body to the content returned by the scraping action.

Also set the extraction prompt, which is a simple AI prompt specifying what data you need to be extracted. You can set this prompt to be static for all URLs, but it can also be dynamic and fed from the excel sheet for advanced uses.

Once we have the extracted data, we can transform it into a JSON string using the "Transform to JSON" action.

Now that we have the extracted data as a JSON string, we can create a new sheet to store the results (Sheet 2) and send the data to the google sheet using the "Add a Row" Action.

Finally, the scenario is done, activate it from your make.com dashboard and add a new URL to your spreadsheet to see it in the works.

With this we've made a no code scraper using Google Sheets, Make.com and Scrapfly in just a few minutes! This is a simple example of what you can do with these tools, you can add more complexity to the scenario by adding more actions and conditions to suit your needs.

Scrapfly Screenshot API Action

Scrapfly also offers the "Capture Webpage Screenshot" action on make.com. This action allows you to automate capturing screenshots for any wesbite, which can be used to web testing or price monitoring tools.

In its most basic form, the action expects the website URL for which the screenshot will be captured. You can also provide additional options to the action for more customization, like:

Screenshot image format (png, jpg, webp...)
Capture target (full viewport, specific element with CSS selector or XPATH)
Screen Resolution (1920x1080, 2560x1440...)
And more

You can add this action to scenario we built above and store the image on Google Drive.

Instant Scraping Inputs

Beyond Google Sheets, there are many other ways to trigger instant data scraping using change detection. The beauty of Instant Data Scraping lies in its flexibility—you can set up automations to kick off data scraping whenever specific changes are detected in a variety of inputs. Here are some examples:

Email Triggers : Automatically scrape data when specific types of emails are received, such as notifications or order confirmations.

Slack Bots : Use a Slack bot to detect messages in a particular channel and trigger a scraping workflow.

Webhook Triggers : Use webhooks to start scraping whenever data is pushed from another service or app.

Cloud File Uploads : Detect new file uploads to cloud storage (like Google Drive or Dropbox) and initiate a scraping process to extract data from documents or links.

Database Changes : Set up triggers based on changes in a database, such as new records being added or existing ones updated.

With these diverse input options, Instant Data Scraping provides a powerful, adaptable way to gather the information you need—right when you need it. This flexibility makes it ideal for anyone wanting to automate their data collection workflows without spending time on manual processes or coding.

On-Demand Scraping

Another powerful feature of Instant Data Scraping is the ability to initiate scraping workflows on demand. This means you can start data collection whenever it suits you, rather than relying solely on change detection triggers. Here are a few options to consider:

Google Forms Integration : Google Forms can be linked to Google Sheets to easily add new entries that trigger the scraping workflow. For example, users can submit a form with a URL, and that URL is instantly scraped using Scrapfly.
Zapier Browser Extension : Zapier has a browser extension that allows you to manually trigger actions while browsing. With this, you can scrape data from a web page with a single click, making it convenient for quick, on-the-fly data collection.
Make.com Mobile App : Make.com also has a mobile app that allows you to trigger workflows from your phone. This means you can initiate scraping actions from anywhere, making data collection as simple as tapping a button on your device.

These on-demand scraping options provide the flexibility to collect data exactly when you need it. Whether you’re at your desk or on the go, you can always get the data you need instantly and without any coding.

Scheduled Scraping

While Instant Data Scraping provides excellent flexibility with real-time and on-demand data collection, there are times when scheduling scraping tasks makes more sense.

Scheduled scraping is perfect for when you need consistent data at regular intervals without having to manually initiate the process. For example, you can set up scraping to happen daily, weekly, or even hourly, depending on your requirements.

Using tools like Scrapfly, you can easily set up these schedules to ensure your data is always up-to-date. You won't be locked into a single solution either — using platforms like Make.com or Zapier, you can schedule scraping workflows to pull information at your preferred times, giving you the freedom to maintain an automated yet highly controlled data collection process.

Here an example of how scheduling works in make.com:

This alternative provides a balanced approach for users who require both instant access to data and regular updates, ensuring no opportunity is missed, and data accuracy is always maintained.

Summary

To recap what we covered in this guide:

Instant Data Scraping : A no-code concept for web scraping using accessible tools like Zapier, Make.com, and Scrapfly, enabling anyone to gather data without coding skills.
What is Instant Data Scraping? : Explained how tools like browser extensions and Google Sheet integrations can automate web scraping.
Google Sheets Integration : Demonstrated change detection powered scraping using Google Sheets, Scrapfly, and Make.com for an end-to-end automation workflow.
Diverse Inputs for Scraping : Explored different trigger inputs, such as email notifications, Slack bots, database changes, webhooks, and cloud file uploads.
On-Demand Scraping : Showed how scraping can be triggered manually using tools like Google Forms, the Zapier browser extension, or the Make.com mobile app.
Scheduled Scraping : Discussed how scheduled scraping is a viable alternative for consistent data collection, allowing workflows to run automatically at specified intervals.

Whether you need real-time data, scheduled scraping, or on-demand access, Instant Data Scraping gives you multiple flexible options for gathering the data you need—without writing a single line of code.

What is HTTP 499 Status Code and How to Fix it?

Scrapfly — Fri, 29 Nov 2024 08:21:36 +0000

Imagine this: You’re surfing the web or managing your server when, suddenly, you’re greeted with an error you’ve never encountered before— 499. It’s not as famous as the notorious 404 or the dreaded 500, but this cryptic error has been making its way into logs, frustrating developers and users alike.

So, what exactly is the 499 error? Why does it appear, and what can you do to prevent it? In this article, we’ll demystify this client-side HTTP status code, explore its origins, and provide actionable steps to resolve it.

Understanding the 499 Status Code

To understand the 499 status code , we first need to recognize that it doesn’t belong to the standard HTTP status codes outlined by the Internet Engineering Task Force (IETF). Instead, it’s a non-standard, server-specific code introduced by Nginx, one of the most popular web servers globally.

Status code 499, often labeled as "Client Closed Request" , indicates that the client (browser or API consumer) terminated the connection before the server could deliver its response. In simpler terms, the client grew impatient and hung up the call before the server could answer.

Why a Non-standard Error?

499 error's association with Nginx stems from the server’s need to log this specific client-side behavior.

Unlike standard HTTP codes, which aim for universal implementation, the 499 code helps Nginx administrators monitor and debug unique issues caused by client-side interruptions or network latency.

Understanding its origin highlights an important distinction:

The 499 error is not a bug in the server or application but a signal of external factors, such as poor client connectivity or mismatched timeout settings. This makes it an essential tool for diagnosing performance bottlenecks in client-server communication.

By decoding its definition and purpose, we can see how the 499 status code serves as a valuable diagnostic indicator, helping web developers uncover the story behind incomplete requests. But why does it happen, and what does it reveal about the client-server relationship? Let’s explore further.

Causes of the 499 Status Code

The 499 status code is a direct result of interruptions in the client-server communication process. Several common scenarios can trigger this error, each shedding light on different aspects of how requests are handled:

Client-Side Request Cancellations : Users may manually stop loading a page or an API consumer may terminate a request prematurely. This abrupt action cuts the connection before the server has a chance to respond, leading to status code 499.
Network Instability or Interruptions : Unreliable connections, such as weak Wi-Fi or mobile data, can cause requests to drop unexpectedly. The server still processes the request, only to find the client has already disconnected.
Server-Side Delays Leading to Client Timeouts : When a server takes too long to process a request, clients often lose patience. Whether due to large database queries or overloaded servers, these delays can cause the client to close the connection and result in a logged 499 error.
Client-Side Timeout Configurations : Some clients, such as browsers or API integrations, have strict timeout settings. If the server response exceeds these predefined thresholds, the client cancels the request, resulting in a 499 error.
Overzealous Proxy or Firewall Rules : Intermediate systems like proxies or firewalls can sometimes terminate requests if they detect unusual patterns or if a timeout configuration is too aggressive.
Misconfigured APIs or SDKs : When third-party APIs or client-side SDKs are not configured properly, they may inadvertently close connections too soon, especially in high-latency environments.

Understanding these causes is crucial because it highlights the shared responsibility between clients and servers in maintaining seamless communication. Identifying the root cause helps determine whether the solution lies in optimizing client behavior, improving server performance, or addressing network issues.

Impact on Web Scraping and Automation

For those relying on web scraping or automated workflows, encountering 499 errors can present significant challenges. These errors interrupt the seamless flow of data extraction, making it difficult to retrieve the information needed efficiently. When clients terminate requests prematurely, the scraper may fail to capture complete responses, leading to incomplete datasets or broken scripts.

In automated workflows, where tasks are chained and dependent on accurate data retrieval, a 499 error can disrupt the entire process. For example, a timeout in one step of the workflow might cascade into downstream failures, wasting time and resources.

Addressing these issues often requires robust error-handling mechanisms and timeout configurations. Ensuring that automated tools can retry failed requests or gracefully handle incomplete responses is essential to maintaining reliability in scraping and workflow automation.

Strategies to Mitigate 499 Errors

To minimize the occurrence of 499 errors, it’s essential to adopt proactive strategies that enhance the resilience of client-server interactions. Here are key approaches:

Retry Mechanisms with Exponential Backoff

When a request fails due to a 499 error, using a retry mechanism with exponential backoff can prevent repeated abrupt failures. This approach delays successive retries by increasing intervals, reducing the likelihood of overwhelming the server.

Here are some examples on how to implement exponential backoff retries in Python and Javascript:

import time
import requests

def fetch_with_retries(url, max_retries=5):
    delay = 1
    for attempt in range(max_retries):
        try:
            response = requests.get(url, timeout=10)
            if response.status_code != 499:
                return response
        except requests.exceptions.RequestException:
            pass
        time.sleep(delay)
        delay *= 2
    return None

async function fetchWithRetries(url, maxRetries = 5) {
    let delay = 1000; // Start with 1 second
    for (let attempt = 0; attempt < maxRetries; attempt++) {
        try {
            const response = await fetch(url, { signal: AbortSignal.timeout(10000) });
            if (response.status !== 499) return response;
        } catch (error) {
            console.error(`Attempt ${attempt + 1} failed`);
        }
        await new Promise((resolve) => setTimeout(resolve, delay));
        delay *= 2;
    }
    return null;
}

Client-Side Timeout Settings

Timeout settings play a critical role in reducing 499 errors. Misconfigured timeout values can cause clients to terminate requests prematurely, especially for long-running processes. Below are examples of configuring timeouts in common HTTP client libraries:

response = requests.get('https://example.com', timeout=30) # Timeout set to 30 seconds

fetch('https://example.com', { signal: AbortSignal.timeout(30000) }) // Timeout set to 30 seconds
    .then(response => response.json())
    .then(data => console.log(data))
    .catch(error => console.error(error));

axios.get('https://example.com', { timeout: 30000 }) // Timeout set to 30 seconds
    .then(response => console.log(response.data))
    .catch(error => console.error(error));

Stable Network Connections

A stable and reliable network connection is crucial for avoiding interruptions. Consider the following practices:

Use wired connections over wireless for critical tasks.
Implement redundancy in network infrastructure, such as failover mechanisms.
Monitor connection health and latency in real time to preempt issues.

By integrating these strategies, you can significantly reduce the frequency of 499 errors, ensuring smoother communication and more reliable workflows.

Best Practices for HTTP Clients and Web Scrapers

When building reliable HTTP clients or web scrapers, following best practices can significantly reduce the impact of errors like 499. Below are actionable steps to improve resilience and efficiency:

Monitoring and Logging

Accurate monitoring and logging help identify patterns and frequency of 499 errors, enabling you to address their root causes effectively. We will demostrate how to effectively log errors like the http status code 499 error in Python and Javascript.

For Python we will be using the logging module which is built-in in the Python standard library. While for Javascript, we will be using a popular third party library for logging called Winston

import logging
import requests

# Configure logging
logging.basicConfig(level=logging.INFO, filename='errors.log', format='%(asctime)s - %(levelname)s - %(message)s')

def fetch_url(url):
    try:
        response = requests.get(url, timeout=10)
        if response.status_code == 499:
            logging.warning(f"499 error encountered for URL: {url}")
        return response
    except requests.exceptions.RequestException as e:
        logging.error(f"Request failed: {e}")
        return None

# Example usage
fetch_url("https://example.com")

const winston = require('winston');

// Configure winston logging
const logger = winston.createLogger({
    level: 'info',
    format: winston.format.combine(
        winston.format.timestamp(),
        winston.format.printf(({ timestamp, level, message }) => `${timestamp} - ${level}: ${message}`)
    ),
    transports: [new winston.transports.File({ filename: 'errors.log' })],
});

async function fetchUrl(url) {
    try {
        const response = await fetch(url);
        if (response.status === 499) {
            logger.warn(`499 error encountered for URL: ${url}`);
        }
        return response;
    } catch (error) {
        logger.error(`Request failed: ${error.message}`);
        return null;
    }
}

// Example usage
fetchUrl("https://example.com");

Robust Error-Handling

Implement error-handling mechanisms that not only retry failed requests but also log and categorize errors for debugging. This ensures that transient issues like 499 errors are managed without affecting the overall workflow.

Wrap network requests in try-catch blocks or similar structures to gracefully handle exceptions.
Use exponential backoff strategies, as demonstrated earlier, for retries to prevent overwhelming the server.

Ethical Scraping Practices

Ethical scraping practices reduce the chances of overloading servers and triggering client-side terminations like 499 errors. These include:

Rate Limiting : Avoid making too many requests in a short time. Use libraries like time.sleep in Python or setTimeout in JavaScript to introduce delays.
Respecting Robots.txt : Check the site’s robots.txt file to understand which resources are allowed to be scraped.
User Agent Rotation : Use a pool of user agents to mimic legitimate traffic patterns while scraping.

Incorporating these practices ensures smoother scraping operations and fosters a responsible approach to web automation. By monitoring 499 errors and adopting robust handling routines, you can create resilient, efficient systems while respecting the servers you interact with.

Power Up with Scrapfly

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

Summary

The 499 status code, though non-standard, plays a significant role in diagnosing issues in client-server communication, particularly in Nginx environments. It arises primarily from client-side interruptions, unstable networks, or server delays. This makes it a unique but essential tool for debugging and performance monitoring.

By understanding the causes and adopting best practices, developers and web scrapers can handle 499 errors effectively, ensuring seamless communication between clients and servers while maintaining ethical and efficient operations.

How to Use cURL to Download Files?

Scrapfly — Mon, 25 Nov 2024 09:51:46 +0000

Curl, short for "Client URL," is a versatile command-line tool used for transferring data with URLs. It's widely favored by developers and system administrators for its ability to interact with a multitude of protocols such as HTTP, HTTPS, FTP, and more.

Using curl to download files simplifies the process by enabling direct command-line interaction with web resources. Curl is not only efficient and lightweight — operating without the need for a graphical interface — but also cross-platform, working seamlessly on Linux, macOS, and Windows systems.

In this article, we'll explore how to use curl to download a file from the web, covering various use cases and demonstrating the tool's versatility.

Why Use Curl to Download Files?

Curl stands out as an exceptional file downloading tool, offering a robust set of features that make it indispensable for developers. Here's what makes curl particularly powerful for downloading files:

Multi-Protocol Support

Handles various protocols like HTTP, HTTPS, FTP, and SFTP.
Eliminates the need for multiple tools when working with different protocols.

Resume Interrupted Downloads

Use the -C - option to continue downloads from where they left off.
Saves time and bandwidth by avoiding the need to restart downloads.

Bandwidth Management

Limit download speeds using --limit-rate to manage bandwidth usage.
Prevents downloads from consuming all available network resources.

Proxy Support

Easily configure proxies using options like -x or --proxy.
Supports various proxy types, including HTTP, HTTPS, SOCKS4, and SOCKS5.

Authentication Handling

Supports a range of authentication methods, including Basic, Digest, NTLM, and OAuth.
Access protected resources seamlessly.

Secure Transfers

Supports SSL/TLS protocols for secure file transfers.
Verify SSL certificates and use secure authentication methods.

Cross-Platform Compatibility

Available on Linux, macOS, Windows, and more.
Consistent functionality across different operating systems.

Automation and Scripting

Easily integrates into scripts for automated tasks.
Ideal for scheduled downloads using cron jobs or Windows Task Scheduler.

Curl's robust feature set makes it an excellent choice for downloading files, whether you're handling simple tasks or complex download operations. Its flexibility and efficiency empower users to manage downloads effectively in various environments.

You can learn more about curl and its options in our article about using curl for web-scraping

Now let's explore the basic usage of curl for downloading files and then dive deeper into more complex and unconventional scenarios.

Curl Basic File Download Options

By default, when curl is run on a file URL without any extra options, the file content is displayed is the terminal.

curl https://web-scraping.dev/assets/pdf/tos.pdf

However, you can use curl to save to file with its original name using the -O (uppercase "O" for Output) option:

curl -O https://web-scraping.dev/assets/pdf/tos.pdf

This command saves the file as tos.pdf, retaining the original filename.

Custom File Name on Download

To save the downloaded file with a custom name, use the -o (lowercase "o") option followed by the desired filename:

curl -o [filename] [URL]

Example:

curl -o web-scraping-tos.pdf https://web-scraping.dev/assets/pdf/tos.pdf

This command downloads tos.pdf and saves it as web-scraping-tos.pdf on your local machine.

Show Progress Bar / Download Silently

Curl show a progress meter by default. However, you can suppress the progress meter and show a simple progress bar instead.

Show Progress Bar

Replace the default progress meter with a simple progress bar using --progress-bar:

curl -O --progress-bar https://web-scraping.dev/assets/pdf/tos.pdf

Download Silently

To suppress all output, including progress and error messages, use the -s or --silent option:

curl -O -s https://web-scraping.dev/assets/pdf/tos.pdf

Silent Mode with Error Messages

If you want to hide the progress meter but still see error messages, combine -s with -S:

curl -O -s -S https://web-scraping.dev/assets/pdf/tos.pdf

Retry for Unstable Connections

For unreliable network connections, you can configure curl to retry downloads automatically:

Set Number of Retries

Use the --retry option followed by the number of retry attempts:

curl -O --retry [number] [URL]

Example:

curl -O --retry 5 https://web-scraping.dev/assets/pdf/tos.pdf

This command retries the download up to 5 times upon failure.

Specify Retry Delay

To add a delay between retries, use --retry-delay:

curl -O --retry 5 --retry-delay [seconds] [URL]

Example:

curl -O --retry 5 --retry-delay 10 https://web-scraping.dev/assets/pdf/tos.pdf

This adds a 10-second pause between each retry attempt.

Retry on All Errors

By default, curl retries on transient errors. To make it retry on all errors, use --retry-all-errors:

curl -O --retry 5 --retry-all-errors [URL]

Example:

curl -O --retry 5 --retry-all-errors https://web-scraping.dev/assets/pdf/tos.pdf

Handling Large File Downloads

Downloading large files can pose challenges such as network congestion or impacting other users on the same network. Curl offers options to manage these issues effectively.

To prevent a large download from consuming all your available bandwidth, you can limit the download speed using the --limit-rate option:

curl -O --limit-rate [speed] [URL]

Example:

curl -O --limit-rate 500k https://web-scraping.dev/assets/pdf/tos.pdf

This command limits the download speed to 500 kilobytes per second. You can specify the speed using suffixes:

k or K for kilobytes (e.g., 500k)
m or M for megabytes (e.g., 2M)

Benefits:

Bandwidth Management : Ensures other network activities aren't slowed down.
Network Stability : Reduces the risk of connection drops due to high bandwidth usage.

Insecure Downloading

In some cases, you might need to use cURL to download a file from a server with an invalid or self-signed SSL certificate. Curl verifies SSL certificates by default, which can block these downloads.

Disable SSL Certificate Verification

Warning: Disabling SSL verification can expose you to security risks like man-in-the-middle attacks. Use this option only when you're certain about the server's trustworthiness.

To bypass SSL certificate checks, use the -k or --insecure option:

curl -O -k https://web-scraping.dev/assets/pdf/tos.pdf

This command tells curl to ignore SSL certificate validation and proceed with the download.

Verifying File Integrity

Ensuring that a downloaded file hasn't been tampered with is crucial, especially for important or large files. You can verify file integrity using checksum tools like sha256sum.

Using sha256sum to Verify Downloads

Steps:

Download the File and Its Checksum

curl -O https://example.com/file.zip
curl -O https://example.com/file.zip.sha256

Verify the Checksum

sha256sum -c file.zip.sha256

The -c option tells sha256sum to check the file against the provided checksum.

Manual Verification:

If the checksum isn't provided in a file:

Get the Expected Checksum

Obtain the checksum value from the website or provider.

Calculate the Downloaded File's Checksum

sha256sum file.zip

This command outputs a checksum that you can compare with the expected value.

Example Output:

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 file.zip

Benefits:

Security : Confirms the file hasn't been altered maliciously.
Data Integrity : Ensures the file isn't corrupted due to network issues.

Handling Authentication

When downloading files from protected resources, authentication is often required. Curl supports various authentication methods to access these resources.

Authorization Header

To include an authorization token or API key in your request, use the -H option to add a custom header:

curl -O -H "Authorization: Bearer your_token_here" https://api.example.com/securefile.zip

This example uses bearer token authentication, but you can use any other authentication method supported by curl.

Cookie Session

If authentication relies on session cookies, you can manage cookies using curl:

When logging in, save the session cookies to a file using the -c option:

curl -c cookies.txt -d "username=user&password=pass" https://example.com/login

The -d option sends POST data for login credentials.
Cookies received during login are saved to cookies.txt.
Use Saved Cookies

Use the saved cookies for subsequent requests with the -b option:

curl -O -b cookies.txt https://example.com/securefile.zip

Benefits:

Session Management : Maintains login sessions across multiple requests.
Automated Workflows : Scripts can handle login and file download processes seamlessly.

Utilizing these options enhances the reliability of your file downloads, ensuring efficiency, security, and smoother operations even with unstable internet connections.

Curl Command Builder

To simplify the process of creating cURL commands for file downloads, we've created a curl command builder tool. This interactive form allows you to select various options and generate the corresponding curl command instantly:

Check it out here

Automating Curl Downloads with Crontab

Automating file downloads ensures you always have the latest data without manual effort. By integrating curl with crontab, you can schedule downloads to run at specified times, enhancing efficiency and productivity.

What Is Crontab?

Crontab is a time-based job scheduler in Unix-like operating systems. It allows users to schedule scripts or commands to run automatically at predefined times or intervals.

Steps to Automate Downloads Using Crontab

1. Create a Download Script (Optional)

Write the Script

Create a shell script (e.g., download.sh) that contains your curl command:

#!/bin/bash
# Navigate to the desired directory
cd /path/to/download/directory

# Download the file using curl
curl -O https://example.com/file.zip

Make the Script Executable

chmod +x /path/to/download.sh

2. Edit the Crontab File

Open Crontab Editor

crontab -e

Add a New Cron Job

Insert a line following the cron syntax:

* * * * * /path/to/command

Example: Schedule the Script to Run Daily at 2 AM

0 2 * * * /path/to/download.sh

Fields Explained:

Minute: 0
Hour: 2 (2 AM)
Day of Month: * (Every day)
Month: * (Every month)
Day of Week: * (Every day of the week)

3. Save and Exit

After adding your cron job, save the file. The cron service will automatically pick up the new schedule.

Automating curl downloads with crontab streamlines your workflow, ensuring timely and consistent data retrieval. Whether you're updating datasets, synchronizing files, or performing regular backups, this combination offers a robust solution for scheduled tasks.

Bypassing Download Blocks

When attempting to use curl to download files, you might encounter situations where the download is blocked or fails. This can be due to various reasons such as network restrictions, server configurations, or security measures that prevent automated requests.

The most common reason for download blocks is that the server is blocking automated requests. To bypass this, you can add a custom browser user-agent string to your request headers to mimic a real browser request.

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" https://example.com/file.zip

This example uses the -A option to set a custom user-agent string. You can replace the string with any other user-agent string that mimics a real browser request.

Changing the user-agent string is the most basic method to bypass download blocks. However, some servers are sophisticated enough to still block requests with custom user-agent strings. In these cases, you may need to use a more advanced tools like curl-impersonate.

Curl-impersonate is a modified version of cURL that simulates the TLS fingerprint of major web browsers, like Chrome, Firefox, Edge and Safari, by mimicing their TLS and HTTP2 configuration. It also overrides the default cURL headers, such as the User-Agent, with regular header values. This makes the cURL Impersonate requests look like those sent from the browsers, preventing the firewalls from detecting the usage of HTTP clients.

You can learn more about curl-impersonate in our dedicated guide on using curl-impersonate for web-scraping

Power Up File Downloads with Scrapfly

Downloading files programmatically can quickly become a cumbersome task. Especially when the files are protected against automation and bots using sophisticated bot protection systems that cannot be bypassed with tools like curl-impersonate.

Scrapfly has millions of proxies and connection fingerprints that can be used to bypass protection against automated traffic and significantly simplify your file download process.

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

For example, here is how to use Scrapfly's web scraping API to download a file, we will use Scrapfly's Pyhton SDK to call the API:

from scrapfly import ScrapflyClient, ScrapeConfig
import base64

scrapfly = ScrapflyClient(key="YOUR SCRAPFLY KEY")

FILE_URL = "https://web-scraping.dev/assets/pdf/tos.pdf"

response = scrapfly.scrape(
    ScrapeConfig(
        url=FILE_URL,
        asp=True,
    )
)

## decode base64 file data
file_data = base64.b64decode(response.result.content)

with open("tos.pdf", "wb") as f:
    f.write(file_data)

Scrapfly's API automatically detects that the requested URL is a file and return the binary content of the file encoded with base64. Which is why we decoded the content returned by the API before we saved it to a file called tos.pdf.

FAQ

Wrapping up, here are some common questions concering downlaoding files with curl:

Can I resume an interrupted download with `curl`?

Yes, you can resume an interrupted download by using the appropriate option in curl that allows you to continue from where the download stopped, which is especially useful for large files or unstable connections.

Is wget a better alternative to curl for downloading files?

wget is another command-line tool specifically designed for downloading files. While curl is versatile and supports various protocols and features, wget is often preferred for its simplicity in handling recursive downloads and its ability to download entire websites. You can learn more about the differenced between curl and wget in our dedicated curl vs wget article

How do I download multiple files at once using `curl`?

You can download multiple files simultaneously by specifying multiple URLs in a single command or by using scripting methods to loop through a list of URLs, allowing for efficient batch downloads.

Summary

Curl is a versatile tool when it comes to downloading files, offering:

Multi-Protocol Support : Works with HTTP, HTTPS, FTP, and more.
Resume Capability : Restarts interrupted downloads with ease.
Proxy and Bandwidth Management : Supports proxies and limits download speed.
Authentication Support : Handles cookies, tokens, and secured resources.
Automation : Integrates with scripts and scheduling tools like crontab.

For advanced needs, tools like curl-impersonate or services like Scrapfly can bypass sophisticated bot protections, offering:

Enhanced Bypass Capabilities : Overcomes anti-bot systems.
API Flexibility : Simplifies complex file downloads with robust solutions.

Curl’s feature-set make it essential for managing simple to complex downloads efficiently.

What is Error 1015 (Cloudflare) and How to Fix it?

Scrapfly — Wed, 20 Nov 2024 08:28:50 +0000

Encountering Cloudflare Error 1015 can be frustrating, especially when you're trying to access important content or services online. This error indicates that you've been rate-limited, but what does that mean, and how can you fix it?

In this article, we'll explore the causes of Error 1015 and provide practical solutions to mitigate it.

What is Error 1015?

Cloudflare Error 1015, commonly displayed as _ "You are being rate limited" _, occurs when a user exceeds the number of requests allowed by a website within a specific time frame.

The error can include one of the following messages:

"error 1015 rate limited"
"service resource is being rate limited"
"error 1015 you are being rate limited"
or simply "this resource is being rate limited"

All of these messages relate to the same error 1015 error just different in presentation.

This rate-limiting is a protective mechanism to safeguard websites from malicious activities such as Distributed Denial of Service (DDoS) attacks, brute-force login attempts, or web scraping bots.

Some reasons that trigger a Cloudflare 1015 error include:

Excessive Requests : Making too many requests in a short period, intentionally or unintentionally.
Automated Scripts or Bots : Using automated tools that send multiple requests rapidly.
Shared Networks : Being on a network (like a corporate or school network) where multiple users access the same service simultaneously.

Where is Error 1015 Encountered?

Since error 1015 "rate limited" is Cloudflare specific, it's only encountered if a website uses Cloudflare Web Application Firewall (WAF) or Cloudflare's anti-bot services.

Cloudflare Web Application Firewall (WAF)

Cloudflare WAF acts as a shield between your website and the internet. It monitors and filters incoming traffic based on predefined security rules.

Functions of WAF:

Blocks Malicious Traffic: Filters out attacks like SQL injection, cross-site scripting (XSS), and other web exploits.
Customizable Rules: Allows website owners to set specific rules tailored to their application's needs.
Real-Time Monitoring: Provides insights into the nature of threats and traffic patterns.

WAF blocks malicious traffic by returning a 1015 Error page to the client indicating they are being rate-limited.

Cloudflare Anti-Bot System

The Anti-Bot system is designed to detect and mitigate automated bot traffic that could be harmful.

Features of Anti-Bot System:

Behavioral Analysis: Monitors user behavior to distinguish between humans and bots.
Challenge Mechanisms: Implements CAPTCHA or JavaScript challenges to verify users.
Machine Learning: Utilizes algorithms to adapt to new bot behaviors over time.

Cloudflare Anti-Bot System treats bot traffic as malicious traffic, even if the bot is not spamming the website, so just the detection of a client as being automated is enough to result in a 1015 error.

If you are trying to scrape a website with Cloudflare anti-bot protection, check out our Cloudflare bypass guide.

Mitigating error 1015

As already mentioned, 1015 error is either an indication of being rate-limited or being detected as automated traffic (bot). Each cause has its own means of mitigation.

Rate Limiting

Rate limiting controls the number of requests a user can make to a server within a set time frame. Understanding how rate limiting works can help you adjust your behavior to avoid triggering it.

Common Axes Used for Rate Limiting:

IP Address : Limits based on the user's internet protocol address.
User-Agent Header : Identifies the application, operating system, vendor, and version of the requesting user agent.
Cookies and Session Identifiers : Tracks user sessions to enforce limits.
JavaScript Fingerprinting : Uses browser characteristics (like screen resolution, installed plugins) to create a unique user profile. You can learn more about JS fingerprinting through our CreepJS fingerprinting guide

To avoid getting rate-limited, you can either reduce request frequency by spacing out your requests to the server or use proxies to distribute the workload over multiple IP addresses. We explain proxies in depth in our Complete Guide To Using Proxies For Web Scraping

Bot Detection

Most websites have robust bot protection measures in place to prevent web scraping and automated traffic. When bot traffic is detected, websites usually throw a 403 error or 429 error, and in Cloudflare's case, the HTML shows a 1015 "You are being rate limited" error.

In this case, the bot would be rate-limited but with no plan from the server to remove that limit, meaning it is permanently blocked.

Bot detection is usually achieved using complex fingerprinting techniques that find loopholes in requests to leak their automated nature. There are many tools that block those leaks and help bypass fingerprint detection:

curl_cffi: a Python HTTP client library built on top of curl-impersonate. It modifies the request configuration to mimic normal browsers, which helps bypass bot detection. You can learn more about curl_cffi in our curl impersonate guide.
undetected-chromedriver: a modified selenium webdriver that has built-in measures to combat websites that block automated headless browsers scraping their pages. Get to know more about it in our in-depth guide on web scraping with undetected chromedriver.

Error 1015 Popular Examples

Many popular websites use Cloudflare's services under the hood, giving them the ability to rate-limit users when suspicious traffic is detected. Let's look at some common scenarios where you can get 1015 error on popular websites:

Discord

Scenario: Rapidly sending messages or joining/leaving servers.
Solution: Slow down activities and avoid using bots that automate actions.

Crunchyroll

Scenario: Refreshing pages excessively or using downloaders to save content.
Solution: Use the service as intended and consider subscribing for offline access.

ChatGPT

Scenario: Making too many API requests in a short time.
Solution: Implement rate-limiting in your application and monitor your usage.

Red Robin

Scenario: Placing multiple orders rapidly or refreshing the menu pages.
Solution: Complete one order before starting another and minimize page refreshes.

USTravelDocs

Scenario: Constantly checking for visa appointment availability.
Solution: Limit checks to a reasonable frequency and avoid using automation tools.

Chess.com

Scenario: Rapidly starting and ending games or refreshing leaderboards.
Solution: Engage in games at a normal pace and avoid unnecessary page reloads.

Power Up with Scrapfly

Bypassing Cloudflare's Error 1015 while possible is very difficult - let Scrapfly do it for you!

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale. Each product is equipped with an automatic bypass for any anti-bot system and we achieve this by:

Maintaining a fleet of real, reinforced web browsers with real fingerprint profiles.
Millions of self-healing proxies of the highest possible trust score.
Constantly evolving and adapting to new anti-bot systems.
We've been doing this publicly since 2020 with the best bypass on the market!

It takes Scrapfly several full-time engineers to maintain this system, so you don't have to!

FAQ

Wrapping up, here are some common questions about Cloudflare's 1015 error.

How long does Error 1015 last?

The duration can vary based on the website's settings. It can range from a few minutes to several hours. It's best to wait and try again later.

Why am I getting Error 1015 when I haven't done anything unusual?

Factors like shared IP addresses, background applications making requests, or malware can cause unexpected rate limits. Check your system for any unwanted activity.

Can Error 1015 mean I am permanently blocked?

While Cloudflare Error 1015 typically indicates a temporary rate limit due to excessive requests within a short period, under certain circumstances, it can signify a more prolonged or even permanent block due to repeated violations or suspicious activity.

Summary

Cloudflare Error 1015 serves as a protective measure for websites against excessive or malicious requests. Understanding the reasons behind this error helps users adjust their online activities to prevent future occurrences.

Whether you are being rate limited for frequently requesting a website or you are permanently blocked due to bot detection, Scrapfly's powerful proxy pools and anti-scraping protection measures can help you bypass this error.

What HTTP Error 412 Precondition Failed and How to Fix it?

Scrapfly — Fri, 15 Nov 2024 05:40:04 +0000

Encountering an HTTP 412 error can be perplexing, especially when you're unsure why it happened. This article sheds light on what the HTTP 412 error means, explores its common causes, and discusses whether it can indicate blocking.

What is HTTP 412 Error?

The HTTP 412 error, known as "Precondition Failed," is a client error response status code. It signifies that the server cannot fulfill one or more conditions specified in the request's headers.

Conditions are typically specified in the request's headers using conditional headers where the header name typically follows the format If-X.

Common conditional headers include:

If-Match
If-None-Match
If-Modified-Since
If-Unmodified-Since

Let's take a look at each of these http 412 error causes next in greater detail.

What are HTTP 412 Error causes?

The HTTP 412 "Precondition Failed" error arises when the conditions specified in the client's conditional headers are not met by the server. So, what are these conditions?

Understanding Conditional Headers

Conditional headers help manage resource versions and ensure data integrity during client-server interactions. These headers are often used for caching and concurrency control — like multiple clients working on the same resource.

Understanding these headers is essential to grasp why a 412 error might occur. Here's the breakdown of each:

If-Match

The If-Match header makes the request conditional on the client's provided ETag matching the server's current ETag for the resource. An ETag (Entity Tag) is a unique identifier representing a specific version of a resource, like version of HTML page or a file.

How It Works : When a client includes an If-Match header with an ETag value, the server compares this ETag with the current ETag of the requested resource. If they match, the server processes the request. If not, it responds with a 412 error.
Use Case : This is commonly used when updating a resource (PUT or PATCH requests) to ensure that the resource hasn't changed since the client last retrieved it.

If-None-Match

The If-None-Match header is essentially the opposite of If-Match. It makes the request conditional on the resource's ETag not matching any of the ETags provided in the header.

How It Works : The server checks if the resource's current ETag matches any of the ETags listed. If there's a match, and the request method is GET or HEAD, the server responds with a 304 Not Modified. For other methods, it returns a 412 error.
Use Case : This header is used to fetch a resource only if it has changed, optimizing network usage by reducing unnecessary data transfer.

If-Modified-Since

The If-Modified-Since header makes the request conditional on the resource having been modified after the specified date.

How It Works : If the resource hasn't been modified since the given date, the server responds with a 304 Not Modified. If it has been modified, the server processes the request.
Use Case : Useful for caching mechanisms, where the client can avoid downloading unchanged resources.

If-Unmodified-Since

This header is used to ensure that the resource hasn't changed since a specific date.

How It Works : If the resource has been modified after the date provided, the server returns a 412 error. If not, it processes the request.
Use Case : Often used when deleting or updating resources to prevent conflicts due to concurrent modifications.

Common Causes of 412 HTTP code

ETag or Modification Date Mismatch

A primary cause is when the ETag or the modification date provided by the client doesn't match the server's current value. This mismatch indicates that the resource has changed since the client last accessed it.

Scenario : A client attempts to update a resource using an If-Match header with an outdated ETag. Since the resource has been modified by another client or process, the ETags don't match, resulting in a 412 error.

Concurrent Resource Modifications

In environments where multiple clients can modify resources, a 412 error helps prevent conflicting updates.

Scenario : Two clients attempt to update the same resource simultaneously. The first update succeeds, altering the resource's ETag. When the second client sends its update with the old ETag in the If-Match header, the server rejects it with a 412 error.

Avoiding HTTP 412 Errors

To prevent encountering a 412 error:

Ensure Synchronization : Keep client-side caches and system clocks synchronized with the server.
Handle ETags Properly : Always update stored ETags after a successful resource retrieval or modification.
Implement Conditional Headers Correctly : Understand the semantics of each header and use them appropriately in your requests.
Manage Concurrent Access : In applications where multiple clients access the same resources, implement proper version control or locking mechanisms.

By carefully managing conditional requests and maintaining up-to-date information about server resources, clients can minimize the chances of triggering a 412 Precondition Failed error.

412 HTTP Code in Web Scraping?

HTTP Status 412 is usually encountered when using condtional headers with POST, PUT, UPDATE requests. For example, when updating a resource concurrently with another user or a resource that is cached on the client and not updated. Such scenarios are very rare in web scraping.

If you encounter it when make a GET request or any request where conditional header are not used, it could mean that the error is returned delibirately by the server to block web scraping and deceive the scraper in thinking there's a technical issue.

Power Up with Scrapfly

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

It takes Scrapfly several full-time engineers to maintain this system, so you don't have to!

Summary

The HTTP 412 error indicates that the server couldn't meet the conditions specified in your request headers. Understanding this error involves:

Recognizing the role of conditional headers.
Identifying mismatches between client requests and server resources.
Acknowledging that while it can rarely resemble blocking, it's not its primary purpose.

By grasping the nuances of the HTTP 412 error, you can better troubleshoot and ensure smoother interactions between your client and the server.

How to Use Yelp API to Extract Business and Review Data

Scrapfly — Thu, 07 Nov 2024 15:48:07 +0000

Considered one of the most popular domains for business directories in the US, Yelp contains valuable company details, including addresses, emails, phone numbers, and reviews. But what's the most efficient way to extract these data?

In this guide, we'll take an extensive look into Yelp API, its key features, pricing, and limitations. Furthermore, we'll discuss potential alternatives. Let's dig in!

Legal Disclaimer and Precautions

This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect and here's a good summary of what not to do:

Do not scrape at rates that could damage the website.
Do not scrape data that's not available publicly.
Do not store PII of EU citizens who are protected by GDPR.
Do not repurpose the entire public datasets which can be illegal in some countries.

Scrapfly does not offer legal advice but these are good general rules to follow in web scraping

and for more you should consult a lawyer.

Yelp API Overview

The Yelp API is a product provided by Yelp for developers to automate certain actions or extract specific business data. Yelp APIs come with different endpoints, covering a wide range of workflows and resources.

In terms of data extraction, the Yelp API offers the following popular features:

Search for businesses based on terms, location, categories, or phone numbers.
Search for business service offerings.
Search for businesses with food delivery services.
Retrieve businesses' engagement metrics and review data.
Retrieve a business's details based on its ID or alias.

Note that the Yelp API capabilities aren't limited to the above features. Refer to the official Yelp API documentation for more.

Yelp Fusion API

Yelp includes thousands of business details and millions of equivalent reviews across different sectors and indentures. This makes the queries required to find specific businesses quite complex. Therefore, Yelp has introduced the Fusion API.

The Fusion API provides an easier business search experience to find the best matching results powered by an AI chat interface. It answers prompted queries and categorizes them into business questions or search results. Then, it retrieves the relevant results with related information, reviews, photos, and more.

For further details on Yelp Fusion API, refer to the official introductory tutorial.

Why Use Yelp API?

Yelp has information and review data for millions of businesses across various sectors and industries. Extracting this data using Yelp empowers different use cases, including:

Market Research

Navigating business services on Yelp allows business owners to evaluate their offerings based on the current market trend, which supports decision-making and helps businesses remain competitive.
Lead Generation

Considered one of the largest business directories in the US, Yelp API enables easy retrieval of contact information. Details like names, addresses, phone numbers, and emails make building outreach and marketing campaigns easier.
Sentiment Analysis

Using third-party tools and software has made it much easier to utilize LLMs for RAG applications and sentiment analysis models. Hence, extracting data from Yelp reviews is an excellent way to train these language models for context-aware applications.

For further details, have a look at our introduction to web scraping use cases.

How to Use Yelp API?

In the following sections, we'll explore using the Yelp API for data extraction. We'll cover each related resource endpoint and the core parameters.

Search for Businesses

One of the most popular Yelp API endpoints is used to search for businesses. The generic search endpoint retrieves basic business data based on the provided search query. Below is the search API endpoint schema:

curl --request GET \
     --url 'https://api.yelp.com/v3/businesses/search?sort_by=best_match&limit=20' \
     --header 'Authorization:Yelp API key' \
     --header 'accept: application/json'

The business search endpoint accepts different URl parameters to refine and narrow down the retrieved results. Below are the most common query parameters:

Parameter	Type	Description
`term`	string	Search term to use
`sort_by`	string	Sorting algoirthm to use
`categories`	[]string	Categories to filter search results by
`location`	string	Geographic area to filter search results by
`latitude`	number	Latitude of the location to search from
`longitude`	number	Longitude of the location to search from
`price`	[]integer	Pricing levels to filter the search result with
`attributes`	[]string	Bussiness attributes to filter by
`limit`	integer	Number of results to retrieve
`offset`	integer	Pagination cursor to start from

Above, are the commonly used query parameters when searching for businesses. Below is an example of the business details retrieved.

{
  "businesses": [
    {
      "alias": "golden-boy-pizza-hamburg",
      "categories": [
        {
          "alias": "pizza",
          "title": "Pizza"
        },
        {
          "alias": "food",
          "title": "Food"
        }
      ],
      "coordinates": {
        "latitude": 41.7873382568359,
        "longitude": -123.051551818848
      },
      "display_phone": "(415) 982-9738",
      "distance": 4992.437696561071,
      "id": "QPOI0dYeAl3U8iPM_IYWnA",
      "image_url": "https://yelp-photos.yelpcorp.com/bphoto/b0mx7p6x9Z1ivb8yzaU3dg/o.jpg",
      "is_closed": true,
      "location": {
        "address1": "James",
        "address2": "Street",
        "address3": "68M",
        "city": "Los Angeles",
        "country": "US",
        "display_address": ["James", "Street", "68M", "Los Angeles, CA 22399"],
        "state": "CA",
        "zip_code": "22399"
      },
      "name": "Golden Boy Pizza",
      "phone": "+14159829738",
      "price": "$",
      "rating": 4,
      "review_count": 903,
      "transactions": ["restaurant_reservation"],
      "url": "https://www.yelp.com/biz/golden-boy-pizza-hamburg?adjust_creative=XsIsNkqpLmHqfJ51zfRn3A&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=XsIsNkqpLmHqfJ51zfRn3A",
      "business_hours": {
        "open": [
          {
            "is_overnight": false,
            "start": 15,
            "end": 130,
            "day": 0
          },
          {
            "is_overnight": false,
            "start": 630,
            "end": 1730,
            "day": 1
          },
          {
            "is_overnight": false,
            "start": 45,
            "end": 500,
            "day": 2
          }
        ],
        "hours_type": "REGULAR",
        "is_open_now": false
      }
    }
  ],
  "region": {
    "center": {
      "latitude": 37.76089938976322,
      "longitude": -122.43644714355469
    }
  },
  "total": 6800
}

Each request to the Yelp API for business search retrieves the matching businesses, with a maximum of 240 entities in each call. However, the results don't include business reviews.

Retrieve Business Data

The business details endpoint retrieves the detailed business content. It follows the below schema:

curl --request GET \
     --url https://api.yelp.com/v3/businesses/business_id_or_alias \
     --header 'Authorization: Your Yelp API key' \
     --header 'accept: application/json'

To retrieve a specific business details, this API endpoint accepts either the business ID or its alias as the business_id_or_alias path parameter.

As for the endpoint query parameters, they are limited to the below properties:

Parameter	Type	Description
`locale`	string	Locale code in the langauge and country code foramt
`device_platform`	string	The platform to use for the `mobile_link` property

Here's a sample output of details retrieved by the business API:

{
  "alias": "golden-boy-pizza-hamburg",
  "categories": [
    {
      "alias": "pizza",
      "title": "Pizza"
    },
    {
      "alias": "food",
      "title": "Food"
    }
  ],
  "coordinates": {
    "latitude": 41.7873382568359,
    "longitude": -123.051551818848
  },
  "display_phone": "(415) 982-9738",
  "distance": 4992.437696561071,
  "id": "QPOI0dYeAl3U8iPM_IYWnA",
  "image_url": "https://yelp-photos.yelpcorp.com/bphoto/b0mx7p6x9Z1ivb8yzaU3dg/o.jpg",
  "is_claimed": false,
  "is_closed": true,
  "date_opened": "",
  "date_closed": "",
  "location": {
    "address1": "James",
    "address2": "Street",
    "address3": "68M",
    "city": "Los Angeles",
    "country": "US",
    "display_address": [
      "James",
      "Street",
      "68M",
      "Los Angeles, CA 22399"
    ],
    "state": "CA",
    "zip_code": "22399"
  },
  "name": "Golden Boy Pizza",
  "phone": "+14159829738",
  "photos": [
    "https://s3-media2.fl.yelpcdn.com/bphoto/CPc91bGzKBe95aM5edjhhQ/o.jpg",
    "https://s3-media4.fl.yelpcdn.com/bphoto/FmXn6cYO1Mm03UNO5cbOqw/o.jpg",
    "https://s3-media4.fl.yelpcdn.com/bphoto/HZVDyYaghwPl2kVbvHuHjA/o.jpg"
  ],
  "photo_details": [
    {
      "photo_id": "CPc91bGzKBe95aM5edjhhQ",
      "url": "https://s3-media2.fl.yelpcdn.com/bphoto/CPc91bGzKBe95aM5edjhhQ/o.jpg",
      "caption": "Meat",
      "width": "710,",
      "height": "47,",
      "is_user_submitted": "false,",
      "user_id": "null,",
      "label": "food"
    },
    {
      "photo_id": "FmXn6cYO1Mm03UNO5cbOqw",
      "url": "https://s3-media4.fl.yelpcdn.com/bphoto/FmXn6cYO1Mm03UNO5cbOqw/o.jpg",
      "caption": "Dessert",
      "width": 585,
      "height": 78,
      "is_user_submitted": false,
      "user_id": "null,",
      "label": "food"
    },
    {
      "photo_id": "HZVDyYaghwPl2kVbvHuHjA",
      "url": "https://s3-media4.fl.yelpcdn.com/bphoto/HZVDyYaghwPl2kVbvHuHjA/o.jpg",
      "caption": "Dessert_2",
      "width": 710,
      "height": 53,
      "is_user_submitted": false,
      "user_id": null,
      "label": "food"
    }
  ],
  "photo_count": 50,
  "price": "$",
  "rating": 4,
  "review_count": 903,
  "hours": {
    "open": [
      {
        "is_overnight": false,
        "start": 15,
        "end": 130,
        "day": 0
      },
      {
        "is_overnight": false,
        "start": 630,
        "end": 1730,
        "day": 1
      },
      {
        "is_overnight": false,
        "start": 45,
        "end": 500,
        "day": 2
      }
    ],
    "hours_type": "REGULAR",
    "is_open_now": false
  },
  "special_hours": [
    {
      "date": "2019-02-07",
      "end": "2000",
      "is_closed": null,
      "is_overnight": false,
      "start": "1600"
    }
  ],
  "transactions": [
    "restaurant_reservation"
  ],
  "url": "https://www.yelp.com/biz/golden-boy-pizza-hamburg?adjust_creative=XsIsNkqpLmHqfJ51zfRn3A&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=XsIsNkqpLmHqfJ51zfRn3A",
  "attributes": {
    "business_temp_closed": 1657868400,
    "outdoor_seating": false,
    "liked_by_vegans": false,
    "liked_by_vegetarians": true,
    "hot_and_new": "2022-12-10"
  },
  "messaging": {
    "url": "https://www.yelp.com/raq/AA5cAADa-F9f5DPqZ-PADA?adjust_creative=5374ujususZtKiSNEg7uhg&utm_campaign=yelp_api_v3&utm_medium=api_v3_graphql&utm_source=5374upadasZtCvMLBg7uhg#popup%3Araq",
    "use_case_text": "Request a Quote",
    "response_rate": 1,
    "response_time": 791,
    "is_enabled": true
  },
  "yelp_menu_url": "https://www.yelp.com/menu/golden-boy-pizza-hamburg",
  "rapc": {
    "is_enabled": true,
    "is_eligible": true
  }
}

Despite having the full business details retrieved by this Yelp API endpoint, the review data aren't included.

Retrieve Service Offerings

We have explored the endpoints responsible for retrieving business details. However, the services provided are retrieved through a dedicated endpoint for service offerings.

Below is the service offerings API endpoint schema:

curl --request GET \
     --url https://api.yelp.com/v3/businesses/business_id_or_alias/service_offerings \
     --header 'Authorization: Your Yelp API key' \
     --header 'accept: application/json'

The above Yelp API endpoint requires a business_id_or_alias path parameter to identify the related business.

The related query parameters are used to define localization settings:

Parameter	Type	Description
`locale`	string	Locale code in the langauge and country code foramt

Here's an example of what the service offering results look like:

{
  "active": [
    "bathtub_shower_installation",
    "drain_repair",
    "emergency_services",
    "garbage_disposal_repair",
    "gas_line_services",
    "offers_electric_water_heater_installation"
  ],
  "eligible": [
    "backflow_services",
    "bathtub_shower_installation",
    "bathtub_shower_repair",
    "drain_installation",
    "drain_repair",
    "emergency_services"
  ]
}

Retrieve Business Reviews

The business review data is among the most frequently requested information on Yelp. For this, Yelp provides a dedicated review API endpoint with the below schema:

curl --request GET \
     --url 'https://api.yelp.com/v3/businesses/business_id_or_alias/reviews?limit=20&sort_by=yelp_sort' \
     --header 'Authorization: Your Yelp API key' \
     --header 'accept: application/json'

Similar to the previous business-related API endpoints, passing the business ID or its alias as a business_id_or_alias path parameter is required to identify the business entity for retrieving reviews.

The Yelp review API provides a few query parameters for locality settings and pagination:

Parameter	Type	Description
`locale`	string	Locale code in the langauge and country code foramt
`offset`	integer	Pagination cursor to start from
`limit`	integer	Number of results to retrieve
`sort_by`	string	Sorting algoirthm to use

The review snippet results include details about the review text, rating, and user details:

{
  "possible_languages": [
    "en"
  ],
  "reviews": [
    {
      "id": "xAG4O7l-t1ubbwVAlPnDKg",
      "url": "https://www.yelp.com/biz/la-palma-mexicatessen-san-francisco?hrid=hp8hAJ-AnlpqxCCu7kyCWA&adjust_creative=0sidDfoTIHle5vvHEBvF0w&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_reviews&utm_source=0sidDfoTIHle5vvHEBvF0w",
      "text": "Went back again to this place since the last time i visited the bay area 5 months ago, and nothing has changed. Still the sketchy Mission, Still the cashier...",
      "rating": 5,
      "time_created": "2016-08-29 00:41:13",
      "user": {
        "id": "W8UK02IDdRS2GL_66fuq6w",
        "profile_url": "https://www.yelp.com/user_details?userid=W8UK02IDdRS2GL_66fuq6w",
        "image_url": "https://s3-media3.fl.yelpcdn.com/photo/iwoAD12zkONZxJ94ChAaMg/o.jpg",
        "name": "Ella A."
      }
    },
    ....
  ],
  "total": 3
}

The above example response includes complete review data for each snippet. However, each review API call limits the number of reviews retrieved to only three.

Yelp API Evaluation

So far, we've explored the technical aspects of the Yelp API. However, a common question arises: is using the Yelp API suitable for extracting business and review data? For this, we must explore two crucial factors: pricing and limitations.

Pricing

Yelp offers different subscription tiers, each varying in pricing and the data fields that can be retrieved. Let's consider the minimum plans required to retrieve both basic business and review data:

Business data: Starter plan, with pricing of $7.99 /1,000 API calls
Review data: Plus plan, with pricing of $9.99 /1,000 API calls

To better explore this, let's evaluate the cost of extracting 1000 business and review data using Yelp API.

Data	Plan	Pricing per 1,000 calls	Max results per API call	Cost per 10,000 results
Business	Starter	$7.99	1	$79.9
Reviews	Plus	$9.99	3	$33.33

Above is a rough estimation of the Yelp API costs for extracting 10,000 business and review entities. However, the subscription plans considered for this cost estimation only cover very basic attributes. The full data attributes are only available under the enterprise plan, which comes at a much higher cost of $14.99 per 1,000 API calls.

For further details on Yelp API pricing, refer to the official documentation.

Limitations

So far, we have explored the available Yelp APIs for business and reviewed data extraction, including their specifications and result schema. However, we have identified the below limitations:

Subscription plans are expensive and allow for a limited number of API calls.
Full data attributes are only accessible through the enterprise plan, which is even more expensive.
Gathering complete business information requires requesting data from multiple API endpoints.

Yelp API Alternatives: Web Scraping

An alternative to using Yelp API for data extraction is using web scraping. This approach enables extracting data from Yelp's public web pages. Instead of requesting the API endpoints for direct JSON data retrieval, we can parse the HTML or replicate background API calls to extract what we are looking for!

For example, let's replicate and parse Yelp's public search pages as an alternative to the paid search API endpoint:

import json
import asyncio

from parsel import Selector
from typing import List, Dict
from urllib.parse import urlencode
from httpx import AsyncClient, Response

# initialize an async httpx client
client = AsyncClient(
    # enable http2
    http2=True,
    # add basic browser like headers to prevent getting blocked
    headers={
        "Accept-Language": "en-US,en;q=0.9",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, br",
        "Cookie": "intl_splash=false"
    },
    follow_redirects=True
)

def parse_search(response: Response) -> List[Dict]:
    """parse listing data from the search XHR data"""
    assert response.status_code == 200, "Request is blocked, use ScrapFly to bypass Yelp's blocking"
    search_data = []
    selector = Selector(text=response.text)
    script = selector.xpath("//script[@data-id='react-root-props']/text()").get()
    data = json.loads(script.split("react_root_props = ")[-1].rsplit(";", 1)[0])
    for item in data["legacyProps"]["searchAppProps"]["searchPageProps"]["mainContentComponentsListProps"]:
        # filter search data cards
        if "bizId" in item.keys():
            search_data.append(item)
        # filter the max results count
        elif "totalResults" in item["props"]:
            total_results = item["props"]["totalResults"]
    return {"search_data": search_data, "total_results": total_results}

async def scrape_search(keyword: str, location: str):
    """scrape single page of yelp search"""

    def make_search_url(offset):
        base_url = "https://www.yelp.com/search?"
        params = {"find_desc": keyword, "find_loc": location, "start": offset}
        return base_url + urlencode(params)
        # final url example:
        # https://www.yelp.com/search?find_desc=plumbers&find_loc=Seattle%2C+WA&start=1

    first_page = await client.get(make_search_url(1))
    data = parse_search(first_page)
    return data


async def run():
    search_data = await scrape_search(
        keyword="plumbers", location="Seattle, WA"
    )
    with open ("data.json", "w", encoding="utf-8") as f:
        f.write(json.dumps(search_data, indent=2, ensure_ascii=False))

if __name__ == " __main__":
    asyncio.run(run())

Here, we request Yelp search pages given a search term and location. Then, we extract all business results from the page hidden web data. Here are what the results look like:

{
  "total_results": 240,
  "search_data": [
    {
      "bizId": "_Wv9uLrzQ1dZ6fgMYjgygg",
      "searchResultBusiness": {
        "ranking": null,
        "isAd": true,
        "renderAdInfo": false,
        "name": "Rooter-Man",
        "alternateNames": [],
        "businessUrl": "/adredir?ad_business_id=_Wv9uLrzQ1dZ6fgMYjgygg&campaign_id=jpQhVJGCG8ILi71Et0XDdQ&click_origin=search_results&placement=vertical_0&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fbiz%2Frooter-man-orting%3Foverride_cta%3DGet%2Bpricing%2B%2526%2Bavailability&request_id=9d07f67caab0b9cc&signature=919646537897431750a9ec41b1240262d0e8596eed098a3d2776e9c9afe1898f&slot=0",
        "categories": [
          {
            "title": "Plumbing",
            "url": "/search?find_desc=Plumbing&find_loc=Seattle%2C+WA"
          }
        ],
        "priceRange": "",
        "rating": 0.0,
        "isClickableReview": false,
        "reviewCount": 0,
        "formattedAddress": "",
        "neighborhoods": [],
        "phone": "",
        "serviceArea": null,
        "parentBusiness": null,
        "servicePricing": null,
        "bizSiteUrl": "https://biz.yelp.com",
        "serviceOfferings": [],
        "businessAttributes": {
          "licenses": [
            {
              "license_number": "ROOTE**792MT",
              "license_expiration_date": "2025-08-12",
              "license_verification_url": "https://secure.lni.wa.gov/verify/Detail.aspx?UBI=602584774&LIC=ROOTE**792MT&SAW=",
              "license_verification_status": "verified",
              "license_verification_date": "2023-11-17",
              "license_issuing_authority": "WA DLI ",
              "license_type": "Journey Level",
              "license_source": "biz_owner",
              "licensee": null
            }
          ]
        },
        "alias": "rooter-man-orting",
        "website": {
          "href": "/adredir?ad_business_id=_Wv9uLrzQ1dZ6fgMYjgygg&campaign_id=jpQhVJGCG8ILi71Et0XDdQ&click_origin=search_results_visit_website&placement=vertical_0&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fbiz_redir%3Fcachebuster%3D1701335261%26s%3D853c7f42baedaddb12d3a47cbf0c7c30e7bb3cf5d0408740f1e1ee56ec69c2a7%26src_bizid%3D_Wv9uLrzQ1dZ6fgMYjgygg%26url%3Dhttp%253A%252F%252Fwww.rooterman.com%26website_link_type%3Dwebsite&request_id=9d07f67caab0b9cc&signature=8842df091a5b79be57b4bf6122644039b1f6c07fac0c2fdb235c8ff076ce2520&slot=0",
          "rel": "noopener nofollow"
        },
        "city": "Orting"
      },
      "scrollablePhotos": {
        "isScrollable": false,
        "photoList": [
          {
            "src": "https://s3-media0.fl.yelpcdn.com/bphoto/-31eN7ypNCIJCHRO0Xjf3g/ls.jpg",
            "srcset": "https://s3-media0.fl.yelpcdn.com/bphoto/-31eN7ypNCIJCHRO0Xjf3g/258s.jpg 1.03x,https://s3-media0.fl.yelpcdn.com/bphoto/-31eN7ypNCIJCHRO0Xjf3g/300s.jpg 1.20x,https://s3-media0.fl.yelpcdn.com/bphoto/-31eN7ypNCIJCHRO0Xjf3g/348s.jpg 1.39x"
          }
        ],
        "photoHref": "/adredir?ad_business_id=_Wv9uLrzQ1dZ6fgMYjgygg&campaign_id=jpQhVJGCG8ILi71Et0XDdQ&click_origin=search_results&placement=vertical_0&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fbiz%2Frooter-man-orting%3Foverride_cta%3DGet%2Bpricing%2B%2526%2Bavailability&request_id=9d07f67caab0b9cc&signature=919646537897431750a9ec41b1240262d0e8596eed098a3d2776e9c9afe1898f&slot=0",
        "allPhotosHref": "/biz_photos/_Wv9uLrzQ1dZ6fgMYjgygg",
        "isResponsive": false
      },
      "childrenBusinessInfo": null,
      "searchResultBusinessPortfolioProjects": null,
      "searchResultBusinessHighlights": {
        "bizSiteUrl": "https://biz.yelp.com/business_highlights?utm_source=disclaimer_www_searchresults",
        "businessHighlights": [
          {
            "bizPageIconName": "",
            "group": {},
            "bizPageIconV2Name": "40x40_locally_owned_v2",
            "iconName": "18x18_locally_owned",
            "id": "LOCALLY_OWNED_OPERATED",
            "title": "Locally owned &amp; operated"
          },
          {
            "bizPageIconName": "",
            "group": {},
            "bizPageIconV2Name": "40x40_family_owned_v2",
            "iconName": "18x18_family_owned",
            "id": "FAMILY_OWNED_OPERATED",
            "title": "Family-owned &amp; operated"
          },
          {
            "bizPageIconName": "",
            "group": {},
            "bizPageIconV2Name": "40x40_workmanship_guaranteed_v2",
            "iconName": "18x18_workmanship_guaranteed",
            "id": "WORKMANSHIP_GUARANTEED",
            "title": "Workmanship guaranteed"
          },
          {
            "bizPageIconName": "",
            "group": {},
            "bizPageIconV2Name": "40x40_years_in_business_v2",
            "iconName": "18x18_years_in_business",
            "id": "YEARS_IN_BUSINESS",
            "title": "20 years in business"
          },
          {
            "bizPageIconName": "",
            "group": {},
            "bizPageIconV2Name": "40x40_veteran_owned_v2",
            "iconName": "18x18_veteran_owned",
            "id": "VETERAN_OWNED_OPERATED",
            "title": "Veteran-owned &amp; operated"
          },
          {
            "bizPageIconName": "",
            "group": {},
            "bizPageIconV2Name": "40x40_free_estimates_v2",
            "iconName": "18x18_free_estimates",
            "id": "FREE_ESTIMATES",
            "title": "Free estimates"
          }
        ],
        "numGemsAllowed": 2
      },
      "tags": [],
      "serviceOfferings": [],
      "snippet": {
        "readMoreText": "more",
        "readMoreUrl": "/adredir?ad_business_id=_Wv9uLrzQ1dZ6fgMYjgygg&campaign_id=jpQhVJGCG8ILi71Et0XDdQ&click_origin=read_more&placement=vertical_0&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fbiz%2Frooter-man-orting%3Foverride_cta%3DGet%2Bpricing%2B%2526%2Bavailability&request_id=9d07f67caab0b9cc&signature=919646537897431750a9ec41b1240262d0e8596eed098a3d2776e9c9afe1898f&slot=0",
        "text": "Give us a call for a free consultatiom.",
        "thumbnail": {
          "src": "https://s3-media0.fl.yelpcdn.com/bphoto/A0e8SoYZthSqITMDTjB0sA/30s.jpg",
          "srcset": "https://s3-media0.fl.yelpcdn.com/bphoto/A0e8SoYZthSqITMDTjB0sA/ss.jpg 1.33x,https://s3-media0.fl.yelpcdn.com/bphoto/A0e8SoYZthSqITMDTjB0sA/60s.jpg 2.00x,https://s3-media0.fl.yelpcdn.com/bphoto/A0e8SoYZthSqITMDTjB0sA/90s.jpg 3.00x"
        },
        "id": "",
        "type": "specialty"
      },
      "searchActions": [],
      "markerKey": "ad_business:below_organic:U5LNtOZST6_9gpNAbqw8Lg",
      "searchResultLayoutType": "scrollablePhotos",
      "verifiedLicenseInfo": {
        "licenses": [
          {
            "licensee": null,
            "licenseNumber": "ROOTE**792MT",
            "issuedBy": "WA DLI ",
            "trade": "Journey Level",
            "verifiedDate": "2023-11-17",
            "expiryDate": "2025-08-12"
          }
        ],
        "bizSiteUrl": "https://biz.yelp.com/verified_license?utm_source=legal_disclaimer_www"
      },
      "verifiedLicenseLayout": "BadgeAndTextBelowBizName",
      "yelpGuaranteedInfo": {
        "yelp_guaranteed_status": false,
        "yg_info_modal_url": "https://www.yelp.com/yelp-guaranteed"
      },
      "adLoggingInfo": {
        "placement": "vertical_0",
        "slot": 0,
        "placementSlot": 1,
        "opportunityId": "9d07f67caab0b9cc",
        "adCampaignId": "jpQhVJGCG8ILi71Et0XDdQ",
        "flow": "search",
        "isShowcaseAd": false
      },
      "offerCampaignDetails": null
    },
    ....
  ]
}

From the sample output above, we can see that we have retrieved full business data from search pages directly in JSON. Some attributes returned are even only available for the Yelp API through the enterprise plan!

For further tips and tricks on Yelp web scraping, refer to our dedicated guide.

Scrape Yelp at Scale With ScrapFly

We have seen that web scraping is a much better alternative to using Yelp API. However, there's a catch: web scraping blocking.

Yelp is able to identify our web scraping requests as being automated and hence requiring us to solve CAPTCHA challenges or even blocking us entirely:

Yelp scraping blocking

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

Using ScrapFly to bypass Yelp scraping blocking is fairly straightforward. All we have to do is replace our HTTP client with ScrapFly client:

# standard web scraping code
import httpx
from parsel import Selector

response = httpx.get("some yelp.com URL")
selector = Selector(response.text)

# in ScrapFly becomes this 👇
from scrapfly import ScrapeConfig, ScrapflyClient

# replaces your HTTP client (httpx in this case)
scrapfly = ScrapflyClient(key="Your ScrapFly API key")

response = scrapfly.scrape(ScrapeConfig(
    url="website URL",
    asp=True, # enable the anti scraping protection to bypass blocking
    country="US", # set the proxy location to a specfic country
    render_js=True # enable rendering JavaScript (like headless browsers) to scrape dynamic content if needed
))

# use the built in Parsel selector
selector = response.selector
# access the HTML content
html = response.scrape_result['content']

Refer to our official official Yelp scraper on GitHub for ready-to-use data extraction scripts for various datasets.

FAQ

To wrap up this guide, let's look at some of the frequently asked questions about using Yelp API for data extraction.

Is Yelp API free?

No, Yelp API is provided under paid subscription tiers. Each tier supports specific data resources.

Are there are alternatives for Yelp API?

Yes, web scraping Yelp is a competitive alternative to Yelp API. This approach extracts Yelp data using HTLML parsing or by replicating background requests. Refer to our Yelp scraping guide for more.

How to get Yelp API key?

Yelp requires API tokens to authorize its endpoints. To get your Yelp API key, subscribe to any of the available plans.

Summary

In this guide, we went through an in-depth guide on Yelp API. We started by exploring the available API endpoints, their schemas, parameters, and outputs.

We have seen that using Yelp API for data extraction has limitations, including expensive subscription plans and limited data attributes supported.

As an alternative, we have explored ScrapFly web scraping API. It provides antibot bypass capabilities, enabling Yelp data extraction at scale.

What is Charles Proxy and How to Use it?

Scrapfly — Wed, 06 Nov 2024 16:53:47 +0000

Charles Proxy is a web debugging proxy that enables developers to monitor , inspect and modify HTTP/HTTPS traffic between their applications and remote servers.

By acting as a man-in-the-middle proxy, Charles Proxy provides deep visibility into network communications that is essential for debugging APIs, analyzing web application behavior, and optimizing performance.

This guide demonstrates how to effectively use Charles Proxy for intercepting web traffic, debugging network issues, and testing applications. We'll cover setup configuration, key features, and practical examples of using Charles's inspection capabilities in real development workflows.

What is Charles Proxy?

Charles Proxy functions as a web debugging proxy server that runs locally on your machine. It positions itself between your web browser/application and the internet, enabling developers to monitor, inspect and modify all HTTP/HTTPS network traffic.

When configured, your browser routes all requests through Charles before they reach their destination servers, allowing you to examine the complete request and response cycle.

The proxy server captures critical debugging data including HTTP headers, cookies, SSL certificates, and request/response bodies. This visibility into the network layer helps developers troubleshoot API integrations, analyze web application behavior, and optimize network performance.

Charles Proxy can also capture Android and iOS app traffic though in this tutorial we'll focus on web traffic interception.

Man In the Middle Proxy Uses

Charles Proxy's man-in-the-middle capabilities enable several critical debugging and development workflows:

Network Traffic Analysis

By intercepting requests between clients and servers, developers can inspect the exact data being transmitted, including headers, parameters, and payloads. This visibility helps diagnose API integration issues, validate request/response formats, and verify data handling.

Security Testing

Security engineers use Charles to analyze application security by examining encryption implementations, testing for vulnerabilities, and validating certificate handling. The ability to modify requests allows testing of input validation and security controls.

Performance Optimization

The proxy enables measurement of request timing, payload sizes, and connection behavior. Developers can identify slow requests, unnecessary data transfers, and connection bottlenecks. The bandwidth throttling feature helps test application behavior under constrained network conditions.

Mobile App Development

Charles facilitates debugging of mobile applications by providing visibility into network calls made from iOS and Android devices. This helps developers troubleshoot API integrations, analyze caching behavior, and optimize mobile data usage.

API Development and Testing

When building APIs, Charles allows developers to inspect the exact requests clients make and responses servers return. This helps validate API implementations, debug client integration issues, and ensure proper data formatting.

Web Application Debugging

Front-end developers use Charles to debug AJAX requests, analyze REST API calls, and troubleshoot browser-server communications. The ability to modify requests helps test different scenarios without changing application code.

Setting up Charles Proxy

Getting started with Charles requires configuring both your system and the applications you want to debug. The setup process involves installing Charles, configuring your browser/device to route traffic through Charles, and setting up SSL certificates for HTTPS inspection.

First, download and install Charles from the official download page. By default, Charles runs on port 8888 and attempts to automatically configure your system's proxy settings. You can verify Charles is working by opening your browser - you should see web traffic appearing in Charles's main window.

As you can see, all the intercepted requests fall under the "Encrypted" category. This is because most of the websites we visit today use HTTPS, which encrypts both requests and responses to prevent spoofing or tampering.

However, Charles proxy allow us to install an SSL certificate that is able to decrypt HTTPS requests and make them inspectable.

SSL Certificates

To inspect HTTPS traffic, Charles generates a unique root certificate for your installation that must be trusted by your system and devices. This enables Charles to decrypt and inspect encrypted traffic by acting as a trusted certificate authority.

The certificate installation process on windows and macOS are very similar, so we are going to demonstrate it on macOS.

Help > SSL Proxying > Install Charles Root Certificate
Enter your device credentials to authorize the installation
Install the certificate under "login" keychains
Double tap the certificate > Expand the "Trust" tab > Set "When using this certificate" to "Always Trust"
Close the window and save the changes.

For Mobile Devices:

iOS: Visit chls.pro/ssl while connected through Charles
Android: Export the certificate from Charles and manually install it in device settings

The certificate is uniquely generated for your Charles installation to maintain security. Charles maintains end-to-end encryption by establishing separate SSL connections with both client and server. After installing the certificate:

You can enable SSL proxying on specific domains by right-clicking the domain name and click "Enable SSL Proxying"
Go to Proxy > SSL Proxying Settings and add entries to the list of domains you want to include in SSL Proxying.

Using Charles Proxy to Intercept Web Traffic

Charles Proxy provides comprehensive web debugging capabilities through its traffic interception features. The proxy server captures all HTTP/HTTPS requests and responses, displaying them in a structured interface that enables detailed inspection and analysis.

When traffic flows through Charles, each request appears in the main window with key details like the host, path, method, and status code.Selecting a request reveals the complete headers, parameters, and body content in the request/response panels.

This granular visibility helps developers identify issues in API calls, validate data formats, and troubleshoot network errors.

There two options to view the requests intercepted by Charles Proxy:

The Structure view organizes requests by domain and displays them in a tree format, making it easy to focus on specific hosts or endpoints.
The Sequence view shows requests chronologically with timing information to analyze request patterns and identify performance bottlenecks.

For modifying traffic, Charles provides breakpoints that pause requests before they reach the server or responses before they return to the client. This enables testing different scenarios by editing parameters, headers, or response data. Map Local and Rewrite rules offer persistent request/response modifications without manual intervention.

Charles also includes specialized tools for web debugging like:

Throttling bandwidth and adding latency to simulate poor network conditions
Validating response content against W3C standards
Comparing requests to identify differences
Recording and replaying captured traffic for reproducible testing

These capabilities make Charles an invaluable tool for debugging web applications, APIs, and network communications during development and testing.

Web Scraping with Charles Proxy

Charles Proxy streamlines web scraping development by providing detailed visibility into web requests. The proxy captures all request components including headers, cookies, and parameters that need to be replicated in scraping code. This visibility eliminates guesswork when reproducing browser requests programmatically.

The proxy's traffic interception reveals how websites implement anti-bot measures like dynamic tokens or session handling. Developers can analyze these protection mechanisms and implement appropriate countermeasures in their scraping logic.

Practical Example

We can demonstrate how Charles Proxy can be used for web scraping tasks by intercepting the requests sent by the testimonials page in web-scraping.dev.

The testimonials page initially includes 10 testimonials and loads more testimonials as the user scrolls to the bottom of the page.

We can use Charles Proxy to intercept the requests sent by the page to load more testimonials.

We can see that the website sends requests to https://web-scraping.dev/api/testimonials?page=[pageNumber] to load more testimonials on each scroll; and we can copy this request as a cURL command (by right-clicking on it) to replicate it with the same configuration sent by the parent website (headers, parameters, body...)

Scrapfly provides a useful tool that converts cURL commands into executable code using Python's most popular HTTP clients. You can use it to integrate requests intercepted by Charles Proxy into your python web crawler.

Power Up with Scrapfly

Charles Proxy is a great tool for debugging and analyzing web traffic though to take advantage of this information for web scraping or web automation tasks you'll still need to write code to replicate the requests and handle the responses. This is where Scrapfly can assist you!

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

Common Charles Proxy Issues

When troubleshooting Charles Proxy, these issues commonly arise:

SSL Certificate Problems

Browser warnings appear when Charles's root certificate isn't properly installed/trusted
Chrome on localhost requires enabling insecure localhost in chrome://flags
Mobile devices need manual certificate installation through chls.pro/ssl
Certificate pinning in some apps prevents SSL inspection

Network Configuration Issues

Applications fail to connect when proxy settings are incorrect (verify 127.0.0.1:8888)
Forgetting to disable proxy settings after debugging causes connection failures
iOS devices experience network problems if proxy config isn't removed post-debugging
External proxy conflicts when Charles is configured with another proxy server

Performance and Memory

SSL proxying on many hosts causes slowdown - limit wildcards (*) usage
Large response bodies impact memory usage - clear session periodically
High traffic applications may require increased memory allocation
Throttling feature can unexpectedly slow all network traffic

Application-Specific Challenges

Android apps require custom network security config for certificate trust
Some applications use certificate pinning requiring code modifications
WebSocket connections may fail without proper SSL proxying configuration
HTTP/2 traffic sometimes requires specific SSL proxying settings

Summary

Charles Proxy is a versatile tool that empowers developers with the ability to intercept, inspect, and manipulate network traffic. Its robust features make it ideal for debugging web applications, testing APIs, and supporting web scraping efforts. Key capabilities include:

Traffic Inspection : View detailed request/response data to troubleshoot client-server interactions.
SSL Support : Intercept secure HTTPS traffic with certificate handling.
Request Mapping & Modification : Offers tools to map requests to different endpoints or local files, enabling testing under diverse scenarios without altering the actual server.
Throttling & Breakpoints : Simulate network conditions and pause requests for precise debugging.

With careful configuration, Charles Proxy is a powerful addition to any web developer’s toolkit.

Axios vs Fetch: Which HTTP Client to Choose in JS?

Scrapfly — Tue, 29 Oct 2024 06:57:49 +0000

Making HTTP requests is a fundamental feature in any programming language, and JavaScript is no exception. As the development community has moved away from using XMLHttpRequest, making HTTP requests in JavaScript typically involves either the built-in Fetch API or the axios package.

In this article, we’ll explore Fetch vs Axios — compare their key differences, pros, cons, and help you determine which option is best for your project.

What is Fetch?

Fetch is a built-in browser API that allows you to make HTTP requests from the client side, providing a more modern and flexible alternative to the older XMLHttpRequest. It is known for its promise-based syntax, making it easy to handle asynchronous operations. While Fetch was initially designed for the browser environment, it has also been officially supported in Node.js v18. This means developers can now use Fetch in both the browser and server-side environments, making it a versatile tool for modern JavaScript development.

ℹ️ Fetch is also supported by other JS runtimes like Bun and Deno as well as serverless runtimes like Cloudflare Workers and Edge Runtime

Example of a basic Fetch request using .then() syntax and async/await syntax:

fetch('https://httpbin.dev/json')
  .then(response => response.json())
  .then(data => console.log(data))
  .catch(error => console.error('Error:', error));

async function main() {
  try {
    const response = await fetch("https://httpbin.dev/json");
    const data = await response.json();
    console.log(data);
  } catch (error) {
    console.error("Error:", error);
  }
}

main();

The Fetch API’s integration in Node.js allows developers to work with the same API on both client and server sides, simplifying the codebase and making Fetch a powerful tool across the entire stack that is now considered a default option for handling HTTP requests in many JavaScript applications.

What is Axios?

Axios is a popular third-party HTTP client library that simplifies making HTTP requests in JavaScript. Built on top of JavaScript’s XMLHttpRequest interface in the browser and Node.js’s http module on the server, Axios provides a cleaner and more intuitive syntax for making requests and handling responses.

Example of a basic Axios request using .then() syntax and async/await syntax:

const axios = require('axios');

axios.get('https://api.example.com/data')
  .then(response => console.log(response.data))
  .catch(error => console.error('Error:', error));

const axios = require("axios");

async function main() {
  try {
    const response = await axios.get("https://api.example.com/data");
    console.log(response.data);
  } catch (error) {
    console.error("Error:", error);
  }
}

main();

Fetch vs Axios - How are They Different?

Both Fetch and Axios are capable of handling HTTP requests, but they differ in several key areas.

Availability

Fetch is a built-in API available in the browser, Node.js, and most JS runtimes and serverless environments making it one less dependency for your project.In contrast, Axios is a third-party library, meaning it requires installation via a package manager or inclusion through a CDN.

While both axios and fetch are isomorphic, meaning that they can be used in both server and client-side environments, each supports a different yet overlapping set of browsers, node.js versions, and JS runtimes.

Feature	Axios	Fetch
Client-side Availability	Available via npm/CDN	Built into modern browsers
Browser Compatibility	Fully Compatible (including IE11)	All Modern browsers (No IE11 support)

Polyfill Requirements	Requires ES6 promises polyfill for non ES6 environments	Requires polyfill for IE11 and older browsers
Server Runtime Support
Node.js	✅	- Native support since Node.js 18 (May 2022) - Stable since Node.js 21 (Nov 2023) - Earlier versions require `node-fetch`
Deno	✅	✅
Bun	✅	✅
Serverless Support
AWS Lambda	✅	✅
Azure Functions	✅	✅
Google Cloud Functions	✅	✅
Cloudflare Workers	🚫	✅
Netlify Functions	✅	✅
Netlify Edge Functions	✅	✅
Vercel Serverless Funcitons (Node.js)	✅	✅
Vercel Edge Functions (Edge Runtime)	🚫	✅
Deno Deploy	✅	✅

Features

While fetch Provides basic functionality for making HTTP requests, it lacks many built-in advanced features that axios supports by default. Let's explore some of those features.

1. Interceptors
Interceptors in Axios allow you to modify requests and responses before they are processed. They are particularly useful for tasks like adding authentication tokens, logging requests, or handling global error responses.

Using interceptors you can centralize logic that applies to all requests or responses, enhancing maintainability and flexibility in your application.

// Add a request interceptor
axios.interceptors.request.use(function (config) {
    // Do something before request is sent
    return config;
  }, function (error) {
    // Do something with request error
    return Promise.reject(error);
  });

// Add a response interceptor
axios.interceptors.response.use(function (response) {
    // Any status code that lie within the range of 2xx cause this function to trigger
    // Do something with response data
    return response;
  }, function (error) {
    // Any status codes that falls outside the range of 2xx cause this function to trigger
    // Do something with response error
    return Promise.reject(error);
  });

2. Timeouts
Timeouts in Axios provide a way to control how long the client should wait for a server response before aborting the request. This is essential when dealing with unreliable networks or APIs that may take too long to respond.

With Axios, you can easily configure request timeouts to ensure your app doesn't hang indefinitely waiting for a response.

// Setting a timeout of 5 seconds (5000 milliseconds)
axios.get('https://httpbin.dev/json', {
  timeout: 5000
})
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    if (error.code === 'ECONNABORTED') {
      console.error('Request timed out');
    } else {
      console.error('Error:', error.message);
    }
  });

3. Query parameters serialization
Query parameters allow you to pass data in the URL of a GET request. They are typically used to pass additional configuration options for an endpoint, such as filters and pagination.

Axios makes it simple to serialize query parameters into the proper format, ensuring they are correctly appended to the URL. It also supports serlializing nested objects without reaching out to a third party library like qs.

const params = {
  page: 2,
  order: 'desc',
  category: 'apparel'
};

axios.get('https://web-scraping.dev/api/products', { params })
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.error('Error:', error.message);
  });

4. Automatic request body serialization
Request body serialization depends on the type of content sent in the body. Whether it's JSON, URL-encoded data, or multipart form data, each type has its very own serialization technique.

Axios automatically converts the request body to the appropriate format based on the content type, eliminating the hassle of manually handling it using built-in language features like JSON.stringify or third-party libraries.

const data = {
  name: 'John Doe',
  age: 30,
  occupation: 'Developer'
};

// Default (application/json)
axios.post('https://httpbin.dev/post', data)
  .then(response => console.log(response.data))
  .catch(error => console.error('Error:', error.message));

// 'application/x-www-form-urlencoded'
axios.post('https://httpbin.dev/post', data, {
  headers: {
    'Content-Type': 'application/x-www-form-urlencoded'
  }
}).then(response => console.log(response.data));

//'multipart/form-data'
const formData = {
  ...data,
  profile_picture: document.querySelector('#fileInput').files[0] 
}
axios.post('https://httpbin.dev/post', formData, {
  headers: {
    'Content-Type': 'multipart/form-data'
  }
}).then(response => console.log(response.data));

5. Automatic Response Parsing
Automatic response parsing refers to Axios's ability to automatically parse the server's response into the appropriate data format.

If the response is in a valid JSON object, Axios will parse it into a JavaScript object without requiring you to manually call .json(), making it easier to handle responses with minimal effort.

axios.get('https://httpbin.dev/json')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.error('Error:', error.message);
  });

// compared to fetch
fetch('https://httpbin.dev/json')
  .then(response => response.json()) // a bit more verbose compared to axios
  .then(data => {
    console.log(data)
  })
  .catch(error => {
    console.error('Error:', error.message);
  });

6. Progress capturing
Tracking the progress of data uploads and downloads in real-time is a very common task encountered by developers. Unfortunately, fetch doesn't support progress tracking and implementing a solution around will lead you into a bunch of browser compatibility issues.

Using onUploadProgress and onDownloadProgress callbacks in Axios, you can easily monitor how much data has been transferred to and from the server, which is especially useful when dealing with large files or slow network conditions.

axios.post('https://httpbin.dev/post', formData, {
  headers: {
    'Content-Type': 'multipart/form-data'
  },
  onUploadProgress: (progressEvent) => {
    const percentCompleted = Math.round((progressEvent.loaded * 100) / progressEvent.total);
    console.log(`Upload Progress: ${percentCompleted}%`);
  }
}).then(response => console.log(response.data));


axios.get('https://api.example.com/content.txt'
  onDownloadProgress: (progressEvent) => {
    const percentCompleted = Math.round((progressEvent.loaded * 100) / progressEvent.total);
    console.log(`Download Progress: ${percentCompleted}%`);
  }
}).then(response => console.log(response.data));

Although axios has built-in support for all those features, they can all be easily implemented in fetch with some boilerplate code.

Developer Experience

When comparing Fetch and Axios from a user experience perspective, Axios offers a more streamlined and feature-rich approach providing:

automatic JSON parsing
built-in error handling
and more features like interceptors and request cancellation. This reduces boilerplate code and makes it easier to handle complex HTTP requests.

In contrast, Fetch is more minimalistic but more readily available and requires manual handling for tasks like transforming data and checking for non-2xx status codes, leading to more verbose code.

While Fetch offers flexibility and is natively supported in browsers, Axios enhances productivity by offering a cleaner, more user-friendly API, particularly for complex requests.

Axios’s ability to handle global configuration, error management, and response transformations makes it a more developer-friendly choice for larger applications, while Fetch might be more suited for simpler, straightforward tasks.

Performance

In terms of performance, there is generally no significant difference between Fetch and Axios for most typical HTTP requests. Both tools handle HTTP requests efficiently, and any performance differences are usually negligible in real-world applications.

Key Points to Consider:

Fetch is a native browser API, which means it’s lightweight and directly integrated into the browser’s environment. This can make Fetch slightly faster in some cases due to the lack of external dependencies, especially for simple requests.
Axios , being a third-party library, introduces a tiny amount of overhead because it has additional features built on top of the basic request-handling mechanisms. However, this overhead is minimal and usually doesn't impact performance significantly, especially when the added convenience and functionality of Axios are considered.

For basic use cases, Fetch might have a slight performance advantage due to its lightweight nature, but for more complex tasks, Axios provides a smoother experience with minimal performance impact.

In most cases, the performance difference is so small that the choice between Fetch and Axios should be based on functionality and developer experience rather than speed alone.

Axios Alternatives

While Axios remains one of the most popular HTTP libraries, several modern alternatives provide similar functionality, often with added features or optimizations. Some modern alternatives include:

Superagent: A flexible and powerful HTTP request library that provides a simple API for making requests in both Node.js and browsers. It supports promises, streams, and file uploads.
Got: A lightweight and feature-rich HTTP request library for Node.js. It’s known for its powerful error handling, high-performance, and promise-based API. It also offers advanced features like retry mechanisms, hooks, and custom agents.
Ky: By the maker of Got. A smaller, modern wrapper around Fetch designed for browsers. It’s promise-based, tiny in size, and offers features like automatic JSON parsing, retry logic, and simpler syntax than vanilla Fetch.

Each of these alternatives provides different features, and the best choice depends on your project’s specific requirements.

Power Up with Scrapfly

While both Fetch and Axios are excellent HTTP clients they are not ideal for tasks like web automation and web scraping.

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

FAQ

Before we wrap this article up let's take a look at some frequently asked questions regarding fetch and axios comparisson that we haven't covered in this article:

Is axios better than fetch?

Whether Axios is better than Fetch depends on your needs. Axios offers a more feature-rich, user-friendly experience making it a powerful choice for handling complex HTTP requests. However, Fetch is a native browser API, which makes it lightweight and ideal for simple requests. It’s flexible, built into modern browsers, and requires no additional dependencies, making it more suitable for smaller projects or when simplicity is preferred.

What is the difference between node-fetch and fetch?

Before Fetch was natively supported in Node.js as of version 18, node-fetch was required for Fetch-like functionality on the server. node-fetch is a lightweight module that brings the Fetch API to Node.js environments, allowing server-side JavaScript to make HTTP requests in a similar way.

Can Fetch and Axios be used in web scraping?

Just like any http client, Fetch and Axios can both be used to request webpages or REST APIs for the purpose of scraping data. Check out our dedicated article on web scarping with Node.js and Javascript.

Summary

In summary, both Fetch and Axios are powerful tools for making HTTP requests in JavaScript, each with unique advantages:

Fetch

Native to browsers, Node.js (from v18 onwards), and mainstream JS runtimes and serverless environments
Lightweight and ideal for simpler requests with minimal configuration.
Requires more manual handling for JSON parsing, error handling, and advanced features.

Axios

Third-party library with more features for handling complex requests.
Provides automatic JSON parsing, built-in error handling, request cancellation, and interceptors.
More user-friendly and efficient for large-scale applications needing advanced configurations.

Ultimately, the choice between Fetch and Axios should be based on the specific needs of your project, balancing simplicity with functionality.

What is HTTP 415 Error? (Unsupported Media Type)

Scrapfly — Thu, 24 Oct 2024 15:24:53 +0000

Encountering an HTTP error can disrupt your web scraping or automation tasks, and HTTP error 415 is one such issue that indicates a problem with the type of data being sent.

In this article, we’ll explore what HTTP 415 is, the common causes behind it, how to replicate it, and whether it could be used as a blocking mechanism.

What is HTTP Error 415?

HTTP error 415 Unsupported Media Type, occurs when the server refuses to process a request because the format or media type of the data being sent is not supported. For example, if you try to send JSON data to an endpoint that only accepts XML, you would encounter a 415 error.

What are HTTP 415 Error Causes?

The most common cause of a 415 error is sending data in an unsupported format. This can happen when the Content-Type header in the request does not match the media type that the server expects.

For instance, if you're sending a POST request with application/xml but the server expects application/json, you’ll trigger a 415 error.

Practical Example

Let's explore how to configure headers, specifically Content-Type headers, in common tools like python's httpx library, and cURL.

import httpx

url = "https://httpbin.dev/json"
headers = {
    "Content-Type": "application/json",
}

response = httpx.post(url, headers=headers)
print(response.status_code)
print(response.text)

curl -X "POST" -H "Content-Type: application/json" https://httpbin.dev/json

In the examples above, the client is specifying that the body content is in application/json format. If the server expects a different media-type than the one that the client specified, a 415 error might occur.

To avoid 415 errors, ensure that your Content-Type header is set appropriately for the endpoint your are sending data to.

415 in Web Scraping

Http status 415 in web scraping is usually encountered when scraping POST or PUT type endpoints like search queries or form submissions. For these cases it's important to set the correct Content-Type header that not only matches the sent content type but the type server expects. To verify what content type the server expects, you can use Browser Developer Tools and inspect browser requests.

Another posibility is that the server is blocking your scraper requests and returns status code 415 purposefully to block your scraper. This is quite rare but here are some indicators that http code 415 is a block:

415 is returned on GET or HEAD requests
415 error cannot be replicated for the same identical requests

If you suspect that you are being blocked take a look at our guide on web scraping blocking or try Scrapfly Web Scraping API.

Power Up with Scrapfly

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

It takes Scrapfly several full-time engineers to maintain this system, so you don't have to!

Summary

HTTP 415 errors occur when the data format or media type is not supported by the server. While this error usually results from incorrect content types, it’s important to consider the possibility of blocking. With Scrapfly’s advanced scraping tools and IP rotation, you can bypass such blocks and continue scraping without interruptions.

What is HTTP 422 Error? (Unprocessable Entity)

Scrapfly — Thu, 24 Oct 2024 07:51:07 +0000

HTTP error 422 Unprocessable Entity occurs when the server understands the request but finds the content syntactically correct yet semantically invalid. Essentially, the data you’re submitting may be well-formed, but something about it is incorrect or incomplete, making it impossible for the server to process.

What are HTTP 422 Error Causes?

The primary cause of a 422 error code is sending data that, while properly formatted, is invalid according to the server's expectations. This often happens with POST requests when submitting form data, JSON, or XML that contain formatting errors.

For example, submitting an invalid or even a well-formed JSON document that lacks required fields or contains invalid values could trigger a 422 error.

To avoid this error, it’s important to ensure that the content being sent matches the server’s requirements, such as validation rules, data types, or required fields.

Practical Example

To demonstrate how a server might return an HTTP 422 status code, let's build a simple Flask API with a /submit endpoint that accepts POST requests. This example mimics submitting data to an API and returns a 422 error when the submitted data does not meet the server's validation rules (e.g., invalid email format).

from flask import Flask, jsonify, request

app = Flask( __name__ )

# A simple validation function to check for a valid email format
def is_valid_email(email):
    return "@" in email and "." in email

@app.route("/submit", methods=["POST"])
def submit():
    data = request.json
    email = data.get("email")

    # Check if email is provided and valid
    if not email or not is_valid_email(email):
        # Unprocessable Entity: Invalid email format
        return jsonify({"error": "Invalid email format."}), 422

    # Otherwise, process the request
    return jsonify({"message": "Data submitted successfully."}), 201

if __name__ == " __main__":
    app.run(debug=True)

In the example above, we simulate a /submit endpoint that accepts POST requests containing JSON data. The server expects a valid email address in the request. If the email is missing or does not meet the simple validation check (containing "@" and "."), the server returns a 422 error, indicating the request is well-formed but semantically incorrect (i.e., invalid email). If the email is valid, the server processes the request and returns a success message.

We can test this server with a http client like python's httpx:

 import httpx

# Test successful submission with a valid email
response = httpx.post("http://127.0.0.1:5000/submit", json={"email": "valid@example.com"})
print(f"Successful Submission: {response.status_code}, {response.json()}")

# Test failed submission with an invalid email
response = httpx.post("http://127.0.0.1:5000/submit", json={"email": "invalid-email"})
print(f"Failed Submission: {response.status_code}, {response.json()}")

422 in Web Scraping

In web scraping 422 http code is usually encountered when an error is made in POST or PUT data generation. So, ensure that posted data is of valid format be it JSON, HTML or XML to avoid this error.

Furthermore, as scrapers don't know exactly how the server reads the received data, it can be difficult to debug the exact cause. For this, Browser Developer Tools can be used to inspect exactly how a website formats the data like symbol escaping, indentation etc all of which can play a part in data processing. Replicating the exact behavior will decrease chances of encountering http status 422 while scraping.

The 422 error could also mean that the server is blocking your requests and deliberately returning a 422 status code to signal that you are not allowed to access the resource. If you're receiving this status code on a GET request, then that could be a sign of blocking.

Power Up with Scrapfly

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

It takes Scrapfly several full-time engineers to maintain this system, so you don't have to!

Summary

HTTP 422 errors typically result from submitting well-formed but invalid data, often in POST requests. While it's unlikely that 422 errors are used to block scrapers, it’s always best to test with rotating proxies if the issue persists. Using Scrapfly’s advanced tools, you can bypass these potential blocks and ensure your tasks continue without disruption.

What is HTTP 409 Error? (Conflict)

Scrapfly — Wed, 23 Oct 2024 06:49:44 +0000

When automating web tasks or scraping data, HTTP errors can disrupt your workflow, and HTTP 409 is no exception. The 409 error signals a conflict with the request you're sending, often caused by improper configuration.

In this article, we’ll explain what HTTP 409 means, common causes, and whether it could indicate blocking. We’ll also explore how Scrapfly can help you bypass this error.

What is HTTP Error 409?

409 HTTP code Conflict occurs when the server detects a conflict with the current state of the resource. This often happens when you're attempting to modify data that doesn't align with the server’s expectations or the resource's current state. For example, attempting to update a resource that has been changed or deleted since your last request might trigger a 409 error.

What are HTTP 409 Error Causes?

The most common cause of a 409 error is a conflict between the request and the server’s current data. This error can arise from various scenarios, such as:

Concurrent Updates : Two requests attempting to modify the same resource simultaneously can cause a conflict.
Version Mismatch : If the server is expecting a specific version of a resource and the request tries to modify an outdated version, a 409 error may occur.
Resource State Conflicts : Attempting to delete a resource that is referenced by another active resource could trigger a conflict.

To avoid 409 errors, it's important to ensure your requests are correctly configured and aligned with the server’s current state.

Practical Example

To demonstarte how a server would return a HTTP 409 status code, let's build a simple Flask API with a /register endpoint that accepts POST requests to mimic registering a new user to a database.

from flask import Flask, jsonify, request

app = Flask( __name__ )

# Sample data to mimic existing resources
existing_users = ["john_doe", "jane_smith"]

@app.route("/register", methods=["POST"])
def register():
    username = request.json.get("username")
    if username in existing_users:
        # Conflict: Username already exists
        return jsonify({"error": "Username already exists."}), 409

    # Otherwise, proceed with registration
    existing_users.append(username)
    return jsonify({"message": "User registered successfully."}), 201

if __name__ == " __main__":
    app.run(debug=True)

In the example above, we use an in-memory list to simulate a database of existing users. The /register endpoint receives the username sent by the client in the request body and checks if it already exists in the existing_users list. If the username is already taken, the server returns a 409 error, indicating a conflict between the data provided by the client and the existing resources. If the username is available, it is added to the list of users.

We can test this server with a http client like python's httpx:

import httpx

# Test successful registration
response = httpx.post("http://127.0.0.1:5000/register", json={"username": "new_user"})
print(f"Successful Registration: {response.status_code}, {response.json()}")

# Test failed registration (conflict)
response = httpx.post("http://127.0.0.1:5000/register", json={"username": "john_doe"})
print(f"Failed Registration: {response.status_code}, {response.json()}")

409 in Web Scraping

HTTP status 409 in web scraping is usually encountered when scraping POST or PUT method requests that create objects or update resources. For example, scraping websites with persistent sessions can yield 409 errors if the session data is outdated or conflicts with the server’s current state.

The 409 error could also mean that the server is blocking your requests due to rate limiting or other restrictions and deliberitely returning a 409 status code to signal that you are not allowed to access the resource. If you're receiving this status code on GET type request then that could be a sign of blocking.

Power Up with Scrapfly

ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass - scrape web pages without blocking!
Rotating residential proxies - prevent IP address and geographic blocks.
JavaScript rendering - scrape dynamic web pages through cloud browsers.
Full browser automation - control browsers to scroll, input and click on objects.
Format conversion - scrape as HTML, JSON, Text, or Markdown.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

It takes Scrapfly several full-time engineers to maintain this system, so you don't have to!

Summary

HTTP 409 errors are typically caused by conflicts between the request and the server’s current state, often due to concurrent modifications or outdated resource versions. While blocking is an unlikely cause of 409 errors, it’s important to test with proxies to rule out intentional blocking. Scrapfly’s automated tools, including ASP and rotating proxies, can help you bypass these issues and keep your scraping tasks on track.

DEV Community: Scrapfly

Comprehensive Guide to OkHttp for Java and Kotlin

What is OkHttp?

Main OkHttp Features

Setting Up OkHttp

Basic Usage

GET Requests and Response Details

Handling Errors

Synchronous and Asynchronous Requests

Advanced Usage

POST, PUT, HEAD, PATCH Requests

POST

HEAD

PUT

PATCH

Adding Query Parameters

Adding Headers

Adding Cookies

Adding Timeout

Adding Retries

Disabling HTTPS

WebSocket

Common OkHttp Errors and Issues

Power Up With Scrapfly

FAQ

Can OkHttp be used for Crawling?

Can OkHttp handle large file uploads or downloads?

Is OkHttp thread-safe?

Summary

Instant Data Scraper Guide - Web Scraping with No Code

What is Instant Data Scraping?

Instant Scraping with Google Sheets

Connecting Google Sheets and Scrapfly with Make.com

Scrapfly Screenshot API Action

Instant Scraping Inputs

On-Demand Scraping

Scheduled Scraping

Summary

What is HTTP 499 Status Code and How to Fix it?

Understanding the 499 Status Code

Why a Non-standard Error?

Causes of the 499 Status Code

Impact on Web Scraping and Automation

Strategies to Mitigate 499 Errors

Retry Mechanisms with Exponential Backoff

Client-Side Timeout Settings

Stable Network Connections

Best Practices for HTTP Clients and Web Scrapers

Monitoring and Logging

Robust Error-Handling

Ethical Scraping Practices

Power Up with Scrapfly

Summary

How to Use cURL to Download Files?

Why Use Curl to Download Files?

Curl Basic File Download Options

Custom File Name on Download

Show Progress Bar / Download Silently

Retry for Unstable Connections

Handling Large File Downloads

Insecure Downloading

Verifying File Integrity

Handling Authentication

Curl Command Builder

Automating Curl Downloads with Crontab

What Is Crontab?

Steps to Automate Downloads Using Crontab

Bypassing Download Blocks

Power Up File Downloads with Scrapfly

FAQ

Can I resume an interrupted download with curl?

Is wget a better alternative to curl for downloading files?

How do I download multiple files at once using curl?

Summary

What is Error 1015 (Cloudflare) and How to Fix it?

What is Error 1015?

Where is Error 1015 Encountered?

Cloudflare Web Application Firewall (WAF)

Cloudflare Anti-Bot System

Mitigating error 1015

Can I resume an interrupted download with `curl`?

How do I download multiple files at once using `curl`?