Connection errors are a common occurrence in HTTP requests. Especially in web scraping where servers may impose strict rate limits or block repeated requests. To handle this — retrying is often an effective strategy.
In this guide, we’ll explore how to retry in Axios, a widely used HTTP client for JavaScript (NodeJS, Deno and Bun). Why retries are essential for error handling and best practices for retry logic. How to analyze status codes and implement exponential backoff and overview common troubleshooting scenarios.
What is Axios?
Axios is a popular, promise-based HTTP client for JavaScript and all of it's flavors (NodeJS, Bun, Deno). It simplifies the process of sending HTTP requests through convenient developer expererience features and for this reason it is widely used in both browser and server environments.
Key Features of Axios
Axios is a powerful and versatile HTTP client library that simplifies making API requests in JavaScript applications. Below are some of its standout features:
- Promise-based design: Works seamlessly with
async/awaitfor cleaner, more readable code. - Automatic JSON handling: Automatically parses JSON responses and converts request data to JSON when needed.
- Interceptors: Allows you to intercept requests or responses for logging, modifying headers, or error handling.
- Broad compatibility: Works in Node.js, Deno, Bun, browsers, React Native and even most Javascript workers used in serverless environments.
Why Developers Choose Axios?
Axios has become a favorite among developers due to its simplicity, flexibility, and feature-rich design. Axios provides a reliable and efficient way to handle HTTP requests. Here’s why developers consistently turn to this library:
- Intuitive API: Easy for beginners to learn and integrates well into advanced workflows.
- Customizability: Supports request cancellation, timeouts, and data transformation for tailored functionality.
- Error Handling: Interceptors and robust error management simplify debugging and enhance reliability.
These features make Axios an indispensable tool for many developers, empowering them to write efficient, maintainable, and scalable code with ease.
How to Retry in Axios?
Retries are critical when dealing with unpredictable environments like web scraping or API integrations.
When making HTTP requests, you might encounter:
- Network issues: Interruptions or timeouts due to unstable connections.
- Server errors: Temporary issues on the server side, such as 500 Internal Server Error.
- Rate limits: APIs often limit the number of requests you can make within a specific timeframe.
Retries help ensure that your application remains functional even in these scenarios.
Next, let's explore different retry methods in Axios.
Manual Retry Logic
For developers who prefer control, manual retry logic is a great way to understand how retries work. Here’s an example in nodejs and axios:
// import axios
const axios = require("axios");
async function fetchWithRetry(url, retries = 3) {
let attempt = 0;
// use for loop to retry 3 times at most
while (attempt < retries) {
try {
const response = await axios.get(url);
return response.data; // Success: Return data
} catch (error) {
attempt++;
console.log(`Attempt ${attempt} failed. Retrying...`);
if (attempt === retries) {
throw error; // Fail after exhausting retries
}
}
}
}
This example demonstrates implementing manual retry logic by retrying a request within a while loop. We make our request and retry any thrown errors blindly which is not ideal but it's a good starting point.
Using the "axios-retry" Library
For more complex applications, axios-retry simplifies retry logic with minimal setup and great default options.
This library automatically retries requests that fail due to specified conditions.
const axios = require("axios");
const axiosRetry = require("axios-retry");
// Configure axios-retry
axiosRetry(axios, {
retries: 3, // Number of retries
retryDelay: axiosRetry.exponentialDelay, // Use exponential backoff
// attach callback to each retry to handle logging or tracking
onRetry: (err) => console.log(`Retrying request: ${err.message}`),
// Specify conditions to retry on, this is the default
// which will retry on network errors or idempotent requests (5xx)
retryCondition: (error) => axiosRetry.isNetworkOrIdempotentRequestError(error)
});
axios
.get("https://example.com")
.then((response) => console.log(response.data))
.catch((error) => console.error(error));
The axios-retry library automates retry logic with configurable conditions. In this example, we:
- Set retries to maximum 3 attempts.
- Use exponential backoff to increase the delay between retries. So, first retry will be quite rapid while following ones wait a bit longer.
- Attach a callback to log each retry attempt.
- Specify conditions to retry on network errors or idempotent requests (5xx).
While the default retry conditions are suitable for most cases, you can customize them to fit your specific needs so let's take a look at that next.
Retrying Condition Logic
Effective retry logic ensures that retries are executed under appropriate conditions without overwhelming the server or violating its rate-limiting policies.
By default we should definitely retry all network errors and 5xx status codes. Though, you can also customize the retry conditions to fit your specific needs which is especially applicable in web scraping.
To start, we can expand the default retry conditions to include more http status codes.
HTTP Status Codes to Retry
When designing retry logic, it’s essential to focus on status codes that indicate temporary issues. These include:
Status Code
Description
Why Retry?
408
Request Timeout
Indicates the server didn’t receive the request in time, likely due to network issues. Retry as the request might succeed on subsequent attempts.
500–599
Server Errors
Covers various server-side errors (e.g., 500 Internal Server Error, 503 Service Unavailable). These are often temporary and resolve after some time.
429
Too Many Requests (Rate Limiting)
The server is enforcing rate limits. Retry after a delay, adhering to the Retry-After header if provided.
Retries should not be used for client-side errors (e.g., 400 Bad Request, 401 Unauthorized) because these typically indicate a problem with the request itself that won’t be resolved by retrying.
Retrying Delay
Retrying failed requests immediately can lead to unecessary server overload and eventually client being blocked. To avoid this, the best delay strategy is to implement exponential backoff — a strategy where the delay between retries increases exponentially.
Why Use Exponential Backoff?
Exponential backoff is a critical technique for managing retries effectively in scenarios where network instability or server-side issues occur.
- Reduces Server Load: By increasing the delay, you prevent excessive traffic to the server during downtime.
- Improves Success Rate: Provides more time for transient issues to resolve before retrying.
- Respects Rate Limits: Avoids spamming the server, reducing the risk of being banned or blocked.
How to Implement Exponential Backoff
In axios-retry, you can configure exponential backoff like this:
const axiosRetry = require("axios-retry");
axiosRetry(axios, {
retries: 5, // Number of retry attempts
retryDelay: (retryCount) => {
return Math.pow(2, retryCount) * 1000; // 1s, 2s, 4s, 8s, etc.
},
});
This setup uses axios-retry to implement exponential backoff, doubling the delay with each retry (1s, 2s, 4s, etc.). You can adjust the multiplier to fit your needs, and the retry limit prevents infinite loops, ensuring reliable and efficient error handling.
Retry-After Header
When a server returns a 429 Too Many Requests or 503 Service Unavailable status code, it may include a Retry-After header. This header specifies the amount of time (in seconds) the client should wait before making another request.
Why Respect the Retry-After Header?
The Retry-After header provides crucial guidance from the server on when to attempt the next request, helping your application avoid unnecessary retries.
- Server Compliance: Following the
Retry-Afterheader ensures your application aligns with the server’s rate-limiting policies. - Avoid Repeated Failures: Ignoring the header can lead to repeated failures and potential server bans.
- Efficient Resource Usage: Saves computational resources by delaying retries only when instructed.
Proxy Rotation for Enhanced Reliability
In scenarios like web scraping or API integrations, relying solely on retries may not suffice. Servers often block requests based on repetitive patterns, IP addresses, or user-agent headers.
Proxy rotation is a technique used to mitigate these restrictions by dynamically changing the IP address for each request.
You can learn more about How to Rotate Proxies in Web Scraping in our dedicated article:
[
How to Rotate Proxies in Web Scraping
what is proxy rotation and how it helps to avoid blocking as well as some common rotation patterns, tips and tricks.
](https://scrapfly.io/blog/how-to-rotate-proxies-in-web-scraping/)
Why Retries Are Needed?
Retries are a crucial mechanism in software systems to improve reliability, especially when dealing with external dependencies like APIs, web services, or databases. These systems are often prone to temporary failures caused by network instability, server issues, or rate limiting.
Here’s a more detailed look at why retries are important:
Handling Transient Errors: Many errors, such as timeouts or temporary server overloads, are transient and resolve themselves after a short period.
Dealing with Unreliable Networks: In scenarios like mobile applications or IoT systems, unstable network connections are common. Retrying failed requests can help ensure data is sent or received once the connection stabilizes.
Adhering to Rate Limits: APIs often impose limits on the number of requests allowed within a given timeframe (e.g., 100 requests per minute).
Preventing Data Loss: In critical operations, such as financial transactions or order processing, a failed request without a retry could result in data loss or incomplete operations. Retries provide an additional layer of assurance.
Retries act as a safety net, ensuring applications remain resilient and functional even in unpredictable environments, making them a fundamental part of error-handling strategies.
Axios Retry Troubleshooting
Retry logic is an essential part of building resilient applications, but its implementation must be carefully designed to avoid introducing new issues. If improperly configured, retries can exacerbate problems rather than solving them.
Here’s a closer look at why retry logic might fail and how to address these challenges effectively.
Common Challenges with Retry Logic:
Challenge
Cause
Impact
Solution
Insufficient Backoff
Retries are attempted too quickly, overwhelming the server or causing blocks.
Repeated failures, server bans, or degraded application performance.
Implement exponential backoff to gradually increase retry delays.
Lack of Proxy Rotation
Repeated requests from the same IP address trigger server blocks.
Access denial, ineffective retries.
Use proxy rotation to distribute requests across multiple IPs.
Ignoring Retry-After
Retry logic doesn’t respect the server-provided Retry-After header.
Premature retries, unnecessary failures, and potential server bans.
Parse and honor the Retry-After header for accurate retry timing.
Infinite Retry Loops
No limit on retry attempts leads to endless retries on failure.
High resource usage, server costs, degraded performance, or crashes.
Define a maximum retry limit to prevent infinite loops.
Retrying Non-Retryable Errors
Retrying for errors like 400 (Bad Request) or 401 (Unauthorized).
Increased latency, server load, and no resolution of the issue.
Configure retries for specific, retryable errors like 408, 429, or 5xx responses.
Frequent Retry Overhead
Aggressive retry configurations with too many retries in a short time.
High resource consumption and potential application slowdown.
Balance retry attempts and delays to optimize reliability and resource usage.
Power-Up with Scrapfly
ScrapFly provides web scraping, screenshot, and extraction APIs for data collection at scale.
- Anti-bot protection bypass - scrape web pages without blocking!
- Rotating residential proxies - prevent IP address and geographic blocks.
- JavaScript rendering - scrape dynamic web pages through cloud browsers.
- Full browser automation - control browsers to scroll, input and click on objects.
- Format conversion - scrape as HTML, JSON, Text, or Markdown.
- Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.
FAQ
To wrap up this guide, here are answers to some frequently asked questions about Axios retry.
Does Axios have built-in retry capabilities?
No, Axios doesn't have default retry mechanisms, but libraries like axios-retry can add this functionality.
How do I prevent infinite axios retry loops?
Limit the number of retries using a counter or configuration options, like the retries property in axios-retry. If you're using manual retries, make sure the retry policy prevents infinite loops.
Why are my retries failing with a 429 status?
Ensure you respect the Retry-After header, throttle your HTTP requests, or consider using proxy rotation to bypass unfair rate limiting.
Summary
Implementing retry logic in Axios improves error handling for web scraping and API integrations and axios-retry is a great tool that simplifies the process offering:
- Easy retry condition configuration
- Retry callbacks
- Exponential backoff strategy
As for retries themselves, the best practices include:
- Using exponential backoff to prevent overwhelming servers
- Respecting
Retry-Afterheaders to avoid premature retries - Rotating proxies to prevent IP blocking.
For advanced needs, tools like Scrapfly's Web Scraping API can simplify the process even further.
Top comments (0)