Using proxies with Node-Fetch

#tutorial #proxies #automation

How to Use Proxies with Node-Fetch for Reliable HTTP Requests

When working on web scraping or any server-side HTTP requests in Node.js, you might find yourself needing to route traffic through proxies — whether to handle geo-restrictions, manage rate limits, or just mask your IP. If you’re using node-fetch for making HTTP calls, integrating proxies is straightforward and can greatly improve stability and reliability.

In this article, we’ll explore how to set up proxies with Node-Fetch, including how to configure residential proxies from providers like DataImpulse to ensure smooth, uninterrupted data fetching.

What is Node-Fetch?

Node-Fetch is a popular library that mimics the browser fetch API within Node.js environments. It lets you perform HTTP requests asynchronously, supporting typical operations such as GET and POST, custom headers, credentials, and more.

This makes it especially useful for any backend task that involves fetching API data, downloading resources, or scraping web pages.

Setting Up Node-Fetch

Before diving into proxies, you need to get Node-Fetch installed and running.

Install Node.js

If Node.js isn’t installed on your machine, download the latest LTS release from Nodejs.org.

Install Node-Fetch

Since Node-Fetch v3 is ESM-only and might not be compatible with all projects yet, this guide uses version 2 (supports CommonJS).

Run this in your project folder:

   npm install node-fetch@2

Create a test file

Create a file called check-ip.js and add this:

   const fetch = require("node-fetch");

   (async () => {
     try {
       const response = await fetch("https://ip-api.com/");
       const data = await response.text();
       console.log(data);
     } catch (error) {
       console.error(error.message);
     }
   })();

Run your test:

   node check-ip.js

This will output your current IP address as seen by the ip-api.com service.

Adding Proxy Support with `https-proxy-agent`

To route your HTTP requests through a proxy, Node-Fetch can accept an HTTP(S) agent. The https-proxy-agent package provides such an agent that supports proxying requests.

1. Install https-proxy-agent

npm install https-proxy-agent

2. Create a Proxy-aware Script

Create a file named proxies.js and set it up like this:

const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');

// Replace with your proxy credentials and server
const proxyUrl = `http://<YourProxyPlanUsername>:<YourProxyPlanPassword>@gw.dataimpulse.com:823`;

(async () => {
  try {
    const proxyAgent = new HttpsProxyAgent(proxyUrl);
    const response = await fetch("https://ip-api.com/", { agent: proxyAgent });
    const data = await response.text();
    console.log(data);
  } catch (error) {
    console.error(error.message);
  }
})();

Replace <YourProxyPlanUsername> and <YourProxyPlanPassword> with your actual DataImpulse proxy credentials.

Understanding Residential Proxies from DataImpulse

Residential proxies route your requests through real residential IP addresses, making your connections appear like ordinary user traffic. This is beneficial for avoiding blocks and CAPTCHAs that target datacenter proxies.

DataImpulse offers residential proxies with automatic rotation:

IPs can change randomly for each request.
Alternatively, they can remain stable for up to 30 minutes.
No manual rotation needed in most cases, simplifying your code.

This makes them ideal for scraping or any geo-sensitive requests you might have.

Proxy Rotation Strategies

If you work with a set of proxies and want to rotate them yourself, you can manage an array of proxy URLs and cycle through them for each request.

Here’s an example using Node-Fetch and https-proxy-agent for rotating proxies from a list:

const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');

const proxyUrls = [
  'http://user:pass@127.0.0.1:10000',
  'http://user:pass@127.0.0.2:10001',
  'http://user:pass@127.0.0.3:10002',
];

(async () => {
  for (const proxyUrl of proxyUrls) {
    try {
      const proxyAgent = new HttpsProxyAgent(proxyUrl);
      const response = await fetch('https://ip-api.com/', { agent: proxyAgent });
      const data = await response.text();
      console.log(`Proxy ${proxyUrl} IP: `, data);
    } catch (error) {
      console.error(`Error with proxy ${proxyUrl}: `, error.message);
    }
  }
})();

You can adjust this to pick proxies randomly or in any sequence depending on your use case.

Why Use Proxies with Node-Fetch?

Bypass geo-restrictions: Access content restricted to certain regions.
Avoid IP bans: Rotate IPs to reduce the risk of being blocked.
Improve anonymity: Hide your originating IP for safer browsing or scraping.
Balance traffic load: Distribute requests across multiple proxies for smoother performance.

Conclusion

Integrating proxies with Node-Fetch is easy and opens up many possibilities for more advanced scraping and API interaction needs. Using tools like https-proxy-agent along with residential proxy services such as DataImpulse allows you to manage IP rotation, bypass restrictions, and maintain reliable access.