Using proxies with Node-Fetch

#tutorial #proxies #automation

How to Use Proxies with Node-Fetch for Smarter HTTP Requests

When building Node.js applications that interact with external web services, you often rely on HTTP clients to fetch data. One popular choice is node-fetch, a package that brings the familiar window.fetch API from browsers to the Node.js environment. Whether you're gathering JSON from APIs, downloading images, or scraping websites, node-fetch provides a simple and flexible way to do it.

Sometimes, direct requests aren’t enough—especially if you need to manage geo-restrictions, avoid rate-limits, or handle unreliable connections. This is where proxies come into play. Integrating proxies with node-fetch can help you route your HTTP requests through different IP addresses, increasing privacy, reliability, and reach.

In this article, we’ll walk through how to set up proxies with node-fetch, including how to use proxy credentials and rotate proxies efficiently. We’ll also introduce DataImpulse as a proxy provider to make your proxy management easy and cost-effective.

Getting Started with Node-Fetch

Before diving into proxies, let's quickly set up node-fetch. Note that node-fetch v3 is ESM-only, which might not suit environments that use CommonJS. For this guide, we'll use node-fetch v2 for compatibility.

Setup Steps:

Install Node.js

Make sure you have Node.js installed. You can download it from Nodejs.org.
Install node-fetch

In your project directory, run:

   npm install node-fetch@2

Basic Node-Fetch Example

Create a file named check-ip.js with the following content:

const fetch = require("node-fetch");

(async () => {
  try {
    const response = await fetch("https://ip-api.com/");
    const data = await response.text();
    console.log(data);
  } catch (error) {
    console.error(error.message);
  }
})();

Run it with:

node check-ip.js

You should see your current IP address output in the terminal.

Adding Proxy Support with https-proxy-agent

For many projects, especially web scraping, you want to send requests through a proxy. This masks your IP and helps with geo-located content or rate-limiting avoidance.

To enable proxy support with node-fetch, we’ll use the https-proxy-agent package.

Installing the Proxy Agent Package

npm install https-proxy-agent

Constructing a Proxy URL with Credentials

Proxies often require authentication. For example, DataImpulse’s residential proxies require a username and password. You build the proxy URL like this:

const proxyUrl = `http://${username}:${password}@proxyserveraddress:port`;

For instance, if you are using DataImpulse residential proxies, you might have:

const proxyUrl = "http://<YourProxyPlanUsername>:<YourProxyPlanPassword>@gw.dataimpulse.com:823";

Replace <YourProxyPlanUsername> and <YourProxyPlanPassword> with your actual DataImpulse credentials.

Using Proxies with Node-Fetch: Complete Example

Here’s how to merge node-fetch with a proxy agent:

const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');

const proxyUrl = "http://<YourProxyPlanUsername>:<YourProxyPlanPassword>@gw.dataimpulse.com:823";

(async () => {
  try {
    const proxyAgent = new HttpsProxyAgent(proxyUrl);
    const response = await fetch("https://ip-api.com/", { agent: proxyAgent });
    const data = await response.text();
    console.log(data);
  } catch (error) {
    console.error(error.message);
  }
})();

Run this script, and it fetches your IP address as seen through the proxy server.

Proxy Rotation for Node-Fetch

If you need to rotate between multiple proxies, DataImpulse’s residential proxies support automatic rotation—changing your IP with every request or reusing one for up to 30 minutes, helping avoid bans or throttling without manual rotation.

Manual Proxy Rotation Example

If you want to rotate proxies manually for any reason, here is a basic approach:

const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');

const proxyUrls = [
  'http://user:pass@127.0.0.1:10000',
  'http://user:pass@127.0.0.2:10001',
  'http://user:pass@127.0.0.3:10002'
];

(async () => {
  try {
    for (const url of proxyUrls) {
      const proxyAgent = new HttpsProxyAgent(url);
      const response = await fetch("https://ip-api.com/", { agent: proxyAgent });
      const data = await response.text();
      console.log(`Response using proxy ${url}:`, data);
    }
  } catch (error) {
    console.error(error.message);
  }
})();

This script cycles through each proxy URL and makes a request through it.

Why Choose DataImpulse for Proxies?

Using residential proxies like those from DataImpulse provides several benefits:

Automatic IP Rotation: No need to manage IP rotation yourself.
Reliable Connectivity: Avoid blocks and geo-restrictions.
Cost-Effective: Affordable plans starting from $1 per GB.
Simple Integration: Easily use with node-fetch via standard proxy URLs.

Wrapping Up

Adding proxy support to your HTTP requests in Node.js enhances your projects by improving reliability, privacy, and access. Using node-fetch along with https-proxy-agent makes proxy integration straightforward.

Whether you need occasional proxy requests or full-scale proxy rotation, combining node-fetch with DataImpulse proxies offers a streamlined solution tailored for web scraping, automation, and more.