Node.js Web Scraping With Proxies: Axios and Fetch

Which “Node.js Proxy” Do You Mean?

For web scraping, a Node.js proxy is an outbound network intermediary used by an HTTP client such as Axios or fetch. It is not the Proxy metaprogramming object described by MDN, and it is not a reverse proxy that accepts traffic for your application. This article focuses only on outbound requests to public or authorized targets.

Nstdata exposes proxy gateways that can be used by Node HTTP clients. The client library determines how the gateway is attached, how credentials are read, and whether connections are reused. The existing Axios proxy guide covers Axios-specific patterns; this guide compares them directly with current Node fetch behavior.

Node documents fetch as a browser-compatible global in Node.js global fetch documentation. Node's current enterprise networking guidance also describes opt-in environment-proxy support in recent releases, but library-specific configuration remains the most portable choice across supported runtimes.

Prerequisites and Project Setup

A practical Node web scraping proxy example needs a supported Node runtime, an HTTP client, an authorized target, and proxy credentials. The examples use ES modules and avoid embedding secrets.

mkdir node-proxy-demo
cd node-proxy-demo
npm init -y
npm install axios undici

Configure environment variables in your shell or deployment secret store:

export PROXY_HOST="gateway.example"
export PROXY_PORT="8000"
export PROXY_USER="channel-id"
export PROXY_PASSWORD="replace-with-secret"

Never print the completed credential URL. Log a non-secret route ID, hostname label, latency, and acceptance state instead.

Detailed Tutorial

Method 1: Configure an Axios proxy

Axios can pass a proxy object on each request, which keeps credentials separate from the target URL. The current Axios request configuration documents proxy, auth, timeout, and the ability to disable proxy handling with proxy: false when supplying custom agents.

Step 1: Create an Axios client

import axios from "axios";

const client = axios.create({
  timeout: 20_000,
  proxy: {
    protocol: "http",
    host: process.env.PROXY_HOST,
    port: Number(process.env.PROXY_PORT),
    auth: {
      username: process.env.PROXY_USER,
      password: process.env.PROXY_PASSWORD,
    },
  },
  headers: { "User-Agent": "authorized-monitor/1.0" },
  validateStatus: () => true,
});

validateStatus: () => true lets the application inspect every HTTP status in one place instead of turning some responses into transport-like exceptions. This is useful only if you then enforce acceptance yourself.

Step 2: Validate a proxied response

const response = await client.get("https://example.com/", {
  responseType: "text",
});

const accepted =
  response.status === 200 &&
  typeof response.data === "string" &&
  response.data.includes("Example Domain");

if (!accepted) {
  throw new Error(`Unaccepted response: ${response.status}`);
}

console.log({ status: response.status, bytes: response.data.length });

A 200 challenge page is not an accepted record. Test a stable field, selector, schema, or content marker that belongs to the authorized destination. The JavaScript web scraping guide adds parsing patterns once transport is reliable.

Method 2: Use a proxy with Node fetch

Node fetch can route a request through an Undici ProxyAgent supplied as dispatcher. The official Undici ProxyAgent documentation shows the agent URI, dispatcher option, and explicit close() lifecycle.

Step 1: Build the credential URL safely

import { ProxyAgent } from "undici";

const user = encodeURIComponent(process.env.PROXY_USER);
const password = encodeURIComponent(process.env.PROXY_PASSWORD);
const proxyUrl = `http://${user}:${password}@${process.env.PROXY_HOST}:${process.env.PROXY_PORT}`;
const dispatcher = new ProxyAgent(proxyUrl);

Encoding credential components prevents @, :, or / from changing the proxy URL structure. Keep the URL scoped to the process and out of errors.

Step 2: Fetch, validate, and close

try {
  const response = await fetch("https://example.com/", {
    dispatcher,
    signal: AbortSignal.timeout(20_000),
    headers: { "user-agent": "authorized-monitor/1.0" },
  });

  const html = await response.text();
  if (response.status !== 200 || !html.includes("Example Domain")) {
    throw new Error(`Unaccepted response: ${response.status}`);
  }
  console.log({ status: response.status, bytes: html.length });
} finally {
  await dispatcher.close();
}

Closing the dispatcher releases pooled sockets when a short-lived job ends. A server process can reuse a dispatcher for its intended identity and close it during graceful shutdown.

Rotate Proxies by Job, Not by Line of Code

Node proxy rotation is an orchestration decision: choose a route for one independent job, retain it for related requests, then close its network resources.

import { ProxyAgent } from "undici";

const routes = [
  "http://127.0.0.1:9001",
  "http://127.0.0.1:9002",
];

async function fetchAccepted(url, proxyUrl) {
  const dispatcher = new ProxyAgent(proxyUrl);
  try {
    const response = await fetch(url, {
      dispatcher,
      signal: AbortSignal.timeout(15_000),
    });
    const body = await response.text();
    if (response.status !== 200 || !body.includes("Example Domain")) {
      return { accepted: false, status: response.status };
    }
    return { accepted: true, status: response.status, bytes: body.length };
  } finally {
    await dispatcher.close();
  }
}

for (const [index, url] of ["https://example.com/", "https://example.org/"].entries()) {
  const result = await fetchAccepted(url, routes[index % routes.length]);
  console.log({ job: index, ...result });
}

The loop is bounded and does not retry a denial. A production pool should store consecutive failures, cooldown-until time, recent latency, and last accepted timestamp. Retries need exponential backoff plus jitter, and only idempotent requests should be replayed automatically. The batch scraping guide explains why checkpoints and accepted-output accounting matter when the URL set grows.

Nstdata Residential Prime Proxies suit authorized Node jobs that need HTTP/HTTPS gateways plus rotating or sticky session control. The product can centralize route selection while Axios or Undici remains responsible for timeouts, socket lifecycle, and semantic validation. Current product materials describe HTTP, HTTPS, and SOCKS5 protocols and session persistence options. Choose rotation for independent URLs and a sticky session for paginated or login-free journeys that must maintain continuity; proxy access never overrides target policy.

• Residential Prime proxy sessions: Map one provider session ID to one logical Node job and renew it only at a clean boundary.
• Client compatibility: Use Axios's proxy object or Undici's dispatcher rather than assuming every Node package reads environment variables identically.
• Measured acceptance: Track valid records, challenge pages, transport failures, and latency separately so route quality is observable.

Axios vs Fetch Proxy Decision Table

Axios and fetch can both use an authenticated HTTP proxy, but their configuration surfaces and response defaults differ.

Decision	Axios	Node fetch + Undici
Proxy attachment	Request proxy object	ProxyAgent dispatcher
Non-2xx default	Axios rejects by default	fetch resolves; inspect ok or status
Timeout	Axios timeout	AbortSignal.timeout()
Socket lifecycle	Managed by Axios/agents	Reuse and close dispatcher deliberately
Credentials	proxy.auth object	Encoded agent URI or documented token option

Choose Axios if the project already uses its interceptors and error model. Choose native fetch with Undici when reducing wrapper dependencies and explicit dispatcher control matter. Do not mix Axios's built-in proxy field with a custom tunneling agent unless the client is configured with proxy: false; double configuration causes confusing routes.

Troubleshooting Node Proxy Errors

Node proxy failures need separate transport, authentication, timeout, and content diagnostics.

Symptom	Check
ECONNREFUSED	Proxy host, port, listener, firewall
ETIMEDOUT or abort	Connect/read budget, target latency, proxy health
HTTP 407	Proxy credentials or allowlist; do not retry blindly
HTTP 403 or challenge HTML	Authorization and target policy; stop rather than rotate aggressively
Axios ignores agent	Ensure proxy: false when a custom agent owns routing
Process does not exit	Close Undici dispatchers and other open handles

When pages require browser rendering, an HTTP client is the wrong execution model. Review extracting content from JavaScript-heavy sites and move to an authorized browser workflow instead of fabricating DOM content from incomplete HTML.

This article is also maintained in the Nstdata proxy knowledge base.

Conclusion

A dependable Node.js web scraping proxy setup begins by choosing the correct client boundary: an Axios proxy object or an Undici dispatcher. Add secret-safe configuration, timeouts, semantic response checks, bounded rotation, and resource cleanup before increasing throughput. Nstdata can supply managed sessions, while your application retains responsibility for authorization and accepted data.

FAQ

Q: Does Node.js fetch support proxies?

Node fetch can use an Undici ProxyAgent passed as the request dispatcher. Environment-proxy behavior varies by Node release and opt-in settings, so explicit library configuration is clearer across mixed deployments.

Q: How do I authenticate an Axios proxy?

Set proxy.auth.username and proxy.auth.password in the Axios request configuration. Read both values from a secret store or environment variables and never log the resulting configuration object.

Q: Is JavaScript Proxy related to HTTP proxies?

No, JavaScript Proxy intercepts object operations inside the language runtime. An HTTP proxy is a network intermediary used by Axios, fetch, a browser, or another client.

Q: Should I create a new ProxyAgent for every request?

Reuse one ProxyAgent for requests that share a stable identity, then close it when that job or process ends. Create a new agent when the route or session identity intentionally changes.

Q: Why does Axios work but fetch fails through the same proxy?

Axios and fetch use different proxy configuration surfaces. Confirm that Axios uses proxy, fetch uses an Undici dispatcher, credentials are encoded correctly, and both clients apply the same timeout and acceptance checks.

Q: Can a proxy scrape every website?

No, a proxy only changes the network route. It does not grant access, execute JavaScript in an HTTP-only client, solve CAPTCHAs, or replace compliance with site terms and applicable law.