DEV Community

Valentina Skakun for HasData

Posted on

Puppeteer + Proxies: How to Stay Unblocked

Using proxies in Puppeteer is clearly a topic that many developers need. I’ve put together this post with small code examples for most common cases - static proxies, per-page routing, authentication, and rotation.

Table of Contents

Static Proxy with --proxy-server

Use a single proxy for all pages. This is simple and native.

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://HOST:PORT']
});
const page = await browser.newPage();
await page.goto('https://httpbin.org/ip');
console.log(await page.evaluate(() => document.body.innerText));
await browser.close();
Enter fullscreen mode Exit fullscreen mode

This will show the IP of your proxy instead of your local IP.

Page-Level Proxy / Request Interception

Puppeteer doesn’t support per-page proxies natively. You can use puppeteer-page-proxy to route requests individually.

npm install puppeteer puppeteer-page-proxy
Enter fullscreen mode Exit fullscreen mode

Example:

import puppeteer from 'puppeteer';
import useProxy from 'puppeteer-page-proxy';

const browser = await puppeteer.launch();
const page = await browser.newPage();

await page.setRequestInterception(true);
page.on('request', req => useProxy(req, 'http://user:password@host:port));

await page.goto('https://httpbin.org/ip');
await browser.close();
Enter fullscreen mode Exit fullscreen mode

Each request goes through the proxy you specify.

Handling Proxy Authentication

This method of authentication won’t work:

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://host:port]
});
Enter fullscreen mode Exit fullscreen mode

Use the previous example with puppeteer-page-proxy, or for HTTP/HTTPS proxies that require login, use page.authenticate().

const browser = await puppeteer.launch({
  args: ['--proxy-server=http://host:port]
});
const page = await browser.newPage();
await page.authenticate({ username: 'USER', password: 'PASS' });

await page.goto('https://httpbin.org/ip');
console.log(await page.evaluate(() => document.body.innerText));

await browser.close();
Enter fullscreen mode Exit fullscreen mode

Always authenticate before navigating to the page.

Proxy Rotation

Rotate proxies to avoid bans. Combine with random user-agents for better results.

const pool = [
  'http://user:password@host1:port',
  'http://user:password@host2:port',
  'http://user:password@host3:port'
];

let i = 0;
function getNextProxy() {
  return pool[i++ % pool.length];
}

const proxy = getNextProxy();
const browser = await puppeteer.launch({ args: [`--proxy-server=${proxy}`] });
const page = await browser.newPage();
await page.goto('https://httpbin.org/ip');
console.log(await page.evaluate(() => document.body.innerText));
await browser.close();
Enter fullscreen mode Exit fullscreen mode

You can expand this with headers rotation or retry logic for blocked proxies. Here’s a link to a list of latest User Agents.

Troubleshooting

Most common errors:

  • 407 Proxy Authentication Required. Check username/password and call page.authenticate(). This can also happen if you used the wrong authentication method.
  • ERR_PROXY_CONNECTION_FAILED. Test host and port, try connecting with curl first.
  • 403/CAPTCHA. Rotate proxies, headers, or user-agent.

Notes for Devs:

Full Guide on How to Set Up Proxy in Puppeteer
More examples on Github
Join our Discord
If you want any examples I might have missed, leave a comment and I’ll add them.

Top comments (0)