DEV Community

Practical Puppeteer: Using proxy to browse a page

Sony AK on December 15, 2019

Today Puppeteer topic will be related to proxy. Using proxy when browse a page is useful when we want to hide our origin access location. That's on...

Read full post

Peter Hansen • Feb 5 '20

Hi Sony,

Cool Article!

I have also been looking for a proxy solution when using Puppeteer. One of the easiest solutions I found is using API proxy services. You don't have to worry about finding and setting up proxies.

The basic example will look like this:

import puppeteer from 'puppeteer';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const url = 'https://proxybot.io/api/......?url=https://example.com';

  await page.goto(url);
})();

What do you think?

Sony AK • Feb 14 '20

Hi Peter,
Thanks, this is also good way and reduce headache for setting up a proxy :) Thanks for the addition.

Gajus Kuizinas • Jan 29 '20

You can use github.com/gajus/puppeteer-proxy to set proxy either for entire page or for specific requests only, e.g.

import puppeteer from 'puppeteer';
import {
  createPageProxy,
} from 'puppeteer-proxy';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const pageProxy = createPageProxy({
    page,
    proxyUrl: 'http://127.0.0.1:3000',
  });

  await page.setRequestInterception(true);

  page.once('request', async (request) => {
    await pageProxy.proxyRequest(request);
  });

  await page.goto('https://example.com');
})();

To skip proxy simply call request.continue() conditionally.

Using puppeteer-proxy Page can have multiple proxies.

Sony AK • Jan 30 '20

Great addition :) Thank you very much.

But hey, today I learn Playwright, similar like Puppeteer even from the same team, what do you think about that?

Gajus Kuizinas • Jan 30 '20

Playwright is a new project. I would not consider it for production at this time. The only (?) advantage is support for different browser engines. I only see a limited use case for that.

Sony AK • Jan 30 '20

ic ic, but the advantage is the developer is the same like Puppeteer, move out from Google to Microsoft :) So I think they will learn from Puppeteer and improve a lot.

Cuadrix • Feb 9 '20

I created a module for that. It's very simple to use,
First install it:

npm i puppeteer-page-proxy

Then require it:

const useProxy = require('puppeteer-page-proxy');

And then simply use it. To set a proxy for an entire page, do this:

await useProxy(page, 'http://127.0.0.1:771');

Or if you want to set it per requests, just do this:

await page.setRequestInterception(true);
page.on('request', req => {
    useProxy(req, 'socks5://127.0.0.1:9000');
});

Repository: github.com/Cuadrix/puppeteer-page-...

Sony AK • Feb 14 '20

Hi Cuadrix, thanks for the addition. This is cool and more natural for human :)

assender • Jun 17 '20

Great article! I would also add that proxies are very important when you have to bypass various restrictions and access the content you want/need or when you're working with web scraping and similar services.
But it's always tricky to choose the right proxies for yourself, so I've made a short review of residential proxy providers for anyone in need.

theincognitotech • Dec 20 '19

Solid article, you have <3 from me! The thing is that between your references I see free proxy sites and that's not a thing you can trust, believe me. There are many decent proxy providers you can trust for affordable prices, one of my favorite is this one, I even made a review and recommend it.

Sony AK • Dec 21 '19

Hi @theincognitotech thanks for the comment, I agree with you, my list maybe not trusted but that's for quick test purpose :) BTW I will add your list here :) I think that's good one :)

MichaelSwerston • Jun 19 '20

It's crazy how many use cases proxy technology has, I wrote an article on some of the use cases of residential proxies for businesses specifically.

LunaLopezz • Mar 17 '20 • Edited

Great article and simple explanation! I must agree that free proxy services, in this case, might not be an option, but since we have a wide variety of secure and high-quality paid services, such as Smartproxy, Netnut, Microleaves or other it's always a good idea to invest into such services and to stay safe while being/working online.

Sony AK • Apr 21 '20

yes correct, totally agree on this and free proxy is just for temporary solution or just for proof-of-concept, for serious task we must get commercial proxy services.