How to configure Swiftproxy proxy server in Puppeteer?

Puppeteer is a Node library that provides a high-level API to control Chromium or Chrome browsers through the DevTools protocol. When performing tasks such as web scraping and automated testing, configuring a Swiftproxy proxy server is a common requirement because it can help bypass geo-restrictions and prevent IP bans. This article will detail how to configure a Swiftproxy proxy server in Puppeteer.

Basic Configuration

1. Install Puppeteer

First, make sure you have installed Node.js and npm. Then, install Puppeteer via npm:
npm install puppeteer

2. Configure the proxy

There are usually two ways to configure the proxy in Puppeteer: through startup parameter configuration and through code interception and modification of requests.

Method 1: Configure via startup parameters

When starting the Puppeteer browser, you can specify the proxy server via the --proxy-server parameter. This method is simple and direct, but you need to restart the browser every time you change the proxy.

const puppeteer = require('puppeteer');

(async () => {
  const proxyServer = 'proxyserver.com:8080'; // Replace with the proxy server address and port obtained from Swiftproxy
  const browser = await puppeteer.launch({
    args: [`--proxy-server=${proxyServer}`],
    headless: false // Set to false to see browser actions
  });
  const page = await browser.newPage();
  await page.goto('https://ipinfo.io'); // Visit the IP information website to verify that the proxy is working
  // ... Perform other operations
  await browser.close();
})();

If the proxy server requires authentication, you can use the page.authenticate() method to authenticate before accessing the page.

await page.authenticate({
  username: 'your_username',
  password: 'your_password'
});

Method 2: Configure through interceptor

Another more flexible way is to use Puppeteer's request interception function to dynamically set the proxy for each request. This method does not require restarting the browser, but it is relatively complicated to implement.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

  // Listening for requests
  await page.setRequestInterception(true);
  page.on('request', (interceptedRequest) => {
    const url = new URL(interceptedRequest.url());
    // Modify the request URL or other properties as needed
    // For example, to set up a proxy server
    interceptedRequest.continue({
      url: `http://your-proxy-server:port/${url.pathname}${url.search}${url.hash}`,
      headers: {
        // Add necessary proxy authentication or other header information
      }
    });
  });

  await page.goto('https://ipinfo.io');
  // ... Perform other operations
  await browser.close();
})();

‌Note‌: The above method of setting up a proxy through an interceptor is a simplified example and may actually need to be adjusted based on the specific requirements and protocols of the proxy server.

Advanced Configuration

For more complex scenarios, you may need to use a third-party library to assist in setting up the proxy, such as puppeteer-extra and its plugin puppeteer-extra-plugin-proxy.
npm install puppeteer-extra puppeteer-extra-plugin-proxy
Then, import and use these plugins in your code:

const puppeteer = require('puppeteer-extra');
const ProxyPlugin = require('puppeteer-extra-plugin-proxy');

puppeteer.use(ProxyPlugin({
  proxy: 'http://your-proxy-server:port',
  proxyBypassList: [], // Optional, bypass certain domains that do not require a proxy
  auth: {
    username: 'your_username',
    password: 'your_password'
  }
}));

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://ipinfo.io');
  // ... Perform other operations
  await browser.close();
})();

Conclusion

Configuring a proxy server in Puppeteer is a flexible process that can be achieved in a variety of ways. Which method you choose depends on your specific needs, the type of proxy server, and your familiarity with Puppeteer. Whether through startup parameters, interceptors, or third-party plugins, you can easily configure a proxy to improve the efficiency of your crawlers or automated tests.