You probably already know or have heard about residential proxies. In short, they are the top answer on how to hide an IP address by covering it up with another residential (legitimate) IP. Even though residential proxies are very reliable and block-free, they’re not as fast as datacenter proxies. But what if there's a solution that combines the two – legitimacy and speed?
This is where static proxies, also known as static residential proxies, come in. Static proxies are connected to a single IP home address that doesn’t change. In this article, we’ll go over what they are, how they work, and how they could benefit your use case. Let’s jump right in.
What are static proxies?
Static proxies are datacenter IPs assigned via a contract with an internet service provider (ISP). And as the name “static” implies, it allows you to operate as a residential user with the same IP for as long as required. Additionally, a static IP proxy allows you to avoid IP address rotation, making their use significantly simpler.
Unlike regular datacenter proxies, static proxy services are not created by using virtual machines. These proxies (also known as sticky IP addresses) will look like genuine consumers for most websites. As most websites will only see an average consumer browsing their services, static proxies will be significantly less likely to be flagged as potential bots.
Static proxy benefits
Let’s go through the key benefits a proxy with static IP offers:
- Fast – originating from datacenter proxies, static proxies are just as fast as their counterparts. Usually hosted on virtual servers, static proxy IPs boast excellent bandwidth and high-speed internet connections.
- High anonymity – being assigned by an ISP, static IP proxies offer high anonymity and legitimacy.
- Reliable – Oxylabs’ static IP proxies are as reliable as they come – not only being assigned by an ISP but also coming from a reliable and proven source.
- Guaranteed up to 99% uptime – with the power of datacenter proxies and residential proxies – static proxies have the power of both.
- Unlimited number of targets – static IP proxies have no limits or restrictions when deployed for scraping.
Note that the benefits above usually don’t apply to free static proxy lists due to various feature cuts. As a rule of thumb, costly features can hardly be offered for free.
As the name suggests, a static IP proxy retains the same address over multiple sessions. Remember that maintaining the same IP addresses can be both beneficial and detrimental. Whether a static IP proxy service should be used depends on your goals.
What are static IP proxies used for?
Static proxies (AKA static residential proxies, static IP proxies) can be used when IP rotation is not a viable option. A static proxy service is used whenever changing addresses could lead to blocks or bans.
Sometimes a rotating IP can cause issues or get banned from sites. This usually happens when making a purchase on e-commerce sites, creating social media accounts, or managing business operations on classified sites. Static IP proxy service is best suited for working with websites that require logging in. A static IP ensures a continuous session and lets you avoid interruptions such as re-entering user credentials. Static proxies allow you to continue using target websites and reap all the benefits of staying anonymous at the same time.
Static proxies create a unique solution for businesses looking to protect their brand. Those who try to illegitimately profit from abusing a brand’s image will often try to block those companies from accessing their content. Static proxies allow businesses to afford incredibly high speeds to scrape the internet for possible copyright infringements or other illegal activities related to their brand and, at the same time, to resemble genuine consumers.
But the most common reason to use static proxies is quite apparent. In the words of Oxylabs’ experts:
A static proxy allows you to operate as a residential proxy user with the same IP address for as long as required. Simply put, you get a performance of a datacenter proxy along with the online reliability of a residential proxy.
Conclusions
If you search for reliable proxy servers, static proxies with high anonymity of residential proxies and the speed of datacenter proxies are the way to go. A static IP proxy will be a key player in any project that relies on maintaining the same address and retaining online anonymity.
This returns one element from the page. For this particular element, all we need is text. Text can be easily extracted with this line of code:
document.querySelector('#firstHeading').textContent
The text can now be returned using the return keyword. The next step is to surround this in the evaluate method. This will ensure that this querySelector can be run.
await page.evaluate(() => {
return document.querySelector("#firstHeading").textContent;
});
The result of the evaluate() function can be stored in a variable to complete the functionality. Finally, do not forget to close the browser. Here is the complete script:
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto("https://en.wikipedia.org/wiki/Web_scraping");
title = await page.evaluate(() => {
return document.querySelector("#firstHeading").textContent.trim();
});
console.log(title);
await browser.close();
})();
Scraping multiple elements
Extracting multiple elements would involve three steps:
1. Use of querySelectorAll to get all elements matching the selector:
headings_elements = document.querySelectorAll("h2 .mw-headline");
2. create an array, as heading_elements is of type NodeList.
headings_array = Array.from(headings_elements);
3. Call the map() function can be called to process each element in the array and return it.
return headings_array.map(heading => heading.textContent);
This of course needs to be surrounded by page.evaluate() function. Putting everything together, this is the complete script. You can save this as wiki_toc.js:
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto("https://en.wikipedia.org/wiki/Web_scraping");
headings = await page.evaluate(() => {
headings_elements = document.querySelectorAll("h2 .mw-headline");
headings_array = Array.from(headings_elements);
return headings_array.map(heading => heading.textContent);
});
console.log(headings);
await browser.close();
})();
This file can now be run from your terminal:
node wiki_toc.js
Bonus tip: Array.from() function can be supplied with a map function directly, without a separate call to map. Depending on the comfort level, the same code can thus be written as:
headings = await page.evaluate(() => {
return Array.from(document.querySelectorAll("h2 .mw-headline"),
heading => heading.innerText.trim());
});
Scraping a hotel listing page
This section will explain how a typical listing page can be scraped to get a JSON object with all the required information. The concepts presented in this section will be applicable for any listing, whether it is an online store, a directory, or a hotel listing.
The example that we will take is Airbnb. Apply some filters so that you reach a page similar to the one in the screenprint. In this particular example, we will be scraping this Airbnb page that lists 20 hotels. To scrape all 20 hotels, the first step is to identify the selector for each hotel section.
root = Array.from(document.querySelectorAll("#FMP-target [itemprop='itemListElement']"));
This returns a NodeList of length 20 and stores it in the variable root. Note that so far, text or any attribute has not been extracted. All we have is an array of elements. This will be done in the map() function.
hotels = root.map(hotel => ({
// code here
}));
The URL of the photo of the hotel can be extracted with a code like this:
hotel.querySelector("img").getAttribute("src")
Getting the name of the hotel is a little trickier. The classes used on this page are some random words like _krjbj and _mvzr1f2. These class names appear to be generated dynamically and may change later on. It is better to have selectors which do not rely on these class names.
The hotel name can be extracted by combining parentElement and nextElementSibling selectors:
hotel.querySelector('ol').parentElement.nextElementSibling.textContent
The most important concept to understand here is that we are concatenating querySelectors. Effectively, the first hotel name is being extracted with this line of code:
document.querySelectorAll("#FMP-target [itemprop='itemListElement']")[0].querySelector('ol').parentElement.nextElementSibling.textContent
Finally, we can create an object containing both of these values. The syntax to create an object is like this:
Hotel = {
Name: 'x',
Photo: 'y'
}
Putting everything together, here is the final script. Save it as bnb.js.
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch(url);
const page = await browser.newPage();
await page.goto(url);
data = await page.evaluate(() => {
root = Array.from(document.querySelectorAll("#FMP-target [itemprop='itemListElement']"));
hotels = root.map(hotel => ({
Name: hotel.querySelector('ol').parentElement.nextElementSibling.textContent,
Photo: hotel.querySelector("img").getAttribute("src")
}));
return hotels;
});
console.log(data);
await browser.close();
})();
Run this file from the terminal using:
node bnb.js
You should be able to see a JSON object printed on the console.
Summary
In this Puppeteer tutorial, various examples of web scraping have been explained. The examples ranged from extracting one element from a website and moving on to extracting hotel listings from a popular website. We recommend that you look at the official Puppeteer documentation for more detailed information.
Top comments (0)