In these times of Covid-19, gaming consoles are more sought-after than ever before. Since people are staying home most of the time, many have turned to video games as a way to pass the time, resulting in almost instantaneous sell-outs of the newest and hottest gaming consoles released. If you weren't among the lucky few who managed to get their hands on one of these when they were first released, your options are to 1) stalk the retailers' websites daily in the hopes that that item with just happen to be in stock when you're there or 2) buy it from a reseller on eBay for a 200% markup, neither of which sound remotely appealing.
However, if you're a clever software engineer, you'll realize that you can use your software engineering skills to your advantage in this situation. Why not use your knowledge to build a simple program that alerts you whenever the item you want is back in stock?
There exists a Node.js library called Playwright, which enables automation across most browsers. Setup is easy, and a simple web-scraping script can be built in under an hour.
As an example, let's write a program using Playwright that sends an alert when a PlayStation 5 console comes back in stock at Best Buy. A slight downside to this whole thing is that you'll have to write a custom scraper for each retailer that sells the particular item you're looking for, but really, how many of those retailers are there? Probably fewer than 10. And the process for writing these scripts is identical anyway.
Step 1: Installation and Getting Started
First, ensure that you have Node.js installed. To check whether you have Node.js installed, run node -v in the terminal. If you get back a version of Node.js (for example, something that looks like v14.13.1), then you're good to go. Otherwise, you can install it here. Once that's installed, run npm i -D playwright in your terminal.
Next, in your script, require Playwright and launch a browser (in this case, firefox) in an asynchronous function.
const { firefox } = require('playwright');
(async () => {
const browser = await firefox.launch();
// do something here...
})();
This function will be invoked as soon as the script is run.
The code written in this script will heavily rely on the async/await pattern. If you would like more information about that, this documentation on the topic explains it well.
Step 2: Go to Retailer's Site
Let's write a function tailored to searching for a PlayStation 5 Console at Best Buy and pass it the browser. We can then call it in the async function.
(async () => {
const browser = await firefox.launch();
searchBestBuyForPS5(browser);
})();
Inside the searchBestBuyForPS5 function, the first thing that needs to be done is to go to the site of the retailer
var searchBestBuyForPS5 = async (browser) => {
const context = await browser.newContext();
const page = await context.newPage();
await page.goto('https://www.bestbuy.com');
// more code to come...
}
Step 3: Search For Product
From here onwards, your function is going to look a bit different depending on the DOM of the site that you are working with. To search for a PlayStation 5 console on Best Buy's Site, let's first inspect the page and grab the element that contains the search input.
For Best Buy, the search input's id is 'gh-search-input'. We can use that id to specify where to type the search term.
await page.type('#gh-search-input', 'playstation 5 console', {delay: 100});
await sleep(3000);
await page.press('.header-search-button', 'Enter');
await sleep(1000);
(This still goes in the searchBestBuyForPS5 function, right after the code in Step 2.)
Let's break down the code in this code block. In the first line of this code block, the type function of Playwright takes in 2 or more parameters. The first parameter is the selector, which selects the element that we want. In this case, we specify that we want the element with the id of 'gh-search-input'. The second parameter is the text, or the search term. In this case, that's 'playstation 5 console'. (Before moving on, it would be a good idea to actually type in your search term on the retailer's site and ensure that the search results give you the product that you're looking for.) For the third parameter, I have here an optional delay parameter. All this does is delay the typing of each letter in the search bar by 100 ms to better imitate a human. If you don't do this, the site could get suspicious that you're using a bot, which.. you technically are.
The second line in the code block above allows time for the program to type in the full search term before moving on. The sleep helper function called in that line looks like this:
const sleep = (milliseconds) => {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
The third line selects the search bar's submit button with a class name of 'header-search-button' and presses enter. After that, it sleeps for another second for the same anti-bot-detection reasons described above.
Once these lines of code execute, we should have access to a page that displays the search results:
Step 4: Identify the Target Item(s)
Or, more specifically, grab the innerHTML of the target item.
From this search results page, it seems that the items that we want are li elements with a class name of 'sku-item', so we can identify it using that information. Before we can find it, however, we have to be sure that those DOM elements have fully rendered.
await page.innerHTML('li.sku-item'); // wait for elements to render
const results = await page.$$('li.sku-item');
for (let i = 0; i < results.length; i++) {
// get name of item
const skuHeader = await results[i].$('h4.sku-header');
const html = await skuHeader.innerHTML();
// check whether item's name contains "playstation 5" and "console"
if (html.toLowerCase().includes('playstation 5') && html.toLowerCase().includes('console')) {
// check in-stock status...
}
}
(Note: .$ and .$$ are both query selectors. The difference is that .$ returns null if it doesn't find anything that matches while .$$ returns and empty array.)
Step 5: Check Whether Item is In Stock
Inside the conditional in the code block above, we can check whether an item is in stock. First, we must select the element that gives us information about the item's in-stock status. For this particular page, the "Sold Out" button is the same as the "Add to Cart" button, just disabled. Therefore, it still has a class name of 'add-to-cart-button', so we can use that to query for the button.
const button = await results[i].$('button.add-to-cart-button')
const buttonText = await button.innerText()
if (buttonText !== "Sold Out") {
// alert user!
}
Step 6: Alert User That Item is Back In Stock!
At this point, if our program has determined that an item is back in stock, it must alert us so that we can grab it before it sells out again. One way to do this is to send a text alert using Twilio. To do this, you do need to make an account with them and purchase a number that you'll use to send these alerts from.
Complete Code
If you would like to see all the code in one place, here it is:
// require playwright and launch browser
const { firefox } = require('playwright');
(async () => {
const browser = await firefox.launch({ headless: false });
searchBestBuyForPS5(browser);
})();
// helper function for sleeping
const sleep = (milliseconds) => {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
// search whether PS5 is in stock at Best Buy
var searchBestBuyForPS5 = async (browser) => {
// go to Best Buy's site
const context = await browser.newContext();
const page = await context.newPage();
await page.goto('https://www.bestbuy.com');
// enter search term "playstation 5 console"
await page.type('#gh-search-input', 'playstation 5 console', {delay: 100});
await sleep(3000);
await page.press('.header-search-button', 'Enter');
await sleep(1000);
// wait for result products to render and put them in a results array
await page.innerHTML('li.sku-item');
const results = await page.$$('li.sku-item');
// iterate through results array
for (let i = 0; i < results.length; i++) {
// get product's name
const skuHeader = await results[i].$('h4.sku-header');
const html = await skuHeader.innerHTML();
// check whether product's name contains "playstation 5" and "console"
if (html.toLowerCase().includes('playstation 5') && html.toLowerCase().includes('console')) {
// get Sold Out/Add to Cart button
const button = await results[i].$('button.add-to-cart-button');
const buttonText = await button.innerText();
// if the product is not sold out, alert the user
if (buttonText !== "Sold Out") {
console.log("Available!");
// alert the user!!!
}
}
}
};
And there you have it! A simple step-by-step process for writing a program that will allow you to know when an item you want is back in stock. Although the program shown here is specifically for searching for PlayStation 5 consoles at Best Buy, it can easily be altered for other purposes. To search for a different item, simply replace the search term. Searching for an item at a different retailer is slightly more complicated because the program accesses specific elements in the DOM of that page, but the process for writing the program is the same.
These steps can also be used for other purposes. For example, a similar process can be used to alert you when an item goes on sale. Simply alter the code so that you get an alert when the price changes instead of when the "Sold Out" button changes to "Add to Cart."
Lastly, it's worth noting that this script needs to be called consistently and frequently to be effective. For example, if this script were only run once a week, that would be no better than you checking the retailers' sites manually. To do this, you can use a job scheduler, such as cron, to invoke this script every few minutes. Here is an article on how to use cron. Good luck and happy shopping!
Top comments (15)
Can this be repurposed to follow product availability on other sites? Wondering cause I've been following this one product for months and every time it becomes available it disappears!
Anywho, this is super cool. 🙌
if you want to add new retailers you can modify the code in :
Step 2: Go to Retailer's Site
you can for sure
Yes, this can absolutely be repurposed for following any product on any site! Just change the code accordingly :)
Uh, the whole reason they’re so consistently out of stock is that people are using scripts like this to scalp them lol.
So this isn’t doing anything to solve the problem, and the more people use this solution, the less effective it becomes.
Thank you for your share! But I can not afford the ps5.😂
It doesn't matter, the concept can be applied not just to PS5 but to any product on any e-commerce store. I once built a service like that and saved lots of dime. That's a great article Marisa.
I also teach web scraping indepth, perhaps you'd want to check that out:
webscrapingzone.com
Thank you for reading!
Great minds think alike! I just posted about a scraper for the new Xbox using a different library, but I can't wait to try this one out!
I'll definitely be checking out your post too! Thanks for reading!
Very informative and well written article that actually explains what each line or block of code is doing. :)
Thank you!
All is good, until the shop implements a captcha.
With the amount of time I spend configuring my Arch, I barely have time for anything else. The last PS console I played was PS2 lol
I rather spend money and time on my own PC.
Very good read though!
I am really impressed with the creativity here, but how will you handle this problem twitter.com/MarcosBL/status/133645...