Practical Puppeteer: Doing currency converter with Google search result

Today fun web scraping will scrap the Google search result of currency converter. Basically it will scrape the result of this Google search URL.

https://www.google.com/search?hl=en&q=BTC+to+USD

Above URL will convert 1 BTC to USD and Google will display the result in knowledge card form like below.

We will create a small Javascript code to scrap the currency converter result with the help of Puppeteer.

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium. You can read more info about it at https://pptr.dev.

Let's start.

Preparation

Install Puppeteer

npm i puppeteer

The code

This code will try to convert from 1 BTC to USD. You can adjust with your own currency code on the source code.

File google_currency_converter.js

const puppeteer = require('puppeteer');

(async () => {
    let launchOptions = { headless: false, args: ['--start-maximized'] };

    const browser = await puppeteer.launch(launchOptions);
    const page = await browser.newPage();

    // set viewport and user agent (just in case for nice viewing)
    await page.setViewport({width: 1366, height: 768});
    await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36');

    // prepare source and target currency
    let currencyCodeSource = 'BTC', currencyCodeTarget = 'USD';

    // use Google to do currency exchange
    // currently data in Google provided by Morningstar for Currency and Coinbase for Cryptocurrency
    await page.goto(`https://www.google.com/search?hl=en&q=${currencyCodeSource}+to+${currencyCodeTarget}`);

    // wait until the knowledge about currency is ready on DOM
    await page.waitForSelector('#knowledge-currency__updatable-data-column');
    await page.waitFor(2000);

    // get the currency exchange data
    const currencyExchange = await page.evaluate(() => {
        return {
                'currency_source_value': parseFloat(document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('input')[0].getAttribute('value')),
                'currency_source_freebase_id': document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('select')[0].options[document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('select')[0].selectedIndex].getAttribute('value'),
                'currency_source_name': document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('select')[0].options[document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('select')[0].selectedIndex].text,
                'currency_target_value': parseFloat(document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('input')[1].getAttribute('value')),
                'currency_target_freebase_id': document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('select')[1].options[document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('select')[1].selectedIndex].getAttribute('value'),
                'currency_target_name': document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('select')[1].options[document.getElementById('knowledge-currency__updatable-data-column').querySelectorAll('select')[1].selectedIndex].text
               }
    });

    // show the results
    console.log(currencyExchange);

    await page.waitFor(1000);
    await browser.close();
})();

Run it

node google_currency_converter.js

If everything OK then it will run the Chrome browser and will display the result on console like below.

Ow nice, 1 BTC equal 7483.75 USD at the time of writing.

I hope you enjoy it. Thank you very much.

Repository of this sample is available at https://github.com/sonyarianto/currency-converter-using-google-search-result-and-puppeteer

Reference

• https://pptr.dev
• https://www.google.com/search?hl=en&q=BTC+to+USD

Comments

[Andy Danzo]

Good info but I'm trying to figure out what API/Data does currency.wiki or transferwise.com uses to fetch rates since it's not using Google currency rates for sure when comparing conversion values.

[Sony AK]

Hi Andy, thanks for the comment.

Actually I don't have any idea what source they are using. For currency.wiki there is "Links" section and maybe they get the source from one of them. I've check the currency.wiki with Google result the same. The URL shortcut to convert USD to IDR for example is this currency.wiki/usd_idr?value=15 so it's easy to scrape later.

For transferwise.com I don't know since they are not pure currency converter service. Oh wait, the currency converter URL at transferwise.com/gb/currency-conve...

[Andy Danzo]

I appreciate you looking into this, I found one of the forex data source "Xignite" but I'll check comparison tomorrow. Thank you again!

[Sony AK]

Great, never heard about Xignite before, but anyway thanks, I learn something new today.