DEV Community: Evan

Making Better Use of Lighthouse with Puppeteer

Evan — Fri, 15 May 2020 13:02:00 +0000

There are tools we can use to evaluate the performance, accessibility, and the use of web developer best practices on our websites and in our web apps. One of these tools is built into Google Chrome. It's called Lighthouse.

Lighthouse analyzes web apps and web pages, collecting modern performance metrics and insights on developer best practices.

The metrics you get back tell you how well a particular webpage is setup and configured to give the best possible user experience on multiple vectors, like page performance, accessibility, and SEO. It even gives you actions to implement that will improve your score and thus your webpage's or web app's user experience.

It's easy to run because it's built directly into Google Chrome and can even emulate different device types. If you are using Chrome and reading this article now, take some time and run Lighthouse:

1) Right click on this webpage and select Inspect.
2) Go to the Audits tab and click Generate Reports.

I ran it on my About page and my results are as follows:

Full disclosure, this is the latest score. I ran this the first time a few days ago and had a disappointing, 79, as my accessbility score. It recommended that I add alt text attribute to my profile photo and aria-labels to the icons in the top right. I did and now I have a perfect score and more importantly, this webpage is more accessible.

You can also run it at the command line, by installing and running the Node module

npm install -g lighthouse
lighthouse https://evanhalley.dev/about

The output is a HTML report that resembles the report generated in Chrome Dev Tools.

Being a Node module, it's pretty easy to run Lighthouse programmatically.

const lighthouse = require('lighthouse');
const chromeLauncher = require('chrome-launcher');

function launchChromeAndRunLighthouse(url, opts, config = null) {
  return chromeLauncher.launch({chromeFlags: opts.chromeFlags}).then(chrome => {
    opts.port = chrome.port;
    return lighthouse(url, opts, config).then(results => {
      // use results.lhr for the JS-consumable output
      // https://github.com/GoogleChrome/lighthouse/blob/master/types/lhr.d.ts
      // use results.report for the HTML/JSON/CSV output as a string
      // use results.artifacts for the trace/screenshots/other specific case you need (rarer)
      return chrome.kill().then(() => results.lhr)
    });
  });
}

const opts = {
  chromeFlags: ['--show-paint-rects']
};

// Usage:
launchChromeAndRunLighthouse('https://example.com', opts).then(results => {
  // Use results!
});

What if we needed to run Lighthouse on a webpage that is behind some type authentication, like many web apps are? This is where Puppeteer comes in handy.

Scenario

For this scenario, I am imaging I am a web developer at HungerRush. HungerRush (aka Revention) provides an online ordering platform like DoorDash or Postmates. I picked them because a pizza place local to me, Salvio's, uses their platform to enable online ordering. I want to write a script that will run Lighthouse on the account management page, which is behind an authentication wall.

The webpage I will be testing is https://salviospizza.hungerrush.com/Order/OrderType.

The goal here is to use Puppeteer to log into the website. When this happens, the website will use cookies to remember the fact that I logged in. I will then pass the Google Chrome browser instance used by Puppeteer and the URL to the account page to Lighthouse. Lighthouse will open a new tab in this Chrome instance and do it's thing. The output is a report data object containing all of the information generated by Lighthouse.

Let's get started:

Logging In

This part is not particularly important. Your implementation will definitely differ depending on how users are able to log into your website. Using Puppeteer, I can log into the account page with the following code:

const fs = require('fs');
const puppeteer = require('puppeteer');
const lighthouse = require('lighthouse');
const config = require('lighthouse/lighthouse-core/config/lr-desktop-config.js');
const reportGenerator = require('lighthouse/lighthouse-core/report/report-generator');

const PIZZA_PROFILE_URL = 'https://salviospizza.hungerrush.com/Account/Manage';
const browser = await puppeteer.launch({ headless: true });

console.log('Navigating to Pizza Profile...');
const page = (await browser.pages())[0];
await page.goto(PIZZA_PROFILE_URL, { waitUntil: 'networkidle0' });

console.log('Starting login, entering username and password...');
await page.type('#UserName', process.env.USERNAME);
await page.type('#Password', process.env.PASSWORD);

console.log('Logging in....');
await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.click('#btnLogin'),
]);

console.log('Pizza profile unlocked!');

Running the snippet logs me in to my profile!

The Handoff to Lighthouse

The handoff to Lighthouse is straightforward. Lighthouse communicates with Google Chrome in the same way Puppeteer does, over CDP (Chrome DevTools Protocol). I simply get the port of the running Chrome instance and give that to Lighthouse, along with the URL to evaluate, and some other parameters.

const config = require('lighthouse/lighthouse-core/config/lr-desktop-config.js');

console.log('Running lighthouse...');
const report = await lighthouse(PIZZA_PROFILE_URL, {
    port: (new URL(browser.wsEndpoint())).port,
    output: 'json',
    logLevel: 'info',
    disableDeviceEmulation: true,
    chromeFlags: ['--disable-mobile-emulation', '--disable-storage-reset']
}, config);

The method, browser.wsEndpoint(), returns a value that resembles ws://127.0.0.1:63980/devtools/browser/666ea71c-a4e4-4777-962c-e26b6cf41ccd.

Once we have the raw report object we can generate HTML and/or JSON versions of the information and save it to disk.

const json = reportGenerator.generateReport(report.lhr, 'json');
const html = reportGenerator.generateReport(report.lhr, 'html');
console.log(`Lighthouse scores: ${report.lhr.score}`);

console.log('Writing results...');
fs.writeFileSync('report.json', json);
fs.writeFileSync('report.html', html);
console.log('Done!');

You probably wouldn't need both types of reports, depending on your use case. In continuous integration environment that has implemented this type of per-build Lighthouse analysis, I can see the JSON version being more usable (and machine-parsable) than the HTML version.

This entire code sample has been uploaded to GitHub.

Original posted at evanhalley.dev.

Hey look, it's Playwright, like Puppeteer but with more browser (support)

Evan — Mon, 17 Feb 2020 21:07:06 +0000

I've written extensively about the many things you can do with Puppeteer. A deal breaker for some workflows, like automated testing + coverage, is that it only officially supports Google Chrome & Chromium browsers (with some experimental Firefox support sprinkled in). If you needed browser diversity in your automated modern workflow, your choices were limited. Recently, your options open up a bit.

The developers who worked to bring Puppeteer to life are the ones behind a very similar framework with official support for Chrome(mium), Webkit, and Firefox browsers on Linux, macOS, and Windows.

Playwright is a Node library to automate the Chromium, WebKit and Firefox browsers with a single API. It enables cross-browser web automation that is ever-green, capable, reliable and fast.
Our primary goal with Playwright is to improve automated UI testing by eliminating flakiness, improving the speed of execution and offering insights into the browser operation.

I want to take the time to step through getting all set up and running in 5 minutes or less, then leverage it's cross-browser-ness.

Install

The following snippet will bootstrap a Nodejs project and download Playwright and it's dependencies.

mkdir hello-playwright
cd hello-playwright
npm init // accept all of the defaults
npm install --save playwright
touch index.js

Take Screenshots

Let's do an easy thing and see what this website, evanhalley.dev looks like in Chromium, Firefox, and Webkit. Open index.js and enter the following:

const playwright = require('playwright');

(async () => {
    for (const browserType of ['chromium', 'firefox', 'webkit']) {
        const browser = await playwright[browserType].launch();
        const context = await browser.newContext();
        const page = await context.newPage('https://evanhalley.dev/');
        await page.screenshot({ path: `evanhalleydev_${browserType}.png` });
        await browser.close();
    }
})();

Run the script with node index.js. After the script finished, you should have three images in the current directory showing you what this website looks like in 3 different browsers.

With Playwright, iterating through the three browser types is pretty easy, 'chromium', 'firefox', 'webkit'. This can be pretty powerful for those looking to quickly generate screenshots of their web app or to validate their UI designs across different browsers.

One difference between Puppeteer and Playwright deals with how new pages are created. In Puppeteer:

const page = await browser.newPage();
const response = await page.goto('https://evanhalley.dev');

Creates a new page in the default context of the browser. What's not immediately apparent is that all pages created this way will share their cookies and cache.

With Playwright, the first thing you need to do is to create a BrowserContext.

Playwright allows creation of "incognito" browser contexts with browser.newContext(). "Incognito" browser contexts don't write any browsing data to disk.

const context = await browser.newContext();
const page = await context.newPage('https://evanhalley.dev/');

There is no way to create a page as part of the default browser context with Playwright, which is important for making sure you start off with a blank slate when running your automated tests.

Execute a Google Search

Here's how I executed a Google Search in my first Puppeteer article "Getting Started with Puppeteer".

let searchBox = await page.$('#tsf > div:nth-child(2) > div > div.RNNXgb > div > div.a4bIc > input');
let searchButton = await page.$('#tsf > div:nth-child(2) > div > div.FPdoLc.VlcLAe > center > input[type=submit]:nth-child(1)');
await searchBox.type('Avengers Endgame reviews NO SPOILERS!');
await searchButton.click();

What about Playwright?

const { firefox } = require('playwright');

(async () => {
    // get a browser, context, and page objects...also start Firefox with a "head" so we can see what's happening
    const browser = await firefox.launch({ headless: false });
    const context = await browser.newContext();
    const page = await context.newPage('https://google.com');

    // execute the search, the CSS selectors have changed since my first Puppeteer article
    const searchBox = await page.$('#tsf > div:nth-child(2) > div.A8SBwf > div.RNNXgb > div > div.a4bIc > input');
    const searchButton = await page.$('#tsf > div:nth-child(2) > div.A8SBwf > div.FPdoLc.tfB0Bf > center > input.gNO89b');
    await searchBox.type('Weather in Greensboro, North Carolina');
    await searchButton.click();

    // wait for the page to finish navigating before taking a screenshot
    await page.waitForNavigation();
    await page.screenshot({ path: 'a-google-search-with-firefox.png' });
    await browser.close();
})();

Emulating an iPhone 11 Pro

Having programmatic access to different browsers means we can emulate certain conditions, like a Webkit browser with the iPhone 11 Pro's device characteristics. Here I want to see how many requests are executed when the New York Times thinks its loading it's homepage on an iPhone versus a desktop browser.

const { webkit, devices } = require('playwright');
const iPhone11 = devices['iPhone 8 Pro'];

(async () => {
    const devicesToUse = [ iPhone11, null ];

    for (let i = 0; i < devicesToUse.length; i++) {
        const browser = await webkit.launch();
        let context; 

        if (devicesToUse[i]) {
            context = await browser.newContext({
                viewport: iPhone11.viewport,
                userAgent: iPhone11.userAgent
            });
        } else {
            context = await browser.newContext();
        }
        const page = await context.newPage();
        await page.setRequestInterception(true);
        let numRequests = 0;
        page.on('request', request => {
            numRequests++;
            request.continue();
        });
        await page.goto('https://nytimes.com');
        console.log(numRequests);
        await browser.close();
    }
})();

Outro

If you've used Puppeteer in the past, this all looks really familiar, and thats a good thing. The API and capabilities are similar, but now you can use other browsers. Playwright is still pre-1.0 so expect a lot of changes going forward.

Playwright is being actively developed as we get to the feature parity across Chromium, Firefox and WebKit. Progress on each browser can be tracked on the Is Playwright Ready? page, which shows currently failing tests per browser.

✌🏿

(originally posted at EvanHalley.dev)

Automate Your Hugo Website Deploys

Evan — Mon, 27 Jan 2020 18:07:47 +0000

My website, EvanHalley.dev, exists as a bunch of Markdown files on GitHub*. I use Hugo to turn the markdown into HTML, which then gets deployed to Firebase. The process is super straightforward.

Write new content, like this post, and commit it to GitHub.
Build the HTML files and associated assets using hugo on the command line.

hugo --minify --gc --cleanDestinationDir

Using the Firebase CLI, deploy everything to Firebase.

firebase deploy --only hosting

This is not that much work, but then I heard of GitHub actions.

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.

GitHub Actions make it really easy to trigger an event based on some repository action. Naturally, I thought it would be nice if I could deploy my website to Firebase whenever I pushed new commits it's GitHub repository. It was pretty easy, leveraging two GitHub Actions, GitHub Actions for Firebase and Hugo Setup.

My build and deploy action is intuitive.

The trigger is a push to master.

name: Build & deploy

on:
  push:
    branches: 
    - master

First step, checkout the master branch.

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@master

Next, download Hugo using the Hugo Setup GitHub Action.

- name: Setup Hugo
  uses: peaceiris/actions-hugo@v2.2.0
  with:
    hugo-version: '0.63.1'

Then, build the HTML and associated assets.

- name: Build with Hugo
  run: hugo --gc --minify --cleanDestinationDir

Finally, deploy the generated HTML to Firebase, using the GitHub Actions for Firebase.

- name: Deploy to Firebase
  uses: w9jds/firebase-action@master
  with:
    args: deploy --only hosting
  env:
    FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}

The cool thing about this is that GitHub actions exist as source code in the repository they service, so you can edit them in your development environment. Once you push them to GitHub, they are live and actionable.

If you are in the niche of writers who host their Hugo site on Firebase, I hope you find this useful. Please find the full build script on Git Hub

An aside: I've been writing on the web in various forms since the early 2000s. Most of my work is still up at https://emuneee.wordpress.com and https://emuneee.com. I got tired of having to keep my Wordpress installation updated. I also wanted to free myself from needing to do server maintenance as well (ie. updating Linux or MySQL). Right before thinking of transitioning to a static HTML website, I began doing my notetaking in Markdown.

With all of this in mind, moving to a static HTML website was perfect. I found Hugo, which enabled me to turn my markdown into HTML to host on the web. I found Firebase Hosting for hosting my HTML content. Using Firebase allows me to host my site for free and benefit from Google's global content delivery network.

Puppyteer Crawler

Evan — Wed, 08 Jan 2020 16:39:15 +0000

The ability to use Puppeteer to automate Google Chrome makes it awesome for doing web crawling. Chrome executes the JavaScript and many times this yields more URLs to crawl. In this month's Puppeteer Experiment, I combine a Puppeteer powered web crawler with some #MachineLearning to crawl a pet shelter’s website for all of the adorable dog pictures. I call it, the Puppyteer Crawler (alternate title: The Puppeteer Collar). 🐶

Overview

This is going to be less of a guide and more of a journey on the "what" and the high level "how" of putting my Puppyteer Crawler together to find all of the adorable dog pictures.

You can jump straight to the source code on GitHub.

Components

I used about 7 or so libraries, but here are the important ones.

Headless Chrome Crawler

Headless Chrome Crawler is a Node.js/JavaScript dependency that you can configure to crawl websites. It differs from some other web crawlers in that it uses Google Chrome as the conduit through which webpages (and JavaScript) are loaded and executed.

Crawlers based on simple requests to HTML files are generally fast. However, it sometimes ends up capturing empty bodies, especially when the websites are built on such modern frontend frameworks as AngularJS, React and Vue.js.

Powered by Headless Chrome, the crawler provides simple APIs to crawl these dynamic websites with the following features...

It's not difficult to get up and running. The following snippet crawls Kevin Bacon's Wikipedia page, printing page titles and information along the way.

const HCCrawler = require('headless-chrome-crawler');

(async () => {
    let crawler = await HCCrawler.launch({
        maxDepth: 2,
        evaluatePage: (() => ({
            title: $('title').text(),
        })),
        onSuccess: (result => console.log(result)),
    });
    await crawler.queue('https://en.wikipedia.org/wiki/Kevin_Bacon');
    await crawler.onIdle();
    await crawler.close();
})();

My use case for using the crawler was finding all of the images loaded by Chrome while crawling a pet shelter's website. I implemented a customCrawl. A customCrawl allows you to, among other things, interact with the Puppeteer Page object.

customCrawl: async (page, crawl) => {
    await page.setRequestInterception(true);

    page.on('request', request => {
        let requestUrl = request.url();

        if (request.resourceType() == 'image' && !imageUrls.has(requestUrl)) {
            imageUrls.add(requestUrl);
            request.abort();
        } else {
            request.continue();
        }
    });
    let result = await crawl();
    result.content = await page.content();
    return result;
}

With access to the Page, I can use request interception to record the URLs that lead to images. I'm saving each url leading to an image for classification by Tensorflow in a later step.

Tensorflow.js

TensorFlow.js is a JavaScript wrapper around the popular machine learning framework, TensorFlow. TensorFlow is a framework for building, training, and using machine learning models to do advanced computation, like text-to-speech or image recognition. Normally, you'd write all of your TensorFlow logic in Python. TensorFlow.js allows you do accomplish your machine learning tasks with JavaScript. This means you can easily load models in the browser or server-side via Node.js.

TensorFlow.js also comes with a handful of pre-built machine learning models, so you don't need in PhD to get up and recognizing quickly.

My implementation takes a URL to an image we recorded in a previous step, gets the binary data from the web server, then provides it to a pre-built object recognition model, coco-ssd.

More about coco-ssd:

Object detection model that aims to localize and identify multiple objects in a single image.

This model is a TensorFlow.js port of the COCO-SSD model. For more information about Tensorflow object detection API, check out this readme in tensorflow/object_detection.

This model detects objects defined in the COCO dataset, which is a large-scale object detection, segmentation, and captioning dataset. You can find more information here. The model is capable of detecting 90 classes of objects. (SSD stands for Single Shot MultiBox Detection).

This TensorFlow.js model does not require you to know about machine learning. It can take input as any browser-based image elements (img, video, canvas elements, for example) and returns an array of bounding boxes with class name and confidence level.

The cool thing about coco-ssd is that it will detect as many things in an image as it can and generate a bounding box that identifies where in the image an object is located. The detect method will return an array of predictions, one for each object detected in the image.

const tf = require('@tensorflow/tfjs');
const tfnode = require('@tensorflow/tfjs-node');
const cocoSsd = require('@tensorflow-models/coco-ssd');
const request = require('request');

function getImagePixelData(imageUrl) {
    return new Promise((resolve, reject) => {
        let options = { url: imageUrl, method: "get", encoding: null };

        request(options, (err, response, buffer) => {
            if (err) { reject(err); } 
            else { resolve(buffer);}
        });
    });
}

(async () => {
    let model = await cocoSsd.load({ base: 'mobilenet_v2' });
    let predictions = [];

    try {
        let url = 'https://www.guidedogs.org/wp-content/uploads/2019/11/website-donate-mobile.jpg';
        let imageBuffer = await getImagePixelData(url);

        if (imageBuffer) {
            let input = tfnode.node.decodeImage(imageBuffer);
            predictions = await model.detect(input);
            console.log(predictions);
        }
    } catch (err) {
        console.error(err);
    }
})();

Here is a picture of a dog.

Passing it into the coco-ssd model yields:

[
  {
    bbox: [
      62.60044872760773,
      37.884591430425644,
      405.2848666906357,
      612.7625299990177
    ],
    class: 'dog',
    score: 0.984025239944458
  }
]

Get Up & Running

Step 1 - Clone the repository

git clone git@github.com:evanhalley/puppyteer-crawler.git

Step 2 - Download the dependencies

cd puppyteer-crawler
npm install

Step 3 - Find the photos of dogs

node . --url=spcawake.org --depth=1 --query=dog

Output

Searching https://spcawake.org for images containing a dog...
The domain for the URL is spcawake.org...
Starting crawl of https://spcawake.org...
Crawled 1 urls and found 25 images...
Classifying 25 images...
 ████████████████████████████████████████ 100% | ETA: 0s | 25/25
Images that contain a dog
https://spcawake.org/wp-content/uploads/2019/11/Clinic-Banner-2-820x461.jpg
https://spcawake.org/wp-content/uploads/2019/03/Dog-for-website.jpg
https://spcawake.org/wp-content/uploads/2019/03/volunteer-website-pic.jpg
https://spcawake.org/wp-content/uploads/2019/12/Social-Dog-250x250.jpg
https://spcawake.org/wp-content/uploads/2019/12/Alhanna-for-blog-v2-250x250.jpg

Recap

This experiment allowed me to use two libraries to accomplish a task that is normally kind of intense if done manually, depending on the size of the website. Using Tensorflow.js allows you to leverage models already created and trained to identify different types of objects. You could even train a model yourself to detect, for example, all of the pictures of 1992 Volkswagen GTIs on a classic car website.

Using a web crawler that leverages Puppeteer ensures you render the JavaScript and crawl URLs that result from the processed JavaScript. This makes collecting the data to feed to your model easy and painless.

✌🏿

(Originally published at evanhalley.dev

Turning Markdown into PDFs using Puppeteer

Evan — Tue, 26 Nov 2019 02:55:37 +0000

Puppeteer is useful in many other contexts besides doing automated things on the web. It's particularly useful for generating PDFs. I want to highlight an NPM module / command line utility for turning Markdown into PDFs, Markdown to PDF. It leverages Puppeteer and Google Chrome.

What is Markdown?

Backing up a bit, Markdown is a light plain text markup language and text-to-HTML conversion tool. It's really useful for writers, in particularly, for developers.

Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML).

I love it because I take a plenty of notes in text files when attending talks, taking notes for meetings, for research, and more. It has a light and easy to learn formatting syntax that allows writers to easily make their writing look well. The thing I found most useful is the there's formatting for including code snippets, with syntax highlighting.



let message = 'Built in syntax highlighting is great';

Markdown is pretty popular in the development community because of it's feature set and is a natively format for documentation on popular software repository hosting services, like GitHub and Bitbucket. For example, GitHub will appropriately render Markdown for files hosted on it's platform, like the readme file for Get-Me-The-Gif.

I love Markdown so much, that I use it as the content format for my website, EvanHalley.dev. I write all of my articles in Markdown, and using Hugo, convert all of my markdown to HTML that is then uploaded to the web. Dev.to also uses Markdown as their content source of choice.

Installing



npm install --global md-to-pdf

Using Markdown to PDF

Markdown to PDF allows you to easily turn markdown files in PDFs. This is immediately useful if I wanted to send some developer documentation to a product manager via email, but want my Markdown to render correctly.

Usage on the Command Line

I want to convert the following Markdown to a PDF. I've saved it as shopping.md. The md extension denotes a markdown file.



# This is a markdown file

## Shopping List
* Macaroni noodles
* Sweet potatoes
* Pie crust
* Turkey
* Stuffing
* Collard greens

> "Don't forget the butter"

I can use Markdown to PDF to convert this to a PDF.



md-to-pdf shopping.md shopping.pdf

A screenshot of the generated PDF:

Usage in a Nodejs App



const markdownToPdf = require('md-to-pdf');

(async () => {
    let pdf = await markdownToPdf('shopping.md', { dest: 'shopping.pdf' });
    console.log(pdf.filename);
})();

The outcome of the the use is the same, however, it's very easily integrated into larger JavaScript applications. The example that immediately comes to mind is generating PDFs of all the documentation in a GitHub repository that can be distributed or uploaded elsewhere.

Thanks for reading! Reach out to me on Twitter if you have any questions.

If you like this article, please subscribe to my newsletter, The PuppetHero Digest, where I share articles just like this with you, sent directly to your email inbox!

How Serverless, Puppeteer, FFMPEG, and AWS Lambda can help you get the GIFs from Twitter

Evan — Thu, 14 Nov 2019 18:42:24 +0000

Running Puppeteer on Amazon Web Services' (AWS) serverless computing platform, Lambda, is it a bit of a challenge. Puppeteer and it's bundled version of Chromium easily exceeds the lambda size limitation. There are a few options you opt for instead:

Run or maintain your own farm of Google Chrome / Chromium instances
Use a Google Chrome / Chromium-as-a-Service offering like Browserless
Use an NPM module, npm i @serverless-chrome/lambda to handle this for you

I'm going to be walking you through how I built "Get Me the GIF" (referred to as GMTG hereafter) with Puppeteer, Serverless, Chromium, and Nodejs.

The Idea

The idea behind GMTG is an easy one. When Twitter user's tweet GIFs out, Twitter will show these GIFs as videos instead of GIFs.

// Detect dark theme var iframe = document.getElementById('tweet-985862643159363584-813'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=985862643159363584&theme=dark" }

I'm somewhat of a GIF collector. You never know when a GIF is more useful in communicating an idea than words are. When I see GIFs on Twitter that are funny, I like to save those for later. Twitter for the web and Android make this impossible. One day, I had more free time than usual and decided to open up Chrome DevTools and explore the code behind how a tweet is displayed in the browser. Long story short, it's relatively easy to capture and download these GIFs (videos). I wrote a little extra code to use FFMPEG to convert the video I capture from Twitter into a GIF I can add to my collection.

I'm not going to highlight all the pieces I used, but just a few pieces of code I thought were useful in helping me reach my end goal, getting all of those GIFs.

The Flow

You give GMTG a URL to a tweet containing the GIF, like:

https://twitter.com/EvanHalley/status/1130891914675445760

It should give you a GIF.

The overall flow:

Open the Tweet
Intercept the network request that rendered the MP4 preview frame
Parse out identifier of the MP4 preview frame
Build the URL to access the MP4 and download it
Use FFMPEG to convert the MP4 to a GIF
Make the GIF available to the user

Getting Chrome to Run in AWS Lambda

Note: For this project, I am using the Serverless Framework to debug and deploy my serverless app to AWS Lambda. I'm not going to dive into Serverless, but checkout this great Serverless tutorial.

As mentioned earlier, getting a Puppeteer + Chrome based Nodejs app running in AWS Lambda is difficult because of the deployment package size limits. In order to get around this limitation, I used a NPM module, serverless-chrome.

Without jumping to far into the details, serverless-chrome handles everything needed to get Chrome up and running in a serverless environment and manages to get around the deployment package size limitations.

The aim of this project is to provide the scaffolding for using Headless Chrome during a serverless function invocation. Serverless Chrome takes care of building and bundling the Chrome binaries and making sure Chrome is running when your serverless function executes. In addition, this project also provides a few example services for common patterns (e.g. taking a screenshot of a page, printing to PDF, some scraping, etc.)

Once you have a Nodejs, Puppeteer, and Serverless project bootstrapped, you can easily add serverless-chrome:

npm install --save @serverless-chrome/lambda

In your source code, connect to a Chrome instance running in a serverless environment:

const launchChrome = require("@serverless-chrome/lambda");

async function getChrome() {
    let chrome = await launchChrome();

    let response = await request
        .get(`${chrome.url}/json/version`)
        .set("Content-Type", "application/json");

    console.log(JSON.stringify(response.body));
    let endpoint = response.body.webSocketDebuggerUrl;

    return {
        endpoint,
        instance: chrome
    };
}

The code snippet above, calls launchChrome() to start a Chrome process. Once it's launched, we can query the Chrome instance to find the URL to the Chrome DevTools Protocol (CDP) socket. Puppeteer uses this URL to connect to Chrome.

Making a GET request to this URL + /json/version returns:

{
   "Browser": "HeadlessChrome/78.0.3904.97",
   "Protocol-Version": "1.3",
   "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/78.0.3904.97 Safari/537.36",
   "V8-Version": "7.8.279.23",
   "WebKit-Version": "537.36 (@021b9028c246d820be17a10e5b393ee90f41375e)",
   "webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/browser/0fbe3418-968a-4d57-9b53-0cf20d590eec"
}

We are after webSocketDebuggerUrl.

Once we have a URL to the CDP socket, connecting with Puppeteer is straighforward:

const puppeteer = require("puppeteer");
...
let cdpSocketUrl = 'ws://127.0.0.1:9222/devtools/browser/0fbe3418-968a-4d57-9b53-0cf20d590eec';
let browser = await puppeteer.connect({
    browserWSEndpoint: cdpSocketUrl
});

There is a caveat to know about when using serverless-chrome. It has not been updated in over a year, which means the latest prebuilt version of Chromium (69.0.3497.81) it uses is over a year old. This means it is pretty much only guaranteed to work with older versions Puppeteer.

Intercepting Requests with Puppeteer

Puppeteer has a handy API for intercepting ALL network requests the browser makes when loading a web page. You can intercept these requests and either continue or abort them. There are some really useful cases where this level of control is desired, such as capturing a webpage screenshot, but not processing any of the image or javascript. In my case, I just wanted to identify the MP4 thumbnail network request.

I discovered that the URL to the MP4 thumbnail looks like:

https://pbs.twimg.com/tweet_video_thumb/1234567890.jpg

The link to the MP4 looks like:

https://video.twimg.com/tweet_video/1234567890.mp4

Using Puppeteer I am able to write request interception code that looks for this URL.

const VIDEO_THUMBNAIL_PREFIX = 'https://pbs.twimg.com/tweet_video_thumb/';
let videoUrl = null;
page = await browser.newPage();
await page.setRequestInterception(true);

page.on('request', request => {

    if (request.url().startsWith(VIDEO_THUMBNAIL_PREFIX) && request.url().endsWith('.jpg')) {
        let thumbnailUrl = request.url();
        let assetId = thumbnailUrl.replace(VIDEO_THUMBNAIL_PREFIX, '')
            .replace('.jpg', '');
        videoUrl = VIDEO_URL_PREFIX + assetId + '.mp4';
    }
    request.continue();
});
await page.goto(tweetUrl);

Once I had the URL to the video thumbnail, I'm easily able to build a URL to the video so I can download it later.

Converting the Video

FFMPEG is one of the most popular command line utilities for transcoding a wide area of video, audio, and still images. It's written in C. However, like many things nowadays, you can instrument it with JavaScript. I discovered a GitHub Gist that was tackling a similar problem, converting a video into a GIF.

Using traditional command line FFMPEG, you could execute the operation with:

ffmpeg -i input_video.mp4 output.gif

Using a Nodejs library, fluent-ffmpeg, the same operation looks like:

const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path;
const ffmpeg = require('fluent-ffmpeg');
ffmpeg.setFfmpegPath(ffmpegPath);

ffmpeg('input_video.gif')
    .on('end', () => {
        // done, do something with output.gif
    })
    .on('error', err => {
        // oh noe error!
    })
    .save('output.gif');

The use of fluent-ffmpeg requires the use of another Nodejs libray, @ffmpeg-installer/ffmpeg. It installs a version of ffmpeg compatible with the underlying platform (Linux, Mac, or Windows). You then tell fluent-ffmpeg where it has been installed and it takes over from there.

The End

I've deployed GMTG to AWS Lambda. You can make HTTP GET calls to it. The value of the URL parameter should be a URL to a Tweet containing a GIF (short movies probably work as well).

https://1ehn2dwcfk.execute-api.us-east-1.amazonaws.com/dev/?url=

In this example, we will extract the GIF out of this Tweet

// Detect dark theme var iframe = document.getElementById('tweet-1194435785082187778-218'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1194435785082187778&theme=dark" }

Using the following GMTG API call:

https://1ehn2dwcfk.execute-api.us-east-1.amazonaws.com/dev/?url=https://twitter.com/ThePracticalDev/status/1194435785082187778

The GIF:

The source code has been uploaded to GitHub.

https://github.com/evanhalley/get-me-the-gif

Let me know if you have any questions by leaving a comment or on Twitter, @EvanHalley.

If you like reading articles about all the cool things you can do with Puppeteer, please sign up for my email newsletter, The PuppetHero Digest.

This article was originally posted at EvanHalley.dev

GroceryJS: JavaScript Your Groceries

Evan — Wed, 02 Oct 2019 19:37:07 +0000

Puppeteer is a JavaScript / Nodejs library that instruments Google Chrome or Chromium browers using the Chrome DevTools Protocol. Think of it as a way to write JavaScript to control every aspect of a Chrome browser. I wrote an article Getting Started with Puppeteer that walked through the process of boostrapping a Nodejs / Puppeteer app and turning a webpage into a queryable API using Puppeteer.

I originally became interested in Puppeteer because I had some inefficiences in my everyday life. One of those inefficiences was how tedious and monotonous grocery shopping is. Puppeteer was instrumental in this endeavor.

For what it's worth, grocery shopping isn't THAT bad. It can be kinda crazy when young kids are tagging along. GroceryJS has the additional advantage of being an interesting technical challenge, so here we are.

[![](/images/grocer_bot_in_action.gif)](/images/grocer_bot_in_action.gif)

Breaking Down Grocery Shopping

Shopping for groceries is a procedural activity:

You look through your fridge and pantry for things you need and make a list
You walk, bike, or drive to the store
You walk the aisles adding things to your cart as you see them
Finally you pay and bring your groceries home

This process tends to happen every week or two, for large grocery orders.

Translating this to a computer program, the primary actions are:

Read items from a list
Search and add items to your cart
Prepare the cart for checkout & checkout

With this in mind, I built GroceryJS.

Breaking Down the Script

GroceryJS is broken up into several pieces of code after plenty of experimentation:

Source

The source is the data backend, it's where the grocery list is held. It also is the holding place for the results of a grocery store run. For, GroceryJS, I started with a text (YAML) file, then transitioned to a Google Sheet. I found the Google Sheet to be something that is accessible from everywhere, desktop & mobile, without needing a bunch of UI. Google provides a pretty robust set of Nodejs libraries you can use to interact with the Google Drive and Sheet APIs.

My grocery list is stored in the first 2 columns and first 50 rows (arbitrary) in the first sheet.

[![](/images/groceryjs_grocery_list.jpg)](/images/groceryjs_grocery_list.jpg)

The sheets-grocery-source.js has two primary actions after initializing a few objects.

The first gets the grocery list, getGroceryList().

async getGroceryList() {
    let spreadsheetId = this._spreadsheetId;
    let sheetsService = this._sheetsService;

    return await new Promise((resolve, reject) => {
        sheetsService.spreadsheets.values.get({
                spreadsheetId: spreadsheetId,
                range: 'A1:C50'
            }, (err, result) => {
            if (err) {
                reject(err);
            } else if (result && result.data && result.data.values) {
                let items = [];

                for (let i = 1; i < result.data.values.length; i++) {
                    let value = result.data.values[i];
                    items.push({ name: value[0], quantity: value[1] });
                }
                resolve(items);
            } else {
                resolve([]);
            }
        });
    });
}

The second adds the results of a particular shopping run to the Sheet, addShoppingResults().

async addShoppingResults(title, sheetId, results) {
        let sheetsService = this._sheetsService;
        let spreadsheetId = this._spreadsheetId;

        return new Promise((resolve, reject) => {
            let requests = [];
            let idx = 1;

            // convert results to an array we can write
            let data = [];

            let headers = [
                { userEnteredValue: { stringValue: 'Requested' } },
                { userEnteredValue: { stringValue: 'Item' } },
                { userEnteredValue: { stringValue: 'Price' } },
            ];
            data.push({ values: headers });

            for (let i = 0; i < results.length; i++) {
                let result = results[i];
                let row = [];
                row.push({ userEnteredValue: { stringValue: result.requested } });

                if (result.result) {
                    row.push({ userEnteredValue: { stringValue: result.result.title } });
                    row.push({ userEnteredValue: { numberValue: result.result.price } });
                }
                data.push({ values: row });
            }

            // add the sheet
            requests.push({ 
                addSheet: {
                    /* removed for brevity's sake */
                }
            });

            // updateCells request
            requests.push({ 
                /* removed for brevity's sake */
            });

            // auto size things
            requests.push({
                /* removed for brevity's sake */
            });

            // execute the batch update
            sheetsService.spreadsheets.batchUpdate({
                spreadsheetId: spreadsheetId,
                resource: { requests: requests }
            }, (err, result) => {

                if (err) {
                    reject(err);
                } else {
                    resolve(`https://docs.google.com/spreadsheets/d/${spreadsheetId}/edit#gid=${sheetId}`);
                }
            });
        });
    }

Shopper

The Shopper contains all the code and actions that make up a successful trip to the grocery store. It's built on top of utility library I wrote called puppet-helper.js.

Puppet Helper

The Puppet Helper contains all of things needed to interact with a modern web app, like clicking a button, given a CSS selector:

async clickButton(selector, clickCount = 1) {
    this.assertPageOpen();
    let button = await this._page.$(selector);

    if (button) {
        await button.click({ clickCount: clickCount });
    } else {
        throw new Error(`Unable to click ${selector}`);
    }
}

Or getting text from an HTML element:

async getTextFromElement(element) {
    this.assertPageOpen();
    return await this._page.evaluate(el => el.innerText, element);
}

As you can imagine, you can string enough of these actions together to mimic a user, shopping for groceries online.

The Lowes Shopper

More and more grocery stores offer online shopping services on the internet, allowing customers the convenience of shopping from their computer, tablet, or mobile phone. We shop at Lowes Foods, a North Carolina-based grocery store chain. Lowes Foods offers an online shopping service, Lowes Foods To Go. For $49-$99 annually (or $4 to $5 per order), you can order your groceries using their web app. Once you place your order, a Lowes Foods employee will shop your order and call you when they are finished (or if they had any questions). When the order is complete, you can pick it up or have it delivered.

Dad/mom-hack, shop for groceries online if you can help it. Taking young kids to the grocery store get's wild! 😅

I spent a lot of time examining the front-end code for Lowes Foods To Go. I have determined that it is an Angular based progressive web app. It lends itself very well to automating with Puppeteer. Using puppet-helper.js, I can string together a few methods to get several things done.

Searching for groceries

async searchImpl(query) {
    this._logger.info(`Searching for ${query}`);
    let productDivs = null;
    await this._puppetHelper.clearText('#search-nav-input');
    await this._puppetHelper.enterText('#search-nav-input', query);
    await this._puppetHelper.wait(SHORT);
    await this._puppetHelper.clickButton('#search-nav-search');
    await this._puppetHelper.wait(MID);
    // body > div:nth-child(5) > div > div > div.content-wrapper > div > lazy-load > ol
    let resultsDiv = await this._puppetHelper.getElement('ol.cell-container');

    if (resultsDiv) {
        productDivs = await this._puppetHelper.getElementsFromParent(resultsDiv, '.cell.product-cell');
    }
    return productDivs;
}

Logging in

async login(email, password) {
    this._logger.info(`Logging into account ${email}...`);
    await this._puppetHelper.goToUrl(SHOPPING_URL);
    await this._puppetHelper.clickButton('#loyalty-onboarding-dismiss');
    await this._puppetHelper.wait(SHORT);
    await this._puppetHelper.clickButton('#shopping-selector-parent-process-modal-close-click');
    await this._puppetHelper.wait(SHORT);
    await this._puppetHelper.clickButton('#nav-register');
    await this._puppetHelper.wait(SHORT)
    await this._puppetHelper.enterText('#login-email', email);
    await this._puppetHelper.wait(SHORT)
    await this._puppetHelper.enterText('#login-password', password);
    await this._puppetHelper.wait(SHORT)
    await this._puppetHelper.clickButton('#login-submit');
    await this._puppetHelper.wait(XLONG);
}

Showing and emptying your cart

async showCart() {
    this._logger.info(`Opening the shopping cart...`);
    await this._puppetHelper.clickButton('#nav-cart-main-checkout-cart');
    await this._puppetHelper.wait(MID);
}

async emptyCart() {
    this._logger.info(`Emptying cart...`);
    await this.showCart();
    await this._puppetHelper.clickButton('#checkout-cart-empty');
    await this._puppetHelper.wait(NANO);
    await this._puppetHelper.clickButton('#error-modal-ok-button');
    await this._puppetHelper.wait(MINI);
}

Putting it Together

With all the pieces mentioned, I can have GroceryJS, prepare a shopping cart full of groceries. When it's done, it sends me an email with a link to the cart (so that I can quickly checkout) and a link to the Google Sheet for tracking purposes.

(async () => {
    let shopper = null;

    try {
        let sheetSource = new SheetGrocerySource(logger, credential.client_email, credential.private_key, config.source.sheetId);
        await sheetSource.init();
        let list = await sheetSource.getGroceryList();

        // login and create a blank slate to shop
        shopper = new LowesShopper(logger);
        await shopper.init(config.shopper.headless);
        await shopper.login(config.shopper.email, config.shopper.password);
        await shopper.emptyCart();

        // do the shoppping
        let shoppingResults = [];
        for (let i = 0; i < list.length; i++) {
            let requestedItem = list[i];
            let shoppedItem = await shopper.addItemToCart(requestedItem.name, requestedItem.quantity);
            shoppingResults.push({ requested: requestedItem.name, result: shoppedItem });
        }

        // notify 
        let dateStr = moment().format('MMMM Do YYYY @ h:mm a');
        let title = `Shopping Trip on ${dateStr}`;

        let urlToCart = 'https://shop.lowesfoods.com/checkout/cart';
        let urlToSheet = await sheetSource.addShoppingResults(title, moment().unix(), shoppingResults);
        let emailBody = `
            <span><b>Shopping Cart:</b> ${urlToCart}</span><br />
            <span><b>Shopping Results:</b> ${urlToSheet}</span>`;
        let mailOptions = {
            service: config.email.sender.service,
            user: config.email.sender.email,
            password: config.email.sender.appPassword
        };
        mailUtil.sendEmail(config.email.recipeint.sender,
            config.email.recipeint.email,
            title, emailBody, mailOptions);
    } catch (e) {
        logger.error('Error while shopping', e);
    } finally {

        if (shopper) {
            await shopper.shutdown();
        }
    }
})();

Conclusion

So that's it. GroceryJS isn't done yet. The real work is actually in the details, like the algorithm for adding groceries from the search results to your cart. Lowes Foods To Go has it's own search algorithm for determining the relevance of a result to a search. In many cases their algorithm will not match expectations, but it can be augmented:

Should GroceryJS prefer groceries that are on sale?
Should GroceryJS prefer groceries for a specific brand?
Should GroceryJS prefer groceries I've purchased before?

There are a ton of calculations we make every time we grocery shop that I didn't realize until I started working on GroceryJS.

Hit up the GitHub repository for all the source code. Be sure to read the README file before jumping in.

github.com/emuneee/groceryjs

I'm really interested in hearing what interesting things folks are using Puppeteer for.