A better puppeteer.evaluate

Execute JavaScript with puppeteer

To execute JavaScript code on your puppeteer instance, you can use the provided evaluate method. e.g.

const title = await page.evaluate(() => document.title);

This returns the document title from the puppeteer instance back to the node process.

But the way this works is actually quite limited and error prone.
This already errors at runtime:

const selector = `p`;
const paragrapahs = await page.evaluate(() => {
    const elements = document.body.querySelectorAll(selector); //runtime error
});

(╯°□°)╯︵ ┻━┻

Why?

Because the callback that you provide in the evaluate method gets serialized.
Meaning by the time it arrives at the browser it will have lost all information about closures/imports etc. What a bummer.

If you're like me and want to separate your evaluate code to an external file, use node modules etc, you're out of luck. It just doesn't work.

Worse, you will get no indication that this will not work, the IDE says all good, go ahead run it, only to be hit with a runtime error :(

To be fair though, the above fairly simple example can be made to work, as you can also specify arguments:

const selector = `p`;
const paragrapahs = await page.evaluate(selector => {
    const elements = document.body.querySelectorAll(selector);
}, selector /* <- pass selector as argument */);

Still, this is very cumbersome to work with, as you just expect closures to be available, after all, that's how JavaScript works.

A better puppeteer evaluate

For that reason I published a little library(my first) that lets me do just that:

Aptly named puppeteer-evaluate2, it allows me to write my puppeteer callback code like any other JavaScript code, where closures/imports are available.

It's signature looks like this:

export function evaluate2<T = any>(page: puppeteer.Page, jsPath: string): Promise<T>

You just pass in your puppeteer page object as the 1st parameter and the path to your JavaScript file as the 2nd parameter.

The library then does the rest, it will create a JS bundle, taking the 2nd parameter as the entry point. The finished bundle is then passed on to the actual page.evaluate method. Closures and imports are then available.

Here's an example importing lodash:

//code.js
export default function() {
    const chunk = require("lodash/chunk");

    return chunk([1, 2, 3, 4], 2);
}

let response = await evaluate2(page, `./code.js`);
console.log(response); //[[1, 2], [3, 4]]

voila 🎉

The only requisite is that your entry file must export a default function!
Thanks to the TypeScript API there are checks in place that tell you if you're missing a default function export 💪

Creating a bundle, entry file,.. this sounds a lot like webpack, and you're right! It is actually using the webpack API under the hood to create the bundle, but in memory and on the fly.

Final

If you're asking why did I import lodash with require and not via import? Because I still consider it more of a POC, hence it only has a beta version right now, as I also want to have TypeScript support. But I wanted to share it early to get some feedback, so let me know what you think :)