Browser Automation With Node Red

Node Red is a fantastic, open source, "low code" automation framework. I have been using it for years as the automation engine for Home Assistant as well as a few other tasks. If you're coding-inclined, you can use Node Red's function nodes to run whatever JavaScript you want. Today I'm going to show you how to use these function nodes to run browser automation tasks using Playwright or Puppeteer.

Note: I'm using Docker to run Node Red and Browserless as a browser instance I can access via API from any machine on my network, but similar steps can be followed if you're running Node Red locally or installed on a server another way.

Step 1: Install the Playwright/Puppeteer module

Navigate to the Node Red data directory. On most systems that defaults to $HOME/.node-red, and in the Docker container it's /data. If you're running Node Red in Docker, you can connect by doing docker exec -it node-red-container-name /bin/bash or if you have NodeJS installed on the host, you can just navigate to wherever the Node Red data directory is mapped.

Now run the appropriate command for the tool(s) you want to install:

• Playwright: npm i playwright
• Playwright Core: npm i playwright-core (if you're using Browserless)
• Puppeteer: npm i puppeteer
• Puppeteer Core: npm i puppeteer-core (if you're using Browserless)

Step 2: Add the tools to Node Red's global context

Inside Node Red's data directory should be a file named settings.js. Open that in your editor of choice and find the functionGlobalContext property. Add whatever tool(s) you installed and then restart Node Red so that the changes are picked up.

functionGlobalContext: {
    env: process.env,
    playwright: require("playwright-core")
    // os:require('os'),
    // jfive:require("johnny-five"),
    // j5board:require("johnny-five").Board({repl:false})
}

Note: I have a Browserless instance running in Docker, so I just installed playwright-core since I don't need to install the browsers typically installed with Playwright.

Step 3: Use your tool(s) in Function nodes

const playwright = global.get('playwright');

(async () => {
    node.warn('Started');
    const browser = await playwright.chromium.connect({
        wsEndpoint: "ws://browserless:3000/playwright"
    });
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.goto('https://iknowwhatyoudownload.com/en/peer/');
    node.warn(page.url());
    const downloads = await page.$$('td a');
    msg.payload = downloads.length;
    await browser.close();
    node.send(msg);
})();

Note: if not using Browserless you would just instantiate a browser with const browser = await playwright.chromium.launch(); or use firefox or webkit.

For those that may be new to Async/Await, it allows for cleaner asynchronous programming without callbacks. Javascript doesn't allow the await keyword unless you're inside an async function, so that's why I wrapped my code in:

(async () => {
  //...
})();

Also, since we're using asynchronous code, we have to use node.send(msg); instead of return msg;.

When creating a new browser automation, you're definitely better off testing things on a non-headless machine. When creating a browser instance for testing, you can run it non-headless and in "slowmo" to allow you to watch what is happening. Here's how to do it in Playwright:

const browser = await chromium.launch({
  headless: false,
  slowMo: 100
});

Adding other NPM modules

You can add other npm modules using this same process to make your life a little easier when writing custom functions and is much easier than developing a custom module. Some potentially handy modules might be: date-fns, mathjs, cheerio, execa.