DEV Community: Joel Griffith

Announcing The Browser Conference

Joel Griffith — Mon, 11 Sep 2023 18:46:07 +0000

We've thought about doing this for years… with the explosive growth of providers and the need for developers to have browser automation as a core skill set, we wanted to create an industry conference that is all about sharing the most cutting edge technologies and best practices for browser automation. The conference will focus on libraries such as Puppeteer, Selenium, Webkit, Playwright and many more. We will also include speakers from both tool providers as well as practitioners that are in the trenches day-to-day crafting unique solutions to the hardest problems.

The first year is virtual and 100% free to join. We would greatly appreciate you taking time out of your day to support this effort. We have worked together with Oxylabs to bring this event to life and have an amazing speaker line-up. You can sign up here: https://www.accelevents.com/e/the-browser-conference-23

About the Event

While the first year is going to be smaller, we hope to partner with industry leaders and turn this into a much larger event in the coming years. If you want this to become something, help us out by signing up and showing your interest in learning more about what Browser Automation can do for your career and company.

Here is an overview of the amazing speakers we will have at The Browser Conference:

Fara Ashiru: Fara Ashiru Jituboh is the co-founder and CEO/CTO of Okra, Inc., a fintech Nigerian-based startup that allows for the exchange of real-time financial information between customers, applications, and banks. Fara is a software engineer and entrepreneur, who is fluent in over 20 programming languages.

Dario Kontratiuk: Dario Kondratiuk has been a web developer since 2001. He won the Microsoft MVP (most valuable professional) award in 2020 for his contributions to the developer’s community. Dario has been working with Puppeteer since the beta versions, back in 2017. He is the author of Puppeteer-Sharp, a Puppeteer port to .NET, and Playwright-Sharp, a Playwright port to .NET. He writes about web automation in his blog https://www.hardkoded.com/, and he’s active on Stack Overflow.

Ben Fellows: Ben is the Founder and CEO of Loop where he helps startups & enterprise organizations run successful test automation teams. Ben is an evangelist of Playwright and a realist when it comes to continuous development. Overall fan of QA & promoting the industry.

Joel Griffith: Founder and CEO of browserless.io, Joel has seen just about every kind of platform, library and headless browser binary this is. These days he spends his time operationalizing and writing about best practices with regards to headless automation.

Final Tidbits

Can't join? No worries! Sign-up and we'll record the event for future enjoyment. A collectors item for years to come!
Can't join won't join? No problem! If you wanted to tweet or tell some friends about it, that'd help tremendously.
Can't join won't join won't X-it? That's ok too, thanks for reading!

Vercel + Puppeteer

Joel Griffith — Mon, 17 Apr 2023 17:28:09 +0000

Vercel is the platform for frontend developers, providing the speed and reliability innovators need to create at the moment of inspiration. At the time of this writing, Vercel powers more than 35+ frameworks. For the purposes of this post, however, we’ll be mostly focusing on NextJS. Puppeteer is a NodeJS library that allows developers to programmatically control a web-browser. As of version 19, Puppeteer supports both Firefox and Chrome.

Combining these technologies is the task for today! We’ll use Puppeteer to generate a PDF of a website and Vercel.JS to host it. When done locally this is a pretty straightforward task, however, you'll likely run into a few issues regarding bundle size and memory limits once you deploy your project to Vercel’s cloud. Below we detail how you can get around these issues easily with browserless. If you want to see code right away, feel free to check out our example project here.

Best practices and setting up

We're using Next.JS as our main technology since it's incredibly effective and popular for Vercel deployments.
You should be somewhat familiar with how environment variables work in NextJS to properly store secrets. Feel free to read more here.
For the purpose of this blogpost we'll create a basic route that generates a PDF of a URL. Feel free to alter this, but it'll be enough to get you started!

Setting up Puppeteer for Vercel

To get started, create a new Next.js application through the CLI:



$ npx create-next-app@latest

This command will create your basic "Hello World" app and then exit. For NextJS apps in particular, you'll likely have a "pages" directory with an "api" folder as well. This is where we'll be spending most of our time since our functionality is created inside the NodeJS environment.

Once your application is set up then we'll also install Puppeteer. We'll opt for the 'puppeteer-core' module to avoid the first issue in Vercel: the 50MB file-size limit. Puppeteer by default ships with headless Chrome bundled together, which is easily over 50MB in size. By using 'puppeteer-core' we can get around this by only having puppeteer's code installed and not the browser binary, which shrinks this module down to about 4MB. To install Puppeteer for Vercel, simply do the following:



$ npm i puppeteer-core

Writing our Puppeteer API

Next we'll set up our API handler to generate these PDFs. By default NextJS adds a "/api" path prepended to whatever our route's name is. For this post, we'll have a "pdf.ts" route in the "pages/api" directory. We'll explain everything below but here's the code to get started:



// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next'
import puppeteer from 'puppeteer-core'

type Json = {
  message: string
}

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse<Json | Buffer>
) {
  const { searchParams } = new URL(
    req.url as string,
    `http://${req.headers.host}`
  )
  const url = searchParams.get('url')

  if (!url) {
    return res
      .status(400)
      .json({ message: `A ?url query-parameter is required` })
  }

  const browser = await puppeteer.connect({
    browserWSEndpoint: `wss://chrome.browserless.io?token=${process.env.BLESS_TOKEN}`,
  })

  const page = await browser.newPage()
  await page.setViewport({ width: 1920, height: 1080 })
  await page.goto(url)

  return res.status(200).send(await page.pdf())
}

To get started we import some type information for NextJS and then import the puppeteer module from puppeteer-core. The typings aren't required if you're using plain NodeJS, so feel free to delete those in that case.

Next is a "Json" type to describe our responses back to the client. Since our handler can convert any URL into a PDF we need to make sure users supply one! Feel free to alter this block to fit your use-case or needs with puppeteer.

Finally we have our route file. This handler will check to see if there's a "?url" query-parameter for the site to convert to a PDF. If it's missing we'll return a 400 message and a response indicating that it's required. After that we simply connect to a live browser on browserless, create a page, and generate that PDF.

Making puppeteer work on Vercel with browserless

The reason we're able to get puppeteer and Chrome working on Vercel is by avoiding having to download and run Chrome inside of our Vercel app. Separating your puppeteer code from Chrome is actually a great best practice as it cleanly separates Chrome from your application code.

In order to connect to browserless and use headless Chrome you'll need to get an API token. We offer free accounts with no credit card required here. Once you have your API key, you'll want to create an env.local file with it:



# Browserless Token
BLESS_TOKEN=YOUR-API-KEY-HERE

Now that that's set up, let's start our application and visit the route to see if it works properly. From the root of the project, run the following command. You'll want to restart this as any environment variable changes do require a restart of the dev server:



npm run dev

Then, once it's up and ready, go to http://localhost:3000/api/pdf?url=https://www.browserless.io/docs/start. You should get a PDF file downloaded!

Deploy your Puppeteer Code on Vercel

Once we've verified everything is working properly, the next step is getting our Puppeteer + NextJS application onto Vercel. There's two ways of doing this: either the Vercel CLI or via Github. For now we'll use GitHub as a deployment method. You'll want to create a new app on Vercel by clicking "Create a New Project":

This will prompt you for a few options, but we'll want to deploy from GitHub. After adding access here's the permissions and screen grab of our GitHub selection:

Vercel should automatically recognize that this is a NextJS application, which means we won't have to inform it how to build or bundle Puppeteer, npm run build, or anything else. We will want to input our browserless token, however, as it's necessary for browserless to work:

Once in you'll click "Add" then "Deploy." After a few seconds your page should load and display their "Hello World" Screen since we never altered it:

From here you can edit the page's URL and add "/api/pdf?url=https://example.com" to it. This will verify that our Puppeteer code on Vercel works swimmingly!

Closing thoughts

With Browserless, it's easy to use a headless browser in serverless environments like Vercel. With browserless you can easily get past any file-size limits, memory limits, and have a great developer experience without the headache. Feel free to sign-up for a hosted account and get started today, or check out our example repo here.

7 Tips to Making Your Puppeteer Scripts More Bulletproof

Joel Griffith — Fri, 21 Oct 2022 18:16:05 +0000

Puppeteer comes out of the box with a lot of great features and makes it dead simple to get started automating your web workflows. In fact, you can likely get everything up and running locally within minutes. Because of its fairly simple API you can likely automate whatever it is you're doing with relatively few lines of code.

But, like any other piece of software, the devil is in the details. Since puppeteer (and playwright) are both heavily async-heavy, there's a lot of opportunity for things to go wrong. If you're aggregating data or automating sites without APIs then the ever-changing nature of the web can easily show up to ruin the fun. To be clear: it's just as easy for things to go wrong with puppeteer as it is for them to go right. A lot of times it's even easier 😢

I'm no stranger to puppeteer, or even web automation, and have even written my own headless web drivers in the past. It's been the longest standing passion of mine since I started programming over 10 years ago. With that said, today I'd like to go over a list of things every developer should do to make their puppeteer scripts more ironclad and graceful. It won't solve all your problems, but it'll give you enough situational awareness to make the issue(s) a lot easier to diagnose and fix. With that, let's jump in!

Change your page.goto calls

Puppeteer will default timeout after 30 seconds in a page.goto call when the specified "load" event doesn't happen. Even though there's some great loading events you can key off of (like networkidle2 events) we generally recommend using the domcontentloaded option when you know what you're doing. Especially if you're after a particular page element or network call to wait for:

// DON'T do this:
// This will wait for network requests to be idle for 2 seconds before proceeding.
await page.goto('https://www.example.com', { waitUntil: 'networkidle2' });
await page.waitForSelector('h1');

// Do this instead:
// This will navigate to the page and *immediately* begin waiting for the h1 selector
// once the page's initial HTML is returned. Often much quicker.
await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1');

This essentially is just optimizing your puppeteer code to run as fast as possible and not let slow network requests get in the way. Often times we (browserless.io) will see a network request NOT respond, which causes your whole script to break as goto never completes successfully within networkidle2.

When shouldn't you use this? Well, the only time you really shouldn't is when you're dealing with the unknown. If you're not keenly aware of the pages layout, network calls, or DOM selectors, then doing a networkidle0 or networkidle2 is about as good as you can get to when the page is "interaction ready."

Log your failed network calls

Lots of times we see that it's a simple network call that fails an entire workflow. This can be differences between running puppeteer locally versus on a cloud server, a site being down, or even networking issues elsewhere. Though the internet generally works, it still relies on physical devices being online and running, which isn't always the case.

The best thing you can do is to simply log failed requests and attempt to proceed on:

// DON'T do this: there's nothing here catching _any_ network errors!
await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1');

// Do this instead:
page.on('response', (res) => {
  if (!res.ok()) {
    console.error(`Non-200 response from this request: [${req.status()}] "${req.url()}"`);
  }
});

await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1');

While we're at it, we might as well wire up some kind of page error handler as well and get a utility in place for these kinds of logs/warnings.

// Drop in your favorite logging technology here. We're using console.error to illustrate.
const logWarning = (message) => console.error(message);

// Page errors here might be deal-breakers
page.on('pageerror', (err) => {
  logWarning(`Page error emitted: "${err.message}"`);
});

page.on('response', (res) => {
  if (!res.ok()) {
    logWarning(`Non-200 response from this request: [${req.status()}] "${req.url()}"`);
  }
});

await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1');

Monitor the browser process

Depending on how you're connecting to the browser, there are a few things here that can help us make sure our scripts are resilient. The first thing is doing a listener on the disconnected event as it indicates that either the browser itself has crashed or we've been disconnected. Doing this is pretty straightforward:

browser.once('disconnected', () => logWarning(`Browser has closed or crashed and we've been disconnected!`));

Now combine that with our prior code snippets for the following:

const logWarning = (message) => console.error(message);

browser.once('disconnected', () => logWarning(`Browser has closed or crashed and we've been disconnected!`));

const page = await browser.newPage();

page.on('pageerror', (err) => {
  logWarning(`Page error emitted: "${err.message}"`);
});

page.on('response', (res) => {
  if (!res.ok()) {
    logWarning(`Non-200 response from this request: [${req.status()}] "${req.url()}"`);
  }
});

await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('h1');

Intermission

As you might have noticed, we've started to add a lot of listeners and other handlers! It's starting to become a bit un-wieldy from a best-practices standpoint, and for the most part, all of this code is essentially just boilerplate. To say this another way: all of our code here has almost nothing to do with the automation side of puppeteer. Because of this, we should start thinking about making a higher-order class that abstracts all of this logic away and lets our business-oriented code (the code that's doing the actual automation) do its thing.

Let's define two modules: one to handle the events and logging/monitoring, and the second that just holds our business logic and consumes the first module. This also makes it easier for us later if we need to add more scripts as we've abstracted away all the management aspects of puppeteer into a reusable module. I'll illustrate this in TypeScript since it better aligns with our goal of having a more deterministic puppeteer experience.

// puppeteer-helper.ts
import { Browser, Page } from 'puppeteer';

export class PuppeteerHelper {
  private page?: Page;

  static log(...messages: string[]) {
    // Use your logging library here
    console.warn(...messages);
  }

  private disconnectListener = () => {
    PuppeteerHelper.log(
      `Browser has closed or crashed and we've been disconnected!`
    );
  };

  constructor(private browser: Browser) {
    browser.once('disconnected', this.disconnectListener);
  }

  public close = async () => {
    this.browser.off('disconnect', this.disconnectListener);
    this.browser.close();
  }

  public newPage = async () => {
    const page = await this.browser.newPage();

    page.on('pageerror', (err) => {
      PuppeteerHelper.log(`Page error emitted: "${err.message}"`);
    });

    page.on('response', (res) => {
      if (!res.ok()) {
        PuppeteerHelper.log(`[${req.status()}]: "${req.url()}"`);
      }
    });

    // Capture the page so we can track stuff later
    this.page = page;

    return page;
  }
}

Now, with that out of the way we can simply incorporate this easily into our simple script:

import puppeteer from 'puppeteer';
import { PuppeteerHelper } from './puppeteer-helper.ts';

(async() => {
  const browser = new PuppeteerHelper(await puppeteer.launch());
  const page = await browser.newPage();

  await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('h1');
  await browser.close();
})();

Perfect! Now that we've got a bit of a management layer in place, let's move on.

Avoiding 'file://' requests

More of a security precaution here, but we really (ideally) want to avoid chrome making calls to the local file system if we can help it. That, plus there's a slew of IP addresses that most cloud providers use to read hosting metadata that we also want to firewall away from Chrome.

I want to reiterate how important it is that we NEVER allow file-system access as it's crucial that things like /etc/passwd remain hidden. Unless you want someone taking over your cloud machines, it's best to add this right away.

static: string[] urlDeny = [
  'file://',
  'http://169.254',
  'https://169.254',
  '169.254'
];

static: string[] ipDeny = [
  '169.254',
  '192.168',
  '0.0.0.0',
];

// Later, let's use these and make sure they get reject right away:
public newPage = async () => {
  const page = await this.browser.newPage();

  page.on('request', (req) => {
    if (PuppeteerHelper.urlDeny.some((url) => req.url().startsWith(url))) {
      PuppeteerHelper.log(`Blocking request and exiting: "${req.url()}"`);
      this.close();
    }
  });

  page.on('response', (res) => {
    const responseUrl = res.url();
    const remoteAddressIP = res.remoteAddress().ip;

    if (!res.ok()) {
      PuppeteerHelper.log(`[${res.status()}]: "${res.url()}"`);
    }

    if (responseUrl && PuppeteerHelper.urlDeny.some((url) => responseUrl.startsWith(url))) {
      PuppeteerHelper.log(`Blocking request URL and exiting: "${responseUrl}"`);
      this.close();
    }

    if (remoteAddressIP && PuppeteerHelper.ipDeny.some((ip) => remoteAddressIP.startsWith(ip))) {
      PuppeteerHelper.log(`Blocking request IP and exiting: "${responseUrl}"`);
      this.close();
    }
  })

  page.on('pageerror', (err) => {
    PuppeteerHelper.log(`Page error emitted: "${err.message}"`);
  });

  this.page = page;

  return page;
};

With any project there can be security things we'll need to come up along the way, so it's best to always treat any technology's security as a permanent work-in-progress. Here we can have some assurances that things will get killed before network responses back if the resulting IP address or URL is sensitive and we're dealing with a bad actor.

While we're on the subject of security, let's do one more thing...

Separate Chrome from your application

It's a common best-practice to generally separate concerns in software. A good example is ensuring your application code runs on separate hardware than, say, your database. This gives the programmer a lot of flexibility on how to scale but also allows tighter controls on things like firewalls and connection pools for your database. Since we're dealing with a blackbox (Chrome) we should also exercise caution in working with it.

Instead of launching Chrome locally with puppeteer let's start Chrome elsewhere and connect to it. browserless makes this trivial as it's a sandboxing layer of sorts for Chrome, meaning you can run it in an ephemeral fashion totally segregated from your app code and data. This, combined with the network helper up above, makes it easy to have more confidence in your deployment.

I'll leave the docker aspects of running browserless as an exercise to you (we've got a lot of resources on that), but once you have it running (we assume localhost and port 3000 below), simply connect to it:

// DON'T launch locally!
const browser = new PuppeteerHelper(await puppeteer.launch());

// DO replace launch with connect:
const browser = new PuppeteerHelper(await puppeteer.connect({ browserWSEndpoint: 'ws://localhost:3000' }));

Now that we've got Chrome all situated and away from our codebase, let's do one last thing in the case of failures.

Take a picture, it lasts longer!

One of the MOST helpful things you can do is screenshot the webpage in the case of a failure. This, combined with network logging, can more easily give you clues as to what's going on. Scraping and automating the web isn't as deterministic as we'd like, however giving yourself as much context and evidence can make experience a whole lot better.

In order to do that and save on same network bandwidth, let's just do a quick JPEG image at 50% quality. Setting dimensions at 1024x768 should also keep sizing matters down as well, but you can do whatever you'd like. We'll adjust our 'close' method to capture a last screenshot so we can also keep track of page changes over time as well. This will help you immensely if you get notified at 2AM that something's wrong since you'll have a photo in your mind of the page as it should be

public close() {
  this.browser.off('disconnect', this.disconnectListener);
  if (this.page) {
    // screenshot here is a base64-encoded string of the jpeg file.
    // We don't do anything here with it since this greatly depends on your tools and tooling.
    // We'll catch issues here and just return `null` in cases of total failure
    const screenshot = await this.page.screenshot({ type: 'jpeg', quality: 50, fullPage: true }).catch(() => null);
  }
  this.browser.close();
}

In order for this to take effect, we'll need to update our business-level code to catch errors and attempt this screenshot. This, too, is pretty straightforward:

import puppeteer from 'puppeteer';
import { PuppeteerHelper } from './puppeteer-helper.ts';

(async() => {
  const browser = new PuppeteerHelper(await puppeteer.connect({ browserWSEndpoint: 'ws://localhost:3000' }));

  try {
    const page = await browser.newPage();
    await page.goto('https://www.example.com', { waitUntil: 'domcontentloaded' });
    await page.waitForSelector('h1');
    await browser.close();
  } catch (err) {
    PuppeteerHelper.log(`Error running script: ${err.message}`);
  } finally {
    browser.close();
  }
})();

Now that there's a visual history of what the page looks like over time, we can begin to paint a better understanding of how things will change over time and feel more confident in our automation. Putting it all together, here's the module we've written:

// puppeteer-helper.ts
import { Browser, Page } from 'puppeteer-core';

export class PuppeteerHelper {
  private page?: Page;

  constructor(private browser: Browser) {
    browser.once('disconnected', this.disconnectListener);
  }

  // Add some static url's and IPs to our class:
  static urlDeny: string[] = [
    'file://',
    'http://169.254',
    'https://169.254',
    '169.254'
  ];

  static ipDeny: string[] = [
    '169.254',
    '192.168',
    '0.0.0.0',
  ];

  static log(...messages: string[]) {
    // Use your logging library here
    console.warn(...messages);
  }

  private disconnectListener = () => {
    PuppeteerHelper.log(
      `Browser has closed or crashed and we've been disconnected!`
    );
  };

  public close = async () => {
    this.browser.off('disconnect', this.disconnectListener);
    if (this.page) {
      // screenshot here is a base64-encoded string of the jpeg file.
      // We don't do anything here with it since this greatly depends on your tools and tooling.
      // We also catch and return null here in case something fails
      const screenshot = await this.page.screenshot({ type: 'jpeg', quality: 50, fullPage: true }).catch(() => null);
    }
    this.browser.close();
  };

  public newPage = async () => {
    const page = await this.browser.newPage();

    page.on('request', (req) => {
      if (PuppeteerHelper.urlDeny.some((url) => req.url().startsWith(url))) {
        PuppeteerHelper.log(`Blocking request and exiting: "${req.url()}"`);
        this.close();
      }
    });

    page.on('response', (res) => {
      const responseUrl = res.url();
      const remoteAddressIP = res.remoteAddress().ip;

      if (responseUrl && PuppeteerHelper.urlDeny.some((url) => responseUrl.startsWith(url))) {
        PuppeteerHelper.log(`Blocking request URL and exiting: "${responseUrl}"`);
        this.close();
      }

      if (remoteAddressIP && PuppeteerHelper.ipDeny.some((ip) => remoteAddressIP.startsWith(ip))) {
        PuppeteerHelper.log(`Blocking request IP and exiting: "${responseUrl}"`);
        this.close();
      }
    })

    page.on('pageerror', (err) => {
      PuppeteerHelper.log(`Page error emitted: "${err.message}"`);
    });

    page.on('response', (res) => {
      if (!res.ok()) {
        PuppeteerHelper.log(`[${res.status()}]: "${res.url()}"`);
      }
    });

    this.page = page;

    return page;
  };
}

Feel free to treat this as a starting place of sorts to contain your management code for puppeteer. There's a lot you can do here, for instance tracking DOM changes over time or even all network requests. While these are definitely suggestions we make it's entirely up to your use-case on what is or isn't interesting to note.

Where do we go from here?

This is just the starting point! We haven't talked about a lot of things you'll likely need to consider when running puppeteer:

How do we ensure we're not running too much traffic?
Can we make our scripts more reliable?
Tools and other things to consider?! Be sure to follow us on our blog to get the best practices and more. If you're thinking about running your own Chrome cluster of instances, we'd love to hear your thoughts and offer some help!

I'm the creator of browserless.io -- let's talk about starting your own company

Joel Griffith — Fri, 19 Apr 2019 16:57:09 +0000

Almost 19 months ago I started a business called browserless.io. Mostly came from my attempts at writing a headless driver and scaling it for production. There's been a lot of challenges and wins, so much so that there's a wealth of information now that I'd like to get out.

But I really want this to be about what you're building. What's keeping you from building it? Is it not ready yet? Feel free to ask questions or get ideas!