Run Playwright tests with hardware acceleration on a GPU-enabled EC2 instance with Docker support

#ec2 #gpu #chrome #playwright

Introduction

As part of our development workflow, I wanted to run Playwright tests in headless mode inside a Linux environment. I also needed to run it inside a Docker container. Another thing I wanted was for the EC2 instance to have GPU support so the tests can run with hardware acceleration.

As far as I can see, the way Google Chrome developers chose to support hardware acceleration under Linux is through Vulkan (here and here)
According to Nvidia, there's no official support for Vulkan inside Docker. Although it seems that FAQ hasn't been updated because I was able to find a Docker container with Vulkan support here.

The ingredients of our "recipe" are the following:

GPU enabled EC2 instance
Nvidia drivers
Docker Engine
Nvidia Container Toolkit
Vulkan enabled Docker container

Steps

Start by launching an EC2 instance with the following features:

Instance type: g4dn.xlarge
AMI: ami-058165de3b7202099 (Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2022-06-09)¹
Storage: At least 20 GiB

SSH to the instance and refresh the system:
sudo apt-get update -y && sudo apt upgrade -y
Don't forget to reboot after the upgrade finishes

The installation of the Nvidia drivers is documented here. I went with installation option 2 (Public NVIDIA drivers). As I'm using a G4dn instance, I did a search using the following parameters:
As you can see in the image, the latest version at the time of this writing is 515.65.01. Now we can download and install this version:

wget https://us.download.nvidia.com/tesla/515.65.01/nvidia-driver-local-repo-ubuntu2204-515.65.01_1.0-1_amd64.deb (To get the link, select the driver version in the search page. Click on the Download button. A new page should show up with a button labelled "Agree & Download". Copy the link from that button)
sudo dpkg -i nvidia-driver-local-repo-ubuntu2204-515.65.01_1.0-1_amd64.deb
sudo cp /var/nvidia-driver-local-repo-ubuntu2204-515.65.01/nvidia-driver-local-22D4AC2B-keyring.gpg /usr/share/keyrings/ (the previous dpkg step suggested to copy the keyring)
sudo apt-get update && sudo apt-get -y install cuda-drivers

Reboot the machine after the driver installation. To make sure the drivers are working as expected, run nvidia-smi. You should get an output similar to this:

To verify that the Vulkan drivers are available, we need to install vulkan-tools: sudo apt-get -y install vulkan-tools. After running vulkaninfo you should get something like this:

Next step is to install the latest Docker Engine, following these steps

sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt update && sudo apt-get install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
Check docker is installed successfully: sudo docker run hello-world
Add the user to the docker group so we don't use sudo to execute Docker commands: sudo usermod -aG docker $USER. Log out and log back in so the changes take effect
Make sure Docker is enabled after a reboot: sudo systemctl enable docker.service && sudo systemctl enable containerd.service

Next, we need to install the Nvidia Container Toolkit (instructions here)

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker
Verify that Docker has access to the GPU (notice the --gpus all flag): docker run -it --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi. There should be some text similar to the nvidia-smi command executed previously
Verify that Docker has access to the Vulkan drivers: docker run --rm -it --gpus all nvidia/vulkan:1.2.133-450 vulkaninfo. There should be some text similar to the vulkaninfo command executed previously

Now to run Playwright inside the container, run the following steps:

Start the container with a mounted volume to the current folder, so files are saved: docker run -v $PWD:/tests -w /tests --rm -it --gpus all nvidia/vulkan:1.2.133-450 bash

Inside the container run the following commands to install NodeJS:

apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub (it seems they need to update the container with new keys. If we don't do this, the next commands will fail)
wget https://deb.nodesource.com/setup_16.x -O nodesource_setup.sh
sh nodesource_setup.sh
apt install -y nodejs

To verify that Playwright is working correctly I use the following files:
package.json:



{
  "devDependencies": {
    "@playwright/test": "^1.24.2"
  }
}

playwright.config.js:



const config = {
  use: {
    channel: "chromium",
    launchOptions: {
        args: [
            '--enable-features=Vulkan,UseSkiaRenderer',
            '--use-vulkan=swiftshader',
            '--enable-unsafe-webgpu',
            '--disable-vulkan-fallback-to-gl-for-testing',
            '--dignore-gpu-blocklist',
            '--use-angle=vulkan'
        ]
    }
  }
}
module.exports = config

gpu.spec.js:



import { test, expect } from '@playwright/test';

test('It should take a snapshot of the GPU Chrome page', async ({ page }) => {
    await page.goto('chrome://gpu', { waitUntil: 'domcontentloaded' });
    await page.screenshot({ path: 'gpu.png' });
    await expect(page.locator('text=Graphics Feature Status').first()).toBeVisible();
});

Place those 3 files in the /tests folder
To install Playwright run npm install

Now we need to install the Chromium browser which we'll use for our tests. I'm adding to the command a custom path to install the browsers:

PLAYWRIGHT_BROWSERS_PATH=/tests/.cache/ms-playwright npx playwright install chromium
Install needed dependencies: apt install -y libglib2.0-0 libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdbus-1-3 libxcomposite1 libxdamage1 libpango-1.0-0 libcairo2 libasound2 libatspi2.0-0
Run the Playwright test: PLAYWRIGHT_BROWSERS_PATH=/tests/.cache/ms-playwright npx playwright test gpu.spec.js

As a result the test will save a file called gpu.png. Opening that file should show something like this
If instead you see "Software only, hardware acceleration unavailable" then something went wrong and Playwright is not using hardware acceleration. Go through the steps again, making sure nothing is missing. Good luck!

There are AMIs with Nvidia drivers already installed. For example this one. But I couldn't use it because I lacked the permissions in my AWS account. If you use it, it might make things easier. Let me know in the comments if you do. ↩