DEV Community

Ali Nemati
Ali Nemati

Posted on AI-assisted

Recording a Chrome extension's side panel with Playwright and ffmpeg

Playwright can load an unpacked Chrome extension, but the side panel belongs to the browser window rather than a page, so recordVideo never sees it. Its video is VP8 at about 1 Mbps, which also blurs small UI text. For the guide video of version 0.0.252 of the INDOX browser extension we wanted the web page and the panel side by side and sharp, so our recorder treats the panel as a second page and rebuilds the window with ffmpeg. The code below runs on Playwright 1.64.

Load the extension

import { chromium } from 'playwright';

const context = await chromium.launchPersistentContext(profileDir, {
  channel: 'chromium',
  viewport: { width: 2560, height: 1288 },
  args: [`--disable-extensions-except=${ext}`, `--load-extension=${ext}`],
});
const sw = context.serviceWorkers()[0] ?? await context.waitForEvent('serviceworker');
const panelUrl = `chrome-extension://${new URL(sw.url()).host}/src/sidepanel/index.html`;
Enter fullscreen mode Exit fullscreen mode

Extensions load only in a persistent context. The MV3 service worker starts with the extension, and its URL holds the extension id, which gives you the panel's address. The same worker can write settings before the first frame, so onboarding screens stay out of the take:

await sw.evaluate((s) => chrome.storage.sync.set({ 'indox.settings': s }), settings);
Enter fullscreen mode Exit fullscreen mode

Open the panel in its own window

const opened = context.waitForEvent('page');
await sw.evaluate(() => chrome.windows.create({ url: 'about:blank', focused: false }));
const panel = await opened;
await panel.goto(panelUrl);
Enter fullscreen mode Exit fullscreen mode

Point chrome.tabs.query at the page

Panel code usually finds the page the user is on with chrome.tabs.query({ active: true, currentWindow: true }). Inside the separate window that call returns the panel's own chrome-extension:// tab, so Convert or Translate acts on the wrong tab. An init script, added before panel.goto, sends that one query to the site tab:

await panel.addInitScript(() => {
  const query = chrome.tabs.query.bind(chrome.tabs);
  chrome.tabs.query = async (info) =>
    info?.active && info?.currentWindow
      ? (await query({})).filter((t) => !t.url?.startsWith('chrome-extension://')).slice(0, 1)
      : query(info);
});
Enter fullscreen mode Exit fullscreen mode

Capture screencast frames

await page.screencast.start({ size, quality: 92, onFrame: ({ data }) => {
  writeFileSync(join(dir, `${String(frames.length).padStart(6, '0')}.jpg`), data);
  frames.push({ at: now() });
} });
Enter fullscreen mode Exit fullscreen mode

Each JPEG keeps the time it arrived. When the shot ends, an ffconcat list gives every frame its real duration, and ffmpeg encodes the result with libx264 at CRF 14. One capture runs on the site page and one on the panel.

Keep the account off screen

The panel shows who is signed in. A second init script replaces the account email in the panel's text with "Demo", and before each shot is kept the recorder reads document.body.innerText and fails the run if the email still appears.

Rebuild the window with ffmpeg

This is the filter graph our code builds for one 14-second shot, minus the timing parts. Input 0 is a rendered browser frame with the tab title and address bar, input 1 the site capture, input 2 the panel capture:

ffmpeg -framerate 30 -loop 1 -t 14 -i frame.png -i site.mp4 -i panel.mp4 -filter_complex \
  "[1:v]crop=2560:1288:0:0[s];[0:v][s]overlay=0:152[a];[2:v]crop=800:1288:0:0[p];[a][p]overlay=1760:152[v]" \
  -map "[v]" -t 14 -c:v libx264 -crf 16 -an shot.mp4
Enter fullscreen mode Exit fullscreen mode

The panel is 400 CSS pixels wide at device scale 2, so it lands at x 1760 on a 2560 by 1440 canvas, under a 76 pixel address bar. Every shot becomes one segment, and the concat demuxer joins the segments with -c copy. Captions, the end card and the audio go on in a later pass. This is the finished guide:

The extension in the video is INDOX 0.0.252: https://indox.org/download/extensions?utm_source=devto&utm_medium=community&utm_campaign=extension-0.0.252

Playwright's page on Chrome extensions: https://playwright.dev/docs/chrome-extensions

Top comments (1)

Collapse
 
demovexa profile image
DemoVexa •

Treating the panel as a second page is a neat way around recordVideo, and failing the run when the account email shows up is a guard more recorders should copy. One thing I'd watch with two screencasts is drift: frames arrive at slightly different moments on each page, so the panel can trail the site by a frame after a click. Stamping both captures against one shared clock and cutting each segment from the same start keeps them in step.