Using AI in Playwright Tests

It's been a long while since I've posted anything. Excuse the clickbaity title, but rest assured that this is content that I think will deliver. And this post isn't written by AI!

I'll keep this pretty short and sweet though. This idea was first inspired by the mobile automation framework Maestro. Other than its solid capabilities as a mobile test framework (using YAML of all things), I was impressed to see an API named assertWithAI. Imagine just prompting an LLM with "assert that a blue-colored Login button appears at the bottom". A real game changer in testing if you ask me!

That API is gatekept behind a paid subscription so I didn't experiment much from there. But I was curious to see if Playwright was anywhere close to implementing anything like that, since it is maintained by Microsoft, and Microsoft has its own large stake in OpenAI. At the time of this writing, it did not.

I gained access to an Enterprise version of OpenAI so I decided to experiment with its API. Here is a helpful util that will integrate AI with visual testing.

import OpenAI from 'openai'
import fs from 'fs'
import path from 'path'
import { expect, Page } from '@playwright/test'
import { uniqueId } from './stringHelper'

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY as string,
})

export const askAI = async (page: Page, prompt: string): Promise<string> => {
  const fileName = `screenshot-ai-${uniqueId()}.png`
  const absPath = path.resolve(fileName)
  await page.screenshot({ path: absPath })
  const imageBase64 = fs.readFileSync(absPath, 'base64')

  try {
    const completion = await openai.chat.completions.create({
        model: 'gpt-4o-mini',
        messages: [
        {
          role: 'user',
          content: [
            { type: 'text', text: prompt },
            {
              type: 'image_url',
              image_url: { url: `data:image/png;base64,${imageBase64}` },
            },
          ],
        },
      ],
    })

    const answer = completion.choices[0]?.message?.content ?? ''

    console.log('\n[AI PAGE CHECK]')
    console.log('Prompt:', prompt)
    console.log('Answer:', answer)
    console.log('Screenshot file:', absPath, '\n')

    return answer
  } catch (err: any) {
    // Check rate limit
    if (err.status === 429) {
      console.log(err)
      throw new Error('OpenAI rate limit reached.')
    }
    throw err
  }
}

export const checkAIResponse = async (aiResponse: string, expected: string) => {
  const normalizedResponse = aiResponse.toLowerCase().trim()
  const normalizedExpected = expected.toLowerCase().trim()

  try {
    expect(normalizedResponse).toContain(normalizedExpected)
  } catch {
    throw new Error(
      `Assertion failed against AI response.\n` +
      `Expected: "${normalizedExpected}"\n` +
      `Received: "${normalizedResponse}"`
    )
  }
}

And then writing the tests is really straightforward, powerful, and actually pretty fun:

test('AI confirms a map is visible on the page', async ({ page }) => {
  const mapLink = 'https://www.mysite.com/tracking/b3cd29b39b'
  await page.goto(mapLink)

  const aiAnswer = await askAI(page,
    'Answer YES or NO only. Is there a map and a visible plotted route that starts in Ann Arbor and ends in Detroit?'
  )

  await checkAIResponse(aiAnswer, 'yes')
})

Sometimes Playwright snapshots can only take you so far, and they can become flaky if those snapshots are constantly changing. If there are assets in your app that frequently change and are difficult to automate (but easy to visually confirm), then these are the best spots for this kind of AI assist.