Leveraging Playwright MCP Server for Test Generation by LLMs

Introduction to Playwright MCP

In the evolving landscape of web development, ensuring the reliability and performance of web applications across different browsers is paramount. This is where the Playwright Model Context Protocol (MCP) Server can be helpful, enabling LLMs to interact with web pages through structured accessibility snapshots, bypassing the need for screenshots or visually-tuned models.

The Goal

This article embarks on a journey to explore the setup and utilization of the Playwright MCP Server within Windsurf IDE, specifically targeting the automation of tests for the web app Conduit. Through this exploration, we aim to compare the outcomes produced using this advanced tool.

Utilized LLMs for Enhanced Test Generation

1. GPT-4.1
2. Claude 3.7 Sonnet (Thinking)
3. SWE-1
4. DeepSeek R1
5. xAI Grok-3

Setting the Stage: Prerequisites

Before diving into the waters of Playwright MCP Server, certain prerequisites must be met:

1. Node.js: Version 20.13.1 or newer is required for optimal performance.
2. npm: Ensure you have version 10 or later installed, laying the foundation for further installations.

Installation Guide

You can find the full guide in the following github repository

Playwright MCP Server Configuration

To configure the Playwright MCP Server, refer to the detailed instructions provided in the official documentation. This step is crucial for ensuring optimal integration and functionality.

Enhanced Test Generation: Custom Instructions

Windsurf IDE requires a unique setup to harness the full capabilities of the Playwright MCP Server. Create a .windserf/rules folder in the root of your project and add a custom instruction file named playwright-mcp-server-test-generator.md. These instructions should be set to "Always On" to ensure continuous assistance from the LLMs in generating and refining test cases based on your specifications.

Example for instructions:

1. You are a playwright test generator.
2. You are given a scenario and you need to generate a playwright test for it.
3. DO NOT generate test code based on the scenario alone.
4. DO run steps one by one using the tools provided by the Playwright MCP.
5. Only after all steps are completed, emit a Playwright TypeScript test that uses '@playwright/test'. Below are instruction for generating the test:
5.1. You are an expert in TypeScript, Frontend development, and Playwright end-to-end testing.
You write concise, technical TypeScript code with accurate examples and the correct types.
-   Create Class for all locators and methods for all actions
-   Avoid using `page.locator` and always use recommended build-in and role-based locators (`getByRole`, `getByLabel`, etc)
-   Prefer to use web-first assertions (`toBeVisible`, `toHaveText`, etc) whenever possible
-   Use build in config objects like `devices` whenever possible
-   Avoid hardcoded timeouts
-   Reuse Playwright locators by using variables
-   Follow the guidance and the best practices described on playwright.dev
-   Avoid commenting the resulting code
6. Save generated test file in the tests directory
7. Execute the test file and iterate until the test passes

Preparing Test Data

With the environment ready, the creation of a test file named "yourTestName.spec.ts" is the next step. Populate this file with the necessary test data, including URLs, login credentials, and content details for the web app Conduit. This data serves as the foundation for generating meaningful and effective test cases using the various LLMs.

Example:

const url = process.env.URL || 'https://conduit.bondaracademy.com/';
const email = process.env.EMAIL || 'yourEmail';
const password = process.env.PASSWORD || 'yourPassword';
const articleTitle = `Test Article for Playwright MCP Server ${Date.now()}`;
const articleAbout = 'This is a test article about Playwright MCP Server';
const articleContent = 'This article is created by Playwright MCP Server test automation';
const updatedArticleTitle = `Updated ${articleTitle}`;
const updatedArticleAbout = 'Updated article about';
const updatedArticleContent = 'This article has been updated';

What's next?

Stay tuned for subsequent articles in this series, where we will delve deeper into comparisons between different LLMs' generated POMs and tests.
Please, do not hesitate to start conversation regarding the topic, since it is just the beginning.

Disclaimer

This repository is intended for informational and educational purposes only. The results and analyses presented do not imply any endorsement or disapproval of any specific LLM.

🙏🏻 Thank you for reading! Building robust, scalable automation frameworks is a journey best taken together. If you found this article helpful, consider joining a growing community of QA professionals 🚀 who are passionate about mastering modern testing.

Join the community and get the latest articles and tips by signing up for the newsletter.