*This article summarizes the FEConf2024 presentation titled 'Dreaming of Easy and Convenient E2E Test Automation'. The presentation is divided into two parts. Part 1 explores E2E testing, tools that support it, and how to build efficient test code by reducing maintenance costs. Part 2 will cover test code reuse, modularization, and enhancements to Playwright. All images included in the article are taken from the presentation slides of the same title, and separate citations are not provided.
'Dreaming of Easy and Convenient E2E Test Automation'
Buseok Baek, CTO at Stibee
- Dreaming of Easy and Convenient E2E Test Automation - pt 1
- Dreaming of Easy and Convenient E2E Test Automation - pt 2
Hello, I'm Buseok Baek, and today I'll be discussing "Dreaming of Easy and Convenient E2E Test Automation." I consider myself a pragmatic developer. With experience in large-scale SI projects and e-commerce platforms like home shopping, I’ve focused on automating repetitive tasks and minimizing manual labor. Currently, I serve as the CTO of Stibee, a service that helps users create and send marketing emails. Leveraging our experience of sending over 2.8 billion emails, I'll share insights on email compatibility optimization, which will serve as the foundation for our deep dive into E2E testing.
The topics covered in this article include:
- The evolution of E2E test automation tools
- Improving efficiency through E2E test automation
- Frontend project setup strategies
However, the following topics will not be covered in depth:
- TDD (Test-Driven Development)
- Mobile-specific E2E testing
- Test CI/CD pipeline automation
Now, let’s dive in.
E2E
What is E2E?
When operating a service, it’s best practice to write unit tests for both the backend and frontend. In reality, though, unit testing is often skipped or only partially implemented. That’s where E2E testing becomes essential—validating a complete user journey to ensure the system works as expected from the user’s perspective.
That said, E2E testing is notoriously challenging. As shown in the diagram below, unit tests are relatively fast and inexpensive to run, and developers can typically manage them independently. In contrast, E2E tests combine integration and UI testing, making them slower to execute and more time-consuming to write.
Despite these drawbacks, their reliability makes E2E tests indispensable, so we decided to maintain a baseline level of test coverage.
E2E Test Process
The E2E testing process involves test design, implementation, execution, and result validation. Since test design typically falls outside a developer’s responsibilities, we’ll skip that part and instead focus on how to write better tests and streamline the process.
Similarly, while test execution optimizations are important, they won’t be the main focus here either. When it comes to reviewing test results, some teams rely on video recordings, but I’ve found that comparing screenshots is often more effective—and that's the approach I use.
Even though E2E test scripts cover the full user journey, they almost always begin on the frontend. So I started thinking about ways to automate that process and make debugging, updates, and reuse through modularization easier.
My E2E Journey: The Beginning
My interest in improving E2E testing began during an SI project. At the time, different browsers rendered HTML, CSS, and JavaScript very differently. This was before WebKit became widespread, so even minor code changes would cause issues across multiple browsers. Debugging these problems was time-consuming and frustrating, so I decided to find a better way.
Selenium
My first approach involved using Selenium, a popular tool for web application testing and automation. Selenium can automatically generate test scripts by recording user interactions. These scripts are stored in a database and replayed to verify behavior.
To make Selenium testing more efficient, I explored screenshot-based visual comparison. During a hackathon, I implemented a feature that captured screenshots at various points during test execution. Comparing screenshots from different browsers helped pinpoint exactly where layout or rendering issues occurred.
However, this method also had limitations, as shown in the presentation slide.
Puppeteer
While working with Selenium, I discovered Puppeteer—a Chrome-based headless testing tool. Right away, I found it much more powerful than Selenium. Puppeteer’s superior network control and speed made it easier to write efficient tests.
But Puppeteer had a major downside: it didn’t support native code generation. I even tried building a browser extension to automate test creation, but it wasn’t stable. Eventually, I switched to using a third-party code generation library.
Another challenge was test replay. If one step failed—say, during checkout after login and cart actions—you had to restart from the login step and repeat every subsequent action.
To fix this, I implemented a way to store test progress in sessionStorage or localStorage and resume from specific checkpoints. Even so, Puppeteer had enough pain points that I wouldn't recommend it today.
Limitations of Selenium and Puppeteer
Both tools had serious drawbacks for scalable test automation. One major issue was selector reliability. Frameworks like React and Vue dynamically generate class names, making traditional CSS or XPath selectors unreliable in many cases.
Playwright
Then came Playwright—the tool that solved many of these problems. Developed by the original Puppeteer team at Microsoft, Playwright feels like Puppeteer 2.0, but better in almost every way.
Playwright supports multiple browsers and executes tests in parallel, which dramatically reduces test execution time. It also supports multiple programming languages and provides robust network control for intercepting or modifying requests.ple programming languages and offers powerful network control for intercepting or modifying requests.
For me, the most important feature is built-in code generation.
Even better, Playwright supports advanced selectors that go beyond CSS or XPath, making it easier to work with modern frontend frameworks. It also includes native support for features like automatic screenshots and persistent browser storage.
Let’s take a closer look at how this works.
Once you install Playwright in VS Code and click the record button, it begins generating test scripts automatically within the editor. While Puppeteer offers a similar feature, Playwright makes it far easier to get a working script on the first try.
Playwright’s reporting tools are also a standout feature. It saves screenshots at every test step, lets you inspect the DOM, and provides console and network logs—making debugging faster and more effective.
E2E Example
E2E Test Environment
Now, let’s look at a real-world example. We’ll explore how E2E testing works using common flows like user registration and login, as well as workflows specific to Stibee, like creating an address book, sending emails, and confirming receipt.
There are generally two types of test environments: temporary and persistent.
A temporary environment spins up Docker containers for databases and servers just to run tests. This setup minimizes test flakiness but is hard to integrate into most real-world scenarios.
Instead, I opted for a persistent testing environment—something that runs against an already active staging or test server. This allowed me to run tests even on accounts created a month or a year ago. It also made it possible to run automated tests on live data periodically, ensuring continuous quality assurance.
Writing Test Code
As you know, maintaining test code can be costly. When writing E2E tests, I focused on minimizing areas with high maintenance overhead. That meant avoiding extra API endpoints or manipulating data just for test purposes.
If test code isn’t written with maintainability in mind, it can quickly become unmanageable—and teams may abandon testing altogether.
Authentication Test Examples
Let’s look at some authentication examples. In E2E tests, you’ll often encounter login flows like Gmail, Daum, or Naver. These platforms tend to block automated bots, so I created an email service that verifies email links in real-time using WebSockets—without adding new APIs.
Next comes reCAPTCHA. While there’s a Puppeteer plugin for solving reCAPTCHA, it’s unreliable. Instead, I used an external service that runs in-browser for about a minute and successfully solves reCAPTCHA, though it requires proper timeout handling.
Then there’s SMS verification. Although I generally avoid building custom APIs, I couldn’t find an off-the-shelf SMS solution. So I built a lightweight API that checks if a 6-digit code exists in the DB.
To make this easier to manage, I used a no-code tool called n8n, similar to Zapier.
With n8n, you can automate workflows like converting a Slack emoji into a Notion task—without writing a line of code.
For SMS verification, I configured n8n to detect a phone number in the DB, retrieve the matching verification code, and return it for validation. Maintenance is simple, thanks to the visual drag-and-drop editor.
By integrating tools like n8n with Playwright's code generation capabilities, you can build highly maintainable and efficient E2E test suites.
So far, we've explored the evolution of E2E testing—from Selenium to Puppeteer to Playwright—and discussed how to handle complex authentication scenarios. We've also seen how to write efficient test code that reduces maintenance costs.
In Part 2, we'll dive into modularizing test code and improving maintainability even further with Playwright’s advanced features.
- Dreaming of Easy and Convenient E2E Test Automation (Part 1)
- Dreaming of Easy and Convenient E2E Test Automation (Part 2)
Top comments (0)