Time to add some end to end tests for your next application. Today I have a new browser automation tool to share.
This is written from the perspective of the team heavily using JavaScript as the development language. I evaluated a few browser automation frameworks for end-to-end application testing and thought to share things the current landscape of tooling.
An End to end test is also known as functional-tests or happy-path tests which automates the sanity or spot checks before or after a software release.
I needed to pick one considering ease of development, long term maintainability, test framework integration. Let's explore a few interesting players out there.
- The good old Selenium
- The Chromium based browser automation API Puppeteer
- The most hyped Cypress
- Taiko - The new kid on the block (It's ok if you have never heard of this. I didn't too)
TL;DR; I ended up picking Taiko for its ❤️ beautiful API and it just works like magic!
1. Selenium ❌
Selenium was built in 2004. The major pain point of Selenium is its architecture.
A system that behaves differently in each run, also termed as flakiness is the major concern with Selenium with its layered architecture. Let's see why Selenium tests are flaky.
In the diagram, it is obvious that the commands have to pass through multiple layers. The drivers are built external to the browser which results in driving the browser "blind" with no feedback on whether the command has successfully been executed.
2. Puppeteer ❌
Fast forward from 2004, 14 years later, Google released v1.0 of Puppeteer.
One thing you will notice from the previous diagram is there are fewer layers between the controller and the browser. This is the key difference between the modern end to end testing frameworks.
Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol.
This is powerful. It uses the DevTools protocol, the same low-level protocol which the Chrome Developer tools uses to interact with the browser.
A code example:
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
console.log(await page.content());
await page.screenshot({path: 'screenshot.png'});
await browser.close();
However, it is geared towards browser automation and the end to end web application testing is only one of them. This means you have to explicitly wait for a page to load, assert things you care about which will soon make writing and maintaining tests not so fun.
3. Cypress ❌
Cypress automates the browser with its own unique architecture. Cypress takes a less traveled and hard path to make runs the test from inside the browser and communicates the actions to an outside Node process which makes testing very capable.
It comes with electron baked in. Many times, I found very simple tests work in electron which they bundle by default however fails to run in Chromium or Chrome in a headless environment.
Rave reviews, Marketing, Podcast talks, Beautiful documentation, however, it does not live up to the high expectations you would expect.
I gave it enough energy and wanted this to work. Wanted this to work so badly. However, realized it has dangerous browser proxy mods which fail to set the redirections when run inside a corporate proxy environment and the most annoying limitation, if you are on one domain you cannot visit another domain.
Each test is limited to only visiting a single super-domain.
So if you have an application with a login page which is in another domain get ready to cry 😭. Cry so loud that Cypress team hears. I wish these change in the future. I gave up.
4. Taiko ✅
Taiko is from ThoughtWorks. The same place where Selenium was born. It shines ✨.
It shines really well with its ❤️ beautiful API.
Show me some code...
await openBrowser();
await goto("google.com");
await write("Dennis Mathew Philip");
await click("Google Search");
You will notice there is no waiting for an element to appear. No CSS selectors. It is smart to identify what input field.
✅ Beautiful documentation
✅ Integration with Jest and other popular testing frameworks
✅ No explicit wait needed. Less code
✅ Headless and CI friendly
It also has a REPL which makes the development a breeze.
A straightforward gif from the Taiko team:
I loved using Taiko. I am having fun writing end to end tests. I hope you will too.
To conclude...
End to end testing improves the confidence with which you release the software. We compared some of the free and open-source end to end testing solutions - Selenium, Puppeteer, Cypress and Taiko. The opinion written here is straight what I experienced. Consider your use-case closely and pick the right tool what works best for you. Browser automation tools is a lot of power. Use responsibly.
If your web application does not have at least one functional test, start today.
Please find my other writings here:
Oldest comments (9)
Hi Dennis!
Can you, please, demonstrate how to click in Taiko in Salesforce in Lightning mode 'New Contact' or something similar?
Hi Dennis,
Does your team use the Gauge+Taiko framework in Saleforce?
Compared to cypress, what's the advantage of Gauge+Taiko?
What testrunner to you use together with Taiko? Jest? Gauge?
Do you know if it's possible to use Taiko with another test automation framework?
Their website recommends using Gauge, but I'd like to use it with SauceLabs.
Hi Wiley,
Good day. Did you get answer for this. I am also interested in integrating with SauceLabs.
Unfortunately, no. I didn't invest time to know more about this topic.
Hello ,
As part of continuous learning I came across Taiko, given attempt to try. Impressed how the objects are identified as we type...after writing same code provided in the taiko blog when I tried to run the .js I am getting below error
'openbrowser cannot be called again as there is a chromium instance open taiko'
How to resolve this?
Appreciate your help.
Any thoughts on the future of Taiko? My understanding is that Thoughtworks is "stopping the funding" i.e. handing the project over to "the community". That can go well like with Selenium (also from Thoughtworks). Or it could go not so well :-)
I am new to Taiko so it's hard to get a feel for it, but I'm about to decide on which testing tool/framework to use for a brand new test suite partially by migrating from selenium.
You’re right. I saw there are just two contributors were active for the last month. In my opinion, it doesnt mean that project is dead, it’s just not highly active like other frameworks. But I think thats the beauty of the open source world! If you need something about the project, instead of waiting to get it done, you can implement it by yourself! (Even it’s hard 😛)
My advice is: try some basic tests and then if you’re happy with it, keep using it. If not, switch to more active frameworks. Keep in mind that highly active products can die too, yup I’m talking about Angular 1 😂