This article is one of the “Bad Test Automation Frameworks/Tools” series.
I started this article 10 months ago, it was sitting in my drafts and just noticed it today. After my three articles on Cypress,
- Why Cypress Sucks for Real Test Automation? and Part 2: Limitations
- Why JavaScript Is Not a Suitable Language for Real Web Test Automation?”
I thought there is no need to talk about Cypress, which, IMO, is doomed to fail (for reasons, read the above). So far, every Cypress test automation (that I am aware of) failed badly, often embarrassingly. I noticed a trend though, after failures, fake automated testers blamed Cypress and then switched to another JS framework Playwright, just like earlier some of them switched to Cypress from the hyped (and now deprecated) Protractor. Playwright is not good either, maybe slightly better than Cypress, then failed again. The best is still raw Selenium WebDriver in Ruby.
The article An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’ tells an interesting and real story of an intern (my mentee, freshly out of Uni) who silenced one fake senior test automation engineer with indisputable results, implemented automated UI tests in both Selenium and Playwright at the productivity this large company have never seen. (Reason: this intern developed tests in raw Selenium WebDriver first, then converted to Playwright). Eventually, this fake senior Playwright tester did dirty tricks to sabotage. A good read.
Today, I saw a LinkedIn feed to recommend using Cypress for test automation, and it was posted in a Selenium Group. So I decide to complete and publish this article.
Anyway, the aim of this article is about an objective comparison of Cypress and Selenium syntax, using a very simple example, from a tester’s perspective. To be fair, I used the Cypress test script example posted by Gleb Bahmutov (known as the authority on Cypress). So, the comparison is as objective as it can be.
“Test Scripts are the core of test automation, naturally. Ignore others who tell you otherwise” — Zhimin Zhan
Cypress vs Selenium WebDriver (Ruby)
I will use a simple example by Gleb Bahmutov (“Avoid Cypress Pyramid of Doom” posted on 2021–07–28): a simple calculator on a web page.
The HTML (from Gleb’s post):
<body>
<p>Calculator</p>
<div>a = <input name="a" type="number" value="1" /></div>
<div>b = <input name="b" type="number" value="5" /></div>
<div>a + b = <span id="result">6</span></div>
</body>
Here is Gleb’s ‘good version’ (there were a couple of not-so-good versions shown) Cypress test script.
it('adds numbers via aliases', () => {
cy.visit('public/index.html')
cy.get('[name=a]').invoke('val').then(parseInt).as('a')
cy.get('[name=b]').invoke('val').then(parseInt).as('b')
cy.get('#result')
.invoke('text')
.then(parseInt)
.as('result')
.then(function () {
expect(this.a + this.b).to.eq(this.result)
})
})
My version in raw Selenium (RSpec):
it "adds numbers" do
driver.get("public/index.html")
a = driver.find_element(:name, "a")["value"].to_i
b = driver.find_element(:name, "b")["value"].to_i
result = driver.find_element(:id, "result").text.to_i
expect(a + b).to eq(result)
end
It is a lot better, isn’t it? Please take a stand from an independent testing perspective.
“Users do not care how the website was coded in Java, JS, C#, PHP or Ruby. The same applies when end-to-end testing a website from a user perspective” — Zhimin Zhan
Comparative Analysis
Let’s see some differences. If you are using Cypress or other JS framework, pause, and think objectively as a team member (e.g. Business Analyst or Manual Tester) who might want to use your automated tests:
Get the value of an input text field
Cypress: invoke('val')
Selenium: ["value"]
Verdict: Selenium WebDriver is more natural, as it gets the value attribute in HTML tag: <input name="a" type="number" value="1"/> . The invoke and 'val' in Cypress make the simple things complex. The keyword invoke, in particular, is bad. Where is 'val' in the HTML fragment? (I know as I programmed JS professionally before, again, consider for others who don’t know JS at all)
Convert string to an integer
Cypress (JavaScript): parseInt
Selenium (Ruby): to_i
Verdict: the Ruby syntax is better as it is concise and more readable.
Promise
(unneeded in software testing, if thinking outside JS)
Cypress: then
Selenium: no such concept. (Please note, achieving the same testing result).
Verdict: Selenium is better. JavaScript’s Promise is confusing and totally unnecessary from the testing perspective.
Alias
(unneeded as well)
Cypress: as('a'), also, later this.a
Selenium: no such concept, just naturally assign a variable, a = .
Verdict: Selenium is a lot better, the alias usage in Cypress is totally unnecessary from the testing perspective.
The winner, clearly, is raw Selenium with Ruby.
“Ruby is the best for scripting automated tests” is not news or secret, why on earth do people fail test automation again and again in Java/C#/JavaScript?
Human Factors
Besides tech comparison, Selenium WebDriver’s version is more concise and consistent, which are important for maintenance. The effort for test script maintenance far exceeds that of test creation. That’s why most JS test automation attempts started OK (and excitedly), then failed badly (and quietly): unable to maintain the tests.
There are other factors too, such as Gleb’s article showing several different (worse) versions, which may cause problems in a team environment. With Selenium WebDriver, if the team stays with the raw Selenium, it is quite easy for the team to reach a convention.
Selenium WebDriver, syntax-wise, is much better than all others for a simple reason: it was created by W3C experts after many rounds of reviews and refinement. — Zhimin Zhan
More importantly, real test automation engineers know that the audience of automated test scripts is the whole team, not limited to programmers. Even the simple syntax that JS programmers take for granted, such as () => {, this.a, and }) may confuse the business analysts and manual testers.
FAQ
I think some Cypress/Playwright/JS testers disagree with my recommendation of raw Selenium WebDriver with Ruby. Before raising your questions, watch the below 37-second video, which is extracted from this great presentation: “Continuous Integration at Facebook”.
Video on YouTube: https://youtu.be/D5Zk5ylPrGk
Facebook’s Testing Pyramid image credit: “Continuous Integration at Facebook”
I added two more references.
“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.
Microsoft dumped its “Coded UI Test” and recommended Selenium WebDriver
So, Selenium WebDriver works, and works very well! Accept this, first. This does not mean excluding others. But if someone wants to promote a new framework/tool X, people shall know the free, open-source, W3C-standard compliant Selenium WebDriver is excellent already for web test automation, when used by real test automation engineers, with extremely high productivity. Also, under proper guidance, raw Selenium WebDriver is the easiest to learn, only takes a few hours.
By the way, my continuous testing process (running automated end-to-end UI tests regression) enables me to deploy all my apps on a daily basis, like Facebook, since 2012. I am NOT interested in debating specific features of a framework or tool, rather than the fundamental, whether your test automation is useful for the team (check out Definition of End-to-End Test Automation Success). My Q&As below are for the engineers who are motivated for real test automation.
(1). I acknowledge Selenium WebDriver (Ruby) syntax is better, however, Cypress is more than a framework, its tool makes creating tests easier?
Wrong. The tool is after the framework, and always should be, in the context of software development. We talk about here are the test scripts (what really matters).
(2). Cypress seems easier to use, compared to Selenium.
Wrong. Cypress is a combination of a framework and tool. You cannot compare a framework with a vendor-provided tool demo. HP’s QTP and IBM’s RFT had good demos too, further, some top magazines or consulting companies (such as Gartner) sang for them. The result: complete failures.
I think Selenium is much easier, TestWise help real automated testers be more productive and fun. Check out my daughter’s “Set up, Develop Automated UI tests and Run them in a CT server on your First day at work”.
(3). People tell me that Selenium is hard to learn.
Wrong. My daughter started writing raw Selenium test scripts at the age of 12, with the right mentoring, your child could do that too. Check out my article: “Why Selenium WebDriver is So Easy to Learn?”.
As a matter of fact, some people hesitated for signing up for my one-day “Web Test Automation with Selenium WebDriver” training because it was too short. My reply: “ learning Selenium WebDriver is just that easy, in fact, during that 8 hours, I spent over 50% on Maintainable Automated Test Design and Functional Test Refactoring”. By the way, no prior test automation/programming experience is required. Again, that’s because raw Selenium WebDriver is that simple and easy to learn. Many so-called ‘better’ frameworks such as failed Protractor actually made it harder.
(4). I think developing Cypress scripts (with the Cypress tool) is more productive than that of Selenium.
Wrong. I am at least 10X more productive than any Cypress and Playwright testers I met. I often completed a planned 3-month Test Automation Proof of Concept (POC) in a matter of days, typically ~ 3 days. The staff at those companies were shocked by my productivity. The reason: I am developing raw Selenium WebDriver test scripts in TestWise, following good test-automation practices such as Maintainable Automated Test Design, Keep browser open, Functional Test Refactoring, .., etc. Special mention: Ruby is great for scripting automated tests.
Not convinced? Check out the screencast in this article: “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes”, and some of the test suites I have been maintaining over the years.
- ClinicWise, 611 raw Selenium WebDriver tests (since 2012).
- TestWise, 307 raw Appium + WinAppDriver tests (since 2019)
- WhenWise, 551 raw Selenium WebDriver tests (since 2018)
all in my spare time.
Any motivated manual tester can achieve high productivity (might take a while to match mine, but shall be much much higher than the so-called best JS automated tester) quite quickly. Many manual testers I mentored became the test automation authority in their companies.
(5). I read the above articles and watched the tutoring, it did look very simple and easy. How come automated testers generally are associated with “difficult” to Selenium?
There are many reasons and I covered those in other articles, such as “Evil Mudslingings against Selenium WebDriver”. Here, I want to just point out key ones concisely.
- Vendors defame Selenium (free, open-source) for commercial interests
- Test automation is often started by programmers who don’t really understand (but pretend), they often chose the wrong language such as Java, C# or JavaScript.
- Not aware of a good Selenium tool, such as TestWise.
- Never experienced a successful Continuous Testing, often using CI servers such as Jenkins to run UI tests. Instead, they should use a CT server such as Facebook’s Sandcastle or BuildWise.
(6). I have used Selenium-based frameworks (created by our test architects), but it was not as easy as you showed us. Why?
That’s because those people did not understand test automation, making Selenium harder. Check out my article “Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver”. Often they used the wrong languages such as Java/C#, or the wrong test syntax frameworks such as Gherkin. Just use raw Selenium WebDriver + RSpec first and most popular BDD framework).
(7). Cypress’s new version has better ….?
This is exactly what the problem is. I have been using Selenium WebDriver exactly the same way since 2011 (Selenium v4 has some new features). Don’t you want this stability, or rather being dependent on others? Furthermore, it is really important to have consistent test script syntax when working in a team. As you have seen Gleb’s “Avoid Cypress Pyramid of Doom”, there are so many ways, a recipe for chaos.
(8). How come so many testers claimed doing Cypress well? Are they all fake?
I only can comment on people I met. Yes, every Cypress/Playwright testers I met was a fake. This is not belittling them, automated end-to-end testing is difficult (except HelloWorld level demos), even for Microsoft (see below). Using Cypress/Playwright makes it even much harder, unnecessarily.
“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.
Related articles:
- My eBook: “Practical Web Test Automation with Selenium WebDriver”
- Why Do Most UI Test Automation Fail? (Part 2: Wrong Choice of Test Syntax Framework)
- Testing Pyramid Clarified
- User Story Card Clarified
- Continuous Testing Clarified
- “Done, Done” in Agile
- AgileWay Test Automation Formula
- Selenium vs X, Y, Z, …, all these years
Top comments (0)