DEV Community: Zhimin Zhan

Cypress vs Selenium WebDriver Comparison by Example

Zhimin Zhan — Fri, 16 Sep 2022 21:17:08 +0000

This article is one of the “Bad Test Automation Frameworks/Tools” series.

I started this article 10 months ago, it was sitting in my drafts and just noticed it today. After my three articles on Cypress,

I thought there is no need to talk about Cypress, which, IMO, is doomed to fail (for reasons, read the above). So far, every Cypress test automation (that I am aware of) failed badly, often embarrassingly. I noticed a trend though, after failures, fake automated testers blamed Cypress and then switched to another JS framework Playwright, just like earlier some of them switched to Cypress from the hyped (and now deprecated) Protractor. Playwright is not good either, maybe slightly better than Cypress, then failed again. The best is still raw Selenium WebDriver in Ruby.

The article An IT Graduate’s frustration with a Fake ‘Senior Test Automation Engineer’ tells an interesting and real story of an intern (my mentee, freshly out of Uni) who silenced one fake senior test automation engineer with indisputable results, implemented automated UI tests in both Selenium and Playwright at the productivity this large company have never seen. (Reason: this intern developed tests in raw Selenium WebDriver first, then converted to Playwright). Eventually, this fake senior Playwright tester did dirty tricks to sabotage. A good read.

Today, I saw a LinkedIn feed to recommend using Cypress for test automation, and it was posted in a Selenium Group. So I decide to complete and publish this article.

Anyway, the aim of this article is about an objective comparison of Cypress and Selenium syntax, using a very simple example, from a tester’s perspective. To be fair, I used the Cypress test script example posted by Gleb Bahmutov (known as the authority on Cypress). So, the comparison is as objective as it can be.

“Test Scripts are the core of test automation, naturally. Ignore others who tell you otherwise” — Zhimin Zhan

Cypress vs Selenium WebDriver (Ruby)

I will use a simple example by Gleb Bahmutov (“Avoid Cypress Pyramid of Doom” posted on 2021–07–28): a simple calculator on a web page.

The HTML (from Gleb’s post):

<body>
  <p>Calculator</p>
  <div>a = <input name="a" type="number" value="1" /></div>
  <div>b = <input name="b" type="number" value="5" /></div>
  <div>a + b = <span id="result">6</span></div>
</body>

Here is Gleb’s ‘good version’ (there were a couple of not-so-good versions shown) Cypress test script.

it('adds numbers via aliases', () => {
  cy.visit('public/index.html')
  cy.get('[name=a]').invoke('val').then(parseInt).as('a')
  cy.get('[name=b]').invoke('val').then(parseInt).as('b')
  cy.get('#result')
    .invoke('text')
    .then(parseInt)
    .as('result')
    .then(function () {
      expect(this.a + this.b).to.eq(this.result)
    })
})

My version in raw Selenium (RSpec):

it "adds numbers" do
  driver.get("public/index.html")
  a = driver.find_element(:name, "a")["value"].to_i 
  b = driver.find_element(:name, "b")["value"].to_i
  result = driver.find_element(:id, "result").text.to_i
  expect(a + b).to eq(result)
end

It is a lot better, isn’t it? Please take a stand from an independent testing perspective.

“Users do not care how the website was coded in Java, JS, C#, PHP or Ruby. The same applies when end-to-end testing a website from a user perspective” — Zhimin Zhan

Comparative Analysis

Let’s see some differences. If you are using Cypress or other JS framework, pause, and think objectively as a team member (e.g. Business Analyst or Manual Tester) who might want to use your automated tests:

Get the value of an input text field

Cypress: invoke('val')
Selenium: ["value"]
Verdict: Selenium WebDriver is more natural, as it gets the value attribute in HTML tag: <input name="a" type="number" value="1"/> . The invoke and 'val' in Cypress make the simple things complex. The keyword invoke, in particular, is bad. Where is 'val' in the HTML fragment? (I know as I programmed JS professionally before, again, consider for others who don’t know JS at all)

Convert string to an integer

Cypress (JavaScript): parseInt
Selenium (Ruby): to_i
Verdict: the Ruby syntax is better as it is concise and more readable.

Promise

(unneeded in software testing, if thinking outside JS)

Cypress: then
Selenium: no such concept. (Please note, achieving the same testing result).
Verdict: Selenium is better. JavaScript’s Promise is confusing and totally unnecessary from the testing perspective.

Alias

(unneeded as well)

Cypress: as('a'), also, later this.a
Selenium: no such concept, just naturally assign a variable, a = .
Verdict: Selenium is a lot better, the alias usage in Cypress is totally unnecessary from the testing perspective.

The winner, clearly, is raw Selenium with Ruby.

“Ruby is the best for scripting automated tests” is not news or secret, why on earth do people fail test automation again and again in Java/C#/JavaScript?

Human Factors

Besides tech comparison, Selenium WebDriver’s version is more concise and consistent, which are important for maintenance. The effort for test script maintenance far exceeds that of test creation. That’s why most JS test automation attempts started OK (and excitedly), then failed badly (and quietly): unable to maintain the tests.

There are other factors too, such as Gleb’s article showing several different (worse) versions, which may cause problems in a team environment. With Selenium WebDriver, if the team stays with the raw Selenium, it is quite easy for the team to reach a convention.

Selenium WebDriver, syntax-wise, is much better than all others for a simple reason: it was created by W3C experts after many rounds of reviews and refinement. — Zhimin Zhan

More importantly, real test automation engineers know that the audience of automated test scripts is the whole team, not limited to programmers. Even the simple syntax that JS programmers take for granted, such as () => {, this.a, and }) may confuse the business analysts and manual testers.

FAQ

I think some Cypress/Playwright/JS testers disagree with my recommendation of raw Selenium WebDriver with Ruby. Before raising your questions, watch the below 37-second video, which is extracted from this great presentation: “Continuous Integration at Facebook”.

Video on YouTube: https://youtu.be/D5Zk5ylPrGk

Facebook’s Testing Pyramid image credit: “Continuous Integration at Facebook”

I added two more references.

“Facebook is released twice a day, and keeping up this pace is at the heart of our culture. With this release pace, automated testing with Selenium is crucial to making sure everything works before being released.” — DAMIEN SERENI, Engineering Director at Facebook, at Selenium 2013 conference.

Microsoft dumped its “Coded UI Test” and recommended Selenium WebDriver

So, Selenium WebDriver works, and works very well! Accept this, first. This does not mean excluding others. But if someone wants to promote a new framework/tool X, people shall know the free, open-source, W3C-standard compliant Selenium WebDriver is excellent already for web test automation, when used by real test automation engineers, with extremely high productivity. Also, under proper guidance, raw Selenium WebDriver is the easiest to learn, only takes a few hours.

By the way, my continuous testing process (running automated end-to-end UI tests regression) enables me to deploy all my apps on a daily basis, like Facebook, since 2012. I am NOT interested in debating specific features of a framework or tool, rather than the fundamental, whether your test automation is useful for the team (check out Definition of End-to-End Test Automation Success). My Q&As below are for the engineers who are motivated for real test automation.

(1). I acknowledge Selenium WebDriver (Ruby) syntax is better, however, Cypress is more than a framework, its tool makes creating tests easier?

Wrong. The tool is after the framework, and always should be, in the context of software development. We talk about here are the test scripts (what really matters).

(2). Cypress seems easier to use, compared to Selenium.

Wrong. Cypress is a combination of a framework and tool. You cannot compare a framework with a vendor-provided tool demo. HP’s QTP and IBM’s RFT had good demos too, further, some top magazines or consulting companies (such as Gartner) sang for them. The result: complete failures.

I think Selenium is much easier, TestWise help real automated testers be more productive and fun. Check out my daughter’s “Set up, Develop Automated UI tests and Run them in a CT server on your First day at work”.

(3). People tell me that Selenium is hard to learn.

Wrong. My daughter started writing raw Selenium test scripts at the age of 12, with the right mentoring, your child could do that too. Check out my article: “Why Selenium WebDriver is So Easy to Learn?”.

As a matter of fact, some people hesitated for signing up for my one-day “Web Test Automation with Selenium WebDriver” training because it was too short. My reply: “ learning Selenium WebDriver is just that easy, in fact, during that 8 hours, I spent over 50% on Maintainable Automated Test Design and Functional Test Refactoring”. By the way, no prior test automation/programming experience is required. Again, that’s because raw Selenium WebDriver is that simple and easy to learn. Many so-called ‘better’ frameworks such as failed Protractor actually made it harder.

(4). I think developing Cypress scripts (with the Cypress tool) is more productive than that of Selenium.

Wrong. I am at least 10X more productive than any Cypress and Playwright testers I met. I often completed a planned 3-month Test Automation Proof of Concept (POC) in a matter of days, typically ~ 3 days. The staff at those companies were shocked by my productivity. The reason: I am developing raw Selenium WebDriver test scripts in TestWise, following good test-automation practices such as Maintainable Automated Test Design, Keep browser open, Functional Test Refactoring, .., etc. Special mention: Ruby is great for scripting automated tests.

Not convinced? Check out the screencast in this article: “Step by Step showing how to learn to write raw Selenium WebDriver test scripts in minutes”, and some of the test suites I have been maintaining over the years.

ClinicWise, 611 raw Selenium WebDriver tests (since 2012).
TestWise, 307 raw Appium + WinAppDriver tests (since 2019)
WhenWise, 551 raw Selenium WebDriver tests (since 2018)

all in my spare time.

Any motivated manual tester can achieve high productivity (might take a while to match mine, but shall be much much higher than the so-called best JS automated tester) quite quickly. Many manual testers I mentored became the test automation authority in their companies.

(5). I read the above articles and watched the tutoring, it did look very simple and easy. How come automated testers generally are associated with “difficult” to Selenium?

There are many reasons and I covered those in other articles, such as “Evil Mudslingings against Selenium WebDriver”. Here, I want to just point out key ones concisely.

Vendors defame Selenium (free, open-source) for commercial interests
Test automation is often started by programmers who don’t really understand (but pretend), they often chose the wrong language such as Java, C# or JavaScript.
Not aware of a good Selenium tool, such as TestWise.
Never experienced a successful Continuous Testing, often using CI servers such as Jenkins to run UI tests. Instead, they should use a CT server such as Facebook’s Sandcastle or BuildWise.

(6). I have used Selenium-based frameworks (created by our test architects), but it was not as easy as you showed us. Why?

That’s because those people did not understand test automation, making Selenium harder. Check out my article “Please, Not Another Web Test Automation Framework, Just Use Raw Selenium WebDriver”. Often they used the wrong languages such as Java/C#, or the wrong test syntax frameworks such as Gherkin. Just use raw Selenium WebDriver + RSpec first and most popular BDD framework).

(7). Cypress’s new version has better ….?

This is exactly what the problem is. I have been using Selenium WebDriver exactly the same way since 2011 (Selenium v4 has some new features). Don’t you want this stability, or rather being dependent on others? Furthermore, it is really important to have consistent test script syntax when working in a team. As you have seen Gleb’s “Avoid Cypress Pyramid of Doom”, there are so many ways, a recipe for chaos.

(8). How come so many testers claimed doing Cypress well? Are they all fake?

I only can comment on people I met. Yes, every Cypress/Playwright testers I met was a fake. This is not belittling them, automated end-to-end testing is difficult (except HelloWorld level demos), even for Microsoft (see below). Using Cypress/Playwright makes it even much harder, unnecessarily.

“95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”.

this interview from Microsoft Test Guru Alan Page (2015)

Why Gherkin (Cucumber, SpecFlow,…) Always Failed with UI Test Automation?

Zhimin Zhan — Thu, 15 Sep 2022 22:48:32 +0000

This article was originally posted on Medium (2021-01), and featured in Medium's largest publication 'The Startup' and some software testing newsletters. This is also included in “Be aware of Fake Test Automation/DevOps Engineers” series.

Many software projects tried or are trying to use Cucumber for test automation, commonly with Selenium Driver for testing web apps. Some might wonder whether my title is just a personal and radical view for attention. No, I just reworded the view from Aslak Hellesøy, the creator of Cucumber:

“So really, what is Cucumber? As a test tool it sucks. There far better automated test tools” (source)

Some Gherkin fans may say: “this might be one mis-comment”. Oh well, here is another one on Aslak’s home age.

“If all you need is a testing tool for driving a mouse and a keyboard, don’t use Cucumber. There are other tools that are designed to do this with far less abstraction and typing overhead than Cucumber.” (source):

With my 10+ years of test automation and continuous testing consultation, every test automation attempt with Gherkin-style syntax (Cucumber, SpecFlow, JBehave, Concordion, Gauge, Spinach…) failed, with no expectations! The biggest failure I heard, “The project spent 3 times of development efforts (measured time and money) trying to maintain those cucumber tests, eventually, dumped to the bin!”

Why does Cucumber fail in test automation? Let’s hear from DHH (Creator of Ruby on Rails, Founder & CTO at Basecamp & HEY, NYT best-selling author):

DHH is correct, as usual. I have never met one business customer who actually read (or even run) cucumber tests.

Once we established that, it is clear that Cucumber tests have little value for customers (and, therefore, business analysts). From the technical perspective, the effort to support the extra layer of test-specific parser for English is going to cost the team a lot. (please note, again, there are no or little business values of doing that)

Some might still argue: “I disagree. The creator of Cucumber and DHH were wrong, I implemented Cucumber successfully on the last project”. For every person who said this to me, and when I had the chance to assess their work, just all plain lies. Think about it, “the last project”, how about this project? Show us. For web test automation, the knowledge is pretty much fully transferable. For every new project, I created at least one working core test (given the environment is ready) and mostly ran in a Continuous Testing server (e.g. BuildWise), on the first day yes, you read right, on Day 1.

What is the main technical reason for Cucumber (i.e. Gherkin) failing on test automation? Test Script Maintenance. The first few test cases are easy to do and relatively simple, and static (e.g. login, sign up). Some get excited, executable specifications that like English, woo-hoo! With more tests, as we know, test cases may get more complicated, and the maintenance effort (existing and new) will grow quickly (exponentially if lacking the tools and capability to meet the challenges). That’s the case for all UI test automation, even with good syntax frameworks such as RSpec.

If someone thinks maintaining automated UI tests is easy, please read this interview from Microsoft Test Guru Alan Page: “95% of the time, 95% of test engineers will write bad GUI automation just because it’s a very difficult thing to do correctly”. If you think GUI automation is easy, then you are at 1 out of 400 SET (software engineers in Test) level, according to Alan Page. Please note, this is judged by the Microsoft Engineer standard. How many test engineers in your city (or state) can meet the standard? Not many, right? Only 1/400 of them can do real GUI test automation well, and they probably won’t think it is easy. This is not just Microsoft’s view, Google VP Patrick Copeland said this: “In my experience, great developers do not always make great testers, but great testers (who also have strong design skills) can make great developers. It’s a mindset and a passion. … They are gold”.

Some might feel intimidated, “if test automation is that hard, we probably shall give up, just do manual testing”. Please don’t, without a solid E2E test automation, there is no real Agile. Test Automation (and later Continuous Testing) is challenging, but not impossible. Face the challenge, and take proper actions such as seeking help from a qualified test automation coach (self-learning is possible, but will be more difficult), choosing correct frameworks (such as raw Selenium WebDriver + RSpec), highly-effective testing tool such as TestWise and dedicated Continuous Testing server such as BuildWise. (By the way, you can start with all the above frameworks/tools, FREE). It can be a very rewarding journey, with positive results in a short time.

However, it is very easy to make bad decisions along the way, choosing Gherkin syntax is one of them.

Cucumber tests will require more maintenance efforts, with that extra useless (except for demos) layer, a lot more. That’s why, from my knowledge over 10 years, every Gherkin automation failed.

A recent big failure was at a large finance organization (claiming Agile for over 12+ years) in my city. Its Gherkin solution (in Java) was so bad that about 3 times of all development effort (time and money) were spent on trying to maintain those ‘Gherkin BDD tests’. Of course, eventually, the tests were abandoned. The excuse the management used was “the contractors worked on this left, so failed to maintain”. Of course, this was not the truth. The root problem was a bunch of mediocre programmers, who mistakenly over-estimated their knowledge of test automation, and made a bad choice based on a naive idea of executing “Given-When-Then” user stories that Business Analysts wrote. Sadly, these kinds of mistakes are keeping repeating.

Besides the human factors, what are the technical reasons why DHH and the creator of Cucumber are against BDD with Gherkin tests? Below is a comparison of the test tiers (based on Maintainable Automated Test Design) between a good test syntax framework RSpec and the bad Gherkin (when used for test automation).

The extra effort (right graph) comes from the ‘test-specific English parser’, the part DHH was referring to. Let’s look at an example Cucumber test.

1. Test (Gherkin) Layer

Feature: Select Flights 
  As a registered user
  I can select flights

  Scenario: Oneway Trip
    Given I am signed in as "agileway"
    When select oneway trip
    And depart from "Sydney" to "New York" on "07" of "May 2021"
    And click "Continue"
    Then I should see "2021-05-07", "New York" and "Sydney"

2. Step Definitions Layer!

Given /^I am signed in as "(.*?)"$/ do |user|
  sign_in("agileway", "testwise")
  @flight_page = FlightPage.new(driver)
end
When /^select oneway trip$/ do
  @flight_page.select_trip_type("oneway")
end
When /^click "(.*?)"$/ do |arg1|
  sleep 0.5
  @flight_page.click_continue
end
When /^depart from "(.*?)" to "(.*?)" on "(.*?)" of "(.*?)"$/ do |from, to, day, month_year|
  @flight_page.select_depart_from(from)
  @flight_page.select_arrive_at(to)
  @flight_page.select_depart_day(day)
  @flight_page.select_depart_month(month_year)
end
Then /^I should see "(.*?)", "(.*?)" and "(.*?)"$/ do |t1, t2, t3|
  the_page_source = driver.page_source
  expect(the_page_source).to include(t1)
  expect(the_page_source).to include(t2)
  expect(the_page_source).to include(t3)
end

3. Helper and Pages Tier

Helper: support/step_helper.rb, included in support/env.rb

def sign_in(user, pass)
  driver.find_element(:id, "username").send_keys(user)
  driver.find_element(:id, "password").send_keys(pass)
  driver.find_element(:xpath, "//input[@value=\"Sign in\"]").click end

Page class: pages/flight_page.rb

require File.join(File.dirname(__FILE__), "abstract_page.rb")
class FlightPage < AbstractPage
  def initialize(driver)
    super(driver, "")
  end
  def select_trip_type(trip_type)
    driver.find_element(:xpath, "//input[@name='tripType' and @value='#{trip_type}']").click
  end
  # more functions ... 
  def click_continue
    driver.find_element(:xpath,"//input[@value='Continue']").click
  end
end

Please note the helper and page classes, if designed well, can be 100% reusable, regardless of what the top syntax framework you use, Cucumber, Capybara or RSpec.

As a comparison, below is the test (top) layer for RSpec.

 before(:all) do
   sign_in("agileway", "testwise") 
 end
 it "One-way trip" do
   flight_page = FlightPage.new(driver)
   flight_page.select_trip_type("oneway")
   flight_page.select_depart_from("Sydney")
   flight_page.select_arrive_at("New York")      
   flight_page.select_depart_day("07")
   flight_page.select_depart_month("May 2021")
   flight_page.click_continue
   expect(page_text).to include("2021-05-07 Sydney to New York")
 end

There is no middle tier (helper/page class tier is the same), therefore, the test script is much easier to maintain. You can get the above test scripts from Github. For a more comprehensive example, see this article: WhenWise Regression Test Suite Reaches 500 Selenium tests and ~300K Test Executions.

RSpec is the most popular “Behaviour Driven Development for Ruby”. RSpec v3.8.0 alone has over 193 million downloads on RubyGems. While RSpec may also be used for unit or integration tests, its download count is quite impressive. As a comparison, the most-downloaded Cucumber v3.1.2 is merely 8.8 million.

Cucumber is not the first failed test framework that uses English-like syntax for automated testing (it may be for other uses, but definitely not real test automation). Do you still remember FitNesse (it was quite big about 10 years ago, an example here)? Now it is hardly mentioned.

Some frustrating Gherkin ‘test engineers’ might grudge: “Maybe you just don’t understand Cucumber”. Sorry, I do know Cucumber well.

Cucumber was developed in Ruby; I am a winner of the 10th Ruby Award. I also worked for many years as a senior software engineer (contractor) using Java, C# and JavaScript.
TestWise, a next-gen functional testing IDE I created, supports Cucumber. (Kent Beck, the father of Agile, once said: ‘I hated the idea so I had to try it.’)
BuildWise, an international award-winning Continuous Testing server I created, supports executing Cucumber tests too.
It shall be fair to say that my Cucumber/Gherkin knowledge is better than most ‘cucumber automation engineers’.

Real functional test automation is far more than a fancy demo. If you truly believe Gherkin automation tests are the way to go, please do it well, don’t ruin the reputation of test automation. Make test automation visible and relevant to the team daily, enabling the team to release to production multiple times a day. That’s what I can do with raw Selenium WebDriver tests in RSpec for my web apps: ClinicWise, SiteWise, and WhenWise.

If you are an architect/manager of an organization doing Cucumber Test automation (usually fooled by fake agile coaches), I suggest you write a concerned email but not too direct and save it (that’s important) or even start preparing your scapegoat. When the shit hits the fan, you can say: “I told you so” or “that fake agile coach’s fault’.

FAQ

1. Do you mean BDD is just hype?

No, Check out BDD Clarified: BDD ≠ “Given-When-Then” (Gherkin).

2. My team is trying to introduce Gherkin, what can I do to stop that?

Check out A Practical Advice on Rejecting Gherkin for Test Automation.