Pytest vs Cypress: A fair fight in UI testing?

Right now, we're witnessing the greatest disruption of the test automation industry that we've ever seen. New tools being developed, and existing tools are getting significant improvements. Today, let's talk about two major test frameworks that have lots of users according to GitHub:

Some of you may already be familiar with cypress: a known framework for end-to-end testing. Unlike cypress, pytest is mostly known as just being a unit-testing framework (for Python). Less known about pytest is that it has a major plugin ecosystem for expanding its abilities. Just like Batman has a tool belt for letting him do things that he can't do with his bare hands, pytest has plugins for allowing it to do much more, such as browser testing, multi-threading tests, and creating test reports. Before we can go into the pytest vs cypress comparison, let's talk about an essential component needed for UI testing with pytest, which is the selenium browser automation library.

In the past, you've probably heard of comparisons being done between cypress and selenium. This raw comparison might not have been fair because Selenium clearly states on their website: "Selenium automates browsers. That's it! What you do with that power is entirely up to you." (source: selenium.dev) Nowhere on their website did Selenium say anything such as them being "a complete test framework", etc. They specifically wanted to give developers and test automation engineers the ability to create their own test frameworks that use selenium. A more fair comparison would be cypress vs a test framework that uses selenium.

Fast-forward to today: Lots of companies want to get a lot of testing done very quickly, but they don't have the time they need to build a complete UI testing framework for their testing. This has led to a lot of corner-cutting (aka sloppy code), and false blame against perfectly solid automation libraries that work great when used properly. One thing to keep in mind when using raw Selenium methods is that if you try interacting with page elements that haven't fully loaded (or elements that haven't been re-found after a page reload), then you may see errors like this:

• NoSuchElementException
• ElementNotVisibleException
• MoveTargetOutOfBoundsException
• StaleElementReferenceException
• ElementNotInteractableException

These errors are completely preventable if your test framework wraps selenium methods with the appropriate code that smart-waits for page elements to be fully interactable before interacting with them (or timeout/fail the test if the element still doesn't appear after a set number of seconds). The good news is that a test framework with selenium smart-waiting already exists as a pytest plugin: SeleniumBase!

SeleniumBase bridges the gap between pytest and selenium for UI testing. With pytest, you have a unit-testing framework without a built-in web automation component. With selenium, you have a web automation library without a built-in test framework. Both sides of the bridge have long been very popular and very powerful, but now there's a special bridge that seamlessly brings the two sides together in one very powerful, but simple, package. And this package, SeleniumBase, exists as a pytest plugin, which uses the selenium library effectively.

Here's an example SeleniumBase test:

from seleniumbase import BaseCase

class MyTestClass(BaseCase):
    def test_basic(self):
        self.open("https://store.xkcd.com/search")
        self.type('input[name="q"]', "xkcd book\n")
        self.assert_text("xkcd: volume 0", "h3")
        self.open("https://xkcd.com/353/")
        self.assert_title("xkcd: Python")
        self.assert_element('img[alt="Python"]')
        self.click('a[rel="license"]')
        self.assert_text("free to copy and reuse")
        self.go_back()
        self.click_link_text("About")
        self.assert_exact_text("xkcd.com", "h2")

That test is from the latest version of SeleniumBase/examples/my_first_test.py, which you can run using pytest:

pytest my_first_test.py

As you can see, it's very easy to open a URL, type text, click, and assert various elements on a page. For a more advanced example (with hovering dropdowns, iFrames, select options, sliders, finding broken links, and detecting JavaScript errors), see SeleniumBase/examples/test_demo_site.py

Now let's compare that with a Cypress example:

describe('My First Test', () => {
  it('Gets, types and asserts', () => {
    cy.visit('https://example.cypress.io')
    cy.contains('type').click()
    cy.url().should('include', '/commands/actions')
    cy.get('.action-email')
      .type('[e-mail]')
      .should('have.value', '[e-mail]')
  })
})

Aside from the obvious differences that Cypress tests are written in JavaScript, and SeleniumBase tests are written in Python, you'll see that both examples are written using simple code, both examples are easy to run, and both examples run reliably.

For test reports:

• cypress has the Cypress Dashboard Service.
• pytest has the pytest-html plugin, which SeleniumBase modifies to include screenshots of failing tests at the moment they failed:

SeleniumBase doesn't have its own dashboard service, but you can easily run your SeleniumBase tests from any cloud service you desire. For open-source projects, I've personally used three free tools: First Travis-CI, then Azure Pipelines, and finally GitHub Actions, which allows you to keep all your work and test results in GitHub. Here's an example of SeleniumBase results from GitHub Actions:

When using SeleniumBase to test applications that aren't open source, I've primarily used Jenkins. If you're doing test automation on a tight budget for closed-source projects, the most cost-effective Jenkins solution I've found so far is Google Cloud's $13.61 USD/month Bitnami Jenkins instance, which supports unlimited users and unlimited test runs (limited only by its CPU, which is enough to handle five simultaneous browser tests without any noticeable lag). As for Cypress tests, those can also be run with GitHub Actions and on Jenkins.

Now that we've covered the basics, lets move onto limitations when it comes to Cypress vs Pytest. Cypress actually has a page on its website dedicated to its limitations, aka "trade-offs": docs.cypress.io -> trade-offs.html.

These Cypress limitations include:

• Limited iFrame support.
• No multiple tabs.
• No multiple browsers at the same time.
• Each test is bound to a single origin domain.
• Missing mobile support.
• Not a general purpose automation tool.

With pytest and SeleniumBase, all of the following features exist:

• Full iFrame support.
• Open as many tabs as you want in tests.
• Open as many browsers as you want in tests.
• No limitations on multi-origin domain navigation.
• Uses Chrome's mobile-device emulator for mobile tests.
• Can be used for general purpose automation.

The only limitation with SeleniumBase is that you must use Python! (And for Python users that's not a limitation... that's a feature!) The good news is that: You can also make JavaScript calls with SeleniumBase!
This is done with:

self.execute_script(JAVASCRIPT)

This ability to execute JavaScript inside SeleniumBase Python scripts has led to some powerful new features, such as the website tour builder that's part of SeleniumBase (learn more about the SeleniumBase tour-builder here):

As for SeleniumBase's mobile mode, just add:
--mobile to the pytest run command of any SeleniumBase test to run it using Chrome's mobile device emulator:

The fastest way to get started with SeleniumBase is by cloning it from GitHub so that you can run all the included example tests, but you can also install seleniumbase from PyPI with pip, then do:
seleniumbase mkdir TEST_FOLDER to create a folder anywhere that includes example tests for you to run.

At the time this article was published, the latest version of seleniumbase was "v1.40.2", so if you're using an older version of SeleniumBase, some of the features mentioned here might not exist until you upgrade.

Here are some links to material discussed in this article:

• Cypress on GitHub
• Cypress Docs Site
• SeleniumBase on GitHub
• SeleniumBase Docs Site

Now that you've had a chance to look at both Cypress and Pytest (with the SeleniumBase plugin), you must decide for yourself where to go from here. I'm biased of course (being a Python developer). I started learning Python in 2008 at a company called ITA Software, which was later acquired by Google. And I started learning Selenium in 2011 at a company called HubSpot, which funded the creation of the test framework that I would later rename to SeleniumBase in 2014 after the project got approval to be open-sourced. (Trivia: I decided on the name "SeleniumBase" on the day Google acquired Firebase. I heard about that acquisition on the news, and I liked the sound of "Firebase", so I took that idea and added "Base" onto "Selenium", and that became the name of the framework ever since.)

I hope that you got a few key takeaways from this article: Mainly, that when used correctly, Selenium is awesome for UI testing. When paired together with pytest (and a good "bridge" like SeleniumBase), Selenium is even more awesome. People have spent a lot of time unjustly trying to replace Selenium, when in reality, all they needed to do was replace their implementation strategy.

Thank you to both the pytest and Selenium teams for all the hard work you've done in making great testing tools. Here's a shout-out to some of the major contributors of these projects: Simon Stewart, Bruno Oliveira, Jim Evans, Ronny Pfannschmidt, David Burns, Florian Bruhin, Dave Hunt, Jason Huggins, Diego Molina, and many more key contributors that I haven't listed here. There will always be competition in the test automation space, but together, we can continue to create a legacy for test automation engineers of today and the future.

Comments

[codewander]

What do you think about cypress's arguments about being able to more precisely detect browser events since it runs inside of the same browser that is running app under test? I have been wondering if there is any inherent advantage there or if that only helps for cypress component tests.

[Michael Mintz]

Selenium/WebDriver has access to chromedevtools.github.io/devtools-..., which allows it to detect and control browser events precisely. No advantages there if both frameworks can see the browser events equally.

[codewander]

I have been trying to understand statements like this (taken from cypress docs):

Most end-to-end testing tools are Selenium-based, which is why they all share the same problems. To make Cypress different, we built a new architecture from the ground up. Whereas Selenium executes remote commands through the network, Cypress runs in the same run-loop as your application.

...

Cypress knows and understands everything that happens in your application synchronously. It’s notified the moment the page loads, the moment the page unloads. It’s impossible for Cypress to miss elements when it fires events. Cypress even knows how fast an element is animating and will wait for it to stop! Additionally, it automatically waits for elements to become visible, to become enabled, and to stop being covered. When pages begin to transition, Cypress will pause command execution until the following page is fully loaded. You can even tell Cypress to wait on specific network requests to finish.

[Amy St. John]

I'm curious, have you tried out elsnoman.gitbook.io/pylenium/ ? I'm really interested to know how it compares to SeleniumBase, but since Pylenium is so much newer there doesn't seem to be anyone talking about how it compares. I'm curious if the only different is that it's lighter weight than SeleniumBase at this point. They both seem to accomplish the same thing of making end to end tests more simple to write with Python and Selenium. What are your thoughts?

[Michael Mintz]

Hi Amy, In terms of lightness, both are light weight (under a MegaByte). SeleniumBase can do far more though: UI Testing, Visual Testing, Website walkthroughs, Charts, Presentations, etc. And the syntax when using the sb pytest fixture is very clean and flexible:

# "sb" pytest fixture test in a method with no class
def test_sb_fixture_with_no_class(sb):
    sb.open("https://google.com/ncr")
    sb.type('input[title="Search"]', 'SeleniumBase\n')
    sb.click('a[href*="github.com/seleniumbase/SeleniumBase"]')
    sb.click('a[title="seleniumbase"]')


# "sb" pytest fixture test in a method inside a class
class Test_SB_Fixture():
    def test_sb_fixture_inside_class(self, sb):
        sb.open("https://google.com/ncr")
        sb.type('input[title="Search"]', 'SeleniumBase\n')
        sb.click('a[href*="github.com/seleniumbase/SeleniumBase"]')
        sb.click('a[title="examples"]')

See seleniumbase.io/help_docs/features... for the list of features.

[pawan deore]

legends are using playwright

[Michael Mintz]

Another strong competitor. It would be interesting to see a Playwright vs SeleniumBase competition on the Python side. The "legends" are on all sides of the battlefield.