What E2E testing framework are you using?

Photo courtesy of Flickr user dkuperman

‪What E2E testing framework are you using these days? I'm going to be looking into Puppeteer and Cypress.

I like that Puppeteer uses async/await in their API, but you're limited to just Chrome. Cypress is coming out with Firefox support, so +1 to them.

In the past I've used webdriver.io and other Selenium based ones, but webdriver based ones are generally slower.

Comments

[Shi Ling]

I use UI-licious for E2E testing.

Disclaimer: I'm one of the creators, so I obviously have an Opinion™ ;)

I've used UI-licious to test UI-licious itself. And for a very convoluted reason, also used it to automate Docker deployments for UI-licious via Rancher's UI, but... that's leave that story for another time.

Free and open-source tools that I like

Codecept: The biggest win for me here is how easy to learn and readable the tests are. This makes the tests really easy to maintain, which is important to me as I worked on enormous projects with heavily complex user interfaces and very deep user journeys. From time to time however, I find Codecept not as elegant is I hoped it to be, and had to fallback to hard coding CSS selectors and using magic waits to get around async events. And as far as I know, there isn't a way for me to split my tests and run them in parallel to finish my enormous test suite faster. It's still the best free open-source acceptance testing framework I know of, and I'd recommend it.

Webdriver.io: This is what I end up using when I can't wraggle Codecept into testing certain cases for me. Webdriver.io is the underlying library for Codecept.js, and exposes Webdriver protocols for you to do the really nitty gritty things, but it's not something I'd recommend for everyone and every project.

Cypress.io: What I really like about Cypress is that conceptually they are building the architecture from the ground up rather that simply creating yet another a framework that adds syntactic sugar over Selenium, which solves a lot of problems that made tests flakey. Reports look great. And syntax is like Mocha, so it should be easy to learn. It is free unless you want to better reporting, which will start from $99/month after beta. Which means that there's someone working their ass off full time to make sure this testing tool works - reliably - because they are not starving open-source devs. The only deal-breaker might be that they only support Chrome at the moment.

Warning, opinion™ ahead

I created UI-licious because I have an opinion on how E2E front-end testing should be that was not being fulfilled by other testing frameworks:

1. You shouldn't be hard coding tests to the UI: And hard-coding CSS selectors and magic waits are a huge no-no to me. Hard-coding is a code smell, and guess what? Tests are code! Hard coding is what's making your tests flakey. And in huge applications, ID-ing all elements for testing is a pain, and the UI designs are probably going change (and the IDs will too).

2. E2E testing is about the users' journey: Making sure your app works isn't simply about filling forms and pressing buttons. It's really about whether the user get what they want from your app. That's why I prefer BDD-style tests. They describe the user's journey and are essentially living-documentation of your app. This makes it really easy to author and maintain tests. I'm no fan of the PageObject concept which comes from Selenium because it's difficult to read and maintain. And if you go to Selenium's website, you'll find that it's not tool meant for testing, because here's what it first says "Selenium automates browsers. That's it!". It's simply a browser automation tool. Also, while the modern async/await is great my front/back-end work, it makes tests terribly unreadable, and using a Javascript testing library that does co-routine sync like Codecept would be better.

What I've done with UI-licious is to create tests that describe the user journey from their perspective, without any knowledge of the actual implementation of the UI.

Here's what logging into Github looks like:

I.goTo("https://github.com");
I.click("Sign in");
I.fill("Username or email address", "brucewayne");
I.fill("Password", "iloveselina");
I.click("Sign in");
I.see("brucewayne");

It's simple, concise, readable.

And independent from your UI code, so you can stop fixing broken tests. And this means you can write less and test more, because the same test can be used for multiple designs. Just write one test for testing the same flow on desktop or mobile.

UI-licious works by using static code and contextual analysis to infer which element on the screen to test.

But what if there's several things on the page that looks the same, how does the test engine know what do interact with? We've got that covered.

And that's my two cents.

[Valentin Berlier]

Wow that's really neat! I didn't know about UI-licious before I just checked out the website and it looks really cool. I'm definitely bookmarking it for the next time I'll need to do E2E testing.

[Theqadiva]

So, does it integrate easily with CI/CD pipelines? We use VSTS and, while this seems to be an easy-to-use, easy-to-maintain product, will I be able to use this with VSTS for CI/CD?

[Shi Ling]

Yes, you can use the CLI to trigger test runs as part of your CI/CD pipeline, like this:

npm install uilicious-cli -g
uilicious-cli run <project_name> <test_path> -u <username> -p <password>

Here's the docs.

And for the lazy, UI-licious also has built-in monitoring features for you to schedule tests and send email / Slack / webhook alerts when an error is found.

[Arnaud Morisset]

I used NightwatchJS on previous projects, but I'm tired of Selenium/Java dependency (and I don't really like the Nightwatch API). Now, I mostly use a library built by one of my coworkers: Wapiti.

It's based on Puppeteer, have a clean API and that do the job.

What else? ;-)

fenntasy.github.io/Wapiti/

github.com/Fenntasy/Wapiti

[Nick Taylor (he/him)]

Thanks for posting Arnaud, A+.

So, so far you're team has been enjoying Puppeteer? I imagine your co-worker wrote Wapiti to wrap some repetitive code your team was writing with Puppeteer?

[Vincent Billey]

Author here :)

There are still some pain points with Puppeteer (inside Docker is not optimal and I can't install it on windows for a friend) but it is sooo much better than selenium.

To be honest, my main focus writing Wapiti was to deal with VCR in tests. Puppeteer is great for running things and I added a mode to change fetch to be able to save API calls and to redo them in a CI environment (we had the problem with a frontend and an API in separate projects, testing in CI by cloning the other project and launching it was not really usable).

After that, I added a feature that I personally like: the possibility of "capturing" several value inside one test which I show in the first example in the doc:

test("it should get the content of elements of the page", () => {
  return Wapiti.goto("http://localhost:3000/index.html")
    .capture(() => document.querySelector("h1").textContent)
    .capture(() => document.querySelector("h2").textContent)
    .run()
    .then(result => {
      expect(result).toEqual(["content of h1", "content of h2"]);
    });
});

Using capture only one will produce a single-value result but using it several times will produce an array of value.
As EtE tests are quite expensive, I wanted to be able to test several things at the same time on a page.

As a bonus, Arnaud made me add a way of dealing with tabs ^'

And the future goal is to be able to start a local server while running tests to embark everything needed to test a pure frontend project.

[Nick Taylor (he/him)]

You should write a post on why you created Wapiti if you haven't already. 😉

[Vincent Billey]

It's in my todo 🙂
But in the meantime, I wrote a longer explanation here fenntasy.github.io/Wapiti/docs/why...

[Oliver Libutzki]

I use tapir for E2E testing.

Disclaimer: I am one of the tapir core developers.

With tapir we address the problem of unstable and unmaintainable UI regression tests. It helps the user not to write boilerplate code to have stable tests. There is no need for waits in order to synchronize the test and the System Under Test (SUT).

We invested a lot of time evaluating existing solutions, but none of them pursued our vision how to develop UI regression tests for complex business applications.

tapir is not that wide-spread so far as it has been launched last month. If you are interested consult the tapir website tapir-test.io/.

[Dinesh Pandiyan]

I've used Selenium in the past, but Cypress and TestCafe are the E2E frameworks to lookout for these days.

Cypress looks all great and polished, works very well for basic setup and faster than other frameworks but the documentation isn't very clear when you run into issues or need to look something up.

TestCafe on the other hand, might have an unpolished site, but their documentation and the framework is really good and reliable. We're using TestCafe in more than one project at work, especially because we need to test file uploads and TestCafe does it very well.

[rhymes]

I'm yet to start doing e2e on my new project but I'm going to start from Cypress

This discussion contains a few pointers:

Which framework use for an e2e tests of an SPA?

Rafal Pienkowski

#discuss#webdev#testing#question

[DimaKramskoy]

I would start from Puppeteer , as it feels more code oriented , rather than using a framework/saas platform which for me Cypress is .. today I am using the "old" but working nice set of CasperJS and PhantomJS to cover our e2e UI suites ..

I know that PhantomJS is an old and there is no versions will be released , since the new headless Chrome is taking over the market .. but it does the work and it is pretty easy to develop and include into the CI pipeline which is CircleCi for us ..

[Andre Vasconcelos]

Been experimenting with Cypress a bit and loved it so far, setting up tests didn't take too long and I got to see what the web pages looked like as the tests were running in real time.

When support for Firefox comes out things will really start looking good

[Nick Taylor (he/him)]

Nice to hear it's going well so far. Thanks Andre!

[Christian Vasquez]

Other than Selenium, I find TestCafe to be quite good with lots of browsers support.

[Nick Taylor (he/him)]

Thanks Christian, I'll take a look at it.

[Tiago Celestino]

At work we’re using Codecept (codecept.io/), it’s supports in our template server.

[Jonathan Boudreau]

At work we're going to be using WebdriverIO with fibers enabled.

[Ibne Nahian]

Sticking with Selenium yet.

[art]

Don't use any frameworks for UI and e2e. Don't clone shit you will go crazy after number of tests grows.
Use C#\Java + custom page object based framework + selenium + NUnit (or other test framework).

All this e2e "frameworks" are not about real testing and maintenance, but about 100500 methods of access to html and 100500 steps in a test case, instead of 5-6 methods like

Dashboard.Open();
var rows = Dashboard.OrdersGrid.GetRows();
Assert.IsNotEmpty(rows);

[Ben Polinsky]

I couldn't disagree more.

1) Selenium is a framework.
2) Custom page object based framework is... a framework.
3) No user says "Dashboard.Open" -> They click links, observe changes, and react to those changes.

What are you actually testing in your example? That there's a Grid with Rows? Does the user care if there is a Grid with Rows? No. They care that there's information/data inside of the table or grid or whatever that they can observe and/or take action based upon.

If you switch from a Grid to a Table or to a different way to display data, you're rewriting your test for no reason.

[art]

1. Selenium is not a framework. It's a library and\or tool which gives you an access. It's a finished thing for end user. It's a difference.
2. Page Object is not a framework. Looks like you are not familiar enought with architecture and design patterns in general. Page Object is a pattern. Like Factory, Repository, Specification, Wrapper and others.
3. It absolutelly doesn't matter how do common people name it. Today they name it with this verb and noun, tomorrow with another (if they don't have a vocabulary of the project. And in 9\10 times they don't). The only next things metters in software development and\or automation testing:

a) Easy maintenance
b) Clear and simple as much as possible expression of thought through commands. To reduce number of possible errors which will be made next developer after you. It's McConnel.
c) contineous refactoring

Page object and Dashboard.Open is 100% more easy to sopport and refactor then
a series of consecutive commands with parameters like:

selenium.Find("[name='submit']").click();
Asser.True(selenoum.Find("[div .someclass > div .someother class]").Display)

Or analogs from this shitty frameworks for e2e testing

Does the user care if there is a Grid with Rows? No.
I am amused by such youthful absolutism. Sure they care that it's a grid and rows. Becuase grid is convinient approach to display structured information. And people see a well known tool, having skills how to work with it.

If you switch from a Grid to a Table or to a different way to display data, you're rewriting your test for no reason

First. If control has changed - the test should be changed. Must be changed. Otherwise there was no need to change control. Be we did it for some reason.

Second. If the framework is made by tester like you, then 99% -may be yes.
While it would be my framwork, then only difference will be the change of the interface implementation. Like:

void OpenDetails(int number)
{
IGridTableObject grid = new Grid()
grid.CheckRow(number)
}
to
void OpenDetails(int number)
{
IGridTableObject table = new Table()
table.CheckRow()

}
And for final test it will the same:

Dashboard.Open();
Dashboard.OpenDetails(123);

which is made with 2 cliks with Resharper.
Because I don't loose the Liskov principles in code.

Generally developers are much better in code writing, system deising and so on, then test automation teams, becase we desigin system everyday for years. And my experience was to work as framework developer for automation test developers for last 3 years just because I'd liked to improve the quality, while the number of people is limited and we required own framwork. Why I judge so? Because I see how my subordinates, pure qa autmation developers, write code and design. It's like a sky and a ground.

[Ben Polinsky]

Your reply is pretty offensive. I made no assumptions about who you were or your experience level. I'll leave it at that and get back to talking about code.

1 Whether or not Selenium is a framework, I don't care. The wikipedia entry for it says it is. Some people say it's not.

2 You specifically in your original post say "custom page object based framework" and "NUnit (or other test framework)" - that's all I was pointing out. You like your tools and others like their tools. They're just tools for accomplishing a goal. Your way is not "right" nor is mine. As long as they get the job done, who cares.

3 E2E tests ARE about the user experience and not about your code. If a user should care that data is structured in a certain way (say a table, or a data row, or some 3rd party data structure yet to come out) then test that behavior. Don't test the implementation details. If a user should be able to sort data - test that. If a user should be able to filter data - test that. If you're abstracting that behind a Page Object - great! It's one way to abstract things.

Users don't know about code. They don't know that a Dashboard object is being opened. They enter URLs and click links.

I don't care how you write tests as long as they are fast and give you the confidence that your application is working.