Puppeteer vs Selenium

Klaus on July 15, 2019

Puppeteer and Selenium are both popular, powerful and widely used solutions for automating web applications. But which one should YOU use? The an... [Read Full]
markdown guide
 

Hey, your link above isn't working for me - SSL error. Just FYI

 

I don't know which link you're talking about, all of them work.

 

oh, okay. I've retried it and it worked. I ended on the www version (which didn't work). Anyways. Interesting idea

Interesting.
The www version redirects me to the non-www version.
What browser are you using?

Firefox 68. I've just tested it and it doesn't redirect www to non-www for me.

 

Hi!

Just wanted to say that you have a typo in selenium cons:

  • A bit slower than Selenium

should be A bit slower than Puppeteer ?

 

Thank you for pointing that out. Fixed it.

 

@Klaus,

your post sounds quite biased.
1) Every now and then (for the last 20 years) a new tool pops up that promises easy to use test specification without any coding knowledge needed. None of those tools have lived up to their promises.
2) You should compare your tool with Serenity BDD, CodeceptJS or any other high level test suite. Selenium and Puppeteer are merely the drivers that speak with the browser
3) Given that more and more teams learn to work agile, testers will be able to code and coders will be able to test. That renders a dedicated tool for people that can not code useless
4) I can code. I have experimented with a few point and klick test tools. Your approach might be of some use for tiny test suites which have no relevance in the real world. Once a test suite becomes a bit bigger there is no alternative to code tests.

 

Thank you for taking the time to write that comment.

CodeceptJS and Serenity BDD are just some libraries.
Selenium is also just a library.

The open source and zero license cost nature of such libraries is attractive when compared to the license or subscription costs of commercial automation solutions.

But there is more to it than just that when you look at the cost of implementing an automation solution.

For example, Selenium is open source and free, however building and using an automation solution on this โ€˜freeโ€™ software comes with some real costs:

1) The open source tools/libraries provide a starting point but you still have to write code for test creation, management, execution, handling errors and reporting. Note that this is over and above the test logic that you want to automate.

2) If you are looking to test integrations with back-end systems and external systems via API, you have to build in support for that into your framework.

3) If you are looking to implement a continuous testing solution, you must add to the above effort, the effort of building integration with DevOps or CI/CD tools.

4) Also to be considered are the costs of building and managing an infrastructure to run these tests. These costs are likely to go up as you scale your infrastructure in order to support increased test volumes or to increase the pace of testing.

 

It's an interesting implementation of a Keyword/Action driven automated testing framework. I've built similar using Java, Selenium, SauceLab and a few add-on libraries for screenshot comparisons. I could see the pull your platform would have on some QA organizations or Directors of Engineering.

Since you've put your product out here I'm assuming you're cool if I ask questions about your platform.

1) I've reviewed your documentation and what I didn't see was how your test framework handles executing the same tests against different environments or URLs. Current software dev shops have multiple environments that they want to run their automated test against (Dev, QA, Stage, ect). How is this handled in your platform using variables?

2) It would be nice to see a demo of how you can have 1 test or suite execute against against multiple browsers without having to duplicate tests for each browser.

3) Do you have any comprehensive examples of your structured results reporting? Your documentation is lacking in this area and as we all know, upper management folks like pretty pictures. ;-)

4) Does your platform offer a way to call external APIs or programs? When I build my automation for a GUI, I like to pre-populate my tests with data before the UI executions and validations without having to drive the UI all the time. Is this possible?

5) When you integrate your platform with Jenkins, Github or others, are the test results pushed back to the calling platform?

6) With your platform retaining all of test actions, video recordings, screenshots and basically the keys to a software teams kingdom with your constant analyzing the sites/apps, what is your security model to protect your customers data?

Thanks for letting me prob into your platform.

Cheers!
Ken

 

Hi Ken,

I really appreciate the fact that you tried Endtest.

My opinion is that you could have obtained some of those answers simply by digging more into the platform.

But here are the answers:

  1. You can easily run tests on different environments (Dev, QA, Stage, etc) with Endtest.
    Those environments are obviously going to have different URLs, different usernames, different passwords, etc.
    You can use the "Set Variable" action to create or generate different sets of variables and you can use IF Statements to see which environment you're on and to use the corresponding usernames and passwords.
    You can even define variables directly in the API call for starting a test execution, which makes it even easier.
    There are multiple ways in which you can handle different environments, it depends on what works best for you.

  2. You can run the same test suite on any operating system, browser or mobile device.
    All you have to do is select the desired configuration in the "Run Test Suite" modal or in the API call.
    You don't have to duplicate or modify a test suite in order to run it on different configurations.

  3. You can export the test results directly from the Results page. In that XLSX report, you have all the numbers, all the details, everything that can be extracted from a test execution.
    Since it's in XLSX format, you can easily create your own reports in Microsoft Excel by simply selecting the relevant columns.

  4. Yes, you can use the "Execute JavaScript" step to call any API you want or you can use the "Connect to Database" and "Execute SQL Query" actions to write that data directly in your database.

  5. Absolutely, the integration with services like Jenkins and Github is done through webhooks, you just add the API calls for starting a test execution and fetching the results in the Webhooks sections from those services.
    Additionally, Endtest provides it's own Webhook functionality, this allows you to automatically send the results to a URL after each test execution.

  6. All the traffic on the Endtest platform is encrypted.
    The test data from the execution machines is deleted right after the test is finished.
    The Machine Learning functionality from Endtest does not constantly analyse the sites and the apps, it only analyses them while the tests are being executed.
    There are multiple Security Standards that are being followed, including the famous ISO/IEC 27001.

 

Klaus,

Thanks for the quick response. It was very helpful.

 
 

It says:
BEWARE: Experimental. Just for preview. Installation and usage will change.

We can talk about it when it's production ready.

 

Still, it got featured on Chrome's YouTube changelog, so that may indicate some measure of stability

code of conduct - report abuse