In this article, I want to talk to you guys about Top 3 mistakes that I have seen every test automation engineer makes or have made at some point in their career. So let's get started -
Mistake #1: Using "Sleep" in your code -
This is one of the most common mistake I have seen everyone makes whenever they need to "fix" a test. Now, may be you already know or have heard that using "sleep" is bad in your code but let's look into why exactly this is bad -
Tests take longer to complete
So this one is obvious, you are adding hardcoded sleep of course it will take longer to complete. This is important because you want faster feedback right, that's the whole point of Agile approach and if few of your tests are taking like 30 mins to an hour or even more, that's adding a lot of extra time in your build pipeline.
And if you are thinking I know I am adding this 2 second here, who cares no one will notice, well that takes us to our next point.
Using "sleep" without being aware of it
In the image below, we are using a method called .open() and right after that we are doing a 2 seconds wait to make sure the page loads.
But, let's understand what is happening inside the .open() method. Here, we are waiting for 2 seconds again. So most likely, the person that added 2 seconds wait in the previous wasn't aware of the wait that's already within the method.
While the total 4 seconds wait might not seem that bad but think of large projects where you have 10s and 100s of files, it will be a big problem if you are using sleep commands everywhere.
Makes your tests unstable (flakiness)
Adding sleep makes your test unstable because you don't know how long to wait for a particular page or an element to load. Taking a look at our previous example once again -
- Person A added 2 sec when they wrote the original test
- Person B had issues with 2 sec due to slowness so they ended up adding 2 seconds more
Now imagine if you are running your tests in the env that is slow then these tests might fail again, so all you are doing is going back and adding more time to your tests which takes us back to this whole problem once again!
Ok by now I hope you have realized the problems with using sleep commands, so what should we do instead?
99% of the time you can replace 'sleep' commands with appropriate 'wait' commands
And if you are thinking why only 99%? That's because you might run into some scenarios where the wait commands are just not working, and this is really extreme case but I will be honest that yes sometimes you will encounter those. In that particular case it is fine to use sleep but come back to that problem again and think if theres a better way to implement that solution.
Mistake #2: Over Complicated Tests
Another common mistake that I have seen over the years is writing over complicated long & complex test as shown in the image above. One key thing to notice in the above image is that at the bottom we have 180k ms i.e. 3 minutes of timeout added as the test takes that long to complete.
So if you are writing tests like this then lets talk about the disadvantages of writing such tests -
No idea what test is trying to do
So this one is funny because there are times where I have written long and complex tests and in couple of months when I came back to it, I had no clue what my test was trying to do. And of course, you can imagine how other team members would feel when they read this kind of code (all I can say is I don't want to be near them at that time!)
Long time for test to complete
This is obvious, when you write long tests, it will take long time to complete as well that's why we saw that 3 min timeout in the image above.
Longer tests causes test flakiness
What happens when we write long tests? Well, long tests are generally lot more unstable because simply theres a lot of things going on and due to that, it has a lot more chances to fail.
Difficult to debug the code
Which takes us to our last point, when the tests fail then oh boy! Good luck trying to debug this. Essentially, you will be running a test which takes 3–5 mins to complete and you are trying to find out in which line where exactly the issue is and how you can fix it. If you haven't run into this problem then I will say you are lucky as this is quite painful to work with.
So what should we do instead? Well here's what I think -
Test should focus on doing 1 thing at a time.
Now, don't take this statement to heart, by 1 thing it could be something that you and your team decides - can be 1 feature, 1 component, 1 E2E flow which completes in a reasonable amount of time (ideally less than a min).
As long as the test has a single purpose which everyone understands, I think that's good enough to work with.
Mistake #3: Test Dependency
In the example above, the second test is dependent on the first test as that's where we open up the url for the page we are testing. This is bad because of few reasons:
Unable to run individual test on failure
If the second test fails due to some reason, you will not be able to run that test only as it depending on the first test where we are opening up the url. The only options you have is to run both the tests which will take longer time to execute or you will have to refactor your tests which we will talk about shortly.
Changing the order of the test will cause the test to fail
If someone else comes and simply change the order of these tests, it will start failing again because their test buddy is not in the same order as before. This is another big issue as now you will need to know the order of each of these tests to run them in future.
Makes it difficult to refactor the code
Now, when you do decide to make refactor in your tests, it would be quite painful as you will need to understand how all these dependencies work and have to fix all that to be able to do any kind of refactor which will end up taking a lot more of your time.
So what should we do instead?
Tests should be Isolated / Independent.
Your goal should be to write tests which can be ran individually without relying on any other tests or even any other data. This will give you a lot more flexibility if you want to do any refactor or simply reorganize your tests in future.
Let's Review
Let's do a quick sum up of everything we covered in this article -
- Avoid using 'Sleep' in your code
- Do not write long and complex tests
- Tests should not depend on each other
Hopefully if you avoid these mistakes you can create a stable and efficient test framework.
If you enjoyed this article & would like to learn more about Test automation & Best Practices then you should check out my new course on WebdriverIO where I teach you how to do Web Automation using the industry standard best practices.
You can check out the promo video of the course below:
Top comments (5)
Awesome.. I don't use sleep but I learnt a thing.. 👏
I don't back the point of having e2e test cases less than a minute.
Let's talk about practical scenarios like...
Account creation
Application setup
Order payment flow ...
How in the world can we have e2e tests that too UI tests take less than a minute ..
Its the ground reality .. We can set up few tests with some seeded data.
But end of the day there are going to be long running user flow test cases in any typical automation suite
Thanks for the feedback! I agree with your point that not all tests can be less than a minute and that's why I said "ideally" try to keep it less than a minute. The main focus is to make your test do 1 thing instead of mixing multiple flows together.
In some of the examples above, I've seen people do this ->
All these in 1 test when they wanted to only test the payment flow and that ends up making your test around 4-5 mins now. So, we need to be careful of such scenarios when writing e2e tests.
And, to your point "How in the world can we have e2e tests that too UI tests take less than a minute .." - with majority of the JavaScript frameworks nowadays, tests runs extremely fast and if setup properly by taking advantage of app state + mocked/seeded data, you can easily keep them under a minute including the flows you have mentioned above.
Still, the point of the article is not the "time" of the tests but more so knowing what exactly your tests are doing. If you can have long running stable tests and it works for you and your team then that's what matters at the end of the day.
it's impressive, I am new to selenium and thinking that I always make these mistakes!
thanks for your insightful article 👏
Thanks for reading!