DEV Community: Lars Frantzen

Why Tests are Flaky - External Dependencies

Lars Frantzen — Thu, 18 Apr 2019 10:02:51 +0000

When dealing with test automation - especially when doing web testing with a full-blown browser interface - one of the biggest issue is what is called flaky tests. You may also call it unstable tests, and often this notion is overloaded with several meanings. Let us define a flaky test as a test, which leads to different verdicts when being run several times against an unchanged environment and system.

Having such a behavior could mean two different things:

the system really shows different behaviour when running the test several times, even though the environment and system setup is unchanged. So it is nondeterministic in the sense of this post:

Nondeterminism In Testing - How To Do It Wrong

Lars Frantzen ・ Apr 14 '19 ・ 6 min read

#testing #webdev #beginners #computerscience
the test case is not able to bring the system in each run always in the same state for each step it takes

The first option means that the system shows a nonderterministic behaviour. This is a special topic, and usually not what is meant with having a flaky test. With flaky test we usually refer to the second option. And that option can have very many root causes. A common (bad) approach to deal with flaky tests is to rerun failed test cases, and see if they pass the second (or later) time. This takes time, and is not the way to go. Instead, make tests stable. I will note here some practical findings why test cases may be flaky, and what can you do about it.

The root cause I am looking at in this post is having external dependencies within your test cases. In web testing that could mean that your test cases do not only access the web pages and corresponding systems you want to test, but also other pages or services you utilise to execute your tests.

As an example, I was asked to have a look at a test case which was quite flaky. The test case was migrated from a former test tool to our recent, PHP-based tool Codeception. The test case has to compute a hash code for some string. Since the former test tool had no option to compute hashes, the writer of the test case solved the issue by accessing an external web page to compute the hash there. Since this test case was migrated exactly in this manner to Codeception, it did the same there.

Using this external page was the root cause for the flakiness. It was very unstable to wait for, and access the computed result on that external page. So people started adding more waits, and all kinds of weird stuff, which ended up in the flaky helper method to compute hashes. Furthermore you add an dependency to an external page. If that page is down, your tests cannot run.

So what was the solution? In this case, simply use plain PHP to compute the hash. This is one of the great advantages of having a test tool which uses a common programming language. You have the full power of the language available for your tests. This, on the other hand, is also a danger, since people may introduce too complex and unreadable code. But that is a different topic.

So, at the end, I could just replace this messy code with one line of PHP:

This may appear as a very obvious and easy to solve case. And it is, but in general having external dependencies can contribute a lot to unstable tests. They often involve network latency, unforeseen changes of the external system, and so on. So, get rid of external dependencies as much as possible.

There are many more root causes for flaky tests, like wrong session handling, different browser behaviors, inability to deal right with failed test cases, and so on. I will write about those topics in upcoming posts.

Nondeterminism In Testing - How To Do It Wrong

Lars Frantzen — Sun, 14 Apr 2019 10:29:28 +0000

One of the most misunderstood concepts in testing is nondeterminism. So let's first quickly clarify some notions. We are testing a system by executing test cases on it. The test cases are derived from some kind of specification or model of the system. A test case gives inputs to the system, and observes outputs. Based on the outputs, the test case can either pass or fail. Failing means that an output has been observed, which does not conform to the specification out of which the test case has been derived.

A system is always in one or more states. Such a state comprises all aspects which are relevant for the observable behavior of the system. For instance, a coffee machine may be turned on and having enough water and beans to serve a coffee. That is a state of the coffee machine. If I now press the coffee button I expect to get a coffee. If, instead, the machine is turned off, or there is no water, I expect to not get a coffee when pressing the button. These are other states of the coffee machine.

A system is called deterministic when it is always in exactly one state. A system is called nondeterministic when it can be in several states at the same time. And this is where the confusion starts.

People may say "every system is deterministic by definition". This is true in some sense, but very wrong in very many pragmatic testing situations. For instance, imagine that the coffee machine is some commercial one in some office where you do not even see if there is enough water and beans in it. So even though the machine is in one specific state, you simply do not know that state. But you can still deal with it: you press the button, and when you do not get coffee, you ask the service to refill the water and beans (or you just kick the machine and walk away). Doing so is what you can call a nondeterministic test case. It is a test case, which can deal with several states of the system, meaning depending on what the test case observes (coffee or no coffee), the test case proceeds in one direction or another.

Let's look at some more common examples. For instance, consider you are testing an online shop. So you query a product. Now you may get two answers (i.e., outputs): the item is in stock, or it is sold out. When it is in stock, you proceed with testing the order process. If it is sold out, you try another product. Also here, the system is in one state, but you do not know that state. To be able to deal with both states, you need nondeterminism.

Or, consider you are testing a system at some observation point, where you do not know the exact ordering in which events occur. This is very common for complex systems. But you also simply do not care in which ordering some events occur. If, when booting your operating system, your messenger starts first, and then the calendar application, or the other way round, does not matter. But both must start, that matters. Also here you need nondeterminism to deal with it.

So, nondeterminism is a very important and useful means to specify and test systems. Unfortunately, many tools are not capable of dealing with nondeterminism. You can only specify one allowed output in your test case, and all others are considered as wrong. This is a very fundamental limitation of these tools.

But, even worse, when the tool allows to do nondeterminism, people tend to use it wrong. Look at an example from a project I worked for as a test architect: we were developing a web page which allows two ways to log into the site, either via some customer number or via email. You can switch between the two options by clicking some GUI element in the login module. Initially when loading the page it is always in the "login via customer number" option.

While I was working on a test case which first logs a member in (via customer number) I checked the test code for this login method, and saw there basically something like this:

function loginMemberWithCustomerNumber($member) 
{
  if (the login module expects an email)
  {
    click the GUI element to switch to login via customer number;
  }
  login the $member via its customer number;
}

Why did the tester do this? Simple, this function is robust in the sense that it can deal with both cases:

the login module expects an email
the login module expects a customer number

Take a second to think why this may be bad testing code, though!

First of all, this code introduces nondeterminism to the test. We can now deal with a system being in two different states. Depending on which state we are in, we act differently. But we have also said, that this is only useful when we do not know in which state the system is! But, for our web page example, we know in which state the login module is, since initially it is always in the "login via customer number" state. And this is even a requirement since most customers use this variant to log in, so it must be the default. So, it actually is even a defect when we load the page and the login module is not in the "login via customer number" state, but in the "login via customer email" state. The problem is now, that we lost the power to find this defect by adding nondeterminism to the test code. If the login module is in the wrong state, the test case corrects it by switching to the correct login method.

Let me end this example by a very little bit of theory for all this. We are in the lucky position to have very solid test theories at hand to give all this considerations some foundation. The next picture shows the above example by referring to transition system based theories, like the most famous one - ioco - by Jan Tretmans.

You see here a specification saying "!x must come first" (see !x as the login module is in the "login via customer number" state). Next to that you see two implementations. Impl1 doing it right, Impl2 doing it wrong by starting with "!y must come first" (see !y as the login module is in the "login via customer email" state). In a conformance relation like ioco, of course, Impl1 would be correct, Impl2 wrong. But now look at the nondeterministic test case. It does what our function does, it allows both states ("tau" means an unobservable step). But having this test case, also Impl2 is correct now, we lost our power to spot that Impl2 is wrong.

This may appear a bit nitpicking, but I find this wrong approach all the time, and I always need to argue since the tester thinks she or he wrote a very fancy method, which can deal with all kind of states of the system. And this is true! But it sucks for testing when we do not really need it, since it removes defect detection power.

Another very common example is test data. When testing a system, you should do everything for being able to control your test data (like which customers are in the system, what kinds of transactions has the customer done, which offers are shown to one specific customer, etc.). Only when you really cannot control this, you need nondeterminism - and then it is very very useful! But when you can control your test data, forget about nondeterminism here! And again, testers tend to write very fancy methods to somehow discover the available test data by analyzing the web page or database, trying for instance to find a specific customer transaction to continue with the test, etc. And then you always see disappointing faces when saying: we do not need that! We know the test data, we can even hardcode it in our tests, we know the state of the system, we do not need to learn it! Since learning always means allowing more states than just one.

Nondeterminism always adds complexity to your test code, you get all these conditionals, maybe exception handling and quite complex SQL statements, etc. So you definitely need to start writing tests for your tests and you lose stability and readability of your test code.

Summarized:

keep test code as simple as possible
control as much as you can
avoid nondeterminism whenever possible
love nondeterminism when you really need it

Howto Become a Tester for Web Applications

Lars Frantzen — Thu, 11 Apr 2019 08:54:40 +0000

We frequently search testers for our agile Web Development teams. Therefore I sit very many times in job interviews, and very many times we have to reject the candidate. In this post I want to sum up the prerequisites for having a successful application talk in this domain. Hopefully this helps to give candidates a clearer view on what to expect and what is expected.

The world I describe here will of course not match to all companies. I am coming from a setup with agile/DevOps development teams. In such teams you will find some specific characteristics. I name two crucial ones next.

Tools are as open as possible and accessible to all team members

The times where the tester in the team had a specific testing tool, which only the tester understood, are blessedly over. Such tools are usually heavy-weight, do not follow any established (coding) language, live in their own repository, etc. What you want is a tool that is light-weight (e.g. a framework for an already existing language), which lives in the same repository as the development code, etc. In our current setup I was confronted with such a heavy-weight tool called Tosca. So one of the major achievements was to migrate from Tosca to the wonderful open source Codeception test framework for PHP, a language which was heavily used already for the web development. So all team members had a common language from that point on.

The gap between a developer and a tester is as small as possible

When using a tool which is code-based it implies that the tester can code in that language. I am coming to that point in more detail below. In general, the tester must also understand code, and the developer must also understand testing. Having a tool based on a common coding language, no one can give excuses anymore like "I do not know that testing tool". And please do not come up with that weird argument that a developer does not have that mystical "tester-mindset". A developer who can only think in happy cases is simply a bad developer.

Having this said, here are the most important prerequisites when applying for a tester position in an agile web development team:

A tester must also be a decent coder (not a developer)

It is not enough to have had a foundation course on Java five years ago. You need to be fluent in the language type of the test tool (usually some OO language). You do not need to be already fluent in PHP when there will be a PHP-based tool at the new job, but you need to be fluent in some OO language. Fluent does not mean that you are a developer. But you must be able to code small programs using all common OO techniques. You can read code, and you know where to find answers to coding problems. You know how to debug a program, and you are not scared to deep-dive into a complex coding issue. You really like code. If you do not like code, you will never be a good tester.

A web tester must understand (not master) all basic web technologies

It is really annoying when someone sits in a job interview for a web tester position, and we show some web page, and then the candidate does not even know the basic constituents of a web page. You really must know how to inspect a web page element and have a basic understand of the common web-related technologies, like:

HTML
CSS
Javascript
Responsive Design
PHP
SQL

Again, you do not need to be a developer and master these topics. But you must know how they work. So it may be sufficient to do one of the countless online web developer courses out there. You will probably know enough to be a very good web tester afterwards, even if you are still far away from already being a full stack web developer. And you do not need to be.

I would have really liked if people were aware of this before applying for the job. Then they know how to prepare, and when it is realistic to get the job. I hope that more people stop aiming at this ancient ISTQB-defined tester role, where a tester does not touch code. You still need to know your testing techniques, of course, so the ISTQB stuff is still valid. But you also need to like code and web technologies. And that makes you as valuable as any other role in the agile team.

DEV Community: Lars Frantzen

Why Tests are Flaky - External Dependencies

Nondeterminism In Testing - How To Do It Wrong

Lars Frantzen ・ Apr 14 '19 ・ 6 min read

Nondeterminism In Testing - How To Do It Wrong

Howto Become a Tester for Web Applications

Tools are as open as possible and accessible to all team members

The gap between a developer and a tester is as small as possible

A tester must also be a decent coder (not a developer)

A web tester must understand (not master) all basic web technologies