DEV Community: Istvan Forgacs

Shift-Left vs Test-First, Which One To Choose?

Istvan Forgacs — Thu, 25 Jul 2024 08:01:18 +0000

Twenty years ago, the test-first method became popular thanks to Kent Beck’s famous book Test Driven Development: By Example. Test-driven development is a software development method where the test code is written before the application code. It consists of the following steps:

Write test code for the unit.
Run all tests and validate that the tests fail.
Write the code of the unit.
Run tests and refactor code.
Repeat until the tests pass.

Case studies such as the one conducted by Microsoft and IBM proved that applying TDD reduces the number of bugs in the code. While the development time of three Microsoft and one IBM projects increased by 15-35%, the defect density decreased by 40-90%. The conclusion was that TDD is worth doing. There are other test-first methods, such as BDD or specification by examples for agile teams. In these methods, the test cases became understandable for all the stakeholders.

Nowadays, shift-left has replaced test-first. It may be a surprise for you that the concept of shift-left testing is also more than 20 years old, published by Larry Smith in 2001. However, shift-left only means that let’s start testing as early as possible. The question is, what does ‘as possible’ mean? Unfortunately, no definition for this exists. However, this question is exceptionally important in the era of DevOps and Agile. It’s obvious that developers can use TDD or BDD, which are test-first methods, thus here, shift-left equals test-first. However, TDD and BDD concentrate on units or modules, and we know that unit testing can be executed before the system or e2e testing. Hence, the important question is, can e2e testing be test-first or only just shift-left?

Executing e2e testing in a test-first way is difficult. A manual test-first solution is impossible as the application should be ready. Unfortunately, creating test automation code entirely before the application is not an easy task, as we should know the selectors in advance. We know that selectors identify the UI objects. The only way to do this is to plan the selector names before the implementation. In addition, the test automation engineer should know all the tricks before the implementation. For example, some UI elements may be covered by others so the test automation engineer should modify the test code such as adding force: true if using Cypress. We can say that test execution automation is not test-first in practice.

We know, that test automation does not just mean test execution automation but should involve test design as well. Without test design, you have no chance to find potential bugs in a complex system. Let’s try testing our Pizza application. Here I created 30 realistic mutants, i.e., 30 alternatives that are slightly different from the original. Let’s use exploratory testing first. You probably will detect 30-45% of the artificial but realistic bugs – I found only 37%. However, by applying a good test design technique such as action-state testing, you can also detect 100% as I did.

Model-based testing (MBT) is a method of test design automation. In addition, the test-first approach can be satisfied if the model has been made before coding. However, to make a model, we should know everything about the application to be implemented. The reason is that the model in traditional MBT is computer-readable, thus, every action made by the user should be incorporated into the model, and every necessary output validation should also be involved. As a consequence, we can make the model in a test-first way, but the result is that the model should be remade when the application is ready. Thus, the modeler makes a double work that is inefficient.

Does that mean that the test-first method is not possible for e2e testing? Fortunately not. The requirement for a model to be computer-readable is false. The new generation of MBT is the two-phase model-based testing. Here there are two models: the first is a high-level model that is not computer- just human readable. The computer cannot make executable test code from this model but can generate abstract tests for the tester. This is a huge difference: a human has significant domain knowledge and will know how to execute the tests independently of the implementation manually.

For example, if the high-level model consists of a user action Register, a tester can execute it independently of the implementation. In other words, the implementation can be anything, a tester can register. This means that the high-level model can be as abstract as possible so that a tester with acceptable domain knowledge can execute the generated test. A high-level model can be textual or graphical, and may or may not contain states, but should be implementation-independent.

On the other hand, a computer-readable model should contain every detail. The problem is that if the model and the implementation differ, the generated test will fail. That’s why a good low-level model can be made when the implementation is ready.

It’s obvious that the high-level model and the generated abstract tests can be made before the implementation starts, thus this part is test-first. This is a big advantage as both the model and the tests can be validated by the whole team. This validation is also strong defect prevention. My personal experience is that when making the model based on the requirements, most of the mistakes, inconsistencies, incompleteness, etc. are revealed. The cause is simple. You cannot make a good model from a bad specification.

When you make a model, you may need to cover more requirements with a single test. This is the point when inconsistent requirements are detected. Note that a perfect specification is only a dream, but high-level modelling makes it good enough to be implemented so that users love it.

Okay, the high-level model is ready, what’s next? From here the method is not test-first as the application should be ready. When this happens, the application is manually tested with the abstract test cases validated so far.

During the manual test execution, every action such as inserting text, clicking, checking, selecting, or anything is immediately logged as a test step. For example, when clicking on Submit button, our tool generates a test step:

When Submit IS clicked

where is the selector of the button and clicked is the action/operation.
The only additional step to be made by the tester is that the necessary output is not just visually validated, but marked for the test automation tool. This means that if the result is a number 100, then this number should be marked, and the test tool logs this validation event. For example, the tester hovers the mouse over the result that is 100, clicks on that value, and the tool generates a test step such as:
Then Total price IS 100

These steps starting with When or Then are the elements of the low-level model in the second phase and these are machine readable for example Total price is a selector. From this low-level model, any necessary test code can be generated. The difficult part of this generation is to solve that executing the whole test without any human interaction should pass without any necessity of debugging or fixing the generated test. If so, then this non-test-first phase becomes very short, our experience is that only about 1/3 – ¼ of the modeling/test-first part.

Conclusion

Shift-left and test-first approaches are close to each other, but test-first is better though more difficult to use. If possible, use the test-first approach over shift-left. Fortunately test automation process can be almost entirely test-first. As nowadays test automation is a must, doing it in a test-first way not only reduces the number of bugs in the system, but it’s also cost-effective.

Two-phase Model-based Testing

Istvan Forgacs — Thu, 15 Dec 2022 13:00:43 +0000

Most test automation tools just do test execution automation. Without test design involved in the whole test automation process, the test cases remain ad hoc and detect only simple bugs. This solution is just automation without real testing. In addition, test execution automation is very inefficient.

If you write the test code in a simple way you hurt the DRY (Don’t Repeat Yourself) principle. For test automation, it means that the maintenance cost will be higher. If you use the page object model — a design pattern where the page objects are separated from the automation test scripts., the code will be much longer and takes more time to write.

That’s why advanced test automation tools include test design. As Bloor Research states: ‘Test design automation provides a potentially massive boost to testing speed and efficiency. Properly leveraged, it can also dramatically improve the end-user experience.’

The tools involving test design are the model-based testing (MBT) tools. ISTQB defines model-based testing as “Testing based on or involving models”. Thus, instead of creating test cases one by one manually, we create an appropriate test model from which an MBT tool is able to generate test cases. However, a model itself is not enough. Traditional test design necessitates the application of test selection criteria, which make it possible to know when to stop creating test cases. Therefore, an MBT tool generates test cases from the model and is based on the appropriate test selection criteria.

Modelling is a divide-and-conquer activity. Its main goal is to collect those elements of a system (at that time the requirement specification) by which the system can be reliably tested. We can make relatively simple models in a hierarchical way, and from these models, a tool can generate abstract test cases.

Systems planned by requirement specifications can be modelled in different ways. For testing, we may need special test models but in practice, software engineering models such as state transition diagrams, UML activity diagrams, EFSM, BPMN, use case diagrams, etc. are used. These are not ideal representations for creating reliable test sets.

Check this out: A Complete End to End (E2E) Testing Tutorial- Comprehensive Guide With Examples and Best Practices

Challenges of Model-Based Testing

A common characteristic of these models is that each can be represented as graphs (even textual models such as use cases have graph representations). Traversing the graphs in different ways, different levels of test selection criteria can be selected. The test cases can be generated to satisfy the selected criterion. Here is the advantage of using test design automation: instead of creating the test cases one by one in an ad-hoc way, they are generated based on an accepted model and test selection criterion.

One problem is that traditional models should involve information about the implementation for test code generation. This means that every event , input, and output should be in the model. If the model is created before implementation, then lots of work should be repeated according to the real implementation. Making models after implementation on the other hand contradicts the shift left principle.

Another common problem is that traditional methods require coding. Traversing a graph invalid test cases are generated. For example, adding the second item to the shopping cart, then deleting the third one is not possible. Guard conditions are added to address this problem. However, guard conditions are code. If a guard condition consists of output, then this output should be coded. If it’s a difficult function, the guard code is also difficult, and how to test it?

Check this out: Test On Android Online Emulator- Test your web and mobile apps on Android Emulators online.

Generating Test Cases in Two Phases

The most popular solution is using stateless models. They can be generally used and that’s why they are the most popular and widely used MBT tools. A specific problem is that stateless models will detect only the simplest bugs even if a stronger test selection criterion has been selected (such as the all-transition-pairs criterion) see this blog post.

Stateful models such as state transition diagrams detect trickier bugs, however, they can’t be used for many systems to be implemented (states are difficult to identify or there are too many states).

All these problems can be addressed by double-phase model-based testing.

Phase 1: Generate High-Level Test cases

Here the first model is a high-level model. Abstract test cases are generated from the model. These test cases can validate the requirement before implementation. In this way, most of the bugs, misunderstandings, etc, can be removed or fixed which is very cheap at this phase of the SDLC. We call these test cases abstract as they can be executed by humans, but executable test code cannot be generated.

The high-level model consists of textual steps. Each step contains a (user action), a (system) response — optional, and a test state — optional. Actions and responses are well-known in use case testing, however, test states are significantly different from program states. The solution to reducing the huge number of states is that if two actions are the same, but the system calculates the response in different ways, then we have arrived at different states.

Example. The initial state is ‘paying is not possible’. If we add any items so that the total price remains below EUR 10, then we remain in the same state as we remain under the pay limit. However, reaching 10 we arrived at a new state ‘paying is possible’.

The reason behind the test states is that each different calculation should be tested as any of them may be faulty. In this way, the number of states remains manageable, yet the necessary tests will be involved. We validated this assertion when solving the exercises of our website and all the bugs have been found by this model.

Here is an example for modeling:

 INITIAL STATE cart empty
R4 paying is possible when price 10:
    add items so that price remains below 10 => check that price <10 STATE paying is not possible
        add items so that price reaches 10 => check that price =10 STATE paying is possible

The first line is the initial state. The second line is a label that connects the requirement R4 to the model and the test case. The third line consists of a model step. The first part ‘add items so that price remains below 10’ is a user action, the second part: ‘check that price <10 *’ is a validation of the system response, and the third part ‘*paying is not possible’ is the resulting test state. A tester knowing boundary value analysis will add items to the cart reaching the closest value below 10. When she validates the result, she will check that paying hasn’t been possible yet.

Executing this test step is easy for the tester knowing the prices of the items, but currently, it’s almost impossible for a machine that should know the test design techniques, understand the text of the model steps and identify the prices on the screen (or by reading and interpreting the specification).

A single test is generated from this model above consisting of the two model steps. The model can be easily modified to execute these steps in two individual test cases:

 INITIAL STATE cart empty
R4 paying is possible when price 10:
    add items so that price remains below 10 => check that price <10 STATE paying is not possible
    add items so that price reaches 10 => check that price =10 STATE paying is possible

You can see that tabulation makes it possible to continue a test case or start a new one.

Model steps are truly implementation-independent. You can implement the shopping feature of these model steps in any way. The model steps are also very compact. They should contain less information which is enough for the tester for manual execution. For example, a model action change password successfully

may involve five steps to be implemented:

click on the password change menu item
insert current password
insert new password
insert a new password for the second time
click on the submit button

In this way, the model remains compact and can be understood and reviewed easier.

Phase 2: Generate Low-Level Test Cases

The next phase is the generation of the executed test code. This can be done by manually executing the test cases. During the execution a low-level model is generated on the fly, see the figure below:

Figure 1. A low-level model is generated during the execution of an abstract test step ‘add items so that price remains below 10’

The tester executes the current actions or responses, here ‘add items so that price remains below 10’.

Here the low-level model is a Gherkin-like description but can be anything. Considering the first step:

WHEN Go shopping IS #pressed

the Go shopping*is the selector that identifies the Go shopping UI element, and *#pressed means a click on that. The validation is a little bit more difficult as you should select the validation type or the result itself. For example, if you want to validate that the Pay button is inactive, then you can select the pay button first, then that it is a validation (THEN), and finally the *#non-active *command

Figure 2. Validation of the Pay button is inactive

In this way, you can easily generate a low-level model that contains every piece of information for test code generation. A tool can generate test code for any programming language and any test runner.

Check this out: How To Use IOS Emulators On PC

Conclusion

A clear advantage of this method is that it’s a DRY testing solution. It means that the test cases may consist of some steps more times, but the tester will manually execute them only once as the second occurrence of the same step will be executed automatedly.

It’s a method where the automation part is entirely automated, and testers can do test design at the highest-level making models. It’s not an easy task especially if test states are involved, but it’s fun and results in higher-quality code.

The two-phase modeling is a true shift-left solution that can be used for stateful and stateless cases. It’s also an efficient defect prevention method. Test automation in this way becomes not only fast, but it’s very comfortable as testers don’t need to calculate the results in advance. They just need to check them, which is much easier. Finally, this method is for testers as it needs test design but doesn’t need coding knowledge.

Fault-Based Testing and the Pesticide Paradox

Istvan Forgacs — Thu, 15 Dec 2022 12:26:52 +0000

In some sense, testing can be more difficult than coding, as validating the efficiency of the test cases (i.e., the ‘goodness’ of your tests) can be much harder than validating code correctness. In practice, the tests are just executed without any validation beyond the pass/fail verdict. On the contrary, the code is (hopefully) always validated by testing. By designing and executing the test cases the result is that some tests have passed, and some others have failed. Testers do not know much about how many bugs remain in the code, nor about their bug-revealing efficiency.

Unfortunately, there are no online courses available where after entering the test cases the test course platform tells which tests are missing (if any) and why. Thus, testers cannot measure their technical abilities (ISTQB offers to measure non-technical skills only). The consequence is that testers have only a vague assumption about their test design efficiency.

Fortunately, the efficiency of the tests can be measured. Let us assume that you want to know how efficient your test cases are. You can insert 100 artificial, but still realistic bugs into your application. If the test cases find 80 bugs, then you can think that the test case efficiency is about 80%. Unfortunately, the bugs influence each other, i.e., some bugs can suppress some others. Therefore, you should make 100 alternative applications with a single-seeded bug in each, then execute them. Now, if you find 80 artificial bugs, your efficiency is close to 80%, if the bugs are realistic. This is the way how mutation testing, a form of fault-based testing works.

This is the basic concept of fault-based testing, i.e., selecting test cases that would distinguish the program under test from alternative programs that contain hypothetical faults. If the program code contains a fault, then executing the output (behavior) must be different. Therefore, in order to be able to distinguish the correct code from all its alternatives, test cases should be designed in a way that some output is different concerning the correct code and all its faulty alternatives. Each alternative is a textual modification of the code. However, there is an unmanageable number of alternatives, and thus we cannot validate the ‘goodness’ of our test set. Fortunately, it was shown by Offutt that testing a certain restricted class of faults, a wide class of faults can also be found. The set of faults is commonly restricted by two principles, the competent programmer hypothesis, and the coupling effect.

Coupling effect hypothesis means that complex faults are coupled with simple faults in such a way that a test data set detecting all simple faults in a program will detect a high percentage of the complex faults as well. A simple fault is a fault that can be fixed by making a single change to a source statement. A complex fault is a fault that cannot be fixed by making a single change to a source statement. If this hypothesis holds, then it is enough to consider simple faults, i.e., faults where the correct code is modified, (mutated) by a single change (mutation operator). Thus, we can apply mutation testing to get an efficient test design technique with an appropriate test selection criterion.

Check this out: Cross Browser Test Cloud - Browser & app testing cloud to perform both exploratory and automated testing across 3000+ different browsers, real devices and operating systems.

Mutation testing

As mentioned, mutation testing is the most common form of fault-based testing, in which by slightly modifying the original code we create several mutants. A reliable test data set should then differentiate the original code from the well-selected mutants. In mutation testing, we introduce faults into the code to see the reliability of our test design. Therefore, mutation testing is essentially not testing, but “testing the tests”. A reliable test data set has to “kill” all of them. A test kills a mutant if the original code and the mutant behave differently. For example, if the code is y = x and the mutant is y = 2 * x, then a test case x = 0 does not kill the mutant while x = 1 does.

If a mutant hasn’t been killed, then the reasons can be the following:

Our test case set is not good enough and we should add a test that will kill the mutant.
The mutant is equivalent to the original code. It’s not an easy task to decide the equivalence. And what is more, the problem is undecidable in the worst case.

In the case of a first-order mutant, the code is modified in one place. In the case of a second-order mutant, the code is modified in two places, and during the execution, both modifications will be executed. Offutt showed that the coupling effect holds for first and second-order mutants, i.e., only a very small percentage of second-order mutants were not killed when all the first-order mutants were killed. Moreover, Offutt showed that the second-order mutant killing efficiency is between 99.94% and 99.99%. This means that if we have a test set that will kill all the first-order mutants, then it will also kill the second-order mutants by 99.94–99.99%. Thus, it is enough to consider only simple mutants, which is much easier, and there are much fewer of them.

The real advantage is that if we have a test design technique that kills the first-order mutants, then this technique kills the second and higher-order mutants as well. Anyway, we can assume that the bigger the difference between the correct and the incorrect code is, the higher the possibility is to find the bug.

Excellent, we have a very strong hypothesis that if we have a good test design technique to find the first-order mutants as bugs, we can find almost all the bugs, and our software becomes very high quality. Consider the numbers again. Based on the comprehensive book of Jones and Bonsignour (2011), we know that the number of potential source code bugs in 1000 lines of (Java, JavaScript, or similar) code is about 35–36 on average. Thus, a code with one million lines of code may contain 35000–36000 bugs. A test design technique that finds at least 99.94% of the bugs would not detect only 22 faults in this huge code base.

Ok, we made mutation testing and we found that 95% of the mutants have been killed. What’s next? The first solution is to create the missing test cases. We should consider the basic code and a non-killed mutant. We should find an input for which the mutant is killed. This is our new test case. Here the method is reversed concerning testing. We start from the faulty code, the mutant, and search for a test that reveals this bug. If we cannot find a test though we know the fault, then we can assume that a user without this knowledge will not detect the bug either. In other words, we consider this mutant as equivalent even if we cannot prove it.
If the tester hasn’t killed some mutants, then instead of the non-killed mutants, the missing test cases appeared. Knowing your missing tests, you can improve your test design knowledge.

Sometimes, finding the missing test cases for the non-killed mutants is too costly. Unfortunately, knowing that we killed, for example, 95% of the mutants doesn’t mean that our test case efficiency is 95% (i.e., that we can find 95% of the bugs in the code). Generating mutants (bugs) is quite different from making mistakes by the developers. However, historical data can help. If in the past our mutant killing efficiency was 95%, the code quality was good enough for a given risk level, and now it’s only 80% for the same risk, then our test case set is not good enough. In this case, we should find additional test cases based on the mutants until we reach 95%.

We note that fault-based testing differs from defect-based testing since the latter is a test technique in which test cases are developed from what is known about a specific defect type (see ISTQB glossary).

Check this out: Emulator vs Simulator vs Real Device Testing: Key Differences

Pesticide paradox

Most testers know and believe in ‘the 7 principles of software testing’. One of them is the pesticide paradox. Originally, Beizer (1990) wrote: “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.” Now this paradox is described as ‘If the same set of repetitive tests is conducted, the method will be useless for discovering new defects.’ or ‘if the same set of test cases are executed again and again over the period of time then these set of tests are not capable enough to identify new defects in the system.’ In simple words: tests wear out.

Let us assume that we have a good test case set that kills all the first-order mutants. This is not entirely perfect as some bugs haven’t been found (0,06%). Assume that only one bug cannot be found with our tests. Now assume that the specification remains but the code is refactored several times. Except for the only bug, all the others made during the code modification will be detected. Let us assume again that the test case set doesn’t kill a single mutant only. The argument is the same. Any faulty code modification will be detected except for these two bugs.

Therefore, we can say that the error-revealing capacity of a test set is determined by the mutation-killing efficiency that is constant until the requirements remain unchanged. This is independent of whether the mutation testing is done or just could have been done. Therefore, if the cause of the new defects is faulty refactoring, and the requirements remain unchanged, then the good test set remains as good for the modified code as for the original. On the other hand, when the requirements change, or new requirements are added, then new/modified tests must be written. But it does not mean that the original tests wear out; on the contrary, we use (some of) them in our regression test pack. The conclusion is that the pesticide paradox principle is not valid.

Check this out: Test on Safari Browser Online with Real Mac Machines - Test on Latest Safari Desktop and Mobile Browsers for Cross Browser Compatibility.

A Deeper Insight into Test Design

Istvan Forgacs — Fri, 04 Nov 2022 12:02:02 +0000

Test automation is a hot topic, test design isn’t. However, without appropriate test design, test automation is worthless as you will execute test cases several times without detecting the bugs in your software. Test design is one of the most important prerequisites of quality. Test design is necessary for:

evaluating the quality of the product with regards to customer expectations and needs (quality control)
finding defects in the product (testing).

Fortunately, or unfortunately, it is a creative process on its own, but also one that requires technical expertise.

“More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded — indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest.” Boris Beizer Software Testing Techniques (1990).

Steps of test design

First, the test approaches are worked out, which means that it should be planned how the product quality together with the cost optimization goals can be achieved, taking into consideration the elicited test conditions, code complexity, and risks. At this point, the test approaches are approximated from a technical point of view: technical consistencies with standards, processes, flows, measures, user experience, etc.
After planning the test approach, the test design techniques are selected that meet the testing objectives and the result of risk and complexity analysis. In general, it is advisable to select test design techniques understandable by other stakeholders. Real systems usually require using more techniques in combination. Below you can see the test design classes and the most important test design techniques. There are some techniques (in bold) that are not involved in any ISTQB material. I will make a separate blog for each of them, and here I include only a very short overview of them.

TEST DESIGN CLASS	TEST DESIGN TECHNIQUE
Specification-based	Equivalence partitioning
Specification-based	Boundary value analysis
Specification-based	General predicate testing
Specification-based	Decision table testing
Specification-based	Cause and effect graphing
Specification-based	Classification tree method
Specification-based	Combinatorial testing
Specification-based	Combinative testing
Specification-based	State transition testing
Specification-based	Action-state testing
Specification-based	Scenario-based testing
Specification-based	Random testing
Specification-based	Metamorphic testing
Specification-based	Syntax testing
Structure-based	Statement testing
Structure-based	Branch testing
Structure-based	Path testing
Structure-based	Decision testing
Structure-based	Branch condition testing
Structure-based	Modified condition/decision coverage
Structure-based	Data flow testing
Structure-based	Error guessing
Structure-based	Error guessing
Structure-based	Exploratory testing
Structure-based	Session-based testing
Structure-based	Checklist-based testing
Model with execution-based	Use case testing with execution
Model with execution-based	Action-state testing with execution
Fault-based	Mutation testing

Check this out: Online Automated Testing Platform - Accelerate your release velocity with blazing fast test automation on cloud

General predicate testing (GPT) is a reliable extension of boundary value analysis when there are more logically dependent requirements given. By applying general predicate testing all the potential predicate bugs will be revealed. It’s easy to use as almost the entire method can be automated. GPT’s test selection criterion requires that each border be covered by four test cases: (1) a test input “closest” to or on the boundary that is inside the examined domain (ON point), (2) the closest test to the border that is outside the examined domain (OFF point), (2) a data point inside the examined domain that differs from the ON point (IN point) and (4) a data point outside the examined domain that differs from the OFF point (OUT point). Many points for different borders can be merged.

Combinative testing: The rationale behind the combinative technique is to assure that for a single variable each computation is calculated in more than one context. The basic variant of combinative testing is called diff-pair testing. Informally, it requires that each value p of any parameter P be tested with at least two different values q and r for any other parameters. In this way, the number of test cases remains linear with input space, yet trickier bugs can be detected.

Action-state testing is a step-by-step technique where abstract steps are created one by one, controlled by a simple algorithm. Instead of making the whole model at once, it is made gradually. It can be considered as a union of use case testing and state transition testing. Action-state testing consists of steps. One step consists of a user action, a system response, and a test state. The method does not need guard conditions, therefore it’s a true codeless solution for test design automation.

Metamorphic testing “is an approach to both test case generation and test result verification. It is a property-based software testing technique, which can be an effective approach for addressing the test oracle problem and test case generation problem. The test oracle problem is the difficulty of determining the expected outcomes of selected test cases or determining whether the actual outputs agree with the expected outcomes. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm concerning multiple inputs and their expected outputs.”

Model with execution technique: The first step is to create an abstract model. Abstract test cases can be generated from this model. The abstract test cases contain the minimum information for a tester to execute the tests. For example, add items to reach EUR 30 is appropriate, as a tester knowing the requirements can do it. When the application is ready, the tester executes the test cases and validates the results. The concrete inputs and outputs are recorded. Therefore, there is no need for exact input/output as in the case of specification-based techniques. Only validation is necessary.

Mutation testing is actually a method for ‘testing the tests’. During mutation testing alternative codes, i.e., mutants are created. The test cases we’ve designed should distinguish the basic program under test from all the alternative programs so that for one test some outputs can be different. Therefore, to distinguish the correct code from all its alternatives, test cases should be designed to differentiate the correct code from all its faulty alternatives. Each alternative/mutant is a textual modification of the code and can be generated or created manually.

3- The next step is to determine the test case selection criteria (for simplicity we refer to this as test selection criteria). The test selection criteria determine when to stop designing more test cases, or how many test cases have to be designed for a given situation. The test selection criterion for the equivalence partition method, for example, may result in designing a different number of test cases for each equivalence class. It is possible to define different (“weaker” or “stronger”) test selection criteria for a given condition and design technique.

4- The next step is to establish the test data. Test data are used to execute the tests, and can be generated by testers or by any appropriate automation tool (can be produced systematically or by using randomization models). Test data may be recorded for re-use (e.g. in automated regression testing) or may be thrown away after usage (e.g. in error guessing).
Typically, test data are created together with the test case they are intended to be used for. Test data can be created manually, by copying from production or legacy sources into the test environment, or by using automated test data generation tools. Note that concrete test cases may take a long time to create and may require a lot of maintenance.

5- Now we can finalize the test design. A test case template contains a test case ID, a trace mapped to the respective requirement/test condition, test case name, test case description, precondition, postcondition, dependencies, test data, environment description, expected result, actual result, priority, status, expected average running time, comments, etc. Of course, you can omit what is not relevant.

6- The next step is to design the test environment. The test environment consists of items that support test execution with software, hardware, and network configuration. The test environment design is based on entities like test data, network, storage, servers, and middleware. The test environment management has organizational and procedural aspects: designing, building, provisioning, and cleaning up test environments requires well-established procedures.

7- Finally, you should validate all the important artifacts produced during the test design including the control of the existence of the bi-directional traceability between the test basis, test conditions, test cases, and procedures.

Check this out: Appium Cloud For App Automation - Test your Native, Hybrid, and Web Apps on Appium mobile device cloud of 3000+ different real devices.

Characterization of test design techniques

To evaluate the different test design classes, we should characterize them. I tried to collect the main characteristics that determine a test design class.

Input — the key element of designing the test case is to plan the appropriate inputs. For example, considering the Boundary Value Analysis technique we need input on the border and another input that is very close to the border but on the opposite side. Calculating the input is not easy, sometimes difficult. For specification-based class, the input should be calculated before test execution. For an experience-based class, the input is not pre-calculated just selected during the execution. Similar is the case for the model with execution class, but some high-level description gives some guidelines for input selection. Selecting the input during test execution is much easier. For structure-based class, the input should be calculated based on domain constraints which are very difficult and sometimes impossible. Finally, for mutation, there is no input needed.
Output — For specification-based class, the output is also calculated. For experience-based, structure-based and model with execution classes, the output is only validated, i.e., you don’t need to calculate before, only check during the program execution which is much easier. For mutation testing output is just for comparison, thus it’s neither calculated nor validated.
Implementation — We know that we should start testing as early as possible. Therefore, implementation-independent methods are better. Considering these classes, specification-based and model with execution classes are implementation-independent, the others are not.
Exclusivity — Structure-based techniques are never used without other techniques as they would be very difficult and time-consuming. Mutation testing only validates the tests and thus cannot be used in itself.
Effectiveness — This refers to how many bugs are detected with respect to the total number of bugs in the application. For specification-based and model with execution classes the effectiveness is high (by applying a strong test design technique in the class), for experience-based is medium. For structure-based the effectiveness is low and for mutation testing, it is not applicable as it only measures the effectiveness.
Regression testing — It is very important as the code is continuously changing which requires re-testing the whole application. Only experience-based testing is not a regression testing method.

Based on this characterization here is a table for the test design classes:

Based on this table and the above explanation we can say that the structure-based technique is not a real test design class. It is for validating the test cases designed by other techniques and for creating missing test cases due to incomplete test design or implementation-dependent code that isn’t related to any requirement. I would call it a test creation/validation method.

Fault-based mutation testing is also not a real test design technique as it is not for designing, it is only for validating the test cases.

Considering the remaining test design classes, experience-based is the easiest and fastest solution but it’s not appropriate for regression and its effectiveness is lower than the two others. The model with execution class is the best as it is effective and efficient. It can also be combined with specification-based techniques such as use case testing and action-state testing.

Check this out: Jest Tutorial- Complete Guide to Jest Testing

Conclusion

Here I overviewed test design as a very important part of software testing. I made characterizations of the different techniques and compared them based on these characteristics.

The Golden Age of Software Testing

Istvan Forgacs — Wed, 02 Nov 2022 15:47:18 +0000

LambdaTest asked me to write blogs for them. I liked the idea and accepted this kind request. I’ve been dealing with software testing since 1986. Originally, I was a researcher, and later I worked in the industry; thus, I’m knowledgeable about both the academic and the industrial side of software testing. My idea is to write articles starting from the basics to the solution of concrete problems practitioners face every day, but Google search gives poor results. Each article will be understandable as the episodes in Friends or The Big-Bang Theory, but they are also connected. The basic difference is that I cannot guarantee a happy end; it’s your turn to reach.

There are good books and documentaries about mathematics and physics, such as The Story of Maths, that is a four-part British television series outlining aspects of the history of mathematics. We are testers, and testing also has history, even if it is much shorter! Unfortunately, there is neither a complete book nor any documentary about the history of testing. In this article, I emphasize only one period, the Golden Age of Software Testing. This period starts in 1975 and ends in 1980. I know that it’s subjective, and you may disagree. But it’s my view, and I will try to tell you what happened during this period; unfortunately, I was too young to play a part in it.

The earliest conception was that you “write a program and then test and debug it.”

We know that Grace Hopper invented the word ‘debugging,’ a US admiral and computer scientist who is a key contributor to COBOL, perhaps the most successful programming language ever, whose codebase is in production and assumed to hit 800+ billion lines.

Testing was considered a follow-on activity and embraced the effort to discover errors and correct and remove them. A number of the earliest papers on “testing” actually covered “debugging,” and the difficulty of correcting and removing errors was long thought to be the more interesting problem. However, it was not until 1957 that program testing was clearly distinguished from debugging. Even after this separation, software testing was a side effect of programming, and testers should wait until 1972 for the first-ever software testing conference. It was organized by William C. Hetzel and was held at the University of North Carolina, to bring together a broad cross-section of people interested in software testing. The first software testing book ‘Program Test Methods’ was published as an outgrowth of this conference and established the view that “testing” encompassed a wide array of activities all associated with “obtaining confidence that a program or system performed as it was supposed to.”

On a side note, to make your life of testing easy and fuss-free while debugging, you can try the LT Debug Chrome extension.

And now we arrived at 1975 when the golden age started. The trigger was the first-ever theoretical software testing research paper by John B. Goodenough and Susan L. Gerhart: Toward A Theory of Test Data Selection published in the journal IEEE Trans. Software Engineering. This paper introduced some very important principles and methods of software testing.

Check this out: Blackbox Testing Tutorial- A Comprehensive Guide With Examples and Best Practices

The first concept is the data selection criterion (now we call it the test selection criterion), which determines when to stop designing more test cases, or how many test cases have to be designed to reach an acceptable software quality. The test selection criterion for exhaustive testing requires testing each possible input in the entire input domain. The criterion for the equivalence partition method is to select one test case for each equivalence class. It’s very important to choose an appropriate test selection criterion (or criteria) based on risk analysis (I will give details in a subsequent article.)

The next concept is the reliability of test selection criteria. Here I use a simpler description to be more understandable. A reliable test set T has the property that either for each t∊T, t *passes if and only if the code is correct or there is at least one test t for which the software fails. A test selection criterion *C is reliable if any generated test T(C) satisfying C is reliable. We will show that only the test selection criterion resulting in exhausting testing is reliable for each software. Despite this, test selection criterion reliability is a very important concept. There may exist reliable test selection criteria for a class of programs or a class of programming language elements. For example, there is a reliable test selection criterion for testing predicates. If we use this criterion, we can be sure that each erroneous predicate will be detected. Unfortunately, the ISTQB glossary doesn’t contain any definition for test set or test selection criterion reliability.

I think that these authors mention first equivalence partition testing. Then, they suggested that a solution can be to weaken exhaustive testing so that the error detection capability remains while the number of test cases is lower. Here they mention:

‘Although some programs cannot be exhaustively tested, a basic hypothesis for the reliability and validity of testing is that the input domain of a program can be partitioned into a finite number of equivalence classes such that a test of a representative of each class will, by induction, test the entire class, and hence, the equivalent of exhaustive testing of the input domain can be performed.’

Check this out: Usability Testing- A Comprehensive Guide With Examples And Best Practices

The following primary result is from Bill Howden’s famous paper ‘Reliability of the path analysis testing strategy.’ He showed that ‘there is no procedure which, given an arbitrary program P and output specification, will produce a nonempty finite test set T C D such that if P is correct on T, then P is correct on all of D. The reason behind this result is that the nonexistent procedure is expected to work for all programs, and thus familiar noncomputability limitations are encountered.’ What does it mean? The sad truth is that except for exhausting testing, there is not a general method resulting in a reliable test set for all the programs. Therefore, we testers cannot tell that we have found all the bugs after testing the software. But this is not an excuse for poor testing, as we can find most bugs.

Though Goodenough and Gerhart introduced a class of program errors, Howden introduced a better one; we can use it today. According to his classification, three types of errors exist:

Computation error — some calculation is faulty, such as x = 1 instead of x = 0
Domain error — the control is faulty, such as if (x > 1) instead of if (x >= 1)
Subcase error — something is missing from the code, such as a missing part if (x === 1) y = 2

If we know the error classes, we can find better test design methods or coverage criteria to detect them. For example, a test should traverse the fault to detect a computation error. But it’s not enough as the faulty result should propagate to some output to manifest the fault as a failure. Domain errors can be detected by applying general predicate testing will be described in a subsequent article. The most difficult error class is the subcase error, fortunately, most of them can also be detected by applying some advanced test design techniques using states.

While test selection criteria are defined concerning the test design, i.e., independent of the implementation, test data adequacy criteria are defined concerning program execution. ISTQB calls them coverage criteria. Almost all the test data adequacy criteria were introduced during this golden age. The two main classes of criteria are control flow and data flow. The most straightforward control flow criterion is statement coverage, but the minimum acceptable one is branch/decision coverage. Nowadays, several tools support these criteria.

The other class of test data adequacy criteria is based on data flow. These are not spread and almost unknown to testers. However, to find computation errors, data-flow coverage is necessary. The primary criterion is all d-u pair criterion, where d *is a definition, and *u is a use. A definition is an assignment to a variable; a use is a reference. A d-u pair is the following:

x=1 //definition for x
… //there is no assignment to x
y = x //use of x

Here y gets the value of x. If y is output and x=1 is a faulty assignment statement, the bug will be detected. If both statements are inside an if conditional statement, satisfying branch coverage may not detect the bug, but the d-u coverage will.

The result of Howden stating that there is no ideal method for creating reliable test cases doesn’t mean that software testing is dead. Fortunately, in the golden age, some hypotheses are introduced based on which test selection criteria and test case sets can be reliable, and the software can be close to bug-free.

The first hypothesis is the competent programmer hypothesis (CPH) identified by DeMillo et al. in their famous paper Hints on test data selection: Help for the practicing programmer published in 1978. They observed that “Programmers have one great advantage that is almost never exploited: they create programs that are close to being correct.” Developers do not implement software randomly. They start from a specification, and the software will be very similar to their expectations, hence, close to the specification.

CPH makes test design possible. If the implemented code could be anything, we should test the application to differentiate it from an infinitive number of alternatives. This would need an infinitive number of test cases. Fortunately, based on CPH, we only have to test the application to separate it from the alternative specifications very close to the one being implemented.

To demonstrate CPH, let’s consider the following requirements:

R1. Input two integers, say x and y, from the standard input device

R2. If x and y are both negative, then print x + y to the standard output device

R3. If x and y are both positive, then print their greatest common divisor to the standard output device

R4. If x and y have opposite signs (one of them is positive and the other is negative), then print their product to the standard output device

R5. If x × y is zero, print zero to the standard output device.

The test set T = { ([-2, -3]; -5), ([4, 2]; 2), ([2, -4]; -8), ([2, 0]; 0) } is adequate. However, an alternative specification can be the following:

R1. Input two integers, say x and y from the standard input device

R2. If x and y are both negative, then print x + y to the standard output device

R3. If x and y are both positive and less than 100,000, print their greatest common divisor to the standard output device; otherwise, print the smaller value to the standard output device.

R4. If x and y have opposite signs (one of them is positive and the other is negative), then print their product to the standard output device.

R5. If x × y is zero, print zero to the standard output device.

Assume that the developer implemented this specification. In this case, T would not be adequate as all test cases would pass, yet the code is wrong. What’s more, any positive pair of test input below 100,000 would not be error revealing. Assuming that the implementation can be anything, we easily modify the specification again so that selecting any number of positive test input pairs would not be reliable.

Fortunately, because of the competent programmer hypothesis, it is unrealistic to implement the modified R3 since this requirement is far from the original, and the probability that a developer swaps them is minimal.

The coupling effect hypothesis has been introduced in the same paper. It means that complex faults are coupled to simple faults in such a way that a test data set detecting all simple faults in a program will detect a high percentage of the complex faults as well. A simple fault is a fault that can be fixed by making a single change to a source statement. On the other hand, a complex fault is a fault that cannot be fixed by making a single change to a source statement. If this hypothesis holds, then it is enough to consider simple faults, i.e., faults where the correct code is modified by a single change.

Empirical studies show that this hypothesis hold for most cases. This is very good as, based on this hypothesis, we can have test design techniques that will detect almost all subcase errors. I will show you in another article.

Here I only listed the most important results of the golden ages of software testing, and I omitted some significant results, such as domain testing by White and Cohen (1980). If you are interested in the history of testing, I suggest two articles: The Growth of Software Testing and The History of Software Testing.

Check this out: Automated Functional Testing- What it is & How it Helps?

Efficient Test Design – The Why, Which, and How ?

Istvan Forgacs — Tue, 01 Nov 2022 11:22:42 +0000

Why good test design is inevitable?

Many books, blog posts, etc. state that either you spend lots of money and time for testing to reach high quality or you save money but your software quality remains poor. It’s not true. The reality is that there are two significant cost elements in the software development lifecycle (SDLC) that the teams can modify: testing and bug fixing. Neither of them is linear.

The cost of testing increases when we want to find more bugs. In the very beginning, the increase is more or less linear, i.e., executing twice as many test cases mean the number of detected bugs will be doubled. However, by adding more test cases the cost becomes non-linear. The reason is that after a certain level we need to combine (and test) the elements of the input domain by combinatorial methods. For example, we first test some partitions, then their boundaries, then pairs or triples of boundary values, etc. Suppose that the number of test cases is not negligible in the project. There is no reason to design and execute too many tests since we are unable to find a significant number of bugs even with many additional tests.

The other factor is the cost of defect correction (bug fixing). The later we find a bug in the software lifecycle, the more costly its correction. According to some publications, the correction cost is “exponential in time” but clearly over linear, most likely showing polynomial growth. Therefore, if we can find faults early enough in the lifecycle, then the total correcting costs can drastically be reduced. As you can see below, considering these two factors together, the total cost has an optimum value. (The figures are not real, only demonstrate the concept)

This means that it’s not a good strategy to reduce testing and testing costs as you should consider these two factors together. The main question is — does this cost optimum result in an acceptable production code quality? This is not a simple question. Assume that you apply an inefficient test design. In this case, with a high testing cost, the number of undetected bugs remains high and even if you reach the optimum, the software quality may not be acceptable. On the contrary, if you apply very efficient test design techniques, the optimum will be moved to a point where the detected bug ratio is much higher resulting in an acceptable quality, see the figure below:

Thus, an efficient test design technique is necessary for the SDLC cost to decrease and the code quality to become acceptable at the same time. However different features require a different level of test design. Let’s consider two implemented features, and assume that the code of the first one is more complex (in some aspects). Hence, it contains faults with a higher probability. If we test both at the same expense and thoroughness, then after testing, more faults will be left in the first code. It means that the total correcting cost will be higher.

We can use similar arguments for considering risk. Suppose that we have two features having the same complexity and testing cost (the distribution of faults is similar), but the first one is riskier. For example, this feature is used more often, or the feature is used in a “critical” flow. Therefore, more bugs will be detected and fixed in this function with a higher correcting cost during the lifecycle. Note that we are speaking about the detection of bugs here, not about the presence of bugs.

Roughly speaking, more complex code and higher risk raise the bug-fixing costs if the testing effort/cost remains unchanged. Since this additional bug fixing occurs later in the lifecycle, the code quality at release will be poorer. Therefore, a riskier or more complex feature must be tested with more thoroughness, i.e., with higher testing costs. This higher testing cost is inevitable to achieve the optimal total cost, see this figure below, where HRF is a high-risk function and LRF is a low-risk function.

For traditional risk analysis, there are two factors, impact and probability. Here we can consider complexity as a third factor.

By mapping the size of the risk to the features, we can make the test design close to the optimum. For riskier code, we should select stronger test design techniques. A good option is to use the same test design technique with different test selection criteria. For example, if you use state transition testing or action-state test design technique you can select the minimum test selection criterion, all-transition testing for low-risk features, or riskier features a stronger all-transition pairs criterion. You can use even stronger criteria, if necessary.

Let’s assume that you do opt for stronger test design techniques for riskier features, but how can you check that you are close to the optimum? This is not an easy task. The only way is to collect as much data as you can. For example: let’s measure your testing effort for the features separately. Apply a simpler criterion first, then extend it and measure both efforts. Then measure the bugfix cost for the detected bugs. Finally, measure the bugfix cost for the bug detected after release. Now you can calculate the sum of testing and fixing costs for applying the weaker and the stronger criteria by approximating the bugfix cost for the simpler criterion (for the bugs detected by the stronger criterion). Finally, consider the risk. If the total cost for the simpler criterion is lower then you can select this criterion for that risk value. Using these data, you can select the appropriate test design level for a given risk.

If you are looking for Selenium IDE Tutorial you might find this blog useful, check this out- Most Comprehensive Selenium IDE Tutorial

Which tests should be automated?

Nobody automates all the test cases. Some features are not very important or the probability of changing the functionality is close to zero (assuming this feature is independent of the others in the static analysis sense). Test automation cost is significant with respect to single manual test execution. Is it reasonable to automate these features? Of course not. There may also be features that will be changed only once or twice. If test automation costs five times that of a single manual test then automation is worthless.

On the other hand, if a bug would block the use of the software or the feature will be modified several times or influenced by many frequently changed features, then test automation is a must, no doubt about it.
Again, we can do a risk analysis. If the impact is very high or high, then we should automate the test even if no change will occur. The reason is that even if a feature remains unchanged, some other features may influence it. If these ‘influencer’ features are changing, the unmodified may go wrong. The question is how to calculate risks. Obviously, the impact is a key factor. Probability is another to consider. However, instead of complexity, we should consider another type of probability, i.e., the probability of change.

Thus, risk analysis happens once and all the four risk items (impact, probability, complexity, and probability of change) should be determined by the team. You can use the well-known risk poker. According to the risks, you can decide which features to be automated. The risk analysis also helps if you want to make regression testing in different intensities. For example, you can execute some feature’s tests with high risk after each code modification and some others with lower risk just once a week.

The final step is the validation to improve the process. Here you can validate whether the automation of features was reasonable. You know the bugs detected during automated test execution, the bug fixing cost related to these bugs, you measure the test automation cost and you have historical data about the average bugfix cost of the bugs detected after release. Based on these data you can calculate whether automation was reasonable or not. If not, you can fine-tune your risk-based strategy.

Check this out: Appium Automation Tutorial- A Detailed Guide To Appium Testing

How tests should be automated?

Excellent! We have a good test design and we know what to automate. The only thing to do is the test automation itself. At the first glance, tests are easy to automate. Test cases consist of user actions and system responses. In both cases, you should identify the UI object under test. For any test automation framework, the solution is similar. For actions, you use a selector/locator and a keyword. Here is an example using Cypress:

cy.get("#add Coke").click()

Here #add Coke selects the ‘+’ button of adding a coke, and click() is the keyword to click on this button. For validation the solution is quite similar:

cy.get('#birthday').should('have.value', '11/01/1983')

You can write the test code very fast, however, there is a significant problem: DRY, i.e., Don’t Repeat Yourself. DRY ‘is a principle of software development aimed at reducing repetition of software patterns’. As probably more test cases contain clicking on the add coke button, the same statement will be repeated. During maintenance, if the test is changing you should change it in many locations.

To avoid this, the page object model (POM) is suggested. The page object model is a design pattern where the page objects are separated from the automation test scripts. You should create separate classes for every page in the application. Then individual methods for each action and response are created and also the selectors are separated. For example, here is the POM version for adding coke:

const addCokeSelector = "#add Coke";
class Example {
  addCoke() {
    cy.get(addCokeSelector).click();
  }
}

it('add coke', () => {
    const example = new Example();
    cy.visit('[https://site_of_add_coke](https://site_of_add_coke))
    example.addCoke();
})

Now if the selector or action code should be modified, you can do it at a single location. Very good, we are done. Oops, there is a problem: the POM code is much larger than the original. Instead of a single statement, there is a variable declaration for the selector, there is a method for the action/response and there is a call to the method. Roughly speaking we tripled the code size to keep DRY. However,

some actions/responses are never modified.
some actions/responses are in a single test case and only once. In this case, modification occurs just in a single location.

You may think that ‘no problem, let’s use risk analysis again’. Unfortunately, here we should analyze each action/response separately which is very costly. What can be done now? Fortunately, there is a good solution: model-based testing!

Model-based testing (MBT) is a technique that inherently involves DRY solution. The essence of MBT is that instead of creating test cases one by one manually, we create an appropriate test model from which an MBT tool can generate test cases based on the appropriate test selection criteria. The model can be textual or graphical. Models are based on test design techniques. For example, statecharts are the representations doing state transition testing. Here is the statechart of the well-known ATM authentication:

Satisfying the all-transition pair criterion, the action ‘insert valid Card’ is involved three times. In the model, this action occurs only once. This means that this action is implemented only once, and in case of any modification, you should modify it in a single location. Models are obviously more compact than the objects the models represent. My experience tells me that the textual model consists of a third of the simplest (non-POM) test code. Thus, the POM version is approximately nine times larger than the model. In another blog, I will compare the different MBT methods, but every method is better than using the original POM.

Another advantage of using MBT is that test design and test execution automation are merged. We informally proved that advanced test design is a must. Making a separate test design, then writing the tests one by one is entirely superfluous. Making a model and then generating the tests is a fast and efficient solution. Maintenance is also efficient as you never touch the code, only the model. The test code is generated after each model modification.

This means that using MBT and a test execution framework together is the best approach resulting in high software quality and lower SDLC cost.

Check this out: A Complete End to End (E2E) Testing Tutorial- Comprehensive Guide With Examples and Best Practices

Conclusion

There are many misconceptions about software testing. Here I showed that using an efficient test design doesn’t only lead to better quality but also decreases the total SDLC cost. I tried to answer the question of which test cases should be automated. Finally, I showed that using MBT makes test automation work more efficient. Don’t throw your test automation framework away, but extend it with test design automation tools. In another blog, I will show how to do it.

I hope I answered all the questions raised in the title.

OK, Model-based Testing, but Which One?

Istvan Forgacs — Mon, 31 Oct 2022 11:27:06 +0000

In my preceding blog on efficient test design, I showed that using model-based testing not only improves software quality, but it’s more efficient than coding test cases. Great, but there are so many model-based testing (MBT) alternatives, how can you select among them? I show you the different approaches and their advantages and disadvantages.

There are different classifications of the MBT methods. For example, the modeling languages, textual vs graphical, mode, i.e., online or offline, etc. Here I introduce another classification based on efficiency and usability in test automation. In this classification there are only three classes:

Stateless
Stateful
Aggregate

Here I only consider the first two and in a subsequent blog the third one.

Check this out: A Complete End to End (E2E) Testing Tutorial- Comprehensive Guide With Examples and Best Practices

Stateless model-based testing

The first class of the MBT methods is the stateless MBT. Here the business processes are modeled, describing the dynamic aspects of the systems. Examples of these models are BPMN, UML activity diagrams, use cases, etc. All these solutions have different notations, but there are very similar with respect to the information they involve. All of them consist of user actions, events, and maybe system responses. It’s more important what they don’t consist of: states. Most MBT tools use this technique. These models are successfully used in software engineering and you can think that it’s a perfect solution for MBT. Unfortunately, when used for testing there are some issues.

Any stateless model can be transformed into a similar graph, see the example below. The tests are generated based on some graph traversal and test selection criteria. The first problem is that there are infeasible paths in the graph. To avoid this problem, MBT tools offer the usage of constraints. Constraints are conditions to prohibit invalid paths. For example, a constraint is when transition b cannot precede transition a.

However, sometimes guard conditions are also needed. A guard condition here describes when a given action/event can happen. This means that the modeling requires some coding. Using this method an MBT tool cannot be entirely codeless.
However, the bigger problem with these MBT methods is that as they do not consider states, they may not find even a simple bug. For example, a frequent bug is when a code location has a correct state for the first time it’s traversed but becomes incorrect during some subsequent traverses. For example, paying is not possible below 20 Euros, but adding food reaching 21, then deleting an item to go below 20, the paying remains possible.

To demonstrate stateless MBT, here is a simple example.

A rental company loans cars (EUR 300), and bikes (EUR 100) for a week.
R1 The customer can add cars or bikes one by one to the rental order.
R2 The customer can remove cars or bikes one by one from the rental order.
R3 If the customer rents vehicles for EUR 700, then they can rent one bike for free. In case of discount:
R3a If the customer has selected some bikes previously, then one of them becomes free.
R3b If the customer hasn’t selected any bike previously, then one free bike is added.
R3c When the discount is withdrawn (see R4) but given again, and no bike was added meanwhile, the customer gets the previous discount back.
R3d When the discount is withdrawn and some bikes are added, when the discount is given again, then one of them becomes free.
R4 If the customer deletes some cars or bikes from the order so that the discount threshold doesn’t hold, then the free bike will be withdrawn.

Here is a simple stateless (or flow) model of the requirement specification above. The edges are the user actions and the nodes are the system responses.

This model permits any valid test case as any add car / add bike / delete car / delete bike sequence can be traversed in the graph. From start, only a car or a bike can be added. Excellent, you can think that the model is good, but it isn’t.

As mentioned, there are invalid paths leading to non-realizable tests. For example, you can traverse a path of adding a car and then deleting two cars. However, the related test would result in a negative car number in the cart. In general, you can only delete existing elements from the cart. Hence, several invalid paths exist in the model and the usage of constraints is not enough.

Instead, guard conditions are needed, i.e., the modeling requires some coding. For example, the guard condition for a transition ‘delete car’ starting from other than node ‘car added’ is:

number_of_cars ≥ 1.

Hence, variables must be handled and keep them up-to-date, e.g.:

number_of_cars++ or number_of_cars–

We should add similar code and guard conditions to transitions when deleting a bike happens.

As mentioned, a more significant problem is that this method will not find some bugs that other methods will. To see why, let’s select the ‘all-transition-pairs’ criterion, where all the adjacent transition/edge pairs should be covered. Though the number of such pairs in the extended graph is numerous (16), still the tests are not reliable, i.e., will not find the bugs.

From the requirements it’s obvious that you should test the following:

Add some vehicles to add a free bike.
Delete some vehicles until the free bike is withdrawn.
Add some vehicles again to see if the free bike is given back.

However, the transition pairs needed to satisfy the criterion is:

add car, add car, delete car (2)
add car, add bike, delete car (2)
add bike, add car, delete bike (2)
add bike, add bike, delete bike (2)
add car, delete car, add car (1)
add car delete car, add bike (1)
add bike, delete bike, add car (1)
add bike, delete bike, add bike (1)
add bike, add bike, delete bike, delete bike (1)
add bike, add car, delete bike, delete car (1)
add bike, add car, delete car, delete bike (1)
add car, add car, delete car, delete car (1)

The first four lines cover two-two pairs, the others just one (with bold). You can see that a test set satisfying the criterion may not cover the first step of the test (add car, add car, add bike).

Therefore, even the first step of T1 will be missing. Even a stronger criterion will not detect the bug which T1 detects.

Yet, the advantage of the method is that it can be generally used, and if states are not relevant (see next chapters), then it can be efficiently used. That’s the reason that most of the model-based testing tools (CA Agile Requirements Designer, Eggplant, Perfecto, Curiosity) apply this technique.

Check this out: Selenium Tutorial- A complete guide on Selenium automation testing

Stateful model-based testing

Let’s do a Google search for ‘state transition testing’! All the examples contain systems with a very limited number of states such as ATM authentication, setting time and date, and switching a lamp on and off. In practice, the number of (program) states are huge and cannot be used for state transition testing resulting in millions of test cases. The only solution is to reduce the number of states. This can be done if we consider only ‘inner states’ and guard conditions.
For example, considering our requirement specification, program states involve the number of bikes and cars and some inner states. In this case, there are several states/nodes in the graph, resulting in too many test cases. We can reduce the number of states but how? An appropriate solution is to consider only the inner or test states.

In our example we have five inner/test states:

No discount, no bike.
No discount, bike included.
Discount, bike added.
Discount, bike converted.
Stop.

The advantage is obvious: as the minimum requirement from a test set is to cover each state, the discount will be tested. The state transition graph is here.

Again, let us assume that the test selection criterion is the all-transition pairs. Let’s consider our test case again:

Add some vehicles to add a free bike.
Delete some vehicles until the bike is withdrawn.
Add some vehicles again to see if the free bike is given back.

The first step is covered by reaching the state Discount, bike added. From here, there is a transition delete car that should be traversed. With this, step 2 is covered. As all the adjacent transition pairs should be covered, we should select add car that goes back to the state Discount, bike added. This means that to satisfy our criterion, test T should be added, see this pair emphasized below:

paths are generated. For example, from the starting point, traversing add bike to go to state Discount, bike converted is invalid as before ‘add car’ should be traversed twice. Thus, guard conditions are required.
For example, the guard condition for the add car transition starting and going back to the No discount, no bike state is:

total price < 400.

However, the total price is output, thus you should code it according to the requirements.

What has happened? You modeled an application that computes the total price of items in the cart. However, while making the model you should code the total price that is the task of the implementation. It’s obvious that you can make mistakes while making this code and the tests may become wrong. It’s a simple example, and there are cases when coding the output can be more difficult. When there are many transitions, adding the necessary guard conditions is time-consuming and error-prone.

If the guard conditions contain only inputs, then the graph will not contain the output values as in a state it can be different according to the path traversed. When the tests are generated, you should add the correct outputs for each test case.

Another problem is that when there are no inner states in the system, how can the states be handled? It’s not easy as you should ad-hoc cut the states not knowing whether the tests based on the reduced graph remain reliable. I think in this case the stateless solution is simpler and leads to the same result considering defect detection.

That may be the main reason why state transition testing is not widely used among testers and much fewer tools implementing it exists. Such tools are Opkey and Conformiq Creator.

Check this out: Execute Playwright testing in parallel and cut down your test execution time by multiple folds

Conclusion

We reviewed two classes of model-based testing. The simplest and most widely used technique is the stateless solution. These tools can be generally applied and easy to use, but there are two shortcomings:

they require constraints/coding;
they detect only simple bugs for complex systems including inner states.

The state-transition testing method is more difficult to use but more reliable, i.e., it can detect tricky bugs for more complex systems. These methods also have two shortcomings:

they require coding, sometimes the expected output should be coded;
for less complex systems, determining the necessary states is difficult, and so is the state transition graph

An obvious question is which one should I select. The answer is neither of them. The third class, ‘aggregate’ is better. A specific MBT method referred to as action-state testing addresses all the issues of these methods. In my next blog, I will show this technique and its usage.

And the Winner Is: Aggregate Model-based Testing

Istvan Forgacs — Fri, 28 Oct 2022 06:50:24 +0000

In my last blog, I investigated both the stateless and the stateful class of model-based testing. Both have some advantages and disadvantages. You can use them for different types of systems, depending on whether a stateful solution is required or a stateless one is enough. However, a better solution is to use an aggregate technique that is appropriate for each system. Currently, the only aggregate solution is action-state testing, introduced in the book Paradigm Shift in Software Testing. This method is implemented in Harmony.

Check this out: Blackbox Testing Tutorial - A Comprehensive Guide With Examples and Best Practices

Where can we use Action-state testing?

Action-state testing can be used for both stateful and stateless cases or even a mixed solution is possible if a long test case consists of inner states at some test steps, but at some other steps don’t.

The ‘action-state’ model is textual, containing ‘action-state’ steps. Test cases are generated from the action-state model. An action-state step is a triple containing

a (user) action,
a (system) response,
the inner (test) state.

Actions and responses must be implementation-independent and as abstract as possible. Action is described at a high level so that humans can understand. For example, an action can be as:

add items so that price remains below 10
This is more abstract than adding two more concrete actions:

add item for 7
add item for 2

Even more, concrete actions would be:

click on the ‘+’ icon next to Pizza Mexicana
click on the ‘+’ icon next to Coke

However, these are implementation-dependent actions and can be created when the implementation is ready. Making an implementation-independent model is a huge advantage as it can be done before coding and the team can validate it against the specification detecting almost all the problems in it.

Modeling is easy: a subsequent step (step 2) is executed immediately after a step (step 1) if step 2 is indented to the right:

step 1
step 2

If two steps’ indentation is the same then they will be executed in different test cases:

step A
step B.

step A may contain an action and a response (stateless step):

action A => response A

or may contain a triple involving the inner state:

action A => response A STATE state for A

Models can be easily mapped with the requirements:

R1 my requirement A:

action A => response A STATE state for A

One model can embed other models. In this way, very complex systems can also be modeled.

You can easily build a model related to a feature or user story. From this model, the tool can generate the statechart and the test cases immediately. The statechart is another model representation, see later. By applying statecharts, you can easily detect missing actions and states. On the other hand, a statechart contains less information. The information on whether a transition is traversed more times is missing. For example, entering the wrong PIN for the first and second time is modeled in this way:

Add wrong PIN => wrong PIN added STATE waiting for PIN
Add wrong PIN => wrong PIN added STATE waiting for PIN

The related statecharts are the same independently of only once or twice the wrong PIN was added. Action-state models contain more information than statecharts and this is the reason why this technique doesn’t need guard conditions and consequently coding.

However, to avoid coding the models are a little bit larger.

You can also set an appropriate test selection criterion. A tool can offer the missing action-state steps to satisfy the criterion. There may be steps to be offered that are invalid. Fortunately, you can easily recognize an invalid step and it can be rejected. Accepting or rejecting an offered step is the replacement of adding guard conditions and some other coding. It’s much easier and non-coder testers can also do it.

Let’s consider the same requirements as in my previous blog:

The most difficult part is how to find appropriate states. A reasonable solution is that if two actions are the same, but the system calculates the response in different ways, then we arrived at different states. Here is an example. The initial state is ‘No discount, no bike’. From here we can add a car for which we go back to the initial state as no discount happens. Adding a second car, the calculation is the same as a car added to the cart. However, when ad the third car, the calculation will be different as a bike is also added for free. Thus we arrived at a new state ‘Discount, bike added ’. If we start with adding a bike and then adding three cars in a row, then there is another different calculation as the bike becomes free. Thus this is a new state: ‘Discount, bike converted ’. It’s obvious that different calculations can be tested as any may contain an error. In this way, you can select a minimum number of test states.

Adding a bike at the same level as the first car, we go to another state ‘No discount bike included’. All these steps cover requirement R1. This means that covering a single requirement with a single test is usually not enough. Here is the first version of the model:

INITIAL STATE No discount, no bike 
R1 The customer can add cars or bikes one by one: 
    add car => one car STATE No discount, no bike
        add car => two cars STATE No discount, no bike 
    add bike => bike added STATE No discount bike included

and the related statechart:

You can see that the statechart doesn’t show that we added two cars.
For covering requirement R2, we should delete a car/bike that should be a child step of a parent step where at least one car/bike is available:

INITIAL STATE No discount, no bike 
R1 The customer can add cars or bikes one by one: 
    add car => one car STATE No discount, no bike
        add car => two cars STATE No discount, no bike 
          R2 The customer can remove cars or bikes one by one:   
            delete car => one car STATE No discount, no bike     
    add bike => one bike STATE No discount bike included 
      R2 The customer can remove cars or bikes one by one:  
        delete bike => cart empty STATE No discount, no bike

To cover R3a, we should add two cars to the single bike:

add bike => one bike STATE No discount bike included 
      R3a discount for EUR 700 when bike converted free:
        add two cars => two cars and a bike STATE Discount, bike converted

R3b can be covered if we add a third car:

add car => two cars STATE No discount, no bike 
          R3b For discount a bike is added when only cars are in the cart:
            add car => three cars and a bike STATE Discount, bike added

Now it’s reasonable to continue with R4. To cover it we remove the third car:

add car => three cars and a bike STATE Discount, bike added 
              R4 going below the discount limit it is withdrawn:
                delete car => two cars STATE No discount, no bike

Considering R3c, we should add the deleted car again:

delete car => two cars STATE No discount, no bike 
                  R3c reaching the limit again, previous discount back:
                    add car => three cars and a bike STATE Discount, bike added

Our last requirement is R3d. We cover it by deleting the second car, adding a bike, and adding the second car again:

delete car => two cars STATE No discount, no bike 
                  R3d reaching the limit again adding a bike before:
                    delete car => one car STATE No discount, no bike 
                      add bike => one car and a bike STATE No discount bike included 
                        add car => two cars and a bike STATE Discount, bike converted

The whole model is the following:

INITIAL STATE No discount, no bike 
R1 The customer can add cars or bikes one by one: (1)
    add car => one car STATE No discount, no bike
        add car => two cars STATE No discount, no bike 
(2)       R3b For discount a bike is added when only cars are in the cart:
            add car => three cars and a bike STATE Discount, bike added 
(3)           R4 going below the discount limit it is withdrawn:
                delete car => two cars STATE No discount, no bike 
(4)               R3c reaching the limit again, previous discount back:
                    add car => three cars and a bike STATE Discount, bike added
(5)               R3d reaching the limit again adding a bike before:
                    delete car => one car STATE No discount, no bike 
                      add bike => one car and a bike STATE No discount bike included 
                        add car => two cars and a bike STATE Discount, bike converted
(6)       R2 The customer can remove cars or bikes one by one:     
            delete car => one car STATE No discount, no bike       
    add bike => one bike STATE No discount bike included 
(7)   R3a discount for EUR 700 when bike converted free:
        add two cars => two cars and a bike STATE Discount, bike converted   
(6)   R2 The customer can remove cars or bikes one by one:  
        delete bike => cart empty STATE No discount, no bike

Hurray, we covered the requirements with only five test cases (the ones (1) — (5)). Are we ready? Let’s see. The related statechart is here:

Check this out: Usability Testing- A Comprehensive Guide With Examples And Best Practices

It’s a great help as missing actions can be detected easily. For example, from the state ‘Discount, bike added’ we can delete or add a bike. Considering ‘No discount bike included’, there is a missing edge going to itself, when deleting a bike from two. We can also add bike going to itself and going to ‘Discount, bike converted’. Finally, considering ‘Discount, bike converted’, there are several missing actions, i.e., adding/deleting a car and adding/deleting a bike to itself, delete bike going to ‘No discount no bike’ and delete car/bike going to ‘No discount bike included’.

Traversing each valid action/edge is the minimum test selection criterion, thus just covering the requirements is way insufficient. Involving the missing actions, we get the following improved model:

INITIAL STATE No discount, no bike 
R1 The customer can add cars or bikes one by one: 
    add car => one car STATE No discount, no bike
        add car => two cars STATE No discount, no bike 
          R3b For discount a bike is added when only cars are in the cart:
            add car => three cars and a bike STATE Discount, bike added 
              add bike => three cars and two bikes STATE Discount, bike added 
              delete bike => three cars STATE No discount, no bike  
              R4 going below the discount limit it is withdrawn:
                delete car => two cars STATE No discount, no bike 
                  R3c reaching the limit again, previous discount back:
                    add car => three cars and a bike STATE Discount, bike added
                  R3d reaching the limit again adding a bike before:
                    delete car => one car STATE No discount, no bike 
                      add bike => one car and a bike STATE No discount bike included 
                        add car => two cars and a bike STATE Discount, bike converted  
          R2 The customer can remove cars or bikes one by one:     
            delete car => one car STATE No discount, no bike       
    add bike => one bike STATE No discount bike included 
      add bike => two bikes STATE No discount bike included 
        add five bikes => seven bikes STATE Discount, bike converted 
          add bike => eight bikes STATE Discount, bike converted 
            delete bike => seven bikes STATE Discount, bike converted 
          delete bike => six bikes STATE No discount bike included 
        delete bike => one bike STATE No discount bike included  
      R3a discount for EUR 700 when bike converted free:
        add two cars => two cars and a bike STATE Discount, bike converted  
          delete car => one car and a bike STATE No discount bike included 
          add car => three cars and a bike STATE Discount, bike converted  
            delete car => two cars and a bike STATE Discount, bike converted 
          delete bike => two cars STATE No discount, no bike 
          add bike => two cars and two bikes STATE Discount, bike converted 
      R2 The customer can remove cars or bikes one by one:  
        delete bike => cart empty STATE No discount, no bike     
      delete bike => cart empty STATE No discount, no bike

The number of test cases increased from 5 to 13. The new statechart is the following:

Now all the actions are involved. We have seven requirements and we carefully covered by 13 test cases. You can think that 13 tests are too much, but it depends on the risk analysis. If the risk is high, you should test it carefully, otherwise tricky bugs remain undetected. You cannot merge cars and bikes as vehicles as they behave differently, only bikes are added freely or converted for free, and cars are not.

Considering the statechart, if you simply traverse it, some invalid tests are generated. For example, adding a bike from the initial state, then adding five bikes we arrive at ‘Discount, bike converted’. However, having only six bikes we have no discount, a contradiction. The state action-state model consists of only valid steps.

As mentioned, by applying a stronger test selection criterion, new steps are offered. For example, when applying all-transition-pairs criterion the following step is offered:

This is because the edge/action pair (add car, add bike), where add car is the action starts from and arrives at the initial state hasn’t been covered. If the risk is very high you should add these steps as well.

Check this out: Automated Functional Testing: What it is & How it Helps?

Conclusion

Action-state testing is an aggregate MBT method. This is the only one that requires no coding at all, just modeling. The model can be converted to a statechart, by which the original model can be improved and made complete. It can be widely used, and tricky bugs can be detected. From the models, abstract test cases are generated that can be manually executed. The model consists of implementation-independent steps, thus it’s a true shift left MBT method.