Introducing yet another method for developing functional test cases. Explore what happens when you base your approach on the architectural diagrams of the system being tested.
By reading this blog post, you will
- Master the Technique for Simplifying Complex Structures, which involves deconstructing systems into components, making it easier to prepare functional test plans.
- Enhance your Analytical Thinking through a step-by-step problem analysis framework.
- Gain a solid understanding of Test Design Strategies by considering both positive and negative scenarios in functional test case design.
- Add a systematic and comprehensive approach to test design to your skillset, essential for anyone interested in test design and software testing.
Introduction
I work in the field of automated testing and have repeatedly gone through the process of entering mature projects. Unfortunately, this usually takes more time than desired because a tester must have a good understanding of the business requirements, logic, and technical architecture of the systems being tested.
During one of my interviews, I was asked a rather common question in the field: "How do you write test cases, or in other words, what is your method for developing tests that cover the functionality of the product?" Surprisingly, the question stumped me. I had read about the traditional approaches and an interesting concept from the book "How Google Tests Software," but none of them were fully applicable in my case.
That's when I decided to formulate my own methodology and share it with others. I hope you'll find this article helpful, even if you're already using something similar.
Meet my lovely ginger cat to brighten your mood 😺
The Problem
Imagine you've just joined a project. Everyone is smiling, welcoming the newcomer, and always ready to help. As a first step, you open the documentation and diagrams to understand how everything works. And you're faced with something like this:
Gathering your strength, you start deciphering the "magic" hidden behind all these technicalities. Often, the system's API is not documented at all, and even when it is, it's hard to find in the architectural diagrams. The same applies to the business entities your project is dealing with. It's nearly impossible to find them within the diagrams. Lastly, we all know from experience that components can be in different states (simply available or not), as can the models: the mere presence of a status field automatically adds several states to an object. Listing all these states in the diagrams is something I've never seen (call it bad luck).
Now, let me tell you what I do in such situations.
The Essence
I start with the idea that any system being tested can be viewed as a set of components (these could be microservices, packages/modules, classes, etc.) and models that are either stored within it or passed through it. With this premise:
- The components operate with a set of models, such as a user or order.
- Components and models can have states. For example, a user can be logged in, deleted, and so on. Some components may be available, unavailable, or "slow".
- Components have some interface (GUI, REST API, GRPC, event subscriptions, CLI, a set of public methods, etc.), which consists of methods (or event subscriptions), their parameters, and the results of their calls – i.e., returned responses, additional calls, dispatched events, or changes to the state of models or the component itself.
Thus, if you create a system diagram consisting of components, marking their models, interfaces, and states, you can, first and foremost, visualize how the system works and what can affect it. Secondly, you can use combinatorics to develop a functional test plan for that system.
Let's examine an example
I. Creating a scheme
Let's imagine that we need to test a component of our system responsible for handling orders: the "Order component".
a. First, let's determine the components that interact with our "Order component"
It has a database with orders, interacts with an external "Payment provider" service for processing payments, and there is also an event bus where our component sends events about API calls.
All our components may be in one of the following states: available, unavailable, slow (requests are processed slowly). This includes the event bus.
b. Now let's picture the models this part of our system operates with
Undoubtedly, it is the Order entity. If we think a little, only a user can create an order in our application, meaning that the order stores information about the user, and the API of our service can also be used by the user. So the second model is User. Moving forward, orders also contain information about their contents, i.e., the products, so the third model is Product.
Let's figure out in which states our models can be:
User can be in one of the following states: non-existent, logged in, not logged in, deleted. Also, if they exist, they may be in one of 2 states: bank card information is provided or not.
Product can also be in one of the states: non-existent, has a quantity available (0 or more), deleted.
Order can be in the states: non-existent, created, deleted, paid, and closed.
c. Let's determine what interfaces our system components have
-
I. The "Order" service we are testing:
- We can create an order;
- Cancel or delete an order;
- Pay for an order;
- Close it.
For simplicity, let's leave only the "payOrder" method for analysis. It accepts the order ID as an argument. However, from the business logic perspective, only a user can call this method. Additionally, the order contains information about the product and the user who owns the order by its ID. So the complete list of business arguments for the payOrder method is:
User who called the method;
Order;
Products in the order;
Now let's figure out what happens when the payOrder method is called:
Validation of arguments;
Calling an external service to make a payment and awaiting a response;
Updating the order status if it has been successfully paid;
Sending an event to the data bus about whether the payment was successful or not;
Returning a response: the payment was successful or not.
An important note: as testers, we don't know the exact order of these events. Keep this in mind and take it into account.
Models that the service works with, based on the arguments of the tested method, are User, Order, Product.
- II. The "Orders DB" database:
It supports CRUDL for orders. Orders contain information about products and their owner users. Responses to calls are success or failure, as well as the absence of response. So the models are the same: User, Order, Product.
- III. External integration - "Payment provider":
Used by the Orders component only when calling the payOrder method. It has an important API for us, in the form of a single method called payByCard, which accepts the order number, its amount, and the user's bank card information. Responses to calls are also success or failure, as well as the absence of response. So, as per our business understanding, the models used are User and Order.
- IV. Also, we have the "Event Bus":
The Order component sends events about API calls to it. For payOrder calls, there will be two events depending on the status: orderPaid and orderPayFailed. These events contain information about the order and the reason for failure for orderPayFailed.
Phew, it seems to be ready. Now we have a clear scheme in front of us that shows how our system works, its components and their connections, the business entities they manipulate, and their interfaces. The only thing left is to add a scheme for what happens when the tested component methods are called.
II. Test Design
We have already looked at what actions should take place when we call the tested method. Let us recall and analyze the order of these actions:
a. Argument validation
Let's go step by step. We pass metadata about the user who called the method. We pass the order ID, which in turn also contains the user who created the order. Thus:
Our method should return an error and attempt to send an
orderPayFailedevent if the user who called the method is invalid, meaning, for example, deleted;Next, the order must be validated to collect information about it. The component needs to access the database. If the database is unavailable, we must return an error and send an event to the bus that the payment failed (and if we think about it for a second, we should also trigger alerts and metrics about this because it seems abnormal, so we noticed another moment worth checking);
We collected the order data, compared it, and the order must exist, that's the first thing. Secondly, it must be in only one state - Created, otherwise we send the failure event and return an error. And finally, the user who called the method must match the one who created the order, otherwise we have an error and a failure event.
It seems ready, let's look over everything one more time, and… We notice that the order contains products, and products can also have different states. What if we are trying to place an order containing products that have already been deleted? And if the number of products in the order is greater than what we have left? Perhaps it is worth adding another component to our scheme and adding a step similar to checking with the database, only we will send a request to reduce the quantity of remaining products. If this call is unsuccessful, we must go through the negative scenario. This way, we found out that we missed an entire interaction, but by describing the scheme competently, we were able to notice this at an early stage - when composing test cases.
That's better now. If all checks have passed, we move on. Next, we must call an external service to pay for the order using a card. Stop. We have two more user states: card information present or not. This also needs to be validated, and if there is no card data, go through the negative case again.
That's it for now. With validation finished, let's move on to the next step. Additionally, for negative scenarios, we must check that further steps have not been executed, subsequent calls have not been made, and nothing has changed in the database.
b. Calling an external service to make a payment
Let's start with the states in which the service can be. It can be available or not (in fact, it can also respond slowly, this option should also be considered, but we will skip it to speed up).
If the service is unavailable, the negative branch is executed, returning an error and sending an
orderPayFailedevent. In real life, it would be worth adding an alert and metric check and adding components to the scheme responsible for this.If the service is available, we need to check the two possible responses - success and failure. If we get an error in response, the negative branch goes on; if success, we move on. It's worth thinking about the cases where we might get a negative response. It could be an invalid card, payment amount issues, or defects on the payment provider's side. These cases need to be checked.
The third option, which is also possible, is the absence of a response. It is necessary to check how our system and the tested Order component will react to this situation.
If this part is successful, the following actions should be performed in parallel; the order does not matter much to us here, but with one nuance. Our component must succeed in the next steps because the user has already paid for the order.
c. Updating the order status
The tested component must access the database and update the order status to Paid. Here again, it is necessary to understand how the Order component will behave, depending on the state of the database. What will happen if it is available? If not? Once again, I note that in all the negative cases we have considered so far, we should check that the tested application did not try to update the order.
d. Sending an event to the data bus about whether the payment was successful or not
We have already checked the sending of payment error events in the negative cases reviewed earlier. I will only point out here that we should check what exactly is being sent in the orderPayFailed event. Is the error there correct? If everything went well, we need to send an orderPaid event about successful payment and check its body. Again, all these cases, both positive and negative, should take into account both a good scenario where the bus is available and a bad one where it is unavailable or slow. Do we have a buffer for such cases, and will we not lose events?
e. Returning a response: whether the payment was successful or not
Should happen in any case. Again, it is worth checking the error texts and codes in any of the cases.
Conclusion
We've obtained a set of scenarios that can arise when working with the tested component. We have described the main business segments of our system. These include models, component interfaces, states in which components and models can be, as well as interactions that occur when using methods from the tested interface. Then, we can notice that we used combinatorics for models, interfaces, components, and their states to form a set of test cases.
Thanks to this approach, we got a fairly extensive test plan. Since we used diagrams for everything and applied them when studying the functionality, we were able to notice that we initially missed a couple of points.
When I use it
This approach has been particularly helpful for me when I had to test critical parts of functionality where I wanted to be highly confident.
However, I actually refer to it always: both when I need to thoroughly test a feature, and when I need to figure out something new. Even when I need to quickly check something, I also use it but draw very abstract diagrams on paper or just keep them in my head.
Unit, Integration, E2E?
The considered example focused on component and integration testing to a greater extent.
But we could easily turn the cases, for example, into end-to-end tests. For the diagram drawn above, we would need to consider who is subscribed to the orderPaid and orderPayFailed events. Investigate what happens to subscribers upon receiving these events. We'd add, for example, withdrawal from reservation or, on the contrary, writing off products in the product component, sending notifications in the notification component using external services, and so on.
To turn the set into unit tests, we simply need to mock and stub everything around (although some say there are no stubs in unit tests) 👍
In addition, a UML diagram or a set of screens from an app can serve as a scheme. How to draw this is a matter of taste, convenience, and the level of the tested component.
Closing thoughts
Did I confuse you too much? :) Even so, this methodology only seems complicated at first glance. You can read it again and simultaneously put a piece of your system next to it. Break down everything exactly the same way for it. I'm confident that everything will go smoothly. In this case, practice is easier than theory 😉
I hope my article will be useful to you!
Share how you approach test design. I would also be happy to hear your opinion on how my thought process works. Drop me a message and let's discuss this topic.
And visit my website to find my public activities or read about my projects.
Good luck to everyone!
Top comments (0)