DEV Community: Evgeniy Ivanchenko

How many testers should ensure Dodo IS high quality

Evgeniy Ivanchenko — Thu, 26 Oct 2023 09:10:28 +0000

How many testers should ensure high quality in an IT company? I have been asked this question so many times that I have finally decided to answer it.

First of all, I have to admit that asking "How many testers does a company need to ensure high quality?" is too abstract, because there is no right answer. Anyway, I did a little bit of research and found some tips such as "one tester for every three developers". Such answers are very similar to tips like: "You should drink 2 liters of water" or "You should take 10k steps per day". These tips apply to everyone and to no one at the same time. There is only one correct answer to this question - "It depends.

Let's have a look together at what influences the ratio of testers to developers.

Critical products, where people's lives, safety and health depend, need more testers. Your ratio of testers to developers needs to be higher if you are developing a software for self-driving cars than if you are developing apps for selling pizzas.
Level of developers and testers. If you have three senior developers, a junior tester is not going to take over the tasks of those three developers. And the other way round, a senior tester is over-qualified for a team of three junior developers.
The culture of the company. If there is no culture in the company where all team members are responsible for product quality, but developers are only responsible for writing code and testers are responsible for testing this code and product quality. You need more testers than a company that has this culture. By the way, I have no idea how testers who don't write code for a product can be responsible for its quality.
The size and the complexity of the product. If you have a huge and complex product, you will need more testers than if you have a small and simple product. But remember that without deep knowledge of the product and its context, it's easy to underestimate the size and complexity of a product.
Process maturity If the whole team is responsible for quality and developers can test the product themselves before the tester starts testing: In this case you only need a few testers. You will need even fewer testers if the developers write not only unit tests, but also integration and end-to-end tests. You will need fewer testers if you have good monitoring, canary releases, or other ways of detecting problems quickly.
How to deploy. If you have a complex way of distributing your products, e.g. software for cars, and your customers need to visit the service center to get their software updated, you will need more testers compared to a company that only updates its products online.
The automation potential of the product. If your product is difficult to automate, you need more testers. Consider a company making printers. You can cover drivers with tests, but it's hard to automate integration of hardware and software. I really don't know how to automate the next case: Checking the text printed on the paper. Maybe we can use computer vision. But how much does it cost? And what about the ROI of this job?

Speaking of how many testers we need to ensure the high quality of Dodo IS: Let me give you some examples.

We have two teams with two developers and no testers. These teams own non-critical services. These services are mostly covered by unit, integration and end-to-end tests. These services can be rolled back in a few minutes. The developers in these teams can test the services themselves. Sometimes, before they build some features, they call in testers to help them with acceptance criteria.
There is another team that has three developers and one tester. This team owns our system's core service. But the engineering culture in this team is high: fast releases, fast rollbacks, and high coverage by tests - after every problem in production, this team will create a new set of automated tests that will cover that problem. Great monitoring and alerting allow testers from this team to not only test their own features, but also do a lot of work for the QA guild.
There are a number of teams that consist of 6 developers and 1 tester. Those teams develop mobile applications. Here the situation is a bit more complex. The developers are not as active in creating automated tests and in most cases rely on the testers for all quality and testing issues. For mobile apps, the cost of failure is higher than for web apps. You can't easily roll back your app on customers' devices. But for such teams, one tester is still enough because they have two backend developers, two Android developers, and two iOS developers, which means this team can develop two features simultaneously instead of a team of three fullstack developers developing three features simultaneously. That is why a single tester will be sufficient for this team.
And we have another option: two teams (7+3 devs) and 1 tester. In this case, one tester is enough, because the devs can partially cover some testing tasks. But there is one thing that might be unpleasant: the tester has to be part of two PBRs, two daily scrum meetings, two planning meetings, etc.

So, as you can see, there is no right answer in general. As far as I'm concerned, I'm sure that every two-pizza team should have at least one tester/QA. This person provides testing and QA expertise, but testing and creating autotests can be done by the whole team. Then review the above criteria and determine whether additional testers are needed. Now we have 22 testers/QA and 34 development teams. We haven't reached my absolute minimum number of testers. But we're moving in that direction. What is your ideal ratio? Or what kind of unique ratio have you found?

6 reasons why Dodo IS developers go to Gemba

Evgeniy Ivanchenko — Wed, 28 Jun 2023 09:51:59 +0000

As a QA engineer, I strongly believe that Gemba is a great opportunity for understanding your product. It helps to answer the question: “Am I doing bullshit or not?” Today I want to tell you more about its advantages and the insights that we got after visiting a pizzeria.

What does Gemba stand for

Gemba (Japanese 現場), Genchi Genbutsu (Japanese 現地現物) translates as “the real location, real thing” (meaning “the situation onsite”) denotes an approach specific to the Japanese kaizen management practice. According to it, it is necessary to come to Gemba — the place where the work process is performed, collect the facts, and make a decision directly on the spot in order to fully understand the situation.

Wikipedia

I will start from the beginning to talk about why we need Gemba. Our product is the Dodo IS — information system, which unites all parts of our business:

· B2C (website and mobile apps);

· B2B (apps for shift control, point of sale management, warehouse balances, cash register interfaces, etc.)

It is quite difficult to create a high-quality product without knowing how it is used and who works with it. To assess the quality, we use 2 approaches:

· Eat your own dog food (for B2C) — a method of probing user experience when employees in a company use the products they produce. This approach is quite easy to use in real life. Like all customers, we order pizza when we are going by subway, walking in the rain, at -30C, or lying on the beach. Flaws, bugs, and bad UX are easy to feel.

· Gemba (for B2B) — a method of probing user experience when employees in a company use the products they develop. But in our case, it implies a deeper immersion into the product and the environment because you cannot go to work as a pizza maker at any time or become a manager in a pizzeria in order to understand how the system works.

Who should go to Gemba

No one has to, but everyone can. Going to Gemba is a right. No one forbids going to a pizzeria, coffee shop, or doner, but no one forces you either. Testers, developers, and product managers can go there. The main thing about it is the difference in focus.

· For products, this is one of the main activities. In a pizzeria, they find ideas, get insights, and can see the whole picture. Product managers use Gemba for hypothesis testing, in-depth interviews, customer development, customer journey mapping, and so on.

· Developers and QA engineers go to Gemba with the aim of “touching” the product they have made. A side effect of this is uncontrolled insights and ideas for product development. After returning from Gemba, not a single engineer said that it was a waste of time.

Why go to Gemba

1. Cheap and cheerful

Going to Gemba is easy to implement. The developer goes to a pizzeria or coffee shop and uses the product. That’s all. This is easier than simulating a pizzeria or coffee shop in the office, inviting actual employees of the pizzeria, and fictitiously demonstrating their user experience in the conditions of a spherical cow in a vacuum.

2. Automation Insights

There can be insights about business process automation. One of our Android developers went to Gemba and saw what the marking process was like. Employees were COUNTING IN HEAD and HANDWRITING labels with the expiry dates of the products. For each product, they count and write the start date of defrosting, the validity date, and the expiry date. Product labeling is a boring routine process in which careless mistakes constantly occur.

The developer realized that this was absolute bullshit. To automate this was “half an hour of a programmer’s work,” so he went ahead and automated it. This is how the mobile application for printing labels was born. The operation is reduced to the following: an employee selects a heading on the tablet, makes a tap, and a label with all the terms is printed, that’s it.

3. Flaws in the field

In Gemba, the defects of the product are more evident. Dodo IS has features for auditing. These features were developed for the tablet. During the audit, an employee can enter data directly into the tablet, but it is too cold in the freezer and refrigerator, the tablet refuses to work, sometimes the connection is lost, and the interface itself is not very convenient. That’s why the results of the audit are written down on paper, and entered into Dodo IS later. Double work!

Before rewriting the interface of the auditor, the development team went to night revisions in order to understand the process. They returned with ideas for improvement. For example, if the connection is lost, then the input of information must be made asynchronous, the data must be stored in local storage, and sent to the server when the connection is restored.

Nighttime. Meanwhile, the developers are counting the tomatoes

4. Real working conditions

In a pizzeria, you can see the working conditions in which the applications are used. One of the development teams created Pizzeria Tracking. It's a set of six tablets displaying the pizza, the recipe with ingredients added or removed, and the location of the pizza in the kitchen at the moment.

Dough rolling tablet

Packing tablet

The team turned on the new tracking in one of the pizzerias and went to see how it worked. In the pizzeria, the developers saw that pizza makers were in a hurry to tap on the tablet very quickly, and they were doing it with their hands dirty with sauce or cheese. As a result, it was not always possible to tap on the tablet the first time. The team was very surprised. Surely, in the office, we clicked on the tablet with clean hands and there was no hurry. After going to Gemba, the team got back and redesigned the tap method in such a way, so that users could mark products even when they were doing it very quickly and with dirty hands.

5. Problem-solving

After going to a pizzeria, you offer the most suitable solutions. You know how the business works, how processes are set up in real life. When the business comes to you with a problem, having a broad view of the business and knowing how the system works inside, you can offer a simpler or more systematic solution to the problem.

Happy developers in Gemba

6. Product Impact

You can see how the inaccessibility or complex interface of the product affects employees and customers. You are standing at the checkout, and it is getting slow, there is a queue, and some customers are getting nervous or leaving. Without this valuable knowledge, the developer might not have noticed the increase in response latency.

QA engineer is taking orders at the checkout

Our CTO @pritchin is happy to deliver your pizza

Why not go to Gemba

I have never been able to imagine any disadvantages of this approach except for one: our developers who go to Gemba are not financially justified. We receive a lot of feedback that we are doing expensive bullshit, we are paying a developer for spinning pizza, distracting him from work and his immediate tasks. My opinion is simple — we pay developers for their work, for their product. They just perform work in a pizzeria, look at the product with the eyes of a client and answer themselves in real-time to the question “Have we done bullshit or not?”.

The End

At Dodo Brands, going to Gemba is a common thing. This is a cool experience, which helps us to check that we make a high-quality product.

I liked how one of our developers wrote: “For me, as a developer who has never gone to Gemba, a developer who went to Gemba looks like a bearer of secret knowledge. During a discussion at a general meeting, such a person may say: “The feature you propose will not work, because I was in a pizzeria and saw how it really was there. In general, you need to go to Gemba at least for this”.

Two most common questions about Gemba and IT people

Do you go to a pizzeria without a medical certificate? No way. You can’t go to Gemba without a medical certificate. We are normal people and strictly observe the standards of cleanliness. There can be no exceptions. Before you go to Gemba, you need to pass a medical examination and obtain the appropriate work permit.

Do developers need to clean the toilets? We go to pizzerias in order to understand how our product works in “combat” conditions. But when we come to a pizzeria, we become full-fledged trainees, so we can still be asked to clean up, and we will do it.