How I Measured The Software Testing Quality

Kashif Gul Kazi — Mon, 11 Dec 2017 12:33:47 +0000

How do I know if my team is producing the quality outcome and if they are improving as they marching forward?

It’s doable with a smart and sweet KPI (Key Performance Indicator). KPI like below make it happen. It explains to me how to quickly and timely understand the product quality and identify areas for further improvement. KPIs are nothing but extracted from the Metrics and they have a narrow focus.

Let me explain what are metrics and KPI and how they help and relate to each other.

Software quality metrics are the quantifiable measure used to track, monitor and assess the success or failure of various software quality processes.

In a world of Software quality metric, there are multiple statistics that can be measured to understand the overall health of development processes as well as the product in development.

For example, think of a vehicle quality metrics, it has suspension, gearbox, ignition, plugs, wiring etc. A whole lot of features and processes. A typical car garage would measure all metrics of a vehicle to make sure they are within their normal ranges for a fitness certificate. Let's call this whole calibration as car quality metrics.

On another hand, KPI is a most important piece to be looked at, for example, a car mileage KPI (a single indicator) is of interest when it’s come to a driver (the end user or a product manager in our case responsible for product success). It can tell a driver if everything is well or not - perhaps need a deeper level investigation by an expert. Let's call it a car KPI.

KPI is something we keep an eye all the time even in software development and the testing world. For example, I can collect multiple of metrics but be monitoring all of them daily with the same level of accuracy isn't easy. The one must add some context to collected metrics to make them meaningful. Am I looking at the right metric?

The Bugs discovery scorecard comes handy, shown below. It's measurable and quantifiable. It's telling if there is a problem and where is the problem.

To construct this efficient and easily understandable KPI for any product, we need to understand the 2 key ingredients

Origin: It defines when a bug was injected into the system.

Detected: It defines at what stage a bug was caught or identified.

Usually, the way I obtained this scoring is by adding 2 custom fields to my favorite bug tracking system to record the bug origin and the bug detection phase. Not to mention, it’s a good for post-release sessions and discussion like release retrospective.

Some examples to get the meaning from this data formation

There are multiple ways to get benefit from this scorecard

For example, it’s perfectly fine to have a bug originated in the “Requirements” phase but it’s surely not right to discover it at later stages of Production. Finding a requirement or a design level issue late in the game may warrant a lot of rework or lead towards the product or a feature failure.

The Bugs discovery scorecard matrix isn't just for development or QA team. It helps all teams at all levels starting from analysts, designers, developer, QA testers, and management to measure quality as well as identify the gap and improve continuously. Like here in above image, 1 coding bug caught in PROD (by customers) and 1 REQ bug was caught in SQA (but not in the coding phase).

From my experience of managing teams, I look for ways to praise my team too. For example, if the design bug has caught in the development phase but not in the design phase, it needs significant rework but rework would still be less when a bug caught in the SQA or the PROD phase

I can easily grab the maturity level of my teams working on requirements, design and development phases. And, I can measure quality standard for the testing team.

DEV Community: Kashif Gul Kazi

How I Measured The Software Testing Quality