Part 1
Last year I spent a good amount of time teaching about Software Engineering and the 3 core paths of Engineering that help with consistent and reliable software development.
Engineering Cycle
Talking about software and engineering is talking about process, development cycle, troubleshooting which in the end has one goal: quality.
There is a caveat though, what is software quality or even better how can software quality be measured? Truth is that there is no way to quantify the quality of software and by quantifying I mean that there is no way to apply something such as Six Sigma to measure the quality of software.
Without getting into many details, especially because I’m not a 6σ Black Belt, Six Sigma is a continuous process to define one thing: In a manufacturing event, only 6 parts in a million can contain defects.
DMADV — https://en.wikipedia.org/wiki/Six_Sigma
Considering any manufacturer, such as a nail manufacturer, it is possible to measure that in a million nails produced that only 6 had defects. This means that manufacturing is optimized in a way that there are no real measured losses during the process.
Along with 6σ there other measuring certifications, compliances that are recognized worldwide for quality such as ISO 9001.
Unfortunately, there isn’t a way to the same with software, not as far as I know. Software quality has been measured in perceived quality for as long as software development became a profession and a science.
For a web application you can define that it is 70% unit test covered, that there is an acceptance of 70% of your audience, that QA successfully catches 99% of the logical bugs, the application follows the best coding practices and standards or our current time, but there is no way to quantify any of those numbers, there is no way to say that in a million lines of code only 6 had defects. Unfortunately, it doesn’t work that way.
If you present your software idea to any person that is willing to invest on it the logical question is to determine how good is the software so one can measure the risk of the investment, and, subsequently the return of the investment (a subject for another article).
In manufacturing a floor engineer can come to the CEO and affirm: We got ISO’s, 6 Sigma and we haven’t had an accident in the past 300 days. Without presenting many numbers it is the same as affirming that there aren’t any losses during the manufacturing process, therefore, the risk is small and profits can be high (not correlated).
When the same is asked for a Software the common reply is based on percentages and although they give some base on quantifying the quality of the software it doesn’t not give true assurance. What does it really mean that 70% of the code is covered by unit tests? Translating that to an investor or a business person is fairly impossible without some explanation of what are unit tests, to begin with.
Facing that problem the logical next step is to count successful deployments, count how many bugs, count many critical or high priority bugs, count how many hotfixes. There is always a need to quantify software quality but there is still not a single way to do it in a way that is accepted worldwide.
Measuring software quality becomes a problem.
During the many years working as a software engineer and all the ever-moving phases that it has gone through, the software has increased in quality but still not fully measure, which leads us to accepted or perceived risk.
911 text — Google Image Search
Consider a team that is working on the software that will make calls to 911 how can the team report that the software will work all the time?
Software can’t be quantified, presenting percentages is fairly hard to explain, magic as an explanation will just not work, the only thing left to present is perceived risk.
Perceived risk is the possibility that an unexpected behavior can happen and in software, it is presented mostly at pre-deployment meetings.
It is considered to be perceived risk because of the same reason why you can’t quantify software quality, therefore, to determine the quality of the software you determine if all the information about the software development and testing can generate an issue and how high is the probability of that happening.
Risk Assesment
For example, in a software that is 70% unit tested, following standards, is fully tested by the quality team, the question is what is the risk in deployment that software to production, as, what is the chance of an unexpected behavior (and note this is not necessarily a bug) can happen.
The usual response for that is either: minimal, considerable or high.
If we take into consideration the scenario that was presented about the 911 calls, since there is no way to quantify the quality of the software the best that can be done is to determine the risk of it not working, or better said, the perceived risk of it not working. In the end, it is important that all calls regardless of what happens reaches the dispatch from 911. This scenario is a real scenario from the telecom industry and, as would anyone say from that industry, it is a life that is on the line so that call better reaches the dispatch.
Till the day that we can get something like 6 Sigma for Software Quality, assessment of risk or perceived risk might be the only good enough measurement that software engineers can give in regards the quality of their software.
Part 2
Now that we know the problem, how to fix it so we can finally quantify quality on a software application.
To first quantify quality, we need to define what is quality and there is a whole bunch of perception that comes into play in this. For example, the Toyota Corolla and the Kia Optima are equivalent cars, therefore, they have similar quality but Toyota is always perceived as having a superior quality for reasons which are usually hard to understand when explained by customer versus the manufacturer.
As a driver, I love my Kia Optima and I know that I would have a hard time selling the Optima as a better car to a current or former Toyota owner. If both cars have similar quality why is one perceived as a better car than the other? The answer for this is quite simple: humans aren’t machines that process binary logic, a lot of emotion is placed into our decision making.
Quality is a matter of perception and it will be different for each of audiences Product Owner, Clients, and Investors and, although there are different messages to be given, they must, in the end, confirm that your software application has quality.
The perception of quality for a product owner is in regards to risk. If the application has a lower risk to be in production this means that it will have a lower cost of technical debt giving more time on the development of new features. Although I made the association of risk and technical debt, the truth is that risk management has a lot of other facts into it to be accounted for. Unit tests, technical debt, complexity, the likelihood of generating a bug, etc.
What is really important is the mathematical association of the higher the risk the higher the odds of technical debt and therefore the higher will be the cost.
Measuring quality from a client perspective has to be done on a daily basis. That is done by the count of tech support calls it generates and by the churn rate (the rate which customers stop subscribing to your application).
An application that has a low churn rate and a low number of tech support calls means that it has a high success rate, therefore, high quality. Tech support calls consume time and a large amount will consume more personal which means that costs that could be used for a given improvement are now being directed to the support of production issues (technical debt). A high churn rate means that the application has something that is generating a high drop in the average daily subscriptions for the application, which has a direct effect on the revenue that the application produces.
The fun part is measuring quality for an investor. If you are familiar with TV shows such as Shark Tank you will know that all of the investors are asking is the same thing: what is the risk and what are the current growth. That is asked so they can successfully measure the ROI (return over the investment).
When you are pitching your application to an investor so you can successfully grow it, you will be provided to him mainly the churn rate and the current profit and this is where it becomes hard.
When the application is starting is very hard to get a large investment or an investor to join (not saying that it is impossible), which makes the team responsible for it the 1st investors, but when attempting a secondary investment for growth, then you must present data and that means:
- How much gross revenue the application has done so far
- What is the overall client acquisition (subscriptions)
- What is the predicted client acquisition for the next 5 years
- What is the current churn rate (same churn rate that we measured with clients)
With all of this one can use the following as a way to quantify quality from the perspective outside of the engineering team:
Average Net profit vs Churn rate
Where the average net profit is the amount of average of gains already discounted from the losses and the churn rate is the percentage of losses that you have on subscriptions.
A high average net profit with a low churn rate is the equivalent of a successful software application and that is the equivalent of a quality product.
When it comes to the Engineering Team, all the points mentioned in the 1st part are the least necessary to ensure a sound quality product.
These articles were first published at:
Top comments (2)
"What does it really mean that 70% of the code is covered by unit tests?"
It means that 70% of the code guarantees basic correctness. Unit testing is good for languages that do not provide contract support. For those languages that do provide contract support (Eiffel, D, Spec#, Sing#, C++20) unit tests fill the small gap that are not covered by contracts.
It does not mean that any of that code will work properly together -- for that you need other kinds of testing, such as one-or-more of integration tests, acceptance tests, system tests, performance tests, security tests, fuzzing tests, stress tests, specifications expressed in a DSL that can be executed as part of a BDD test suite, et cetera.
Sure and that is exactly the point. Unit tests and some other forms of tests will only cover so much and if you have a perfect application based on all available standards and tests modalities out there, its quality is measured by how the end-user will perceive it.
A quality product could be defined by many perceptions. For a team lead it could be 70% of code coverage, for a manager it could be 5% risk of deployment, for a CIO it could be how much revenue it generates and for the user, it could mean anything even if the color of the link is wrong.
So if you consider risk as a baseline for quality, regardless of any other perception, that is a message that can be passed among any other portions of the organization in different forms and even then, risk can have different ways of being measured.
The idea of the post is to bring some light that the way that we measure software quality today can't be translated to the industrial world in the same way because you wouldn't be able to measure 6 Sigma for software, for example, but you could measure risk.