I have to say that this type of very complicated process strikes me as inherently problematic. In my experience, reducing defects is actually a fairly simple process consisting of two main steps:
Formal Analysis: If an algorithm is mathematical in nature, then it is amenable to a formal mathematical analysis. Cryptography is a nice clear case in point. If I were to come up with a new algorithm for encryption, I'd write a paper describing the algorithm and I'd provide proofs of its characteristics. Then I'd publish the paper and offer it up for peer review and public scrutiny. Once the paper passed that step, then presumably the algorithm would be eligible to be implemented in actual code. Of course someone clever might find a fundamental loophole in the algorithm (as opposed to any specific code) later on, but that's true of everything. For example, crypto algorithms that rely on factoring large numbers into primes are based on the assumption that P != NP, which most experts believe to be true, but no one has been able to prove one way or the other. If someone finds a counterexample, such algorithms will become instantly obsolete. One can think of other situations where this applies. For example, let's say we have a system where multiple threads communicate in real time. Formally determining that the system will perform as expected given various starting assumptions is probably a good idea. I don't think one wants to hack something like that together piecemeal. ;)
Testing: If an algorithm or design is based on somewhat arbitrary business rules, then I would say up-front analysis is probably not very effective. In this kind of case, I think its effectiveness tends to drop off exponentially with time. If you have spent several days as a team trying to think of issues with a particular design, spending an additional month is unlikely to be the most efficient use of time. Instead, in such cases I recommend very thorough testing. We can test individual modules in isolation; we can develop end-to-end tests; we can do property-based testing; we can run performance tests; and so on... I find that thorough and careful testing will uncover problems much more quickly than just staring at a document and trying to think really hard about what may be wrong with it. So for business logic, I recommend trying to get something up and running as quickly as possible and then using all the applicable testing techniques to find issues with it. Then feed any problems that are found back into the design process. This may mean doing some significant refactoring of the design to fix an issue, or it may simply mean adding edge cases ad-hoc to the existing logic. The team has to decide what is appropriate. Even if this takes time, it's still more efficient than spending that time on analyzing a design that exists on paper.
I really don't believe anything more can be done to reduce defects in a general sense. Sometimes one can simplify things. For example, one can use a domain-specific language that isn't Turing-complete to make some potential problems go away. Another option is to use an off-the-shelf framework and plug one’s application-specific logic into it.
However, if one is doing normal programming in a general-purpose language, I believe one must simply rely on the above two methods as appropriate (and hire the best people one can find, needless to say).
I'm a small business programmer. I love solving tough problems with Python and PHP. If you like what you're seeing, you should probably follow me here on dev.to and then checkout my blog.
Formal methods, yes. I 100% agree. If you can use formal methods to prove the properties of some algorithm, go for it.
However, testing is a super-expensive way to find defects. That's sort of the whole point of PSP: if you can reduce the number of defects you inject and find and remove the defects you do inject as soon as possible, you can drastically reduce the effort it takes to deliver high quality software. Finding a defect through testing, debugging it, fixing it, and then retesting is expensive.
The actual techniques Humphrey recommends aren't radical: collect requirements, review requirements, make a design, review your design, implement in code, personally review your code, peer review your code (or do an inspection), and then if you follow all those steps, you should have a lot fewer defects that need to be found and fixed during testing.
The part that's hard for people to swallow is all the tracking and analysis Humphrey wants you to do to prove that you're getting better over time. But you don't have to do that. You could just take it on faith that code reviews are a good idea, do them, and hope for the best.
We do that with all kinds of stuff in programming. For example, have you proven that pair programming is effective in your organization? If you have, you would be one of the few people who has gone to the trouble. The research is equivocal.
Let me quote straight from Humphrey (page 142 of the PSP book):
Data for several thousand programs show that even experienced developers inject 100 or more defects per KLOC. Typically, about half of these defects are found during compiling and the rest must be found in testing. Based on these data, a 50,000-LOC product would start with about 5,000 defects, and it would enter testing with about 50 or more defects per KLOC, or about 2,500 defects. Again, about half of these defects would be found in unit testing at a rate of about two or three defects per hour, and the rest must be found in system-level testing at a cost of 10 to 20 or more hours each. Total testing times would then be about 13,000 to 25,000 hours. With roughly 2,000 working hours in a year, this would require 12 to 18 or more months of testing by a team of five developers.
With the PSP, you will use design reviews and code reviews to find defects before testing. Then, on a TSP team, you will also use inspections. When they do proper reviews and inspections, TSP teams typically find close to 99% of the defects before even starting system-level testing. Test times are then cut by ten times or more. Instead of spending months in testing, TSP teams need only a few weeks (Davis and Mullaney 2003).
Testing just isn't a very efficient way to ensure the quality of your software. There's lots of research to back that up. Unless your software is safety-critical, it is very unlikely that you have anything near 100% statement coverage, never mind the modified condition/decision coverage (MC/DC) you'd need to be able to make any claims about quality.
So, I'm not sure where you're coming from on that, Nested Software. Or have I misinterpreted your remarks (as happens from time to time)?
@bosepchuk
, the part I disagree on, at least if there was no misunderstanding, is that basically every line of code written by everyone on the team should be subjected to analysis and code review. That's the impression I got about PSP, but maybe I got the wrong idea. I think that would be significantly less efficient than a more conventional approach combined with testing (unit testing and other methods), as well as adding some amount of analysis and code review where it matters most; and possible addition of things like pair programming into the mix as well.
As far as research goes, while I totally support research into software process, I think we must be skeptical of its actual applicability, given the current state of the art. To me, the fundamental problem is that if you do research in a very "clean" academic setting, there is a natural objection of "that's nice, but does it apply to the real world?" Then if you try to do research with real projects and organizations, there's the equally natural objection of "there are thousands of uncontrolled parameters here that could be polluting the results!" At the end of the day, I wouldn't be inclined to deeply trust any research into software process - even if I 100% agreed with its conclusions based on my own experience and intuition.
That said, I think there are probably many things we can agree on, even though our viewpoints may be somewhat divergent here:
When we write code as developers, we should be thinking hard about ways our code could go wrong, and we should try to generate automated tests for these possibilities as much as possible/reasonable. The idea here of course is to make sure the code works as expected today, but will also continue to work in regression.
As we write code, we should be thinking both about the immediate functionality we're developing, but we should also consider the bigger picture and make sure things we're doing fit into the broader context of the application logic being developed.
Practices like code and design review, as well as "real time" versions thereof like pair programming, can help to improve the quality of code, clean up weaknesses or bad habits, and foster communication among team members.
One thing I wonder about is whether the PSP really takes into account the expectations that people have of software today. More than ever software is a living, breathing thing. Many applications are online, and users want both bug fixes and features to be added more or less continuously. It's a far cry from the world of 1980s programming that PSP seems to have come out of in that sense...
I'm a small business programmer. I love solving tough problems with Python and PHP. If you like what you're seeing, you should probably follow me here on dev.to and then checkout my blog.
I think I need to clarify one point. Humphrey is actually encouraging us to follow a process, track its effectiveness, and do experiments to improve it over time. He makes it clear that the process he describes in the book, is what works for him and is just a convenient starting place to have a discussion. But it's up to each programmer to find and do what works for them.
If you wanted to follow Humphrey's process, then he would agree with you idea that "every line of code written by everyone on the team should be subjected to analysis and code review."
We do that on my team and we find it catches a crazy number of problems before we put code in production. We also work in a legacy environment where much of the existing code cannot be reasonably covered with automated tests. So your mileage may vary.
Yes, research is a huge problem in our field for all the reasons you mentioned and more. Humphrey would encourage you to run your own experiments and figure out what works for you.
I absolutely agree with your three points.
I don't see any reason PSP would have trouble coping with the expectations of today's software. Humphrey was very much in favor of small changes.
Just out of curiosity, what kind of programming do you do?
I find that testing gets more expensive the further away it is from the developer. If a separate qa group finds issues, that's going to take more time and money to solve. If an end-user finds issues, that's going to be worse again. However, I believe strongly in developers taking ownership of thoroughly testing their work before it ever goes on to these other stages. That includes both automated tests of whatever variety are relevant as well as just manually working with the software. As developers we should be going through workflows manually on our own and trying to break our own software. That's actually a very fast way to find many issues that are likely to occur (when I find a problem like that, I make every effort to automate a test for it first so that it stays fixed in regression).
I personally find that monkeying around with an application like this is very efficient because it decouples one from thinking in terms of the code, and just focuses attention on the actual behaviour of the software. So I think this kind of developer testing is an efficient way to improve quality. It doesn't mean a QA team isn't needed, but hopefully it means the bugs they find will really be of a different order - things that are hard to reproduce; complex interactions or integrations of the technologies involved; or running the software in a bunch of different environments. In my experience, it's really not hard to miss problems when just looking at code, or looking at a design on paper.
I've mainly worked in enterprise software as well as full stack web development, along with some dabbling in embedded systems.
I'm a small business programmer. I love solving tough problems with Python and PHP. If you like what you're seeing, you should probably follow me here on dev.to and then checkout my blog.
Lots of people have trouble "switching hats" from developer to tester to find problems with their own code. Going full black box is quite effective in that regard.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
I have to say that this type of very complicated process strikes me as inherently problematic. In my experience, reducing defects is actually a fairly simple process consisting of two main steps:
Formal Analysis: If an algorithm is mathematical in nature, then it is amenable to a formal mathematical analysis. Cryptography is a nice clear case in point. If I were to come up with a new algorithm for encryption, I'd write a paper describing the algorithm and I'd provide proofs of its characteristics. Then I'd publish the paper and offer it up for peer review and public scrutiny. Once the paper passed that step, then presumably the algorithm would be eligible to be implemented in actual code. Of course someone clever might find a fundamental loophole in the algorithm (as opposed to any specific code) later on, but that's true of everything. For example, crypto algorithms that rely on factoring large numbers into primes are based on the assumption that P != NP, which most experts believe to be true, but no one has been able to prove one way or the other. If someone finds a counterexample, such algorithms will become instantly obsolete. One can think of other situations where this applies. For example, let's say we have a system where multiple threads communicate in real time. Formally determining that the system will perform as expected given various starting assumptions is probably a good idea. I don't think one wants to hack something like that together piecemeal. ;)
Testing: If an algorithm or design is based on somewhat arbitrary business rules, then I would say up-front analysis is probably not very effective. In this kind of case, I think its effectiveness tends to drop off exponentially with time. If you have spent several days as a team trying to think of issues with a particular design, spending an additional month is unlikely to be the most efficient use of time. Instead, in such cases I recommend very thorough testing. We can test individual modules in isolation; we can develop end-to-end tests; we can do property-based testing; we can run performance tests; and so on... I find that thorough and careful testing will uncover problems much more quickly than just staring at a document and trying to think really hard about what may be wrong with it. So for business logic, I recommend trying to get something up and running as quickly as possible and then using all the applicable testing techniques to find issues with it. Then feed any problems that are found back into the design process. This may mean doing some significant refactoring of the design to fix an issue, or it may simply mean adding edge cases ad-hoc to the existing logic. The team has to decide what is appropriate. Even if this takes time, it's still more efficient than spending that time on analyzing a design that exists on paper.
I really don't believe anything more can be done to reduce defects in a general sense. Sometimes one can simplify things. For example, one can use a domain-specific language that isn't Turing-complete to make some potential problems go away. Another option is to use an off-the-shelf framework and plug one’s application-specific logic into it.
However, if one is doing normal programming in a general-purpose language, I believe one must simply rely on the above two methods as appropriate (and hire the best people one can find, needless to say).
Formal methods, yes. I 100% agree. If you can use formal methods to prove the properties of some algorithm, go for it.
However, testing is a super-expensive way to find defects. That's sort of the whole point of PSP: if you can reduce the number of defects you inject and find and remove the defects you do inject as soon as possible, you can drastically reduce the effort it takes to deliver high quality software. Finding a defect through testing, debugging it, fixing it, and then retesting is expensive.
The actual techniques Humphrey recommends aren't radical: collect requirements, review requirements, make a design, review your design, implement in code, personally review your code, peer review your code (or do an inspection), and then if you follow all those steps, you should have a lot fewer defects that need to be found and fixed during testing.
The part that's hard for people to swallow is all the tracking and analysis Humphrey wants you to do to prove that you're getting better over time. But you don't have to do that. You could just take it on faith that code reviews are a good idea, do them, and hope for the best.
We do that with all kinds of stuff in programming. For example, have you proven that pair programming is effective in your organization? If you have, you would be one of the few people who has gone to the trouble. The research is equivocal.
Let me quote straight from Humphrey (page 142 of the PSP book):
Testing just isn't a very efficient way to ensure the quality of your software. There's lots of research to back that up. Unless your software is safety-critical, it is very unlikely that you have anything near 100% statement coverage, never mind the modified condition/decision coverage (MC/DC) you'd need to be able to make any claims about quality.
So, I'm not sure where you're coming from on that, Nested Software. Or have I misinterpreted your remarks (as happens from time to time)?
@bosepchuk , the part I disagree on, at least if there was no misunderstanding, is that basically every line of code written by everyone on the team should be subjected to analysis and code review. That's the impression I got about PSP, but maybe I got the wrong idea. I think that would be significantly less efficient than a more conventional approach combined with testing (unit testing and other methods), as well as adding some amount of analysis and code review where it matters most; and possible addition of things like pair programming into the mix as well.
As far as research goes, while I totally support research into software process, I think we must be skeptical of its actual applicability, given the current state of the art. To me, the fundamental problem is that if you do research in a very "clean" academic setting, there is a natural objection of "that's nice, but does it apply to the real world?" Then if you try to do research with real projects and organizations, there's the equally natural objection of "there are thousands of uncontrolled parameters here that could be polluting the results!" At the end of the day, I wouldn't be inclined to deeply trust any research into software process - even if I 100% agreed with its conclusions based on my own experience and intuition.
That said, I think there are probably many things we can agree on, even though our viewpoints may be somewhat divergent here:
When we write code as developers, we should be thinking hard about ways our code could go wrong, and we should try to generate automated tests for these possibilities as much as possible/reasonable. The idea here of course is to make sure the code works as expected today, but will also continue to work in regression.
As we write code, we should be thinking both about the immediate functionality we're developing, but we should also consider the bigger picture and make sure things we're doing fit into the broader context of the application logic being developed.
Practices like code and design review, as well as "real time" versions thereof like pair programming, can help to improve the quality of code, clean up weaknesses or bad habits, and foster communication among team members.
One thing I wonder about is whether the PSP really takes into account the expectations that people have of software today. More than ever software is a living, breathing thing. Many applications are online, and users want both bug fixes and features to be added more or less continuously. It's a far cry from the world of 1980s programming that PSP seems to have come out of in that sense...
Good points, Nested Software.
I think I need to clarify one point. Humphrey is actually encouraging us to follow a process, track its effectiveness, and do experiments to improve it over time. He makes it clear that the process he describes in the book, is what works for him and is just a convenient starting place to have a discussion. But it's up to each programmer to find and do what works for them.
If you wanted to follow Humphrey's process, then he would agree with you idea that "every line of code written by everyone on the team should be subjected to analysis and code review."
We do that on my team and we find it catches a crazy number of problems before we put code in production. We also work in a legacy environment where much of the existing code cannot be reasonably covered with automated tests. So your mileage may vary.
Yes, research is a huge problem in our field for all the reasons you mentioned and more. Humphrey would encourage you to run your own experiments and figure out what works for you.
I absolutely agree with your three points.
I don't see any reason PSP would have trouble coping with the expectations of today's software. Humphrey was very much in favor of small changes.
Just out of curiosity, what kind of programming do you do?
Cheers.
I find that testing gets more expensive the further away it is from the developer. If a separate qa group finds issues, that's going to take more time and money to solve. If an end-user finds issues, that's going to be worse again. However, I believe strongly in developers taking ownership of thoroughly testing their work before it ever goes on to these other stages. That includes both automated tests of whatever variety are relevant as well as just manually working with the software. As developers we should be going through workflows manually on our own and trying to break our own software. That's actually a very fast way to find many issues that are likely to occur (when I find a problem like that, I make every effort to automate a test for it first so that it stays fixed in regression).
I personally find that monkeying around with an application like this is very efficient because it decouples one from thinking in terms of the code, and just focuses attention on the actual behaviour of the software. So I think this kind of developer testing is an efficient way to improve quality. It doesn't mean a QA team isn't needed, but hopefully it means the bugs they find will really be of a different order - things that are hard to reproduce; complex interactions or integrations of the technologies involved; or running the software in a bunch of different environments. In my experience, it's really not hard to miss problems when just looking at code, or looking at a design on paper.
I've mainly worked in enterprise software as well as full stack web development, along with some dabbling in embedded systems.
I 100% agree with your comments.
Lots of people have trouble "switching hats" from developer to tester to find problems with their own code. Going full black box is quite effective in that regard.