Part of the issue with research is that we don't know what we should be measuring? Metrics for quality and productivity are ellusive in the software world. Test setup is also an issue as all software projects tend to be unique in some way. There are a few domains with lots of inner similarities, like full stack business pages, but this touches on only a fraction of programming.
I think this is closely related to your point about demonstrating skills, but I'm focusing more on the achievement of the team instead.
I'm a small business programmer. I love solving tough problems with Python and PHP. If you like what you're seeing, you should probably follow me here on dev.to and then checkout my blog.
Is the challenge of finding acceptable quality and productivity metrics really that big of an issue for CS researchers?
When I think of all the crazy-hard questions scientists study (in all kinds of fields) and all the years they spend on one little area, I don't think CS research is particularly challenging or unique.
I just think that too few people and businesses really care about CS research.
I read about a hospital adopting a decision tree to triage patients with chest pain (seilevel.com/requirements/visual-m...). It saves lives, saves money, and increases quality of care. But the point I want to make is that someone took the time to study the subject and come up with a decision tree based on the evidence. It's not perfect but it's better than sending everyone with chest pain to the cardiac unit.
Why can't we have this kind of thing for TDD? It doesn't have to be 100% accurate, it just has to be better than what we have now, which is basically no scientifically validated guidance.
Imagine if we had a decision tree for TDD with 3-5 questions and it would spit out the probability that your project would benefit from TDD.
Medicine is surprisingly predictable and routine compared to software development. You can classify health issues, and outcomes, in a relative straight forward manner. The hospitals across a region, the country, or continent, all generally deal with the same types of issues (naturally regional variance), as there are always humans in the equation with a similar culture.
In software we just don't know what indicates something is good or bad. Is it user satisfaction? Money made by the company? Data breaches, or security incidents? What exactly is the quality that we are measuring?
We don't have reliable ways of saying the complexity of a feature, or the significance of defect. There are too many variables involved.
I would compare this to the field of art. Do they have good metrics on what constitutes a good artist?
I'm a small business programmer. I love solving tough problems with Python and PHP. If you like what you're seeing, you should probably follow me here on dev.to and then checkout my blog.
In software we just don't know what indicates something is good or bad
I disagree. For the most part we desire software that meets all its functional and non-functional requirements. When you turn on the auto-pilot in a plane, you expect it not to fly the plane into the ground. When you activate WPA-2 wireless encryption, you expect your wireless connection to your router to be encrypted.
That's very different that art.
All things being equal:
cheaper software is better than more expensive
faster delivery is better than slower delivery
fewer defects in production is better than more
higher mean time between failure is better than lower
cheaper to maintain is better than more expensive
cheaper to extend is better than more expensive
fewer known security defects is better than more
available is better than not available
reliable is better than unreliable
We don't have reliable ways of saying the complexity of a feature, or the significance of defect. There are too many variables involved.
We have money. The only reason business people care about software quality is because low quality means loss of money and possibly risk and liability. It's non-trivial to convert defects, technical debt, and new features into money but it can and should be done. The business people supervising your project are doing that all the time.
So, given two identical applications produced by ACME Software Inc., the better one is the one that has the greatest positive effect on the bottom line of the business over the life of the system.
Software developers go on and on about quality because they think low quality slows them down and makes their projects more costly than they need to be. But, when you zoom out, money is the only thing that matters for most commercial software projects.
I disagree. For the most part we desire software that meets all its functional and non-functional requirements. When you turn on the auto-pilot in a plane, you expect it not to fly the plane into the ground. When you activate WPA-2 wireless encryption, you expect your wireless connection to your router to be encrypted.
There is no standard by which functional and non-functional requirements are defined for software. When we say user data should be "secure" we don't have a clear definition of what that means. We also don't have any true way of testing that, other than just waiting for incident reports.
Waiting for issues is also not a good measure since popularity plays a major role here: some products have fewer users to discover issues, some services have less interesting data, thus attracting less attackers.
I agree on the money argument, it's what I said in my article about doomed software quality. Major breaches, failures, and otherwise seem to have no impact on the money flow at the moment. With money as our only metric there is simply no pressure to have what most people would qualify as good quality software.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Part of the issue with research is that we don't know what we should be measuring? Metrics for quality and productivity are ellusive in the software world. Test setup is also an issue as all software projects tend to be unique in some way. There are a few domains with lots of inner similarities, like full stack business pages, but this touches on only a fraction of programming.
I think this is closely related to your point about demonstrating skills, but I'm focusing more on the achievement of the team instead.
Is the challenge of finding acceptable quality and productivity metrics really that big of an issue for CS researchers?
When I think of all the crazy-hard questions scientists study (in all kinds of fields) and all the years they spend on one little area, I don't think CS research is particularly challenging or unique.
I just think that too few people and businesses really care about CS research.
I read about a hospital adopting a decision tree to triage patients with chest pain (seilevel.com/requirements/visual-m...). It saves lives, saves money, and increases quality of care. But the point I want to make is that someone took the time to study the subject and come up with a decision tree based on the evidence. It's not perfect but it's better than sending everyone with chest pain to the cardiac unit.
Why can't we have this kind of thing for TDD? It doesn't have to be 100% accurate, it just has to be better than what we have now, which is basically no scientifically validated guidance.
Imagine if we had a decision tree for TDD with 3-5 questions and it would spit out the probability that your project would benefit from TDD.
Our profession really is in the dark ages.
Medicine is surprisingly predictable and routine compared to software development. You can classify health issues, and outcomes, in a relative straight forward manner. The hospitals across a region, the country, or continent, all generally deal with the same types of issues (naturally regional variance), as there are always humans in the equation with a similar culture.
In software we just don't know what indicates something is good or bad. Is it user satisfaction? Money made by the company? Data breaches, or security incidents? What exactly is the quality that we are measuring?
We don't have reliable ways of saying the complexity of a feature, or the significance of defect. There are too many variables involved.
I would compare this to the field of art. Do they have good metrics on what constitutes a good artist?
I'm enjoying our little debate.
You wrote:
I disagree. For the most part we desire software that meets all its functional and non-functional requirements. When you turn on the auto-pilot in a plane, you expect it not to fly the plane into the ground. When you activate WPA-2 wireless encryption, you expect your wireless connection to your router to be encrypted.
That's very different that art.
All things being equal:
We have money. The only reason business people care about software quality is because low quality means loss of money and possibly risk and liability. It's non-trivial to convert defects, technical debt, and new features into money but it can and should be done. The business people supervising your project are doing that all the time.
So, given two identical applications produced by ACME Software Inc., the better one is the one that has the greatest positive effect on the bottom line of the business over the life of the system.
Software developers go on and on about quality because they think low quality slows them down and makes their projects more costly than they need to be. But, when you zoom out, money is the only thing that matters for most commercial software projects.
There is no standard by which functional and non-functional requirements are defined for software. When we say user data should be "secure" we don't have a clear definition of what that means. We also don't have any true way of testing that, other than just waiting for incident reports.
Waiting for issues is also not a good measure since popularity plays a major role here: some products have fewer users to discover issues, some services have less interesting data, thus attracting less attackers.
I agree on the money argument, it's what I said in my article about doomed software quality. Major breaches, failures, and otherwise seem to have no impact on the money flow at the moment. With money as our only metric there is simply no pressure to have what most people would qualify as good quality software.