loading...

Can Developer Productivity be Measured?

nickhodges profile image Nick Hodges Updated on ・6 min read

Introduction

If you go to Google and search for "measuring software developer productivity" you will find a whole lot of nothing. Seriously -- nothing.

Well, okay, not exactly nothing. You'll get a ton of links. But almost all of the links you find will talk about how measuring the productivity of software developers can't be done effectively. Some people will even argue that it shouldn't be attempted at all. Some others will describe techniques to measure developer productivity, that, well, everyone else knows don't really work.

There have been many valiant attempts to measure developer productivity, but all seem to end in less than successful territory. (We all know to laugh at "Lines of Code" as a productivity measure). Virtually any objective measurement you can apply to the software development process can be "gamed" into submission.

There are just too many variables to account for everything that goes into the development process. Measuring any one of them, or even a combination of them, simply cannot begin to capture effectively everything that is involved. You can't pick a surrogate thing to measure, but because of human nature, developers will react in ways that will skew the measurement.

Many studies have been done on this topic, with little to show for it. Every software development manager who has to fill out a developer's evaluation or determine who gets bonuses has struggled with this. Many a leading expert and guru has thought about and researched this thorny topic.

Experts Agree

And not one of the experts has come up with a good idea or method to measure individual productivity. There are plenty of good and effective ideas on measuring team and project productivity, but there is a severe if not complete shortage of ways to measure individual performance and productivity.

For instance, Robert D. Austin wrote an entire book on this subject called Measuring and Managing Performance in Organizations and came to the conclusion that unless you can measure 100% of something -- that is, every aspect of it -- then there is no point in measuring at all. If you attempt to measure less than 100% of the things involved in a given activity, you will get what he refers to as "dysfunction", or what you and I might call gaming the system. And since software development is an incredibly complex and creative endeavor, it's virtually impossible to measure every aspect of it. Often it quite literally involves sitting and staring at a computer screen. How do you measure the productivity of that?

Here's Martin Fowler's conclusion in an article he entitles " CannotMeasureProductivity":

"This is somewhere I think we have to admit to our ignorance."

Neil McAllister of InfoWorld writes in an article entitled "The Futility of Developer Productivity Metrics"

"For most other companies, however, it might be best simply to forget about the idea of measuring developer productivity and rely instead on tried and true methods. That's right: A highly effective, productive developer workforce is often the result of high-quality, effective management. Unfortunately, nobody has developed a metric for that yet. Coincidence?"

Joel Spolsky writes:

"It's rather hard to measure programmer productivity; almost any metric you can come up with (lines of debugged code, function points, number of command-line arguments) is trivial to game, and it's very hard to get concrete data on large projects because it's very rare for two programmers to be told to do the same thing."

Steve McConnell, a widely regarded academic, author, and consultant says:

"So while I see the value of measuring individual performance in research settings, I think its difficult to find cases in which the effort is justified on real projects."

Like I said, it's hard to find a leading expert making the case for objective measurements, and easy to find plenty that say you shouldn't even try.

So I don't think that it's a tough leap to agree that there is no objective way to measure productivity in software developers. I know this is controversial, but it seems quite clear to me that software is more art than science, and measuring productivity in art is, well, simply not done. One does not tell Michelangelo that it took him too long to paint the Sistine Chapel because he should have been painting 1.4 square feet per day instead of 1.2. That would be preposterous.

So, What Are We to Do?

Anyone who is charged with managing and leading developers is left in a quandary. We want to be fair, but if everything is subjective, how can we be?

The fact that there is no objective measure of developer productivity doesn't mean you can't measure it. It just means that you have to measure it subjectively . And we do that, don't we. You can look around your shop and just know who the productive developers are. There are probably developers you know that simply crank out good code faster and fix bugs more efficiently than the rest. For ease of discussion, I'll give this developer the name "Jason".

And just how do you know Jason is more productive and more valuable? You just do. You can just tell. You know it when you see it. But if you stop and think about it, Jason has some characteristics that you can figure out and write down.

What I mean is that if you stop and examine Jason and his work, there are things that you can say about him that describe what he is all about. They are subjective, sure, but you can write them down and give an "objective" measure to these subjective things.

A Modest Proposal

So yeah, what I am proposing is a way to measure developer productivity and value via a subjective set of measurements. Think of it as a way to qualify the notion of "you just know that Jason is productive".

Here is the list:

Productivity

  • Does the developer get a reasonable amount of work done in a given period of time?
  • Is the developer's velocity on bug fixing sufficient?

Engagement

  • Is the developer dedicated to his/her craft?
  • Is the developer committed to delivering software on time?
  • Is the developer dedicated to company success?

Attention to Quality

  • To what degree does the developer's code work as designed?
  • Does the developer thoroughly test code and believe it to be correct before checking it in?
  • Do a minimal number of bugs get reported against his/her code?
  • Does the developer write unit tests for all new code?
  • Does the developer follow the Boy Scout Rule and leave a module cleaner than it was before he or she worked on it?

Code Base Knowledge and Management

  • To what degree does the developer understand the code base assigned to him/her?
  • Does the developer take responsibility for his/her team's code base, improving it at every opportunity?
  • Adherence to coding guidelines and techniques
  • Does developer's code routinely meet coding standards?
  • Do code reviews reveal a minimum of problems and discrepancies?
  • Does the developer use Dependency Injection to ensure decoupled code?

Learning and Skills

  • Is the developer constantly learning and improving his/her skills?
  • Does the developer show a passion for the craft of software development?

Personal Responsibility

  • Does the developer first assume that the error lies within his or her code?
  • Does the developer understand that he or she is solely responsible for their code working correctly?
  • Does the developer take pride in their code, ensuring it is clean, tested, easy to read, and easy to maintain?

Again, these seven items are all subjective measurements. All the questions require the evaluator to make subjective judgments. But a good lead/manager is presumably in their position because of their good judgment about such things, and so they should be able to make fair and accurate assessments about Jason's productivity using these "measurements".

Most of the questions ask for a yes or no answer, but all can spark a discussion with management or even the developer being evaluated. You could use these measures on a formal evaluation form. Whatever you do with them, it's probably better than the "just know it when I see it" approach you are probably taking now.

Conclusion

In the end, any evaluation, judgment, or other assessment of the productivity or value of a single developer will be subjective. No one has yet found an objective way to measure individual productivity in our excessively complex profession. But I am saying that it is possible to qualify and make judgments about a specific set of criteria as a small and humble step towards being able to do the thing that our industry has so far found very difficult.

My suggestion is that we simply accept reality and embrace the subjectivity -- and lack of objectivity -- in measuring developer performance.

Discussion

pic
Editor guide
Collapse
jillesvangurp profile image
Jilles van Gurp

I don't think it is this black and white. Loc are too easily dismissed in this industry together with a lot of other metrics in favor of something even worse: opinions and gut feelings. Sure, Loc are not a great metric but they are better than measuring nothing. And it's not the only metric. You can look at number of commits, size of those commits, etc. Frequency of commits per time unit, tickets closed, test coverage, etc. Together they tell a story; especially when you track them over time. It's usually not the whole story and in the end it is just numbers. But they tend to even out over time and big changes tell you something just happened.

One glance at the github statistics of any project will usually tell you who are the movers and shakers and who are definitely not having a lot of impact. When metrics vary by magnitudes between developers, it's pretty hard to ignore. In my experience the numbers can vary dramatically between developers doing similar work in one project. 10x is pretty rare, but 3-5x within one project is pretty common and it is usually not temporary.

With those kind of differences, effectively 70-80% of all the code is being written by a handful of people. Amount of code varies of course with technologies used, quality of that work, etc. But it varies particularly with uncertainty and confidence. Somebody that is struggling will not get a lot done and the quality of what they get done might not be great either. If somebody is struggling a lot, their impact is going to be pretty low; or even negative. It's going to show in the numbers.

That doesn't necessarily mean you have a slacker but you do have an issue with somebody needing a little help in the form of coaching, pairing or training, or a gentle nudge to a different type of role.

Collapse
docsbydesign profile image
Bob Watson

I agree with the approach, but I would modify the terminology. Measurement evokes a sense of objectivity, which the introduction describes as difficult to impossible to apply. I'd call your rubric more of an evaluation. It's subjective but also transparent. If done well, it describes what you value as a manger and an organization and can be used to have constructive discussions with subordinates. At the same time it requires a degree of involvement with them that LOC and commit-counters might prefer to avoid due to a lack of time, interest, and/or people skills.

LOC are objective, easy, and make for pretty progress charts but they are too easily gamed and encourage the wrong goals

My personal experience with LOC came after a large refactor that eliminated bugs and improved reliability, but cut LOC by 30%. In the minds of LOC counters, I had just reduced the value of the product and was "rewarded" accordingly. So, full disclosure, I'm not a fan and In my experience they are worse than nothing when they encourage bad code over good. I've also worked with "movers and shakers" who just churned the code, not really advanced it.

You can't really apply process-based metrics to something until you have a repeatable process with consistent units of input and output.

Collapse
optimite profile image
Optimite

There are truly objective metrics out there now, which take into account the contextual difference every developer works in, that analyse revisions from multiple dimensions such as it's Volume, Complexity and Interrelatedness altogether.

As Jilles said, it's true that gut feelings and opinions are part of the process with other metrics but now there are measures that eliminate that altogether. It's possible to directly compare productivity (one metric) and maintainability (another metric from a quality perspective) of code provided by a developer, which starts to make comparison of developer effort on every project fair and allows everyone to move forward together to improve the rate at which code is delivered and simultaneously advance it.

They're useful as they give you an indication of where to look, especially if you're going to be working on different projects with many teams wouldn't you say?

Anyway the metrics I was talking about earlier and I've come across so far is provided by BlueOptima (blueoptima.com).

Collapse
joeyhub profile image
Joey Hernández

Measuring up productivity and quality is extremely important but also extremely hard. Quality is what is also missing here because that code produced by those who are also more productive was of better quality. This means it ran quicker, was written in less code, had less defects, etc. Many people believe that code should be easy to read by those who cannot read code well at all or code that is converted into or padded with plain English as much as possible. This is definitely not how you get that 10X performance. You cannot get good quality effective code by reducing everyone's coding standard to that of the poorest or even the most inadequate performer. I think often what happens is a crap coder happens upon some good code they can't hack and then its all the fault of the code for being too advanced.

There are quite a few points here that definitely are subjective or circumstantial. Many of the subjective ones don't leave too much room for error although as I have pointed out the one about readability has the potential to go entirely wrong where you end up undermining your top performers. Quite a few points you presented are popular opinion but the problem is that the 90% are not the 10% that does 90% of the work.

If a developer write unit tests isn't useful at all for measuring productivity. It is 100% meaningless and is typical of the kind of superficial approach you'll see when deeper understanding is lacking. Productivity is productivity and a productive developer applies solutions appropriately as and when they contribute to that. Doing unit tests for the sake of it and simply assuming based on theory that extra work will give ROI is exactly how you would go about being unproductive. It isn't a case of what is or isn't used but if what is used is used to the greatest effect.

A good developer will certainly test their code but there are umpteen different approaches to achieving that. Unit tests stick out because they are taught in schools as they are very simple but they can also be the most inefficient way to test code. In some cases even manual testing can beat unit tests although few people would realise that because they haven't the empirical data.

I see this a lot. People looking for signs such as function length. Strange beliefs such as doing things that involve more work will always result in less work. Very often people mistake dumbed down code for productive code.

It is possible to measure productivity but it takes an expert to look at that code and the outcome. That you're going by numbers and things that tells me you're not quite there yet. You need to be very careful because if you propose an incorrect measure of productivity but people stick to that then it will drain productivity.

Real judgement wouldn't just sign off on a bunch of check lists. A productive coder starts out with code that for the most part only does what it needs to do and only has as much code as there need be. This is basic minimalism.

You shouldn't be scrutinising what it doesn't have. You should first be scrutinising what it has. Only when there is actually some deficiency that presents or real tangible opportunity should you consider if something should be added.

A productive developer wont add anything extra unless it makes sense to do so. They wont preempt either if they aren't sure. They'll let a process become tedious and expose itself as needing optimisation rather than prematurely optimising. This is based on a very simple but fundamental logic. If you try to fix something ahead of time but aren't sure, then you'll never know if you had an impact or the problem never occurred. If you let problems manifest you can see them for real, need not rely on speculation and have a point of reference to compare your improvement to. That is not to say you never adapt for problems down the line but you only do so when you know what you are doing. The most fundamental concept behind this is that if you do too much you will never find out. If you do too little you will always find out.

It isn't an easy problem to solve. The only way is to have a productive developer do a fair amount of the same work as a less productive developer then compare the two. You'll actually find nearly everything the productive developer does is objectively and often quantifiably better but you'll also find hundreds of aspects to that for a reasonably sized codebase. It isn't something you can always easily reduce into a few simple rules.

I have often taken a few thousand lines of code and turned it into a few hundred that performs faster, uses less resources all around, is defect free, more accurate, maintainable, adaptable/flexible, readable, error tolerant, etc and that is not accomplished by adding anything special. That is done by simply reducing the code down to what it needs to be and writing good code. To know what that is you have to look at individual lines, the whole and the outcome.

There's no magic sign or label that can tell you what good code is. You just have to understand the code, what it does and how it does it.

Collapse
alesdonoso profile image
Ale Donoso

Without a doubt, this topic raises many doubts and questions, mainly because variables of a quantitative and qualitative nature are mixed. At Scope we started asking ourselves how complicated would it be to measure how much time it takes for a developer, depending on their skill level, to learn the documentation of a specific framework? It would be pernicious on our part to try to generate a single answer to this question.

We at Scope -where we provide metrics and visualizations so that a team leader can know the complete workflow of their teams-, we know all this casuistry and, therefore, we have developed an algorithm to generate qualitative value for the work done by a software developer. Through this algorithm, which you can read in this article, we weigh variables that directly affect the quality of the code of a repository, through the Pull-Requests.

Our speed metrics give us appropriate indicators on the entire flow and linked to the quality of the tasks measured through the algorithm, conclusions can be drawn on how well a developer is working and what exactly the workflow is, where they are bottlenecks, etc.

Furthermore, it is not only a matter of measuring individual productivity, as they have mentioned in some answers, but also measuring the progress of projects over time. That is why at Scope we provide metrics both at the developer level and at the repository level.

We try to create an approach based on data and metrics to measure in a way as close to reality as people work in software development.