This article originally appeared on IG's blog
I was quite surprised at a recent blog post by Uncle Bob Martin, titled: "Type Wars", in which he writes: "Therefore, I predict, that as Test Driven Development becomes ever more accepted as a necessary professional discipline, dynamic languages will become the preferred languages. The Smalltalkers will, eventually, win."
This statement didn't sit well with some people in the static typing community, who argued that in a sufficiently advanced statically typed language, types are proofs and they make unit tests mostly redundant. Haskell even claims that "once your code compiles it usually works"!
Be it safer refactoring, better documentation, more accurate IDE support or easier to understand, for me all these claims translate to a simple promise: less bugs.
And I really hate bugs. I find them to be one of the worst wastes of time and energy for a project, and there is nothing that annoys me more than getting to the end of iteration demo and the team being somehow proud of saying, "We did X story points and we fixed 20 bugs! Hurray!"
To me it sounds like, "In the last iteration we wrote more than 20 bugs, but our clients were able to find just 20! And we were paid for both writing and fixing them! Hurray!"
Charting bugs
With that in mind, I tried to find some empirical evidence that static types do actually help avoid bugs. Unfortunately the best source that I found suggests that I am out of luck, so I had to settle for a more naïve approach: searching Github.
The following are some charts that compare the "bug density" for different languages. By bug density I mean the average number of issues labelled "bug" per repository in GitHub. I also tried removing some noise by just using repositories with some stars, on the assumption that repositories with no stars means that nobody is using them, so nobody will report bugs against them.
In green, in the "advanced" static typed languages corner: Haskell, Scala and F#.
In orange, in the "old and boring" static typed languages corner: Java, C++ and Go.
In red, in the dynamic typed language corner: JavaScript, Ruby, Python, Clojure and Erlang.
Round 1. Languages sorted by bug density. All repos
Round 2. Languages sorted by bug density. More than 10 stars repos
Round 3. Languages sorted by bug density. More than 100 stars repos
Whilst not conclusive, the lack of evidence in the charts that more advanced type languages are going to save us from writing bugs is very disturbing.
Static vs Dynamic is not the issue
The charts show no evidence of static/dynamic typing making any difference, but they do show, at least in my humble opinion, a gap between languages that focus on simplicity versus ones that don't.
Both Rob Pike (Go creator) and Rich Hickey (Clojure creator) have very good talks about simplicity being a core part of their languages.
And that simplicity means that your application is going to be easier to understand, easier to change, easier to maintain, and more flexible. All of which means that you are going to write less bugs.
What characterizes a simple language? Listing the things in common between Go, Erlang and Clojure, we get:
- No manual memory management
- No mutex-based concurrency
- No classes
- No inheritance
- No complex type system
- No multiparadigm
- Not a lot of syntax
- Not academic
Maybe all those shiny things that we get in our languages are actually the sharp tools that we end up hurting ourselves with - creating bugs and wasting our time - and that all they do is bring a lot of additional complexity, when what we really need is a simpler language.
As Tony Hoare said:
There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.
Latest comments (80)
How is Python vastly more complicated than Ruby?
I've been noticing similar things. Statically typed compiled languages do eliminate the possibility of certain classes of errors getting through to production simply because the compiler will catch the errors. However, a lot of the time, these compiled languages introduce complexities that raise the likelihood of other classes of errors that may actually be more advanced to troubleshoot (looking at you, Haskell, with your lazy evaluation and unpredictable performance). Another language that makes me question the perceived value of static typing is Scala. Until I had worked with Scala almost exclusively for the better part of a year, I actually spent more time fighting with the compiler over code that was logically correct (but due to type-erasure and other things, the compiler couldn't verify) than actually working on real bugs in the application.
I wonder, if all of this is true, what it's impact is going to be in the success or demise of Rust?
I think that for a programming language it is far more important the marketing around it than the technical qualities.
¿Have you recalculated the data by dividing the total number of bugs by the age of the projects?
This is necessary to remove the age variable as obviously the older the project, the more bugs it will have.
But surely you have found a very valuable source of information to provide more insight in the static vs dynamic typing debate...
Thanks for this post Dan. It generated a lot of good discussion on this topic!
Is the bottom line "simplicity appears to be important, and static-vs-dynamic typed languages less important"?
That is what I would like to have a conversation around! What is your feeling about it?
We have been fighting over the static-vs-dynamic thing for too long.
Thanks for reading!
Dan
My feelings -- just my feelings, not backed by any hard data -- is that the most important thing is both simplicity and writing the source code for maintainability and legibility. What Uncle Bob wrote about in his book Clean Code.
Some languages lend themselves to simplicity. For example, I'm impressed with D, Python, Lua and F# ... all of which have a clean syntax and are rather free of excessive "ceremony". Which is why I have a soft spot in my heart for those languages.
But the languages I use that pay the bills are C++ and C#, and I have a love-hate relationship with both of those languages. (More vehemence for C++, because I've been using it for a very long time.)
Bugs can be written in any language. But languages like C++ that have so many areas of undefined behavior that are easy to accidentally stumble into do no one any favors.
Languages that have contract programming, like Eiffel, D, and Ada 2012, make unit testing a lot less important because the contracts can be specified directly in the code instead of being encoded in unit tests. (That's what unit tests do: they express contracts.)
In my experience, statically typed languages -- like Go, C++, D, F#, Swift, TypeScript -- don't have much better protection from the duck typed languages like Python, JavaScript, Boo for "not making bugs". What the static typing does provide is scaling. Small applications gain little benefit from static typing. But as applications grow helps to make sure the pieces are fitting together correctly.
Case in point is Google's Angular that was converted from JavaScript to TypeScript, they had discovered that there had been a good number of bugs in their code that were caught once they had the static typing of TypeScript. (TypeScript transpiles to JavaScript, and the type annotation information is erased. It's a transpile time safety net.)
But, I've also worked with large system based in Objective-C which has a mix of static type checking and runtime duck typing, due to the nature of it using message passing to objects. (The message passing is reminiscent of SmallTalk.)
When I think of duck typed languages, I usually think of scripting languages. When I want to do something quick-and-dirty I reach for Python. When I want to make something application-like, I reach for a static typed compiled language.
But there are languages out there that bridge the two worlds of sorts. Languages that minimize the ceremony around the static typing, like OCaml, F#, and Swift. They're still all strongly typed, but the burden is more on the shoulders of the compiler, rather than forcing the developer to dot all the i's, and cross all the t's.
So I'd say that static typing catches a small category of bugs. For smaller applications, those kinds of bugs are few. For larger applications, those kinds of bugs can be crippling.
I don't know of any scripting language that supports contract programming as part of the core language. (Educate me if you know of any!)
A vastly bigger source of bugs in programs I work in is mutable global state. By which I am also including local mutable member variables in a class instance... that's a smaller scope global state. Programs that I've seen and I've written that emphasize immutability and segregate immutable data from functions and side-effect free functions seem to produce a lot less bugs.
I'm not sure if the "less bugs" I'm seeing is because I'm a better programmer with those kinds of languages, or if I make less bugs in those languages because it is easier to reason about the correctness of the code. Doesn't have to do with all those languages being statically typed. I believe it does have to do with immutable data and lack of global state has more simplicity.
Another vast source of bugs I've run into is null pointers. (Damn you Tony Hoare for adding in the null reference to ALGOL W!). That's another area where Haskell, F#, OCaml, Swift outshine C, C++, C#. Objective-C sort of sidestepped the problem with its treatment of the nil object quietly eating messages (well, almost quietly... the eaten message is output to the console log).
This analysis is flawed. The input variables are not controlled, nor do the conclusions logically follow from the data. I could make an almost opposite conclusion with the same data:
Based on bug density we clearly see that static typed languages are the best for identifying bugs.
The fact that the "data" can be used to draw very opposing conclusions would indicate a fatal flaw in the analysis. This is a sensational piece with no merit as research.
Thanks for the comment!
I would completely agree with your conclusion if the bugs from statically typed languages were all compilation errors. I suspect they are not.
Also, I could agree if the bug density of statically typed languages weren't all over the place. Note that Go is statically typed and one of the languages that I would call simple.
The post is not research. It doesn't say so anywhere, the post says "naïve", "not conclusive" and "opinion".
For research, read the link near the word naïve.
Also I would suggest to read the comments and watch the videos from Blaine. They are very cool and probably closer to your taste.
Thanks again!
Dan
Have you considered that you may have actually measured known bug-count vs unknown? To me this is like like comparing pennies in a jar vs missed pay-checks... It's the unknown long-term problems with systems (rounding errors, off-by-one's, partial API regressions, and design flaws) that lead to the biggest problems.
It's for sure interesting, I'd love for there to be an answer, but I've been making the transition from dynamic -> static yo-yo'ing without any evidence for or against either for the general-case since the 90's.
Thanks for the article
Hi Lewis,
Not sure if I understood you about the known vs unknown. Do you mean that for dynamically typed languages, there are bugs that have not been reported or found, while those same bugs would have been reported in a staticly typed lang?
I love the pennies vs paychecks analogy. I will steal it for a future blog post ;).
I am with you in the static vs dynamic debate, that is why I wanted to propose a different one: simple vs complex. On this one, I would position myself on the "simple-by-default" camp, were doing complex things was painful and non-idiomatic. What about you?
Thanks!
Dan
Hi Dan,
You got the known vs unknown in one. Not knowing about a bug (it not being in issues) doesn't mean it doesn't exist as we found with the OpenSSL bugs a few years back.
I'm glad you enjoy my analogies I love using them as they generally help ;)
On simple vs Complex. I'm sure it's a false dichotomy overall but I definitely love the idea I keep being sold RE: simplicity.
I would expect something similar.
At the summary of studies that I link in the blog, there is reference to this talk were "The speaker used data from Github to determine that approximately 2.7% of Python bugs are type errors".
I was quite surprised.
Yes!
As I am somehow fascinated with Haskell, I would love to go through all the fixed bugs in some Haskell repos to see if there are some common patterns.
What would you expect to find?
I've used static and dynamic languages and I agree with the hypothesis that static languages, when used well, help you reduce the probability of bugs. In many cases, people write poor code. If you use a static language, like F# or Haskell, but use it like it is JavaScript or the old C, C++ it is normal that bugs will arise. Most programmers are "Primitive obsessed" which is a source for some bugs. Many like to cast all over the code too. Programming in a way which makes invalid states impossible to represent helps a lot and also saves you a lot of testing. Usually when I get a code to compile I have very few bugs and most of them are caused by a bad communication of the requirements.
fsharpforfunandprofit.com/series/d...
Hi Dan!
Nice analysis.
When you calculate bug density, is it # of bugs divided by # of lines of code?
If so, this is really surprising! I read somewhere that bug density was pretty much a constant. So more concise languages had an advantage by being shorter.
However, your analysis shows that Java and C++ have more bugs per line of code!! So more code * more bugs = more more bugs! Ouch!
It would be cool to see the distributions of these languages. How wide are the curves? Does bug density vary widely in Java projects? What about Haskell?
Rock on!
Eric
Hi Eric,
I have updated the post to make it clear: "By bug density I mean the average number of issues labelled "bug" per repository in GitHub"
The assumption is "I do expect is that roughly all developers, no matter the language, have to solve the same problems, so the open source libraries available have roughly the same functionality.". David seems to disagree on this assumption. What are you thoughts?
I also remember reading somewhere that bugs are constant per lines of code, but maybe what was constant was the number of lines produced or the number of lines that you can keep in your head. I unable to find the reference right now.
Steven McDonnell in the "Code Complete" book says: "the number of errors increases dramatically as project size increases, with very large projects having up to four times as many errors per line of code as small projects"
Great idea for another pet project. Maybe one for PurelyFunctional.tv? ;)
Thanks a lot!
Dan