Not invented here syndrome (NIHS) is a slightly tongue-in-cheek name for the tendency of both individual developers and entire organizations to reject suitable external solutions to software development problems in favor of internally developed solutions.
This phrase came up in a discussion I was a part of recently, and I wanted to speak to it. Whether to build something yourself or choose an already-built solution is one of the most vital discussions we can have, so more perspective is always useful. This phrase is a great shorthand for the kind of bias we might have in software development, where we arbitrarily trust our own solution without taking into account the cost and pain of development, and more importantly, maintenance.
Even thought this is a valid concept, it's also important to not give this idea too much extra weight. A similar concept is "don't reinvent the wheel". These might be somewhat interchangeable. I want to examine where these ideas should and should not be applied. Your mileage may vary on all of these ideas.
Where this phrase applies best
Fundamental low-level software, like databases and frameworks should generally not be re-invented without a painful need and a lot of thoughtful consideration. At DEV some examples are PostgreSQL, Ruby on Rails, Preact, Redis, etc. Other utility libraries probably are safe to not be re-invented, like HTTParty which provides a clean interface over the core HTTP functionality provided by Ruby. Sometimes new things need to be invented along these lines, but it's not always a good idea.
Many of our core external services, such as logging, payments, monitoring, should probably also not be reinvented. There is so much complexity, infrastructure, and perhaps regulation, that reinvention seems incredibly unwise.
Where this phrase doesn't apply as well
"Duplication is far cheaper than the wrong abstraction" - Sandi Metz
This is a fabulous quote and blog post which applies really well here. This debate is always about whether the topic at hand is the right abstraction.
The aforementioned PostgreSQL is simultaneously robust and incredible software, while also being the wrong abstraction for a lot of types of data. That is why other databases get invented. They, too, are leaky abstractions, as is all software. We accept this abstraction because the pain of building a concrete solution is obviously a bad idea. Natural selection among those who do and do not attempt this is probably enough to weed out this notion from our debates.
When the topic at hand is "business logic", e.g. the type of code we write to solve user problems, and deliver on the ultimate purpose of the project, "inventing here" is a really good idea a lot of the time. And when going for an external solution, having the idea in mind that you may need to migrate off of it is always worth keeping in mind. The right abstractions are incredibly hard to come by. At DEV we use plenty of libraries "not invented here", and there are always limitations that beg the question of whether we should have written the logic ourselves.
A couple of libraries, acts-as-taggable-on and acts_as_follower handle a lot of the logic of tags and followers. It wouldn't be practical to migrate away from these any time soon, but they provide us with opinions which are sometimes just different enough that we may have been better off writing these concepts into the app ourselves. Tagging and following are higher abstractions which do not rise to the complexity of databases and frameworks. Since these are areas we care about in terms of user experience, I've since adopted more of a "Yes, invent here!" approach even when the current functionality we are looking for is available in a library.
Anything related to our own team's effective workflow, especially in terms of moderation and administration, typically deserves to be written by our team and community. The way we interact with the software, especially anything with possibly destructive consequences, is begging to be written in-house in our world. It helps that we are a technology-driven organization with solid funding and an open source community to boot.
In a few cases I've copied and pasted library code into a project instead of including a package in order to adopt some of the functionality now, but also allow for clean internal hackability when it inevitably becomes needed. This is not "clean" if done for the wrong reasons, but it's not inherently dirty either.
NPM & left-pad: Have We Forgotten How To Program?
This is another article that has stuck with me. Perhaps because I personally suffered the pain of "Left Pad" — A day of reckonning for the JavaScript community.
The article takes on a lot of the same ideas I've outlined here, more specifically in reference to "npm culture" which is known for taking this idea to extremes.
Take on a dependency for any complex functionality that would take a lot of time, money, and/or debugging to write yourself. Things like a database access layer (ORM) or caching client should be dependencies because they’re complicated and the risk of the dependency is well worth the savings and efficiency.
But, for the love of all that is programming, write your own bloody basic programming functions.
In conclusion
"Not invented here syndrome" is real, but it's easy to swing the pendulum too far. Technology-driven companies should do a lot of invention, even when it feels like reinvention. It's what we do. These named patterns we identify with should always be the start of a conversation or debate, not the thing that ends the debate.
A lot of the specifics of my argument relate to our own organizational circumstance, but just like the phrase itself, there are broad lessons that can be applied. Individual decisions are always about resource constraints and culture, and your organization may have different ones. Google, famously, invents a lot of stuff in house, and always has. Their decision-making process is unfathomably different than ours, but the in-house invention is as much a cultural thing as it is a resource issue for them as far as I can tell.
Please let me know what you think in the comments!
Latest comments (18)
My personal philosophy is to look at the business value. Does reinventing the wheel provide better value for our customers and is more important than what we could be doing otherwise?
I generally like to use external resources until they begin to cause problems. Only then do I think about investing something to replace them. This way I make sure not to invent the wrong thing at the wrong time.
I work for a critical messaging company.
If I were to suggest we should create our own database, this would be clearly NIH at its worst. We use databases, of course. We also use web servers, and all sorts. I wouldn't advocate trying to duplicate any of those.
If I were to suggest we should create our own messaging server, this makes a lot more sense - of course we could use another existing one, but our whole company is about messaging, so it makes sense that we should fully own our solution. We can, of course, survive on an existing server for some time, though.
If I were to suggest that we should use a pre-existing messaging client, though, that would be bizarre - we obviously need to write our own mobile apps. But we use existing libraries to do so - perhaps we'll start to commit back to those, and perhaps we might eventually choose to write our own.
So to me, the argument isn't about "high" or "low" level, it's about whether it makes sense to own a particular component given your mission. I think of this as a critical path through the stack, though I freely admit I'm mixing technical metaphor here.
Good point ... over-reliance on third party libs for pretty trivial pieces of functionality (application/ business logic, as opposed to "infrastructure") can easily lead to unnecessary "vendor lock-in". In these cases I'd also be inclined to just build a home-grown custom solution "at the app level".
There's an objective balance you can use:
As the value of the code is closer to the core of your business domain, it's more likely you'll want to write it yourself (if you have the skills) other than delegate to a third party.
If the code is closer to the core value of a technical domain outside of your core business domain, then it's more likely you'll want to use somebody else's code.
It's more of a risk management decision than an actual coding decision. It turns out many developers don't understand risk management in the context of the organisation or the project
"These named patterns we identify with should always be the start of a conversation or debate, not the thing that ends the debate."
Very well said.
I've seen lots of programming "best practices" come and go. Many years ago it was "comment everything". Later on it was "comment nothing". Both are wrong.
As our experience grows, we realize that dogmatic oversimplifications like these are naive. They fail to take into account context and nuance. They may be correct in most cases, but when invoked as sacred truth their overuse is detrimental to a code base.
Another example is YAGNI (You Ain't Gonna Need It). I've been in discussions where, as you suggest, invoking it was like playing a trump card -- subsequent discussion was futile. YAGNI has an extreme beyond which it should not apply; if you wrote a left-pad method (speaking of left-pad methods) and so far you always needed to pad to a width of 16, would you write this?:
def left_pad_16(string)
No, of course not, you would parameterize the width:
def left_pad(string, width)
And yet the unquestioning application of YAGNI would dictate against that. It's a silly extreme, but it illustrates the point.
So, as you say, "not invented here syndrome" raises an important and useful question, but applying it unconditionally without considering the context and nuance is unhelpful.
This raises the question of whether code is the best place for business logic. One of the things we have done as developers is feather our beds by raising barriers to certain tasks or roles, such as specifying business rules, data structures, user interfaces.
In many ways we retreated from tools like Delphi which made design more accessible. Developing basic business apps requires substantially more skill sets than they ever have. My own thesis is that we should write software that decouples from the domain as far as possible, just as databases have, and allow people without development skills to leverage technology to solve their own problems.
The spreadsheet is perhaps the best example of this approach, where managers can create their own solutions without knowing how to code. Most development tools follow the orthodoxy of domain binding, forcing you to statically link to data structures at design time.
We already accept the wisdom of domain decoupling when it comes to databases or email, or even TCP. With all of these the payloads are a separate concern. Obviously there is a domain, so I'm not suggesting that it does not exist, only that developers need to release their death grip over it and give it back to users.
Hi Ben, great article!
It is also fine to reinvent something just to learn how it works.
Following the slogan:
Bundle size and tree-shakeability should also be considered when importing npm packages.
Could you elaborate on licence concerns when copy and pasting a part of the code from a library? What implications does this have?
Is it necessary to add a licence notice about the library in the project? If not done, is it a copyright infringement?
Does it differ when the copied code code is altered slightly?
I assume understanding the given code and coming up with an very similar solution would be legally fine.
Also a good article on that matter with the tendency to rely on proven solutions:
How to stand on the shoulders of giants by Quincy Larson
Invented Here Syndrome
Great article!
I think the key point you touched on is that the the core of your business should always be invented there. Otherwise, your business has no reason to exist over another constructed from the same commodity pieces.
I also find that I reinvent a few lower-level things simply because the big name options that exist out there have created too complex and/or too opinionated of an abstraction. For example, I wrote my own libraries for:
These numbers are from scc.
All of these projects started on the weekend at home out of frustration with the tools available to do the same job. I started working on it just to see if I could do better. The end result turned out useful enough that we use it at work.
Reinventing a wheel works well as an analogy because using a wheel is simple. You bolt it on and it works. If it breaks, you get another one off the shelf and bolt that on. It is the manufacturing of it that is more complicated. But it's already done for you. If wheels start needing significant investment to use correctly, they should be reinvented. Which is what I did with the libraries mentioned above.
Only last week, one of our talented junior developers tried to convince me that utility libraries (in this case lodash or ramda) were worth including because they are tried and tested and have better code quality than anything you could write in a few minutes.
I raised three objections:
Many of those libraries have a track record of having had many exploitable vulnerabilities and bad code quality. Just because something is widely used doesn't mean it is secure and has good code quality. Internet Explorer was the most used browser at one point in history and some of us still remember... though I digress.
You often get a lot more than you think: let's take for example an
isEmptyfunction - he actually brought it up in the discussion, which I am thankful about. In most cases you want to use it, you know exactly what type of value you are checking (if you are using TypeScript, you could even get it inferred if it is not already typed). In most cases, these are Objects, because in every other case, you wouldn't even considerisEmpty. There are a lot of checks for sets, maps, strings, etc., that you won't need for your use case.Using these helpers can save time, but that time is only wasted if used to think about a solution before understanding the underlying problem. Especially if you are less experienced, I challenge you to try this: before you import a helper to write an easy solution, try to understand the problem and how your helper would be solving it (e.g. by reading its sources). Then, only if implementing the solution yourself will take more than twice as long as an npm install, should you add the extra dependency. Even if you don't change your behavior, you will at least have learned how those helpers are doing it.
TL;DR: When using external dependencies is merely a replacement for thinking about the underlying problem, the diagnosis of NIHS does not apply.