Not invented here syndrome (NIHS) is a slightly tongue-in-cheek name for the tendency of both individual developers and entire organizations to reject suitable external solutions to software development problems in favor of internally developed solutions.
This phrase came up in a discussion I was a part of recently, and I wanted to speak to it. Whether to build something yourself or choose an already-built solution is one of the most vital discussions we can have, so more perspective is always useful. This phrase is a great shorthand for the kind of bias we might have in software development, where we arbitrarily trust our own solution without taking into account the cost and pain of development, and more importantly, maintenance.
Even thought this is a valid concept, it's also important to not give this idea too much extra weight. A similar concept is "don't reinvent the wheel". These might be somewhat interchangeable. I want to examine where these ideas should and should not be applied. Your mileage may vary on all of these ideas.
Where this phrase applies best
Fundamental low-level software, like databases and frameworks should generally not be re-invented without a painful need and a lot of thoughtful consideration. At DEV some examples are PostgreSQL, Ruby on Rails, Preact, Redis, etc. Other utility libraries probably are safe to not be re-invented, like HTTParty which provides a clean interface over the core HTTP functionality provided by Ruby. Sometimes new things need to be invented along these lines, but it's not always a good idea.
Many of our core external services, such as logging, payments, monitoring, should probably also not be reinvented. There is so much complexity, infrastructure, and perhaps regulation, that reinvention seems incredibly unwise.
Where this phrase doesn't apply as well
"Duplication is far cheaper than the wrong abstraction" - Sandi Metz
This is a fabulous quote and blog post which applies really well here. This debate is always about whether the topic at hand is the right abstraction.
The aforementioned PostgreSQL is simultaneously robust and incredible software, while also being the wrong abstraction for a lot of types of data. That is why other databases get invented. They, too, are leaky abstractions, as is all software. We accept this abstraction because the pain of building a concrete solution is obviously a bad idea. Natural selection among those who do and do not attempt this is probably enough to weed out this notion from our debates.
When the topic at hand is "business logic", e.g. the type of code we write to solve user problems, and deliver on the ultimate purpose of the project, "inventing here" is a really good idea a lot of the time. And when going for an external solution, having the idea in mind that you may need to migrate off of it is always worth keeping in mind. The right abstractions are incredibly hard to come by. At DEV we use plenty of libraries "not invented here", and there are always limitations that beg the question of whether we should have written the logic ourselves.
A couple of libraries, acts-as-taggable-on and acts_as_follower handle a lot of the logic of tags and followers. It wouldn't be practical to migrate away from these any time soon, but they provide us with opinions which are sometimes just different enough that we may have been better off writing these concepts into the app ourselves. Tagging and following are higher abstractions which do not rise to the complexity of databases and frameworks. Since these are areas we care about in terms of user experience, I've since adopted more of a "Yes, invent here!" approach even when the current functionality we are looking for is available in a library.
Anything related to our own team's effective workflow, especially in terms of moderation and administration, typically deserves to be written by our team and community. The way we interact with the software, especially anything with possibly destructive consequences, is begging to be written in-house in our world. It helps that we are a technology-driven organization with solid funding and an open source community to boot.
In a few cases I've copied and pasted library code into a project instead of including a package in order to adopt some of the functionality now, but also allow for clean internal hackability when it inevitably becomes needed. This is not "clean" if done for the wrong reasons, but it's not inherently dirty either.
NPM & left-pad: Have We Forgotten How To Program?
This is another article that has stuck with me. Perhaps because I personally suffered the pain of "Left Pad" โ A day of reckonning for the JavaScript community.
The article takes on a lot of the same ideas I've outlined here, more specifically in reference to "npm culture" which is known for taking this idea to extremes.
Take on a dependency for any complex functionality that would take a lot of time, money, and/or debugging to write yourself. Things like a database access layer (ORM) or caching client should be dependencies because theyโre complicated and the risk of the dependency is well worth the savings and efficiency.
But, for the love of all that is programming, write your own bloody basic programming functions.
In conclusion
"Not invented here syndrome" is real, but it's easy to swing the pendulum too far. Technology-driven companies should do a lot of invention, even when it feels like reinvention. It's what we do. These named patterns we identify with should always be the start of a conversation or debate, not the thing that ends the debate.
A lot of the specifics of my argument relate to our own organizational circumstance, but just like the phrase itself, there are broad lessons that can be applied. Individual decisions are always about resource constraints and culture, and your organization may have different ones. Google, famously, invents a lot of stuff in house, and always has. Their decision-making process is unfathomably different than ours, but the in-house invention is as much a cultural thing as it is a resource issue for them as far as I can tell.
Please let me know what you think in the comments!
Top comments (18)
Thanks for the article! I agree that this is a tough issue because so much of it depends on you and your organization's/project's specific circumstances.
I used to try and pull in as many third party dependencies as possible in order to accelerate development. Now, I tend to choose more foundational dependencies and avoid pulling in smaller libraries that bring in functionality I could easily do myself. I've found that it's easier to create something that is suited exactly for my purposes than it is to pull in and manage a dependency that was created for someone else's problems. It's always a trade off ๐
Joel Spolsky famously wrote a "Defense of Not-Invented-Here Syndrome."
In my career, I've more often seen bad cases of not-invented-here syndrome than I've seen cases like Spolsky mentions. After seventeen years, my personal philosophy about software engineering has developed into the idea that software engineers should do the most simple and obvious thing possible as much as possible, because I believe Donald Knuth's stance that code is actually human-to-human communication disguised as "instructions for a computer." This guideline overlaps with the idea of NIH syndrome in the sense that usually you want to reuse the abstractions people know already so that they can understand what your code does.
But it also doesn't rule out inventing your own something. In fact, it supports it in many cases where a domain-specific language makes the code clearer (and therefore "more simple and obvious") to others.
Ironically, even though I linked to another Spolsky post, this article could be a re-invention of the same basic presence he was outlining.
However, in framing it in my own experiences and cultural context, the re-invention was justified ๐
I really like this quote from that post.
I think a good takeaway is definitely to know what types of software you should be inventing and don't invent the other kinds, and that is going to be incredibly context-dependent. I think that's probably the issue with these broad statements in general, they require contextual wisdom and framing.
That's why we hang around on sites like this, so we can noodle on the context.
One benefit of rolling your own stuff (which I agree you should avoid regarding foundational pieces of the stack) is the opportunity to attack a problem from first principles. That allows you to challenge a lot of the assumptions that exist in external libraries in the same space, sometimes coming up with a much simpler bespoke solution.
The obvious danger is when you think you can simplify authentication, so you spin up your own insecure auth library! The same goes for a large number of generic tasks (like you mentioned: ORMs, web frameworks, servers, etc).
As long as there is some intentional thought or dialogue put into writing your own solution vs. pulling in a third-party dependency it's usually easy to determine the better path.
Only last week, one of our talented junior developers tried to convince me that utility libraries (in this case lodash or ramda) were worth including because they are tried and tested and have better code quality than anything you could write in a few minutes.
I raised three objections:
Many of those libraries have a track record of having had many exploitable vulnerabilities and bad code quality. Just because something is widely used doesn't mean it is secure and has good code quality. Internet Explorer was the most used browser at one point in history and some of us still remember... though I digress.
You often get a lot more than you think: let's take for example an
isEmptyfunction - he actually brought it up in the discussion, which I am thankful about. In most cases you want to use it, you know exactly what type of value you are checking (if you are using TypeScript, you could even get it inferred if it is not already typed). In most cases, these are Objects, because in every other case, you wouldn't even considerisEmpty. There are a lot of checks for sets, maps, strings, etc., that you won't need for your use case.Using these helpers can save time, but that time is only wasted if used to think about a solution before understanding the underlying problem. Especially if you are less experienced, I challenge you to try this: before you import a helper to write an easy solution, try to understand the problem and how your helper would be solving it (e.g. by reading its sources). Then, only if implementing the solution yourself will take more than twice as long as an npm install, should you add the extra dependency. Even if you don't change your behavior, you will at least have learned how those helpers are doing it.
TL;DR: When using external dependencies is merely a replacement for thinking about the underlying problem, the diagnosis of NIHS does not apply.
Hi Ben, great article!
It is also fine to reinvent something just to learn how it works.
Following the slogan:
Bundle size and tree-shakeability should also be considered when importing npm packages.
Could you elaborate on licence concerns when copy and pasting a part of the code from a library? What implications does this have?
Is it necessary to add a licence notice about the library in the project? If not done, is it a copyright infringement?
Does it differ when the copied code code is altered slightly?
I assume understanding the given code and coming up with an very similar solution would be legally fine.
Also a good article on that matter with the tendency to rely on proven solutions:
How to stand on the shoulders of giants by Quincy Larson
I'm always voting to use external solutions because such an approach saves a lot of time (usually ๐) and allows us to focus on achieving the goals.
A bright example is to use an external UI library instead of creating your own from scratch.
Great article!
I think the key point you touched on is that the the core of your business should always be invented there. Otherwise, your business has no reason to exist over another constructed from the same commodity pieces.
I also find that I reinvent a few lower-level things simply because the big name options that exist out there have created too complex and/or too opinionated of an abstraction. For example, I wrote my own libraries for:
These numbers are from scc.
All of these projects started on the weekend at home out of frustration with the tools available to do the same job. I started working on it just to see if I could do better. The end result turned out useful enough that we use it at work.
Reinventing a wheel works well as an analogy because using a wheel is simple. You bolt it on and it works. If it breaks, you get another one off the shelf and bolt that on. It is the manufacturing of it that is more complicated. But it's already done for you. If wheels start needing significant investment to use correctly, they should be reinvented. Which is what I did with the libraries mentioned above.
Great article. I would make the argument that for front end development, frameworks and libraries are typically safe to use from external sources. Provided the research was done to make sure it actually delivers and solve a need.
The best use case I have seen for avoiding external code is with components. When developers pull in a calendar widget, a expander, slider, etc it does solve a problem. But as I get older and focus more and more on accessibility and performance, I see these as bloating the client.
The really big cause is design and need. Many components are built for the masses to make things easier. Others are restrictive on patterns.
So what that causes us developers to do is either compromise the design, or choose larger components we can custom to our needs.
The other benefit for a front-end Dev to actually build it themselves, is they learn. So many boot camp devs have only copy and pasted from stack overflow or used libraries. Building it from scratch to meet your organizations needs, while focusing on a11y and performance, really educates.
Not everyone can afford to do this. But itโs a trade off we have made any chance we can, and it has paid off in spades with the development of our teams. But it does have to be thoughtful, and justified.
So very, very true!
Related...
Maybe Reinvent The Wheel
Ben Halpern ใป Jul 12 '16 ใป 2 min read
Please Reinvent The Wheel
Jason C. McDonald ใป Jan 20 '18 ใป 7 min read
This raises the question of whether code is the best place for business logic. One of the things we have done as developers is feather our beds by raising barriers to certain tasks or roles, such as specifying business rules, data structures, user interfaces.
In many ways we retreated from tools like Delphi which made design more accessible. Developing basic business apps requires substantially more skill sets than they ever have. My own thesis is that we should write software that decouples from the domain as far as possible, just as databases have, and allow people without development skills to leverage technology to solve their own problems.
The spreadsheet is perhaps the best example of this approach, where managers can create their own solutions without knowing how to code. Most development tools follow the orthodoxy of domain binding, forcing you to statically link to data structures at design time.
We already accept the wisdom of domain decoupling when it comes to databases or email, or even TCP. With all of these the payloads are a separate concern. Obviously there is a domain, so I'm not suggesting that it does not exist, only that developers need to release their death grip over it and give it back to users.
I really appreciate the thoughts here. There's definitely a spectrum and it's not the same spectrum for every organization because each organization's needs are continually evolving and in different directions. For early iterations, NIHS is a good rule to follow as it helps with speed to market. That said, as a platform matures and grows I think forking or doing it yourself is perfectly reasonable as business logic often outgrows available abstractions.
"These named patterns we identify with should always be the start of a conversation or debate, not the thing that ends the debate."
Very well said.
I've seen lots of programming "best practices" come and go. Many years ago it was "comment everything". Later on it was "comment nothing". Both are wrong.
As our experience grows, we realize that dogmatic oversimplifications like these are naive. They fail to take into account context and nuance. They may be correct in most cases, but when invoked as sacred truth their overuse is detrimental to a code base.
Another example is YAGNI (You Ain't Gonna Need It). I've been in discussions where, as you suggest, invoking it was like playing a trump card -- subsequent discussion was futile. YAGNI has an extreme beyond which it should not apply; if you wrote a left-pad method (speaking of left-pad methods) and so far you always needed to pad to a width of 16, would you write this?:
def left_pad_16(string)
No, of course not, you would parameterize the width:
def left_pad(string, width)
And yet the unquestioning application of YAGNI would dictate against that. It's a silly extreme, but it illustrates the point.
So, as you say, "not invented here syndrome" raises an important and useful question, but applying it unconditionally without considering the context and nuance is unhelpful.
My personal philosophy is to look at the business value. Does reinventing the wheel provide better value for our customers and is more important than what we could be doing otherwise?
I generally like to use external resources until they begin to cause problems. Only then do I think about investing something to replace them. This way I make sure not to invent the wrong thing at the wrong time.
Good point ... over-reliance on third party libs for pretty trivial pieces of functionality (application/ business logic, as opposed to "infrastructure") can easily lead to unnecessary "vendor lock-in". In these cases I'd also be inclined to just build a home-grown custom solution "at the app level".
I work for a critical messaging company.
If I were to suggest we should create our own database, this would be clearly NIH at its worst. We use databases, of course. We also use web servers, and all sorts. I wouldn't advocate trying to duplicate any of those.
If I were to suggest we should create our own messaging server, this makes a lot more sense - of course we could use another existing one, but our whole company is about messaging, so it makes sense that we should fully own our solution. We can, of course, survive on an existing server for some time, though.
If I were to suggest that we should use a pre-existing messaging client, though, that would be bizarre - we obviously need to write our own mobile apps. But we use existing libraries to do so - perhaps we'll start to commit back to those, and perhaps we might eventually choose to write our own.
So to me, the argument isn't about "high" or "low" level, it's about whether it makes sense to own a particular component given your mission. I think of this as a critical path through the stack, though I freely admit I'm mixing technical metaphor here.