Introduction
As a senior developer, I spend a lot of my time looking at tools and libraries to decide if they're something we can or should use as part of our projects. I've developed a strong ability to quickly filter through lists of possible tools to narrow them down, and then do further research on the remaining candidates.
This is definitely not the kind of skill that they teach you in school, which is sad given the way modern software development is strongly dependent on reusing libraries.
To help with that, here's some of the things I take into consideration when evaluating tools.
How to Correctly Evaluate Third-Party Software Libraries and Tools
Starting Point: Search Locations
For software libraries, you should typically start with the appropriate package manager site for the language you're using, such as NPM for JavaScript, PyPi for Python, or Maven Central for Java). If you're looking for non-library tools such as IDEs, Google is your best bet.
Searching for keywords on those sites should turn up a list of things that are potentially relevant.
Potential Evaluation Criteria
When choosing any tool, you have to evaluate it on multiple criteria. Here's an (incomplete) list of things you may want to consider when looking at any software library or tool that you're considering using:
Zeroth Pass: Relevance
The first step is to select a list of tools that actually look relevant to your topic. For example, searching NPM for react number input right now turns up 219 hits. There's probably only a much smaller number that actually match what you're looking for. Skim through the hits and pop open the most likely looking options based on the initial descriptions.
First Pass: Basic Suitability
Now that you've got the initial candidate list, it's time to determine which of these may be a good possibility:
- Does this even solve the problems I have right now?
- Does it solve those problems well?
- What's the license of this tool? Is it compatible with my company's policies?
The goal of using any tool is to solve a problem you have. If a tool doesn't solve your problem, you shouldn't even consider using it in the first place. (Note that this implies that you should understand what problems you're trying to solve before you start investigating tools to help with them :) )
Along with that, there's the question of whether a tool is something you can use based on legal or policy concerns. For example, many companies disallow use of GPL-licensed software libraries to avoid any possible issues mixing their proprietary code with GPL-licensed code and being forced to distribute their own code publicly. Other companies might restrict libraries and applications based on the nationality of the authors, purchased licensing terms, or license costs.
One-line summary of the most common open-source (OSS) licenses:
- MIT, BSD: use the code for whatever you want, just credit the original author
- LGPL: Use it as long as you don't modify the original library
- GPL: If you use it, your own software must be licensed under the GPL as well, and your source must be made available on request
If you know ahead of time that you can't use a particular tool, there's no point in digging into it further.
Second Pass: Detailed Evaluations
Once you've narrowed down the list down to the top 3-4 choices, you can do a more detailed evaluation of each of them.
Here's some of the criteria that I take into consideration. I don't necessarily have these in a formal checklist, but they're usually in the back of my head in some form. (This list is biased towards OSS software libraries, particularly for JavaScript, but most apply more broadly as well).
- How easy is it to use?
- What's the size of the community?
- How many projects appear to use this tool?
- How many downloads does this have weekly/monthly?
- How much info is there on using it outside the core docs?
- What's the size of the ecosystem?
- How many related packages exist?
- If we have an additional use case not solved by the core tool, do addons exist that handle that use case?
- How easy is it to extend or modify this tool?
- How easy is it to understand what the tool does internally and conceptually?
- How much documentation does this have? Is there an actual meaningful docs site, or is it just a Github readme?
- How well tested is this tool?
- How much prior experience does my team have with this tool or something similar?
- Is it well maintained?
- How many bugs does it have?
- Has this ever had any critical vulnerabilities / CVEs? If so, how many, how often, and how long did they take to get fixed?
- How often do new releases occur? When was the last release?
- How many open issues / PRs are there?
- How fast will things get fixed if there's a bug?
- How many maintainers are there? Are they active?
- What's the long-term roadmap for this project?
- How many other dependencies does this tool have?
- How many of them are runtime dependencies vs development-only dependencies?
- Do we need to be concerned about any of them?
- How much does this affect the architecture of our app?
- Can we rip it out if we choose to switch to something else?
- How hard is it to upgrade this tool when a new version comes out?
- Where does this fall in the "hype cycle"?
- Brand new tool that is experimental? It works, but still only used by early adopters? Mainstream? Dying?
- Are people actively starting new projects with this tool, or are they moving away?
- What technical requirements and system constraints does this have?
- What's the minimum runtime version?
- Are there size concerns, such as min+gzip size for a JS library used in the browser?
This isn't an all-inclusive list - there's likely other factors you might want to consider as well.
These all affect a decision on whether using a given tool is a good idea or not for your team. It's also likely that some of these may be more relevant than others depending on your situation.
Note that you could easily spend an infinite amount of time recursing through the transitive dependencies of a given tool, especially if this is a JavaScript package of some kind. For example, a brand new Create-React-App project only has a handful of dependencies: react, react-dom, @testing-library/react, and react-scripts. However, react-scripts itself depends on Jest, ESLint, Webpack, Babel, PostCSS, and a couple dozen other packages, and Webpack in turn ends up pulling in around 700 packages, for a final total of just over 1500 individual packages installed into your node_modules folder. Clearly, inspecting every one of those manually is not feasible, so you have to draw a line somewhere. Given the rise of "supply chain" attacks, where malicious packages are deliberately created to steal information or cause harm, it's critical to make these decisions wisely.
It can sometimes be worth doing a "trade study" to evaluate alternatives, especially if multiple people are involved in this decision. Define several categories, decide on the possible range of numeric scores for each category, and rate each tooling option in each category. You may want to add together the scores for each tool, or average them out. Also, you might want to weight some categories as being more important than others. I suggest defining the weights and scoring approach ahead of time, so that the final decision is more fair.
When searching for critical security vulnerabilities like CVEs, you can use sites like https://cve.mitre.org/cve/search_cve_list.html and https://nvd.nist.gov/vuln/search as references.
For open-source projects on Github, it's worth looking at the number of stars, open issues, and PRs. Those aren't final decision criteria, but they can give you a sense of the popularity and health of a project. Also keep an eye on the recent commits to get an idea how active it is.
NPM shows a weekly download stats graph on the package description page, but I prefer using npm-stat.com, which lets you compare the download stats for multiple packages over a specific period of time, with values shown for daily/weekly/monthly/yearly downloads totals.
For JS client bundle size questions specifically, I strongly recommend using Bundlephobia to determine the true cost of adding a given package to your app. Michel Weststrate's import-size util can also be helpful here.
Further Information
- TLDR Legal: shortened plain English explanations of common software licenses
- Bundlephobia
mweststrate/import-size- Trade Study guidelines
- npm-stat.com
- 17 criteria for evaluating third-party libraries
- The 12 Things You Need to Consider when Evaluating Any New JavaScript Library
- Evaluating 3rd party libraries
Top comments (0)