The troubles of modern software dependency management and what to do about them

In order to help developers tracking dependencies, FASTEN provides a new intelligent package management framework, on top of existing dependency management tools.

The FASTEN European research project wants to support DevOps teams by helping them managing and mastering dependencies, at a finer grain level compare to what they are currently doing.

The FASTEN consortium is preparing an intelligent dependency management solution that will bring innovation such as: a more accurate evaluation of the impact of dependencies, security vulnerabilities, license compliance, risk management, and evaluation of the consequences of library API changes on users.

The project is led by a team of researchers from the Dutch University TU Delft associated with academics in Milan and Athens, industry engineers from SMEs Endocode, SIG, XWiki and OW2 open source community. This article evokes the first results obtained after more than 2 years of cooperation started in January 2019 and funded by the European Commission for three years.

Software produced as part of FASTEN project is open source. As a result, any developers team can provide feedback and help improve the tools.

FASTEN wants to tame dependencies

FASTEN addresses a fundamental problem of developers, that is, the way they manage the dependencies of their projects. Reusing existing dependencies (usually opened source) is quite easy thanks to automation build tools and package managers. However, several risks should be identified and assessed beforehand. Indeed, when a developer re-uses existing third-party tools or libraries to save time, he or she lower the control over the code that will be part of his application. Software systems may depend on multiple external libraries and components that evolve separately, without centralized coordination. Therefore, several questions arise, in relation to the integrated functionalities. Such questions relate to trust, security or missing notifications regarding the evolution of a software component.

How to verify that the functions provided by a dependency correspond to what is expected, without any security implications?

How one can guarantee that a dependency update won’t prevent the end-user application from running?

Costly consequences

In the recent years, we have witnessed several spectacular ecosystem failures with severe implications on client programs, end users and the further adoption of OSS:

1. A dispute over a library name in the NPM ecosystem led to the removal of a library called left-pad. The package removal led to the collapse of thousands of libraries which directly or transitively depended on leftpad, and hence a major disruption for client programs. Even after the leftpad incident, a study estimated that libraries exist whose removal can affect more that 30% of the core components of the network.

2. A company named Equifax leaked over 100.000 credit card records due to a dependency that was not updated. The compromised systems included a vulnerable version of the Apache Struts library, whose update was postponed as the Equifax security team erroneously underestimated the impact of the bug on their codebase.

3. Malicious developers uploaded to the Python package manager repository (PyPI) libraries whose name was deliberately misspelled, being almost identical to the original libraries (e.g.,urllib in-stead of urllib3). The intention was to steal information from client applications of developers who had accidentally mistyped the library name in the dependency file.

Development teams should guard against such technical issues, or legal problems including incompatible open source licenses. In particular, the consequences of the slightest code change should be known by the team in charge of maintaining the applications in operational condition and by the IT cybersecurity team, which is not easily doable today.

A better way to manage dependencies is therefore necessary and this is precisely the goal of the FASTEN project (Fine-Grained Analysis of Software Ecosystems as Networks).

An analysis of dependent functions

FASTEN’s goal is to strengthen software ecosystems through smarter application dependencies analysis. The implementation of its analysis tools is under development for popular programming languages ​​such as Java, Python and C.

At the heart of the project, a knowledge base that will store functions call graphs that cover both library internal calls and inter libraries calls. The developers team can benefit from this knowledge base thanks to the integration provided by FASTEN in popular build automation tools. Schema below illustrate FASTEN architecture:

The network formed from the dependency call graphs provides valuable information on how the dependencies are used and what is their impact. Hence, developers can be aware of how many customers will be really affected by the last change of their code. Likewise, operators will be reassured before updating a particular library. They will also know if a dependency released under the free GPL license, for example, affects their application.

The developers team will therefore be able to check the constraints in relation to the use of new dependencies, on a technical and legal level, each license bringing rights but also obligations.

In particular, engineers will know how many maintainers are involved in the evolution of an external code artifact, how many users have already put a certain library into production and what are the known vulnerabilities that affect the dependency. Many other variables can be also collected, at the function level, and then the team may decide whether or not to perform an upgrade of a particular framework or library.

Integrated with most popular package managers

The FASTEN project will be integrated into popular package managers for the Java, C, and Python programming languages, such as Maven and PyPI. Plugins for PyPi and Maven are already under development and will be ready for public testing soon. Developers will have access to the FASTEN knowledge base by just using their package manager and will be able to explore the FASTEN knowledge base to find information regarding libraries or projects. For instance, when developers want to update a dependency, the estimated impact of this update on their machine will be communicated by the package manager. Developers will also be notified about any security issue (e.g. if they are using the piece of code where the vulnerability is spotted; if any new issue affects transitive dependencies; or if an update is required). Compare to current solution that notify the user of security vulnerability, FASTEN will bring major improvement as it will only notify the user if he/she actually uses the part of the library that include the security issue. The Figure below provides an example of a FASTEN workflow with PyPi and illustrates how the technology can support DevOps teams and help developers making the rights decisions.

FASTEN has received funding from the European Union’s Horizon 2020 research and innovation programme. It is carried out by a Consortium composed of AUEB, TUDelft, University of Milan-Bicocca, Endocode, OW2, SIG, and XWIKI. The project started in January 2019 and will run until December 2021.

Useful links

• FASTEN GitHub repository: https://github.com/fasten-project
• FASTEN website: https://www.fasten-project.eu/
• FASTEN presentation video: https://www.youtube.com/watch?v=4gqPTZ3ASU8
• FASTEN beta-testing campaign on the Java Call Graph Generator: https://www.fasten-project.eu/view/Events/FASTEN_Beta-testing%20_Campaign_%231