DEV Community: Antoine Mottier

The troubles of modern software dependency management and what to do about them

Antoine Mottier — Wed, 05 May 2021 13:01:14 +0000

In order to help developers tracking dependencies, FASTEN provides a new intelligent package management framework, on top of existing dependency management tools.

The FASTEN European research project wants to support DevOps teams by helping them managing and mastering dependencies, at a finer grain level compare to what they are currently doing.

The FASTEN consortium is preparing an intelligent dependency management solution that will bring innovation such as: a more accurate evaluation of the impact of dependencies, security vulnerabilities, license compliance, risk management, and evaluation of the consequences of library API changes on users.

The project is led by a team of researchers from the Dutch University TU Delft associated with academics in Milan and Athens, industry engineers from SMEs Endocode, SIG, XWiki and OW2 open source community. This article evokes the first results obtained after more than 2 years of cooperation started in January 2019 and funded by the European Commission for three years.

Software produced as part of FASTEN project is open source. As a result, any developers team can provide feedback and help improve the tools.

FASTEN wants to tame dependencies

FASTEN addresses a fundamental problem of developers, that is, the way they manage the dependencies of their projects. Reusing existing dependencies (usually opened source) is quite easy thanks to automation build tools and package managers. However, several risks should be identified and assessed beforehand. Indeed, when a developer re-uses existing third-party tools or libraries to save time, he or she lower the control over the code that will be part of his application. Software systems may depend on multiple external libraries and components that evolve separately, without centralized coordination. Therefore, several questions arise, in relation to the integrated functionalities. Such questions relate to trust, security or missing notifications regarding the evolution of a software component.

How to verify that the functions provided by a dependency correspond to what is expected, without any security implications?

How one can guarantee that a dependency update won’t prevent the end-user application from running?

Costly consequences

In the recent years, we have witnessed several spectacular ecosystem failures with severe implications on client programs, end users and the further adoption of OSS:

A dispute over a library name in the NPM ecosystem led to the removal of a library called left-pad. The package removal led to the collapse of thousands of libraries which directly or transitively depended on leftpad, and hence a major disruption for client programs. Even after the leftpad incident, a study estimated that libraries exist whose removal can affect more that 30% of the core components of the network.
A company named Equifax leaked over 100.000 credit card records due to a dependency that was not updated. The compromised systems included a vulnerable version of the Apache Struts library, whose update was postponed as the Equifax security team erroneously underestimated the impact of the bug on their codebase.
Malicious developers uploaded to the Python package manager repository (PyPI) libraries whose name was deliberately misspelled, being almost identical to the original libraries (e.g.,urllib in-stead of urllib3). The intention was to steal information from client applications of developers who had accidentally mistyped the library name in the dependency file.

Development teams should guard against such technical issues, or legal problems including incompatible open source licenses. In particular, the consequences of the slightest code change should be known by the team in charge of maintaining the applications in operational condition and by the IT cybersecurity team, which is not easily doable today.

A better way to manage dependencies is therefore necessary and this is precisely the goal of the FASTEN project (Fine-Grained Analysis of Software Ecosystems as Networks).

An analysis of dependent functions

FASTEN’s goal is to strengthen software ecosystems through smarter application dependencies analysis. The implementation of its analysis tools is under development for popular programming languages such as Java, Python and C.

At the heart of the project, a knowledge base that will store functions call graphs that cover both library internal calls and inter libraries calls. The developers team can benefit from this knowledge base thanks to the integration provided by FASTEN in popular build automation tools. Schema below illustrate FASTEN architecture:

The network formed from the dependency call graphs provides valuable information on how the dependencies are used and what is their impact. Hence, developers can be aware of how many customers will be really affected by the last change of their code. Likewise, operators will be reassured before updating a particular library. They will also know if a dependency released under the free GPL license, for example, affects their application.

The developers team will therefore be able to check the constraints in relation to the use of new dependencies, on a technical and legal level, each license bringing rights but also obligations.

In particular, engineers will know how many maintainers are involved in the evolution of an external code artifact, how many users have already put a certain library into production and what are the known vulnerabilities that affect the dependency. Many other variables can be also collected, at the function level, and then the team may decide whether or not to perform an upgrade of a particular framework or library.

Integrated with most popular package managers

The FASTEN project will be integrated into popular package managers for the Java, C, and Python programming languages, such as Maven and PyPI. Plugins for PyPi and Maven are already under development and will be ready for public testing soon. Developers will have access to the FASTEN knowledge base by just using their package manager and will be able to explore the FASTEN knowledge base to find information regarding libraries or projects. For instance, when developers want to update a dependency, the estimated impact of this update on their machine will be communicated by the package manager. Developers will also be notified about any security issue (e.g. if they are using the piece of code where the vulnerability is spotted; if any new issue affects transitive dependencies; or if an update is required). Compare to current solution that notify the user of security vulnerability, FASTEN will bring major improvement as it will only notify the user if he/she actually uses the part of the library that include the security issue. The Figure below provides an example of a FASTEN workflow with PyPi and illustrates how the technology can support DevOps teams and help developers making the rights decisions.

FASTEN has received funding from the European Union’s Horizon 2020 research and innovation programme. It is carried out by a Consortium composed of AUEB, TUDelft, University of Milan-Bicocca, Endocode, OW2, SIG, and XWIKI. The project started in January 2019 and will run until December 2021.

Useful links

FASTEN GitHub repository: https://github.com/fasten-project
FASTEN website: https://www.fasten-project.eu/
FASTEN presentation video: https://www.youtube.com/watch?v=4gqPTZ3ASU8
FASTEN beta-testing campaign on the Java Call Graph Generator: https://www.fasten-project.eu/view/Events/FASTEN_Beta-testing%20_Campaign_%231

Streaming prerecorded online conference on YouTube live

Antoine Mottier — Mon, 24 Aug 2020 13:17:32 +0000

In this article I'll explain how you can easily stream a prerecorded video on YouTube Live using open source software.

We use it at OW2 for our own online conference: OW2online and as it did run flawlessly I thought it'd be a good idea to share my experience.

YouTube

Obviously the first step is to create an account on YouTube.

Next you need to create a new stream:

Go to YouTube Studio live: https://studio.youtube.com/channel/UC/livestreaming
In the menu in the left click on "Stream" to create a new stream
Configure the new stream:
- Title: choose a short meaningful title.
- For visibility you have 3 options: public, private (only you) or unlisted (only users with the link).
- Provide a description and make sure that it includes keywords that can help potential viewers to find your stream (if public).
- Pick up a category.
- You are probably planning your live event ahead so check the option "Schedule for later" and select a date and time.
- Upload a thumbnail that will be displayed to users when they try to view your stream before it actually starts.
- Select the appropriate option to define the targeted audience.
Click on "Create stream" button.
All the default settings are usually just fine.
Copy the stream key for later use.

With that the YouTube part should be ready and now you can move on to the streaming part.

Broadcast with Open Broadcaster Studio

Open Broadcaster Studio (OBS) is an open source software that will let you stream your prerecorded video content to YouTube.

First you need to download and install OBS.

Once installed start OBS.

Settings

On the first start you will be asked if you want to run the auto-configuration wizard.
Choose no, we will do a manual configuration.

Click on the "Settings" button

and apply the following modifications (assuming you plan to stream a video recorded in Full HD 1080p without downgrading the quality):

General:
- Select "Show confirmation dialog when stopping streams". This will prevent you from accidentally stopping the stream.
Stream:
- Service: "YouTube".
- Server: "Primary YouTube ingest server". You can stream same content from another computer as a backup. On this second computer you should choose "Backup YouTube ingest server".
- Stream key: paste the key you copied when creating the stream in YouTube Studio.
Output:
- Output Mode: "Advanced".
- Streaming tab:
- Encoder: "x264".
- Rate Control: "VBR".
- Bitrate: 8000 Kbps.
- Recording tab (in case you plan to also record locally your streaming):
- Recording Format: "mp4".
Audio:
- Sample Rate: "48kHz".
- Desktop Audio: "Disabled". You usually don't want to stream sounds coming from the OS.
- Mic/Auxiliary Audio: "Disabled". This will prevent from streaming sound from your microphone over your prerecorded video.
Video:
- Base (Canvas) Resolution: 1920x1080 (this assume that your screen is a full HD).
- Output (Scaled) Resolution: 1920x1080 (you want the Aspect Ratio to be 16:9 to match YouTube recommendation).
- Common FPS Values: "30".

Click on "OK" to close settings window and on "Yes" to restart OBS.

Content to stream

In order to control when our playback and stream will start we want to configure two "scenes". Each scene will include a single source.
The first scene will include a source with a static image (usually the same as the thumbnail configured in YouTube).
The second scene will include a source that is actually your video file.

In order to stream the static image you need to create and configure a "source" on the already existing default scene (named "Scene"):

In the "Sources" area of OBS window click on the "+" icon.
Select "Image" in the list of source types.
Choose the option create a new source and keep the default name: "Image".
Click on "OK" button.
Click on the "Browse" button to select the image file to stream.
Click on "OK" button.

In order to stream a video you need to:

Create a new scene by clicking on the "+" icon in the "Scenes" area of OBS window.
Keep default name "Scene 2" and click on "OK" button.
In the "Sources" area of OBS window click on the "+" icon.
Select "Media Source" in the list of source types.
Choose the option create a new source and keep the default name "Media Source".
Click on "OK" button.
Click on the "Browse" button to select the video file to stream.
Select the option "Use hardware decoding when available" for better performance.
Click on "OK" button.

Streaming

And now the final steps to start your live stream:

Select the scene with your static image.
Click on "Start streaming" button.
In YouTube click on "Go live" button.
In OBS, select the scene with your video file.
You are streaming your video live!

Conclusion

Doing live stream video event is quite easy and you can achieve professional results using exclusively open source software.

We use this kind of setup for OW2online live event. In an upcoming article I'll cover other aspect of the live event such as scheduling, participants registration and interaction with speaker.

If you need more information you can reach me on Twitter @antoinemottier or LinkedIn and also checkout OW2 activities.