How we do automated dependency updates in 2020

As the number of projects grew in our company, it became difficult to update manually our dependencies. Early 2019, we looked for a solution to automate this process. We tried a few but ended up being overwhelmed by a flood of PRs and too many branches clogging up our repositories, despite the fact that, in practice, we would always accept pull requests (PRs), but it quickly became a hassle more than a help.

So, we decided to write our own dependency update tool with these requirements in mind:

1. dependencies should be updated on the same branch, one after the other, and in the end, only commits that passed CI should be merged in
2. there's no need for PRs: if the bot can watch update commits run through our CI, it can tell whether it's safe to merge the update branch to the main branch
3. still, we should be able to configure what to do after an update via simple plugins. Send a Slack message, an email, open an issue... you name it.
4. package managers support should be made with plugins so we can add as many as we want
5. we should be able to connect multiple Git servers: we host some projects on our internal Gitlab and others on Github, and we want to view the updates in a single place
6. all our secrets and private registries are already configured in our CI, we should not have to reconfigure that anywhere else

So we wrote it, we've used it, we've named it Pmbot, and we'd like to share it with you ! It's free, and self hosted.

Now, get yourself a cup of coffee, and let's see how to get started !

Installing Pmbot locally with Docker compose

We'll use the following docker-compose.yml:

Note that APP_UI_URL and PMBOT_API_URL are not strictly mandatory, but it'll make your life easier when copy/pasting code snippets from the Pmbot UI. Use the IP or host name of your machine

Now, run docker-compose up -d, and browse http://192.168.1.54:9118. You should get to the setup screen:

Just fill in the registration form and click Create account.

Connect Pmbot to your Git server

We'll start off by selecting the type of Git server we're adding. In this case, we're using a self hosted Gitlab instance, but you can also connect to Github, and we'll be adding support for more Git servers in the coming weeks.

We'll now add Pmbot as a new OAuth2 application in Gitlab so we can get our client ID and client secret, which we can use to complete the Git provider OAuth configuration:

Now, just click Run test. Pmbot will check the connection and synchronize your projects.

Finally, click Add Git provider.

Connect Pmbot to your CI server

We're using Gitlab CI in this case, but you can use Drone (we wrote a blog post on this topic), and we're planning to add compatibility with more CI providers in the coming weeks.

Now, just click Add CI provider, and you'll now be in your dashboard.

Configure updates for a Git repository

We'll add our first project. Click Add first project, and then select the Git repository to update.

Then, we'll tell Pmbot to update this project on our CI server, and we'll configure the settings so Pmbot can trigger jobs on our CI.

We'll tell Pmbot to update the master branch every day and then click Save.

Finally, we'll click Enable project. The UI now shows us what the next steps are.

Add .pmbot.yml

Pmbot uses a .pmbot.yml to configure package managers and actions. We'll add the following one to our Git repository:

Here, we'll be updating our Npm dependencies with the official npm plugin, but you could use any of the existing package manager plugins or even write your own. At the moment, we've written plugins for Npm, Go and a very basic one for Maven, but we'll add more in the coming weeks.

We're also using an action plugin named auto-merge and we're telling Pmbot to execute it when a project update is successful or partially successful. This plugin is pure greatness: it automatically merges the update branch to the branch from which the update started, and it's fully configurable, so checkout our docs for more info. Checkout the complete list of action plugins for more goodness, we've already written cool things like sending a Slack message or calling a webhook. Best of all, you can write your own action plugins with just a few lines of code !

Update your CI config file

We're using Gitlab CI, but you can find instructions for using Drone CI either in our docs or on a blog post we wrote about it.

We'll update our .gitlab-ci.yml:

• add an update job which runs the pmbot CLI
• add jobs for notifying Pmbot of CI output
• conditionally enable/disable jobs so Pmbot can decide which ones to run

Values for PMBOT_URL, PMBOT_PROJECT_ID and PROJECT_TOKEN are displayed in your Pmbot UI, so this is specific to your Pmbot project.

Also, notice how we control which jobs to run by using an environment variable. That's how Pmbot uses your CI to run updates inside single jobs.

Allow Pmbot to commit and push

Finally, we'll have to allow the pmbot CLI to commit and push to our repository. Generate an SSH keypair with

ssh-keygen -b 2048 -t rsa -f /tmp/sshkey -q -N ""

or click Generate for me in the UI:

You'll need to add the public key as a deploy key in your Gitlab project, and provide the pmbot CLI with the private key. We suggest setting a CI secret environment variable of type file named PMBOT_SSH_PRIVATE_KEY which our CLI will pick up automatically, or you can use the --ssh-private-key option.

Tip: any option of the pmbot CLI can be provided using environment variables (uppercase, underscored, starting with PMBOT_). More info in our docs.

Done !

Now, just click Run first update:

The update will start. It may take a few minutes, as every time Pmbot updates a package, it waits for the commit to run through your CI.

When the update is done, you'll see all the actions executed.

Check your Git history, you'll see a new commit from your beloved Pmbot.

Tip: clicking any commit id will open the corresponding page in your Git server, and clicking any pipeline status bubble will open it in your CI platform.

Conclusion

Pmbot takes a relatively new approach to automated dependency updates. We've tried to make it follow the flow that we used to manually update the dependencies of our projects, and we hope you'll like the way we've made it work.

We're aware that the project setup process is a bit tedious at first, and we'll work on making it smoother and reducing the number of manual steps, but know that once you've added your first project, adding the next ones takes a lot less time as you can copy/paste most of the code. You can also leverage global CI variables to reduce configuration of the pmbot CLI.

We're looking forward to having feedback on this tool, and we hope that it'll change your coding life as much as it changed ours.

Thanks a lot, and happy updates !

Read further:

• official website
• documentation
• blog
• bugs and feature requests
• follow us on twitter

Comments

[Alex Barashkov]

Congratulations with the launch! Very glad to be part of the project.