DEV Community: MindaugasLaganeckas

Contributing to open source can be frustrating

MindaugasLaganeckas — Thu, 09 Nov 2023 11:52:46 +0000

Disclaimer

This post is about open source contributions, that happen during business hours and the company is paying for it. Why? Because the company uses open source projects and it is in the company's best interest to get certain/blocking stuff fixed.
This post is not about contributions that happen in people's free time.

Background

I have been a JAVA dev since the start of my career. Started as a desktop application developer in a Danish company, that produces high-end hearing aids for the hearing impaired people. After 7 great years, I switched to another high-tech Danish company, but this time as DevOps engineer. My primary task was to support 400 engineers by providing stable and reliable CI/CD system. This was the time, when my new colleagues introduced me to the Open Source and why it mattered. Great times - I can promise you :) One year into the new job, the company has decided to move from internally hosted Gitlab solution to Github. Yay!!! For our team it meant to support the migration and keep the level of build system at the same or even better performance level. Even though the time has been spent on verifying Github for our organization needs, the true issue came out only after the migration :)

The Problem

We have had many repositories in our organization. More than 1200 actually. Some of them have been built by TeamCity and some of them by Jenkins. There was no overlap between the 2 groups of projects. While all projects in TeamCity have been created by hand (using templates), the projects in Jenkins have been managed by configuration-as-code CasC. Great - it worked well in Gitlab - what could go wrong in Github? The answer is Repository discovery. It is a process performed by build systems to detect projects to build. Suddenly for every Jenkins restart, the repository discovery took 10+ min. These minutes got introduced into startup of Jenkins and caused a series of failures. Just to name a few: timeouts, health check problems, tests failing to start (yes, we validated every Jenkins installation), memory issues. Unfortunately, we could not silently ignore the problem and wait for the fix to appear. Engineers in our organization wanted new Jenkins features, bugfixes. We had to regularly update plugins, resize disks etc and therefore restart Jenkins. The life was not easy.

The Solution

Well, we thought, let's find the plugin responsible for Github repository discovery in Jenkins and fix the problem. Ok. That is not hard. Few days later we came up with the fix. The improvement is huge: the repository discovery went from 10+ min to 2+ min. You might think that the whole community has been waiting for the update, right? Or ...

Here is what happened next

Unfortunately, and to our biggest surprise, no one from the project maintainers was really interested in reviewing the change and get it released. Here is what happened next (look at the dates!):
Oct 1, 2020 Pull request (PR) published
Oct 2, 2020 First ask for a review from the maintainers
Oct 22, 2020 github-source-branch plugin comments on the PR
Oct 22, 2020 First review received
Oct 27, 2020 Review feedback addressed
Oct 27, 2020 First reviewer asks for a second pair of eyes
Jan 11, 2021 Approval for PR received
Sep 7, 2021 A ping to project maintainers
Sep 13, 2021 Third reviewer joins the conversation
Sep 13, 2021 Merge issues detected in the PR
Sep 20, 2021 Merge issues solved
Oct 4, 2021 Changes approved and merged
Dec 27, 2021 A new version of plugin released

What happened behind the scenes

16 releases happened between Oct 1, 2020 (the day that we published the fix) and Dec 27, 2021 (the day, when the fix was made available to everyone). Meaning we had to keep the fork of the project updated: integrate new changes, build and deploy the new version of the plugin. Everyone who has ever tried to keep Jenkins plugins updated in your Jenkins installation, knows what a mess it can be: plugins, that depend on other plugins. Oh, and do not forget transitive dependencies, that has to match. Well, should I have invested into automation? Maybe yes. Except, that my expectation was "I am very close. It will get the fix released soon."

How will I go about contributing to open source next time

Well, myself being an open source project maintainer, I understand, that the issue is in man power. And that is pretty much it. It takes time to correspond, review and release. The only thing, that I would like to improve next time is to align expectations with the project maintainers and as early as possible get in touch with them. Normally open source projects have CONTRIBUTING.md or a public slack channel to get started. As the last resort I would create an issue in the project repository and present my ideas there. Basically I would like to get a feel of:

How active the project is
How busy the people behind the project are
How much my feature is relevant/important for the project etc

This information would help to keep things more transparent with my colleagues and my boss and potentially invest into automation while waiting for the fix to get released.

Acknowledgments

I would like to thank Rasmus Jelsgaard for encouraging to write this post and Sameena Syed for proof-reading.

DevOps development guidelines

MindaugasLaganeckas — Thu, 23 Sep 2021 11:54:10 +0000

Motivation

2 years ago I have switched from software development to DevOps in a Danish high-end technology company. The team, that I had joined, consisted of 5 very talented engineers, who delivered build agents, pipelines, automation scripts etc. to the whole R&D of more than 400 hardware and software engineers. Joining this talented and ambitious DevOps team back then, it seemed that there is no limit to what we can achieve. We could see so many new good solutions out there to ease our customers` day-to-day hurdles. Unfortunately, quite fast we had noticed, that adding more technologies to the stack made us vulnerable to bus-factor and stretched us quite thin. In addition, we had no consensus of where to keep secrets that made it hard managing them, shared pipelines implemented by us, were hard to debug/run on a dev machine, the number of self hosted solutions grew rapidly requiring more and more time to attend them... So we had started working on development guidelines for the team, which we called Katas back then.

Development guidelines

Here are some of the guidelines that we would like to share with you. The idea with these guidelines is to minimize the impact on the organization and deliver high quality tools and infrastructure while keeping a high pace for it. Please, leave a comment, if you have similar experiences or more to add.
The development guidelines are not listed in a prioritized order.

Programming languages

Limit the number of programming languages

It is so fun to solve a specific problem with the best tool there is. It is not fun at all to maintain solutions that use technologies you do not know. In the start we used Groovy, NodeJS, Powershell(Core), DotNet(Core), Bash, Java, Python. Unfortunately, we have some solutions now that we have stopped maintaining and reject any new feature requests, because we do not have the right competencies and no time to acquire them.

How to choose programming languages

For us the deciding factor in this case is our approach to inner source all what we develop. We want to use the same programming languages as other engineers in the company. All our scripts, tools etc are open to contribute for everyone inside the company. We want people to help us out instead of being blocked by us. And that works quite well! :)

Design principles

Always prefer a cloud-based solution

It is so easy to download a docker image, create a VM for it, deploy and use it right away. Especially if a solution is free of charge. No budget negotiations, no approvals needed. Unfortunately, the maintenance of such VM/docker combo should not be underestimated. Your users will ask for new versions of the SW, your IT department will require to comply with the company's security policy. And it will happen when you have the least time for it.

CI/CD must be able to push/release/execute what you are building, not just locally.

How many times have I heard: it works on my computer so I am done, am I? NO. You are done, when you have a pipeline for your project.

Design so that local builds and CI builds work the same way

This one is tricky. The more patches/hack you will add to the pipeline to compensate a faulty tooling, the more time will you have to use when debugging the local builds that do not work.

The image above shows a build stage of a DotNet project in a Jenkins pipeline. A developer will have to use time and effort before she configures local IDE to work the same way as the pipeline.
All in all, you want to develop your pipelines/tooling so that it is easy to setup in the IDE. In the example above a simplification and integration in dotnet build would be a desired solution.

Deployment strategies

This section is mostly relevant to the organizations, that are larger or has a similar size, that I work for. While updating 2 build agents can also be challenging, still the impact of updating 40+ build agents has a much larger impact. In the latter case, the experience suggests to use Canary deployment strategy, when introducing new changes to the VM templates for your build agents. Normally, you do not know how people are using the build machines you provide, what kind of projects they are building. Introducing a change slowly minimizes the impact to the organization.

Open source vs Inner source

Choose to contribute to an open source project instead of developing your own custom solution. Maintenance is a killer. Your team will have to learn your solution. Normally a community of an open source project is much larger than your team. Naturally, there are several risks (worth considering!) associated with this approach:

Your pull requests can get stuck for a year or longer
It can take longer time to develop a contribution than a local hack. But you will get your code reviewed from someone knowledgeable in the field and you will learn something - no doubt!

Managing secrets

Do not distribute your secrets across build systems, environment variables on the deployment machines and other places. It is a mess! Github does not even let you read the secret back, once it is put there. Keep the secrets in one (or several) place and manage access to them.

Monitoring and telemetry

Working without monitoring and telemetry is like walking in the mine field in the dark. You want to stay ahead of the issues, such as servers running out of the disk space, before they hit the organization. In addition, you want to spend the time and energy on the tools and pipelines, that are being used most. Rich telemetry and monitoring data can help you to take decisions, where to put your focus most.

Acknowledgements

I would like to thank Rasmus Jelsgaard and Vaida Laganeckiene for inputs, ideas when materializing development guidelines and proof-reading when working on this post.

A pipeline dashboard for multi repository builds with GitHub Actions

MindaugasLaganeckas — Sun, 10 Jan 2021 15:44:34 +0000

What I built

A pipeline dashboard for multi repository builds with GitHub Actions

Category Submission:

Built for Business

App Link

https://pipeline-graph-viewer-7oa6n.ondigitalocean.app/

Screenshots

Description

Please, see either https://pipeline-graph-viewer-7oa6n.ondigitalocean.app/
or
https://github.com/MindaugasLaganeckas/pipeline-graph-viewer

Link to Source Code

https://github.com/MindaugasLaganeckas/pipeline-graph-viewer

Permissive License

MIT

Background

An actual problem, that we face in our organization.

How I built it

I have learned a lot about ReactJS and its integration with DotNet 5.

Additional Resources/Info

None

Create Pull Request (only)

MindaugasLaganeckas — Wed, 02 Sep 2020 12:43:38 +0000

Create Pull Request against another or the same repository. There is no extra functionality. If the Pull Request already exists, the action exits without failing.

    - name: Make a PR
      uses: MindaugasLaganeckas/create-pull-request@v1
      with:
        token: ${{ secrets.SUPER_SECRET }}
        owner: ${{ env.OWNER }}
        repo: ${{ env.REPO }}
        headBranch: ${{ env.DEST_BRANCH }}
        title: "Install Release Drafter"

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

https://github.com/mindaugaslaganeckas/create-pull-request/

Get Default Branch Name

MindaugasLaganeckas — Tue, 01 Sep 2020 10:49:13 +0000

Get the default branch or an empty string, if there are no branches in the repository.

You can call the action in your workflow like this:

    - name: Get default branch name
      id: defaultBranchName
      uses: MindaugasLaganeckas/get-default-branch@v1
      with:
        token: ${{ secrets.SUPER_SECRET }}
        path: ${{ env.OWNER }}/${{ env.REPO }}

You can access the output like this:

    - name: Change to master branch if repository is empty
      if: ${{ steps.defaultBranchName.outputs.default-branch == '' }}

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

https://github.com/MindaugasLaganeckas/get-default-branch

Percy upload simplified

MindaugasLaganeckas — Fri, 21 Aug 2020 08:42:26 +0000

My Workflow

Wraps percy/agent docker in a Github action. Executes

percy upload

operation. The users of this action does not need to know docker. They get a simplified interface to upload generated images to percy.io for regression testing.

Submission Category:

Maintainer Must-Haves (for image regression testing)

Link to Repository

Percy Upload

Additional Resources / Info

Mermaid CLI uses percy-upload action