Git hooks and Docker for Continuous integration - part 2

Simone — Mon, 04 Oct 2021 15:00:24 +0000

Local Continuous integration flow

User branch - pre-push git hook

A CI build process will be triggered by a User pre-push Git Hook. At the end of each build the CI pipeline status will be displayed on the Git repository (if you want, you can even stop user pushing code change if the CI process fails 🛑 )

Main branch - after merge

If the User branch or Pull request is ✅ , we can use Github or Bitbucket api for merging the code/Pull Request.
Once the Pr has been merged we can trigger an automatic CI pipeline on Main/Master branch.

How to run multiple parallel CI pipelines with Docker

First we should know how to create private sub-networks with Docker engine.
In Docker you can create multiple bridge networks.
Containers connected to the same user-defined bridge network effectively can resolve each other.
Using a user-defined network provides a scoped network in which only containers attached to that network are able to communicate.

Docker engine allows you to create multiple networks.
Example: we want max 254 parallel CI pipelines with max 126 containers for each network.

The Docker configuration will be:

"Subnet": "172.20.X.0/24", // 172.20.X.1 - 172.20.X.254
"IPRange": "172.20.X.0/25", // 172.20.X.1 - 172.20.X.126
"Gateway": "172.20.X.12"

where 1 <= X <= 254

Single network configuration:


const availableNetwork = 1;

const networkConfig = {
  "Subnet": `172.20.${availableNetwork}.0/24`,  
  "IPRange": `172.20.${availableNetwork}.0/25`,
  "Gateway": `172.20.${availableNetwork}.12`
}

const networkOption = {
  "Name": networkName,
  "Driver": "bridge",
  "Labels": {
    "name": networkLabel
  },
  "IPAM": {
    "Config": [networkConfig]
  }
}

Article part 1: Git hooks and Docker for Continuous integration - part 1

Check LocalCI: localci.io

Git hooks and Docker for Continuous integration - part 1

Simone — Sat, 02 Oct 2021 19:59:48 +0000

Continuous Integration is a great software development practice, we can reduce integration problems and we can allow teams to develop cohesive software more rapidly.

There are many interesting CI tools in the market but I think the dev CI experience can still be improved. I see companies and dev teams struggling and disappointed when they have to run and setup CI pipelines.

Security:

Most of the IT companies care about code privacy and they don’t like the idea to run their code on external public CI platforms. It's possible to have CI solutions on-premise but most of the time this kind of solutions are expensive for the company. For example, Atlassian has stopped selling new on-prem server licenses. ref.

Build queue and CI debug:

As a developer and member of a tech team I’d like to see our CI builds running immediately after a code change and I’d like to see an easy way for debugging the code when it’s running on CI. Most of the time I see teams and developers complaining about long builds queues and usually it’s even quite difficult debugging the code if an error happens during the CI process.

Cloud costs:

How many times we see commits failing on CI because the code has not been tested enough locally or the code has not been rebased with the main branch yet. Every time we run a build on Cloud it’s a cost for the team and the company. The good news is we can now easily avoid this scenario. Few years ago, Git has introduced "Git hooks", this new technology allows developers to execute pre-tests before a commit will trigger a new CI process in Cloud.

After a long brainstorming with some friends, we created a small CI tool localci.io to address all these issues. This tool doesn’t fit with any IT infrastructure, but this could be useful at least for some team. Probably, in the future it will be completely free but currently we still need to cover some cloud costs.

Article part 2: Git hooks and Docker for Continuous integration - part 1

DEV Community: Simone