DEV Community: Pierre Smeyers

Is CI/CD stuck on the evolution ladder?

Pierre Smeyers — Tue, 31 Jan 2023 19:59:37 +0000

The Theory of Evolution applied to computer science...

Throughout my career as a developer, I have seen many complex areas of computer science gain in maturity, formalize more and more structured notions, allowing us to manage things with a higher semantic level, freeing us from low-level details, allowing us to focus on more value-added tasks and ultimately becoming a mere commodity.

Think of Maven, which first standardized the life cycle of a Java project (sources layout, dependencies management, build, tests, release, ...).
Before that we had ANT or Makefile. The main purpose of these tools was to build the javac command line to build the Java project. They offered a very low level of abstraction.
Even though the underlying Maven and Gradle implementation still produces a possibly complex javac command, you don't have to worry about it anymore. The Java build is no longer a topic today.

The Infra-as-Code tools have also followed the same evolution:

in the beginning, infrastructure provisioning was done with handwritten shell scripts,
then came semi-imperative languages such as Chef or Ansible,
we now have a huge range of IaC tools, where you simply describe the infrastructure you want, and the tool takes care of applying it (iteratively) to your live environments to match what you have specified (Terraform, CloudFormation, Kubernetes, Helm ...).

We could hardly implement Gitops without these latest generation tools.

Front-end development also followed the same path...

There was a time before jQuery (and yes, I was there :)). Back then, you had to manage the DOM manually, struggle with browser-specific APIs and implementations, and deal with under-tooled JavaScript code.
Then jQuery appeared, which brought a first level of simplification. It gracefully abstracted some browser specificities, introduced a more efficient way of handling the DOM, but still needed to be handled, and jQuery offered no framework
Today with Angular, React, Vue.JS and others, you have more mature frameworks, which offer a very high level of semantics to develop your web applications. You hardly see the underlying mechanisms for interacting with browser APIs...

...

So why on earth isn't this (yet) the case with CI/CD tools?
Why are we stuck with tools that still expect us to write low-level commands to build/test/deploy our code?

# run npm lint
npm ci
npm run lint

# run Python unit tests with JUnit output report
pip install .[tests]
python -m pytest --junit-xml=junit.xml tests/

# Terraform plan & apply
terraform init -backend-config prod.tfbackend
terraform plan -var-files prod.tfvars -out prod.tfplan
terraform apply prod.tfplan

# deploy my prod env with kubectl
kubectl apply -f prod.k8s.yml

Over and over again… why?

What would it look like?

What would it look like, that new-generation CI/CD tool I have in mind?

Well, first of all, I wish I could say:

“““
Dear CI/CD tool,

Here's my Git repository,

it contains Angular (frontend) and Python code (backend),

I wish to containerize the Python code with Docker (or even better with Cloud Native Buildpacks),

I'd like to provision my Kubernetes cluster with Terraform,

I planned to deploy my frontend to AWS (S3 + CloudFront),

I planned to deploy my backend to my Kubernetes cluster,

I run frontend acceptance tests with Cypress,

I test my APIs with Postman/newman,

...

So please, do the necessary to:

build and test my code,

analyze my code and scripts (please select the most appropriate tools, I'm no DevSecOps expert),

package my code,

provision my environments,

deploy my code to my environments,

run my acceptance tests

...

oh! by the way, I want to work according to a basic feature/branch workflow ”””

Instead I still have to write those dumb npm ci, python -m pytest and terraform init commands...

Why should we not be satisfied with current tools?

There are lots of topics that current CI/CD tools have no opinion on (and therefore don't offer any solutions):

The different Git workflows.
What is the difference between a pipeline triggered by a commit on a feature branch and a commit on the master branch?
What is the semantic meaning of a Git tag? Is it legitimate to trigger a server deployment on a Git tag?
What mapping is recommended between my environments and my Git branches?
Is there a preferable/more efficient order to sequence its build, unit test, analysis, packaging, deployment, acceptance testing, publish tasks?
How do you balance the need to shorten the developer's feedback loop with the need to ensure code quality and security?
...

The consequences of this lack of opinion are manifold.
At the level of the practitioner first:

Each of the subjects mentioned above is a real subject, which can be implemented correctly or very badly. A bad Git workflow, for example, can have a disastrous impact on a team's productivity. Not trying to provide a solution in the tool and letting each DevOps engineer manage it is the guarantee of very disparate and often questionable implementations from one project to another.
How long does it take to become a good DevOps engineer? Example: the day when a practitioner understands why to use npm ci rather than npm install (or why to commit the poetry.lock file) is an epiphany. But why should each practitioner make this discovery by himself? Why isn't this intelligence coded once and for all in the tool, for everyone (even beginners)?
And even an excellent DevOps engineer wastes a lot of time in each new project redoing the same thing over and over again. Are there enough different ways of building a Java project, running pytest tests, building a Docker image, deploying a static website in an S3 bucket that we can't put it all in common ?

But there is also a major consequence at the level of our entire profession: it is the absolute lack of standardization. Each project has its own CI/CD implementation. And even 2 projects relying on the same CI/CD tool can have very different pipelines.

I believe that the time has come for CI/CD tools to raise the semantic level of the notions they handle, to have an opinion on these questions, and to bring at least proposals (if not solutions).

A first attempt...

If you're using GitLab and GitLab CI/CD, to-be-continuous could help. It's an open source project providing DevOps, ready-to-use, composable, configurable and overridable templates that modestly tries to tackle those questions:

Git workflows: to-be-continuous supports main-branch based workflows and variants (feature-branch, Gitflow)
generic pipeline stages: attempt to standardize and optimize the ordering of CI/CD tasks
DevSecOps promotion & prescription: each to-be-continuous template comes with the major (open source) DevSecOps tools related with the technology
optimized development cycle: how to shorten the developer's feedback loop and ensure code quality and security at the same time
and of course no more low-level code writing ;)

Get started with `to-be-continuous`

A few take-away links to get started:

available templates
understand key notions and concepts (possibly applies to any CI/CD tool)
kicker: an interactive online configurer for your project pipeline
project samples: code projects using and combining to-be-continuous templates into production-like examples

"to be continuous": opinionated GitLab CI

Pierre Smeyers — Wed, 02 Jun 2021 13:07:21 +0000

Introduction

At Orange, we have been using GitLab for several years (since late 2015).
By the way we have several self-managed GitLab servers. Some in Community Edition, others in Enterprise Edition, but the overwhelming majority of developers at Orange manage their code in a GitLab.

GitLab is a fantastic tool, promoting a very modern and ambitious vision of development (collaboration, Merge Requests, DevOps, continuous integration …).
The monthly releases of GitLab are each time a gift that we cannot wait to unwrap to discover the many new toys that the fox firm has prepared for us.

At Orange, for CI/CD too, we've smoothly moved from Jenkins to GitLab CI.
For my part, I started using GitLab CI in 2017 and I was immediately struck by the simplicity and versatility of its architecture (1 job = 1 container, no API to implement, no nightmare upgrading the product and its plugins, decoupling of runners, …). I immediately adopted it and never came back to Jenkins again.
As a result, after 2 years of practice I had acquired some savoir-faire, and I was able to start sharing my expertise and some pieces of GitLab CI code.

Then in January 2019, the include feature was released in the Community Edition (version 11.7.0). It was truly an epiphany.
We could now go much further than just sharing pieces of code between colleagues: we could share ready to use templates…

At that time, I had just got a position of Developer Advocate at Orange, and supporting Orange developers by providing state-of-the-art GitLab CI templates seemed to be perfectly inline with the job description.
I was also an active member of the "Orange Experts: Software" community, it was only natural that I asked some of my fellow colleagues to work together on this topic.

During 2 years, we developed templates, we polished them, we increased our GitLab CI expertise, we also progressed in our semantic representation of what a pipeline should be, the nature of the tasks, the profound differences between CI and CD, Git workflows, …

From the very beginning we designed these templates as a honeypot. On the one hand, an argument to attract newbies who wanted to set up their continuous integration without spending hours on GitLab CI, natively implementing all the killer-features offered by GitLab, and on the other hand a way to accelerate/enforce the adoption of state-of-the-art tools and practices (DevSecOps tools, modern and consistent workflows, speed and frugality of CI pipelines, sureness and completeness of CD pipelines).

In only 2 years, this personal initiative became a recognized inner source project, attracting more and more users and contributors, bringing the community to nearly 1000 members today (June 2021), and about 30 templates available.
The visible effect of this increasing adoption is the beginning of a CI/CD standardization at Orange.

Orange - like many IT companies - is a heavy consumer of opensource, sharing in return what we do well is just obvious.
The logical continuation of this story was going opensource, which happened in May 2021 under the name "to be continuous".

https://to-be-continuous.gitlab.io/doc

To semantics and beyond …

What does to be continuous bring?

I am a big fan of GitLab and GitLab CI. But one shall admit that building your DevOps pipeline from scratch requires a lot of skills and time if you want to take advantage of the many features.

In addition, GitLab CI is a product that is constantly evolving, and each release brings a lot of new things. Integrating these new features over time can represent a significant recurring burden for projects.

And finally, the most problematic point in my opinion is the lack of semantic framework provided by GitLab CI. GitLab CI is mainly a sophisticated scheduler, allowing to define technical tasks and their sequence, but GitLab CI cares little about the nature of these tasks, and does not give any clues as to the "right" way to build a DevOps pipeline. The consequence of this is a lack of standardization from one project to another, from one company to another. This is a criticism that can be formulated against the majority of CI/CD tools today.
I am quite convinced that this will be the next big topic in the CI/CD products market.

In fact, as a veteran Javaist (fancy phrase to admit that I'm not that young anymore), the current situation reminds me of what was the Java build in the pre-Maven era. Back then, we used non-structuring tools such as Make or Apache Ant, each project built up its own build system, adopted its own conventions, its own code and resource files structure, … in short, it was a happy mess, everyone reinvented the wheel, and when joining another project, it was necessary to relearn "how does the build work here?".
Then in 2004, Maven was released (and Gradle 3 years later). For a while there were heated debates between the proponents of standardization and the defenders of expertise and of "I can control how it works", but today it would not occur to anyone to build a Java project other than with Maven or Gradle. If I join a project developed in Java, I will immediately know how files are organized, how the project is built. To summarize: the Java build has become a non-topic.

I think this is what must happen now in the field of CI/CD: tools shall offer a more opinionated framework so that the CI/CD in turn becomes a non-topic .

In short, this is what to be continuous modestly aims: to lay the foundations for a standardization of the CI/CD through semantic modeling, integrating Git workflows, environment management, the nature of the tasks of a pipeline, the differences between CI and CD …

Like Maven or Gradle, to be continuous comes with conventions (default configuration), but remains open to customization nonetheless. Either by overloading configuration variables, or by overwriting YAML code.

Why opensourcing?

GitLab is a widely used product; by many companies and developers (around 2/3 of the market share of on-premise CI/CD products).

At Orange, we needed to develop these templates to start standardizing our CI/CD and systematizing the adoption of state-of-the-art tools and practices. We believe this is a general need.

In addition, DevSecOps tools (static code analysis tools, acceptance testing tools, security tools, compliance tools, etc.) are constantly evolving. Integrating new tools and keeping up with developments requires time and expertise that we hope to find in the opensource community.

Opensourcing to be continuous is also the hope of creating a larger community, promoting our vision and sharing development efforts.

What is in "to be continuous"? How can I start?

For now, to be continuous has 26 templates of 6 kinds:

Build & Test: Angular, Bash, Go, Gradle, Maven, MkDocs, Node.js, PHP, Python
Code Analysis: Gitleaks, SonarQube Packaging: Docker
Infrastructure (IaC): Terraform
Deploy & Run: Ansible, Cloud Foundry, Google Cloud, Helm, Kubernetes, OpenShift, S3 (Simple Storage Service)
Acceptance: Cypress, Postman, Puppeteer, Robot Framework, SSL test
Others: semantic-release

To be continuous is thoroughly documented (basic notions and philosophy, general usage principles, every template also has its own documentation, how to use to be continuous in a self-managed GitLab).

To get started quickly, to be continuous provides an interactive configurer (a.k.a "kicker") that allows generating the .gitlab-ci.yml file simply by selecting the technical characteristics of your project.

Finally, to be continuous exposes several example projects, illustrating how to use the templates in production-like projects, combining multiple templates.

What's next?

There is still lot to be done:

to mature existing but still incomplete templates,
manage uncovered deployment technologies (AWS, Azure, Heroku, …)
integrate new DevSecOps tools (e.g. dependabot)
support new languages (eg Rust)
acceptance tests (eg k6, DAST tools, …)
…

Users and contributors: we're waiting for you!