The GitLab CI Manual: The Textbook You’ve Always Needed — #2 Basic CI Coverage

The second post in the 5-part series will aim to provide a clear and distinct understanding on the following topics:

• The Anatomy of CI Jobs
• Stages in Jobs
• Job Rules & Dependencies
• Job Inputs

Prerequisites

Before kicking things off, a few prerequisites are needed and will be required for the remainder of this series.

• A GitLab account
• A GitLab repository
• A basic understanding on YAML (which can be learned here: https://youtu.be/IA90BTozdow?si=y8MFH7nwBif2RVni)

CI Jobs — The mechanical organs of a pipeline

CI jobs are the mechanical instruments that are used to execute script commands. By script commands, we mean commands normally executed in, say, a *CLI such as cmd, bash, docker, kubectl, mongo. Plain and simple.

Jobs in a CI/CD pipeline contain a job name, along with the script keyword, which facilitates the execution of script commands:

first_script:
  script:
    - echo "This is a script!"
    - echo "These commands run inside this script."

• first_script: The job’s name. Of course, a job can have any name, other than the glaringly-obvious reserved CI keywords that will be discussed much later in this series.
• script: The keyword used to run commands in a *shell environment. This is a very crucial point to remember, and is expanded upon in later sections of this post.

CI Jobs and Shell Environments

Each job in a CI/CD pipeline runs in a shell environment. These environments are non-interactive in nature, meaning that the user cannot interact with them, as they normally would in a regular CLI, such as setting global users for Git repositories using git config --global user.(name|email). Running interactive commands in a job result in the pipeline being stuck, due to the absence of *TTY.

All jobs execute script commands in the same shell environment. There are some script commands that, despite being in the same job, run in different shell environments.

On this basis, we have 3 distinct keywords that dictate:

• The order in which the script commands in a job are run
• The distinction between each script’s shell environment

CI Keyword	Primary Purpose	Same Shell Environment?
before_script	Runs script commands before the main script	Yes
script	Runs main script commands	Yes
after_script	Runs script commands after the main script commands	No

The after_script job will always run after the main script, whether or not the main script or the before_script passed.

Note: Any variables set inside the before_script and script jobs WILL NOT carry over to the after_script job.

Stages in Jobs

Jobs in a pipeline run in-parallel by default. In order to run them sequentially — one after another, that is — we use stages.

Stages in a pipeline allow jobs to run sequentially. A stage can contain a number of jobs, but a job can only be in one stage.

CI/CD pipelines provide two keywords for this concept:

• stages: A global keyword to define stages in a job.
• stage: A job-level keyword to declare a job inside a stage.

# global keyword
stages:
  - build
  - push
  - deploy

build_image:
  # job-level keyword
  stage: build
  script:
    - echo "Building Docker image..."

push_image:
  stage: push
  script:
    - echo "Pushing Docker image to Git Repository..."

deploy_image:
  stage: deploy
  script:
    - echo "Deploying to AWS..."

Note: All jobs in a pipeline run inside the test stage, if stages are not defined in a pipeline.

job_1:
  script:
    - echo "This is job 1."

job_2:
  script:
    - echo "This is job 2."

job_3:
  script:
    - echo "This is job 3."

A few features of stages are provided below

• By default, for a pipeline to advance forward, all jobs in a stage must succeed. If one job fails, the succeeding stages do not run and the pipeline halts, unless explicitly written.
• Stages run sequentially in the order they are defined.
• When stages are declared in a pipeline, every job MUST be included in a stage!
• Jobs in a stage run in-parallel with each other.

Jobs Rules & Dependencies

Rules dictate the overall flow of jobs — when do they start, if they start at all, and are jobs in a stage allowed to fail so that other stages can run.

Dependencies, on the other hand, inject a *DAG-like relationship between jobs. Jobs do not execute unless the required job is successfully executed first.

Dictating the flow of jobs — Job Rules

We can use the rules keyword to inject rules in a pipeline for jobs.

The rules keyword comes with its own set of sub-keywords:

• rules:if
• rules:when
• rules:exists
• rules:allow_failure
• rules:changes

1. Configure when jobs run based on conditions — rules:if

The rules:if keyword adds each job to a pipeline based on the first match found.

If at-least one match from the list of if conditions is found (the first match), the rest are ignored and the job is added to the pipeline.
If no if condition matches, then the job is ignored.

variables:
  allowed_to_run: false

job_with_rules:
  script:
    - echo "This job has rules and will execute 
       based on the first match found to be true."
  rules:
    - if: '$allowed_to_run == true'
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'

In the above example, we have 2 rules on the satisfaction of which, the job is added to the pipeline.

• The first rule is checked and it is found that the condition is not satisfied.
• The second rule is checked, the condition is satisfied, and the job is added to the pipeline.

Note: CI/CD variables will be discussed much later in the series. For now, consider the above example to be intuitively understood.

2. Configure when jobs run based on number of satisfied conditions — rules:when

When the when: keyword is combined with the rules: keyword, it changes the behaviour of the job:

If any ONE if condition is satisfied in a job, the pipeline then evaluates the when keyword.
Example:

deploy_to_aws:
  script:
    - echo "Deploying to AWS..."
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual

• The condition is checked, and is satisfied.
• The job is then configured to run manually in the GitLab UI.

3. Configure when jobs run based on existence of files — rules:exists

The way rules:exists keyword works is — if the specified file (or a path to the file) exists, then the job is added to the pipeline. Otherwise, it will not run.

deploy_to_aws:
  script:
    - echo "Deploying to AWS..."
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      exists:
        - "Dockerfile"
      when: manual # job is required to run manually if Dockerfile exists

    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: on_success  # runs if the file does not exist

4. Configure jobs to allow them to fail — rules:allow_failure

When the allow_failure: keyword is used with the rules: keyword, it modifies the behaviour of the keyword.

If one match from the pool of if keywords is found, the pipeline then evaluates the allow_failure keyword.

test_job:
  script:
    - echo "Testing build..."
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: on_success
      allow_failure: true # test job runs for main branch and is allowed
                          # to fail

    - when: on_success
      allow_failure: false # test job runs on success and (if) fails for
                           # every other branch to fix errors

5. Configure jobs to run based on file changes — rules:changes

Changes made to any file (or number of files) in a repository can trigger a pipeline to run when the rules:changes keyword is used.

The files and their absolute paths specified inside this keyword can be kept track off for any changes made by the developers.

deploy_to_aws:
  script:
    - echo "Deploying to AWS..."
  rules:
    - when: manual
      changes:
        - Dockerfile
        - docker-compose.yml

Injecting dependencies — Job Dependencies

Some jobs depend on other jobs’ completion in order for those jobs to run. This is where job dependencies come into play.

Job Dependencies can be injected in a pipeline using the needs: keyword.

stages:
  - build
  - test
  - deploy

build_stage:
  stage: build
  script:
    - echo "Building Dockerfile..."

test_stage:
  stage: test
  script:
    - echo "Testing build..."
  needs: ["build_stage"]
  rules:
    # for main branch, allow it to fail
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      allow_failure: true

    # for other branches, it will always run
    # allow it to fail for error-checking
    - when: always
      allow_failure: false

deploy_stage:
  stage: deploy
  script:
    - echo "Deploying to AWS..."
  needs: ["build_stage"]
  rules:
    # manual deployment from main branch
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual
      changes:
        - .gitlab-ci.yml

What’s happening inside the example

• 3 jobs were defined — build_stage, test_stage, deploy_stage — inside 3 stages — build, test, deploy, respectively.
• test_stage needs build_stage to execute successfully. It has two conditions — (A) If commit happens on the main branch, then add the job to the pipeline and allow it to fail; (B) If the commit happens on any other branch → job is added to the pipeline → it always runs and is allowed to fail.
• deploy_stage needs build_stage to execute successfully, as well.

The DAG-like dependency injected on these jobs looks like this:

Note: Try using needs: inside rules:. Note the differences you see between static job dependency (standalone needs: )and dynamic job dependency (rules:needs).

Job Inputs

Inputs in jobs allow for typed validated parameters that can be overridden with new values when retrying or manually running jobs.

The main difference between jobs inputs and job variables is:

• Inputs allow for typed checks
• Variables remove hard-coded values from pipelines altogether.

Note: CI/CD pipeline configuration inputs will be explored much, much later in the series.

Inputs are evaluated using the following syntax:

${{ job.inputs.INPUT NAME }}

Job inputs contain other sub-keywords, with the use of which, further input configuration is possible.

• default: The default value provided to each input. All inputs must have a default value.
• options: Optional. Provides a list of allowed options for the input.
• description: Optional. Provides a description for the input.
• type: Optional. Specifies what kind of values it can take — number, string, boolean, array
• regex: Optional. Specifies the regular expression that a input’s value must match. Mostly used for *semantic versioning.

deploy_to_aws:
  inputs:
    # first input
    target:
      default: aws
      type: string
      # 👉 the default value used must be included inside options!
      options: ["aws", "azure", "gcp", "vercel"]
      description: Target Cloud Provider

    # second input
    environment:
      default: production
      type: string
      description: Target Environment

    # third input
    deployed_containers:
      default: 1
      type: number
      description: Number of containers deployed by this job

    # fourth input
    tag:
      default: v1.0.0
      type: string
      regex: '^v\d+\.\d+(\.\d)?$'
      description: Semantic Versioning

  script:
    # small note: this is called pipelining — running multiple commands
    # at once
    - |
        echo "Deploying to ${{ job.inputs.target }}"
        echo "Environment: ${{ job.inputs.environment }}"
        echo "Containers deployed: ${{ job.inputs.deployed_containers }}"
        echo "Tag: ${{ job.inputs.tag }}"
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual

The output of the above job is described below:

$ echo "Deploying to aws" # collapsed multi-line command
Deploying to aws
Environment: production
Containers deployed: 1
Tag: v1.0.0
Cleaning up project directory and file based variables
00:01
Job succeeded

A note before we end things here for this post: Try to decipher the workflow, the behaviour and output of each code block’s examples. It helps in learning pipeline behaviours much quicker and outlining areas prone to errors!