CICD pipelines: Application Developers perspective

abrarmoiz — Mon, 26 Feb 2024 08:36:55 +0000

This blog presents an alternate view of CICD pipelines with an example of AKS build deployment using GitHub actions.

A simplified view of CI pipeline is to allow developers to relate to and make the CICD pipelines more readable and update or fix any issues quickly in the CICD pipeline. The CI pipeline can be considered as an automated list of steps any developer would follow to get his code built and pushed into a code artifact repo.

Consider a situation where in there is no devops team around or you are just moved to devops team from a development background.

A quick view of the steps a developer would take and their matching automation step Github actions. (A similar action can be found on any other CI/CD tool like Azure Devops)

1. Code Build Phase

Our developer in this first step wants to build the code and save the created code output in a safe location. Previously depending the language the output could be a war / jar / zip file or dll file and it would be saved in versioned format.

Since in this case we deal with a containerized output i.e. a container image. The developer needs to save the image in an image repo like docker hub, ACR or ECR

First the developer needs machine to run the build on it can be a windows or linux machine a VM


runs-on: ubuntu-latest

Next the developer would want to checkout the code


- name: Code checkout
  uses: actions/checkout@v3  
  with:
   ref: ${{github.head_ref}}
   fetch-depth: 0

github.head_ref is a reference to the github repo and branch links saved as a variable in github actions

Developer needs to now authenticate / configure the container registry in our case Azure Container Registry where the code in form of container images would reside


      - name: Login to Azure Container Registry
        uses: docker/login-action@v2
        with:
          registry: ${{ secrets.ACR_URL }}
          username: ${{ secrets.ACR_USERNAME }}
          password: ${{ secrets.ACR_PASSWORD }}

Since we are dealing with credentials of ACR, it would be better saved as a secret.

Refer to this link for using secrets in github.

Developer then would run a build i.e. a docker build and push the created docker image into the aforementioned container registry


- name: Build docker image and push to registry
        uses: docker/build-push-action@v4
        with:
          context: ${{ inputs.PROJECT_PATH }}
          file: ${{ inputs.DOCKERFILE_PATH }}
          push: true
          build-args: |
            build_id = ${{ steps.vars.outputs.sha_short }}
          tags: ${{ secrets.ACR_URL }}/${{ steps.vars.outputs.project_name }}:${{ steps.vars.outputs.sha_short }}
          cache-from: type=registry, ref=${{ secrets.ACR_URL }}/${{ steps.vars.outputs.project_name }}:latest
          cache-to: type=inline

In this action steps is a variable internal to github action used to refer to predefined variables.

With the above list of steps you have a got a basic CI pipeline running. Always remember
CI = checking out code, building / compiling code using the dependencies

We have got an artifact at this stage. A responsible developer however would want to run the test cases to ensure a new PR or code commit doesn’t break the existing functionality.

2. Test Phase

The developer again would need a machine to run the tests it can be a windows or linux machine a VM

runs-on: ubuntu-latest

Developer would again want to checkout the code


       - name: Code checkout
         uses: actions/checkout@v3  
         with:
           ref: ${{github.head_ref}}
           fetch-depth: 0

Developer would need to now set up the local environment to run tests and default test suite to get executed


- name: Execute unit tests
  if: inputs.TESTCASE_INPUT != ''
  defaults:
   run:
    working-directory: ${{ inputs.TESTCASES_FOLDER }}/ ${{inputs.TESTCASE_INPUT}}

In this step **inputs... ** is referred to the variables passed to the github actions

Setup coding language version


- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt

Now execute the tests


- name: Install dependencies
  run: |
   pip install pytest pytest-cov
   pytest tests.py --doctest-modules --junitxml=junit/test-results.xml --cov=com --cov-report=xml --cov-report=html

As this stage we realize why the team lead and the architects have been after us to increase the code coverage. Remember a good code coverage always allows you to catch the errors early. We have heard that multiple times earlier.

3. Code Deploy Phase

Finally, the developer needs to run this application on cloud or on premise. In our case its going to be AKS cluster on azure cloud. The developer pulls out the image pushed into the registry and tries to run it into the compute layer. The compute layer could be Azure web service or a VM here we consider an AKS cluster

Assumption: AKS, DB the application relies on is already setup we deal with only delivery of application code

Developer would again need a VM or machine to deploy the code from

runs-on: ubuntu-latest

Login to a cloud environment in our case an Azure environment


- name: Azure Login
  uses: azure/login@v1
  with:
    creds: ${{ secrets.AZURE_CREDENTIALS }}
    fetch-depth: 0

Developer would need to identify the proper AKS where the application code in form of docker image would reside this is done by setting the AKS context by identifying resource group, subscription group and AKS cluster name


- name: Set AKS Context
  uses: azure/aks-set-context@v3
  with:
    resource-group:
    ${{ secrets.RESOURCE_GROUP }}
    cluster-name: ${{ secrets.CLUSTER_NAME }}

Authenticate into ACR container registry


- name: Login to Azure Container Registry
  uses: docker/login-action@v2
  with:
   registry: ${{ secrets.ACR_URL }}
   username: ${{ secrets.ACR_USERNAME }}
   password: ${{ secrets.ACR_PASSWORD }}

Finally refresh the AKS deployment with the latest version of container image.

This will involve installation of kubectl tool, configuring kubeconfig from the secrets and running the k


- name: Set up Kubectl
  uses: azure/k8s-set-context@v1
  with:
    kubeconfig: ${{ secrets.KUBECONFIG }}
- name: Deploy
  run: |
     kubectl apply -f kubernetes/deployment.yaml
     kubectl apply -f kubernetes/service.yaml

The above step could also be accomplished using helm charts.

The entire CI/CD pipeline therefore can be viewed at a basic level by breaking it up into smaller jobs which are an automation of a developer’s day to day tasks of code build, running unit tests and deployment to a cloud server instead of running on his laptop.

Each of the main sections could be considered as a github action job and every developer activity we went through is a step. Different implementations of CI/CD may call each step with a different name i.e. a job, step, trigger can be referred and have syntactically different ways of representation in different tools. Refer to this link for a comparison of different CICD tools link

However, our pipeline is not production ready! The major difference between these steps and a CICD pipeline fit to be used in a production environment would be:

a. Optimizing steps:

Remove duplicate steps between jobs for e.g. code checkout in build and test jobs to be reduced to only one
Add dependencies between stages to ensure one stage is completed before the next one starts. e.g. deploy cannot start until build is completed

b. Adding security scanners to CICD:

Various measures are implemented to ensure the final output from the CI Pipeline is free from vulnerabilities. This is done by scanning code / code artifact by adding a few additional steps in the CI pipeline. This is a quality control applied on the CI pipeline. For this can use the standard github (or any custom CICD) actions to the pipeline which addresses the following –

Static Application Security Testing (SAST) e.g. SonarQube , Sonar cloud
Dynamic Application Security Testing (DAST) e.g. Snyk, ZAP

c. Event based execution:

The execution of the entire pipeline needs to be all glued to an event it could be PR approval or Commit to a branch or Creation etc.,

d. Continous delivery first and then continous deployment:

In the real world the deploy stage evolves first into a human monitored continous delivery and if the processes mature enough then it moves onto continous deployment something which devops teams would need to ponder over.

An overall view of our jobs in form of a CICD pipeline after implementing all the CICD best practices

Hybrid CI CD: Cloud based CI and On premise CD Solution

abrarmoiz — Mon, 08 Mar 2021 14:01:15 +0000

Use of (cloud-based) Azure Devops CI tool with (on premise) Octopus deploy CD tool. This blog explains steps to move towards a cloud-based solution from an on premise one.

Follow steps below to implement this solution:

Use of on-premise agents: The concern with this implementation was the deployment would happen on premise servers using Octopus. To implement this solution we used the “On-premise” Agents. The type of agents, communication and authentications steps are all discussed in https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser

Setting up default agent: Ensure the user has sufficient permissions and identify a suitable littler on-premises agent. Then follow the steps in https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops to set up the agent.

Azure Devops Plugins: Following plugins are installed :
a. Red gate plugin to facilitate Integration with Redgate SQL Change Automation (https://marketplace.visualstudio.com/items?itemName=redgatesoftware.redgateDlmAutomationBuild)

b. Octopus plugins for deployment (https://marketplace.visualstudio.com/items?itemName=octopusdeploy.octopus-deploy-build-release-tasks)

Setting up of additional Octopus projects: Create additional projects to receive output artifacts from Azure Devops pipeline.

These projects would receive build artifacts from Azure Devops. The additional projects were setup to:
a. Avoid any impact to progress on tickets dependent on Octopus
b. Existing projects in octopus have a Team city specific configuration, this was modified to generic one in

Ensure the code is in a cloud based repository: Either Azure repo / github / gitlab / bitbucket etc..,

Setting up pipelines in Azure Devops: Follow steps to setup azure build pipeline (for e.g. follow https://azuredevopslabs.com/labs/azuredevops/yaml/ to set up a pipeline) however ensure the agent pool in Azure Devops is set to default (as we have defined the agent to be default in step 2) and have the following major tasks in each pipeline:

a. NuGetCommand@2
b. VSBuild@1
c. OctopusPack@4
d. OctopusCreateRelease@4
e. OctopusDeployRelease@4

For Steps c,d and e follow https://octopus.com/docs/packaging-applications/build-servers/tfs-azure-devops/using-octopus-extension which results in packaging/ creating a release and deploying a release in Octopus from Azure Devops

Note : Since we are using Azure Devops as a CI tool we will only be setting up the build pipelines and not the release pipelines

DEV Community: abrarmoiz