DEV Community: Eliise

Pros and cons of using devcontainers in pipelines

Eliise — Thu, 13 Apr 2023 13:39:24 +0000

Context

Given a repository with a local development container aka dev container that contains all the tooling required for development, would it make sense to reuse that container for running the tooling in the Continuous Integration pipelines?

Considered Options

Run CI pipelines in the dev container with a container registry
Run CI pipelines in the dev container via building image locally
Run CI pipelines in native environment

Open questions:

How do the pipeline run time vary between each of these?

Here are below pros and cons for both approaches:

Run CI pipelines in the dev container with image registry

Pros	Cons
Utilities scripts will work out of the box	Need to rebuild the container for each run, given that there may be changes within the branch being built
No surprise for the developers, local outputs (of linting for instance) will be the same in the CI	Not everything in the container is needed for the CI pipeline¹
Rules used (for linting or unit tests) will be the same on the CI	Some tools were installed via a definition in `devcontainer.json`, resulting in a different container from the dev environment²
All tooling and their versions defined in a single place	Some pipeline tasks will not be available ³
Tools/dependencies are already present	Require access to a container registry to host the container within the pipeline⁴
The dev container is being tested to include all new tooling in addition to not being broken

¹: container size can be reduces by exporting the layer that contains only the tooling needed for the CI pipeline. This would require building the image without tasks
²: building dev container with GitHub - devcontainers/ci builds the container with the devcontainer.json. Example here: devcontainers/ci · Getting Started
³: using container jobs in AzDO you can use all tasks (as far as I can tell). Reference: Dockerizing DevOps V2 - AzDO container jobs - DEV Community
⁴: within GH actions, the default Github Actions token can be used for accessing the container registry, see the example below

    - uses: whoan/docker-build-with-cache-action@v5
        id: cache
        with:
          username: $GITHUB_ACTOR
          password: "${{ secrets.GITHUB_TOKEN }}"
          registry: docker.pkg.github.com
          image_name: devcontainer
          dockerfile: .devcontainer/Dockerfile

Run CI pipelines in the dev container with building image locally

Pros	Cons
Utilities scripts will work out of the box	Need to rebuild the container for each run, given that there may be changes within the branch being built
Rules used (for linting or unit tests) will be the same on the CI	Not everything in the container is needed for the CI pipeline¹
No surprise for the developers, local outputs (of linting for instance) will be the same in the CI	Some tools were installed via a definition in `devcontainer.json`, resulting in a different container from the dev environment
All tooling and their versions defined in a single place	Some pipeline tasks will not be available
Tools/dependencies are already present	Building the image for each pipeline run is slow²
The dev container is being tested to include all new tooling in addition to not being broken

¹: container size can be reduces by exporting the layer that contains only the tooling needed for the CI pipeline
²: could be mitigated via adding image caching without using a container registry

Run CI pipelines in native environment

Pros	Cons
Can use any pipeline tasks available	Need to keep two sets of tooling and their versions in sync
No container registry	Can take some time to start, based on tools/dependencies required
Agent will always be up to date with security patches	The dev container should always be built within each run of the CI pipeline, to verify the changes within the branch haven't broken anything

Tools can be Terraform/Terragrunt/linting utilities/unit tests framework. Dependencies are what our Python code will require to run (SDK Purview for instance)

Dockerizing DevOps V2 - AzDO container jobs

Eliise — Fri, 13 May 2022 18:04:10 +0000

This is a follow up to my previous blog post of Dockerizing DevOps, because I've discovered a better way to run containers within AzDO by using container jobs with step targeting:

trigger: none

resources:
  containers:
    - container: devcontainer
      image: $(imageRepository):$(Build.BuildId)
      endpoint: demoacr
      options: --name devcontainer

variables:
  - name: dockerRegistryServiceConnection
    value: "demoserviceconnection"
  - name: imageRepository
    value: "demoacr.azurecr.io/foo-devcontainer"

stages:
  - stage: BUILD
    displayName: Docker build & push
    jobs:
      - job:
        displayName: Devcontainer build & push
        steps:
          - task: Docker@2
            displayName: Login to ACR
            inputs:
              command: login
              containerRegistry: $(dockerRegistryServiceConnection)

          - task: Docker@2
            displayName: Build and push an image to container registry
            inputs:
              command: buildAndPush
              repository: $(imageRepository)
              dockerfile: ".devcontainer/Dockerfile"
              containerRegistry: $(dockerRegistryServiceConnection)
              tags: |
                $(Build.BuildId)
                latest

  - stage: TEST
    displayName: Run tests and linters
    dependsOn:
      - BUILD
    jobs:
      - job: Tests
        steps:
          - task: Docker@2
            displayName: Login to ACR
            inputs:
              command: login
              containerRegistry: $(dockerRegistryServiceConnection)

          - script: |
              make lint
            displayName: "Lint"
            target: devcontainer

          - script: |
              make test
            displayName: "Run all tests"
            target: devcontainer

Dockerizing DevOps

Eliise — Wed, 08 Jul 2020 17:00:13 +0000

In this post I'd like to present how to reuse a devcontainer as the build environment for pipelines. A devcontainer is a Docker container used as full-featured development environment. To learn more about what makes devcontainers awesome, see Visual Studio Code and Devcontainers article by my dear friend, Stuart Leeks.

A typical CI pipeline consists of source -> build -> test stages. Both build and test stages have dependencies (on software and configuration) that need to be configured within the pipeline itself. Since these dependencies are already setup in your devcontainer, why not just use that environment to run your build and tests in?

While the examples below use AzDO YAML pipelines, this can be achieved with any CI environment. Lawrence Gripper's AzBrowze is an example of a repository using devcontainer pipelines with Github Actions.

All code snippets can be found in terraform-pester-devcontainer-example repository.

Requirements

Intermediate knowledge of Docker, devcontainers and CI/CD pipelines
Basic knowledge of AzDO YAML pipelines and Makefiles is helpful

About the project

I've created the terraform-pester-devcontainer-example repository to demonstrate devcontainer pipelines. The project consists of:

terraform to provision resources in Azure Devops
tests covering the terraform written in Pester, a powershell testing framework
- More about testing terraform with Pester in my previous post
devcontainer with the development environment

CI pipeline checks

In a CI pipeline for this project we want to:

validate terraform with tflint
run the Pester tests
validate the devcontainer can be built

Classic pipeline

In a classic or standard pipeline, we first install the dependencies in the pipeline and then run our checks:

.azdo/classic-pipeline.yml

 steps:
      # Install dependencies
      - task: TerraformInstaller@0
        displayName: 'Install terraform'
        inputs:
          terraformVersion: 0.12.25

      - task: Bash@3
        displayName: Install tflint
        inputs:
          targetType: 'inline'
          script: |
            curl -L https://github.com/terraform-linters/tflint/releases/download/v0.17.0/tflint_linux_amd64.zip -o tflint.zip
            unzip tflint.zip
            rm tflint.zip
            sudo mv tflint /usr/local/bin \

      - task: PowerShell@2
        displayName: Install Pester module
        inputs:
          targetType: 'inline'
          script: Install-Module -Name Pester -Force -RequiredVersion 4.10.1

      # Run checks
      - task: PowerShell@2
        displayName: Validate terraform
        inputs:
          targetType: 'inline'
          script: |
            terraform init
            terraform validate
            tflint

      - task: PowerShell@2
        displayName: Run tests
        env:
          AZURE_DEVOPS_EXT_PAT: $(AZDO_PERSONAL_ACCESS_TOKEN)
          AZDO_PERSONAL_ACCESS_TOKEN: $(AZDO_PERSONAL_ACCESS_TOKEN)
          AZDO_ORG_SERVICE_URL: $(AZDO_ORG_SERVICE_URL)
        inputs:
          targetType: 'inline'
          script: Invoke-Pester -EnableExit

      - task: PowerShell@2
        displayName: Build devcontainer
        inputs:
          targetType: 'inline'
          script: docker build -f .devcontainer/Dockerfile -t devcontainer .

Build times

Below we can see the time the build took. Installing dependencies locally is very fast, with majority of the time being spent on building the dev container. Looking at this, the best way to save time on your builds it by not using a devcontainer at all! Luckily, there are other benefits that more than make up for this gluttony.

Devcontainer pipeline

Now lets do what we've all been waiting for and convert the above pipeline to now instead run the tasks inside of the devcontainer:

.azdo/devcontainer-pipeline.yml

steps:
      # Build the devcontainer
      - task: PowerShell@2
        displayName: Build devcontainer
        inputs:
          targetType: 'inline'
          script: docker build -f .devcontainer/Dockerfile -t devcontainer .

      # Run checks inside the devcontainer
      - task: PowerShell@2
        displayName: Validate terraform
        inputs:
          targetType: 'inline'
          script: |
            docker run `
              --entrypoint /opt/microsoft/powershell/7/pwsh `
              -v $(System.DefaultWorkingDirectory):/src `
              --workdir /src `
              devcontainer `
              -c "terraform init && terraform validate && tflint"

      - task: PowerShell@2
        displayName: Run tests
        env:
          AZDO_PERSONAL_ACCESS_TOKEN: $(AZDO_PERSONAL_ACCESS_TOKEN)
          AZDO_ORG_SERVICE_URL: $(AZDO_ORG_SERVICE_URL)
        inputs:
          targetType: 'inline'
          script: |
            docker run `
              -e AZURE_DEVOPS_EXT_PAT=$(AZDO_PERSONAL_ACCESS_TOKEN) `
              -e AZDO_PERSONAL_ACCESS_TOKEN=$(AZDO_PERSONAL_ACCESS_TOKEN) `
              -e AZDO_ORG_SERVICE_URL=$(AZDO_ORG_SERVICE_URL) `
              --entrypoint /opt/microsoft/powershell/7/pwsh `
              -v $(System.DefaultWorkingDirectory):/src `
              --workdir /src `
              devcontainer `
              -c Invoke-Pester -EnableExit

Build times

Below we can see that we've shaved off a few seconds, but nothing amazing.

Devcontainer pipeline with caching

As a bonus, we can also cache our devcontainer image between runs to further reduce the time the builds take. Take a look at the YAML below:

.azdo/devcontainer-caching-pipeline.yml

    steps:
      # Initialize caching 
      - task: Cache@2
        inputs:
          key: docker-image | .devcontainer/**
          path: '.dockercache'
          restoreKeys: docker-image
          cacheHitVar: DOCKER_CACHE_HIT
        displayName: Cache docker layers

      # Load the cached image if cache was found
      - task: PowerShell@2
        displayName: Load cached devcontainer image
        condition: eq(variables.DOCKER_CACHE_HIT, 'true')
        inputs:
          targetType: 'inline'
          script: docker load -i ./.dockercache/devcontainer.tar

      # Build the devcontainer
      - task: PowerShell@2
        displayName: 'Build devcontainer'
        inputs:
          targetType: 'inline'
          script: |
            # Create dockercache directory
            mkdir -p ./.dockercache/
            docker build --cache-from devcontainer:latest -f .devcontainer/Dockerfile -t devcontainer .

      # Cache the docker image file if build succeeded
      - task: PowerShell@2
        displayName: Save devcontainer image
        condition: and(succeeded(), ne(variables.DOCKER_CACHE_HIT, 'true'))
        inputs:
          targetType: 'inline'
          script: docker image save -o ./.dockercache/devcontainer.tar devcontainer

      # Run checks in the devcontainer
      - task: PowerShell@2
        displayName: Validate terraform
        inputs:
          targetType: 'inline'
          script: |
            docker run `
              --entrypoint /opt/microsoft/powershell/7/pwsh `
              -v $(System.DefaultWorkingDirectory):/src `
              --workdir /src `
              devcontainer `
              -c "terraform init && terraform validate && tflint"

      - task: PowerShell@2
        displayName: Run tests
        env:
          AZDO_PERSONAL_ACCESS_TOKEN: $(AZDO_PERSONAL_ACCESS_TOKEN)
          AZDO_ORG_SERVICE_URL: $(AZDO_ORG_SERVICE_URL)
        inputs:
          targetType: 'inline'
          script: |
            docker run `
              -e AZURE_DEVOPS_EXT_PAT=$(AZDO_PERSONAL_ACCESS_TOKEN) `
              -e AZDO_PERSONAL_ACCESS_TOKEN=$(AZDO_PERSONAL_ACCESS_TOKEN) `
              -e AZDO_ORG_SERVICE_URL=$(AZDO_ORG_SERVICE_URL) `
              --entrypoint /opt/microsoft/powershell/7/pwsh `
              -v $(System.DefaultWorkingDirectory):/src `
              --workdir /src `
              devcontainer `
              -c Invoke-Pester -EnableExit

Alternatives to caching

An alternatives to caching is using a container registry to save your image, such as the azure container registry or docker hub. Infact AzBrowze is using docker hub for it's caching.

Build times

With cashing we can see that our run times actually increased the first time the build in run:

But on consecutive runs when we are retrieving the cache, it's lower overall.

Build time overview

All the build times side-by-side:

Execute commands in container locally

For troubleshooting or verifying your commands will run in your container, you can also execute them in your local machine:

## Build the devcontainer
docker build -t devcontainer -f ./.devcontainer/Dockerfile .
## Execute commands in the container
docker run `
        -e AZURE_DEVOPS_EXT_PAT=$env:AZDO_PERSONAL_ACCESS_TOKEN `
        -e AZDO_PERSONAL_ACCESS_TOKEN=$env:AZDO_PERSONAL_ACCESS_TOKEN `
        -e AZDO_ORG_SERVICE_URL=$env:AZDO_ORG_SERVICE_URL `
        --entrypoint /opt/microsoft/powershell/7/pwsh `
        # Replace `<host/source/directory>` with the local path to the repo
        -v <host/source/directory>:/src `
        --workdir /src `
        devcontainer `
        # Replace `Invoke-Pester` with your desired command
        -c Invoke-Pester

Pros and cons

Cons

Building a docker image is slower than just installing dependencies
- This con is negated if you're already using and build a devcontainer
An alternative to running your build pipeline in a devcontainer is extracting dependency installation into a script and using that script in the pipeline and devcontainer.

Pros

Consistent, traceable and fully automated creation of the environment (container) by a docker file
Standardized environment to use for all developers
Easy distribution of the container and its updates for all developers
Reduces the effort to maintain the environment in the pipeline through scripts or similar
Having an identical environment for the developers and the pipeline lowers the probability of errors resulting from inconsistent environments (e.g. different Go compiler version)

Epilogue

While I'm clearly biased towards devcontainers I'd love hear what everyone else thinks on the matter. And do let me know if I've missed anything or if something is broken!

Testing terraform with Pester

Eliise — Fri, 03 Jul 2020 16:55:01 +0000

In this post I'd like show how you can create integrations tests for terraform with Pester, a powershell testing framework.

In this example I'll be using the Azure DevOps terraform provider along with the Azure Devops CLI, but these are just examples that can be replaced as needed for your tests.

All code snippets can be found in terraform-pester-devcontainer-example repository.

Environment

Pester 4.10.1
Powershell Core
Terraform
Azure DevOps terraform provider
Azure Devops CLI
OS: Debian Buster

Terraform under test

The terraform below will be creating a project, a repository and a variable group

`main.tf`

provider "azuredevops" {
  version = ">= 0.0.1"
}

resource "azuredevops_project" "test" {
  project_name = "test-project"
}

resource "azuredevops_git_repository" "test" {
  project_id = azuredevops_project.test.id
  name       = "test-repo"
  initialization {
    init_type = "Clean"
  }
}

resource "azuredevops_variable_group" "test" {
  project_id   = azuredevops_project.test.id
  name         = "test-vg"
  allow_access = true

  variable {
    name  = "example_var_name"
    value = "example_var_value"
  }
}

Pester tests

The tests will:

apply terraform
check validity of state against az devops CLI
tear down the deployed infrastructure

`tfIntegration.tests.ps1`

function Get-ResourceState($address) {
    # Get state and resources after `terraform apply`
    $tfState = terraform show -json | ConvertFrom-Json
    $resources = $tfState.values.root_module.resources

    return $resources | Where-Object { $_.address -eq $address }
}

function Get-Project {
    return Get-ResourceState "azuredevops_project.test"
}

function Get-Repository {
    return Get-ResourceState "azuredevops_git_repository.test"
}

function Get-VariableGroup {
    return Get-ResourceState "azuredevops_variable_group.test"
}

Describe "Terraform Deployment" -Tag 'Deploy' { 
    Context "clean tfstate" {

        It "remove tfstate" {
            $tfStatePath = "./terraform.tfstate"
            if (Test-Path $tfStatePath -PathType leaf) {
                Remove-Item $tfStatePath -Force
            }
        }

        It "terraform initialize" {
            terraform init
        }

        It "terraform apply" {
            $tfResult = terraform apply -auto-approve
            $LASTEXITCODE | Should -Be 0
            Write-Host $tfResult
        }

        It "returns state for all resources" {
            Get-Project | Should -Not -BeNullOrEmpty
            Get-Repository | Should -Not -BeNullOrEmpty
            Get-VariableGroup | Should -Not -BeNullOrEmpty
        }

        It "returns a valid repostiory for the repository ID" {
            $repository = Get-Repository
            $repository.values.id | Should -Not -BeNullOrEmpty
            az repos show -r $repository.values.id -p $repository.values.project_id --org ${env:AZDO_ORG_SERVICE_URL}
            $LASTEXITCODE | Should -Be 0
        }

        It "returns a valid variableGroup for the variableGroup ID" {
            $variableGroup = Get-VariableGroup
            $variableGroup.values.id | Should -Not -BeNullOrEmpty
            az pipelines variable-group show --id $variableGroup.values.id -p $variableGroup.values.project_id --org ${env:AZDO_ORG_SERVICE_URL}
            $LASTEXITCODE | Should -Be 0
        }

        It "returns an empty plan when re-run" {
            # Run a terraform plan and check no changes are detected
            # `-detailed-exitcode` will cause the command to exit with 0 exit code
            # only if there are no diffs in the plan 
            # https://www.terraform.io/docs/commands/plan.html#detailed-exitcode
            #
            # If this test fails it shows an issue with the `read` command returning different data between calls.

            terraform plan -out plan.tfstate -detailed-exitcode

            if ($LASTEXITCODE -ne 0) {
                Write-Host "Detected terraform changes:"
                terraform show plan.tfstate
            }

            $LASTEXITCODE | Should -Be 0 -Because "plan should show no changes"
        }
    }
}

Describe "Terraform Destroy" -Tag 'Destroy' { 
    Context "existing tfstate" {

        It "ensure we have an existing terraform deployment" {
            "./terraform.tfstate" | Should -Exist
        }

        It "terraform destroy" {
            $tfResult = terraform destroy -auto-approve
            $LASTEXITCODE | Should -Be 0
            Write-Host $tfResult

        }

        It "clean up terraform files" {
            Remove-Item ./terraform.tfstate -Force
            Remove-Item ./terraform.tfstate.backup -Force
            Remove-Item ./plan.tfstate -Force
        }
    }
}

Run with

Invoke-Pester
# OR
./tfIntegration.tests.ps1

Debug Terraform (Azure Devops) Provider with VSCode

Eliise — Wed, 01 Jul 2020 16:30:46 +0000

Introduction

In an effort to save someone the pain of having to go through what I did when trying to get debugging to work on the Terraform Azure Devops Provider acceptance tests, here's the solution.

Prior to starting I was already able to run, debug the unit tests and run the acceptance tests. This article is only about enabling debugging for terraform acceptance tests.

Update: Checkout the Issues section for how to enable codelens debugging.

Environment

VSCode
A terraform provider
Golang
Golang extension setup and configured

The environment is also set up in the Azure Devops devcontainer. The code below can also be found in the repository.

Set up

Add the launch.json and .env below. Edit the .env file as needed for your terraform provider secrets.

NB: The buildFlags attribute was only needed for Azure Devops provider, for example, the Databricks provider work without it.

launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch a test",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${file}",
            "args": [
                "-test.v",
                "-test.run",
                "^${selectedText}$"
            ],
            "env": {
                "TF_ACC": "1",
            },            
            "buildFlags": "-v -tags=all",
            "showLog": true,
            "envFile": "${workspaceFolder}/.env"
        }
    ]
}

.env

AZDO_PERSONAL_ACCESS_TOKEN=<your_token>
AZDO_ORG_SERVICE_URL=<your_azdo_org>

Make sure Launch a test is selected in the VSCode debug window

Debug a test

Highlight the name of the test you wish to run test in the test file and press F5 or select Start debugging.

Debugging in action

Issues

The only issue with this solution is that you have to highlight the test name and press F5 instead of being able to just select the debug test option above each test in VSCode. You're able to run non-integration tests via that option and run all tests via the corresponding run test option.

~~I'd love it if anyone could share a solution where they've got it to work with that.~~

Fixed: Enable codelens debugging

The missing piece to enable codelens debugging is to add the below flags to go.testFlags in .vscode/settings.json:

`settings.json`

{
    "go.testFlags": [
        "-v",
        "-tags=all",
        "-args",
        "-test.v"
    ],
}

Here's the same code in our project.

Huge thanks to Thomas Meckel for figuring this out!

Credit

Inspired by https://blog.gripdev.xyz/2019/09/12/easily-debugging-terraform-provider-for-azure-in-vscode/

PowerShell Core as default shell on a Debian devcontainer

Eliise — Wed, 15 Apr 2020 11:35:28 +0000

Introduction

Here we'll cover setting up powershell on a dev container with a debian:buster baseimage.

At the bottom of this article you can also find the full devcontainer.json and dockerimage, which you can skip to.

Credit to: https://www.phillipsj.net/posts/powershell-as-default-shell-on-ubuntu/

All code snippets can be found in terraform-pester-devcontainer-example repository.

Installing Powershell 7

Here's the PowerShell install snippet from our debian dockerfile.

# Install PowerShell 7
RUN wget https://packages.microsoft.com/config/debian/10/packages-microsoft-prod.deb \
    && dpkg -i packages-microsoft-prod.deb \
    && rm packages-microsoft-prod.deb \
    && apt-get update \
    && apt-get install -y powershell \

Set PowerShell as default shell

Next to set powershell as our default shell we must find it in the list of available shells with:

$ cat /etc/shells
# /etc/shells: valid login shells
/bin/sh
/bin/bash
/bin/rbash
/bin/dash
/usr/bin/pwsh
/opt/microsoft/powershell/7/pwsh

The last item in the list is the PowerShell shell location, which we need to use in our devcontainer.json file to set it as our default shell.

"settings": {
        "terminal.integrated.shell.linux": "/opt/microsoft/powershell/7/pwsh",
    },

Optional PowerShell profile set up

If you want to be able to customize your PowerShell like you would with bash, such as to add aliases, you can set up a profile using the below.

# Powershell customization
RUN \
    ## Create PS profile
    pwsh -c 'New-Item -Path $profile -ItemType File -Force' \
    ## Add alias
    && pwsh -c "'New-Alias \"tf\" \"terraform\"' | Out-File -FilePath \$profile"

Complete files

Dockerimage

FROM debian:buster

# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive

# Configure apt and install packages
RUN apt-get update \
    && apt-get -y install --no-install-recommends apt-utils 2>&1 \
    # Verify git, process tools, lsb-release (common in install instructions for CLIs), wget installed
    && apt-get -y install git procps lsb-release wget \
    # Install Editor
    && apt-get install vim -y \
    # Install PowerShell 7
    && wget https://packages.microsoft.com/config/debian/10/packages-microsoft-prod.deb \
    && dpkg -i packages-microsoft-prod.deb \
    && rm packages-microsoft-prod.deb \
    && apt-get update \
    && apt-get install -y powershell \
    #
    # Clean up
    && apt-get autoremove -y \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

# Powershell customization
RUN \
    ## Create PS profile
    pwsh -c 'New-Item -Path $profile -ItemType File -Force' \
    ## Add alias
    && pwsh -c "'New-Alias \"tf\" \"terraform\"' | Out-File -FilePath \$profile"

# Switch back to dialog for any ad-hoc use of apt-get
ENV DEBIAN_FRONTEND=dialog

devcontainer.json

{
    "name": "Debian 10 & PowerShell",
    "dockerFile": "Dockerfile",
    // Set *default* container specific settings.json values on container create.
    "settings": {
        "terminal.integrated.shell.linux": "/opt/microsoft/powershell/7/pwsh",
    },
    // Add the IDs of extensions you want installed when the container is created.
    "extensions": [
        "ms-vscode.powershell"
    ]
}

Known issues

These are the issues that I've run into:

PowerShell Core has fewer modules and commands available when compared to Powershell
Remove-Item command has been unusable due to exasperated results with a known issue: https://github.com/PowerShell/PowerShell/issues/8211

Databricks Operator load testing

Eliise — Tue, 28 Jan 2020 10:24:20 +0000

This blog will provide an high level overview of the methodology we're using when load testing, using the Databricks Operator as an example.
The operator allows applications hosted in Kubernetes to launch and use Databricks data engineering and machine learning tasks through Kubernetes.

The environment

In the above simplified architecture diagram we can see:

Locust, the load testing framework we're using for running the test scenarios
Databricks Operator, the service under test
Databricks mock API, a mock API created to simulate the real Databricks for these load tests
Prometheus, gathers metrics on the above services throughout the test
Grafana, displays metrics gathered by Prometheus

The methodology

The steps for this load testing methodology consists of:

Define scenarios
Run a load tests based on a scenario
Create a hypothesis if unhappy with the results
1. Re-run load tests with the changes from the hypothesis
2. Go to step 3
Repeat until all scenarios are covered

Defining scenarios

To begin the load tests, we first need to define test scenarios we wish to consider and the performance level we would like to achieve. These scenarios are the basis for the tests below.

Here are examples of test scenarios used for the operator load testing:

Test Scenario 1:
1. Create a Run with cluster information supplied (referred to as Runs Submit)
2. Await the Run terminating
3. Delete the Run once complete regardless of status
Notes:
- This scenario is designed to test throughput of the operator under load.
- By deleting the Run after it has complete we ensure we keep the K8s platform as clean as possible for a baseline performance.

Test Scenario 2:
1. Create a Run with cluster information supplied (referred to as Runs Submit)
2. Await the Run terminating
3. DO NOT delete the Run object
Notes
- This scenario is designed to test potential impact of the Operator if the Run objects are not cleaned up.
- The operator should still be performant, even when there are a potentially large number of objects to manage
- This test will also help us understand the acceptable stress limit of the system

Running a scenario

To run a scenario, we'll start by making the load test environment as static as we can to control as many variables as possible between runs. For the operator we achieved this by using automated deployment scripts, code freezes and documenting the images used for each load test. Here's a snippet of the deployment script using specific image tags:

# The following variables control the versions of components that will be deployed
MOCK_TAG=latest-20200117.3
OPERATOR_TAG=insomnia-without-port-exhaust-20200106.2
LOCUST_TAG=latest-20200110.7
LOCUST_FILE=behaviours/scenario2_run_submit.py

Baseline

We document the state of environment before load tests, as seen below. Then proceed with a baseline run, which is the first load test run in a scenario.

Setup
Components
MOCK_TAG=latest-20191219.3
OPERATOR_TAG=metrics-labels or baseline-20191219.1 #See note on Run 1
LOCUST_TAG=latest-20191219.1

Locust
Users: 25
Time under load: 25mins
Spawn rate: 0.03 (1 every 30 secs)

After the run is done, we document what has happened, for example: the state of Grafana graphs, tests we've run if an issue was highlighted, key points discovered.

An example of the metrics:

From the above metrics we discovered these key points:

Run summary
- Run completion time increasing, which is a static value set at 6 seconds, indicating there is a issue with handling the load somewhere
- Requests to the MockAPI are decreasing and are in a spaced out pattern

Hypothesis

Based on the summary we hypothesized the problem is with the operator. The MockAPI is receiving fewer requests as the load increases, meaning the operator is struggling to process the amount of requests.

This lead into an investigation into the operator, where we saw the time.sleep operation is in use. Based on this discovery, we created a fork of the operator and replaced the usage of time.sleep.

The fix for time.sleep can be found here:

Remove time.Sleep from run controller delete #141

stuartleeks posted on Jan 06, 2020

After investigating #140 we found a time.Sleep call for 15 seconds in the run controller when deleting a run.

The original flow was to cancel the run, sleep for 15 seconds and then delete the run.

This PR modifies the behaviour: if the run is not in a terminal state then call cancel and requeue for 5s time, if the run is in a terminal state then call delete. For runs that are already terminated this removes the wait completely and proceeds with deletiong. For an active run that takes 6s to cancel the flow will be

1st time: get state (running), call cancel and requeue fo 5s time
2nd time (after 5s): get state (not yet cancelled), call cancel and requeue fo 5s time
3rd time (after 10s): get state (now cancelled) and call delete

With this change in place the load test now runs smoothly for 10 mins (until a different issue occurs that we will address in a future PR). The following graph shows the work queue depth remaining stabe and low:

View on GitHub

Testing the hypothesis

We then tested this hypothesis by running a new load test and repeating the steps above. The only difference between this load test and the baseline is the image of the operator fork.

With the fix, we can see below that the issues highlighted above have been solved, but has also revealed another issue of requests failing to be sent from the operator.

Repeat

To continue the cycle we'd create another hypothesis and then based on that hypothesis another load test. This would be repeated until we've reached the performance levels we've deemed acceptable when creating the scenario.

Then reassess the scenarios and repeat this for each scenario.

Conclusion

Thanks to the methodology's rigorousness, it's been very easy to provide evidence for the reasons we need to make the changes and to see the progression from the baseline to the end result.

Add monitoring with Prometheus/Grafana in Kubernetes

Eliise — Tue, 31 Dec 2019 17:47:47 +0000

In this post I'd like to give a short overview on the parts needed to add monitoring for a GO API with Prometheus/Grafana in Kubernetes(K8).

Prerequisites

K8 cluster that you can deploy to locally
Intermediate knowledge of K8, Docker and GO
Basic knowledge of what Prometheus and Grafana is used for

Summary

Create a GO API with Prometheus /metrics endpoint
Deploy API, service, service monitor to K8
Deploy Prometheus to K8
Access Grafana dashboards

Create a GO API with Prometheus `/metrics` endpoint

Here's a simple GO API that'll have an welcome page at / and a metrics page at /metrics, which will display metrics from the API. The /metrics will later be used to by Prometheus Operator to scrape data about the API

package main

import (
    "log"
    "net/http"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

type prometheusHTTPMetric struct {
    Prefix                string
    ClientConnected       prometheus.Gauge
    TransactionTotal      *prometheus.CounterVec
    ResponseTimeHistogram *prometheus.HistogramVec
    Buckets               []float64
}

func initPrometheusHTTPMetric(prefix string, buckets []float64) *prometheusHTTPMetric {
    phm := prometheusHTTPMetric{
        Prefix: prefix,
        ClientConnected: promauto.NewGauge(prometheus.GaugeOpts{
            Name: prefix + "_client_connected",
            Help: "Number of active client connections",
        }),
        TransactionTotal: promauto.NewCounterVec(prometheus.CounterOpts{
            Name: prefix + "_requests_total",
            Help: "total HTTP requests processed",
        }, []string{"code", "method", "type", "action"},
        ),
        ResponseTimeHistogram: promauto.NewHistogramVec(prometheus.HistogramOpts{
            Name:    prefix + "_response_time",
            Help:    "Histogram of response time for handler",
            Buckets: buckets,
        }, []string{"type", "action", "method"}),
    }

    return &phm
}

func (phm *prometheusHTTPMetric) wrapHandler(typeLabel string, actionLabel string, handlerFunc http.HandlerFunc) http.Handler {
    handle := http.HandlerFunc(handlerFunc)
    wrappedHandler := promhttp.InstrumentHandlerInFlight(phm.ClientConnected,
        promhttp.InstrumentHandlerCounter(phm.TransactionTotal.MustCurryWith(prometheus.Labels{"type": typeLabel, "action": actionLabel}),
            promhttp.InstrumentHandlerDuration(phm.ResponseTimeHistogram.MustCurryWith(prometheus.Labels{"type": typeLabel, "action": actionLabel}),
                handle),
        ),
    )
    return wrappedHandler
}

func index(w http.ResponseWriter, r *http.Request) {
    _, _ = w.Write([]byte("GO API is up"))
}

func main() {
    phm := initPrometheusHTTPMetric("go_api", prometheus.LinearBuckets(0, 5, 20))

    http.Handle("/metrics", promhttp.Handler())
    http.Handle("/", phm.wrapHandler("Index", "GET", index))

    port := ":8080"
    print("API running on http://localhost" + port)

    log.Fatal(http.ListenAndServe(port, nil))
}

Deploy API, service, service monitor to K8

YAML file to deploy the API, service and service monitor to K8

apiVersion: v1
kind: Namespace
metadata:
  name: prom-go-api
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prom-go-api
  namespace: prom-go-api
spec:
  selector:
    matchLabels:
      app: prom-go-api
  template:
    metadata:
      labels:
        app: prom-go-api
    spec:
      containers:
        - name: prom-go-api
          image: $PROM_GO_API_IMAGE_NAME
          imagePullPolicy: Always
          resources:
            limits:
              memory: "128Mi"
              cpu: "500m"
          ports:
            - containerPort: 8080
              name: api
---
apiVersion: v1
kind: Service
metadata:
  name: prom-go-api
  namespace: prom-go-api
  labels:
    app: prom-go-api
spec:
  selector:
    app: prom-go-api
  ports:
    - port: 8080
      targetPort: api
      name: api
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prom-go-api-servicemonitor
  namespace: prom-go-api
  labels:
    app: prom-go-api
spec:
  selector:
    matchLabels:
      app: prom-go-api
  namespaceSelector:
    matchNames:
      - prom-go-api
  endpoints:
    - port: api
      path: /metrics

Create a local docker image of the GO API and replace IMG with the docker image name to add the above YAML to K8

# Replace ${IMG} with your local docker image name
cat ./manifests/deployment.yaml | PROM_GO_API_IMAGE_NAME=$IMG envsubst | kubectl apply -f -
kubectl apply -f ./manifests/service.yaml

Deploy Prometheus to K8

Install the prometheus-operator helm chart

# prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false means all serviceMonitors are discovered not just 
# those deployed by the helm chart itself
helm install prom-test-api stable/prometheus-operator --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

Check it's running

kubectl port-forward service/prom-azure-databricks-operator-grafana  8090:80 --namespace="default"

Access Grafana dashboards

Access Grafana on http://localhost:8090 with the credentials below

Username: admin
Password: prom-operator

Create charts with metrics such as

increase(go_api_requests_total[1m])

DEV Community: Eliise

Pros and cons of using devcontainers in pipelines

Context

Considered Options

Run CI pipelines in the dev container with image registry

Run CI pipelines in the dev container with building image locally

Run CI pipelines in native environment

Dockerizing DevOps V2 - AzDO container jobs

Dockerizing DevOps

Requirements

About the project

CI pipeline checks

Classic pipeline

.azdo/classic-pipeline.yml

Build times

Devcontainer pipeline

.azdo/devcontainer-pipeline.yml

Build times

Devcontainer pipeline with caching

.azdo/devcontainer-caching-pipeline.yml

Alternatives to caching

Build times

Build time overview

Execute commands in container locally

Pros and cons

Cons

Pros

Epilogue

Testing terraform with Pester

Environment

Terraform under test

main.tf

Pester tests

tfIntegration.tests.ps1

Run with

Debug Terraform (Azure Devops) Provider with VSCode

Introduction

Environment

Set up

Debug a test

Debugging in action

Issues

Fixed: Enable codelens debugging

settings.json

Credit

PowerShell Core as default shell on a Debian devcontainer

Introduction

Installing Powershell 7

Set PowerShell as default shell

Optional PowerShell profile set up

Complete files

Dockerimage

devcontainer.json

Known issues

Databricks Operator load testing

The environment

The methodology

Defining scenarios

Running a scenario

Baseline

Hypothesis

Remove time.Sleep from run controller delete #141

Testing the hypothesis

Repeat

Conclusion

Add monitoring with Prometheus/Grafana in Kubernetes

Prerequisites

Summary

Create a GO API with Prometheus /metrics endpoint

Deploy API, service, service monitor to K8

Deploy Prometheus to K8

Access Grafana dashboards

`main.tf`

`tfIntegration.tests.ps1`

`settings.json`

Create a GO API with Prometheus `/metrics` endpoint