DEV Community: Martin Humlund Clausen

Trunk based development: A Teams experiment - From Idea to Practice: Our TBD Workflow

Martin Humlund Clausen — Mon, 25 Aug 2025 07:37:51 +0000

Since my last post, our team has started experimenting with Trunk-Based Development (TBD). The early results are promising, and we’ve taken the time to document our process and agreements.

First things first. Before starting a new way of working, it is super important to get some agreements about responsibilities, and setting expectations. This is an important step in the process.

📋 Team Agreements

Expectation	Details
Responsibility for Quality	You own your code from commit to production impact.
Small, Frequent Commits	Atomic and meaningful changes only.
Tag Commits Properly	- Task related work are always tagged. Use AB# - Scout commits are not necessary to tag
All New Code Must Be Tested	Preferably with unit tests. Test-Driven Development is Encouraged.
Discretion with Breaking Changes	Plan carefully and minimise impact.
Reviews are Non-Judgmental	Focus on the code, not the coder.
Minor Changes by Reviewer	Reviewers may commit small fixes directly. (Let the author know)
Pull Requests are used selectively	- Breaking Changes - Explicit feedback is requested from author - Change require manual test on non-local dev-environment
Everyone Reviews	Regardless of seniority.
Feature Toggles when Needed - (Still gathering knowledge on how to feature-flag)	Hide incomplete features, do not delay merging.

As for the agreements, it’s vital that the whole team agrees on the basic premise.

🚀 Our Workflow - SOP

So, how does this actually look like in practice?

Direct Commits to Main
- Commit directly to main.
- Use short-lived branches only when absolutely necessary. Those can be:
  - Breaking Changes
  - Explicit Feedback is needed (requested via Pull Request)
  - Required for Manual Tests (if no other way of testing the change)
Commit Tagging Requirement
- Format: AB#<TaskNumber> <Commit Message>
- Example: AB#4321 Add validation for payment processing form
Mandatory Test Coverage for New Code
- All new code must be covered by a test, preferably a unit test.
- Use the most appropriate automated test type if unit tests are not practical.
- When changing code in existing codebases with no associated unit tests, create a new unit test class and test that change specifically.
Use Discretion with Breaking Changes
- Avoid breaking changes between services.
- If unavoidable, minimize the impact and number of breaking commits.
- If a commit is breaking, consider doing a minimal Pull-Request (only 1 commit to rollback if environment breaks).
- Communicate proactively with affected developers.
Automated Testing First
- Build checks, linting, static analysis, unit tests.
- (Optional) Deploy to Development if needed.
Post-Merge Non-Blocking Code Reviews
- Author moves task to “To Review” after merging.
- Author requests explicit review feedback as comments in the Pull Request, which must be resolved by the reviewer.
- Reviewer actions:
  - Reviews tagged commits
  - Reviews Pull-Requests
- Reviewers can:
  - Leave comments
  - Make minor fixes directly with a follow-up commit (inform the author)
Act on Review Feedback Quickly
- Critical issues: fix ASAP.
- Non-critical issues: address in follow-up work.
Collaborative Testing
- Code must be well-tested.
- Encourage manual exploratory testing by another team member.

In short: commit often, keep changes small, ensure tests are in place, and use reviews as a collaborative learning tool.

💭 Room for Improvements

We consider this a first draft, and there are areas we want to refine as we iterate:

Feature Flagging Process

While implementing feature flags is straightforward, the challenge lies in managing them over time.
Open questions we need to solve:
- What is the process for introducing and removing feature flags while avoiding long-term technical debt?
- As a sub-team within a larger organization, how do we align with other teams on consistent use and governance of feature toggles?

Continuous Delivery Maturity

Pain points:
- Currently, we are constrained by rigid approval gates across Development and Live environments.
- Shared environments introduce dependencies and bottlenecks.
- Delivery pipelines are standardized across teams, limiting flexibility.
Next steps:
- Explore ways to streamline approvals while maintaining quality and compliance.
- Evaluate opportunities for team-specific delivery pipelines or improved coordination mechanisms across teams.

Knowledge Sharing & Adoption Across Teams

Our team has been able to adopt Trunk-Based Development quickly due to its maturity and seniority. However, other teams may not yet have the same level of experience or comfort with these practices.
Pain points:
- The company has a strong background in open-source ways of working, which creates some hesitancy and skepticism toward adopting different practices like TBD.
- Limited visibility into how our process works for teams outside our own.
- Risk that TBD practices stay siloed within our team instead of being scaled across the organization.
- Uneven skill levels across teams can slow down adoption.
Next steps:
- Build confidence internally by gathering more data and experience with TBD in our own team.
- Document learnings in lightweight, accessible formats (playbooks, demos, internal blog posts).
- Use side-by-side comparisons with familiar open-source workflows to show how TBD can complement, not replace, existing practices.
- Run knowledge-sharing sessions to explain the “why” behind our approach.

This SOP is a living document, and we’ll keep refining it as we go. If you’ve tried Trunk-Based Development in your team, I’d love to hear what worked for you — and what didn’t.

Trunk based development: A Teams experiment

Martin Humlund Clausen — Fri, 18 Jul 2025 13:31:08 +0000

🚧 Challenging the Pull Request Habit: Our Team’s Trunk-Based Development Experiment

Coming from a background rooted in open source, RFCs, and collaborative workflows, our company has a strong culture of development through pull requests. PRs are central to how we collaborate, review, and deliver software.

But recently, I’ve been reflecting on whether this approach still serves us best—especially as a small, high-performing team.

There’s been a growing body of research, especially from Google’s State of DevOps (DORA) and the book Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim, that highlights an interesting trend:

Teams that practice Trunk-Based Development (TBD)—merging small, frequent commits directly to the mainline—consistently outperform in both speed and stability.

This sparked a question for me: What if we tried something different?

🤔 Why Experiment with Trunk-Based Development?

Let me be clear: this isn’t a rejection of PRs. Pull requests are essential in many contexts—especially when introducing potentially breaking changes, managing infrastructure as code, or contributing to open source, or other controlled environments.

They’re valuable tools, and we’re not throwing them away.

But in our case—a tight-knit team of three developers—we’ve noticed some subtle but persistent friction:

PR’s are often merged within a few hours, but they still introduce context switches and unnecessary overhead.
Review comments tend to focus on minor improvements—naming, formatting, etc. Which could often be addressed later or handled differently.
We rarely catch logical flaws with the code during reviews
We still release bugs (albeit minor) into production despite our best efforts
Each PR brings with it process weight: writing the description, tagging tasks, running manual tests, waiting for reviews, and so on.

Individually, these frictions aren’t catastrophic. But together, they slow us down just enough to matter.

So, what if we could maintain quality, increase flow, and reduce friction—all while keeping our high trust and collaboration intact?

My stomach feeling is that when developing new features, you can code 80-90% by isolating specific feature, and only the last percentage is integrating with existing code and services.

🧪 The Experiment: A Shift to Trunk-Based Development

We’re running a TBD experiment over a few sprints. It’s structured, intentional, and driven by data. Our goals are to improve lead time, reduce cognitive load, and boost developer satisfaction.

✅ What We’re Testing

1. Refined Process

Share a developer brief to introduce the experiment and invite team reflection.
Establish team agreements on how we’ll operate.
Pull requests are still being used—but used selectively, especially when code requires manual testing or broader review, or integration.
Tagging of commits with task IDs for traceability, and to support non-blocking reviews.

2. Lean Into CI/CD

Automated testing is mandatory for all new code—preferably unit tests, but anything that makes sense contextually and is automated
Any commit that passes CI is eligible for deployment.
Emphasise tiny, frequent commits and early integration.

3. Non-Blocking Reviews

Reviews are asynchronous and post-merge where appropriate.
Feedback is delivered directly to authors, especially for more significant changes.
Minor improvements can be committed directly to main by reviewer.
The focus shifts from gatekeeping to enabling flow and improvement.
Boyscout commits, such as removing unused namespaces, updating variable names or whatever leaves the campsite cleaner does not need to be part of the review process.

4. Weekly Team Check-ins

Each week we ask:

What friction points have emerged?
Are we moving faster—or slower?
Have we introduced or reduced bugs?
Can we replace manual steps with automation?
How do we feel about the process overall?

📈 Metrics We’re Tracking

Lead time from commit to production
Deployment frequency
Rate of bugs or rollbacks
Team sentiment and confidence in the process

This isn’t a leap of faith—it’s a measured experiment. We’re testing, learning, and adapting.

🌱 The Feeling of Growth

Pull requests have become more than a workflow—they’re a symbol of caution, structure, and trust. In large or distributed teams, they play an essential role.

But in small, high-trust environments, those same rituals can become bottlenecks. And the truth is: what worked five years ago might not be the right fit for who we are now.

This is a moment to reflect, evolve, and improve.

🧭 In Closing

I don’t know yet if Trunk-Based Development will be the silver bullet for us. (Is anything really?)

But I do know this: it’s worth exploring.

We’re not abandoning quality—we’re doubling down on trust, tooling, and transparency.

If you’re in a small team and feel the drag of over-engineered process, maybe it’s time to ask: What are we optimising for?

For me its all about running the experiement, gather the data and the feedback, and if it does not work, the worst thing that happens is that we go back to the “old” ways of working.

Let’s see what happens. I promise to do a followup

Investigating Azure Regional Capabilities

Martin Humlund Clausen — Sun, 22 Sep 2024 08:35:21 +0000

This week, I was tasked with figuring out whether we could support a new region on our company’s hosting platform.

I started by searching the web for information on Azure capabilities in a specific region. Before I knew it, I had spent an hour combing through resources. The only useful document I could find was a huge, 50-page PDF filled with slick layouts, graphics, and all the marketing fluff you can imagine. While the PDF did contain some technical information, I quickly realised it would take forever to compare the resources in the document with those from an existing region we were already using.

It was time to switch up my approach.

I decided to explore the Azure CLI to see if I could query regions and their capabilities directly. Sometimes when you ask the universe for answers, it delivers. After a bit of planning, I came up with the following approach:

First, we need to list the current capabilities in an existing region to get a list of resourceType we require.
Then, we’ll retrieve the provider information for a specific resourceType and check whether it’s available in the target region. For this example, we’ll use UAE North for this example.

Sound fun? Let’s dive in!

List Resources for existing Region

Sign into azure using Azure Cli and switch to the subscription containing your region.

Note: We have architected Azure to have one subscription per region, so its fairly easy for us to manage regional resources and cost associated with a specific market. I digress..

So…

az login
az account set --subscription "<name of your regional subscription>"

Getting a flat list by querying all types of resources, and then sort them by unique entry

az resource list --query "[].type" --output tsv | sort -u

The output the following list

Microsoft.AppConfiguration/configurationStores,
Microsoft.Automation/automationAccounts,
Microsoft.Automation/automationAccounts/runbooks,
Microsoft.Compute/virtualMachineScaleSets,
Microsoft.ContainerInstance/containerGroups,
Microsoft.ContainerService/managedClusters
... etc

Listing regional capabilities

From here I concocted the following powershell script to iterate over existing resource types, and check whether they were available in the target region, Add the result to a list, and then output the result.

# List of resource providers and types for an regional Stamp
$resourceTypes = @(
    "Microsoft.AppConfiguration/configurationStores",
    "Microsoft.Automation/automationAccounts",
    "Microsoft.Automation/automationAccounts/runbooks",
    "Microsoft.Compute/virtualMachineScaleSets",
    "Microsoft.ContainerInstance/containerGroups",
    "Microsoft.ContainerService/managedClusters"
)

# Region to check (Full name of the region)
$region = "UAE North"
$resultList = @()

# Iterate over the resource types and check availability
foreach ($resourceType in $resourceTypes) {
    Write-Host "Checking $resourceType in $region"

    $providerNamespace = $resourceType.Split("/")[0]
    $resourceTypeName = $resourceType.Split("/")[1]

    # Get provider information
    $providerInfo = az provider show --namespace $providerNamespace --query "resourceTypes[?resourceType=='$resourceTypeName']" --output json | ConvertFrom-Json

    if ($providerInfo) {
        $locations = $providerInfo.locations

        $doesExist = $locations -contains $region
        $symbol = if ($doesExist) { "✔" } else { "❌" }

        $resultList += [PSCustomObject]@{
            ResourceType = $resourceType
            DoesExist = $symbol
        }

    }
    else {
        Write-Output "Could not retrieve information for $resourceType"
    }
}

$resultList | Format-Table -AutoSize

The script outputs the following

Name                                               DoesExist
----                                               ---------
Microsoft.AppConfiguration/configurationStores     ✔
Microsoft.Automation/automationAccounts            ✔
Microsoft.Automation/automationAccounts/runbooks   ✔
Microsoft.Compute/virtualMachineScaleSets          ✔
Microsoft.ContainerInstance/containerGroups        ✔
Microsoft.ContainerService/managedClusters         ✔
Microsoft.DocumentDB/databaseAccounts              ✔
Microsoft.Insights/actiongroups                    ❌
...

Conclusion

While we discovered that we were unable to create certain resources in a specific region, it quickly gave us the knowledge needed to provide the business with an assessment of whether we could launch in that region.

On a personal level, I feel like this exercise gave me more insight into the capabilities available through the Azure CLI command. Additionally, I spent roughly 5 hours on the task in total. Next time the business asks if we can launch in a region, we’ll be able to provide an answer even faster.

Productivity hack of the day: Creating pull requests descriptions from Git diff and ChatGPT

Martin Humlund Clausen — Tue, 10 Sep 2024 06:57:31 +0000

As developers, we create a lot of pull requests, and I must admit that sometimes I neglect to write a detailed description to help my team review the changes efficiently.

To assist my team, I use ChatGPT to generate richer pull request descriptions, giving me a little more time to focus on coding.

Here’s the simple trick:

On your current feature branch, run the following PowerShell script from within your root repository folder. It will output a diff.patch file .
Paste that file into ChatGPT and use the following prompt:
- "As a software developer, I’ve made the changes in the attached git diff file. Please write a pull request description for my team to help them review it efficiently, with an emphasis on {Insert your bullet points}."

Here is the gist

# Check if the git command is available
if (-not (Get-Command git -ErrorAction SilentlyContinue)) {
    Write-Host "Git is not installed or not available in the PATH." -ForegroundColor Red
    exit
}

# Ensure we're in a git repository
$gitStatus = git rev-parse --is-inside-work-tree 2>&1
if ($gitStatus -ne "true") {
    Write-Host "This is not a git repository." -ForegroundColor Red
    exit
}

# Get the current branch name
$currentBranch = git rev-parse --abbrev-ref HEAD
if ($currentBranch -eq "main") {
    Write-Host "You are on the 'main' branch. Please switch to a feature branch." -ForegroundColor Yellow
    exit
}

# Fetch the latest changes for the main branch
git fetch origin main

# Get the latest commit on the main branch
$mainCommit = git rev-parse origin/main

# Get the latest commit on the current branch
$currentBranchCommit = git rev-parse HEAD

# Generate the diff between the latest commit on main and the current branch
$diffFileName = "diff.patch"
git diff $mainCommit $currentBranchCommit > $diffFileName

# Check if the diff file was generated successfully
if (Test-Path $diffFileName) {
    Write-Host "Diff file generated successfully: $diffFileName" -ForegroundColor Green
} else {
    Write-Host "Failed to generate diff file." -ForegroundColor Red
}

As a generel rule of thumb, keep your Pull Request small! Not only does that help your team reviewing your changes and it also makes writing Pull request descriptions easier.

Disclaimer

As alway, when using gpt is that you OWN the output, so dont forget to check it.
The output is only for scaffolding a pull request description, so modify the output accordingly

Deploying Terraform Infrastructure to Azure using Azure DevOps Pipelines

Martin Humlund Clausen — Mon, 28 Aug 2023 10:16:59 +0000

In my experience knowing how to deploy a service intro production is as important as knowing the application itself. As Software Engineers we’ll like to manage the risk of deployments, and deploy as often and early as possible in the release cycle without downtime. However, every now and then you have to deploy infrastructure as well, so why not subject it, to the same level of automation as your applications?

This is the Second part of how to use Terraform with Azure, so if you haven’t already go read my first blog entry on how to setup a terraform repository and create the necessary service principles to get your started. Alternatively you can go to the repository on Github see what is there.

What kind of Infrastructure are we deploying?

When working with bigger Cloud Platforms you quickly discover that not all infrastructure are deployed at the same time, and more often than not, we gave contain multiple layers of infrastructure. To drive the point through consider these two layers

Global Infrastructure, this typically includes changes to an Azure Kubernetes Cluster, a Azure Container Registry, or an Azure Service Bus. Global Infrastructure is the shared components that your applications can leverage. Also global infrastructure can be deployed independently of the rest of your application stack.
Service Infrastructure is the the infrastructure that is needed to run your application, this includes Databases, Redis cache etc. And is deployed together with the service deployment.

In our organisation we have a couple of more layers, but lets just keep it simple for demonstration purpose, so for this article, I would like to focus on the global infrastructure, since we its easy to zoom in on the pipeline itself.

What does our deployment pipeline look like?

For this sample pipeline we are going to build two stages

Validate Terraform - First stage we validate that our terraform configuration is correctly configured. This includes Terraform syntax and connectivity to our Azure Backend Store.
Release to Dev - We have some infrastructure to deploy, so lets release it to our development environment

I have chosen to organizing the pipeline in the following way.



├── azure-pipelines.yaml
├── main.yaml
└── stages
    ├── tf-release.yaml
    └── tf-validate.yaml

azure-pipelines.yaml is the entry point for the pipelines. It contains all the triggers, variables and other pipeline controls
main.yaml can be though of an orchestrator of stages. It contains a list stages that we would like to execute
tf-release.yaml is our terraform release stage, that is responsible for doing the actual release of terraform. Note here that we do not denote the environment as to which we would like to deploy. We can simply reuse the stage and give it other parameters on runtime.
tf-validate.yaml is or Terraform Validation Stage

Deploying your infrastructure

All the magic happens in the tf-release.yaml. Here we basically do the exact same things, as if we had run the command locally.



terraform init
terraform plan
terraform apply

Starting at the file, we define three variables

tf_version - We need to parse in the terraform version into the release stage to ensure that our whole pipeline is using the same version of terraform. Since we use terraform in multiple pipelines this variable is defined in the azure-pipelines.yaml and parsed through all stages
working_directory - this will be our base directory where we are executing commands
environment - is the name of the environment. We can use this to do some magic display names, or reference azure pipelines variables using naming conventions.

Additionally I will be pulling in, the variable group that we created in the to get the service principle along with the access key to our storage account, plus three additional variables to help with location of our terraform artifacts.



parameters:
  tf_version:
  working_directory:
  environment:

stages:
  - stage: release_${{ parameters.environment }}
    displayName: Release to ${{ parameters.environment }}
    variables:
      - group: root-terraform-backend-credentials
      - name: backend_tfvars
        value: "${{ parameters.working_directory }}/tf/backend/backend.ci.tfvars"
      - name: variable_tfvars
        value:  "${{ parameters.working_directory }}/tf/environments/${{ parameters.environment }}/variables.tfvars"
      - name: infrastructure_workingdirectory
        value:  "${{ parameters.working_directory }}/tf"

Install Terraform on the build agent

Until we have explicitly installed terraform on our build agent, we’ll not be able to execute any Terraform. This is however pretty straight forward. Though you might have to install Terraform Extension into your Azure DevOps Organization



- task: TerraformInstaller@0
  displayName: "Use Terraform ${{ parameters.tf_version }}"
  inputs:
    terraformVersion: ${{ parameters.tf_version }}

Terraform Init

When initialising terraform we must provide it the initial backend file that we would like to use. We first have to replace the tokens in our backend.ci.tfvar file. You might also have to install Replace Tokens Extension into your Azure DevOps Organization.

After we have replace the token, we can proceed to initializing terraform



- task: qetza.replacetokens.replacetokens-task.replacetokens@3
  displayName: "Replace tokens in backend.tfvars with variables from the CI/CD environment vars"
  inputs:
    targetFiles: $(backend_tfvars)
    encoding: "auto"
    writeBOM: true
    actionOnMissing: "fail"
    keepToken: false
    tokenPrefix: "#{"
    tokenSuffix: "}#"

- bash: |
    terraform init -backend-config=$(backend_tfvars) -input=false
  env:
    ARM_CLIENT_ID: $(sp_clientId)
    ARM_CLIENT_SECRET: $(sp_clientSecret)
    ARM_SUBSCRIPTION_ID: $(sp_subscriptionId)
    ARM_TENANT_ID: $(sp_tenantId)
  displayName: Initialize configuration
  workingDirectory: $(infrastructure_workingdirectory)
  failOnStderr: true

See the -input=false flag at the end? This tells Terraform that it should not prompt for input, and is important not to miss when running in a CI environment.

Terraform Plan

Terraform plan, is the action of making the change set for our infrastructure. For this command we are parsing providing a -out parameter which is a mechanism to store that change set to a file on disk, and since this is running in a single build agent job, we can use that file to apply the infrastructure in a later task.

💡 You can export the terraform plan file and use it in another stage. This is especially useful if you want to add an additional approval step, or inspect the file manually.




bash: |
terraform plan -var-file=$(variable_tfvars) -input=false -out=tfplan
env:
ARM_CLIENT_ID: $(sp_clientId)
ARM_CLIENT_SECRET: $(sp_clientSecret)
ARM_SUBSCRIPTION_ID: $(sp_subscriptionId)
ARM_TENANT_ID: $(sp_tenantId)
displayName: Create execution plan
workingDirectory: $(infrastructure_workingdirectory)
failOnStderr: true

Terraform Apply

Last thing we need to do is to apply the changes. For this example, we parse the -auto-approve flag, which will just apply the changes that we have, without prompting us. Additionally we are using the tfplan that we created in the previous task.




bash: |
terraform apply -input=false -auto-approve tfplan
terraform output -json > output.${{ parameters.environment }}.json
env:
ARM_CLIENT_ID: $(sp_clientId)
ARM_CLIENT_SECRET: $(sp_clientSecret)
ARM_SUBSCRIPTION_ID: $(sp_subscriptionId)
ARM_TENANT_ID: $(sp_tenantId)
displayName: Apply execution plan
workingDirectory: $(infrastructure_workingdirectory)
failOnStderr: true

Conclusion

In my previous article Getting Started with Terraform and Azure, we started by setting up the the initial terraform, which would serve as the foundation for constructing a Cloud Platform using Terraform.

For this article we have gone through how to apply the infrastructure using Azure Pipelines.

This is a very basic example, and I am sure that you need to do adjustments to fit your development flow. However here are some ideas, that you can consider for your next project

Have the pipeline add an additional approval step before applying the infrastructure structure
Have the multistage Azure Pipeline, Plan your live environment, after you have deployed to Development and see how the change will affect your live environment, before approving it.
Configure a web hook that tells your team what changes are being deployed.

You can find the full code sample on Github, with more examples to come.

References

Github Repository: Cloud.Platform.Foundation
Azure DevOps Extension: ReplaceTokens
Azure DevOps Extension: Terraform

Getting Started with Terraform and Azure

Martin Humlund Clausen — Sat, 19 Aug 2023 09:23:01 +0000

At the heart of any Cloud Platform is Infrastructure, and how we manage infrastructure in large-scale systems has a direct correlation with how effective your development teams are when deploying, securing, and auditing your infrastructure.

For this article, I will go through the simple steps for setting up Terraform and connecting the state to an Azure Blob Storage Account.

A huge disclaimer here is that I am possibly suffering from a severe case of imposter syndrome and that I am treating grounds that so many developers have walked the path before me. There is notes on how to get started, and I hope that you find them useful.

Prerequisites

We are of course using some command line magic to get us started, so before you go any further make sure to have the following installed command line tools on your dev box

Azure CLI
Terraform
An Azure Subscription with Contributor rights, and access to Azure DevOps

The Challenge

Infrastructure as Code is an addiction. You discover that all that complicated stuff, that you have used to setup manually either on Amazon, Azure, or even Cloudflare, can be coded and managed via Infrastructure as Code, you have an unlimited amount of terraform providers, and you discover that it supports everything that you are trying to build! That is freaking awesome!

However, you need to start somewhere and you cannot start using Terraform and not have either a state file locally, or somewhere else. That means, that there is a small amount of infrastructure that you still have to manually create in order to get started.

Since this is an Azure tutorial, this tutorial will cover setting up the foundational stuff that you need in order to execute Terraform from a CI/CD Pipeline.

An Azure Subscription that will contain all the configured resources, that we need to get started
A Service Principle with Contributor rights to the Subscription
A Manually Created Storage Account to Store the Terraform state
An Azure DevOps variable group, to store the service principle and the Storage Account Access Key

Creating Service Principle

In order to execute our terraform you need to be able to access your Azure Subscription with Contributor access rights.

If you want to work with Terraform locally it is enough to log in via the Azure CLI

az login

And switch to the Subscription that you are currently working with

az account set --subscription <SubscriptionId>

However we would like to have our terraform executed via an automated build pipeline, and for that, we need to create a Service Principle. Long story short, a service principle is an identity, that has been assigned scopes to a particular subscription, and that Identity can be used to interact with Azure.

Spin up your favorite command line and lets us create the service principal

az ad sp create-for-rbac --display-name="root-terraform-sp" --role="Contributor" --scopes="/subscriptions/42e5d700***"

{
  "appId": "519ef410****",
  "displayName": "root-terraform-sp",
  "password": "JKX8Q~*****",
  "tenant": "e4794ea3-****"
}

Setting up the Storage Account for the Terraform Backend

By default, Terraform will be storing your state locally on your development box, which isn’t ideal if you want to either manage your Infrastructure via the build pipeline (because it cannot access your local store DUH!). Additionally, you will never be able to recover the state files should the laptop go missing. So let's put it somewhere on the Cloud.

Setting up an Azure Storage Account to manage our state puts us in a chicken and an egg problem. We’d want to use Terraform to manage our infrastructure, but we need some infrastructure to store our state. My recommendation is just to create a resource group and the storage account manually, and not think too much about it.

This is the snipped is shamelessly copied from the documentation linked below.

#!/bin/bash

RESOURCE_GROUP_NAME=tfstate
STORAGE_ACCOUNT_NAME=tfstate$RANDOM
CONTAINER_NAME=tfstate

# Create resource group
az group create --name $RESOURCE_GROUP_NAME --location eastus

# Create storage account
az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob

# Create blob container
az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME

Once that is done, you can get the storage Access Key, and store this somewhere secure

ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query '[0].value' -o tsv)
echo $ACCOUNT_KEY

Setting up a basic Terraform

The initial structure for Terraform project is simple.

.
├── backend
│   ├── backend.ci.tfvars
│   └── backend.local.tfvars
├── environments
│   ├── dev
│   │   └── variables.tfvars
│   └── live
│       └── variables.tfvars
├── main.tf
├── providers.tf
└── variables.tf

On the top level, we have the expected main.tf and providers.tf for configuring our terraform modules, and secondly, we have a list of environments that we will be targeting. Each environment has a set of variables and an associated backend.ci.tfvars file

Assuming that you would like to use Azure as a provider, then I’ve configured the provider.tf to look like this

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.69.0"
    }
  }

  required_version = ">= 1.5.5"
}

provider "azurerm" {
  features {}
}

Backend File

From here it's time to set up terraforms backend.local.tfvars file, with the following. The idea behind the file is to have a local copy that you can work with, so make sure that this file does not leave your machine, and preferably not part of your git repository!


resource_group_name  = "tfstate"
storage_account_name = "<storage_account_name>"
container_name       = "tfstate"
key                  = "terraform.tfstate"
access_key           = "<your account key>

We are going to create a new backend.ci.tfvars file, that contains replaceable strings that are being used from CI/CD


resource_group_name  = "tfstate"
storage_account_name = "<storage_account_name>"
container_name       = "tfstate"
key                  = "terraform.tfstate"
access_key           = "{tf_backend_storage_account_key}"

Creating an Azure Pipelines Variable group

In order to access these variables from our Azure Pipelines we need to create new Azure pipelines variable group, where we can store our service principal and our storage account access key.

Spin up your favorite Azure DevOps Project, Go to Pipeline, Go to Library, and Click the plus sign.

From here fill in the information for our service principle and our storage account access key.

Deploying your Terraform using Azure Pipelines

The whole point of us creating all this infrastructure as code, is that our infrastructure is audited, and not changed by humans directly, so of course we would like our infrastructure to be executed automatically whenever we push infrastructure changes to our repository.

Conclusion

So for the challenges we set up an initial terraform repository, we have configured a backend.ts that is configured to use Azure Storage Account as a backend Store. Additionally, we have configured a Service Principle that we can use to execute Terraform from Azure Pipelines, and set up an Azure Pipelines Variable Group to store the information.

I hope that this was enough to get you started!

References

The documentation information found in this article is already out there

Sample Repo: https://github.com/mclausen/Cloud.Platform.Foundation
Setting up Azure Storage Account for Terraform: https://learn.microsoft.com/en-us/azure/developer/terraform/store-state-in-azure-storage?tabs=azure-cli
Creating Azure Directory Service Principles: https://learn.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest#az-ad-sp-create-for-rbac
Setting up Terraform: https://developer.hashicorp.com/terraform/tutorials/azure-get-started/infrastructure-as-code

Aks from the trenches - Why zone topology might not be the best solution

Martin Humlund Clausen — Thu, 29 Jun 2023 09:11:27 +0000

It is Friday around 4 pm, the sun is shining with the temperature hitting around 27 degrees Celsius outside and all your friends are leaving work to go to the beach. And just after the sprint demos are completed and you are about to leave the office, you get that feeling in your stomach where your subconscious has been brewing on a problem that was reported earlier to another team.

Intermittent failure of requests for our customers, with no real availability pattern. You scramble the team for a late Friday afternoon in your effort to settle your stomach before hitting the door.

Running Azure Kubernetes Services without any challenges for more than a year

Last year we decided to make the big move to a more robust service hosting platform, and coming from a single mother load of VM to handle the workload of around 22 different docker images, was a big relief. No more sleepless nights of what happens to that single VM goes down, yay!

At the time our company had a huge focus on operational sustainability and reliability of services, and for that, Azure Kubernetes Services (AKS), had just the tool we needed: Zone-Affinity!

So after spinning up a new AKS Cluster, we configured the node pool consisting of 3 nodes to be spread across Zone 1, Zone 2, and Zone 3 of our local region.

For us to be highly available we wanted to have 3 API services spread across all three zones and thus applied the following spec to our deployments.

spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: {{ $fullname }}
            version: {{ $imageTag }}

This setup has been running for more than a year, and we have been happy… so far.

Nodes running out of Memory

You’ve scrambled your team together and just let that sinking feeling in your stomach out, and the team decides to have a look at the AKS instance…

Dread sets in, at this point we’ve had around 9000 restarting pods, and it seems like Kubernetes does what it is supposed to do which is keep a quorum of running services and rescheduling pods all over our services node pool.

After some additional investigations we saw that each of our nodes in the service node pull was been given the following taint

[node.kubernetes.io/memory-pressure](http://node.kubernetes.io/memory-pressure):NoSchedule

Which basically means, dont schedule more pods on the node.

Additionally, we saw that the taint was only applied to Node 1 (Zone 1), and Node 3(Zone 3), so it made sense that since our topology said “spread 1 application on all three zones” and we’ve had schedule anyway set, we’ve constrained Kubernetes to try forever to schedule workload on Node 2 (Zone 2).

Needless to say, customer tickets were starting to pop up left and right, and since we were unable to schedule the number of pods necessary to sustain throughput, we had to remedy the situation, so our customers would get happy again.

Topology constraints, Yikes.

The first option that came to mind was increasing the number of nodes that were available in the cluster to 4. And so we did..

After we added the additional node to our cluster, we had one additional node In Zone 1 of the Azure region. Unbeknown to us at the time, we ran through our kubectl command for cleaning up the failed pods

kubectl delete pods --field-selector=status.phase==Failed -n umbraco-cloud

Secondly it was manually needed to remove the node taints, and so we did, by this command

kubectl taint nodes <nodename> node.kubernetes.io/memory-pressure:NoSchedule-

To our surprise AKS continued to re-taint our Nodes and kept trying to schedule workload onto Zone 2. At the time the team was a little bewildered, and it wasn’t until we had a look at our topology.

And what we discovered was, oh well if it wasn’t the consequences of our own actions…

What we have discovered is that due to our topology constraints we forced services to be available in all three zones no matter what, and with no room for more pods on a specific node, aks would basically throw the towel in the ring.

What we ended up doing

In order for us to get to the beach with our friends, we ended up increasing the total node count to 6 in order to achieve equilibrium across all three zones in the cluster. As soon as we did that, Kubernetes automatically recovered, automatically removed the taints from the existing nodes, and automatically distributed the workload as expected.

Of course, this cost us a little extra, but the alternative was first to bump our cluster down to 4 nodes, change our helm charts to relax the topology, and then re-release all our services. But doing that on a Friday at 17 PM would have everybody miss out on the beach trip, and I would rather have 6 nodes during the weekend and not worry about it. A job for the future Platform Team…

Closing remarks

This ended up being quite a good team-bonding experience, and one of those issues where quite fun to solve. Additionally, I do not believe that this issue was something we could not foresee when we introduced Aks. It is just one of those things, you’ve to learn the hard way of operating Kubernetes.