DEV Community: Oskar Mamrzynski

Lessons learned: Azure Reservations

Oskar Mamrzynski — Tue, 05 Nov 2024 04:47:30 +0000

Here's what I learned while using Azure Reservations:

Reserve NOW

If you have doubts whether you should reserve a Virtual Machine, just ask a question: "Will this VM still be there in 6 months?" If yes, reserve it for at least 1 year.

I can't tell you how much money we've wasted using Pay-As-You-Go rates because of indecisive managers and broken promises. I've been in so many situations where after a year of deliberations we still did not reserve anything because the powers that be couldn't decide if the VMs would still be there. This next "migration project" is not a golden bullet to your infra spend.

Many costs are static: Domain Controllers, Build Agents, Virtual Desktops, Gateway servers, IIS servers etc. If they are likely to be still there in 6 months, chances are - it will be longer and you can save at least 30% by reserving for a year.

Reserve for longer

Reservations for Virtual Machines in Azure can be set to 1 year or 3 years. You are able to exchange any set of reservations for another set as long as the new total cost is greater than the remaining value of existing ones.

To give you an example:

I reserve 50 nodes of D8ads_v5 for 3 years.
In 6 months time some teams want to use another VM size.
Use Exchange button in Reservations tab to cancel existing reservation. Add new reservation for 40 nodes of D8ads_v5 and 10 nodes of E8ads_v5 (memory optimised).
The value of 50-node reservation is now (3 years minus 6 months). The new reservation can even be slightly lower in value due to different VM sizes and you can still exchange without any penalty.

Point is: You do not have to fully commit to either 1 or 3 year reservations. Microsoft cares more about keeping you on the rollercoaster than about data centre capacity planning.

Better still: You can cancel (without exchanging) up to $50K worth of reservations in any rolling 12 month period without penalties.

Some reservations are greater than others

Virtual Machines have the greatest return on reservations with savings of anything between 30% to 65%.

Azure Redis Cache can slash costs about the same because it's primarily VM-based.

Azure Managed Disks can save about 10% of PAYG price, but only from larger disks >= 1024 GB.

Azure Databricks requires a lot of usage before you can reserve. I did not manage to hit the break-even point yet.

ALWAYS try to consolidate your Log Analytics workspaces and check consolidated usage. Ingestion costs >$3 per GB at PAYG prices. You can slash that by at least 10% by committing to daily ingestion. Log Analytics is not part of normal Reservation, but rather SKU tier on the resource itself.

Observe your utilisation

You want to hit that 100% utilisation on your reservations. For a lot of static infra this is a given.

Don't beat yourself up. Even if you miscalculate, your utilisation can drop down to 60% before you break even compared to PAYG price. Just exchange for a better number of nodes, or prepare for growth.

You will sometimes hit bizarre scenarios where your monthly utilisation is <100% but you still get charged overage for that VM size in your subscription. This is because reservations are hourly but the utilisation reporting is daily. Link here to the relevant doc.

We found this out because our Azure Databricks clusters would spin up a lot of VMs for 2 hours of the day to process batch jobs. This would spike the hourly utilisation to >100% where overages are charged at PAYG price.
Then, the rest of the day, the cluster would coast at <100%. Daily utilisation was 95% while we were still charged $25 per day for overages.

Same with our Dev/Test environments which we turn off for the weekend. You cannot "accumulate" reservation hours over the weekend to then burst out to higher cores during the working week. The 2 weekend days were essentially wasted reservation compute. Still better than PAYG though.

Consider shared scope and billing subscription

You can choose how to scope your Reservations: to a resource group, subscription or every subscription (top-level management group).

Because our cost is all billed to 1 company we always choose shared scope. It allows me to reserve "All AKS nodes" across all squads and all subscriptions. I don't have to go to each team individually and ask them how many nodes they plan to use. I just reserve the total as-is. If one team scales down by 2 but another team needs more because they're growing - shared scope will make sure the utilisation stays at 100%. Otherwise, I would have to constantly review and exchange lots of little reservations.

This advice won't apply to everyone because you may have strict cost reporting needs per subscription and different teams cannot dip into each other's purse.

All reservations need a billing subscription. This might not be the subscription that the Reservation applies to but it is where the cost will be shown. We have a shared subscription for centralized bootstrap infra as per Azure CAF. This is where our "shared" cost goes, e.g. all Kubernetes nodes. Just be prepared to explain big cost spikes each month to bean counters.

The other side-effect of shared subscription scope is that some subscriptions will have zero Virtual Machine cost despite having VMs. This is because the Reservation in shared sub will be applied to them, bringing their PAYG cost to zero.

Cloudflare Pages and Origin Rulesets

Oskar Mamrzynski — Tue, 03 Oct 2023 19:54:46 +0000

In my current job we use Cloudflare (CF) extensively, most recently to migrate our main website written as an Angular SPA to a series of smaller apps in Remix (SSR) and next.js (SSG). To do this, we use Cloudflare Pages. It's a great way of hosting your site directly on the edge.

It would be cool if we could roll out new apps on new routes without impacting existing Angular app hosted on our mobile domain.

Scenario

All traffic for our mobile website, m.mydomain.com goes to Angular SPA hosted in Azure App Service. Product wants to deploy a brand new blog site on sub-path: m.mydomain.com/blog/.

Blog is a static site generation (SSG) app using next.js. Perfect candidate for hosting on CF Pages. I won't go into detail on how we build and deploy it - here's a link on how to use Wrangler to publish your build output to CF Pages.

Once we deploy our assets to a CF Pages project, we get a <project-name>.pages.dev hostname. Now, the main problem - how to host it on a sub-path of an existing domain?

Attempt 1 - Pages domain

You can bind a domain to your CF Pages project as described here. You need to create a proxied CNAME record:
m.mydomain.com -> my-project.pages.dev and bind it to that project. However, a proxied CNAME already exists! It points to our Azure App Service. Changing it would break the main site. This method is mostly for projects that are entirely served by CF Pages project, and not just part of it.

Attempt 2 - Origin Rules

Origin rules are part of the new ruleset engine and an evolution of Page Rules.

The main difference between origin rules and page rules is that you can use expressions in your rule condition instead of just path-matching patterns. So you can use hostname, IP, country, headers, cookies, paths, query strings etc. in your matching expression.

They are still in beta, but we use them extensively in our Production without issues - except this one! Let's hope they fix it before GA. :)

We can create an origin rule like this:

Now, let's go to https://m.mydomain.com/blog/. Oh no! 522 error. The connection did not time out by the way. I got this error instantly.

After contacting Cloudflare, this turns out to be some security feature preventing you from calling Pages from Cloudflare - even though the hostname we put in the rule belongs to our project. Technically, we do not own the .pages.dev domain - that is shared between all Cloudflare customers, but it would have been good for them to realize we use it!

I also found this doc which explains that your DNS override should be a hostname on your zone. That's OK - I can make a brand new CNAME on my zone: <project-name>.mydomain.com -> <project-name>.pages.dev.

Now update the origin rule:

Still nothing! Always 522 error. Believe me when I tell you - I tried every permutation of DNS records with proxied, not proxied, even tried where brand new DNS record was also a Pages domain like in attempt 1. Nothing worked.

Attempt 3 - Workers

We also have a Cloudflare Workers bundle purchased. Workers are a very powerful tool that allow you to write custom JavaScript to modify your requests, responses, fetch responses from different origins etc. We use it in some places to augment Cloudflare functionality where some rulesets don't work as we expect them. To date, we have been using this trick to implement our scenario:

You can create a CF worker script, read the URL of the incoming request, change the hostname and use fetch() to get it, as if it was a 3rd party URL.

export default {
  async fetch(request, env) {
      const url = new URL(request.url);
      url.host = "<my-project>.pages.dev";
      return fetch(url.toString(), request);
  }
}

You can bind a route to be handled by your worker code. This binding is similar to Page Rules - only path based matching allowed, not full expressions.

And voila! We have our blog site on m.mydomain.com/blog/ while the rest of m.mydomain.com remains the same. I had to change some domain names for the demo, but you can probably guess.

There are some disadvantages with this approach:

You are paying extra for Worker requests on top of your throughput.
Complex and bespoke fetch() behaviours. We put our entire mobile site through Workers for few reasons, but logic in the script becomes complicated and hard to understand.
Performance - it seems a little silly that we need an extra hop through Workers to hop over to our own CF Pages project. This hop always adds just a little bit more milliseconds to our overall latency.

Attempt 4 - Origin Rules (again!), with a workaround

Attempt 2 got me thinking - We use origin rules successfully in other places where DNS host is overwritten. In those cases, we create a CNAME record on our zone to Azure resource hostnames. e.g.
<my-app>.mydomain.com -> <my-app>.azurewebsites.net.

Maybe Cloudflare has issues with it's own domain, but not if an external DNS resolves it. What if I create a CNAME that points to a hostname on Azure DNS that ultimately resolves to my CF Pages project hostname?

Let's create a new public Azure DNS zone. To add insult to injury, I made it a sub-domain of my Cloudflare zone: omtest-sub.mydomain.com.

Now link this sub-domain to root domain using NS records in Cloudflare:

Now I can resolve my custom domain:

PS> dig blog.omtest-sub.mydomain.com

; <<>> DiG 9.14.2 <<>> blog.omtest-sub.mydomain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10431
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;blog.omtest-sub.mydomain.com. IN    A

;; ANSWER SECTION:
blog.omtest-sub.mydomain.com. 600 IN CNAME   <my-project>.pages.dev.
<my-project>.pages.dev.     300     IN      A       188.114.96.7
<my-project>.pages.dev.     300     IN      A       188.114.97.7

;; Query time: 123 msec
;; SERVER: 10.64.140.254#53(10.64.140.254)
;; WHEN: Tue Oct 03 20:33:20 GMT Summer Time 2023
;; MSG SIZE  rcvd: 124

Last step is to put my new CNAME directly into DNS override box in the origin rule from Attempt 2. After all, it's already on mydomain.com zone!

And we're back in business..

I assume this would also work if you used another DNS provider like NS1 or Route53. You obviously have to pay for external DNS resolution. Azure charges by millions of queries, but it's cheaper than having to go with Workers. Ultimately, I hope they fix this problem, so we can directly use .pages.dev domains in origin rules.

Automating high-privilege operations in Azure

Oskar Mamrzynski — Fri, 26 May 2023 07:29:14 +0000

Motivation

In a lot of traditional companies the IT department holds the keys to Azure Active Directory. Many times these teams have no interest in making things easier for developers or follow DevOps mentality to automate. For example, log a ticket in your ITSM of choice and 3 days later someone will manually create an app registration for you. With luck, it may even have been configured correctly.

Understandably, many of these operations require high privileges in AAD and you don't want to be giving these out left and right to developers to solve their own problems.

Examples of high-privilege operations we automated

Creating AAD groups and managing memberships
- We have a group-centric RBAC model in Azure. Using team-based and role-based groups we give out granular permissions to limited scopes.
- Onboarding, offboarding users - managing group memberships helps when people leave or join teams, or need a subset of team's permissions to collaborate.
- Onboarding new product teams - We want to quickly spin up new Azure subscriptions and associated groups, roles etc.
- Consistent structure - each team receives the same set of basic permissions, and any deviations are reviewed and approved.
Creating service principals and app registrations
- Often you will need a service principal to give a team access to Azure resources for automation. You may need to store its Client ID and Secret in a Key Vault or ADO service connection for them to use.
- 3rd party software like Grafana Cloud, Elastic Cloud, GitHub etc. may need service principals configured for SAML / SSO. You can manage access to these by assigning roles in the Enterprise App.
- Developers will want to secure their own APIs with App registrations, define roles, reply URLs and extra permissions.
Creating role definitions and assignments
- Custom role definitions require high privileges to create, but are essential for least-privilege systems.
- Creating consistent permission structure across landing zone subscriptions requires creating role assignments for AAD groups.
- You may need to create management-group level role assignments too.

Solution overview

Instead of creating things manually, use Terraform to create things in AAD.
Put Terraform into a source control repository with branch policies. Disallow direct commits to main branch.
Use pull requests as means to propose, review and approve changes.
Only reviewed and approved changes can be automatically deployed.
Use secure high-privilege service principals to execute Terraform pipeline.
Anyone can submit changes via Pull Requests, including developers. IT / DevOps become reviewers and advisories instead of doers of the work.

Setting up Terraform repo

We use Azure DevOps (ADO) to host our git repositories. I recommend having a dedicated project for centralised IT functions like AAD, firewall, DNS etc. This project can have a small subset of people with rights. We decided to create all 3 categories from above in the same Terraform repository, because it's easier to do onboarding of new teams and Azure subscriptions.

Importantly, you want to set up branch policies on the repo to prevent people from submitting changes directly to main branch. Branch policies can also enforce use of Pull Requests, comment resolution, running of a validation pipeline and number of reviewer votes.

Comment resolution ensures that all discussion is completed with both submitter and reviewers happy. We also have an auto-generated Terraform plan summary comment from this blog post.
Build validation will run a Terraform plan pipeline to check that IAC is valid and will post the plan to PR as a comment. We always sanity check the plan against proposed changes.
DevOps team is automatically added as reviewers. You can add multiple teams or different teams automatically depending on which paths in the repo are modified.

The actual Terraform files set up is quite easy. We use azuread and azurerm providers.

providers.tf

terraform {
  backend "azurerm" {}
  required_providers {
    azuread = {
      source  = "hashicorp/azuread"
      version = "<3.0.0"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "<4.0.0"
    }
  }
}

provider "azuread" {
  use_msi   = true
  client_id = var.group_admin_client_id
  tenant_id = local.tenant_id
}

provider "azuread" {
  alias     = "service-principal-creator"
  use_msi   = true
  client_id = var.service_principal_creator_client_id
  tenant_id = local.tenant_id
}

provider "azurerm" {
  alias                      = "access-admin"
  subscription_id            = "<subscription_id>"
  use_msi                    = true
  client_id                  = var.access_admin_client_id
  tenant_id                  = local.tenant_id
  skip_provider_registration = true
  features {
    key_vault {
      recover_soft_deleted_key_vaults = false
    }
  }
}

data "azuread_client_config" "sp_creator" {
  provider = azuread.service-principal-creator
}

data "azuread_client_config" "group_admin" {}

dynamic-groups.tf (example)

resource "azuread_group" "az-all-technology-staff" {
  display_name     = "az-all-technology-staff"
  security_enabled = true
  owners           = [data.azuread_client_config.group_admin.object_id]
  types            = ["DynamicMembership"]

  dynamic_membership {
    enabled = true
    rule    = "user.department -eq \"Technology\" and (user.accountEnabled -eq true)"
  }
  lifecycle {
    ignore_changes = [members]
  }
}

service-principals.tf (module example)

module "policy_contributor" {
  source         = "../modules/service-principal"
  principal_name = "<conventions_prefix>-azure-policy-contributor"
  providers = {
    azuread = azuread.service-principal-creator
  }
}

service-principals.tf (inside the module)

data "azuread_client_config" "identity" {}

resource "azuread_application" "app" {
  display_name = var.principal_name
  owners       = [data.azuread_client_config.identity.object_id]
  lifecycle {
    ignore_changes = [required_resource_access]
  }
}

resource "azuread_service_principal" "sp" {
  application_id = azuread_application.app.application_id
  owners         = [data.azuread_client_config.identity.object_id]
}

resource "azuread_application_password" "app_client_secret" {
  application_object_id = azuread_application.app.object_id
}

resource "azurerm_key_vault_secret" "client_id" {
  name         = "${var.principal_name}-client-id"
  value        = azuread_application.app.application_id
  key_vault_id = var.key_vault_id
}

resource "azurerm_key_vault_secret" "client_secret" {
  name         = "${var.principal_name}-client-secret"
  value        = azuread_application_password.app_client_secret.value
  key_vault_id = var.key_vault_id
}

role-assignments.tf (example)

resource "azurerm_role_assignment" "az-sub-readers-prod" {
  provider             = azurerm.access-admin
  scope                = "/subscriptions/${var.prod_subscription_id}"
  role_definition_name = "Reader"
  principal_id         = module.az-sub-readers-prod.object_id
}

resource "azurerm_role_assignment" "az-sub-contributors-prod" {
  provider             = azurerm.access-admin
  scope                = "/subscriptions/${var.prod_subscription_id}"
  role_definition_name = "Contributor"
  principal_id         = module.az-sub-contributors-prod.object_id
}

We can put all of these together into a Terraform module to onboard whole sets of product teams with default permissions. Module example below creates AAD groups for readers, contributors, AKS readers, AKS admins, SQL admins, SQL readers, KV readers, KV admins, monitoring contributors, a service principal for team-abc to use in ADO pipelines and assigns roles in Azure on relevant subscriptions.

portfolio-abc.tf (example)

module "team-abc" {
  source = "../modules/group"
  name   = "team-abc"
  active_members = [
    local.user_ids["user1@domain.com"],    
    local.user_ids["user2@domain.com"],
  ]
}

module "abc-default-groups" {
  source                  = "../modules/default-portfolio-groups"
  portfolio_code          = "abc"
  devtest_subscription_id = local.sub_ids["abc-devtest"]
  prod_subscription_id    = local.sub_ids["abc-prod"]
  default_team_object_id  = module.team-abc.object_id
  devops_team_object_id   = module.team-devops.object_id
  dba_team_object_id      = module.team-db-admins.object_id
  kingmakers              = true
  providers = {
    azurerm.kv-admin                  = azurerm
    azurerm.access-admin              = azurerm.access-admin
    azuread.service-principal-creator = azuread.service-principal-creator
  }
}

Setting up automation identities

You may have noticed we use 3 provider blocks in providers.tf file. We manually created a separate user-assigned managed identity (service principal) for each scenario and assigned them the right set of privileges. You need to be a Global Admin on AAD to set these up.

Access admin - We assigned it User Access Administrator role over the tenant root management group. This way it can create role definitions scoped across all subscriptions and manage default role assignments for each sub during landing zone onboarding.

Group admin - This identity needs permissions over Microsoft Graph API to create/delete AAD groups and manage them.

We assigned it Directory.Read.All, Group.ReadWrite.All and RoleManagement.ReadWrite.Directory app roles on Microsoft Graph API. Required roles are described here.

I found this blog post about how to assign extra app roles to managed identities. Execute this Azure PowerShell script as Global Admin:

$roles = @('Directory.Read.All', 'Group.ReadWrite.All', 'RoleManagement.ReadWrite.Directory')
$managed_identity = Get-AzADServicePrincipal -ObjectId '<mi_object_id>'

$access_token = (Get-AzAccessToken -ResourceTypeName 'MSGraph').Token
$graph_sp = Get-AzADServicePrincipal -ApplicationId '00000003-0000-0000-c000-000000000000'

$roles | % {
    $role_name = $_
    $role = $graph_sp.AppRole | ? { $_.Value -eq $role_name }
    $body = @{
        'principalId' = $managed_identity.Id;
        'resourceId'  = $graph_sp.Id;
        'appRoleId'   = $role.Id
    } | ConvertTo-Json -Compress

    Invoke-RestMethod `
        -Method POST `
        -Headers @{Authorization = "Bearer $access_token" } `
        -ContentType 'application/json' `
        -Uri "https://graph.microsoft.com/v1.0/servicePrincipals/$($managed_identity.Id)/appRoleAssignments" `
        -Body $body
}

Service principal creator - This identity also needs MS Graph permissions, as described here and here.

We executed the same script as above, just replacing roles list with Directory.Read.All, Application.ReadWrite.OwnedBy, AppRoleAssignment.ReadWrite.All.

Setting up pipeline

We use managed identities rather than service principals so we do not need to use and rotate client secrets. Our Terraform pipeline should execute on an ADO agent in a trusted location. See my other blog post about how we set up an ADO agent linked to managed identities. The agent can run on a normal VM too and be assigned these managed identities directly instead of using federated credentials. Bottom line is that our ADO project has an agent pool where the agent is able to obtain tokens from these 3 managed identities.

For Terraform to log in with 3 different managed identities with 3 different providers we need to pass in Client ID for each of them as a parameter.

pipeline.yaml (example)

name: $(Rev:rr)
trigger:
- main
pool: aks-azure-rbac

variables:
  tf_dir: $(Build.SourcesDirectory)/terraform
  tf_vars: |
    service_principal_creator_client_id = "<client_id_1>"
    group_admin_client_id = "<client_id_2>"
    access_admin_client_id = "<client_id_3>"

steps:
- checkout: self
  clean: true

- pwsh: |
    $tf_vars = '$(tf_vars)'
    [System.IO.File]::WriteAllText('terraform.tfvars', $tf_vars)
  displayName: set tf vars
  workingDirectory: $(tf_dir)

- task: TerraformTaskV2@2
  displayName: tf init
  inputs:
    provider: azurerm
    command: init
    workingDirectory: $(tf_dir)
    backendServiceArm: <terraform_storage_service_connection>
    backendAzureRmResourceGroupName: <storage_account_rg>
    backendAzureRmStorageAccountName: <storage_account_name>
    backendAzureRmContainerName: <storage_container>
    backendAzureRmKey: <blob_name>

- pwsh: |
    terraform plan -out tfplan
  displayName: tf plan
  workingDirectory: $(tf_dir)

- pwsh: |
    & '$(Build.SourcesDirectory)/scripts/set-tf-plan-pr-comments.ps1'
  workingDirectory: $(tf_dir)
  displayName: set pr comments
  condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest'), ne(variables ['Build.Repository.Provider'] , 'GitHub') )
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

- pwsh: |
    terraform apply -auto-approve tfplan
  displayName: tf apply
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  workingDirectory: $(tf_dir)

- task: DeleteFiles@1
  displayName: clean up tf files
  condition: always()
  inputs: 
    Contents: |
      **/tfplan
      **/*.tfvars
      **/*.tfstate

We set up the pipeline to only run the apply step when running on main branch. Terraform will still plan changes and post PR comments with a summary when running a PR validation build.

Process for making changes

In our ADO project we allow everyone in the tech department to read and contribute to our repository. They must do so via pull requests. A contributor can either clone the repo to their local, make changes on a branch and submit a pull request or do it directly via the ADO UI.

All pull requests go to our Slack channel so our team can review them and approve if everything is OK.

Once approved, the PR is merged into main where automated pipeline picks it up, authenticates using managed identities and creates whatever is necessary via Terraform.

Our own team members raising changes are treated no different from any other contributor. Someone else still has to review and approve.

Security considerations

You need high privilege role to set this up to begin with. At least Application Administrator in AAD and User Access Administrator over management groups. I had to elevate my account to Global Administrator using Privileged Identity Management.
Creating things manually with user accounts can be safer because you would normally have to pass through Privileged Identity Management, approvals, conditional access policy, MFA etc. before you can execute a high privilege action.
Automated identities have fewer safety restrictions than user accounts (lack of MFA for instance). You may be able to set up conditional access policy to only allow obtaining access tokens from 1 trusted location - ADO agent.
"Who guards the guardians" - whoever controls the system responsible for automation can elevate themselves to use these managed identities. This includes Azure DevOps project admins (on that particular project), Project Collection Admins, on Azure - Contributors and Managed Identity Operator roles.
If you have a Managed Identity Role over those managed identities (or equivalent role on a service principal) then you can obtain access tokens with privileges potentially higher than your own. Azure landing zone architecture seems to recommend a dedicated Azure subscription where you create these managed identities and where limited number of people have access. We follow this practice.
Our ADO agents run on an AKS cluster. There are a myriad of ways in which a cluster can become vulnerable and we try our best to secure it, but you also have to be careful who can execute kubectl exec on it.
Terraform state file is stored in Azure blob storage in our case. We create Azure Key Vault secrets via Terraform to put in Client ID and Client Secret, but plain text values are also present in the Terraform state file. Described here. We will soon mitigate this by moving most of our service principals to managed identities. Whoever has access to the state file can read out these secrets.
You may want to set up any Activity Log alerts in AAD and in Azure for when these identities do anything. If the timestamp doesn't coincide with a main-branch pipeline, then something fishy may be going on.

Azure DevOps agents on AKS with workload identity

Oskar Mamrzynski — Tue, 23 May 2023 15:00:06 +0000

Recently I deployed some Azure DevOps (ADO) agents on Azure Kubernetes Service (AKS) and use the new workload identity add-on to authenticate to Azure. We mainly use these agents to deploy other Azure with Terraform / Azure CLI / Azure PowerShell within our private network. Builds are done on Microsoft-hosted agents.

Pros:

Resource utilisation is better than using VMs. Agents used for deployment are not very resource intensive on CPU/Memory as they mostly do network I/O to talk to various APIs and deploy things.

You can use KEDA to auto-scale agents based on number of pending ADO jobs. This lets you scale down during quiet periods to reclaim resources.
You can deploy agents within your private network. Most of our deployment targets are behind either private link or IP-whitelisted resources. You cannot use Microsoft-hosted agents to deploy to those. You must use self-hosted agents on e.g. network-joined AKS cluster.
When running deployment pipelines on ADO agents, you will have to authenticate to Azure to deploy or manage resources. You can use service principals, but they are not the most secure option because they require a Client ID and Secret.
Managed identities are great for keyless authentication to Azure resources. Terraform and all Azure tools support managed identity authentication. You can get rid of the need for principal credential rotation.
Workload identity add-on for Kubernetes is the new way of assigning managed identities to pods in Kubernetes. This can be done between multiple Azure subscriptions (and even cross-tenant), and agent pod doesn't even have to live on Azure Kubernetes Service (could be EKS or GKE). It is less clunky than aad-pod-identity.

Cons:

Anyone who can execute jobs on that agent pool will be able to obtain managed identity access token. If you share agent pools between multiple teams then you shouldn't use this method. Each team will be able to access each other's resources.
For multiple teams you should ideally have 1 managed identity linked to 1 agent pool, and 1 agent pool usable only by 1 team (ADO project).

This is more of a "fill in the blanks" guide than a step-by-step tutorial. I am going to only briefly mention some setup steps because they are better documented elsewhere.

Set up a basic ADO agent on AKS first:

Create a basic ADO agent image. This just involves a base Debian image, some dependencies and a pre-made script from Microsoft. My agents are all based on a Linux distro, so you can find the guide here.
Add your dependencies on top of the base ADO agent image. In my case, I just need Terraform and PowerShell, but you can also install Azure CLI, kubectl etc.

FROM <Base ADO agent image>

WORKDIR /tools

# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes

# Updating paths to tools
ENV PATH="/tools/azure-powershell-core:/tools/terraform:/tools:${PATH}"

# Install basics
RUN apt-get update 
RUN apt-get install curl zip unzip

# Install Terraform
ADD https://releases.hashicorp.com/terraform/1.4.6/terraform_1.4.6_linux_amd64.zip /tmp/terraform.zip
RUN unzip /tmp/terraform.zip -d terraform
RUN chmod +x terraform/terraform

# Install PowerShell Core
ADD https://github.com/PowerShell/PowerShell/releases/download/v7.1.5/powershell_7.1.5-1.ubuntu.20.04_amd64.deb powershell.deb
RUN dpkg -i powershell.deb ; exit 0
RUN apt-get install -f

WORKDIR /azp

Push your ADO agent image to a registry of your choice, in my case ACR:

az acr login -n <acr_name>
docker push <acr_name>.azurecr.io/<ado_agent_image>

You need an AKS cluster with workload identity add-on deployed. You can create one using Azure CLI as described here. In my case, clusters are deployed using Terraform, so I just set these properties to true. If using Terraform, you will need to output OIDC issuer URL to be used in subsequent steps.

resource "azurerm_kubernetes_cluster" "cluster" {
(...)
  oidc_issuer_enabled                 = true
  workload_identity_enabled           = true
(...)
}

output "aks_oidc_issuer_url" {
  value = azurerm_kubernetes_cluster.cluster.oidc_issuer_url
}

Create ADO agent pool and personal access token to join agents to a pool.

Deploy your ADO agent image as a Kubernetes deployment. You can use YAML, though I prefer to create everything in Terraform.

### variables.tf
variable "pool_name" {}
variable "ado_token" {}

variable "replicas" {
  default = 1
}

variable "requests" {
  default = {
    cpu    = "100m"
    memory = "512Mi"
  }
}

variable "limits" {
  default = {
    memory = "2Gi"
  }
}

variable "disk_space_limit" {
  default = "4Gi"
}

locals {
  work_dir  = "/mnt/work"
  ado_agent = "ado-agent-${var.pool_name}"
}

### providers.tf
terraform {
  <backend_configuration>
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "<3.0.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "<3.0.0"
    }
  }
}

### kubernetes.tf
resource "kubernetes_namespace_v1" "namespace" {
  metadata {
    name = "ado-agents-${var.pool_name}"
  }
}

resource "kubernetes_secret_v1" "env_vars" {
  metadata {
    name      = local.ado_agent
    namespace = kubernetes_namespace_v1.namespace.metadata[0].name
  }

  data = {
    # These variables are used by the script from Step 1. 
    # Have a look at the script Microsoft provides in their guide and what other config options they have available. 
    # Each of these secret Key-Value entries is another environment variable mounted to the pod.
    AZP_URL   = "https://dev.azure.com/<your_org_name>"
    # You can source this from input variable, Azure Key Vault, etc.
    AZP_TOKEN = var.ado_token 
    AZP_POOL  = var.pool_name
    AZP_WORK  = local.work_dir
  }
}

resource "kubernetes_deployment_v1" "ado_agent" {
  metadata {
    name      = local.ado_agent
    namespace = kubernetes_namespace_v1.namespace.metadata[0].name
  }

  spec {
    replicas = var.replicas

    selector {
      match_labels = {
        app = local.ado_agent
      }
    }

    template {
      metadata {
        labels = {
          app = local.ado_agent
        }
      }

      spec {
        container {
          name  = local.ado_agent
          image = "<acr_name>.azurecr.io/<ado_agent_image>"

          # We link all variables from a secret ref instead of listing them here directly.
          # so that your PAT token doesn't show up on kubectl describe pod
          env_from {
            secret_ref {
              name = kubernetes_secret_v1.env_vars.metadata[0].name
            }
          }

          volume_mount {
            name       = "temp-data"
            mount_path = local.work_dir
          }

          resources {
            limits   = var.limits
            requests = var.requests
          }
        }

        volume {
          name = "temp-data"

          # Using node's ephemeral disk space for agent storage.
          # Deployments agents do not use much disk space and can be wiped after pod dies.
          empty_dir {
            size_limit = var.disk_space_limit
          }
        }
      }
    }
  }
}

Use terraform plan and apply while your kubectl context is linked to your AKS cluster. This should create a new agent and link it to the pool. Microsoft's start up script downloads a latest version of the agent each time a pod starts up, registers it against the pool and removes it on pod termination.

PS> kubectl get deployment
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
omtest1                  1/1     1            1           11d

This set up is just a normal ADO agent without workload identity linked together. You would still need a Client ID / Secret or Azure DevOps Service Connection to use in your pipelines.

Adding workload identity to ADO agent:

Workload identity works by linking a Kubernetes service account to a managed identity via federated credentials. This establishes trust between a specific Kubernetes cluster + namespace + service account and a managed identity. An identity no longer has to be assigned to AKS nodes like in aad-pod-identity.
Create a user-assigned managed identity and give it some role assignment over Azure resources or resource group, so it can manage it. Here is modified Terraform from above.

### variables.tf
variable "pool_name" {}

variable "ado_token" {}

variable "replicas" {
  default = 1
}

variable "requests" {
  default = {
    cpu    = "100m"
    memory = "512Mi"
  }
}

variable "limits" {
  default = {
    memory = "2Gi"
  }
}

variable "disk_space_limit" {
  default = "4Gi"
}

variable "location" {
  default = "westeurope"
}

variable "managed_identity_scope {}

# Take this from your AKS deployment
variable "aks_oidc_issuer_url" {}

locals {
  work_dir  = "/mnt/work"
  ado_agent = "ado-agent-${var.pool_name}"
}

### providers.tf
terraform {
  <backend_configuration>
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "<3.0.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "<3.0.0"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "<4.0.0"
    }
  }
}

provider "azurerm" {
  features {}
}

### azure.tf
resource "azurerm_resource_group" "group" {
  name     = "rg-identity-${var.pool_name}"
  location = var.location
}

resource "azurerm_user_assigned_identity" "identity" {
  name                = "mi-${var.pool_name}"
  resource_group_name = azurerm_resource_group.group.name
  location            = var.location
}

# This resource will establish trust between Kubernetes cluster, namespace and service account. 
# Only pods assigned that service account will be able to obtain an access token from Azure for this corresponding managed identity.
resource "azurerm_federated_identity_credential" "identity" {
  resource_group_name = azurerm_user_assigned_identity.identity.resource_group_name
  parent_id           = azurerm_user_assigned_identity.identity.id
  name                = "${kubernetes_namespace_v1.namespace.metadata[0].name}-${kubernetes_service_account_v1.service_account.metadata[0].name}"
  subject             = "system:serviceaccount:${kubernetes_namespace_v1.namespace.metadata[0].name}-${kubernetes_service_account_v1.service_account.metadata[0].name}"
  audience            = ["api://AzureADTokenExchange"]
  issuer              = var.aks_oidc_issuer_url
}

# Give the managed identity some role over a resource group it will manage. 
# This does not have to be done in Terraform if you don't have permissions to do it.
resource "azurerm_role_assignment" "deployment_group" {
  scope                = var.managed_identity_scope
  role_definition_name = "Contributor"
  principal_id         = azurerm_user_assigned_identity.identity.principal_id
}

### kubernetes.tf
resource "kubernetes_namespace_v1" "namespace" {
  metadata {
    name = "ado-agents-${var.pool_name}"
  }
}

resource "kubernetes_secret_v1" "env_vars" {
  metadata {
    name      = local.ado_agent
    namespace = kubernetes_namespace_v1.namespace.metadata[0].name
  }

  data = {
    AZP_URL   = "https://dev.azure.com/<org_name>"
    AZP_TOKEN = var.ado_token
    AZP_POOL  = var.pool_name
    AZP_WORK  = local.work_dir
  }
}

# Pods would normally use namespace default service account.
# Instead, we create a specific service account for these pods and link it to Azure. 
# You do not have to automount service account token. 
# Workload Identity will still mount a signed token file different to normal Kubernetes Service account token file. 
# This token can be exchanged with Azure for an Azure access token.
resource "kubernetes_service_account_v1" "service_account" {
  metadata {
    namespace = kubernetes_namespace_v1.namespace.metadata[0].name
    name      = local.ado_agent
    # These annotations are optional but they will set the "default" managed identity for this service account. 
    # This is helpful if you plan to have 1:N SA->MI federated credentials.
    annotations = {
      "azure.workload.identity/client-id" = azurerm_user_assigned_identity.identity.client_id
      "azure.workload.identity/tenant-id" = azurerm_user_assigned_identity.identity.tenant_id
    }
  }
  automount_service_account_token = false
}

resource "kubernetes_deployment_v1" "ado_agent" {
  metadata {
    name      = local.ado_agent
    namespace = kubernetes_namespace_v1.namespace.metadata[0].name
  }

  spec {
    replicas = var.replicas

    selector {
      match_labels = {
        app = local.ado_agent
      }
    }

    template {
      metadata {
        # This label must be specified so workload identity add-on is able to process the pod.
        labels = {
          app                           = local.ado_agent
          "azure.workload.identity/use" = "true"
        }
        # While workload identity works by token-exchange method by default, this is not supported by Azure terraform provider. 
        # If we want to use managed identity authentication in our agent, we must inject a sidecar. 
        # This extra container will spoof a managed identity endpoint and provide us with an access token.
        annotations = {
          "azure.workload.identity/inject-proxy-sidecar" : "true"
        }
      }

      spec {
        # Link the service account to deployment, so all pods created by the deployment have the SA token.
        service_account_name            = kubernetes_service_account_v1.service_account.metadata[0].name
        automount_service_account_token = false
        container {
          name  = local.ado_agent
          image = "<acr_name>.azurecr.io/<ado_agent_image>"

          env_from {
            secret_ref {
              name = kubernetes_secret_v1.env_vars.metadata[0].name
            }
          }

          volume_mount {
            name       = "temp-data"
            mount_path = local.work_dir
          }

          resources {
            limits   = var.limits
            requests = var.requests
          }
        }

        volume {
          name = "temp-data"

          empty_dir {
            size_limit = var.disk_space_limit
          }
        }
      }
    }
  }
}

Once deployed, your agent pod will have an extra sidecar container and init container:

PS> kubectl get pod
NAME                                     READY   STATUS    RESTARTS   AGE
ado-agent-<pool_name>-7b8d79dd65-hwb22   2/2     Running   0          3d22h

PS> kubectl describe pod
Name:         ado-agent-<pool_name>-7b8d79dd65-hwb22
Namespace:    ado-agents-<pool_name>
(...)
Labels:       app=ado-agent-<pool_name>
              azure.workload.identity/use=true
Annotations:  azure.workload.identity/inject-proxy-sidecar: true
(...)
Init Containers:
  azwi-proxy-init:
    Image:          mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v1.0.0
    (...)
    Environment:
      PROXY_PORT:                  8000
      AZURE_CLIENT_ID:             <managed_identity_client_id>
      AZURE_TENANT_ID:             <managed_identity_tenant_id>
      AZURE_FEDERATED_TOKEN_FILE:  /var/run/secrets/azure/tokens/azure-identity-token
      AZURE_AUTHORITY_HOST:        https://login.microsoftonline.com/
    Mounts:
      /var/run/secrets/azure/tokens from azure-identity-token (ro)
Containers:
  ado-agent-<pool_name>:
    Image:          <acr_name>.azurecr.io/<ado_agent_image>
    (...)
    Environment Variables from:
      ado-agent-<pool_name>  Secret  Optional: false
    Environment:
      AZURE_CLIENT_ID:             <managed_identity_client_id>
      AZURE_TENANT_ID:             <managed_identity_tenant_id>
      AZURE_FEDERATED_TOKEN_FILE:  /var/run/secrets/azure/tokens/azure-identity-token
      AZURE_AUTHORITY_HOST:        https://login.microsoftonline.com/
    Mounts:
      /mnt/work from temp-data (rw)
      /var/run/secrets/azure/tokens from azure-identity-token (ro)
  azwi-proxy:
    Image:         mcr.microsoft.com/oss/azure/workload-identity/proxy:v1.0.0
    (...)
    Port:          8000/TCP
    Host Port:     0/TCP
    Args:
      --proxy-port=8000
      --log-level=info
    Environment:
      AZURE_CLIENT_ID:             <managed_identity_client_id>
      AZURE_TENANT_ID:             <managed_identity_tenant_id>
      AZURE_FEDERATED_TOKEN_FILE:  /var/run/secrets/azure/tokens/azure-identity-token
      AZURE_AUTHORITY_HOST:        https://login.microsoftonline.com/
    Mounts:
      /var/run/secrets/azure/tokens from azure-identity-token (ro)
(...)

Obtaining an access token using workload identity:

Guides here and here describe how access tokens are obtained via workload identity. Since we use a sidecar to mimic an IMDS endpoint, getting an access token is described here. Let's try it using kubectl exec.

PS> kubectl exec -it ado-agent-<pool_name>-7b8d79dd65-hwb22 -- pwsh
Defaulted container "ado-agent-<pool_name>" out of: ado-agent-<pool_name>, azwi-proxy, azwi-proxy-init (init)
PowerShell 7.2.6
Copyright (c) Microsoft Corporation.

https://aka.ms/powershell
Type 'help' to get help.

PS /azp> Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -Headers @{Metadata="true"}

StatusCode        : 200
StatusDescription : OK
Content           : {"access_token":"eyJ0eXA...
RawContent        : HTTP/1.1 200 OK
                    Server: azure-workload-identity/proxy/v1.0.0 (linux/amd64) 9893baf/2023-03-27-20:58
                    Date: Tue, 23 May 2023 09:17:48 GMT
                    Content-Type: application/json
                    Content-Length: 1707

                    {"access_to…
Headers           : {[Server, System.String[]], [Date, System.String[]], [Content-Type, System.String[]], [Content-Length, System.String[]]}
Images            : {}
InputFields       : {}
Links             : {}
RawContentLength  : 1707
RelationLink      : {}


PS /azp> (Invoke-RestMethod -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -Headers @{Metadata="true"}).access_token
eyJ0eXA...

We are now ready to run a Terraform pipeline against this new agent pool.

Running a pipeline:

Create a Terraform repo in Azure DevOps to create some example resources.

### main.tf
terraform {
  <backend_configuration>
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "<4.0.0"
    }
  }
}

provider "azurerm" {
  # We can just use this flag to authenticate
  use_msi         = true
  subscription_id = "<subscription_id>"
  features {}
}

resource "azurerm_application_insights" "example" {
  name                = "appins-omtest1"
  location            = "westeurope"
  resource_group_name = "rg-identity-omtest1"
  application_type    = "web"
}

Create YAML pipeline to run Terraform:

name: $(Rev:rr)
trigger:
- main
pool: omtest1

steps:
- pwsh: |
    terraform init
    terraform plan -out tfplan
    terraform apply -auto-approve tfplan
  workingDirectory: $(Build.SourcesDirectory)/terraform-deploy-with-managed-identity

Result:

In summary:
We have our ADO agent running in AKS and use workload + managed identity combination to achieve keyless authentication to Azure for running Terraform pipelines.

Deploy linkerd to Kubernetes using Helm and Terraform

Oskar Mamrzynski — Sun, 31 Jul 2022 10:13:00 +0000

So I've been trying to install Linkerd service mesh on our AKS clusters this week. There are a few options to install: using their command line utility or using Helm.

Command line tool works well, it will generate CA and issuer certificates for you. However, you have to download it when deploying via a pipeline, and there's a small problem with running linkerd check --pre - it will only succeed on first installation and fail after linkerd is installed.

So I chose to install it with Helm, along with linkerd-cni and linkerd viz dashboard. You can find all their charts here. I don't use Helm command line in pipelines, preferring to use Terraform helm provider. This is because I can install and configure many Helm releases using a single set of Terraform commands.

Generating CA and issuer certs
You need to generate your own CA and issuer certificates if you choose to install linkerd with Helm. They give you instructions on how to generate them using step command line utility. However, we can also generate them using Terraform tls provider.

resource "tls_private_key" "ca" {
  algorithm   = "ECDSA"
  ecdsa_curve = "P256"
}

resource "tls_self_signed_cert" "ca" {
  private_key_pem       = tls_private_key.ca.private_key_pem
  is_ca_certificate     = true
  set_subject_key_id    = true
  validity_period_hours = 87600
  allowed_uses = [
    "cert_signing",
    "crl_signing"
  ]
  subject {
    common_name = "root.linkerd.cluster.local"
  }
}

resource "tls_private_key" "issuer" {
  algorithm   = "ECDSA"
  ecdsa_curve = "P256"
}

resource "tls_cert_request" "issuer" {
  private_key_pem = tls_private_key.issuer.private_key_pem
  subject {
    common_name = "identity.linkerd.cluster.local"
  }
}

resource "tls_locally_signed_cert" "issuer" {
  cert_request_pem      = tls_cert_request.issuer.cert_request_pem
  ca_private_key_pem    = tls_private_key.ca.private_key_pem
  ca_cert_pem           = tls_self_signed_cert.ca.cert_pem
  is_ca_certificate     = true
  set_subject_key_id    = true
  validity_period_hours = 8760
  allowed_uses = [
    "cert_signing",
    "crl_signing"
  ]
}

Configuring linkerd Helm releases

Now all I need is to add 3 helm_release resources for linkerd CNI, linkerd and linkerd viz dashboard, providing outputs of certificate resources:

resource "helm_release" "linkerd_cni" {
  name             = "linkerd-cni"
  namespace        = "linkerd-helm"
  repository       = "https://helm.linkerd.io/stable"
  chart            = "linkerd2-cni"
  version          = "2.11.4"
  create_namespace = true
}

resource "helm_release" "linkerd" {
  name             = "linkerd"
  namespace        = helm_release.linkerd_cni.namespace
  repository       = "https://helm.linkerd.io/stable"
  chart            = "linkerd2"
  version          = "2.11.4"
  create_namespace = false
  set {
    name  = "cniEnabled"
    value = "true"
  }
  set {
    name  = "identityTrustAnchorsPEM"
    value = tls_locally_signed_cert.issuer.ca_cert_pem
  }
  set {
    name  = "identity.issuer.tls.crtPEM"
    value = tls_locally_signed_cert.issuer.cert_pem
  }
  set {
    name  = "identity.issuer.tls.keyPEM"
    value = tls_private_key.issuer.private_key_pem
  }
}

resource "helm_release" "linkerd_viz" {
  name             = "linkerd-viz"
  namespace        = helm_release.linkerd_cni.namespace
  repository       = "https://helm.linkerd.io/stable"
  chart            = "linkerd-viz"
  version          = "2.11.4"
  create_namespace = false
}

Namespaces

Helm needs a namespace to create the release in, but you cannot pre-create default linkerd namespaces. Their Helm chart deploys and configures them with appropriate annotations. Instead I create tell Helm resource to create a new namespace called linkerd-helm which houses only Helm releases but no linkerd resources. I also use implicit dependency on helm_release_linkerd_cni.namespace so CNI is deployed before linkerd and viz dashboard.

Once I run through the standard terraform init, plan, apply I end up with this:

kubectl get namespaces
NAME                       STATUS   AGE
<truncated>
linkerd                    Active   2d19h
linkerd-cni                Active   2d19h
linkerd-helm               Active   2d19h
linkerd-viz                Active   37h

helm ls --namespace linkerd-helm
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
linkerd         linkerd-helm    4               2022-07-29 09:59:38.7045481 +0000 UTC   deployed        linkerd2-2.11.4         stable-2.11.4
linkerd-cni     linkerd-helm    1               2022-07-28 14:40:00.021646641 +0000 UTC deployed        linkerd2-cni-2.11.4     stable-2.11.4
linkerd-viz     linkerd-helm    1               2022-07-29 20:26:34.495285272 +0000 UTC deployed        linkerd-viz-2.11.4      stable-2.11.4

P.S. Caveat for Azure Kubernetes Service: Do not use Open Service Mesh add-on if you want to install linkerd. It kept crashing for me.

Obtain Azure access token from a local Docker container

Oskar Mamrzynski — Sat, 30 Jul 2022 02:43:59 +0000

Apps talking to Azure need to obtain an access token at runtime. For example, reading secrets from Azure Key Vault as part of your configuration. There's a whole host of Azure services you can talk to using tokens: Azure Key Vault, Azure App Config, Azure SQL, Azure Event Hubs, Azure Service Bus, Azure Storage Account etc.

All follow the same basic flow: obtain an access token as an Azure Identity and attach that token to API requests for that Azure service. In .NET apps you can use the new Azure.Identity library and it's DefaultAzureCredential type.

This type will automatically try to obtain an Azure access token using various methods, but 3 are of particular interest:

Environment variables - for logging in as a Service Principal using Client ID, Client Secret and Tenant ID environment variables
Managed Identity - for logging in using a system or user-assigned managed identity in Azure systems like Azure App Service, Azure Kubernetes Service, Azure VM etc.
Azure CLI - for logging in on your local machine for development work

Q: So how am I supposed to log in to Azure so that my app can obtain tokens?
A: I tell devs: For local development log in to Azure CLI with your normal user account. It has Contributor over your Dev/Test subscription and you can access secrets and configuration from their Dev/Test Key Vaults.
For staging and production running in Azure (in our case Docker containers running on AKS) we use User-Assigned Managed Identity and aad-pod-identity project. This managed identity has least-privilege permissions over staging and production environments to do it's job at runtime.

Q: I can obtain tokens locally using Azure CLI and Azure.Identity library when I run on the host machine, but not when inside Docker container because it doesn't have Azure CLI installed! What do I do?
A: This has already been asked about by many people here with various interesting solutions here and here.

I decided to write another solution of my own because I would like locally run Docker containers to be as closer to staging as possible, i.e. use Managed Identity flow to obtain Azure access tokens. Obviously I cannot use Managed Identities on a laptop but I can spoof the process.

Let's reproduce the problem with least code.
Made a little sample console app that connects to Azure Key Vault, adds secrets to IConfiguration and then prints all config to console (not a great practice in prod btw!).

Running it directly on my laptop with dotnet is fine because my laptop has Azure CLI installed, and I am logged into it.



$env:AZURE_KEY_VAULT_URI="https://<redacted>.vault.azure.net/"
dotnet run .\SampleConsumerApp.csproj
mysecret=secretsauce (AzureKeyVaultConfigurationProvider)

Q: What happens when I build a Docker image and try to run it?



docker build -t dev:consumer -f .\Dockerfile .
<truncated>
docker run -e AZURE_KEY_VAULT_URI=https://<redacted>.vault.azure.net/ dev:consumer
Unhandled exception. Azure.Identity.CredentialUnavailableException: DefaultAzureCredential failed to retrieve a token from the included credentials. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/defaultazurecredential/troubleshoot
- EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
- ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
- Operating system Linux 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 isn't supported.
- Stored credentials not found. Need to authenticate user in VSCode Azure Account. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/vscodecredential/troubleshoot
- Azure CLI not installed
- PowerShell is not installed.
<truncated>

Q: Why use Azure CLI at all? Why not just configure my Docker container with environment variables for ClientId, ClientSecret and TenantID?
A: While I give devs a service principal for Dev/Test to use in pipelines, this principal's credentials are not supposed to be used for local development. Configuring SP credentials with environment variables would prevent us from rotating them frequently too. Not everyone has a service principal to hand either!

Q: I don't want to install Azure CLI into my dev Docker images just to obtain a token. So what's next?
A: It seems getting an access token using a Managed Identity token endpoint is the easiest because you can just curl a URL and get a token. One problem is that nearly all authentication libraries hardcode this URL as http://169.254.169.254/metadata/identity/oauth2/token?<uri_params>

So, I am going to create a "proxy" managed identity token endpoint and return a token just like the app expects.

Q: How will this proxy provider get a token in the first place?
A: This proxy provider is a tiny ASP.NET Core web API and can use Azure.Identity library just like the sample console app. So let's make it get this token from Azure CLI. I can install Azure CLI into that token provider since it's only a development tool and not my app's production image.

Q: Ok but how would the Azure CLI inside proxy's Docker container get an access token? You'd have to log into Azure CLI every time you start the container!
A: I can mount my laptop's .azure folder as a volume into the token provider's container. As long as I am logged into Azure CLI on my laptop I can obtain tokens using Azure CLI inside proxy's container! Combine that with a little web API and we're in business.

If this is confusing, here's a handy diagram:

There is a small problem with this that someone already spotted. It only works with Azure CLI version up to 2.29 as described here. So you'd need to install that version of Azure CLI on host machine and in the token provider's Docker image.

Here is the least effort code for this token provider. Let's build it and run it.



docker build -t dev:token -f .\Dockerfile .
<truncated>
docker run -e ASPNETCORE_URLS=http://+:80 -p 8080:80 -v C:\Users\OskarMamrzynski\.azure:/root/.azure:rw dev:token
<truncated>
      Now listening on: http://[::]:80
<truncated>
      Request finished HTTP/1.1 GET http://localhost:8080/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F - - - 200 - application/json;+charset=utf-8 1196.6707ms

My container listens on localhost:8080 on the host machine, so we can curl it and obtain a token impersonating the user logged in with Azure CLI:



curl 'http://localhost:8080/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -H Metadata:true -s
{"access_token":"eyJ0<redacted>","refresh_token":"","expires_in":5335.8911044,"expires_on":1659150718,"not_before":1659145382,"resource":"https://management.azure.com/","token_type":"Bearer"}

Q: Fine, but your proxy container is still not using that 169.254.169.254 IP address. How are you going to get that working?
A: We can create a local Docker network and assign a static IP to our token provider container. Consumer app would also run in this network.



docker network create omtest --subnet=169.254.0.0/16
<truncated>
docker run -d --net omtest --ip 169.254.169.254 -e ASPNETCORE_URLS=http://+:80 -p 8080:80 -v C:\Users\OskarMamrzynski\.azure:/root/.azure:rw dev:token
<truncated>
docker run -it -e AZURE_KEY_VAULT_URI=https://<redacted>.vault.azure.net/ --net omtest dev:consumer
mysecret=secretsauce (AzureKeyVaultConfigurationProvider)

In summary
You can use Azure CLI in conjunction with this token provider container just always running there in the background on your laptop. Any Docker containers you spin up locally will be able to use this spoofed token provider endpoint as long as they are on the same Docker network. This makes it appear as if the app is assigned a Managed Identity.