DEV Community

Cover image for Adding a Second User-Assigned Managed Identity to an Azure VM Breaks Apps That Never Named Their Identity
KBY Technologies
KBY Technologies

Posted on Originally published at kbytechnologies.com

Adding a Second User-Assigned Managed Identity to an Azure VM Breaks Apps That Never Named Their Identity

Originally published on KBY Technologies.

If you ship code to Azure virtual machines that fetch tokens from the Instance Metadata Service, this is a trap that hits you the day someone else attaches a second user-assigned managed identity to the same VM. We cover which of your apps and scripts are exposed, the read-only commands that show it, and how to fix it without adding a system-assigned identity or a client secret.


Symptom

An application on an Azure virtual machine that has read from Key Vault or Storage for months stops getting tokens. The change that day was routine: someone attached a second user-assigned managed identity to the VM so another workload could use it. The first identity and its role assignments are untouched. The application never said which identity it wanted, and with one user-assigned identity it didn’t have to. With two, the Instance Metadata Service (IMDS) has to be told which one to use, so the token request fails. Fix the application first by giving it the client ID of the identity that holds its roles. Don’t turn on the system-assigned identity, and don’t go back to a client secret.

This applies to a VM with no system-assigned identity that went from one user-assigned identity to two or more. A VM that already has a system-assigned identity behaves differently, as covered below.

Why attaching an identity looks safe

Microsoft supports this setup. A resource can have a system-assigned identity and one or more user-assigned identities, and Microsoft’s best-practice guidance describes sharing one user-assigned identity across many resources so fewer role assignments are needed. Assigning an identity to a VM changes no role assignments and detaches nothing, so in a change review it reads as additive and low risk. The person making the change usually owns the new workload, not the application that breaks.

Root cause

The application’s token request names no identity, and Microsoft documents that a VM with more than one user-assigned identity needs one named. Code on the VM gets a managed identity token from the IMDS endpoint http://169.254.169.254/metadata/identity/oauth2/token. Microsoft documents three optional query parameters that pick the identity: client_id, object_id and msi_res_id. Each is marked as required if the VM has multiple user-assigned managed identities. A request that names none of them worked while there was only one identity to choose. Attach a second, and the same request no longer says which identity to use.

The Azure Identity libraries pass whatever identity they were configured with. In Python, DefaultAzureCredential takes managed_identity_client_id, which defaults to the AZURE_CLIENT_ID environment variable; if neither is set, it uses a system-assigned identity. In .NET, AZURE_CLIENT_ID, if defined, is the default for ManagedIdentityClientId. An application with neither configured sends no identity at all.

IMDS reports request errors with a 4xx status and a JSON body with error and error_description. Its documentation calls 4xx errors design-time errors and says not to retry them, so retry logic will not get the application working again.

Impact

Every process on the VM that asks for a token without naming an identity fails the next time it needs a new token: the application, scheduled scripts that call IMDS with curl or Invoke-RestMethod, and any agent built the same way. The new workload usually works, because whoever set it up named its identity, so the change looks fine to the person who made it.

Three tempting fixes that make it worse

The first is to turn on the VM’s system-assigned identity. The Python library documents that, with no client ID, it uses a system-assigned identity, so token requests succeed again. But the token now belongs to a new identity with none of the application’s roles, so the failure moves to the Key Vault or Storage call. The next step under pressure is to grant that identity the same roles. Two identities then hold the same access, and Microsoft notes that role assignments are not deleted automatically when an identity is deleted.

The second is to fall back to a service principal secret in AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET. DefaultAzureCredential tries a service principal from environment variables before managed identity, so this works. It also brings back the secret that managed identities exist to remove, and you now have to store and rotate it. The same trade appears when a storage client goes back to account keys, covered in fixing Storage 403 errors after Shared Key access is disabled.

The third is to remove the new identity. That restores the old behaviour and breaks the workload it was added for, and the next person to attach an identity will hit the same failure.

How to confirm the diagnosis

These checks change nothing. Replace rg-app, app01 and the IDs with your own.

  1. List the VM’s identities. You are looking for a type with no SystemAssigned and two or more entries under userAssignedIdentities.
   az vm identity show --name app01 --resource-group rg-app
Enter fullscreen mode Exit fullscreen mode
  1. Get each identity’s client ID and principal ID from its resource ID.
   az identity show --ids <identity-resource-id> \
     --query "{clientId:clientId, principalId:principalId}"
Enter fullscreen mode Exit fullscreen mode
  1. Find the identity that holds the application’s roles. Without --all, only subscription-scope assignments are listed.
   az role assignment list --assignee <principal-id> --all --output table
Enter fullscreen mode Exit fullscreen mode
  1. On the VM, request a token twice: once as the application does, and once with that identity’s client ID. If the first fails and the second returns an access_token, the diagnosis is confirmed. Don’t paste the token into a ticket.
   curl -s -H Metadata:true \
     'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F'
   curl -s -H Metadata:true \
     'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F&client_id=<client-id>'
Enter fullscreen mode Exit fullscreen mode

Correction

Name the identity in the application before you change anything on the VM. That way the VM never has an identity layout the application can’t use. You need access to the application’s configuration. For the identity changes, you also need Virtual Machine Contributor, plus Managed Identity Operator to assign a user-assigned identity.

  1. Save the current state: az vm identity show --name app01 --resource-group rg-app > vm-identity-before.json, and a copy of the application’s configuration or service definition.
  2. Pin the application to the identity that holds its roles. Set AZURE_CLIENT_ID to that identity’s client ID in the application’s environment, or pass it in code as managed_identity_client_id (Python) or ManagedIdentityClientId (.NET). Don’t add AZURE_TENANT_ID or AZURE_CLIENT_SECRET. Restart the application. If token errors continue, check that the client ID matches an identity listed by az vm identity show before you try anything else.
  3. Do the same for every other process on the VM that requests tokens. Scripts that call IMDS directly need client_id= in the query string.
  4. If you removed the second identity to get running again, assign it back only after steps 2 and 3:
   az vm identity assign -g rg-app -n app01 --identities <identity-resource-id>
Enter fullscreen mode Exit fullscreen mode

If anything on the VM starts failing to get a token, remove it again with az vm identity remove -g rg-app -n app01 --identities <identity-name> and find the process that still sends no identity.

Validation

The fix holds when every workload on the VM gets a token for its own identity and nothing extra was granted along the way.

  • The application reads from Key Vault or Storage again, with no token errors in its log.
  • The IMDS request with client_id returns a token, and az vm identity show still lists every identity the VM’s workloads need.
  • No system-assigned identity or client secret was added on the way. If az role assignment list on an identity added during the incident shows copies of the application’s roles, remove them once the pinned application has run cleanly, so only one identity holds that access.

If the pinned request gets a token but the resource still answers 403, the problem is authorisation, not identity selection. The Owner role on an RBAC Key Vault returning Forbidden is one case to check next.

Rollback

  • Application setting: remove AZURE_CLIENT_ID or the code option and restart. This is only safe while the VM has a single user-assigned identity.
  • Identity assignment: az vm identity remove -g rg-app -n app01 --identities <identity-name> removes an identity you added, and az vm identity assign puts back one you removed, using the IDs saved in vm-identity-before.json.

Stop it happening on the next change

Configure the client ID in every application that uses a user-assigned identity, even when the VM has only one. The setting changes nothing today, and it means the next identity anyone attaches can’t break the application. Before you approve a change that attaches an identity to a VM, ask the owners of everything already running there whether their code names its identity. The two token requests above show it without changing anything.


Read the original, with the rest of the Config Traps library, on KBY Technologies.

Top comments (0)