DEV Community: Tochukwu Ohabuike D.

Automate Azure VM Password Rotation with PowerShell and Azure DevOps

Tochukwu Ohabuike D. — Tue, 07 Nov 2023 07:39:49 +0000

Virtual Machines are ideal for hosting applications and sensitive databases. Regularly changing passwords for virtual machines is vital to prevent unauthorized access. However, manually updating passwords across multiple VMs can be tedious and error-prone. Automating password rotation using PowerShell and Azure DevOps improves efficiency.

Why the drama?

Imagine this: you have lots of VMs hosting sensitive data, and for some reason, you have a disgruntled ex-employee who still has access to some of the VMs. Having a script that generates and sets new random passwords at intervals could save you lots of mishaps. Other reasons are:

Mitigating Security Risks: Password rotation shortens the attacker’s window. Even if a password is compromised, its limited validity minimizes potential damage.
Compliance Requirements: Some industry regulations demand password rotation for security. Non-compliance risks severe penalties.
Proactive Security: By automating password rotation, you can ensure a proactive approach to security, rather than a reactive one that responds only after a security breach.

Less time wasted on repetitive manual processes means more time for higher value work that only humans can do.

Tools to achieve this task:

PowerShell. To generate and set random passwords for Azure VMs.
Azure DevOps. Pipeline to run the PowerShell script on a schedule.
Azure KeyVault. To store the generated password so it can be accessed.

Flow of the solution:

I assume you already have an Azure subscription and Azure DevOps account set up, so lets get to it straight away…

Step 1: Set up App Registration

App Registration is required to setup service connection for authentication from Azure DevOps to the subscription(s) hosting our VMs and KeyVault.

Create a new one or use an existing registration.
In your portal, visit Microsoft Entra ID >> App Registrations.
Once created, note the Tenant ID and Application ID.
In Certificates and Secrets, create a client secret and note the value.

Step 2: Set up Service Connection on Azure DevOps

Using the details we retrieved in step 1, we would set up a service connection in Azure DevOps.

In your Azure DevOps, create or navigate to your existing organization
Create a new project >> select project settings at the bottom left corner
In the pipelines section, select service connection and click create new
Select Azure Resource Manager >> select Service Principal (manual)
Fill the fields using details from step 1 and your subscription details (id and name),
Name the service connection >> click verify to validate the details.
Once verified, check the ‘grant access to all pipelines” box >> click Verify and save. It should be similar to the below:

You can add as many connections for other VMs that may be located in other subscriptions.

Step 3: Set Up Azure Key Vault

We use the KeyVault to store the newly generated password so we can access it securely.

In your portal, navigate to KeyVault, create a new or use an existing one. In ‘Secrets,’ click ‘Generate/Import.’ Provide a name (preferably your VM name) and add a placeholder password to the secret field, the script would modify it. Click create.

In the overview page, choose ‘Access Policies.’ Under ‘Secret permissions,’ select all. In ‘Principal,’ find and select the earlier created App Registration. Review and create. This grants ‘KeyVault permission’ to your app.

Step 4: The PowerShell Script

The script simplifies steps for easy understanding with enough comments for clarity.

It has three parts…

A primary script called by the pipeline, it fetches the VMs in a subscription.
The script then calls a function that generates the random password, updates the Azure KeyVault and VM passwords.
A JSON file stores VM details for accurate KeyVault secret matching.

Let’s dive in, beginning with the JSON file containing KeyVault details:

{
    "KeyVaultConfigurations": [
        {
        "VMName": "vm1", //your virtual machine name
        "KeyVaultName": "my-secretss", // your key vault name
        "SecretName": "vm1-secret" // name of the secret used to store your password
        },
        {
        "VMName": "vm2",
        "KeyVaultName": "my-secretss",
        "SecretName": "vm2-secret"
        }   
    ]
}

The JSON file is named ‘configuration.json’.

Next, we create the script called by the pipeline that fetches the VMs from the subscription and calls the function, lets call it “secret-rotate.ps1”:

# Get all VMs in the current subscription context
$virtualMachinesDetails = Get-AzVM | Where-Object { $_.StorageProfile.OSDisk.OSType -eq "Windows" }

# Dot-source the function script to load the function
. ".\vm-kv-updatefunc.ps1"

# Call the function with your parameters
Update-VMPasswords -VirtualMachinesDetails $virtualMachinesDetails

Next, we create the function to generate a strong password and update the VM password and KeyVault, lets call it “vm-kv-updatefunc.ps1”:

# Stage 1: Function to generate password
function Set-RandomPassword {
    param (
        [int]$length = 16
    )

    $lowerCase = "abcdefghijklmnopqrstuvwxyz"
    $upperCase = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    $numbers = "0123456789"
    $symbols = "!@#$%^&*()_-+=<>?/[]{}|"

    $charPool = $lowerCase + $upperCase + $numbers + $symbols
    $password = -join ($charPool.ToCharArray() | Get-Random -Count $length)

    return $password
}

function Update-VMPasswords {
    param (
        [array]$VirtualMachinesDetails
    )

    # Stage 2: Loop through each VM, update keyvault and VM password
    $VirtualMachinesDetails | ForEach-Object {

        # Generate new password
        $generatedPassword = Set-RandomPassword

        $vmName = $_.Name

        # Load VM configurations from the JSON file
        $configurations = Get-Content -Raw -Path "./configurations.json" | ConvertFrom-Json
        $vmConfigurations = $configurations.KeyVaultConfigurations

        # Find the corresponding configuration for the VM
        $vmConfiguration = $vmConfigurations | Where-Object { $_.VMName -eq $vmName }

        # Check if keyvault details exisit for the vm
        if ($null -eq $vmConfiguration) {
            Write-Host "No keyvault configuration found for this VM: $vmName"
            continue
        }
        # Retrieve key vault details
        $keyVaultName = $vmConfiguration.KeyVaultName
        $secretName = $vmConfiguration.SecretName

        # Update the VM and keyvault with the new password
        try {
            $securePassword = ConvertTo-SecureString -String $generatedPassword -AsPlainText -Force
            Set-AzKeyVaultSecret -VaultName $keyVaultName -Name $secretName -SecretValue $securePassword

            Write-Host "Password for user $userName on VM $vmName successfully updated in Key Vault."

            $extensionParams = @{
                'ResourceGroupName' = $_.ResourceGroupName
                'Location' = $_.Location
                'VMName' = $_.Name
                'Name' = 'VMAccessAgent'
                'TypeHandlerVersion' = '2.0'
                'Credential' = New-Object System.Management.Automation.PSCredential ($vmName, $securePassword)
                'ForceRerun' = $true
            }

            Set-AzVMAccessExtension @extensionParams

            Write-Host "Password for VM: $vmName updated successfully."
            Write-Host -ForegroundColor Cyan ".........................................................................................\n"
        } catch {
            Write-Host "An error occurred while updating the Virtual Machine Password: $($_.Exception.Message)"
        }            
    }
}

Step 5: Set Up Azure Pipeline

The pipeline yaml file to run the script at intervals via the Azure DevOps pipeline:

schedules:
- cron: '0 0 * * 7'  # Run every every week on a Sunday
  displayName: 'Every 1 minute schedule'
  branches:
    include:
    - <replace with the branch you want to trigger>

pr:
- '*'

variables:
- name: azureSubscription 
  value: '<replace with the name of your service connection>'

pool: 
  vmImage: 'windows-latest'

stages:
- stage: VirtualMachineSecretUpdate

  jobs:
  - job: VMSecretUpdate

    steps:
    - task: AzurePowerShell@5
      inputs:
        azureSubscription: $(azureSubscription)
        scriptType: filePath
        scriptPath: ./secret-rotate-scripts/secret-rotate.ps1 #replace with the path to your script
        azurePowerShellVersion: latestVersion
        pwsh: true

Put the script and JSON file and script inside a folder. Let your folder structure look similar to this in Azure DevOps:

Final Step: Confirm Setup Works

I already have two virtual machines and their passwords are stored in a key vault. You can create and connect to yours using: Create Azure VM

Once the pipeline triggers, it first updates the key vault and then update the VM password. You should get an output similar to the image below:

With the new password updated in the key vault (current version) I can now login into the VM.

Once you click the “current version” you would see the new password.

Conclusion

Automation makes easy work of complex and laborious tasks. By leveraging PowerShell and Azure DevOps to rotate credentials automatically, organizations can painlessly enforce strict password policies across their environments.

Next, I will cover updating each users in a VM (a VM can have multiple users). Until then, keep automating…

How to Automate Blob Storage Cleanup Using PowerShell and Azure Automate

Tochukwu Ohabuike D. — Sat, 21 Oct 2023 08:23:38 +0000

Blob storage management is a bit like tidying up a busy workspace. Files pile up over time, messing up your storage space with unnecessary items. This article offers a detailed automated solution to clean up older blobs.

Why the drama?

1. Cost Control: Blob storage expenses can accumulate, particularly when retaining unnecessary files. Automated cleanup is budget-friendly.

2. Enhanced Performance: Fewer files mean quicker access times. Your applications and users will appreciate the faster response.

3. Compliance: Certain industries demand data retention for specific periods. Automated cleanup ensures compliance.

Tools to achieve this task:

PowerShell. The script to remove unused data from the blob storage.
Azure Automate. To allow the script run at defined intervals.
Azure KeyVault. To store the Azure Storage access key.

Flow of the solution:

Step 1: Set up App Registration

App Registration is required for authentication from Azure Automate runbook to access Blob storage and KeyVault. Create a new one or use an existing registration. In your portal, visit Microsoft Entra ID >> App Registrations. Once created, note the Tenant ID and Application ID. Then, in Certificates and Secrets, create a client secret and note the value.

Step 2: Set up KeyVault

It’s not good practice to embed sensitive information in scripts, which is why we use KeyVault. Begin by going to the Azure Blob Storage, then Access Keys. Click ‘Show’ and copy any of the access keys.

Next, in KeyVault, create a new or use an existing one. In ‘Secrets,’ click ‘Generate/Import.’ Provide a name and add the storage account’s access key to the secret field, then create it.

Step 3: The PowerShell Script

This is the heart of the operation. I will paste the whole script with just about enough comments to understand it.

# STAGE 1: Function to format duration

function Format-Duration {
    param (
        [double]$durationInHours
    )

    $days = [math]::Floor($durationInHours / 24)
    $hours = [math]::Floor($durationInHours % 24)

    if ($days -gt 0) {
        return "$days days $hours hours"
    } elseif ($hours -gt 0) {
        return "$hours hours"
    } else {
        $minutes = [math]::Round($durationInHours * 60)
        return "$minutes minutes"
    }
}

# STAGE 2: Authenticate using App Registration

$ApplicationId = "Replace-with-your-application-ID"
$TenantId      = "Replace-with-your-Tenant-ID"

# Retrieve secrets from Azure Automate Credentials
$myCredential = Get-AutomationPSCredential -Name "Replace-with-your-credential-name"
$password = $myCredential.GetNetworkCredential().Password

$SecurePassword = ConvertTo-SecureString -String $Password -AsPlainText -Force
$Credential     = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $ApplicationId, $SecurePassword
Connect-AzAccount -ServicePrincipal -TenantId $TenantId -Credential $Credential

# STAGE 3: Retrieve blob key from Azure Key Vault 

$keyVaultName   = "Replace-with-your-keyvault-name"
$secretName     = "Replace-with-your-keyvault-secret-name"
$keyVaultSecretBlobKey = Get-AzKeyVaultSecret -VaultName $keyVaultName -Name $secretName -AsPlainText


# Stage 4: Set your Azure Storage account name

$storageAccountName = "Replace-with-your-storage-account-name"
$containerName = "Replace-with-your-container-name"

# STAGE 5: Connect to storage and delete blobs greater than 30 days

# Get the current date and set the blob age threshold to 30 days (in hours)
$currentDate = Get-Date
$blobAgeThreshold = 30 * 24  # 30 days (in hours)

# Connect to Azure Storage account
$context = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $keyVaultSecretBlobKey

# Get a list of all blobs in the container
$Blobs = Get-AzStorageBlob -Container $containerName -Context $Context
$totalDeleted = 0

# Loops through each of the container and checks for blobs > 30days and remove
$Blobs | ForEach-Object {

    $blobAge = $_.LastModified.DateTime
    $ageDiff = ($currentDate - $blobAge).TotalHours

    if ($ageDiff -gt $blobAgeThreshold) {

        $blobName = $_.Name
        Write-Host -ForegroundColor Yellow "Name of Blob: $blobName; Age of blob: $(Format-Duration -durationInHours $ageDiff)..."
        Write-Host -ForegroundColor Red "Deleting..."

        try {
            Remove-AzStorageBlob -Context $context -Container $containerName -Blob $blobName -Force
            Write-Host -ForegroundColor Green "Delete successful"
            Write-Host -ForegroundColor Cyan "........................................................................................."
            $totalDeleted ++
        } catch {
            Write-Host "Deletion failed for $blobName. Error: $_"
        }
    }  
 }

Write-Host -ForegroundColor Green "Successfully deleted: $totalDeleted"

Stage 1 formats the blob’s age for user display. Stage 2 connects to Azure using App registration details we obtained earlier, with the secret handling discussed in the next step. Stages 3, 4, and 5 handle the remaining tasks

Step 4: Set up Azure Automate Runbook

With Azure Automate Runbook, we schedule script execution. To create a runbook, go to your portal and set up an Automation account. Store the client secret from App Registration in Azure Automate credentials. In your Automation account, select ‘Credentials’ under Shared Resources on the left pane.

On the ‘Credentials’ page, choose ‘Add a credential.’ In the ‘New Credential’ pane, provide a name per your naming standards. Enter the access ID in the ‘User name’ field. For both password fields, use the client secret value.

The above is what we retrieved in stage 2 of our script. Next, we create a runbook, give it a name and hit create.

Next, click on edit in portal at the top bar

Now, paste the PowerShell script in the code field and hit Save >> Publish.

Next, click on schedules >> add a schedule >> fill to your taste how often you want it to run >> Create

Then go back to my current schedule and add the recently created schedule, then click ok.

Now we are done, we wait for it to trigger and then we confirm if it deleted in the blob.

Last step: Confirm our set up works

The state of my blob storage before the Azure Automate runbook triggered:

When the scheduled time triggers, check the ‘Jobs’ section for the status:

If you click on it and review the output, you’ll find the details as configured in our script:

Refreshing my blob container, no blobs are found:

In Conclusion: Embrace a Tidy Cloud

In the ever-expanding realm of the cloud, keeping things tidy isn’t just a suggestion; it’s a necessity. With the blend of PowerShell and Azure Automate, we can simplify the process of managing your cloud storage. It helps keep your cloud organized, efficient, and cost-effective.

Happy Blob Busting!

Managing Azure VMs Using PowerShell Effectively

Tochukwu Ohabuike D. — Thu, 12 Oct 2023 07:07:22 +0000

Imagine this: You’re responsible for overseeing a range of virtual machines (VMs) in your Azure environment. However, you’re also someone who values efficiency. Why spend time on manual tasks when automation can make your life easier? In this article, we’ll introduce a script that streamlines the process of stopping, starting, and checking the status of your Azure VMs.

Why the drama?

Before we dive into the script, let’s answer a crucial question: why perform these operations on Azure VMs? In the realm of the cloud, managing Azure VMs can be similar to tending to digital pets, especially when handling a multitude of them without a fixed schedule for starting, stopping, or status checks.

Cost Control: Every second your VMs run, coins are drained from your Azure treasure chest. Stopping idle VMs saves you shiny gold.
Maintenance Marvel: VMs need TLC. Updates, patches, and configurations often require them to be stopped and started.
Health and Sanity: Sometimes, VMs decide to take a stroll on the wild side. Checking their status ensures awareness of your VM state.

Let’s script

To simplify the process, we’ll utilize Azure Cloud Shell. Let’s assume you have a substantial number of VMs, possibly hundreds distributed across various resource groups and environments (e.g., production, development). An approach will be to filter the VMs for operations based on pre-existing tags. In this article, we’ll instead us an Excel sheet containing the list of VMs for querying. Here is mine called VMlists.csv:

Start VMs:

The below script starts the VMs based on the user input. It loops through each VMs in the CSV file. The operation would be run based on the OS type and the action (stop/start/status) specified.

Write-Host "`nWelcome, this script allows you to start, stop and check status of multiple VMs in various resource groups in parallel. `n" 

Write-Host "`nEnter action (start/stop/status):"

$Action = Read-Host

Write-Host "`nSpecify environment(prod/dev):"

$OS_TYPE = Read-Host

$VirtualMachinesDetails = Import-Csv .\VMLists.csv #Imports csv file from the specified path

if ($Action -eq "start"){

    Write-Host "Please specifiy the environment (Prod/Dev)"

    $VM_Environment = Read-Host

    Write-Host "`nStarting all $OS_TYPE Virtual machines in the $VM_Environment environment..."

    $VirtualMachinesDetails | Where-Object {($_.VM_OS -eq $OS_TYPE) -and ($_.VM_Environment -eq $VM_Environment)} | ForEach-Object -parallel { #filters and fetches the csv files content based on the OS type.

        $checkIfVMExistInRG = get-azvm -ResourceGroupName $_.VM_RG -name $_.VM_Name -ErrorVariable notPresent -ErrorAction SilentlyContinue # Checks if the VM exists in the RG

        if ($notPresent){
            continue
        }
        else {
            Start-AzVM -ResourceGroupName $_.VM_RG -Name $_.VM_Name
        }
    } -ThrottleLimit 10
}

The VMs are started in parallel (10 at once, instead of one by one). The limit was set to 10 to manage your compute resources.

Stop VMs:

The below script loops through the VMs based on user inputs and stop the VMs.

elseif ($Action -eq "stop"){

    $OS_TYPE = "Windows"

    Write-Host "Please specifiy the environment (Prod/Dev/Test):"

    $VM_Environment = Read-Host

    Write-Host "`nStopping all $OS_TYPE Virtual machines in the $VM_Environment environment..."

    $VirtualMachinesDetails | Where-Object {($_.VM_OS -eq $OS_TYPE) -and ($_.VM_Environment -eq $VM_Environment)} | ForEach-Object -parallel {

        $checkIfVMExistInRG = get-azvm -ResourceGroupName $_.VM_RG -name $_.VM_Name -ErrorVariable notPresent -ErrorAction SilentlyContinue

        if ($notPresent){
            continue
        }
        else {
            Stop-AzVM -ResourceGroupName $_.VM_RG -Name $_.VM_Name -Confirm:$false -ErrorAction SilentlyContinue -Force
        }
    } -ThrottleLimit 10
}

Get VM status:

The below script loops through the VMs and get their current state. It checks if they are running properly or in a bad state.

elseif($Action -eq "status"){

    Write-Host "`nGetting status of all Virtual machines..."

    $Rg_Exist = @()
    $Vm_Exist = @()
    $vmStatuses = @()

    $VirtualMachinesDetails | ForEach-Object {
        if ($Rg_Exist.Contains($_.ResourceGroupName) -and $Vm_Exist.Contains($_.VM_Name)) {
            return
        }
        else {

            $VMstatus = Get-AzVM -Name $_.VM_Name -ResourceGroupName $_.ResourceGroupName -Status
            $VMdetails = Get-AzVM -Name $_.VM_Name -ResourceGroupName $_.ResourceGroupName
            $Environment = $_.VM_Environment

            $vmInfo = [PSCustomObject]@{
                ResourceGroupName   = $vmStatus.ResourceGroupName
                Name                = $vmStatus.Name
                Environment         = $Environment
                Location            = $VMdetails.Location
                OsType              = $VMdetails.StorageProfile.OsDisk.OsType
                Disk_Status         = $vmStatus.Disks.Statuses.code
                VMAgentStatus       = $vmStatus.VMAgent.Statuses.DisplayStatus
                PowerState          = $vmStatus.Statuses[1].DisplayStatus     
                Provisioning        = $vmStatus.Statuses[0].DisplayStatus
                MaintenanceAllowed  = $vmStatus.MaintenanceRedeployStatus.IsCustomerInitiatedMaintenanceAllowed
            }
            $vmStatuses += $vmInfo

            $Rg_Exist += $_.ResourceGroupName
            $Vm_Exist += $_.VM_Name
        }        
    }
    $vmStatuses | Format-Table -AutoSize
}
else {
    Write-Host "action not recognized"
}

You can modify the script based on your use case. For the full script, check my GitHub: https://github.com/Toch-vybe/Powershell-scripts/blob/9a993126ba967746db9f0bad0998b1fd933a7004/vm-manager.ps1

Now let’s try the script out in the cloud shell.

You will have to create or select an existing storage account where your scripts would be stored.

Upload the csv file containing the VMs and the PowerShell script into the cloud shell:

Now open the editor to confirm your uploaded files:

Next, run the script in the shell and follow the prompt using:

./<the name of your script>.ps1

Below is how it should look like when you run the scripts:

Conclusion

In the world of Azure, scripts are valuable tools that can simplify your administrative tasks. They allow you to manage your Azure resources efficiently, saving time and effort. So, I encourage you to embrace the power of automation. Your Azure VMs will benefit, and you’ll free up more time for other pursuits, whether it’s conquering new digital challenges or enjoying some well-deserved relaxation.

Until next time, keep scripting, and continue to excel in your Azure endeavors!