Automatically provision Azure Monitor alert rules for your Azure solutions!

Toon Vanhoutte — Tue, 10 Sep 2019 13:47:27 +0000

This article is part of #ServerlessSeptember. You'll find other helpful articles, detailed tutorials, and videos in this all-things-Serverless content collection. New articles are published every day — that's right, every day — from community members and cloud advocates in the month of September.
Find out more about how Microsoft Azure enables your Serverless functions at https://docs.microsoft.com/azure/azure-functions/.

Monitoring is an extremely important aspect of cloud solutions. This blog shows how you can automatically create Azure Monitor alerts for each resource that is created within your subscription. Based on the resource type, the appropriate alert rules are automatically configured. The solution uses Azure Event Grid to react on resource events inside your subscription, Logic Apps to orchestrate the workflows and Azure Function's PowerShell support (in preview) to apply the alert rules, based on best practices. This is the high-level solution diagram, more details are explained below.

Create the Event Grid subscription (1)

Event Grid allows you to react on events that happen within your subscription. In this case, we want to trigger a process when a resource gets created. After consulting the documentation, the Resource Write Success on subscription level seems the correct event type to subscribe on. In the advanced filters, exclude the Metric Alert and Deployment events, as they create too many unwanted events.

Configure the Alert Creator Orchestrator Logic App as the one that receives and processes all events.

Configure the Alert Creator Orchestrator Logic App (2)

The Logic App subscribes with an HTTP request on the incoming events. First of all, an Azure Function is invoked, to which all complex logic is offloaded. In case the outcome of the Function indicates that a certain resource type is not configured for alert rule creation, an email is sent to the responsible, so he/she can take action.

Create the Azure Monitor alert rules (3)

The Alert Creator Azure Function is reponsible to create Azure Monitor alert rules, based on the resource type.

In order to make this fully configurable, a storage account is used. This is the logic that is applied by the Azure Function:

Check if the resource type is blacklisted. In case it is on the blacklist (= a JSON config file), the event can be safely ignored. Return RESOURCETYPE_BLACKLISTED as a response.
If the resource type is not blacklisted, search for an ARM template that has the name of the resource type. Deploy the ARM template that has three input parameters and return ALERTS_CREATED:
-resourceId: the unique Id of the Azure resource
-actionGroupId: the Id of the Azure Monitor action group that handles the alerts
-enabled: a flag to enable/disable alert rules (typically you don't want them enabled for DEV environments)
If no corresponding ARM template is found, return the RESOURCETYPE_NOT_CONFIGURED response.

The Azure Function is written in PowerShell.

using namespace System.Net

# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)

try
{
    #Retrieve input parameters from request body
    $resourceName = $Request.Body.resourceName
    $resourceId = $Request.Body.resourceId
    $resourceType = $Request.Body.resourceType

    #Set Azure Auto Monitor variables from app settings
    $storageAccountConnectionString = $env:storageAccountConnectionString
    $containerName = $env:containerName
    $blackListedResourceTypesFileName = $env:blackListedResourceTypesFileName
    $resourceTypeTemplatesFolderName = $env:resourceTypeTemplatesFolderName
    $actionGroupId = $env:actionGroupId
    $resourceGroupName = $env:resourceGroupName
    $alertsEnabled = [boolean]$env:alertsEnabled

    #Initialize variables
    $resourceType = $resourceType.ToLower() 
    $supportedResourceTypes = @()
    $ErrorActionPreference = "Stop"

    #AzLogin using Managed Service Identity
    Connect-AzAccount -Identity

    #Connect to storage account
    $storageContext = New-AzStorageContext -ConnectionString $storageAccountConnectionString  

    #Get all blacklisted resource types
    Get-AzStorageBlobContent -Container $containerName -Blob $blackListedResourceTypesFileName -Destination $blackListedResourceTypesFileName -Force -Context $storageContext  
    $blackListedResourceTypes = Get-Content $blackListedResourceTypesFileName | ConvertFrom-Json  

    if($blackListedResourceTypes -contains $resourceType)
    {
        Write-Host "Blacklisted resource type, no further actions"
        $body = "RESOURCETYPE_BLACKLISTED"
    }
    else
    {
        #Get all available resource type templates
        $resourceTypeTemplateFiles = Get-AzStorageBlob -Container $containerName -Prefix "$($resourceTypeTemplatesFolderName)/" -Context $storageContext
        foreach ($template in $resourceTypeTemplateFiles | Select-Object -ExpandProperty Name){
            $supportedResourceTypes += $template.Replace("$($resourceTypeTemplatesFolderName)/", "").Replace(".json","").Replace("_","/")
        }

        if($supportedResourceTypes -contains $resourceType)
        {
            $templateName = $resourceTypeTemplatesFolderName + "/" + $resourceType.Replace('/', '_').ToLower() + ".json"
            $templateSasToken = New-AzStorageBlobSASToken -Container $containerName -Blob $templateName -Permission r -StartTime (Get-Date).AddDays(-1.0) -ExpiryTime (Get-Date).AddDays(1.0) -Context $storageContext 
            $templateUrl = $storageContext.BlobEndPoint + $containerName + "/" + $templateName + $templateSasToken
            $guid = [GUID]::NewGuid()

            New-AzResourceGroupDeployment -Name "AzureAutoMonitorAlerts_$($guid)" -ResourceGroupName $resourceGroupName -TemplateUri $templateUrl -resourceId $resourceId -actionGroupId $actionGroupId -enabled $alertsEnabled

            $body = "ALERTS_CREATED"
        }
        else
        {
            $body = "RESOURCETYPE_NOT_CONFIGURED"
        }
    }

    $status = [HttpStatusCode]::OK
}
catch
{
    $body = $_.Exception.Message
    $status = [HttpStatusCode]::InternalServerError 
}

# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
    StatusCode = $status
    Body = $body
})

This Azure Function uses Managed Service Identity to authorize against the Azure Resource Manager:

The Azure Function's Managed Service Identity is granted Contributor rights on the subscription:

This is an example of an ARM template that creates Azure Monitor alerts rules for all Logic Apps. You should create a similar template for each resource type you want to monitor (e.g. SQL Database disk space - memory - cpu).

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "resourceId": {
        "type": "string"
    },
    "actionGroupId": {
        "type": "string"
    },
    "enabled" : {
      "type": "bool"
    }
  },
  "variables": {
    "resourceName" : "[split(parameters('resourceId'), '/')[sub(length(split(parameters('resourceId'), '/')), 1)]]"
  },
  "resources": [
    {
      "type": "microsoft.insights/metricAlerts",
      "name": "[concat('Logic Apps Failed Runs - ', variables('resourceName'))]",
      "apiVersion": "2018-03-01",
      "location": "global",
      "properties": {
        "description": "[concat('There are failed runs for the Logic App ', variables('resourceName'), '.')]",
        "severity": 3,
        "enabled": "[parameters('enabled')]",
        "scopes": [
          "[parameters('resourceId')]"
        ],
        "evaluationFrequency": "PT1M",
        "windowSize": "PT5M",
        "criteria": {
          "odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria",
          "allOf": [
            {
              "name": "Failed runs",
              "metricName": "RunsFailed",
              "dimensions": [],
              "operator": "GreaterThan",
              "threshold": "0",
              "timeAggregation": "Total"
            }
          ]
        },
        "actions": [
          {
            "actionGroupId": "[parameters('actionGroupId')]",
            "webHookProperties": {}
          }
        ]
      }
    },
    {
      "type": "microsoft.insights/metricAlerts",
      "name": "[concat('Logic Apps Failed Triggers - ', variables('resourceName'))]",
      "apiVersion": "2018-03-01",
      "location": "global",
      "properties": {
        "description": "[concat('There are failed triggers for the Logic App ', variables('resourceName'), '.')]",
        "severity": 3,
        "enabled": "[parameters('enabled')]",
        "scopes": [
            "[parameters('resourceId')]"
        ],
        "evaluationFrequency": "PT1M",
        "windowSize": "PT5M",
        "criteria": {
          "odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria",
          "allOf": [
            {
              "name": "Failed triggers",
              "metricName": "TriggersFailed",
              "dimensions": [],
              "operator": "GreaterThan",
              "threshold": "0",
              "timeAggregation": "Total"
            }
          ]
        },
        "actions": [
          {
            "actionGroupId": "[parameters('actionGroupId')]",
            "webHookProperties": {}
          }
        ]
      }
    }
  ],
  "outputs": {
  }
}

As an example, the following Azure Monitor alert rules were automatically added to my newly created Logic App:

Handle the Azure Monitor Alerts (4)

The Azure Monitor action group - to which all alert rules point - is configured to send the alert events to a single Logic App:

Currently, this Logic App is configured to send just an alert email. In future extensions of this approach, this alerting can be extended to send SMS, to call someone, to start a Skype conversation or to notify people through Teams / Slack channels.

Conclusion

By combining the magic of Event Grid, Logic Apps and Azure Functions, you can easily build a budget-friendly solution that automates the monitoring of your Azure solutions. Developers can focus on creating powerful Azure solutions, without the need to worry about monitoring.