DEV Community: Mike Richter

ML to AI with ML.Net and Azure

Mike Richter — Mon, 26 Apr 2021 23:59:15 +0000

Use ML.Net to build a Sentiment Analysis API

Goal

This post will show you how easy it is to build a Sentiment Analysis Prediction Model with ML.NET and Model Builder in Visual Studio.

We will then deploy that prediction model as an API and consume it as AI from a web application.

If you have all the requirements already, this should take about 30 minutes and be completely free.

ML to AI

What's the difference between Machine Learning (ML) and AI? My perspective is that AI is applied ML. When you build applications that use ML to improve the user experience and to make the applications feel more natural and intuitive, that is AI.

We will see this transformation from ML to AI as we go through this post.

Steps

We will build a Machine Learning model using an existing ML.Net tutorial.
We will add a Web API to the project and update the code to wire it all together.
We will deploy the Web API to the cloud.
We will use a simple web-app to demonstrate making the call to the API.

By the end of this post, you'll have a working Sentiment Analysis API and an AI-infused app that you can show to your friends and coworkers!

Build the ML Model

Tutorial

Follow the instructions for this tutorial to get started. Note that you will need Visual Studio installed (the tutorial provides the link) to complete this exercise. You do NOT need Visual Studio to use ML.Net generally, but Visual Studio comes with a feature called Model Builder which provides a nice UI and writes a lot of the code for you.

When you finish the tutorial, you should be able to run the project and see predictions. Congratulations, you're ready for the next step!

Add An API

OK, you built an ML Model and have a way to consume it. But, it is not very usable right now. Your users or customers are not going to want to use the Command Line and these predictions are not very helpful outside the context of an actual application. Let's fix that.

Update Some Code

To make this work, we'll need to change some of the code that Model Builder generated for us. In Visual Studio find the file called ConsumeModel.cs and open it. Around line 30 you should see a line that looks like this:

string modelPath = @"C:\Users\mrichter\AppData\Local\Temp\MLVSTools\BaruchMLDemo11ML\BaruchMLDemo11ML.Model\MLModel.zip"

The directory for the ML Model (MLModel.zip) we produced is hard-coded. This is bad because when we deploy this API, we won't actually know what the directory structure will be. Also, the API will be hosted on a Linux server, so this Windows directory won't make any sense.

We need to update the modelPath variable from a hard-coded directory to a 'relative' directory. Update that line with this code:

string modelPath = System.IO.Path.Combine(System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location), "MLModel.zip");

Now the code will look for the model (MLModel.zip) in the same directory that the application is executing.

Rebuild the solution again (Build --> Rebuild Solution). You should see a message in the output that "3 succeeded". That's because we have 3 projects in the solution.

Add the API Project

In the Visual Studio solution explorer, right-click and choose Add a New Project. See the screen shot below.

Search for API and choose the C# ASP.NET Core Web API project. Click Next.

Name the project something like MLDemo.Web. Leave all the other defaults and click Create.

Prepare the API Project

This API Project comes with some demo classes that we don't need, so let's remove them. In the root directory, remove WeatherForecaster.cs. Just click it and hit the delete button. In the Controllers folder, delete WeatherForecastController.cs.

Under Properties, you'll find launchSettings.json. You'll see 2 entries that look like this.

"launchUrl":"weatherforecast"

change them so they look like this:

"launchUrl":"score"

Now let's build the Score URL endpoint which is how our API will be called!

Build the API

Right-click on the **Controller **folder and choose Add --> Controller

Choose "API Controller - Empty".

Save the name of the file as ScoreController.cs.

Replace the content of the generate filed with the below. It's not 100% necessary but you may want to update your namespace name to match the name of your project.

using Microsoft.AspNetCore.Mvc;
using System.Text.Json;

namespace UPDATEMEIFYOUWANT.Web.Controllers
{
    //This tells the dotnet framework that we are building a web api
    [ApiController]

    //This attribute tells dotnet that we want the URL route for the api to be
    //based on the controller's name.
    //In this case, the controller's name is "ScoreController" so the route will be /score
    [Route("[controller]")]
    public class ScoreController : Controller
    {
        public IActionResult Index()
        {
            return View();
        }

        //This tells dotnet that our API will resond to HTTP GET Verbs
        //the "text" parameter will get passed in from the query string when a request to this API is made.
        // 
        [HttpGet]
        public string Get(string text)
        {
            var input = new ModelInput() { Col0 = text };
            //Let's call our ML model predict the sentiment of our text
            ModelOutput result = ConsumeModel.Predict(input);
            //Return the prediction in the response.
            return JsonSerializer.Serialize(result);
        }
    }
}

You will notice that some of the code has red lines beneath it. That's because we want our API to reference the the Model project in our solution, but we haven't yet created that explicit reference between the projects. If you hover over any of those red-underlined words, you should see a little lightbulb pop up which reveals a menu. At the bottom of the menu there's an option to add a reference to our Model project. Click it.

That's it! You've now built an API around the ML Model. Let's try it out!

Test the API

Try rebuilding the entire solution. Go to Build -> Rebuild Solution. After a few seconds the output should tell you "4 succeeded" (because we now have 4 projects).

Right-Click on the Web project and choose Debug --> Start New Instance.

After a few seconds, your browser should start and point to a URL that looks like https://localhost:44380/score and you should see some text in the browser window that looks like this.

{"Prediction":"1","Score":[0.4424079,0.5575921]}

You can pass in text via the URL and the API will use the model to predict the sentiment. Try updating the URL like this:

https://localhost:44380/score?text=this%20is%20amazing!

(note you can write spaces, the Browser will replace them with %20.)

The prediction should be "1". The model correctly predicted this is a positive statement.

Now update the URL to say this:

https://localhost:44380/score?text=I%20think%20this%20is%20horrible

The model should return "0" and predict that this is a negative statement.

Fantastic! Let's review what you've done so far.

You've built an ML Model.
Consumed the model from the command line.
Built a Web API
Consumed the model from the web api.

But we're still not done. Users aren't going to connect to your computer to talk to this API. You need to deploy it to the cloud!

Deploy the API to the Cloud

To deploy the API you will need some requirements.

Requirements

A Microsoft Azure Account - If you don't have one, get one for free.
Azure Command Line (CLI) installed.
Git client installed.

Add the solution to Git

We need to add the solution to Git because we will be using Git as the deployment vehicle for getting this API into the cloud.

In your command line/terminal, go to the root directory for your solution. You should see four sub directories at this level (one for each project that was created).

Run these commands:

git init
git add -A
git commit -m "Initial Commit"

Log into Azure from the Command Line

Now log into Azure with these commands:

az login
az account set -s "your subscription name"

Build a web app to host our API

You're going to set some default values for Azure resources here (note these commands are for setting environment variables in Windows. Here's a post for setting them on other operating systems. )

The APP name needs to be globally unique and should only contain letters and numbers. Try using your name to make sure it is unique.

setx RG "baruchmldemo11rg"
setx PLAN "baruchmldemo11plan" 
setx APP "baruchmldemo11api"

Try running this command to make sure the environment variables were set correctly: echo %RG%. If you don't see the value you set returned, you may need to open a new terminal window.

Once you've verified that the environmental variables were set correctly, We're ready to start deploying Azure resources!

Create a Resource Group where these resources will live:

az group create -n %RG% -l eastus

Create an App Service plan that will host our app. This will create a free plan in the East US.

az appservice plan create -l eastus -g %RG% -n %PLAN% --sku F1 --is-linux

Now we create the actual Web App that lives in the Free App Service plan and that will host our web API.

az webapp create -g %RG% -p %PLAN% -n %APP% --runtime "DOTNETCORE|3.1"

When we add our API to an app, we want it to be reached from a web browser client, so we need to update the CORS setting)

az webapp cors add -a "*" -n %APP% -g %RG%

We will be using Git to deploy our web application, so we need to create credentials for it.

az webapp deployment user set --user-name %APP% --password m@gicDepl0y

You should change the password. Just remember what it is, you will need it later.

Now let's tell the Web App that we will be deploying via Git.

az webapp deployment source config-local-git --name %APP% -g %RG%

That should return a URL that looks like this:

https://baruchmldemo11api@baruchmldemo11api.scm.azurewebsites.net/baruchmldemo11api.git

Remove the part of the URL between the "https://" and including the @ symbol so it looks like this now:

https://baruchmldemo11api.scm.azurewebsites.net/baruchmldemo11api.git

Use that URL as a "remote" for git by running this command:

git remote add azure https://baruchmldemo11api.scm.azurewebsites.net/baruchmldemo11api.git

Deploy the API

If we've done everything right we should be ready to deploy our web API to the cloud. To do it, run the command below. You will be asked for the credentials. Use the app name value for user name (in my case baruchmldemo99api) and the password we defined as the password (m@gicDepl0y).

git push azure master

This command can take a couple of minutes to run.

Test the Web API.

When the command is successfully complete, we can test the API.

Got to the Azure Portal to find the URL for your new Web API.

Click on the URL, it may take a minute for it start. Update the URL so it looks something like this:

https://baruchmldemo11api.azurewebsites.net/score?text=azure%20is%20the%20best%20for%20APIs!

That should return a positive sentiment prediction. Congratulations, your API is deployed to the cloud!

Cool, we are almost done! 😄

Add the API to an App

The last step is the easiest and the most fun. Let's see how we can use this API in a web application.
Go to this URL:

https://baruchmldemo.z13.web.core.windows.net/

In the Sentinment Analysis API text box, enter in the URL endpoint for your API. It should be something like this:

https://baruchmldemo11api.azurewebsites.net/score

Now, in Text to analyze, enter in some text and click "Analyze!"

Depending on what text you put in, you should see a smiley face appear that reflects the sentiment of the text you entered.

ML to AI

This is a simple app but it shows how we create AI by invoking these ML Models.

Conclusion

In this blog post, we built an ML Model from scratch and operationalized it with an API. We deployed that API to the cloud and used it inside an application. A decade ago this may have taken months to build and cost a lot of money. We did it today in 30 minutes for free.

What's next?

This model can definitely be improved and all the steps to retrain it, build and deploy it can be automated. We built a model that is about 80% accurate in a few minutes. If we want to improve it we need to train it with more data. We can start with this model, deploy it, learn from it and improve it. All the tools and steps for operationalizing the model are there, we just went through them, but that is just the easy part!

Authorization Strategies with Azure Active Directory

Mike Richter — Mon, 08 Mar 2021 01:20:15 +0000

This article will explore four strategies for authorizing users who authenticate into your applications with Azure Active Directory.

Application Roles
User Groups
Azure AD B2C 3rd Party Claims
Azure Web API

We all know the two main elements of identity when it comes to App Development. There's Authentication: this is who I am; and there's Authorization: this is what I can do. Azure Active Directory does the authentication for you, but you can also use it for authorization.

Authentication with Azure Active Directory

In the cloud, one of the most popular systems for authentication is Azure Active Directory (AAD). If you use Office 365, when you sign in to Outlook or use the Web versions of Word or office, you are authenticating against AAD. AAD is the identity system for Office 365. If your company uses popular SAAS solutions like Salesforce, Workday, SAP or ADP and you sign in with your work Office 365 account, then you are using the same AAD to sign in to those applications. And, of course, you can build your own applications that leverages AAD to single-sign-on users with their corporate accounts. There are lots of reasons to enable SSO with AAD and thousands of SAAS providers do just that.

Authorization with Azure Active Directory

Let's say you're building a SAAS app. You can delegate authentication to AAD and that is well documented. But how do you authorize users? When you authenticate users from AAD using OIDC/OAUTH2, you'll get a jwt token with a globally unique identifier for that user composed of a Tenant ID (the AAD tenant the user belongs to) and an Object ID (the User's ID in that tenant). Once you have this information, you can use it to figure out what permissions (if any) the user has in your application.

For the sake of simplicity, authorization or permissions information can be put in two places. Inside AAD or somewhere else like a Bespoke Permissions system you build for your application. You can use either one or use can use both. How do you choose? There are a lot of decisions that go into designing how you authorize users. By leveraging AAD, there's fewer things you need to worry about but you may not get the granularity you need.

Here are some things to think about it when considering where to put your authorization information.

In Azure Active Directory

Control: Customer has a lot control and app developer has less control.
Management: Because your customer will be modifying objects in AAD to determine permissions, an AAD administrator may need to be actively engaged when changes are required. For example, if your SAAS application is for Marketing teams, your customer may need to submit requests to IT to change/add permissions in your system. This could add some friction.
Licensing: if you're licensing your application per user, you'll need to be aware of how many users your customers are adding. You will need to monitor that and potentially implement a way to true-up.
Complexity: There's less for you to build and maintain. Therefore, relying on AAD for permissions can be simpler.
Granularity: You will use some built in AAD features. Depending on your needs this may suffice or you may need a more granular system.
Cost: Since you're leveraging your customer's AAD, there may not be a big cost impact storing authorization information there.

Bespoke System

Control: Customer only has the control that the bespoke system allows.
Management: Generally, the AAD administrator does not need to be actively involved. They will need to greenlight your app, but then the team that's actually using the application can control permissions. If you're building a SAAS app for marketing teams, the marketing team may need to ask IT to allow your application one time, but from then on they can manage permissions themselves.
Licensing: You can more easily control how you do per user licenses because you have access to all the transactions happening in the vendor system.
Complexity: There's more for you to build an maintain. Therefore, relying on your own system can be more complex.
Granularity: You can build as much granularity as you need and evolve it over time.
Cost: You will need to operate a permissions system. If you use a commercial product to manage permissions there may be a cost associated with it.

Let's look at four options for building an authorization system with AAD.

Authorization Strategies

Strategies Leveraging Azure AD

Enterprise Application Roles

Application Roles allow app developers to define roles in their application like Administrator, Contributor, Reader. When customers add the application to AAD so they can SSO to it, the customers' AAD admins can add users to your application and assign them to roles. This is a very simple solution for you. It adds more complexity for your customers and provides the least granular set of permissions.

Authorization with Enterprise Application Roles

Microsoft Groups

Permissions in your application can be aligned to custom groups in your customers' AAD tenants.
There are several ways to retrieve the group information. You can get the user's groups in the token. Or you could also ping Microsoft Graph to get the Groups the customer is a member of. There are few things to think about with groups:

Will your customer be OK with sending information about every group a user is in?
If you want to query Microsoft Graph to see the members of a particular group, your application will request that permission and the customer's AAD admins will have to consent to that.
In either case, you should tell customers to create new groups that are reserved just for the roles in your applications. For example, the groups can be called "My-App Administrators", "My-App Contributors", "My-App Readers".
You wont get the actual group names back from AAD, just their Object ID Guids. So you will need a per AAD Tenant lookup somewhere in your application to map group ids from your customers' tenants.

Authorization with User Groups

Which should you choose? There's a discussion about Roles vs Groups here: https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-add-app-roles-in-azure-ad-apps#app-roles-vs-groups

Here's a Microsoft Learn course to get hands on with groups and roles: https://docs.microsoft.com/en-us/learn/modules/identity-users-groups-approles/

Strategies Leveraging a Bespoke System

AAD gives you couple of options for integrating with a Bespoke permissions system.
You may want to use this, rather than AAD, if you need more granular or advanced authorization features like ACLs, Policies and RBAC.

Azure AD B2C 3rd Party Claims

You can add authentication with AAD into your application directly in your code. You can use the MSAL library to integrate AAD. You can also use a service that acts as a proxy to many identification providers. Let's say you have customers that use AAD and customers that use Google Workspace. There are various Identity Services out there that will make it easy for you to talk to many identity providers. Microsoft's offering for this Azure AD B2C.

During the authentication flow, Azure AD B2C can call out to a 3rd party API to get custom claims. You can build that 3rd party API around your Bespoke Permissions system. B2C can call that API and merge claims from your Bespoke Permissions service into the JWT token it returns to your application. The benefit of this approach is that your application only calls one service, Azure AD B2C and it gets back a consistent set of claims. Azure AD B2C does the hard-work of calling multiple identity providers and 3rd party permissions services.

Authorization with Azure AD B2C

Azure AD Web API

If you have a Bespoke Permissions system and expose it as an API, you can have your application call it directly. You can leverage AAD to make sure calls to the permissions API service are protected. In this case, you're using your own AAD, not your customers. It just takes a few steps to make this work.

In AAD you'll give your application an identity.
In AAD you'll give your permissions service API an identity.
In AAD, you'll configure the permissions service API to allow the application to make requests to it.
In your application, when you make requests to the permissions service, you'll present the identity, so the permissions service knows who's calling and will allow it.

Authorization with AAD Web API

In Conclusion

There are other ways to implement authorization and permissions in your application. In this article I discussed four strategies that leverage AAD. There are lot of elements to consider when deciding which strategy to choose. If using roles and groups inside of AAD is not granular enough there are many 3rd party authorization systems that you can integrate into your application, including open source projects like Casbin. I hope you found this discussion useful!

Easily Create Allow (or Deny) Lists for Azure Resources

Mike Richter — Mon, 22 Feb 2021 02:49:03 +0000

In this post, we will use the Azure CLI and the Azure Resource Graph to quickly generate a list of endpoints for Allow or Deny lists. You can find the source code for this example here:

michaelsrichter / azure-allow-list

Use node and the Azure CLI to quickly generate allow (or deny) lists.

What is an Allow List?

An Allow List is a list of endpoints that you will allow through your firewall. A Deny List is the exact opposite: endpoints you deny through your firewall. You may be more familiar with the archaic terms, white list and black list. They are essentially the same thing. When I use the term endpoints, I am using it as a generic term to refer to URLs or IP Addresses.

Why do I want an Allow List?

Security

Over the years I've worked with many Microsoft Partners who were helping their internal IT teams prepare to move to Azure. Often some services would live in Azure and some would remain on-premises. The internal IT teams needed a list of all the public endpoints in Azure so they could allow-list them or deny-list them.

Building this list in the past was complicated. Imagine if you have thousands of Azure Resources and many of them had public endpoints; it used to be a pain to gather all that information. Almost every Azure service has a public endpoint. Popular services like Azure SQL DB, Azure Storage Accounts, and Azure Web Apps all have public endpoint and there are many, many more. Some services, like storage accounts, even have multiple public endpoints!

Insights and Risks

Even if you're not concerned with creating Allow Lists or Deny Lists, you might be interested to see all of the publicly available endpoints for your Azure services. You might want to track each of them and make sure you understand the risk and risk mitigation efforts for each.

Governance

Just by running this tool you might be surprised or even shocked at what you'll find. If you're an administrator for large deployments in Azure, you may have had suspicions that there a ton of externally available resources. Now you can easily find them all and implement a plan to address them.

How do I easily generate an Allow List?

I built some sample code using node to show you how easy it is. Azure has a cross platform CLI and a couple of years ago the team built the Azure Resource Graph (ARG). ARG lets you query your Azure Resources and return just the data you need, formatted the way you need. Before ARG came along, it was a real pain to pull all of this information together.

ARG gets us most of the way there. I wrote some node code for the finishing touches.

To get a CSV file of all your URLs and IP addresses in an Azure subscription, you just need to run a command like this

npm install

node index.js c8faea8e-b5d3-4f31-bc58-f15f4390309a > azure-allow-list.csv

You can open that csv file in Excel and you'll get something that looks like this.

Notice the value column has the URL or the IP Address for your service. You can now take this list and import it into whatever Firewall or Network appliance you're using.

You can find all the easy steps to get started on the Github repo: https://github.com/michaelsrichter/azure-allow-list. It works great in the Azure Shell too!

Of course, this tool can be improved in many ways. Feel free to make a suggestion or submit your own pull-request. And please let me know if you have any thoughts or feedback. You can use the comments below or on Github too. Thanks!

How to Deploy to Azure with Least Privilege

Mike Richter — Sun, 14 Feb 2021 18:38:36 +0000

In this post we'll walk through the steps you can take to give a Service Principal a role with "Least Privilege" in Azure. After reading this article you will have a very practical method that you can use over and over again. You will be able to create roles for your Service Principals that will only allow them to deploy specific types of resources and only in the specified scopes.

Background

If you build an ARM Template or you get one from a 3rd party software company or services company, you will need permissions to deploy it. If you want to automate the deployment of that ARM Template, you will want to create a Service Principal that will do the deployment for you. Ideally the Service Principal will only have enough permission to deploy that ARM Template and do nothing else. If the Service Principal has broad permissions, like contributor or owner of an entire subscription or resource group, the Service Principal can be exploited.

For instance, the ARM Template can be altered and more services can be added to it. Or the Service Principal's credentials can be compromised or re-used in other automation pipelines.

So, how do you build a "Least Privilege" Service Principal with only the permissions that it needs? Let's find out.

Concepts

Here are the concepts I will discuss in this article.

Service Principal - Essentially a Service Account that you can use to automate Azure.
ARM Template - A declarative json file used for deploying Infrastructure As Code in Azure.
Azure Subscription and Resource Groups are the scopes for where you can deploy Azure services. All Azure services live inside a Resource Group which lives inside a Subscription. You can scope permissions at the individual Resource level, the Resource Group level or for the whole subscription.
Azure Built-In Roles and Azure Custom Roles. Roles are what determine what an identity can do in Azure. A user or a service principal doesn't have permission to do anything in Azure until it is assigned a role. Identities can have more than one role. Roles determine what actions you can perform in Azure and within what scope. Roles are at the heart of what this article is about!
Permissions - there are thousands of permissions that determine what an identity can do in Azure. Building a role composed of only the minimum required permissions and only within the minimum required scope is how we get to least privilege.

Set Up

Here's what you'll need to follow along.

An Azure Subscription with a Resource Group that YOU are the owner of. We are going to create a Service Principal that will be scoped to this Resource Group and this requires that you are an owner of it because you are delegating access to the Resource Group. Note that to create a Resource Group you need to be a Contributor or an Owner of a Subscription. Otherwise an Owner or Contributor will need to create a Resource Group for you and make you an owner.
The Azure CLI. We will need two instances of the CLI running. In one instance you will sign in with YOUR credentials to create Service Principals and Roles. In the other instance you will sign in as the Least Privilege Service Principal.
A simple text editor. I'm using VS Code.
An ARM Template to deploy. You can find 100s of examples on the Azure Quickstart Templates site. I am going to use the Simple Umbraco CMS Web App Template. It uses various services like Azure App Service, Azure Storage, Azure SQL DB and Application Insights. Our Service Principal should ONLY be able to deploy those services in our chosen Resource Group. If we tried to use the same Service Principal to deploy Virtual Machines or Container Instances, it should fail.

Let's Start

Create a Service Principal

Sign in to the Azure CLI with your credentials and create a service principal. I will refer to this as your User CLI instance.

az ad sp create-for-rbac -n "leastsp" --skip-assignment

This will return something like this:

{
  "appId": "55555555-5555-5555-5555-555555555555",
  "displayName": "leastsp",
  "name": "http://leastsp",
  "password": "SuPerSecretP@ssw0rd",
  "tenant": "00000000-0000-0000-0000-000000000000"
}

Make sure to copy and paste these details somewhere; you won't be able to see the password again!

Right now we have a Service Principal that has no permissions to do anything. It's just an identity in your Azure AD. Let's try to login to the Azure CLI.

Start another CLI instance. I will refer to this as the SP CLI. Sign in with this command (of course update the values with YOUR values):

az login --service-principal -u http://leastsp --tenant 00000000-0000-0000-0000-000000000000 -p SuPerSecretP@ssw0rd

You should get a message back that says No subscriptions found for http://leastsp. That makes sense, this principal has no permissions to do anything yet!

We add permissions to a principal by assigning it a role. A role is essentially a container of permissions.

Principals have roles.
Roles have permissions.

Let's look at what roles are assigned to our Service Principal.

In the User CLI run this command (remember to use YOUR values):

az role assignment list --assignee 55555555-5555-5555-5555-555555555555

This returns an empty array []. No roles assigned. 😦

To let the Service Principal login to your Azure subscription, let's give it a Reader role of the Resource Group that you own. If the Resource Group is called Least, The ID for that Resource Group will be /subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least.

Go back to your 1st CLI (the User CLI), the one where YOU are logged in. Type this in (last reminder to use YOUR values):

az role assignment create --role Reader --assignee 55555555-5555-5555-5555-555555555555 --scope "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least"

In the SP CLI try logging in again:

az login --service-principal -u http://leastsp --tenant 00000000-0000-0000-0000-000000000000 -p SuPerSecretP@ssw0rd

This time you should be successful and see something like this get returned:

[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "00000000-0000-0000-0000-000000000000",
    "id": "11111111-1111-1111-1111-111111111111",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Subscription Name",
    "state": "Enabled",
    "tenantId": "00000000-0000-0000-0000-000000000000",
    "user": {
      "name": "http://leastsp",
      "type": "servicePrincipal"
    }
  }
]

And we should see the one Resource Group that our SP is now a Reader of:

az group list -o table

Name    Location    Status
------  ----------  ---------
least   eastus      Succeeded

The SP should ONLY be able to see the Resource Group that it is scoped to and nothing else in the subscription or any other subscriptions.

We can say that leastsp now has the Reader role, scoped to the Least Resource Group.

Now, let's try to deploy the Simple Umbraco template. We'll use the CLI command on that page to do the deployment. Remember to do this in your SP CLI:

az group deployment create --resource-group least --template-uri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/umbraco-webapp-simple/azuredeploy.json --p '@parameters.json'

As you might expect, we get an error:

{"error":{"code":"AuthorizationFailed","message":"The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have authorization to perform action 'Microsoft.Resources/deployments/validate/action' over scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourcegroups/least/providers/Microsoft.Resources/deployments/azuredeploy' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

This makes complete sense. Your SP only has a Reader role. We shouldn't expect a Reader to be able to deploy services! That would require the "write" kind of permissions. 😄

To deploy this template with least privilege, we will need to create a Role with only the permissions that are required.

Let's begin.

Building the Role

This is the main part of the article. Stay with me! 😅

We will create a role and add all the permissions we need to it. When we try to deploy the ARM Template, we will get an error message (like the one above) telling us about additional permissions that we need. We will add the permissions to the role and try to do the deployment again. We may need to repeat this several times as each step of the deployment reveals new permissions that are needed.

Look at the error message above. It's says our Service Principal doesn't have permissions to perform Microsoft.Resources/deployments/validate/action.

Let's create a role that has this permission and assign it to our SP. Start by creating a role definition file.

{
    "type": "Microsoft.Authorization/roleDefinitions",
    "roleName": "leastprivilegeappdeployer",
    "description": "Least Privilege App Deployer",
    "assignableScopes": [
        "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least"
    ],
    "name": "leastprivilegeappdeployer",
    "roleType": "CustomRole",
    "permissions": [
        {
            "actions": [
                "Microsoft.Resources/deployments/validate/action"
            ],
            "notActions": [],
            "dataActions": [],
            "notDataActions": []
        }
    ]
}

Save this file as role.json. You see that the role is called "leastprivilegeappdeployer" and it is assigned to our least resource group. The only permission it has is the perform the deployment validation action.

Let's create this role in your User CLI.

az role definition create --role-definition role.json

And, again in our User CLI, assign this role to our Service Principal

az role assignment create --role leastprivilegeappdeployer --assignee 55555555-5555-5555-5555-555555555555 --scope "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least"

We have now added a role to our SP that allows it to validate deployments. Let's try our deployment again. Switch over to the SP CLI. Before we redeploy, we should logout and log back in to make sure the CLI is updated with the SP's role.

In the SP CLI

az logout

az login --service-principal -u http://leastsp --tenant 00000000-0000-0000-0000-000000000000 -p SuPerSecretP@ssw0rd

Now try the deployment again in the SP CLI

az group deployment create --resource-group least --template-uri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/umbraco-webapp-simple/azuredeploy.json --p '@parameters.json'

We should now get this REALLY LONG error:

{"error":{"code":"InvalidTemplateDeployment","message":"Deployment failed with multiple errors: 'Authorization failed for template resource 'umbracolzybcxduxe526' of type 'Microsoft.Sql/servers'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Sql/servers/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Sql/servers/umbracolzybcxduxe526'.:Authorization failed for template resource 'umbracolzybcxduxe526/umbraco-db' of type 'Microsoft.Sql/servers/databases'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Sql/servers/databases/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Sql/servers/umbracolzybcxduxe526/databases/umbraco-db'.:Authorization failed for template resource 'umbracolzybcxduxe526/AllowAllWindowsAzureIps' of type 'Microsoft.Sql/servers/firewallrules'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Sql/servers/firewallrules/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Sql/servers/umbracolzybcxduxe526/firewallrules/AllowAllWindowsAzureIps'.:Authorization failed for template resource 'lzybcxduxe526standardsa' of type 'Microsoft.Storage/storageAccounts'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Storage/storageAccounts/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Storage/storageAccounts/lzybcxduxe526standardsa'.:Authorization failed for template resource 'umbracolzybcxduxe526serviceplan' of type 'Microsoft.Web/serverFarms'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Web/serverFarms/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Web/serverFarms/umbracolzybcxduxe526serviceplan'.:Authorization failed for template resource 'umbracolzybcxduxe526' of type 'Microsoft.Web/Sites'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Web/Sites/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Web/Sites/umbracolzybcxduxe526'.:Authorization failed for template resource 'umbracolzybcxduxe526/MSDeploy' of type 'Microsoft.Web/Sites/Extensions'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Web/Sites/Extensions/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Web/Sites/umbracolzybcxduxe526/Extensions/MSDeploy'.:Authorization failed for template resource 'umbracolzybcxduxe526/connectionstrings' of type 'Microsoft.Web/Sites/config'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Web/Sites/config/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Web/Sites/umbracolzybcxduxe526/config/connectionstrings'.:Authorization failed for template resource 'umbracolzybcxduxe526/web' of type 'Microsoft.Web/Sites/config'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'Microsoft.Web/Sites/config/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/Microsoft.Web/Sites/umbracolzybcxduxe526/config/web'.:Authorization failed for template resource 'umbracolzybcxduxe526serviceplan-scaleset' of type 'microsoft.insights/autoscalesettings'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'microsoft.insights/autoscalesettings/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/microsoft.insights/autoscalesettings/umbracolzybcxduxe526serviceplan-scaleset'.:Authorization failed for template resource 'umbracolzybcxduxe526-appin' of type 'microsoft.insights/components'. The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have permission to perform action 'microsoft.insights/components/write' at scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least/providers/microsoft.insights/components/umbracolzybcxduxe526-appin'.'"}}

This is scary but it's also great because it gives us everything we need to fix it! Our SP can now validate the deployment but the validation shows that we need more permissions. We can pull all of these permissions out of the error message and update our custom role. Let's add these permissions to our role.json file.

{
    "type": "Microsoft.Authorization/roleDefinitions",
    "roleName": "leastprivilegeappdeployer",
    "description": "Least Privilege App Deployer",
    "assignableScopes": [
        "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least"
    ],

    "id": "/subscriptions/11111111-1111-1111-1111-111111111111/providers/Microsoft.Authorization/roleDefinitions/33333333-3333-3333-3333-333333333333",
    "name": "33333333-3333-3333-3333-333333333333",
    "roleType": "CustomRole",
    "permissions": [
        {
            "actions": [
                "Microsoft.Resources/deployments/validate/action",
                "Microsoft.Sql/servers/write",
                "Microsoft.Sql/servers/databases/write",
                "Microsoft.Sql/servers/firewallrules/write",
                "Microsoft.Storage/storageAccounts/write",
                "Microsoft.Web/serverFarms/write",
                "Microsoft.Web/Sites/write",
                "Microsoft.Web/Sites/Extensions/write",
                "Microsoft.Web/Sites/config/write",
                "microsoft.insights/autoscalesettings/write",
                "microsoft.insights/components/write"
            ],
            "notActions": [],
            "dataActions": [],
            "notDataActions": []
        }
    ]
}

You'll see that I also included the id field in the json. This value was generated for us by Azure and since we are going to update the role, we need to include the generated id going forward. You can get the id via the command az role definition list in the User CLI. Here's an example:

az role definition list --scope /subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least --custom-role-only -n leastprivilegeappdeployer --query [0].id
Result
------------------------------------------------------------------------------------------------------------------------------------------
/subscriptions/11111111-1111-1111-1111-111111111111/providers/Microsoft.Authorization/roleDefinitions/33333333-3333-3333-3333-333333333333

In the User CLI, run the role update command:

az role definition update --role-definition role.json

Let's try the deployment again.
Back in the SP CLI, log out and login again and then try the deployment again.

az logout

az login --service-principal -u http://leastsp --tenant 00000000-0000-0000-0000-000000000000 -p SuPerSecretP@ssw0rd

az group deployment create --resource-group least --template-uri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/umbraco-webapp-simple/azuredeploy.json --p '@parameters.json'

We are SO close! 😉

There's another error, but I promise this is the LAST one.

Azure Error: AuthorizationFailed
Message: The client '22222222-2222-2222-2222-222222222222' with object id '22222222-2222-2222-2222-222222222222' does not have authorization to perform action 'Microsoft.Resources/deployments/write' over scope '/subscriptions/11111111-1111-1111-1111-111111111111/resourcegroups/least/providers/Microsoft.Resources/deployments/azuredeploy' or the scope is invalid. If access was recently granted, please refresh your credentials.

Our role just needs the permission to actually write the deployments. Let's update our role.json one more time to give it this permission.

{
    "type": "Microsoft.Authorization/roleDefinitions",
    "roleName": "leastprivilegeappdeployer",
    "description": "Least Privilege App Deployer",
    "assignableScopes": [
        "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/least"
    ],

    "id": "/subscriptions/11111111-1111-1111-1111-111111111111/providers/Microsoft.Authorization/roleDefinitions/33333333-3333-3333-3333-333333333333",
    "name": "33333333-3333-3333-3333-333333333333",
    "roleType": "CustomRole",
    "permissions": [
        {
            "actions": [
                "Microsoft.Resources/deployments/validate/action",
                "Microsoft.Resources/deployments/write",
                "Microsoft.Sql/servers/write",
                "Microsoft.Sql/servers/databases/write",
                "Microsoft.Sql/servers/firewallrules/write",
                "Microsoft.Storage/storageAccounts/write",
                "Microsoft.Web/serverFarms/write",
                "Microsoft.Web/Sites/write",
                "Microsoft.Web/Sites/Extensions/write",
                "Microsoft.Web/Sites/config/write",
                "microsoft.insights/autoscalesettings/write",
                "microsoft.insights/components/write"
            ],
            "notActions": [],
            "dataActions": [],
            "notDataActions": []
        }
    ]
}

In the User CLI run the role update command

az role definition update --role-definition role.json

And now, in the SP CLI, logout, login and do the deployment.

az logout

az login --service-principal -u http://leastsp --tenant 00000000-0000-0000-0000-000000000000 -p SuPerSecretP@ssw0rd

az group deployment create --resource-group least --template-uri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/umbraco-webapp-simple/azuredeploy.json --p '@parameters.json'

If it worked, congratulations! 👏 You built a role scoped to a single Resource Group with the least amount of permissions required to deploy this ARM template. You assigned the role to a Service Principal and did the deployment. You can visit the web app that got built and start configuring Umbraco. Awesome!

Remember, you can follow these steps with any other ARM Template you want to use.

If it didn't work, let me know what went wrong and I'd be happy to help you figure it out.

Clean Up

When you're done trying this out, remember to delete your Service Principal, your Custom Role and your Resource Group with all the resources from this template. In the User CLI you can use these commands with the appropriate flags:

az ad sp delete 

az role definition delete

az group delete

Final Thoughts

Providing your operations team, your devops pipeline or your customers with an ARM Template is only a part of a secure automated deployment process. You should also provide a role that has the least amount of privileges to deploy that ARM Template. If a new service gets added to the ARM Template, the role should also be updated to reflect that change. Adopting this process helps reduce risk and exposure, especially from a security, compliance and cost control perspective.

If you're building roles that will be doing automated deployments, you can assume that you'll need the Microsoft.Resources/deployments/validate/action and the Microsoft.Resources/deployments/write permissions, so you can always start a role with those permissions.

I hope this was a helpful tutorial and that you'll be able to use this to secure your automated deployments in the future! If you have any feedback or suggestions please share them or leave them in the comments.

Thanks!

The Cloud Solution Architect Checklist

Mike Richter — Mon, 08 Feb 2021 02:12:29 +0000

As a Cloud Solution Architect, I help Microsoft's partners build solutions on Azure. Often the way forward is not clear and using a checklist can help point us in the right direction. Here's a discussion of what checklists are and why they're useful.

Why do we need a checklist?

Here are some interesting health care stats that will set the stage.

Hospital doctors evaluate 250 different disease and conditions per year.
Intensive Care Unit (ICU) patients require, on average 178 daily tasks and about half of these patients experience a complication.
In one study of 41,000 trauma patients:
- Unique injury diagnoses: 1,224.
- Combination of diagnoses: 32,261.
In 2006 there were 230 Million surgeries globally
- 3% - 17% had complications.
- 7 million people died.

What do these stats have to do with checklists and cloud architecture?

First, all of these statistics come from the excellent book "The Checklist Manifesto" by Atul Gawande.

When you look at these stats, you can see how complicated it is to diagnose and treat healthcare issues.

It's also very complicated to design cloud architectures.

Now, to be clear, I AM NOT suggesting that cloud architecture is as complicated as diagnosing and treating healthcare issues. Nor am I suggesting the problems of cloud architecture are as serious or important as saving people's lives. I am merely suggesting that when things get complicated a checklist can come in handy.

Here's an eye chart of the Cloud Native ecosystem from the CNCF.

Think about this eye chart and then think about every other technology domain – security, data, identity, monitoring, etc. They have their own eye charts. There are thousands of options in near infinite combinations. An architecture, especially an existing/legacy one, is going to be a Frankenstein of all these different services. And, as an architect, it’s your job to design and/or validate these architectures for your cloud platform.

And just think about how complicated that cloud platform is on its own! I work at Microsoft and Azure itself has 100s of different services. Check out this eye chart from Azure Charts.

How can a checklist help?

So, how can a checklist help with all this complexity?

Let's go back to those healthcare examples and look what happens when simple checklists were implemented.

In 2001, John Hopkins Hospital implemented a 5 question checklist for ICU doctors. After 15 months they prevented 43 infections, 8 deaths and saved $2 Million.
In 2009, 8 hospital piloted a 2-minute, 19 question surgery checklist. After 3 months, surgery complications fell 36% and deaths fell 47%.

Let's be clear: 5 questions saved a hospital two million dollars over 15 months. Spending 2 minutes at the beginning of a surgery cut the number of deaths in half!

Again, these impressive stats come from "The Checklist Manifesto." A checklist is just asking a few question before starting a process. It forces you to check things every time you start a process. Without a checklist, you might rely on just remmebering to check these things. But if you forget to do this thing just once, it can have a tremendously bad impact. Checklists are used in a lot of other place besides healthcare: Airplane pilots use them as well as non-Cloud architects and investors.

Checklists are a great tool for avoiding errors and for lowering stress. There's a great Freakonomics podcast episode featuring Atul Gawande, the author of The Checklist Manifesto, if you want to learn more about the power of checklists. The book is also a great quick read!

My Checklist

Last year I worked with my colleagues at Microsoft, who are among some of the best partner architects, to compile the checklists we use when working with Microsoft Partners. We shared our Solution Architecture Questions (SAQ) on Github. Anyone is free to use this list, share it and contribute to it. Some of the questions are Azure partnership specific, but they can be used when designing architectures in any cloud environment.

I also started working on a web app for finding these questions, asking them and recording the answers. You can find it at https://saq.monster. Once you create an account, this web app runs completely in the browser. It leverages Blazor Web Assembly and Azure Static Web Apps. So whatever data you put into the app remains in your browser only. Eventually I may build a cloud sync service behind it, but for now it remains completely client-only.

What are your thoughts on checklists and the list we built? Are there strong opinions out there about using checklists in the field of cloud architecture? Do individuals have their own lists? I would love to hear your thoughts on this!

When Their Stuff Doesn't Work With Your Stuff

Mike Richter — Sun, 31 Jan 2021 21:08:14 +0000

Here's the situation: There is an open source library that you love to use that works with an open source platform that your solution is built on. You use this library all the time and it's part of your standard set of tools for operating and managing the platform. Now you move the solution to a new cloud vendor which provides you the platform as a managed service. But, unfortunately the library you know and love no longer works in this new environment!

Let's explore this very real situation that unfortunately comes up too regularly. At the heart of this common problem is the risk that is part of any non-trivial software system.

There are 3 parts to this puzzle.

The Open Source Software or Platform itself.
The Cloud Flavor of this Open Source Software or Platform.
The Ecosystem of tools and libraries around the Open Source Software or Platform.

Note that I wont cite any specific examples to protect the names of the innocent :).

Cloud Flavors of Open Source

Modern software architecture today means using lots of different solutions from a lot of different sources. We use cloud services, 3rd party applications and open-source software. Making things more complicated is that many cloud providers have their own flavors of popular open-source software. For instance: AWS MySQL vs GCP MySQL vs Azure MySQL. You maybe thinking: 'So what? MySQL is MySQL is MySQL.' And, like-wise, 'Kubernetes is just Kubernetes.' Yes, but, and there is a big BUT: each platform brings its own features and integrates them into their own offerings.

These different Cloud Flavors often require specific adjustments, tweaking and testing. Usually it is not the core workload that varies among these Cloud Flavors. Rather, it is in the operationalization (observability, security, fault tolerance, etc.) where there can be lots of differences.

Open Source Ecosystem around Open Source.

Popular Open Source platforms, especially Open Source platforms, often have an ecosystem of other Open Source tools around them. Tools like libraries, plugins and connectors that provide additional features and integrations are widely used by the community. Think of WordPress and the tens of thousands of plugins available for it.

So cloud vendors have their own implementations of open source software. That open source software has its own ecosystem of open source tools. Everything should all work together, right? Ideally, yes, but that is not always the case. And when things don't work as expected who is responsible for fixing it?

Look at the image above. Suppose you want to use "Tool 1" with Open Source Software on Cloud Vendor C and it doesn't work as expected.

Who is Responsible?

Who is responsible for making sure the tool works?

The Open Source Tool 1 maintainers?
The Open Source Software maintainers?
The Cloud Vendor C engineers?

We know it's a problem for you when you want to get your solution working. Who are you likely to expect to fix it?

My experience tells me that customers believe it is the Cloud Vendor who should be responsible. After all, the tool works with the platform in the non-managed service. The only thing different is the cloud vendor's implementation. But it is not that simple.

Usually the cloud vendor is building a specific implementation to simplify management of the software, and that can come with some trade-offs or lack of support for some integrations. Like WordPress, popular open source software and platforms can have an ecosystem of dozens, hundreds or thousands of libraries and plugins. How is the Cloud Vendor supposed to test, let alone support them all?

Look to the Docs!

Often you will find documentation and support in various places and that might begin to help us figure out who should be responsible.

The tool maintainers may provide guidance for installing and running the tool on different cloud vendors. If they do that, then they should encourage the community to open bugs when problems are discovered.
The cloud vendor may also provide specific documentation on how to use its service with popular 3rd party plugins and tools. If they provide this documentation, you can assume it's supported and customers should be able to submit support tickets when things don't work as expected.
Rarely would the Open Source platform itself be responsible for fixing how 3rd party solutions integrate or operationalize the software.

Where is the Risk?

If the cloud vendor doesn't provide specific documentation, you may be using that tool at your own risk. How much risk is there?

Does the tool project website say they specifically support the cloud vendor? That's good.
Does the cloud vendor say they support the tool and provide documentation? That's even better.
If the cloud vendor doesn't support it, things may work today, but may not work tomorrow. When the cloud vendor makes a change, they wont test it against unsupported 3rd party tools. If a change breaks the tool, you'll have to wait for the tool maintainers to update it (or submit a pull request with your own fix!).
If the tool is focused more on the core workload of the platform, there is generally less risk.
if the tool is focused on the operationalization of the platform there is generally more risk.

Final Thoughts

Always try to use supported libraries and tools when building solutions. The open source ecosystem combined with open source platforms and a cloud flavor, can create infinite variations. It may be hard to always build a solution with tools that are 100% supported in all these environments. Make explicit decisions and document what tools you use and for what reasons. Calculate the risk and have a mitigation plan.

Finally, as a Cloud Architect, I look at the world of opinionated software development to get a perspective on this. Think of Java Spring Boot and how it lets you get productive with "minimal fuss." If you choose a Cloud Flavor of an open source platform, why not also choose the built-in capabilities and supported tools? You're choosing a managed service to reduce complexity and ultimately save costs, so try to embrace it's "opinion" on what tools you should use with it.

Let me know what you think!

Parameters vs Architecture

Mike Richter — Mon, 25 Jan 2021 02:29:28 +0000

As an architect that works with Microsoft partners, I often get direct perspectives and opinions about Azure. These insights are sometimes obvious to me but are completely new to partners. Here's something that happened recently and I thought I'd share to help all architects everywhere on their cloud journey.

Automation

We all know that the cloud makes it easy to spin up some service, try it out and shut it down when we're done. The cloud also makes it easy to automate doing these things. If we build some parameterization into our automation then it's easy to spin up services for different kinds of environments.

Basic/cheap tiers for dev and test, and
Premium/expensive tiers for production. Easy right? Not so fast!

The reality is that many times there are architecture implications based on what SKU or tier of a service you are going to deploy. The basic tier usually does not support all the features and integrations of a premium tier. This makes building automation a little more complicated. You can't just swap out parameter values - you will need to adjust other architecture components.

An Example

Azure Database for MySQL is a PAAS database service from Azure. It comes in several tiers - basic, general purpose and memory optimized. The basic tier does not allow for a feature called Service Endpoints.

Service Endpoints are a way for you to lock down who can access the service. With a service endpoint, you can say, 'only allow traffic from this subnet in this network to access my service.' That's a useful feature in a production scenario when you want to really restrict who can access the service. There are other ways to lock down access but that is beyond the scope of this article.

If you set up a Basic mysql database and try to connect from a VNET that has a Service Endpoint enabled. You will get an error.

But, if you set up a General Purpose mysql database and try to connect from a VNET that has a Service Endpoint enabled, it works.

You can build an Azure Resource Manager (ARM) Template to automate deploying these service. But when you want to deploy in a real production environment with a Service Endpoint, you will need to add additional components. See the example below.

        {
            "type": "Microsoft.DBforMySQL/servers/virtualNetworkRules",
            "apiVersion": "2017-12-01",
            "name": "[concat(parameters('servers_mrichtermysqlgeneral_name'), '/serviceendpoint')]",
            "dependsOn": [
                "[resourceId('Microsoft.DBforMySQL/servers', parameters('servers_mrichtermysqlgeneral_name'))]"
            ],
            "properties": {
                "virtualNetworkSubnetId": "[concat(parameters('virtualNetworks_hcm_perf_vnet_eastus_externalid'), '/subnets/serviceendpoint')]",
                "ignoreMissingVnetServiceEndpoint": false
            }
        }

You need a way to build automation that includes this for production environments and excludes this non-production. It's a little more complicated than swapping parameters when deploying this arm template.

Obviously this is just a simplistic example. You can imagine a solution with many different services can get much more complicated.

Just Be Aware

As you can see, targeting different environments can have an impact on your automation. Azure does give you some ways to address this. ARM Templates support conditions, logical functions and nest templates. You can build a template that says If MySQL is Basic tier, use dev template, otherwise use prod template. You can learn more about flexible arm templates from this blog article.

So, yes: it is possible to create a single set of automation for a solution that targets different service tiers, utilizing different features for different environments. It's just a little bit more complicated than swapping parameter values. Be aware of this when planning your Azure cloud journey.

Azure Files Transactions Experiments

Mike Richter — Mon, 18 Jan 2021 21:29:24 +0000

What are Azure File transactions? Can we measure them? Are all transactions equal? Let's find out!

Background

Azure Files is a file share service from Azure. It comes in several tiers: Premium, Transaction Optimized, Hot and Cool. All tiers charge you based on the amount of data in the file share. If you use any tier besides Premium, you are charged for transactions as well. You can find more details on the Azure Files pricing page.

For Transaction Optimized (previously known as the Standard tier), Write transactions cost $0.015 per 10,000. Straightforward, right?

But, what is a transaction and how do you know when they happen and what causes them? A penny-and-a-half per 10 thousand transactions sounds reasonable, but do you know how many transactions your workload is going to need? Are all transactions equal? If I write a 1 MB file, is that the same as writing a 1 GB file when it comes to transactions?

Existing Documentation

There is some documentation on what storage transactions are. But this doesn't give you a complete picture. To be fair to the Azure team, it can be difficult to give a complete picture. The Azure File service can be used multiple ways, all utilizing different transactions in different ways. For example, you can connect to Azure Files over SMB, NFS, or the Rest API. Each way will have an impact on what transactions happen and how often.

My Questions

I wanted to see if there's a way to reduce transactions and optimize costs. Would writing fewer larger files reduce transactions and be cheaper than writing many smaller files?

I did some of my own experiments. Here's what I was looking for:

I wanted to see what the transaction impact is when I add files of different sizes at different times.
I also wanted to see if files with different sizes have an impact on transactions.
How could I see transactions per file share within a storage account. In the Azure Portal, you can only see transaction metrics aggregated at the account level. But you can have many file shares per storage account; is there a way to see transactions per file share?

Set Up

I created a General Purpose v2 Storage Account in Azure and called it transactiontests. Within the storage account I created a Transaction Optimized File Share and called it fileshare1. The URI for the fileshare is \\transactiontests.file.core.windows.net\fileshare1\

Azure Storage accounts collects diagnostic logs and allows you to send them somewhere where you can inspect them.

I enabled Diagnostic Settings to write the logs to an Azure Log Analytics account. I used Log Analytics to query for transactions that were happening in specific time frames.

I created a Windows VM in the same region as my storage account and I mounted the file share to the Windows VM via SMB.

I created dummy files of various sizes using this Online Random File Generator.

Now I had everything I needed to run my experiments

The Experiments

I added different files of different sizes to the file share over time. I started doing this one by one. I added a file, waited a minute for Diagnostic Settings to ship the logs to the Log Analytics workspace. Then in the Log Analytics portal experience, I would query to see what transactions were logged.

StorageFileLogs
| where TimeGenerated > now(-3m)

Here's an example of the output

I kept track of Write transactions since I was adding files to the File share. Any other transactions I just considered 'overhead.'

I added a 1 KB file, a 100 KB file, 1 MB file and so on. You can see the table below in the Results section for all the file sizes and the resulting transaction values.

After adding each file one by one. I added three copies of files of the same size to the file share. I was curious to see if there was any decrease in overhead when doing multiple operations at the same time.

Results

Transactions Per File Count and Size

# of Files	File Size (MB)	# of Transactions	# of Writes	Writers per MB	Transaction Overhead	Overhead / File
1	0.001	34	1	1	33	33.0
1	0.100	48	1	1	47	47.0
1	1.000	38	4	4	34	34.0
1	5.000	48	10	2	38	38.0
1	100.000	136	100	1	36	36.0
1	1024.000	1057	1024	1	33	33.0
3	0.001	53	3	1	50	16.7
3	0.100	58	3	1	55	18.3
3	1.000	62	12	4	50	16.7
3	5.000	85	30	2	55	18.3
3	100.000	364	300	1	64	21.3
3	1024.000	3142	3072	1	70	23.3

Breakdown of Transaction Types

OperationName	Count	% of all
QueryInfo	116	2%
QueryDirectory	46	1%
Create	212	4%
TreeConnect	21	0%
Ioctl	81	2%
Close	103	2%
SetInfo	48	1%
Write	4560	88%
TreeDisconnect	19	0%

Thanks to this tool for converting Excel to Markdown!

Observations

As I suspected, writing files of different sizes do not equal the same number of writes.
It looks like writing 1 MB counts as 1 Write transaction.
For some reason adding an exactly 1 MB file results in four Write transactions and writing a 5 MB file results in 2 Write transactions. I saw this consistently whether I added 1 file at a time or 3.
File sizes that were less than 1 MB results in a minimum of 1 Write transaction.
Some overhead could be saved by adding multiple files at a time.
The bulk of transactions were Writes. Considering many of the overhead transactions like QueryDirectory or TreeConnect cost even less than Writes, there may not be much cost savings for trying to reduce that overheard even more.

So to answer my original questions at the top of this post, here is what I learned:

Larger files result in more Write transactions. Overhead transactions cost less than Writes and are a small over-all percentage of the transactions. Therefore, I do not believe I will save much money by adding fewer larger files as opposed to many smaller files.

I also learned that by viewing the diagnostic logs, I can see transactions per file share.

Conclusion

I feel better now about understanding how Azure File transactions happen. Obviously there are more experiments I can do, like reading files, modifying and deleting them. I am reasonably confident that the transaction counts will be similar in those situations as well; deleting large files will be more transactions than deleting small ones, etc.

I hope this has been helpful!

Azure Hybrid Connection Manager Latency

Mike Richter — Mon, 11 Jan 2021 02:45:51 +0000

In this post we'll look at some non-scientific experiments I ran to measure the performance latency of the Hybrid Connection Manager (HCM) in Azure.

What is the Hybrid Connection Manager (HCM)?

Do you need to connect an on-premises resource to a service in Azure? Does the idea of setting up VPNs or Express Routes, asking IT for dedicated IP addresses, and opening up in-bound ports make you queasy? Then Hybrid Connection Manager is a potential solution. You can read all about HCM in the Azure Documentation.

HCM is built specifically for App Service, Azure's Platform-As-A-Service offering for web apps and APIs. You can use HCM to connect to an on-premises database or other service. HCM has really nice integration with App Service making it really easy to set up.

HCM is built on top of the Azure Relay Hybrid connection capability. You can use Relays to build your own hybrid connections between other Azure services and your services running locally. HCM is specifically for App Service.

The Experiments

I wanted to see what kind of latency was introduced when using HCM. Theoretically it should not introduce much. HCM communicates via Azure Service Bus. As long as the App Service and Service Bus are in the same region, there should be virtually zero latency for that communication. Of course, there will be latency over the internet to the on-premises location. Optimizing for that is outside the scope of this post.

I did two experiments

What was latency like when connecting to servers in different parts of the world?
What was latency like when connecting to servers in the same region as the app service?

In these experiments I set up VMs in different Azure regions to effectively "mock" on-premises environments. I don't have hardware for testing lying around all over the world. ;)

Experiment 1 Set Up

I created 3 VMs (SQL 2019, Win 2019) all the same SKU in Azure (A4v2, all the same configuration, using spot instances) and installed the Northwind sample database.

One in Europe, with HCM installed.
One in East US.
One in Australia, with HCM installed.

I created a simple Web API that will send the same query to any of these databases.

The Web API is deployed to Azure App Service in East US.
The App Service uses HCM to talk to Australia and Europe, the SQL Servers are not exposed to the internet.
The App Service is connected to the VNET where the East US SQL Server is. So that call is over the network via Private IP Address. The database not exposed to the internet.

Experiment 1 Test

I setup a JMeter Test Plan with 100 users, 1 second ramp up, 100 Loops per SQL Server – so 10,000 requests per SQL Server. All running the same query on each database. It took a few minutes to run.

Experiment 1 Results

Here’s the application dependency map. Europe is on top, US in the middle, Australia on the bottom. You can see Australia took the longest. That makes sense, it's the furthest from US East.

I also outputted the results of the SQL Statistics to make sure I was only comparing network time and not latency introduced due to compute time.

customMetrics 
| where name == 'SQLStatistics'
| extend networkTime = toint(customDimensions["NetworkServerTime"]), database = tostring(customDimensions["Database"])
| project timestamp, networkTime, ['database']

Here is the result which is very similar to the application dependency map above.

Experiment 2 Set Up

For experiment 2, I compared the latency between 2 VMs running in the same region as the App Service (US East).

One VM is connected through a virtual network integration
The other VM is connected via the Hybrid Connection Manager (HCM)

Remember that these are both the same size VM SKUs with the same hardware, same OS and SQL versions, running the same query in the same Azure region. So “everything else being equal”, the request time difference should be isolated to just network latency.

Experiment 2 Test

Similar to Test 1. 10,000 runs (100 users, 1 second ramp up, 100 loops).

Experiment 2 Results

It looks like HCM introduced about an average of 9ms of latency (13.1ms - 4.4ms) over connecting directly through a virtual network. In the image below SQL connected via HCM is on top and SQL connected via network is on the bottom.

Results and Final Thoughts

Overall, I found that HCM does not introduce much overhead at 9ms of additional latency inside the same data center. It costs about $10/month per listener, and there is a charge for data transfer over the first 5GB per month. More pricing information here: Azure Service Bus Pricing.

You could build something similar to HCM yourself. For instance you can use Inlets but you'd have to maintain the server yourself and pay for it. Plus, HCM is a breeze to set up and can be automated with the Azure CLI

If anyone is interested in seeing a latency comparison between Inlets and HCM, let me know!