DEV Community: martinjt

Evil Monkeypatching in C# with Rosyln Source Generators

martinjt — Tue, 03 May 2022 15:48:16 +0000

I’ve been working on an OSS project recently where I wanted to seamlessly redirect a call that a developer thinks they’re using to do some additional bits. I couldn’t find any real documentation on this, so I thought I’d investigate some ways to do it. Special Thanks to Mark Rendle for helping me with being evil.

Now I have to fully qualify my classes right back to global:: because you’re evil

Mark Rendle

What is Monkeypatching?

If you’re unfamiliar with the term Monkey patching, it’s a process of changing some code “on-the-fly”, where the author of the code didn’t necessarily want that behaviour. It isn’t, however, something you can do in C# (or .NET) in general. There are some libraries like Harmony that can do it partially, but are based around your code running and doing the patching of the libraries. The general premise is that you want to redirect a call between a method the code thought it was going to call, and some other method. This can be incredibly useful if want to change/fix the functionality of something you don’t control. Khalid (@buhakmeh) has a post on some of these approaches here, and we’re going to look at an alternative that has a very narrow use case.

What are Roslyn Source generators?

If you’re not familiar with source generators, they’re a new bit of functionality in .NET that allow you to generate source code at build time that the developer doesn’t see. They can be used in a few different ways, and specifically, they’re really useful for allowing users to use Attributes to generate additional code. However, that’s not what we’re going to do here as we don’t want the developer to have to change anything.

The Code

There is sample code here, and each commit shows the different phases. In this basic example, we’ll redirect our own code that is using System.Console to our new PrefixConsole. Our new class prepends “WithPrefix: ” to the front of all our console WriteLines.

Phase 1 – We hate ourselves

This is obviously not a real world example of what would someone would do (unless they’re REALLY evil and hate themselves). What we’re proving here is that we can redirect a call in our code from one type (in this case the Console class from the Framework) to another type (PrefixConsole). We’ll come onto something a little more interesting soon.

First, we’ll create a clean new console app

dotnet new console && dotnet run

Hello World!

Next our new Console Prefixer class, you’ll see if a normal C# class, nothing special. We’ll then internally call the System.Console class. The full namespace is important here, otherwise you’ll end up in an infinite loop.


public static class PrefixConsole
{
    public static void WriteLine(string text)
    {
        System.Console.WriteLine("WithPrefix: " + text);
    }
}

Then we’ll add using in our class that will override the usage.


using Console = monkeypatch_test.PrefixConsole;

So, that’s not bad. We’re pointing our own class at this, so we can see where it’s happening, and it’s pretty obvious where we’re sending the calls. When we use our IDE to go to the definition, it will take us to our PrefixConsole class, so there’s nothing dodgy here, just a little indirection that we probably didn’t need to do…

Now, lets take this a step further to annoy the rest of our team

Phase 2 – We hate our team mates (Global `using`)

So now, lets move that using statement as it’s too obvious what we’re doing. Also, we want ALL the usages of Console to be prefixed, and we’re too lazy to go into every class file and do it. So let’s a global using.

Global using statements came in with C# 10. They allow you to really cut down on the bloat in our files. If you’ve used Razor templates before, you’ll have done something similar where you added all the using states to View_start.cshtml.

We’ll add this Globals.cs but the name doesn’t matter. Then remove our using statement from the file.


global using Console = monkeypatch_test.PrefixConsole;

This will point all references to Console (that aren’t fully qualified) to our new PrefixConsole.

So, that’s getting bad now. It’s not obvious from looking at the code of your class that you’re being redirected. At least, however, you’re IDE will navigate you to the write class.

So, now, let’s make things even worse.

Phase 3 – We hate everyone (Source generators)

In the previous 2 phases, we’ve been using our class in our project. That’s annoying, but not that bad. It’s useful if you want to provide some consistency, and even if that class is in a NuGet package, it’s not terrible.

If it’s in a NuGet package though, people need to add a pesky line of code to their solution to do that redirection. That’s pretty bad, why would we want people who use our library to write MORE lines of code?

So lets use a source generator to do that redirection, that will REALLY mess people up and makes us popular everywhere.

First, lets move that class into our new console library called EvilConsolePrefixer (because this is where we get a little evil). We can also remove our Globals.cs too now.

We can then add the SourceGenerator packages to our new library.

dotnet add package Microsoft.CodeAnalysis.Analyzers
dotnet add package Microsoft.CodeAnalysis.CSharp.Workspaces

Now we can add the really evil part. We’re going to tell the generator to add our Global.cs to the main project at compile time, without the developer knowing.


[Generator]
public class EvilConsolePrefixerGenerator : ISourceGenerator
{
    public void Execute(GeneratorExecutionContext context)
    {
        context.AddSource("Globals", "global using Console = EvilConsolePrefixer.PrefixConsole;");
    }
    public void Initialize(GeneratorInitializationContext context)
    {
    }
}

This will add a source file into the compile pipeline with a single line that will do our redirection. Why do it this way, and not just add a global using in thing the EvilConsolePrefixer class? global usings (and usings in general) are scoped to the project, so they wouldn’t transition into the calling project. Using a SourceGenerator like this means that our global using will be added to the main project as if it was code that the developer wrote.

All that remains is to add the EvilConsolePrefixer project as a reference to our main project. As we’re doing this locally (i.e. not through NuGet), we’ll need to add an additional attribute to the import. This isn’t required if we use a NuGet package.


  <ItemGroup>
    <ProjectReference Include="..\EvilConsolePrefixer\EvilConsolePrefixer.csproj" OutputItemType="Analyzer"/>
  </ItemGroup>

The additional attribute is OutputItemType="Analyzer"

What makes this evil you might ask?

Mark Rendle said it’s Evil and I shouldn’t do it.
We shouldn’t redirect calls, it makes code hard to reason about as your context isn’t correct.
Redirecting at compile time like that could break your user’s compilation as your new code may not have all the methods and properties that the original class did.
IDE’s will navigate the user to the original class, not the one being injected at compile time
Decompiling the solution will look like the developers directly referenced your code, which isn’t actually correct.

So why would you do this Martin?

I came across this as I was trying to find a way to add a shim onto a sealed class from the Microsoft BCL. The goal was to provide a package that allowed people using that class to get a wrapper very easy without having to change their code.

Unfortunately, extension methods can’t override normal methods, and because this class (ActivitySource) doesn’t come from any kind of factory, and was sealed I couldn’t inherit and override.

In addition, what I wanted to use in my new code was [CallerLineNumber] and [CallerFilePath], which need to be done at build time as it’s the only time it is available (without PDBs). You couldn’t, for instance, use an existing interface without those properties, and simply add them to your class that implements the interface.

This solution works for that use case, and they’ll be blog post about the library I’m creating soon, despite it being evil, it serves a very real benefit. I just hope that this isn’t misused, and removed at some point in the future.

Estimates are not metrics

martinjt — Sun, 06 Jun 2021 14:44:30 +0000

This post has been inspired by the countless ~~arguments~~ conversations I have with my good friend David Whitney who has produced his own post about estimation that should also read: https://www.davidwhitney.co.uk/Blog/2021/02/10/agile_software_estimation_for_everyone

Estimation is an intrinsic part of software development, it is however, used wrongly in my opinion. Within a small iteration of a development team, the act of estimation itself is an important tool to understand complexity, perform slicing, share understanding, the “number” that may be produced as part of that is less important. In contrast, when you’re trying to establish a Time to Market, or a budget, the rolled up “number” is important. In this post I’m going to discuss what is wrong, and how to frame this with your teams, managers, leaders.

I’ll state this so it’s not ambiguous, I HATE story points. I believe the only unique value they provide is a stick to beat the team with when velocity changes, or they get the number wrong. Every other perceived value can be achieved much easier, and with more advantages, using other means.

Further clarification, if you arrived here with an anticipation that I’d give you a roadmap to remove estimation from your company, move to a “NoEstimates” culture or provide a reference for “Martin said estimation is bad” you’re in the wrong place. NoEstimates doesn’t mean we don’t estimate, it’s just a hashtag that Woody created to start a conversation about the use (and misuse) of estimates in Software Development with like minded people, and how we can do better.

What is “Estimation”?

So what exactly are we referring to when it comes to “Estimation” in software development.

Lets start with estimation as a term in general vocabulary. Estimation isn’t a software development concept, we perform all sorts of estimation in our daily lives. From estimating the impact of rush hour on our travel to work, to how much we’ll need to save each month. We do all of this without really thinking about, and without complaining, and the reason is that we know “why” we’re doing it.

According to Dictionary.com:

to form an approximate judgment or opinion regarding the worth, amount, size, weight, etc., of; calculate approximately
Estimate | Definition of Estimate at Dictionary.com

Essentially an estimate is a “guess” at a specific measure of something based on information you’ve been given. Let that sink in, it’s a GUESS. It’s sometimes influenced by analysis, or investigation on the subject, however, it’s still a guess.

Most importantly, an estimate isn’t just 1 thing, it has “dimensions” such as size, weight, duration, value, etc. Therefore in order to know which dimension we should use, we need to know how our estimate is going to be used. Why do we estimate how long it will take to get to the office in rush hour? it’s to work out when we need to set off, therefore estimating the number of miles we’ll travel around the back streets of London to get around the traffic won’t necessarily help.

Estimating in Software development is no different. Asking a team to come up with arbitrary numbers (story points, t-shirt sizes etc.) are pointless if they don’t know why, and how they’re going to be used. Further, story-points, t-shirt points hold up better at larger abstraction levels than smaller ones.

Bad estimates

The things I’ve seen as what I would call “Bad Estimates” can normally be categorised as estimates that are subsequently used as “metrics” that are monitored. David Whitney defined estimates as “like a shotgun, accurate at close range, and useless at long range”, and this rings true when you start to think about estimates as a measure. If they aren’t accurate beyond the next few iterations, they can’t (and shouldn’t) be monitored beyond that.

In short, don’t take those story points, multiple by 1.5, put that in a Gantt chart and beat the team when they’re not at the right milestones based on those guesses they made. This practice has, over time, forced developers to be over cautious, non-committal, and generally push against the estimation process in general. Resulting in everything from overly conservative estimate “It’ll take 2 years at least”, or a complete refusal to engage.

Estimation is a discovery task

One thing that myself and David agree on is that the act of performing estimation at various levels is an important part of discovery. If this is done with the team doing the work, and facilitated in the right way, you can get accurate ranges at high levels of abstraction.

Performing the estimation work will help the teams understand lots of things about the work being proposed including:

Complexity of the ask (what specialisms might be needed, Automation, performance, compliance)
Slicing (how can we split this up and deliver in smaller chunks)
Dependency (internal or external things that will dictate when it can be completed)
Spikes/Investigations (do we need to run smaller discoveries to prove something)
External factors affecting delivery (Certification, procurement, etc.)

All of these are incredibly valuable to the software development process, and are an important by product of estimation. They can help with the mental wellbeing of the team by ensuring that small victories can be celebrated, and ensure that the team doesn’t over commit in smaller cycles.

Outcomes not Outputs

When Estimates become metrics, you’re actually measuring the output of your team, and not the outcome. That is to say that you’re not measuring the “ask” of the team, but rather metrics that are by products of that in the hope that they’re indicative of the outcome you wanted.

When you move into a product business with product teams, the focus is no longer on the output of the teams, teams are autonomous and measured by the outcomes they’re asked to produce. In these teams, it’s simply not acceptable to say “but we have a predictable velocity”, or “we did all these story points though”, these just aren’t how the teams are measured anymore. The team will test, learn, adapt, and focus on constant feedback with regular value delivery.

In these sorts of teams, the value of an overall estimate based on time is at best irrelevant, at worst detrimental to the teams focus and goals. This is where it becomes important to think about the reason for estimation, and establish what the end goal of the estimation is.

Making Estimation work

Different types of estimation WILL work for different usecases. There is never a one size fits all approach to providing any kind of estimate.

Estimation can work in Software Development, and the key is knowing “Why” you’re either making estimates, or providing estimates to others. This is what allows us to provide the right “kind” of estimate, at the right level, and also inject the right amount of effort into providing the estimate in the first place. These are the sorts of questions you should be asking:

What will it be used to inform? (Project Budgets, Resourcing, Customer commitments)
What’s the risk of it being too long/short/high/low/expensive/cheap? (Project green lights, regulatory, reputation)
What does success look like for the estimate? (Delivery time, Cost, Profit)

What we’re trying to establish is the “value” of the estimate we’re giving, so we can apply the appropriate amount of time and effort to producing it.

Monitoring the estimate

Once you’ve provided an estimate on a large project, you’ll often find that people then want to use that to “track” progress against that “guess”. This is where estimates become toxic. This is specific to Large projects that span multiple months/business cycles.

Estimating the cost/timeline is providing an endpoint of the project. This is the definition of success (or failure) of the project. This doesn’t mean that you can use this to monitor whether a project is on track on a regular basis.

When you start to question the stakeholders for the reason the monitoring, what I’ve found is that what they really want to know is “Are we on track to deliver the project based on what was projected”. I’ve never found that monitoring on a granular level can give the right level of confidence to the stakeholders.

So what’s the answer? Milestones. A milestone is a point in time that you can use to check whether you’ve completed the required things that will get you towards that outcome. These don’t need to be “finished” things, they shouldn’t be a “number of stories” or an “amount of points”, they should, however, give you an indication that you’re on the right road, and you’re heading in the right direction, at the right pace. I would also encourage you to see these not a Date (i.e. 12:36pm on March 25th), but more a vague range like “End of March” or “Before the summit”.

Breaking up a larger project into a series of milestones that provide tangible outputs is BY FAR the best way to give stakeholders the confidence that the project is heading the right direction. These milestones should have defined outcomes that are tangible, demoable, and show a progression to the overall outcome of the project.

Defining the milestones is something that you should do as a collaborative task with the WHOLE team, and include the stakeholders. If the stakeholders don’t want to be involved in that, they likely only care about the end goal, and therefore the monitoring isn’t really important.

There is literally know better way to give the stakeholders the confidence that you know what you’re doing, and you’re on time/budget. There is no amount of numbers, spreadsheets, Gantt charts or (god forbid) powerpoints, that can replace actually talking to people regularly, and showing them the things they asked for.

Conclusion

To summarise my stance on this:

Large project estimation is important, and relatively easy to achieve with a degree of variance.
Monitoring of the progress of Large projects should use different information than estimation.
Estimation shouldn’t be used to produce a number that is tracked

In general, moving to outcome tracking on smaller iterations is where I desire all my teams to be. This is all about empowerment and trusting the teams to deliver value than work to metrics that don’t map to business value.

Deploying .NET 5 Azure functions with Pulumi and GitHub Actions

martinjt — Mon, 03 May 2021 21:08:30 +0000

In this post I’ll show you how to deploy the .NET 5 “Out of process” azure functions using Pulumi. We’ll be using a GitHub action to build the code, which will also create the infrastructure too, then deploy the function to that infrastructure. In this example, we’ll be using a Azure Blob Storage to store the state of our Pulumi stack.

If you’d just like to view the solution, you can find the code here: martinjt/pulumi-dotnet5 (github.com)

What is Pulumi?

Pulumi is an Infrastructure as Code framework that allows you to declare your infrastructure in the same language you do your code. In this case, we’ll be writing it in C# to match the code of our function.

What are Github Actions?

Github Actions are free Build runners provided by GitHub that run against your Github repo. You currently get 2000 free build minutes for each of your personal accounts. You can use them to do anything, and we’ll be using them to build and deploy our code to Azure for free!

Pre-requisites

For this tutorial, you’ll need:

An Azure Subscription to deploy to (You’ll need Contributor rights for Storage Accounts, Blobs, AppService/Function apps)
An Azure Storage blob for storing the state
Access to (or the ability to create) a service principal.
Fork of the repository at https://github.com/martinjt/pulumi-dotnet5

Step 1 – Creating a Service Principal

You’ll need to have a service principal that Pulumi can use to create the resources in Azure. Behind the scenes, Pulumi is hitting the Azure REST API, and for that it needs credentials

If you have a service principal, then you can skip this part.

The easiest way I’ve found to do this is using the Azure CLI. Note that currently, the CLI is creating a Service Principal with Contributor permissions, however, there is a message saying this will not always be the case. You’ll need Contributor permissions for this.


az login
az account set -s <subscriptionId>
az ad sp create-for-rbac --name pulumi-tests

In the above commands, you’ll notice that I’m setting the subscription as default. This is something that I would recommend, but you could equally just pass the subscriptionId on the command to create the principal.

This should return you a JSON blob that looks like this:


{
  "appId": "",
  "displayName": "pulumi-tests",
  "name": "http://pulumi-tests",
  "password": "",
  "tenant": ""
}

Keep hold of this as we’ll need it in a subsequent step.

Step 2 – Hosted State file setup

We’ll be storing our Pulumi state file in Azure Blob Storage, so that will need creating manually.

Pulumi maintains an “internal” dictionary of all the resources it’s created as part of the stack, and how those map to the things it needs to create. As GitHub actions don’t have a shared place for these things, we need a persistent store for them. We’ll be using Azure Blob for this.

You’ll need a storage account (using an existing one is completely fine). I’d recommend storing this in the same subscription as the infrastructure it’s managing (i.e. if you have a subscription for Dev, test and prod, put the state files for each environment in those).

The only recommendation around creating this I would have is disabling the public access.

Screenshot of the “Enable blob public access” checkbox for blob storage

Once you have a Storage account, you’ll need a blob creating within there. Note down the names of both the storage account, and the blob as these will be need in the subsequent steps.

Step 3 – Github Secrets

For the pipeline to run, you’ll need to add a couple of secrets that will grant access to the Azure Blob storage, the Infrastructure, and also encrypt the state created by Pulumi. You’ll also need the variables for the Storage account name, and the blob name.

Within your fork, setup the following secrets in => Settings => Secrets => New Repository Secret.

Note: This works equally well with Organisation or Environment secrets if you’re using them.

Service Principal

The Github Action workflow is setup to use a secret called AZURE_PRINCIPAL


      - name: Azure Login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_PRINCIPAL }}

This comes from the Action provided by azure, and expects the following json format:


{  
  "clientId": "<appId>",
  "clientSecret": "<password>",
  "tenantId": "<tenant>",
  "subscriptionId": "<subscriptionId>"
}

These should all be available from the principal that was setup in Step 1.

Pulumi Secret

You’ll need to provide pulumi with a “key” for which it will use to encrypt the state of your workflow. This is set as PULUMI_PASSPHRASE and can be any string

Storage Account

Next you’ll need to provide a valid storage account (that the principal supplied has access to). This is done with the STATE_STORAGE_ACCOUNT secret

Storage Blob

This is the name of the blob used for the Pulumi State file, and is set as STATE_STORAGE_BLOB. You need to ensure that the Service Principal supplied has Read, Write and List permissions to the blob.

Step 4 – Run the workflow

That’s it, you should now be able to run the workflow from the menus.

Screenshot of the Pipeline

The important bits

There are a few important things to note when creating both Azure functions and .NET 5 Azure Functions with Pulumi.

Functions Runtime

When you’re deploying an Azure Function with .NET 5, you’ll need to make sure that you set the AppSettings FUNCTIONS_WORKER_RUNTIME to dotnet-isolated

Changing Blobs on build

The solution I’ve provided uses the WEBSITE_RUN_FROM_PACKAGE AppSetting that points to a blob in Blob storage. This provides a nice separation of the code and function. However, due to the way that pulumi works, the Blob resource is not marked as updated, and therefore does not change the URLs. As the URL to the blob doesn’t change, the Function App will not pick up the new code.

To workaround this, I’ve added a DateTime to the blob’s name, so it’s updated on every build. I would recommend having this use the commit hash instead.

This does still have limitations in that old Function Apps could start errors as the old blob is being deleted.

Don’t publish your Infrastructure code

When you do a dotnet publish in your application, make sure that you target the application’s project, and not the solution. If your solution file contains both the application and the infrastructure, publishing the solution will result in around 200MB of extra libraries that you don’t need. This quickly fills up your GitHub data allowance.

Conclusion

Deploying an Azure function with Pulumi is pretty easy, adding in the complexity of Github Actions/Workflows is actually pretty easy too. There are a fewhop incantations that you need in order for the Action to access azure, but other than that, it was fairly smooth sailing.

I hope that forking and deploying this repo makes it easier to understand what’s needed.

Grafana On Azure – AzureAD Authentication

martinjt — Sat, 10 Apr 2021 20:37:46 +0000

This is part of a multi-part series on how to deploy and host Grafana safely, and cheaply on Azure, and how to get some decent visbility from Azure Monitor/App Insights through it. Hopefully parts of this will be useful.

Part 1 – Hosting/Configuration
Part 2 – Azure MySQL storage
Part 3 – SSL with LetsEncrypt
Part 4 – Azure AD Login (this post)
Part 5 – Azure Monitor Datasource (Coming Soon)

Pre-requisites

If you’ve followed the previous 3 steps, you’ll have everything setup correctly. Otherwise, you’ll need the following:

Grafana instance (obviously)
Access to the grafana.ini file on that instance
Grafana using SSL (this is a requirement for AzureAD’s response/callback URLs)
AzureAD instance

Under both scenarios, you’ll need:

Access to create App Registrations in the Azure Portal.

Overview

In this post, we’ll be looking at adding Azure Active Directory (AzureAD) support to a Grafana instance. This is what I would advise if you’re hosting on Azure as you’re already likely to have all of your potential Grafana users setup in Active Directory, and either this is AzureAD native, or you have passwords sync’d with a standard Active Directory instance.

You will still be able to have local users, as well as AzureAD, and I’d recommend keeping the admin user with a very strong password for maintainence.

Using AzureAD as your authentcation system for Grafana also allows you to have Two-Factor Authentication (2FA) for Grafana by enabling this within AzureAD.

What is AzureAD?

This is the cloud based authentication system used to access the Azure portal. If you’re using Azure, you likely already have one. It’s the next generation Active Directory which is Microsoft’s centralised IAM system.

It provides interfaces for common authentication protocols like OIDC (OpenIdConnect) and SAML2. This is what Grafana will use to verify the identity of your users.

Step 1 – Create the Azure App

The first step is to create an Azure AD “Application” that will be what is used for Grafana to communicate get access to Azure. For this step, the application will be used to identify user information. We’ll be breaking the Application creation into 2 steps, the first will allow the use the application, then the second will allow you to map Azure AD groups to Grafana roles.

The Name is a friendly name that you users will see the first time they try to login. Use something recognisable to your user base, and also descriptive to ensure that you users trust the login.

The Redirect URI is required for this Grafana integration. You’ll need you domain here and the value should look like this:

https://<domain>/login/azuread

It’s important that this is a domain and not an IP as you’ll need to use HTTPS and have a valid certificate.

Once the app is created, you’ll need to record 2 details. The TenantId and the ClientId:

These will be needed for the grafana config in the next steps.

Next you’ll need to create a “Client Secret” which is how Azure can know that it’s your Grafana instance, rather than a someone else’s.

Client the “New client secret” link, then give this secret a descriptive name. The maximum expiration is 2 years, however, I’d recommend using 6 months and schedule a reminder to update it.

Once you’ve added the secret, you’ll need to copy this out as it will be required in the next steps. You’ll only be able to copy this secret at this stage, so it’s important that you copy it out. before leaving the page.

Step 2 – Grafana Config

Next you’ll need to tell grafana about the config from the Azure AD Application. There is a section specifically for this in the grafana.ini file called [auth.azuread] The important things here are:

Name = Friendly name, it’s not really used anywhere

Enabled = set this to true

client_id = that you copied from the main Azure AD app screen

client_secret = that you copied from the main Azure AD app screen

scopes = openid email profile

auth_url = https://login.microsoftonline.com/{tenant}/oauth2/v2.0/authorize replacing {tenant} with the tenant ID from the main Azure AD App screen

auth_url = https://login.microsoftonline.com/{tenant}/oauth2/v2.0/tokenreplacing {tenant} with the tenant ID from the main Azure AD App screen

[auth.azuread]
name = Azure AD
enabled = true
;allow_sign_up = true
client_id = 
client_secret = 
scopes = openid email profile
auth_url = https://login.microsoftonline.com/{tenant}/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
;allowed_domains =
;allowed_groups =

Restart the service and you should now be able to login with your Azure AD credentials.

sudo systemctl restart grafana.server.service

Conclusion

In this post you’ve seen just how easy it is to enable AzureAD authentication. There is more that you can like enable groups for the users, and removing the ability to have a local login form. Those are all for another post.

In the next post, we’ll look at using this Azure AD application to enable access to Azure Monitor, and Azure Log Analytics.

Grafana on Azure – Enabling SSL with LetsEncrypt

martinjt — Sat, 13 Mar 2021 22:02:59 +0000

This is part of a series of posts about running Grafana on Azure. Checkout the others

Part 1 – Hosting/Configuration
Part 2 – Azure MySQL Storage
Part 3 – Enabling SSL with LetsEncrypt (this post)
Part 4 – Azure AD Login
Part 5 – Azure Monitor Datasource (coming soon)

What is LetsEncrypt?

LetsEncrypt.org is an initiative to promote sites using SSL. Regardless of whether there is a data you feel is critical, SSL should still be enabled. LetsEncrypt provides completely FREE SSL certificates that you can use on any domain. The certificates are limited to 30 days, which means that you need to regnerate the key regularly. However, there is a utility called “certbot” which will automate this for us.

There are multiple ways you can generate the certificates, but for this example, we’ll be use what is referred to as the HTTP Challenge method. It’s generally a little bit more secure to run the DNS Challenge method as this doesn’t rely on you opening up additional ports, however, we won’t be covering that method here as it specific to the DNS provider you’re using.

Step 1 – Setup your domain name

In order to get a valid certificate, we’ll need to have a domain name, this won’t work on an IP address.

Setup your domain name to point to the IP address of your grafana instance. To make sure that you’re not going to hit issues, I’d recommend checking using dig or nslookup to ensure that it’s correctly pointed. Also, try hitting the grafana using that domain on port 3000 (i.e. http://{domain}:3000/).

Step 2 – Open the required ports

You’ll need to open up ports in your Network Security group so that both LetsEncrypt can communicate on Port 80 (http) for the HTTP Challenge, and your users can communicate on Port 443 (https).

Navigate to your Virtual Machine in the Azure Portal, Click “Networking” then “Add Inbound Port Rule”.

For “Destination Port Ranges” enter both 80 and 443 separated by a comma e.g. 80,443, and click Save.

Step 3- Let Grafana Access the certificates on the filesystem

The grafana process will used the certificates as it has it’s own inbuilt HTTP server. Therefore, the process will need access to the location where the certificates are generated.

First, we’ll create a group that will provide access to our ssl certificates:


sudo groupadd sslcerts

Then we’ll create the directories for the certificates, and change the ownership to our newly created group, and modify some permissions


sudo mkdir /etc/letsencrypt
sudo mkdir /etc/letsencrypt/archive
sudo mkdir /etc/letsencrypt/live
sudo chown -R root:sslcerts /etc/letsencrypt/
sudo chmod 755 /etc/letsencrypt/archive
sudo chmod 755 /etc/letsencrypt/live

Finally, we’ll add grafana’s process user to our newly created group.


sudo usermod -G sslcerts -a grafana

Step 4 – Install and run LetsEncrypt Certbot

Next we’ll need to install the certbot which is the application that will communicate with LetsEncrypt. There is a package in APT, so that’s pretty easy.


sudo apt install -y certbot

Then we can run the tool in standalone mode.


sudo certbot certonly --standalone

You’ll need to give LetsEncrypt an email address, and agree to their terms, then add the domain you want to generate the certificate for. This will be the one you setup in Step 1, without https, etc.

Certbot will then setup a temporary http server running on port 80 that will allow LetsEncrypt’s servers to verify that you actually own the domain by literally just hitting the url.

Once that’s done, you’ll find that there is now a certificate file in the folders we created in step 3.

azureuser@grafana-azure:~$ sudo ls -hal /etc/letsencrypt/live
total 16K
drwx------ 3 root root 4.0K Mar 13 21:08 .
drwxr-xr-x 9 root sslcerts 4.0K Mar 13 21:08 ..
-rw-r--r-- 1 root root 740 Mar 13 21:08 README
drwxr-xr-x 2 root root 4.0K Mar 13 21:08 grafanablog.martinjt.me

You’ll notice that the certbot hasn’t honoured our changing of the directories groups, so we’ll need to rectify that. If we change it on the files, certbot will honor them on renewal.

Step 5 – Configure Grafana to use SSL

Open the grafana config for editing


sudo nano /etc/grafana/grafana.ini

Edit the following settings:

protocol = https
domain = <your-domain>
enforce_domain = true
root_url = https://<your-domain>
cert_file = /etc/letsencrypt/live/<your-domain>/fullchain.pem
cert_key = /etc/letsencrypt/live/<your-domain>/privkey.pem

Now restart the grafana service:


sudo systemctl restart grafana-server.service

You should now be able to access your service using:

https://<your-domain>:3000/

Using port 443 for https

by default, Grafana won’t be able to listen on port 443 due to restrictions in Linux. You’ll need to enable this using the following command:


sudo setcap 'cap_net_bind_service=+ep' /usr/sbin/grafana-server

Now we can edit our grafana.ini again:

http_port = 443

Then restart our service


sudo systemctl restart grafana-server.service

Now you should be able to access your grafana instance without the port:

https://<your-domain>

Additional Links

Setup Grafana on Ubuntu 18.04 with LetsEncrypt

Grafana on Azure – Azure MySQL Storage

martinjt — Sat, 13 Mar 2021 22:01:34 +0000

In this multi-part series you’ll learn how to host Grafana safely, and cheaply on Azure, and how to get some decent visbility from Azure Monitor/App Insights through it.

Part 1 – Hosting/Configuration
Part 2 – Azure MySQL Storage (this post)
Part 3 – SSL with LetsEncrypt
Part 4 – Azure AD Login
Part 5 – Azure Monitor Datasource (Coming Soon)

In the last post, you’ve setup a basic Grafana VM on Azure that is using a database on the VM’s disk as it’s datastore.

In this post, we’ll move that database to Azure PaaS.

What is Azure Database for MySQL?

This is Azure’s managed version of MySQL. It’s cheap (~£25/month) and supports everything you’d expect an Azure PaaS platform to do. There’s no patch management, backups, etc. just an endpoint for you to hit. What’s not to love?

This makes it great for our database for Grafana as the point of a monitoring system is be always up, and you don’t want to be monitoring your monitoring system with your monitoring system. Having a single VM, with the database, the frontend, etc. as a single point of failure just isn’t great. Having then to add backups of all your data regularly, and suddenly the quick and easy Grafana instance is now a management overhead in itself.

Step 1 – Azure MySQL Setup

Grafana stores a very small amount of data, and can be stored in a shared server if you already have one. We’ll be setting up a server from scratch for this, but most of the steps will work on an existing server if you have one.

You can get away with the smallest possible instance, which a B1 at ~£20/month, with the minimum base storage (5GB). This will give you the ability to have backups, SSL, and DB maintainance. I’m going to guess that this is likely nothing in comparison to the other things you’re running, and also nothing in terms of the ~1day/month it would take you to verify all backups, etc. if you ran your own.

You’ll need to create a database and a user for Grafana. Later in this post, we’ll look at locking down the Azure MySQL instance, for now, you’ll need to manually add your IP.

Click the link “Add current client IP Address”, and then click save. This will add your public IP Address to the SQL Server.

Now connect to the MySQL instance from the command line (if you don’t want to install mysql, running the docker image is a nice alternative).

mysql -h {servername}.mysql.database.azure.com -u {adminuser}@{servername} -p

Your user will need full privileges to manage the database as grafana has builtin database migrations.

CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
CREATE DATABASE grafana_data;
GRANT ALL ON grafana_data.* TO '<username>'@'%';

That’s it for MySQL. You’ve created the database, and it’s setup so you can access it. Next we’ll make grafana use it.

Step 2 – Configuring Grafana’s DB

The first thing we’ll need to do is allow Grafana to access the database. For now, we’ll enable the “Allow access to Azure Services”, this will enable access from Azure IPs which will work for now, and we can secure this later.

Next we’ll need to tell grafana where to find our database, and how to log in. SSH to your VM and open up the grafana config file

sudo nano /etc/grafana/grafana.ini

In there, find the section labelled [Database] and set the following values, leaving all the others at their defaults.

Note: the ini file uses ; to comment out the settings, remove it to set them.

type = mysqlhost = <servername>.mysql.database.azure.com:3306name = <database-name>user = <username>@<server-name>

ssl_mode = skip-verify

ca_cert_path = /etc/ssl/certs/ca-certificates.crt

Now restart grafana, and you’ll be need to login again with the default credentials.

sudo systemctl restart grafana-server

If you end up with your url timing out, check the logs.

sudo tail -f /var/log/grafana/grafana.log

A common error is accessing the MySQL instance. This will manifest as the log tail above stalling at “Starting DB Migrations”. Likely causes are:

Username hasn’t include @<server-name>
ssl_mode and ca_cert_path aren’t set
MySQL firewall rules haven’t been setup.

If all has gone well, you’ll be presented with the Grafana logon screen, and be asked to input the default admin password and reset it.

Step 3 – Securing MySQL connection

Right now the MySQL instance has been setup so that your public IP and literally anything that runs on Azure can access it. This is not a desireable situation, so we’ll need to secure that further.

There are a few things you can do to secure the MySQL instance.

Option 1: IP restrictions

You can restrict your MySQL instance to just the IP of your VM. This is the simplest approach, but isn’t really scalable as if you kill the VM, you’ll need to update the IP. It also isn’t scalable if you bring on multiple instances/scaling.

That said, to get started, this is a completely viable approach. You can set this by going to you MySQL Server > Connection Security and adding your VM public IP, and then disabling “Allow access to Azure services”

Option 2: VNet Security

This is generally my preferred option. This allows you to make your MySQL Server only accessible from within a specific VNET’s subnet. Unfortuately, this is only available on Standard edition MySQL servers, which come in at ~£50/month.

Rant on “Allow Access to Azure Services”

A quick gripe with this setting. It can be tempting to just enable this setting as Azure Managed services can then access your MySQL instance. However, this is NOT limited to the services in your subscription. Therefore, anyone running their app in Azure can access your MySQL instance. Given the amount of people running in Azure, this doesn’t really provide much security. Use with caution.

Conclusion

In this post we’ve connected our grafana instance to an Azure MySQL instance so that we get backups, security and a persistent database. This means that should our Grafana VM become corrupt, or otherwise become compromised, we can just terminate it and bring up a new one from scratch.

In addition, you can also setup as manage VMs for grafana as you want, add load balancers, host it in containers, anything.

In the next post, we’ll look at securing the frontend using SSL, and generating certificates using LetsEncrypt

Grafana on Azure – Hosting/Configuration

martinjt — Sat, 13 Mar 2021 21:59:50 +0000

This intended to be a multi-part series on how to host Grafana safely, and cheaply on Azure, and how to get some decent visbility from Azure Monitor/App Insights through it. Hopefully parts of this will be useful.

Part 1 – Hosting/Configuration (this post)
Part 2 – Azure MySQL storage
Part 3 – SSL with LetsEncrypt
Part 4 – Azure AD Login
Part 5 – Azure Monitor Datasource (Coming Soon)

Overview

In this first post, we’ll look at building a Grafana Instance with a managed MySQL instance using Azure MySQL.

The aim simplicity, speed of deployment, reduced management overhead and cost.

There as a balancing act to be done here. Speed and simplicity could be achieved using a database on the machine, that’s then backed up, at the expense of management overhead. Reduced Management could be achieved using ACI or App Service, at the expense of cost (unless you already have a container infrastructure). The solution here, I believe, gives a fair balance

Why Grafana

Grafana has been my go-to tool for a long time and therefore where I’m comfortable. I’m sure that some or all of what I’m doing here could be achieved with Azure Dashboard, but I haven’t learnt this yet.

Grafana is a graphing tool that allows you represent data from multiple sources. In modern development, we don’t use just a single provider for our platforms, therefore visualisation needs to be able to draw in multiple sources.

Grafana Cloud

A quick note on Grafana Cloud. This is Grafana’s Managed Service offering and provides an easy way to get up and running without hosting it yourself. The pricing is a little expensive once you get to 20-30 users, which the installation I’m proposing can handle. For 10 users, at $49/month, it’s a steal, and I’d recommend that route. If you want to start pushing these dashboards out to business stakeholders, every developer, it can start to get expensive.

There is also now an AWS Managed version, which is even more expensive at $9/month/user, https://aws.amazon.com/grafana/pricing/. This seems very expensive to me, especially as that doesn’t include the enterprise plugins like Timestream.

_ Note : for Grafana staff, you should should look at adding unlimited “View Only” users, or a 4:1 ratio of free to paid users just for viewing dashboards like stackholders and wallboards._

Step 1 – Create the Azure Virtual Machine

Grafana requires VERY little resources, unless you’re running this at scale. This is because, for the most part, it’s just proxying queries from the user’s browser to the Analytics backend (e.g. InfluxDB, AppInsights, CloudWatch, etc.). This changes as you add support for alerting, but it’s still not a massive amount, and as we’ll be using Azure analytics, most of the alerting will come from there.

For this tutorial, I’ve successfully run this using a B1Ls which is currently ~£5/month. We’ll be using Ubuntu, so all the commands and paths will be based on that. For this we’ll be using Ubuntu 20.04.

Create a standard B1LS virtual machine with Ubuntu 20.04, you’ll need to open up port 22, and have access to the SSH key. It’s out of scope of this post to explain how to create VMs.

Step 2 – Install Grafana

SSH onto the machine


nano install-grafana.sh

Then paste this into the file.


#!/bin/sh
sudo apt install -y software-properties-common

#add grafana to list of allowed software
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

sudo apt update && apt install -y grafana
sudo systemctl enable grafana-server.service
sudo systemctl start grafana-server

You’ll notice from the above that it’s adding a respository. This is adding grafana’s package respository that will override the default ubuntu packages. This means you’ll be getting the very latest version of Grafana.

Save the file (Ctrl+X) and then we need to make the file executable


chmod +x install-grafana.sh

As the script will be installing some things, you’ll need to run it with super user permissions.


sudo ./install-grafana.sh

That will get you a running grafana instance, with a local sqlite database on the machine. Meaning that if the machine dies, and you haven’t used persistent disks, you’ll lose all of the data. Obviously we could stop here and use persistent disks, but that would mean we’d lose the ability to scale-out, and also the ability to image the machine in case of corruption, etc. You’re stuck with “fixing” the machine’s image, which isn’t the cloud way.

Once you’ve run the above you’ll need to open up grafana’s default Port (3000).

Once that port has been opened up, you should be able to access the grafana instance using the following

http://<ip>:3000/

The default credentials are admin/admin, and once you’ve put those in, you’ll need to set a new password for admin. In a further step we’ll add Azure AD credentials for logging, but for now, this will be what is used to login.

Conclusion

This is post shows the basics of getting a Grafana instance working on an Azure VM. In the next posts we’ll work on making this more resilient and secure.

You can stop at this point and have a fully functioning grafana instance that you can configure and use. You can mitigate outages by making the disk persistent so it can be restored into another VM. However, in the next post, you’ll see how we can leverage Azure MySQL PaaS to host the database so the VM will become largely irrelevant.

Pulumi Multiple projects with Custom Backends

martinjt — Thu, 04 Mar 2021 22:19:04 +0000

I’ve been working with Pulumi to create a reference architecture for a client that provides individual team autonomy while providing some shared resources. In addition, the client is also wanting to utilise a custom backend in Azure Blob storage. This presented some issues as the documentation around projects, stacks and stack references for custom backends. Hopefully this post should clear up some of those patterns.

What is a StackReference?

A stack reference in pulumi is a way of your current project accessing the outputs of another project. This is really useful when you’re sharing things like a Load Balancer, API Gateway, AppService Plan, etc, between multiple services.

In Terraform this would be a “Remote State” that you’d bring in as a datasource.

More information on Stack References: https://www.pulumi.com/docs/intro/concepts/stack/#stackreferences

Why Multiple Projects?

When you’re working with multiple teams of people, you will inevitably hit a situation where you’ll want to have multiple, independently versioned and iterated infrastructure projects. Some of the time, there will be outputs from these projects that other projects depend on.

What is a Custom Backend?

Pulumi offers an awesome service for managing the state at https://app.pulumi.com. This provides an intuitive UI, along with some cool features like Deployment Histories, and integration into CI/CD pipelines etc. This is my go to, and if you can justify the cost, it’s something I would highly recommend. However, this isn’t mandatory as you can also host the backend yourself as you would with terraform.

You can choose to store that recorded state, however, in your own storage engine like S3 or Azure Blob Storage. Full information of the storage options are here:https://www.pulumi.com/docs/intro/concepts/state/#logging-into-a-self-managed-backend

Projects and Custom Backends

When using app.pulumi.com for hosting you backend, the providers automatically take into account your project name, as well as the stack name. Therefore if you run this:


pulumi stack init dev

In a project called demoyou’ll have stack calledmy-awesome-org/demo/dev. However, when it comes to using a custom backend, the default syntax doesn’t provide this separation.

When using Azure Blob Storage, that syntax will result in a stack simply called dev, therefore you will need to namespace the stack manually.


pulumi stack init demo.dev

You’ll note that I’ve not added slashes, and instead used periods. This is because the CLI doesn’t allow slashes for stack names. I’ve also not added the organisation name as that’s specific to the pulumi service.

Referencing using a StackReference

Now we have 2 stacks, we’ll need to reference the first in the second. We can do this in C# using the following code, referencing the now qualifed stack reference.

var stackReference = new StackReference("shared", new StackReferenceArgs { Name = "demo.dev" });

Limitations

This doesn’t provide the ability to reference a stack in a separate blob storage container, this isn’t something that’s supported right now as far as I can tell.

Speed up legacy ASP.NET applications with HttpContext.Items caching

martinjt — Sun, 21 Feb 2021 16:06:33 +0000

One of the goals of most websites is to return the page for a user within the fastest time possible. Adding caching of frequently accessed data is a really easy way to do this, however, in legacy applications, there can be a lot of context heavy code making it hard to understand what can be cached, and what can’t.

What I’ve seen over and over again is code that runs the same function multiple times, and within that function using the current users’ information.


        public string GetUserAge()
        {
            var userId = HttpContext.User.Claims.FindFirst(c => c.Name == "UserId");
            var user = context.Users.FirstOrDefault(u => u.Id == userId);
            return user.Age;
        }

In the above block, we’re extracting the UserId from the claims, then getting the whole user from the database and returning the age. This is an example I’ve seen a number of times and it’s easy. It was written with the best of intentions, and when it was first used, a single database hit was absolutely fine. However over time, people have gradually added calls to this function in other places. Those functions are now used in lots of places too.

The biggest example of this is when in a web-page, you start to add multiple components/views. Even worse when it’s a CMS with functional components that aren’t thought about ahead of time. That means that for a single Web Request for a page, that function could be hit 5-10 times, each resulting in a database hit (putting aside that there will be caching if you’re using EntityFramework or other ORMs that support In-Memory entity caching).

So how do we improve the performance of this code that has now become a critical perfomance bottleneck.

Options

Option 1: Rewrite EVERYTHING

Perfectly valid approach, a little nuclear, but if you’ve got this problem, it’s probably in lots of places. That said, this is likely going to be a long-term goal, and not a quick win.

Option 2: Cache all the users

This is normally the solution I see implemented. There will be some kind of global cache, or shared cache like Redis that stores all the users. Alternatively, they’ll be an In-Memory list of the users.

This is, again, a perfectly valid way of doing it. You’re still adding lots of latency, and possibly even serialization costs. On top of that, you’ve also now got a global cache to maintain. It does have the benefit that it reduces the database impact over multiple requests though.

Option 3: Just cache it for the current request.

This is my preferred option as you have only a few lines of code, and now, only a single request to the database for each context based execution (e.g. Web Request).

How?

HttpContext should be familiar to most people in the ASP.NET world. It’s been around since… well I can’t remember so “awhile”. In the full Framework world, it was a static you could reference that had all the properties of the current request, like the URL, the cookies, etc.

One of the under-used parts of the HttpContext object is something called “Items”, with a dictionary that only lives for the life of the current request. This opens up some safe caching options, that will also be fairly memory efficient. Here’s the code above but adding in a Items cache.


        public double GetUserAge()
        {
            if (HttpContext.Current.Items.Contains("UserAge"))
                return (double)HttpContext.Current.Items["UserAge"];

            var userId = HttpContext.Current.User.Claims.FindFirst(c => c.Name == "UserId");
            var user = dbContext.Users.FirstOrDefault(u => u.Id == userId);

            HttpContext.Current.Items["UserAge"] = user.Age;

            return user.Age;
        }

In normal operation, there will likely be no performance degradation, even if there is a single hit, as it’s just adding an item to a dictionary. However, if this is hit multiple times inside a single request, you’ve just reduced the database calls to a single call per-request.

Infrastructure Autonomy using DNS Delegation and internal Top Level Domains

martinjt — Sat, 14 Nov 2020 18:06:58 +0000

In this post we’ll talk about using a specific Top Level domain to separate your internal application infrastructure addresses from what you’re users see. Further, how to provide team level autonomy to using DNS delegation to provide a predictable naming strategy.

The Problem

One of the big issues with DNS management is the security elements of allowing people to add what they want to your prized potential, your front facing domain. I’ve seen this process create teams that simply manage the DNS, which is not great really very cost effective. On the opposite side, I’ve seen organisations where everyone has access to change any DNS entry, or even transfer the domain ownership.

When working with infrastructure as code, and creating things like AWS ALB’s or Azure Load Balancers, their names are… less than predictable. Further, when you’re treating this environments as something that can be torn down and spun up, those services within Azure or AWS will get new, random, names each time. This means that any other teams relying on these services will have to constantly change their configurations.

Providing team autonomy to manage as much of their stack is hard when they have to submit a request, signed by the CEO, COO, Chairman, etc. that takes a week to anction, just to point their subdomain to the new things they’ve created.

The Solution

Providing a predictable, static name for all your internal resources allows for things outside of the immediate teams to be “slow moving”, while still allowing the autonomy of the team to iterate at it’s own pace.

I also like to use predictable DNS entries to allow people to navigate and identify the purpose, team etc. of the service. This provides a lot of consistency when it comes to cross team working, and further allows the “underlying” resource to change, without external notification.

What I’ve done at every business I’ve been into recently is create a “delegation” infrastructure.

I’ve spent quite a lot of time defining and using an approach to combat this in both Azure and AWS. It’s not exactly “news” or “complicated”, what I would say is that this has been a stumbling block in cloud understanding/adoption. Understanding how to disassociate the “old school” static servers with known IP addresses, with transient services with IP addresses you don’t control is where a lot of the old school developers and engineers have struggled.

The Idea

The basic idea starts with separating how you address your “internal” addressing of services with what the end users see.

So, address your services using an internal domain name, and then link them together using CNAME entries.

E.g. if your domain is http://www.martinjt.me, use a new domain internal-martinjt.me

In this example, our users use http://www.martinjt.me However, our server lives on an emphemeral IP inside AWS which is currently 50.50.50.50. The team developing their website DO NOT have control over martinjt.me entries, as this is managed centrally by a shared team for “security and/or compliance” reasons. They do have access to add/delete/amend records on the internal domain though, which they use to setup a consistent record that is updated every time they change their backend service. Finally, the central/shared team has setup a CNAME entry to that internal domain, that won’t change.

The result in this example is that the team has access to change the destination of http://www.martinjt.me, without having direct access to it.

Subdomain Delegation

You can take this a step further, so that you don’t need to setup lots of Top Level domains.

Both Azure and AWS have the ability for you to setup DNS Zones for a Subdomain (e.g. team1.internal-martinjt.me, team2.internal-martinjt.me, etc.). These DNS Zones can then be setup inside of that teams control, and they can create more entries. Personally, I prefer to setup a structure like this:

{application|service}.{env}.{team}.internal-martinjt.me

e.g.

https://webserver.test.booking.internal-martinjt.me

In this example, “booking.internal-martinjt.me” is delegated to the “Booking” team, that manage a series of applications, and they then create DNS entries for each of their applicaitons, in each environment.

Conclusion

Granting access to your frontend domain is risky, and requires a lot of trust. However, providing the team with the autonomy is predicated on granting them access to it.

All the cloud providers are able to provide this ability, and further, you’re also able to use a DNS provider like GoDaddy, etc. that you can give the teams access to.

This doesn’t, however, mitigate the security risk of this new domain being used to redirect the main site. It does stop that from being a permanent issue where that security is breached.

Manage Cross team terraform and azure-cli versions with docker

martinjt — Mon, 07 Sep 2020 18:51:28 +0000

One of the issues with having multiple teams, and pushing for autonomy to choose everything from infrastructure to languages, is making sure that you have the right versions of everything installed.

As developers, we want to be on the bleeding edge, playing with new things, new versions, etc. However, that can have an impact beyond our machine, and can introduce weird issues with cross-team work and even within the team.

I’ve been working on an approach to isolate that, and streamline the workflow with getting new team members up to speed quickly. Essentially, this boils down to using docker for just about everything. In this post, I’ll walk through doing this for Terraform and the Azure-CLI.

Caveat: I will admit that there are generally images in wild, already built with this combination. However, this approach allows for you to customise the image at a repository level. It allows the primary team to provide the other team’s developers an isolated environment for them to enact change in their team in a frictionless way.

Dockerfile


FROM mcr.microsoft.com/azure-cli

ENV TERRAFORM_VERSION 0.12.28

RUN apk add --update wget ca-certificates unzip git bash && \
    wget -q -O /terraform.zip "https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip" && \
    unzip /terraform.zip -d /bin && \
    apk del --purge wget ca-certificates unzip && \
    rm -rf /var/cache/apk/* /terraform.zip

VOLUME ["/data"]
WORKDIR /data

RUN printf "\\nalias tf='terraform'" >> /root/.bashrc

ENTRYPOINT ["/bin/bash"]

Lets have a look at what it’s doing.


FROM mcr.microsoft.com/azure-cli

This means we’re going to pull the latest version of the Azure CLI (or use the version we’ve already pulled). If we want to be a little more specific, you could add a “:” tag to the end.


ENV TERRAFORM_VERSION 0.12.28

RUN apk add --update wget ca-certificates unzip git bash && \
    wget -q -O /terraform.zip "https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip" && \
    unzip /terraform.zip -d /bin && \
    apk del --purge wget ca-certificates unzip && \
    rm -rf /var/cache/apk/* /terraform.zip

This will install the specific version of terraform, then clean itself up afterwards.


VOLUME ["/data"]
WORKDIR /data

RUN printf "\\nalias tf='terraform'" >> /root/.bashrc

ENTRYPOINT ["/bin/bash"]

Finally, we add a volume tag, which is where our terraform code will be mounted and add an alias as typing “terraform” is way too many keystrokes when you can just right “tf”.

We can then build the image


docker build . local/tf:12-28

The additional “:12-28” means that we’re keeping it versioned.

Running

Then run the image


docker run -it --rm -v ${PWD}:/data -v azure-cli:/root/.azure --name tf-inf local/tf:12-28

The secret sauce here is the name and the volumes. The adding of ${PWD} means you’re mounting your current directory as the source for the container.

The azure-cli volume is what allows for persisting your validation, so it’s not lost when you/if you kill the container, and also allows it to be shared with other containers.

Note: the --rm flag will remove the container afterwards. This is very useful due to a bug with WSL2/Docker that deletes the contents of the mount after a restart occasionally.

Conclusion

Managing cross-team work is hard, and when it comes to being an infrastructure consultant, or someone who is working in anyway across multiple autonomous teams, docker is definitely your friend!

DEV Community: martinjt

Evil Monkeypatching in C# with Rosyln Source Generators

What is Monkeypatching?

What are Roslyn Source generators?

The Code

Phase 1 – We hate ourselves

Phase 2 – We hate our team mates (Global using)

Phase 3 – We hate everyone (Source generators)

So why would you do this Martin?

Estimates are not metrics

What is “Estimation”?

Bad estimates

Estimation is a discovery task

Outcomes not Outputs

Making Estimation work

Monitoring the estimate

Conclusion

Deploying .NET 5 Azure functions with Pulumi and GitHub Actions

What is Pulumi?

What are Github Actions?

Pre-requisites

Step 1 – Creating a Service Principal

Step 2 – Hosted State file setup

Step 3 – Github Secrets

Service Principal

Pulumi Secret

Storage Account

Storage Blob

Step 4 – Run the workflow

The important bits

Functions Runtime

Changing Blobs on build

Don’t publish your Infrastructure code

Conclusion

Grafana On Azure – AzureAD Authentication

Pre-requisites

Overview

What is AzureAD?

Step 1 – Create the Azure App

Conclusion

Grafana on Azure – Enabling SSL with LetsEncrypt

What is LetsEncrypt?

Step 1 – Setup your domain name

Step 2 – Open the required ports

Step 3- Let Grafana Access the certificates on the filesystem

Step 4 – Install and run LetsEncrypt Certbot

Step 5 – Configure Grafana to use SSL

Using port 443 for https

Additional Links

Grafana on Azure – Azure MySQL Storage

What is Azure Database for MySQL?

Step 1 – Azure MySQL Setup

Step 2 – Configuring Grafana’s DB

Step 3 – Securing MySQL connection

Option 1: IP restrictions

Option 2: VNet Security

Rant on “Allow Access to Azure Services”

Conclusion

Grafana on Azure – Hosting/Configuration

Overview

Why Grafana

Grafana Cloud

Step 1 – Create the Azure Virtual Machine

Step 2 – Install Grafana

Conclusion

Pulumi Multiple projects with Custom Backends

What is a StackReference?

Why Multiple Projects?

What is a Custom Backend?

Projects and Custom Backends

Referencing using a StackReference

Limitations

Speed up legacy ASP.NET applications with HttpContext.Items caching

Options

Option 1: Rewrite EVERYTHING

Option 2: Cache all the users

Option 3: Just cache it for the current request.

How?

Infrastructure Autonomy using DNS Delegation and internal Top Level Domains

The Problem

Phase 2 – We hate our team mates (Global `using`)