DEV Community: Rob Van Pamel

Find your application's hidden secrets using opentelemetry

Rob Van Pamel — Thu, 23 Feb 2023 00:00:00 +0000

When your project is growing, the need to gain more insights in your application is growing along. You need to know in detail what is happening inside the application, either to resolve a bug or to improve some performance issues. In the past these insights were mainly received by adding extensive logging. The most of us have already done this, and is the easier step to start with. If you would like to go a step further, adding trace and metric information is the way to go. Trace information is extremely valuable when you start to work with distributed applications. They allow you to follow a request across multiple systems.

The combination of logs, traces and metrics are called telemetry data. While there are different ways to collect the data above, OpenTelemetry should be the defacto-standard these days. Opentelemetry is a Cloud Native Computing Foundation (CNCF) project because it provides us with an open standard to collect telemetry data from our applications. (the standard isn’t fully approved yet, but this shouldn’t hold us from using it)

The open telemetry projects consists out of several topics which i would like to explain

Signals
Open Telemetry Collector
Instrumenting your application

Signals

In open telemetry, the signals are the actual collection of different telemetry data that are supported, logs, traces and metrics.

Logs

A log is a timestamped text record, either structured (recommended) or unstructured, with metadata.

Traces

Traces give us the big picture of what happens when a request is made by a user or an application. For example a GET request was executed.

Metrics

A metric is a measurement about a service, captured at runtime.

Open Telemetry Collector

The open telemetry Collector is the process which will act as a gateway to receive the signals (telemetry data), process it and send to your observability tool. Using the open telemetry collector isn’t a mandatory step. It is possible to send the telemetry data to your observability tool, however this isn’t recommended for production environments. The collector can take care of retries and more.

There are 2 collector variants available, the ‘normal‘ and the contrib. As you expected, the contrib variant includes more receivers, exporters and processors built by the community.

Receivers

Here you can define how you will ingest data into your collector, it can be one source, but nothings holds you from using multiple receivers. The default is the Open Telemetry Protocol (OTLP). OTLP runs on HTTP(4318) and on gRPC (port 4317). Other options which are out-of-the-box available are Jaeger and Prometheus, but if you use the contrib variant, you have even more options.

https://opentelemetry.io/docs/collector/configuration/#receivers

Processors

Here is where you can add some magic to your traces. Each trace, log or metric can be tweaked in this space. Some examples are adding tags, filtering logs or traces, sample, … More information can be found here.

https://opentelemetry.io/docs/collector/configuration/#processors

Exporters

Like the name mentions, the exporter is responsible for exporter the telemetry data into your observability tool. Just like at the receiver side, the default is the Open Telemetry Protocol (HTTP and gRPC), but also Jaeger and Prometheus are present. And when using the contrib variant of the collector, much more is possible. In the example below we will be using the datadog exporter, cause we would like to import our data into it.

https://opentelemetry.io/docs/collector/configuration/#exporters

Configuration

The configuration of the Collector is done via a yaml file, in which you can define how it works, lets take a look.

First part that we need to configure is the receiver side. In our example, we will use the OTLP with http and gRPC enabled. We will have to instrument our web-application as well that we ‘export’ the OTLP data over there. For more information see Instrumenting your application

receivers:
  otlp:
    protocols:
      http:
      grpc:

The next part to configure is the processor side. Like mentioned above you can tweak your signals here before sending them to your exporter. The example above has 2 processors. One thing to note over here is that not all the processors are already in a stable phase. I personally don’t mind it as, but you should be aware of this. See this link for more information about the stability levels

The first processor is a filter that we defined for our traces. In the example below we have defined some healthchecks on our application, and we do not want them to ‘pollute’ our traces when everything is configured as it should be (HTTP status = 200). By adding this filter, we can remove these traces, but if the health check fails (HTTP status <> 500), we will still see it. This filter processor has lots of possibilities and examples which can be found on the readme.

The next processor which is available is the batch processor. You can already expect what this does, it is batching the data into payloads before it is exported ( in this case to Datadog). Batches can be created based on size or on time.

There is a time out (eg. 10s), after which a batch will be sent regardless of size. On the other hand, you have the batch size. This specifies the amount of signals a bath contains before being sent, send_batch_size.

processors:
  filter: 
    traces: 
      span: 
       - 'attributes["http.target"] == "/health/ready" and attributes["http.status_code"] == 200'
       - 'attributes["http.target"] == "/health/live" and attributes["http.status_code"] == 200'

  # The batch processor batches telemetry data into larger payloads.
  # It is necessary for the Datadog traces exporter to work optimally,
  # and is recommended for any production pipeline.
  batch:
    # Datadog APM Intake limit is 3.2MB. Let's make sure the batches do not
    # go over that.
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: 10s

The next part in the configuration, are the exporters. Here you specify where your data will be sent to. I have specified 2 exporters here, a file exporter and also our datadog exporter. Here you can find a list of all the exporters provided by the community. For each exporter you have to specify a bit more content, eg the file exporter requires a path to be defined, where the datadog exporter requires a api-key and site. Each exporter has a descent readme file on github where the required attributes are listed. Here you can fine the readme for datadog

exporters:
  file/no_rotation:
    path: ./logging
  datadog:
    api:
      ## The Datadog API key to associate your Agent's data with your organization.
      key: "<Your API key goes here>"
      site: datadoghq.eu
      fail_on_invalid_key: true

The last part to define in the collector is the pipeline. This is the place where you are connecting the dots. For each signal being a trace, metric or log, you can define its receiver, processor and exporter. This is where the power of the collector becomes visible. You can start to export your telemetry to different exporters, you can receive logs, only from specials sources that you output to other exporters, really nice.

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]
    traces:
      receivers: [otlp]
      processors: [batch, filter]
      exporters: [datadog]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog, file/no_rotation]

When you spin up the collector you will see something like this (I manually filtered the output a bit here)

C:\Users\rob.vanpamel\work\OTELCOL> .\otelcol-contrib.exe --config .\configuration\config.yaml
2023-02-23T10:24:56.365+0100 info service/telemetry.go:90 Setting up own telemetry...
2023-02-23T10:24:56.369+0100 info service/telemetry.go:116 Serving Prometheus metrics {"address": ":8888", "level": "Basic"}
2023-02-23T10:24:56.712+0100 info provider/provider.go:41 Resolved source {"kind": "exporter", "data_type": "metrics", "name": "datadog", "provider": "system", "source": {"Kind":"host","Identifier":"BEACC-8MT3MN3"}}
2023-02-23T10:24:56.718+0100 info clientutil/api.go:47 Validating API key. {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2023-02-23T10:24:56.967+0100 info clientutil/api.go:51 API key validation successful. {"kind": "exporter", "data_type": "metrics", 
2023-02-23T10:24:57.334+0100 info clientutil/api.go:51 API key validation successful. {"kind": "exporter", "data_type": "logs", "name": "datadog"}
2023-02-23T10:24:57.350+0100 info logs/sender.go:45 Logs sender initialized {"kind": "exporter", "data_type": "logs", "name": "datadog", "endpoint": "https://http-intake.logs.datadoghq.eu"}
2023-02-23T10:24:57.357+0100 info service/service.go:128 Starting otelcol-contrib... {"Version": "0.70.0", "NumCPU": 16}
2023-02-23T10:24:57.357+0100 info extensions/extensions.go:41 Starting extensions...
2023-02-23T10:24:57.358+0100 info service/pipelines.go:86 Starting exporters...
2023-02-23T10:24:57.359+0100 info service/pipelines.go:90 Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2023-02-23T10:24:57.360+0100 info service/pipelines.go:94 Exporter started. {"kind": "exporter", "data_type": "metrics", "name": "datadog"}
2023-02-23T10:24:57.360+0100 info service/pipelines.go:90 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "datadog"}"datadog"}
2023-02-23T10:24:57.362+0100 info service/pipelines.go:98 Starting processors...
2023-02-23T10:24:57.362+0100 info service/pipelines.go:102 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "metrics"}AppInsights", "pipeline": "traces"}
2023-02-23T10:24:57.373+0100 info service/pipelines.go:106 Processor started. {"kind": "processor", "name": "filter/logs", "pipeline": "logs"}
2023-02-23T10:24:57.374+0100 info service/pipelines.go:102 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "logs"}
2023-02-23T10:24:57.374+0100 info service/pipelin. 
...  
2023-02-23T10:24:57.383+0100 info otlpreceiver@v0.70.0/otlp.go:112 Starting HTTP server {"kind": "receiver", "name": "otlp", "pipeline": "logs", "endpoint": "0.0.0.0:4318"}
2023-02-23T10:24:57.385+0100 info service/pipelines.go:118 Receiver started. {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2023-02-23T10:24:57.385+0100 info service/pipelines.go:114 Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "metrics"}
2023-02-23T10:24:57.387+0100 info service/pipelines.go:118 Receiver started. {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2023-02-23T10:24:57.388+0100 info service/service.go:145 Everything is ready. Begin running and processing data.

Instrumenting your dotnet core application

When you want to collect traces, metrics and logs, you have to make some code changes to your codebase before they can be sent to eg Datadog with OpenTelemetry. Yes, there are possibilities to do the same without code changes, but you are not using the open telemetry standard at that moment, but a vendor specific implementation.

Collecting logs

Collecting logs is the easiest part, if you are already using the ILogger interfaces that dotnet core provides you. The Nuget Packages to add to your project are listed below. Remember that they are still in beta/preview but it shouldn’t hold you back.

After you’ve added the packages you need to extend the startup.cs file and extend the service collection.

public void ConfigureServices(IServiceCollection services)
{
    ... // other service registrations etc
    services.AddLogging(loggingBuilder =>
        loggingBuilder.AddOpenTelemetry(otelLoggerOptions =>
        {
            otelLoggerOptions.IncludeFormattedMessage = true;
            otelLoggerOptions.IncludeScopes = true;
            otelLoggerOptions.ParseStateValues = true;
            otelLoggerOptions.AddConsoleExporter();
            otelLoggerOptions.AddOtlpExporter();
        })
    );
}

Afterwards you can log like you were doing already by injecting the ILogger<> or ILoggerFactory.

public class MyLoggingController
{
    public MyLoggingController(ILogger<MyLoggingClass> logger)
    {
        logger.LogInformation("The logging class is created");
    }
}

And the result should look like

LogRecord.Timestamp: 2023-02-23T09:39:17.1029870Z
LogRecord.TraceId: 87f9c6ac8d1cc7aea9c105c866a5095b
LogRecord.SpanId: a0d1d3ed1cbd01d0
LogRecord.TraceFlags: Recorded
LogRecord.CategoryName: MyApp.Controllers.v1.MyLoggingController
LogRecord.LogLevel: Information
LogRecord.FormattedMessage: The logging class is created
LogRecord.StateValues (Key:Value):
    OriginalFormat (a.k.a Body): The logging class is created
LogRecord.ScopeValues (Key:Value):
[Scope.0]:SpanId: a0d1d3ed1cbd01d0
[Scope.0]:TraceId: 87f9c6ac8d1cc7aea9c105c866a5095b
[Scope.0]:ParentId: 0000000000000000
[Scope.1]:ConnectionId: 0HMOLG7A3HTFA
[Scope.2]:RequestId: 0HMOLG7A3HTFA:00000001
[Scope.2]:RequestPath: /api/v1/default
[Scope.3]:ActionId: a94a60be-d09e-425e-85d9-c5111ca61cf5
[Scope.3]:ActionName: MyApp.Controllers.v1.MyLoggingController.GetAsync (MyApp)

Resource associated with LogRecord:
service.name: MyApp
service.instance.id: cbfe9734-9fe4-4711-9ea2-fdfe45f2251e

Collecting Traces

Traces in your web-application are generated by default whenever your web-application receives a request. This means that you don’t have to create them manually. You do have to catch them and send them to the Open Telemetry Collector.

Enabling this for open telemetry can also be done by extending the services collection. You will see in the configuration that we enable 3 kind of instrumentations.

Instrumentation for:

AspNetCore
SqlClient
HTTPClient

The required nuget packages are the ones below:

Instrumentation will be added automatically afterwards. IF you think it is too much, you can start filtering in the collector.

  var resourceBuilder = ResourceBuilder
                            .CreateDefault()
                            .AddService("MyService");

  services
    .AddOpenTelemetry()
    .WithTracing(openTelemetryBuilder =>
    {   
        openTelemetryBuilder
            .AddConsoleExporter()
            .AddOtlpExporter()
            .AddSource("MyService.*")
            .ConfigureResource(resourceBuilder)
            .AddAspNetCoreInstrumentation()
            .AddSqlClientInstrumentation(
                options => options.SetDbStatementForText = true
            )
            .AddHttpClientInstrumentation(
                options => options.RecordException = true);
                })

When you run the application with the above configuration, you notice that logs and traces are exported towards OTLP but also to your console. I think this console is still useful for local development.

The output on your console should look like this

Activity.TraceId: 178afd54db82d620af89f346dd26eeae
Activity.SpanId: 1d3093c32ceb22be
Activity.TraceFlags: Recorded
Activity.ActivitySourceName: OpenTelemetry.Instrumentation.AspNetCore
Activity.DisplayName: api/v{version:apiVersion}/{tenant:length(1,100)}/blogs
Activity.Kind: Server
Activity.StartTime: 2023-02-23T09:33:42.4927438Z
Activity.Duration: 00:00:07.9412154
Activity.Tags:
    net.host.name: localhost
    net.host.port: 44381
    http.method: GET
    http.scheme: https
    http.target: /api/v1/default/blogs
    http.url: https://localhost:44381/api/v1/default/blogs
    http.flavor: 2.0
    http.user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36
    http.route: api/v{version:apiVersion}/{tenant:length(1,100)}/blogs
    http.status_code: 200
    Principal: 0aa00995-cb51-49ed-b2c6-5e5696170db9
Resource associated with Activity:
    service.name: MyApp
    service.instance.id: 43af6b39-2e41-475d-857c-3b3dd3581748

Remarks on the OTLP exporter

In my situation I’ve deployed my collector in the cloud, but I had some trouble connecting to it. I noticed that data was being sent towards the collector, but it didn’t ended up there. After some digging into Wireshark I saw that mu deployed collector returned a 404 Error status code. After digging around for a solution I got it working by adding the correct path and the correct protocol.

Path for the OTLP Exporter

Signal	Path	Example
Metric	v1/metrics	http://mydeployedcollector.com:4318/v1/metrics
Logs	v1/logs	http://mydeployedcollector.com:4318/v1/logs
Traces	v1/traces	http://mydeployedcollector.com:4318/v1/traces

.AddOtlpExporter(oltpexporter =>
{
        oltpexporter.Endpoint = new System.Uri($"{_configuration["OpenTelemetry:EndPoint"]}/v1/traces");
        oltpexporter.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
});

With this configuration you will be able to use opentelemetry and by doing so, getting more and more insights into your application. Which might lead to faster bug resolutions, less issues, … Thank you for reading with me. You have a comment or found an issue? Ping me on twitter or leave them over here

References

Some resources which aren’t shared yet are mentioned over here.

Looking back to NDC London 2023

Rob Van Pamel — Mon, 30 Jan 2023 00:00:00 +0000

Last week me and my colleagues went to the NDC conference in London. This NDC conference is a 3 day conference where you can attend several talks related to different topics like architecture, cloud, testing, .NET and many more.

Each day you can choose between 7 different tracks, whatever triggers you the most. If you like hands-on, there are workshops available during the conference.

My goal was to get more up to date with latest architecture patterns and look at some AWS tracks.

3 of my favourite sessions are listed below: there is much more good content, but I can’t list them all off course.

Intentional Code - Minimalism in a World of Dogmatic Design by David Whitney

During this talk David told us how we could create a better design for our applications. Not design in the sense that you should add design patterns or best practices but more to the fundamentals. Looking at how we as a developer feel when we look at code. Do we want to close the file, or can we understand the flow at a glance? Most of the examples aren’t hard to grasp or complicated, but it is so easy to overlook them. It starts very easy for example by adding some newlines to the file, or I call it, “Give your code some air to breathe”. This can already improve the readabiliity a lot! But you can go a few steps forward, can this code be minified and simplified? Look closer if you really need that additional level of abstraction which was added. Most likely you can remove it and don’t need it. This lowers the cognitive load, makes it easier to changes which leads in the end again to a better architecture. The talk continues with more examples and a good reading reference would be “Code that fits in your head”.

A perfect match: Dapr & Azure Container Apps by Sander Molenkamp

In this talk I got a good overview of the different possibilities that Dapr provides and how this can be used in combination with Azure Container Apps. My knowledge of Dapr was very limited so it was easy to overwhelm me. You can use Dapr for enabling services like PubSub, State Management, Secret Management, …. You can use the Dapr defaults, but in combination with Azure, you can hook it up with Azure Services like Azure Service Bus, Blob storage etc. Which makes it very powerfull in my opinion. A service that AWS is missing I think, although you can add Dapr yourself.

Don’t Build a Distributed Monolith: How to Avoid Doing Microservices Completely Wrong by Jonathan “J.” Tower

Jonathan provided the audience with a top 10 of things to avoid when building microservices. Although I think almost all of them can be found in the book of “Building Microservices” by Sam Newman, I still think this a good reminder! Let me sum some of them up for you. To learn all of them you should go and listen to his talk

Assuming Microservices are always better
Shared database between microservices
Microservices are too small
Starting your microservices from scratch
Coupling through cross cutting concerns
Use of sync communications

Once again, there were a lot of other interesting sessions but it would take way too long to list all these talks. I must admit that the overall quality of the given talks on the conference was very high. And looking back also includes the nice lunches we had in combination with a good party from the line-breakers on Thursday.

That will be it for today, see you next time!

Use vscode web editor in Azure DevOps or Github repositories

Rob Van Pamel — Sat, 26 Nov 2022 00:00:00 +0000

I am already a fan of visual studio code, but recently I discovered some features which made it only better.

This blog on github is created mainly using visual studio code. But actually I don’t have visual studio code installed. I’m editing it in my browser on my iPad from the kitchen! I love the “new” web editor in github. Actually I really love the fact that you can start editing now from anywhere anytime, almost no prerequisites. You can launch vscode in your browser by pressing the dot . on your keyboard, and a new environment will be created, where your complete repository is cloned and ready to change. How great is that?!

The best part, is that you can not only do this with Github but also on Azure Devops! Go to your repository in the browser and press the dot . and the magic happens. For big changes, it might not be the ideal environment, but if you want to create a small change or file a PR, it might just suit your needs.

If you want an overview of the other options, press the question mark ? and you get the overview. It is something small but I found so usefull, I needed it to share with you!

If you have any questions or comments (including typos ;) ) please leave them over here

Thanks for reading and learning along! See you next time!

References about vscode web

https://code.visualstudio.com/docs/editor/vscode-web ( here, it still mentions readonly for Azure Repos, but this is outdated )

Accessing blob storage using User Managed Identities in Azure

Rob Van Pamel — Mon, 21 Nov 2022 00:00:00 +0000

In my previous blog post, the benefits of Managed Identities are handled. As mentioned over there, they increase the security inside your Azure environment. Now we will take this theorie into practice and start working with it. We’ll create an azure function which access a storage account and writes a stream to it, by using the user Managed Identity.

There will be 2 parts in this blogpost, the first one is to setup up the Azure environment, creating the azure function, the managed Identities etc. The second part where the application will be updated to use the User Managed Identity to write the stream to the blobl storage.

The Azure environment

Creating the Azure environment will be done in combination with Bicep. Let’s start with the Azure Function. The Azure function requires at least an App Service Plan, a storage account and off course a function app.

The service plan is create on the Dynamic tier and as reserved which enabled running on a a Linux environment.

resource appServicePlan 'Microsoft.Web/serverfarms@2022-03-01' = {
    location: location
    name: 'asp-blog-managed-identities'
    kind: 'linux'
    sku:{
        name: 'Y1'
        tier: 'Dynamic'
    }
    properties:{
        reserved: true
    }
    tags:{
        blog: 'blog-managed-identities-storage'
    }
}

The function app and its linked storage account is described below. The functionapp is attached to the App Service Plan via the serviceFarmId. The purpose of the AzureWebJobsStorage which is listed in the appSettings, is to store the binaries of the function. It isn’t involved in the futher process of accessing data via a user managed identity.

resource functionApp 'Microsoft.Web/sites@2022-03-01'={
name: 'fct-blobwriter'
location: location
kind: 'functionapp'
identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
        '${usermanagedIdentity.id}': {}
    }
}  
properties:{
    serverFarmId: appServicePlan.id
    siteConfig: {
        appSettings: [
            {
                name: 'AzureWebJobsStorage'
                value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccountName};EndpointSuffix=${environment().suffixes.storage};AccountKey=${storageAccount.listKeys().keys[0].value}'
            }
            {
                name: 'FUNCTIONS_EXTENSION_VERSION'
                value: '~4'
            }
            {
                name: 'APPINSIGHTS_INSTRUMENTATIONKEY'
                value: applicationInsights.properties.InstrumentationKey
            }
            {
                name: 'FUNCTIONS_WORKER_RUNTIME'
                value: 'dotnet-isolated'
            }
            {
                name: 'AZURE_CLIENT_ID'
                value: usermanagedIdentity.properties.clientId 
            }    
        ]
        minTlsVersion: '1.2'
    }
    httpsOnly: true
}
}

resource storageAccount 'Microsoft.Storage/storageAccounts@2021-06-01' = {
    name: storageAccountName
    location: location
    sku: {
        name: storageAccountType
    }
    kind: 'StorageV2'
    properties: {
    }
    tags:{
        blog: 'blog-managed-identities-storage'
    }
}

In the identity section, an array of user managed identities are added. In this example only one is added.

identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
        '${usermanagedIdentity.id}': {}
    }
}

Next to this, which user managed identity to use, is specified in the appsettings of the function.

{
    name: 'AZURE_CLIENT_ID'
    value: usermanagedIdentity.properties.clientId 
}

Once the function is created, the next step is to create a User Managed Identity, a Role and a RoleAssignment.

The Role is created below, for simplicity an existing is being used here. You can find a full list with their corresponding guid over here

@description('This is the built-in Storage Blob Data Contributor role. See https://docs.microsoft.com/azure/role-based-access-control/built-in-roles#contributor')
resource contributorRoleDefinition 'Microsoft.Authorization/roleDefinitions@2018-01-01-preview' existing = {
  scope: subscription()
  name: 'ba92f5b4-2d11-453d-a403-e96b0029c9fe'
}

Once these are created, it is time to add the RoleAssignment. In this Role Assignment the User Managed Identity and the Role definition are coupled together.

resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
    name: guid(resourceGroup().id, usermanagedIdentity.id, contributorRoleDefinition.id)
    properties:{ 
        principalId: usermanagedIdentity.properties.principalId 
        roleDefinitionId: contributorRoleDefinition.id
    }
}

Now it is time to move over to the application to start using this.

The Application

Before starting using the User Managed Identity, let see how this should typical is done without them.

Without User Managed Identity

When no Managed Identity is used, a SAS token is created on the Azure portal to gain access to the blobcontainer. An important aspect to get the SAS token from the Azure Portal, is to store it in a safe manner, eg Azure Keyvault.

public static async Task WriteWithSas()
{
    var connectionString =
            $"BlobEndpoint=https://{BLOB_ACCOUNT}.blob.core.windows.net/;{BLOB_SASTOKEN}";
    var client = new BlobContainerClient(connectionString, BLOB_CONTAINER);
    ... 
    ... // stream stuff goes here
    ...
    return await blobContainerClient.UploadBlobAsync($"file-{DateTime.UtcNow:yyyy-MM-dd-HH-mm-ss}.txt", stream);
}

Using User Managed Identity

Now when using the User Managed Identity, we don’t have to securely fetch any identities or so, we can just safely use it, which is the whole idea to make it much safer. Now you’ll notice that there is no SAS token, or another secret involved when creating the connection string. The difference between those, is to use the DefaultAzureCredential. This defaultAzureCrendential will use the user managed identity which is specified in the appsettings of the function AZURE_CLIENT_ID.

string containerEndpoint = $"https://{BLOB_ACCOUNT}.blob.core.windows.net/{BLOB_CONTAINER}";

var client = new BlobContainerClient(new Uri(containerEndpoint), new DefaultAzureCredential());
... 
... // stream stuff goes here
...
return await blobContainerClient.UploadBlobAsync($"file-{DateTime.UtcNow:yyyy-MM-dd-HH-mm-ss}.txt", stream);

Using this the Azure function is allowed to access and write a stream to the blob storage!

The complete example can be found over here

If you have any questions or comments (including typos ;) ) please leave them over here

Thanks for reading and learning along! See you next time!

Improve security with Managed Identities in Azure

Rob Van Pamel — Mon, 31 Oct 2022 00:00:00 +0000

When you want to access a resource in Azure like a storage account or a SQL database, there are multiple options available. SharedAccessKeys and connection strings are the most popular one we have nowadays. Who hasn’t used a connectionstring to connect to the sql-database? However, these solutions are based on what you could call “shared credentials” which are not always the most secure way. You have to store the credentials in safe manner, so it doesn’t get compromised.

This is where Azure Key Vault is the obvious solution. The credentials which must remain secret can be stored in a secure way in a key vault. However, it is already stored safely, there is still a possibility that it leaks or gets compromised by human errors. On top of that, you still need to rotate the secrets, which puts load on the responsible team.

In most cases, the resource which we want to access runs in the same Azure environment as the resource  that would like to access it. So it should be possible that we provide access without the hassle of shared credentials etc. The answer to this challenge is using Managed Identities.

Azure Managed Identities

Managed identities make it possible to connect multiple resources without the management of secrets or credentials. You don’t even have access to the secret or credential which is used. Azure mananaged identities can be used for each service or resource that supports Azure AD authentication. And best part for last, you can use them for free!

User-assigned Managed Identity

A user-assigned managed identity is an identity which can be shared accross multiple resources. It is not attached to a given resource, which means that multiple resources, can use that same Managed Identity to access a given resource.

System-assigned Managed Identity

A system-assigned managed identity is an identity which is directly attached to a given resource. It is created together with the resource and if the resource is removed, the managed identity is removed as well.

Managed identities, system or user-assigned, will only define who has the permission to do an action. What and where it can be done, is not defined yet. This is defined inside a Role Assignment.

Role assignments

A role assignments in azure is a definition of 3 principles

What can be done - is defined in the role and its permissions
Who can do it - is defined in the principal
Where can it be done - is defined in the context.

Role

Like mentioned above, the role will define what can be done in a list of permissions, eg the storage blob data reader can access the blob storage for read, but he is not able to write to it. Multiple built-in roles are available, but custom roles can be created as well.

Principal

The principal can be a managed identity, which we discuss currently, but it can also be a user or a group.

Scope

The scope is where the user can use the role, For example in the given resource group, or in the subscription.

Role assignments are maintained individually as well as for user as for system-assigned identities. This implies that in case that a system or a user assigned identity is removed, the role assignment still needs to be updated.

Deployments and Managed Identities

Although both, user or system assigned, managed identities can serve the same needs, user-assigned Managed identities have my preference. This mainly from a pracitcal perspecitice when deploying resources together with the managed identities for that resource. When using system-assigned managed identities, the number of role assignments which are required can increase very rapidly. Be aware that there is a limit when creating these assignments.

system assigned identities - 8 role assignements

In the example above (image from microsoft) already 8 role assignments need to be created. When we are working with User Based identities only 2 are required, see below. This already makes the life a bit easier.

user assigned identities - 2 role assignements

Combining multiple user-assigned Managed Identities is a flexibility that is only possible when working with user-assigned managed identities. We could combine a ‘shared user-assigned identity’ with a more specific user-assigned identity. Although the latter is also possible when combining system and user-managed identities.

It is important to note that combining several user-assigned identities should still adhere to the principle of the least privilege.

Use of user-assigned or system-assigned managed identities?

The preference here goes to the usage of user-assigned above system-assigned managed identities. They are more flexible to be used across multiple resources. This will improve the developer experience, while it isn’t less secure compared to system-assigned managed identities.

User-assigned managed identities have their own lifecycle, what improves creation of resources and resource groups. While the lifecycle of system-assigned managed identities is attached to the resource. This could require multiple runs of a pipeline before a role-assignment can be done, because you can’t create a role assignment for a resource which isn’t created yet. While with a user-assigned managed identity, the role assignments can be created before the resource are created, which makes it easier to use.

That’s it for today, thanks for reading along! In the next blog post I ‘ll explain more in detail how you can create user-assigned identities and how to use them to access a storage account from an appservice. See you there!

References

The next steps in Github Pages

Rob Van Pamel — Sat, 15 Oct 2022 00:00:00 +0000

After the first steps follows … the next steps. I created the first blogpost and made it public, I already realised that it wasn’t ready yet to be launched. It was way too early, but at least it was something.

So while waiting  in the airport on my next flight I continued to work on it. What needed to be done ? A lot of things!

The title of the site was the one from the repository
I didn’t had the social media links
I couldn’t get an excerpt of my blogpost on the main page
…

Adding social media links

Adding these social media links didn’t went as fast as I expected it. It seemed that I was using an ‘old’ version of the minima theme. Due to this, the settings which i applied in the _config.yml didn’t apply.

You can always refer to the remote theme from github by using a plugin. After changing it to use that remote theme, I got a step further and after tweaking a bit more the media links worked. See the _config.yml below for more information

minima:
  social_links:
    twitter: youraccount
    github: youraccount
    linkedin: youraccount-878796454

remote_theme: jekyll/minima
plugins:
  - jekyll-remote-theme

Adding titles

By default the title and description of your page are the name and description of the repository. Which is mostly not what you want. So lets change this as well I saw multiple options online, but not all of them worked. This one worked for me in the _config.yml

title: Rob Van Pamel
description: Rob Van Pamel

Adding Excerpts

Another default setting of the minima theme is that there is no excerpt shown on your post page, but only the title. I like it that you can already see a bit of the post, so I enabled that as well in the

_config.yml

show_excerpts: true

Afterwards add the excerpt seperator in your blogpost header and off course also in the blogpost. Here is an example

title: The next steps in Github Pages
date: 2022-10-15   
excerpt_separator: <!--more-->
---
After the first steps follows ... the next steps. I created the first blogpost and made it public, I already realised that it wasn't ready yet to be launched. It was way too early, but at least it was something. 

So while waiting <!--more-->

That will be it for now, it the next one, i would like to change some more smaller things, like a favicon, look at SEO, add tags to the blogposts, and so on.

The first steps in Github Pages

Rob Van Pamel — Fri, 14 Oct 2022 00:00:00 +0000

“Are you serious, Rob, you don’t know how to create Github pages?”

Yes, I never looked into to be honest. I know you can create them with regular HTML, CSS and some JS, but I want to work on something new, so I’ll start with Jekyll and markdown.

I want something that is easy to maintain, so I am able to proceed with it with not too much effort and overhead. Jekyll and markdown seems to be a good fit.  But as I’ve never used github pages before so, it’s all new for me. As the purpose of the github pages is to serve as a blog, it’s a very good starting point. Allright lets dive into it.

Specify a theme

The first thing you need to do is add a _config.yml, which looks like this in my case. Which just specifies the theme and the installed plugins. As a menu is required, the Jekyll menus are added over here.

theme: minima
plugins:
- jekyll-menus

This gives already a clean look to your page, although it isn’t fancy yet. But that is something that wil be fixed later, content over presentation, right? For those who know me, I’m not that frontend guy which works on that slick webpage. I’m the one sitting quietly in the back trying to figure out why we experience a too high latency 😏 .

Okay, next step is adding the blogs itself, which is actually pretty easy.

Adding blog posts

A blog post can be added very easily by adding a new markdown page in a folder called _posts. The only convention you need to be aware of is the name of your file. This requires the format of <YYYY-MM-DD-TitleOfYourBlogPost>. In this example, it is 2022-10-15-CreatingGitHubPages, so that aint’t too hard, is it?

At the top of your blogpost, you can add more metadata, for example title and date, which can be used by some plugins. See below for the example.

---
title: "The first steps in Github Pages"
date: 2022-10-14
---

About page

Every blog needs an about page, so this one as well. This can be done, by adding a new markdown page where you specify all the contents. If you want to be page to be accessable, you need to tell Jekyll in which menu you want it to appear. That can be done, by adding some metadata at the top of your page again. See the example below.

---
title: About
menus: header
---

I am a .NET consultant with a focus on architecture and cloud. I ....

That’s it for now, and I’ll explore some more about it, and will tell you more in an upcoming post! To be continued…