DEV Community: Wallism

polling after CreateOrUpdate: polling failed

Wallism — Mon, 22 Jul 2024 22:48:07 +0000

When updating an Azure APIM API using Pulumi (pulumi up) we started seeing this obscure error.

First thing to do is get more info via debug:
pulumi up --debug
(if you're running this in a shell, make sure you've increased the buffer for the window (in options) because there's alot of extra logs!)

Searching through that output we can see the PUT that tries to update the Open API (swagger), followed by a few GET polls to the same API. The value of note is the provisioningState, the first couple of calls were:

"provisioningState": "InProgress"

followed by a final:

"provisioningState": "Failed"

But no real info on what the issue actually is. To get that info we need to go into the APIM and manually try to update the Open API.

In the Azure portal, navigate to APIs in the APIM, each API name has an ellipse next to it, click the ellipse for the API that is having the issue, then select "Import", then OpenAPI and select the json file that has your OpenAPI from your machine:

Click Import, then after a short amount of time you should see an error, ours came up like this:

From there should have enough info to resolve whatever the error is!

Github action to deploy to Azure container app fails with LinkedAuthorizationFailed

Wallism — Wed, 07 Feb 2024 20:59:30 +0000

I created my github action from the Azure portal, in the container app (capp) "Continuous deployment" (CD) section.
The first capp I did this with worked fine, but the second one generates the error.
One important detail, I put the second capp into the same container environment as the first (a very normal thing to do).

This error should not occur any more as Contributor access is given on the resource group now, not on the container app and environment.

This is the error:

The client 'guid' with object id 'guid' has permission to perform action 'Microsoft.App/containerApps/write' on scope '/subscriptions/guid/resourceGroups/MyRG/providers/Microsoft.App/containerApps/capp-02'; however, it does not have permission to perform action(s) 'Microsoft.App/managedEnvironments/join/action' on the linked scope(s) '/subscriptions/guid/resourceGroups/MyRG/providers/Microsoft.App/managedEnvironments/capp-environment' (respectively) or the linked scope(s) are invalid.

It's actually a really clear error. The "client" (or in this case, Service Principal) does not have permission to do stuff on the capp environment.

Why?

When we setup CD and the Github action was created, a part of that process is the creation of a service principal (aka client).

The client is what allows the action to securely update our resource(s) in Azure, you can find the client in Microsoft Entra ID -> App registrations. You can also see the permissions granted to the client on the capp and capp environment, under "Access Control (IAM)".

When we create the first CD, the client is granted permission (Contributor) on both the capp and the capp environment. But when we create the second CD on the second capp, a second client is created but this second client is only granted access to the capp, not the capp environment! I don't know why, I guess it's a bug.

The solution

To fix the error, just go to the capp enviroment -> Access Control (IAM), then "Add role assignment" and grant the client Contributor access.

Azure Az.Storage command returns "A task was canceled."

Wallism — Wed, 10 Jan 2024 23:52:30 +0000

Using az.storage version 6.1.0 with this command:

$context = New-AzStorageContext -StorageAccountName accountname

(run after successfully connecting - Connect-AzAccount - and selecting the correct subscription - Set-AzAccount)

I kept getting this error:

New-AzStorageContext: A task was canceled.

Not the most helpful error message! Turns out the fix is very simple, we now need to provide the key as well (the previous version I was running did not require the key)

$context = New-AzStorageContext -StorageAccountName accountname -StorageAccountKey accountkey

Easy...made difficult by the useless error message.

Connection creation correlation with slow response times - in Azure WebApp

Wallism — Mon, 25 Sep 2023 21:27:15 +0000

We were experiencing a strange performance issue with one of our API's running in an Azure Web App. None of the usual metrics were spiking in correlation with the response time spikes, CPU, memory, even request count didn't really correlate.

Then I noticed a distinct correlation with, Connections.

Red line = Response Time, blue line = Connections

I had never seen this link before, so first I had to figure out exactly what "Connections" meant. Essentially it means Connections out, in our case a connection to an SQL Server.

This leads to the question, how are we creating the connections...and are we doing something inefficient? Looking in the code, this is how we get a new connection:

new System.Data.SqlClient.SqlConnection("valid-sql-connection-string")

Nothing tricky there, at first I thought maybe we were creating a new connection every time and we could help the issue by re-using connections. That connection re-use actually happens automagically under the hood in the SqlClient library via pooling.

One key point from the docs for me was:

The connection pooler removes a connection from the pool after it has been idle for approximately 4-8 minutes

Our spikes were happening every 15 min (it's an api, there must be an automated process running every 15 min), so (as you can see in the metrics graph) after a short period of time all of those connections that were created, get cleaned up and the pool drops to the minimum.

Which leads to the next question, "Can I increase that minimum connection count?"

Turns out that's really easy by adding Min Pool Size to the connection string.

So set Min Pool Size=70 in the connection string. Result, it still needs to create more connections sometimes but those spikes were significantly flattened. In fact, looking in App Insights, our average response times for one endpoint went from >10s (yes, seconds) to <800ms!

Creating connections is expensive (I found this SO useful), but having never seen this correlation before, why did it impact this service so significantly and not others? I believe it is due to the fact that the DB is in a different region to the service (most of our other services are in the same region). That slight increase in latency (guessing ~100ms) blows out the connection creation time on a service as busy as this one.

Azure APIM - GET total call count for subscription

Wallism — Thu, 21 Sep 2023 01:13:09 +0000

How do you get the total call count for a given subscriptionId from Azure APIM?
The easy but not overly useful way is in the Azure Portal, go to your APIM, then select the Analytics blade (under Monitoring), select the Subscriptions tab, pick your time range and there they are.

Programatically

How to get this data programatically is far more useful and is also relatively easy, the only tricky part is getting the token.
First, check out the docs.

Your call will look like this:
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ApiManagement/service/{serviceName}/reports/bySubscription?$filter={$filter}&api-version=2023-03-01-preview

You can run that in Postman or similar tool (e.g. open source Hoppscotch), however you will need an access token.

How to get an access token

The easiest way is, hit that "Try It" button on the docs page, after you are logged in, you should see a Bearer token in the Request Preview section. You can use that. I dropped mine into jwt debugger to check the expiry, it lasts one hour.

OR - generate your own...

I followed these instructions :)

Essentially (super brief summary), call this POST https://login.microsoftonline.com/{{AAD_name}}/oauth2/token with client_id and client_secret specified in the body (can't recall why but in my Postman my body is "x-www-form-encoded").

Azure App Configuration task - An unexpected error occurred

Wallism — Wed, 03 May 2023 23:37:58 +0000

In Azure DevOps, using the task, "Azure App Configuration Push", I hit the rather obscure error, "An unexpected error occurred".

First step to dig into this is to edit the release, adding the following debug variable -> system.debug set the value to true. Then re-run and you do get more detail about what actually went wrong. Looking through the log I can now see:

[debug]{"name":"RestError","statusCode":403

There is much more detail but that is all that was enough for me, following these instructions I knew I hadn't given my service principal the "App Configuration Data Owner" role in my Azure App Configuration instance. Once the role was added, the task ran successfully.

pulumi stack - error: getting selected stack: failed to decrypt

Wallism — Thu, 06 Apr 2023 03:48:38 +0000

If you do pulumi stack and get an error like this:

error: getting selected stack: failed to decrypt: incorrect passphrase, please set PULUMI_CONFIG_PASSPHRASE to the correct passphrase or set PULUMI_CONFIG_PASSPHRASE_FILE to a file containing the passphrase

Check what value is being used for the passphrase, in powershell, $env:PULUMI_CONFIG_PASSPHRASE.

Ultimately you simply need to ensure the passphrase is correct. The issue I had was it looked correct but the project I was working on at the time had a different passphrase to our other projects. Turns out the passphrase for this project was "", an empty string.

Best practice suggestion - keep the passphrase consistent across all dev projects and stacks, then different for all prod stacks.

To change the password for a stack you can use this: pulumi stack change-secrets-provider passphrase. Apply this to ALL stacks. Also do it for the PROD stack but you want to use a different passphrase for that stack.

Why make the passphrase consistent? When you start working with multiple Pulumi projects, when you change projects, you have to remember to log into the correct Pulumi backend, plus ensure you are logged into the correct Azure tenant and have the correct subscription selected. Having different passphrases is one more thing to trip up you and your developers and waste time. Some projects require more security, obviously in those cases you want to prefer being more secure vs being more convenient.

Failed to negotiate protocol, waiting for response timed out

Wallism — Thu, 06 Apr 2023 00:44:13 +0000

After upgrading to .Net 6 running dotnet test myproj.csproj failed with this error:

Failed to negotiate protocol, waiting for response timed out after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

I found the solution here.

And in case that link disappears, basically you need to add to following to the test project:
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.1" />

Pulumi with Serilog

Wallism — Fri, 24 Mar 2023 05:27:27 +0000

I use Serilog everywhere and I want to do the same in Pulumi projects. It provides all the flexibility I want/need for logging, so I try and bake it into any new project (unless it's going to be a shared library, in which case I want minimal dependencies.)

When doing this, we want to ensure we don't lose any of the console logging that Pulumi does.

First thing, add the Serilog nuget package. Then what I do is create a method somewhere to configure it. (this can usually go into the config file however with the way Pulumi runs, that doesn't work out so well, didn't for me anyway).

Config method:

    public static void ConfigureLogger()
    {

        Log.Logger = new LoggerConfiguration()
            .Enrich.FromLogContext()
            .Enrich.WithExceptionDetails(new DestructuringOptionsBuilder()
                .WithDefaultDestructurers()
                .WithDestructuringDepth(8)) // essential to limit the depth
            .WriteTo.PulumiLogSink()
            .WriteTo.File(path: $"{Environment.CurrentDirectory}\\log\\log.txt")
            .CreateLogger();

        Log.Information("Logging configured");

    }

Either remove "WithExceptionDetails" or add that package. Same with WriteTo.File, I recommend adding, then you have a history.

Then what we need to do, is implement PulumiLogSink...so we don't lose any of the normal Pulumi output.

Info on building a custom sink is here. Following those instructions our Pulumi sink looks like this:

    public class PulumiLogSink : ILogEventSink
    {
        private readonly IFormatProvider _formatProvider;

        public PulumiLogSink(IFormatProvider formatProvider)
        {
            _formatProvider = formatProvider;
        }

        public void Emit(LogEvent logEvent)
        {
            var message = logEvent.RenderMessage(_formatProvider);
            if (logEvent.Level == LogEventLevel.Debug || logEvent.Level == LogEventLevel.Verbose)
                Pulumi.Log.Debug(message);
            if (logEvent.Level == LogEventLevel.Information)
                Pulumi.Log.Info(message);
            if (logEvent.Level == LogEventLevel.Warning)
                Pulumi.Log.Warn(message);
            if (logEvent.Level == LogEventLevel.Error || logEvent.Level == LogEventLevel.Fatal)
                Pulumi.Log.Error(message);
        }
    }

    public static class MySinkExtensions
    {
        public static LoggerConfiguration PulumiLogSink(
            this LoggerSinkConfiguration loggerConfiguration,
            IFormatProvider formatProvider = null)
        {
            return loggerConfiguration.Sink(new PulumiLogSink(formatProvider));
        }
    }
}

Finally, the first line of our Pulumi should call ConfigureLogger() and voila, serilog logging hooked up so you can add whatever sinks you like.

Tip, you probably want to add this using: using Log = Serilog.Log; to ensure 'Log.' uses serilog and not the Pulumi logger.

generate swagger file from dll

Wallism — Fri, 26 Aug 2022 05:28:13 +0000

To generate a swagger file from a dll you need to run a command like this:
dotnet swagger tofile --output "valid-file-path" "valid-dll-path" v1

To make that work there's a few steps to follow and few issues you might hit.

First, you need Swashbuckle.AspNetCore. Instructions to install are in the link, install the latest version globally, you may need a different, lower version depending on your project (I did).

To install a specific version for a project, go to the root of that project in a ps or cmd window. Do the following:

First, make sure you're running the correct version of dotnet sdk, see this post and follow through to the SO article for how to change the sdk dotnet cli uses.
Then install the correct version of Swashbuckle:
(ensure you are in your project root folder)
dotnet new tool-manifest
dotnet tool install --version 6.2.3 Swashbuckle.AspNetCore.Cli

You should see in the output of the the install command where the dotnet-tools.json files is, that was updated by the command. If you run dotnet swagger in any subfolder, it will use this version, not the globally installed version.

Note on version numbers

when you run
dotnet tool install --version 6.5.0 Swashbuckle.AspNetCore.Cli
ensure the version is the same as the nuget package version that is installed in your api project.
So be careful when choosing your version or you will see an error like this:

Could not load file or assembly 'Swashbuckle.AspNetCore.Swagger, Version=6.4.0.0, Culture=neutral, PublicKeyToken=62657d7474907593'. The located assembly's manifest definition does not match the assembly reference.

dotnet swagger tofile : Could not load file or assembly System.Runtime

Wallism — Fri, 26 Aug 2022 05:00:05 +0000

When running this command:
dotnet swagger tofile --output "valid-path" "valid-dll" v1

I was getting this error:

Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The located assembly's manifest definition does not match the assembly reference. (0x80131040)
File name: 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

The target dll was .net 5.0 but because I have .net 6.0 installed, my system was using that sdk by default. To use the correct sdk when running dotnet cli, this SO answer has a clear explanation. Basically you add a new global.json file at a folder level that makes sense (e.g. project or solution root folder).

Logging Headers with Serilog into Sumo

Wallism — Mon, 25 Jul 2022 10:48:42 +0000

A quick post showing subtle changes to how request headers were being logged and the significant improvement on the end result.

Version 1

Log.Logger.Information("Request {headers}");

manifests like this in Sumo:

Ugly and larger than required.

Version 2

Log.Logger.Information("Request {@headers}");

Simply using the destructuring operator produces this:

Overall probably better but there's alot of expanding required to see all of the header values. Particularly annoying is the need to expand Value.

Version 3

Create a simple class
internal class Header { public string Key { get; set; } public string Value { get; set; } }

Extract the keyvalue pairs into a list of Headers:
var headerList = httpRequest.Headers.Select(kvp => new Header {Key = kvp.Key , Value = kvp.Value.FirstOrDefault()}).ToList();

Then log that list:
Log.Logger.Information("Request {@headersList}");

produces:

Marginally better, at least we don't need to expand Value.

Version 4

Still use the Header class and headerList from Version 3, but change the way we log it to:
Log.Logger.ForContext("Headers", headerList).Information("request")

The key difference being the use of ForContext, produces:

Which to my eye is perfect, all the headers, tightly grouped, one click to see them all.