DEV Community: Andrei Kniazev

Azure DevOps CD Self-Reverting AKS Failed Deployment

Andrei Kniazev — Tue, 10 Sep 2024 15:20:06 +0000

Recently, I was working on a migration from Azure Function Apps to Azure Kubernetes Service. While building a CI/CD pipeline, I decided to try applying a self-healing approach. The idea is that what if we can detect failures earlier and automatically recover from them?

(If you just need a full template, scroll to the end.)

How can problems be detected and reverted in Kubernetes deployment?

We use Deployment to manage workloads in Kubernetes. And the power of deployment that we can check status and rollback.

After applying, we can check the status of the deployment to detect problems.

If we just run this command:

kubectl rollout status deployment/some-deployment

It will exit with 0 if deployment is ready. But if deployment is not ready and fails due to reasons, it will be stuck in this command and wait for deployment to complete. To escape this behaviour and exit with 1 if deployment is not ready on time we need to add a timeout argument:

kubectl rollout status deployment/some-deployment --timeout=1m

This argument will timeout our deployment if it does not succeed within the timeframe.

Reverting changes.

To revert changes in Kubernetes we can use this command:

kubectl rollout undo deployment/some-deployment

Before

After

Rollback Concept & Prerequisites.

The rollback mechanism in Kubernetes is designed to revert a deployment to a previous state if the current deployment is unhealthy or fails to meet the desired criteria.

In this case you should be aware that any changes that are not reversable can lead to a situation where rollback will not help.

Good example of those changes a schema change in database. For example during deployment you deploy schema changes and new code that supports it. But during deployment something went wrong and the new code has error and the app cannot start.

In this case if you rollback to a previous version the app will not work as intended because of the schema changes.

To successfully work with automatic rollbacks your team should always apply concept of Rolling Migrations.

Example Scenario.

Consider an application that requires a database schema update. The rolling migration process would involve the following steps:

Deploy Code Supporting New Schema (Disabled): First, deploy the new application code that supports the new schema, but keep the new features disabled.
Deploy Infrastructure Changes: Apply the database schema changes.
Health Check: Introduce health checks to validate that the application can connect to the updated database schema.
Enable New Features: Once the health checks pass, enable the new features in the application.
Monitor Deployment: Continuously monitor the deployment to ensure everything is functioning correctly.
Rollback if Necessary: If any issues are detected, rollback to the previous version of the application.

In this example the code is compatable with previous schema and even in the case of rollback the app will work as intended.

Better Deployment Status.

The deployment can get stuck and never completed because of the following:

Insufficient quota
Readiness probe failures
Image pull errors
Insufficient permissions
Limit ranges
Application runtime misconfiguration

Most of those problems are due to misconfigured deployments or clusters, but there is one that can be customised and can make our check even more robust. But Readiness probe failures allow us to extend those reasons to a more sophisticated one.

If we specify the URL for Readiness probe failures as our health check and extend it to validate more scenarios, like whether our application is able to connect external services like databases with the needed permissions, it will allow us to improve our Deployment Status Check.

Template Itself.

Azure DevOps yml pipeline template to validate AKS deployment and roll back if needed.

parameters:
- name: environment
  type: string
  displayName: Required. Represents the stage of deployment. Usualy it is Development, Test, Acceptance, Production. It is needed so you can have multiple stages in the same pipeline.
- name: azureServiceConnection
  displayName: Required. Azure service connection that will provide access to ARM.
- name: aksName
  displayName: Required. Name of the AKS.
- name: aksRg
  displayName: Required. Name of the resource group where AKS is.
- name: aksSubscription
  displayName: Required. Azure subscription ID where AKS is.
- name: namespace
  displayName: Required. Namespace of the k8s deployment to check.
- name: deploymentName
  displayName: Required. Name of the k8s deployment to check.
- name: timeout
  displayName: Optional. Timeout for the deployment check. The default is 1m. Example 1m, 2m, 10m
  default: 1m
- name: dependsOn
  type: object
  displayName: Optional. Pass list of previous stages to depend on them.
  default: []

stages:
  - stage: AksDeploymentHealthAndRollback${{ parameters.environment}}
    displayName: ${{ parameters.environment}} — K8S Deployment Health and Rollback
    dependsOn: ${{ parameters.dependsOn }}
    jobs:
      - job: AksDeploymentHealthCheck
        displayName: ${{ parameters.environment}} — K8S Deployment Health
        steps:
        - checkout: self
          displayName: Checkout
          fetchTags: false
        - task: AzureCLI@2
          displayName: K8S Deployment Health
          inputs:
            azureSubscription: ${{ parameters.azureServiceConnection}}
            scriptType: bash
            scriptLocation: inlineScript
            inlineScript: |
              set -e # fail when script is failing
              sudo az aks install-cli

              az aks get-credentials --name ${{ parameters.aksName }} \
                --resource-group ${{ parameters.aksRg }} \
                --subscription ${{ parameters.aksSubscription }} \
                --overwrite-existing \
                --file .kubeconfig-${{ parameters.aksName }}

              export KUBECONFIG=$(pwd)/.kubeconfig-${{ parameters.aksName }}

              # Set default namespace
              kubectl config set-context ${{ parameters.aksName }} --namespace=${{ parameters.namespace }}
              kubectl config get-contexts

              # Pass kubeconfig to kubelogin to access k8s API
              kubelogin convert-kubeconfig -l azurecli

              # Check the rollout status of the deployment
              kubectl rollout status deployment/${{ parameters.deploymentName }} --timeout=${{ parameters.timeout}}

      - job: ManualApprovalOfRollBack
        displayName: ${{ parameters.environment}} — Manual Approval Of Rollback
        dependsOn: AksDeploymentHealthCheck
        condition: failed()
        pool: server
        steps:
        - task: ManualValidation@0
          displayName: Approve Rollback
          timeoutInMinutes: 1440 # task times out in 1 day

      - job: Rollback
        displayName: ${{ parameters.environment}} — K8S Deployment Rollback
        dependsOn: ManualApprovalOfRollBack
        condition: eq(dependencies.ManualApprovalOfRollBack.result, 'Succeeded')
        steps:
        - checkout: self
          displayName: Checkout
          fetchTags: false
        - task: AzureCLI@2
          displayName: K8S Rollback
          inputs:
            azureSubscription: ${{ parameters.azureServiceConnection}}
            scriptType: bash
            scriptLocation: inlineScript
            inlineScript: |
              set -e # fail when script is failing
              sudo az aks install-cli

              az aks get-credentials --name ${{ parameters.aksName }} \
                --resource-group ${{ parameters.aksRg }} \
                --subscription ${{ parameters.aksSubscription }} \
                --overwrite-existing \
                --file .kubeconfig-${{ parameters.aksName }}

              export KUBECONFIG=$(pwd)/.kubeconfig-${{ parameters.aksName }}

              # Set default namespace
              kubectl config set-context ${{ parameters.aksName }} --namespace=${{ parameters.namespace }}
              kubectl config get-contexts

              # Pass kubeconfig to kubelogin to access k8s API
              kubelogin convert-kubeconfig -l azurecli

              # Rollback the deployment
              kubectl rollout undo deployment/${{ parameters.deploymentName }}

      - job: AksRollbackHealthCheck
        displayName: ${{ parameters.environment}} — K8S Rollback Health
        dependsOn: Rollback
        condition: eq(dependencies.Rollback.result, 'Succeeded')
        steps:
        - checkout: self
          displayName: Checkout
          fetchTags: false
        - task: AzureCLI@2
          displayName: K8S Rollback Health
          inputs:
            azureSubscription: ${{ parameters.azureServiceConnection}}
            scriptType: bash
            scriptLocation: inlineScript
            inlineScript: |
              set -e # fail when script is failing
              sudo az aks install-cli

              az aks get-credentials --name ${{ parameters.aksName }} \
                --resource-group ${{ parameters.aksRg }} \
                --subscription ${{ parameters.aksSubscription }} \
                --overwrite-existing \
                --file .kubeconfig-${{ parameters.aksName }}

              export KUBECONFIG=$(pwd)/.kubeconfig-${{ parameters.aksName }}

              # Set default namespace
              kubectl config set-context ${{ parameters.aksName }} --namespace=${{ parameters.namespace }}
              kubectl config get-contexts

              # Pass kubeconfig to kubelogin to access k8s API
              kubelogin convert-kubeconfig -l azurecli

              # Check the rollout status of the deployment
              kubectl rollout status deployment/${{ parameters.deploymentName }} --timeout=${{ parameters.timeout}}

      - job: Clean
        displayName: Clean Up
        dependsOn: [ AksDeploymentHealthCheck, ManualApprovalOfRollBack, Rollback ]
        condition: always() 
        steps:
        - checkout: none
        - script: |
            rm -rf ~/.kube/config
          displayName: Remove kube config

Template behaviour:

🔍 Azure CLI. How to Simplify Output.

Andrei Kniazev — Fri, 19 May 2023 09:38:43 +0000

If you need to extract specific information from the output of an Azure CLI command you can use the --query that will do it for you.

Let's say you want to save a deployement key of the Azure Static Web App in a Bash variable for later use in your CI/CD pipeline. When you run the command:

az staticwebappsecrets list -g rg-name -n static-web-app-name --query

The output will be in JSON format:

{
  "id": "some id",
  "kind": null,
  "location": "West Europe",
  "name": "secrets",
  "properties": {
    "apiKey": "cool key"
  },
  "resourceGroup": "rg-name",
  "type": "Microsoft.Web/staticSites/secrets"
}

But you only need the apiKey value. That's where the --query parameter comes in play. Simply add it to your command:

az staticwebappsecrets list -g rg-name -n static-web-app-name --query 'properties.apiKey'

Now, you can save only the key in a Bash variable:

DEPLOYMENT_TOKEN=$(az staticwebappsecrets list -g rg-name -n static-web-app-name --query 'properties.apiKey')

To remove the surrounding quotes, use the following step:

DEPLOYMENT_TOKEN=${DEPLOYMENT_TOKEN//\"/}

And here you go! You have extracted the key and it is ready to use in your CI/CD pipeline.

Full documentation is here: https://learn.microsoft.com/en-us/cli/azure/query-azure-cli

Azure Web Apps with One Static IP Address via Bicep

Andrei Kniazev — Fri, 18 Nov 2022 13:02:52 +0000

If you need to have multiple Azure Web Apps with one static IP for outbound calls you need:

1 - Create a Virtual Network
2 - Create NAT Gateway
3 - Create Public IP
5 - Create SubNets for each App
6 - Attach SubNets to VN
7 - Use NAT In SubNets
8 - Enable vnetRouteAllEnabled for each App

We will use bicep for all of it.

Apps

Notice: Apps should be S1 or higher to allow subnets - https://learn.microsoft.com/en-us/azure/app-service/overview-vnet-integration#pricing-details

In you apps you need to enable vnetRouteAllEnabledin siteConfig. Example bicep:

First you need to creat two apps. Example bicep file for app-one:

param location string = resourceGroup().location

resource hostingPlan 'Microsoft.Web/serverfarms@2020-12-01' = {
  name: 'hostingPlan'
  location: location
  kind: 'windows'
  sku: {
    name: 'S1'
    capacity: 1
  }
}

resource appService 'Microsoft.Web/sites@2021-03-01' = {
  name: 'app-one'
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    serverFarmId: hostingPlan.id
    httpsOnly: true
    siteConfig: {
      vnetRouteAllEnabled: true 
    }
  }
}

Network Part

This bicep will create VN, NAT Gateway, Public IP and SubNet for each Web App.

param location string = resourceGroup().location
var appOne = 'app-one'
var appTwo = 'app-two'

resource publicIp 'Microsoft.Network/publicIPAddresses@2021-05-01' = {
  name: 'public-ip-name'
  location: location
  sku: {
    name: 'Standard'
  }
  properties: {
    publicIPAddressVersion: 'IPv4'
    publicIPAllocationMethod: 'Static'
    idleTimeoutInMinutes: 4
  }
}

resource natgateway 'Microsoft.Network/natGateways@2021-05-01' = {
  name: 'natgateway-name'
  location: location
  sku: {
    name: 'Standard'
  }
  properties: {
    idleTimeoutInMinutes: 4
    publicIpAddresses: [
      {
        id: publicIp.id
      }
    ]
  }
}

resource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-05-01' = {
  name: 'virtualNetwork'
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '192.168.0.0/16'
      ]
    }
    subnets: [
      {
        name: 'subnet-for-${appOne}'
        properties: {
          addressPrefix: '192.168.0.0/24'
          natGateway: {
            id: natgateway.id
          }
          delegations: [
            {
              name: 'delegation'
              properties: {
                serviceName: 'Microsoft.Web/serverfarms'
              }
            }
          ]
        }
      }
      {
        name: 'subnet-for-${appTwo}'
        properties: {
          addressPrefix: '192.168.1.0/24'
          natGateway: {
            id: natgateway.id
          }
          delegations: [
            {
              name: 'delegation'
              properties: {
                serviceName: 'Microsoft.Web/serverfarms'
              }
            }
          ]
        }
      }
    ]
  }
}

resource prodcutsToSubnet 'Microsoft.Web/sites/networkConfig@2022-03-01' = {
  name: '${appOne}/virtualNetwork'
  properties: {
    subnetResourceId: virtualNetwork.properties.subnets[0].id
    swiftSupported: true
  }
}
resource webhooksToSubnet 'Microsoft.Web/sites/networkConfig@2022-03-01' = {
  name: '${appTwo}/virtualNetwork'
  properties: {
    subnetResourceId: virtualNetwork.properties.subnets[1].id
    swiftSupported: true
  }
}

Now all outbound connections from apps will use one static IP address.

Design Web API endpoint without leaking data to monitoring service

Andrei Kniazev — Sun, 09 Jan 2022 10:45:49 +0000

Right now SSL is a standard, but it is not enough to protect sensitive data. We can make a small mistake that can leak data that should be protected. For example — passwords. Even if you use cryptography to conceal the password bad endpoint design can expose it to a monitoring service or to a logger.

Here is a very common scenario. We decided to introduce the endpoint that will allow our users to restore passwords if they receive a restoration code.

[HttpPut("ChangePassword/{email}/{newPassword}/{code}")]
[ProducesResponseType(200)]
[ProducesResponseType(401)]
[ProducesResponseType(404)]
public IActionResult ChangePassword(string email, string newPassword, string code)
{
    // do logic
    return Ok();
}

Because I am using Azure in this example I will use Application Insights as a monitoring service for this app.

Let's hit the endpoint and check what we will be able to see in logging:

So as you see, It exposes the password of the user. To prevent this we need to change our Web API endpoint. Instead of using the URL itself, we will put data in the body.

[HttpPut("ChangePassword")]
[ProducesResponseType(200)]
[ProducesResponseType(401)]
[ProducesResponseType(404)]
public IActionResult ChangePassword([FromBody] ChangePassword body)
{
    // do logic
    return Ok();
}

Let's check the logs!

As you can see all the sensitive data should be put in the body of the request and not in the URL. SSL will encrypt everything but the monitoring service or logger will expose it.

Thank you and be safe!

TypeScript 'instanceof' interface is it possible?

Andrei Kniazev — Mon, 03 Jan 2022 13:55:26 +0000

I really like the power of interfaces in TypeScript and its duck typing. But what really bothers me is that I cannot check whether is something an instance of interface.

For example, we have decided to build a role model for the user class. User class will have different roles and based on role type it either has permissions or not. And we decided to describe roles as interfaces.

So I want to have code like this:

class User {
    constructor(readonly role: IReader | IWriter){};
}

interface IReader  {
}

interface IWriter {
    readonly permissions: Array<string>
}

function guardAccess(user: User){
    // Oh no it is not working 😱
    // Error TS2693
    if (user.role instanceof IReader){
      throw new AccessDeniedException();
    }
}

But in TypeScript, it won't work. Because types do not exist in runtime. There is no such thing as an interface in JavaScript.

Ok, so how can we fix this?

We can use string literal types in TypeScript to identify the interface. Let's update our code:

class User {
    constructor(readonly role: IReader | IWriter){};
}

interface IReader  {
    readonly name: "reader",
}

interface IWriter {
    readonly name: "writer"
    readonly permissions: Array<any>
}

function guardAccess(user: User){
    // ✔️ no error
    if (user.role.name === "reader"){
      throw new AccessDeniedException();
    }
}

It allows us to get data that is available only in a certain interface.

// lets create a user with role reader
const user = getUser();
if (user.role.name === "reader"){
    user.role.name;
    user.role.permissions; // 🛑 error
}
else if (user.role.name === 'writer'){
    user.role.name;
    user.role.permissions; // ✔️ no error
}

Also it TypeScript will help us to control the values of the role's name.

if (user.role.name === "editor"){
 // 🛑 Error: TS 2367This condition will always return 'false'
 // since the types '"reader" | "writer"' and '"editor"' 
 // have no overlap.
}

That is it. Now we can have similar behavior to the instance of the interface in TypeScript.

Thank you for reading!

Migration from tiptap v.1 to tiptap v.2

Andrei Kniazev — Thu, 23 Dec 2021 09:48:19 +0000

If you are working with tiptap v.1 and want to upgrade to tiptap v.2 here is the story of how I did it for my project LoreHub.

LoreHub's stack
Back - .net 6, ef 6, c#
Front - Vue.js 2, Veutify, Pinia

Initial setup

I have a Vue.js component that does a put request that will update the description on the server. It will do two things:

Updates in the description's snapshot table.
Insert into the description's history table.

Here are parts from the Vue component. As you can see I initialize the editor and on an update, it fires debounce function. Debounce function allows it to do requests only if the user will stop updating the content for 3 seconds. The debounce function is from lodash.

import debounce from "lodash-es/debounce";

function initEditor () {
    this.editor = new Editor({
        extensions: [
          // extensions
        ],
        onUpdate: ({ getJSON }) => {
          this.content = getJSON();
          this.isSaving = true;
          this.updateDocumentDescriptionOnServerDebounce();
        },
    });
}

function updateDocumentDescriptionOnServerDebounce: debounce(async function () {
    await this.updateDocumentDescriptionOnServer();
}, 3000),

async updateDocumentDescriptionOnServer() {
    try {
        this.serverError = null;
        // pinia store action - put to WebApi
        await useDescriptionStore().updateDescription(
          this.settingId,
          this.type,
          this.forId,
          this.content
        );
        this.isSavingValue = false;
      } catch (e) {
        // some error handling
        this.serverError = e;
        this.isSavingValue = false;
      }
}

Time to upgrade

First of all, I do everything that is mentioned in the official upgrade guide - https://tiptap.dev/overview/upgrade-guide

So it took some time, but then I face a problem that requires me to migrate standard extensions names. Imagine I have gigabytes of data in my database that contains JSONs and I need to iterate through all of this to rename extensions types. It is not an option and there should be a better way to do it.

How do I solve it? My idea was to create a migration function that will do it on the front end. But I don't want it to run every time the description is loaded. The solution will be to save the state in the database that the migration was performed and I don't want it to do it again.

I decided to change my WebAPI and database. I introduce a new column 'EditorVersion'. Because I have too many raws to update I set this field as nullable, without a default value.

Example entity framework migration:

// ef 6 migration
migrationBuilder.AddColumn<string>(
    name: "EditorVersion",
    table: "Descriptions_History",
    type: "nvarchar(max)",
    nullable: true);

migrationBuilder.AddColumn<string>(
    name: "EditorVersion",
    table: "Descriptions_Description",
    type: "nvarchar(max)",
    nullable: true);

After this I've created a v2 description.get action on the back that returns not just JSON, but JSON and editorVersion.

Description {
    value   string
    nullable: true
    editorVersion   string
    nullable: true
}

If editorVersion is null it will run this migration on the front:

migrateExtensions(content) {
      for (const node of content) {
        // tiptap 2 migrate extensions type from v.1 to v2.
        // https://tiptap.dev/overview/upgrade-guide#new-names-for-most-extensions
        if (node.type === "bullet_list") node.type = "bulletList";
        if (node.type === "code_block") node.type = "codeBlock";
        if (node.type === "hard_break") node.type = "hardBreak";
        if (node.type === "horizontal_rule") node.type = "horizontalRule";
        if (node.type === "list_item") node.type = "listItem";
        if (node.type === "ordered_list") node.type = "orderedList";
        if (node.type === "table_cell") node.type = "tableCell";
        if (node.type === "table_header") node.type = "tableHeader";
        if (node.type === "table_row") node.type = "tableRow";
        if (node.type === "todo_list") node.type = "taskList";
        if (node.type === "todo_item") node.type = "todo_item";

        // recursion
        if (node.content && node.content.length > 0)
          migrateExtensions(node.content);
      }
    }

After the migration is done it will perform put request that will send updated JSON and set editor version to 'tiptap_v2'.

Conclusion

It took me about three days of work to migrate from tiptap v1 to tiptap v2. It includes migration for a custom extension that uses a Vue router for links. The tiptap's team did a good job with the migration guide, thank you. It was straightforward and easy to do.

Overall I like the new tiptap's API and this little hack allows to do a lazy migration 😊.

I hope this guide will help you do the migration and if you have any questions feel free to ask.