DEV Community: Lord Jake

Service Bus Data not pulling through webhooks

Lord Jake — Thu, 12 Oct 2023 22:33:55 +0000

We had an issue reported from App team where the messages was not going to service bus queue - this got fixed by enabling the private endpoint.

Later there was an issue where the webhook listening to the queue is not receiving messaging.
Changed the messaging type to AMQP and then NSG was modified to accept port 443 and 5671.

References:

https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-protocol-guide

Azure App Proxy Header Based SSO not passing headers to WebAPP

Lord Jake — Thu, 12 Oct 2023 09:36:06 +0000

One of my customer wanted to implement SSO with Azure App Proxy, they had a requirement to display the username, and persist the username and user email back to the cosmos db to retrieve in future for further use cases.

Issue faced.
Headers sent from App Proxy was not reaching the webapp.

We had an architecture similar to below one.

Checks done
First point of check was to intercept App proxy traffic and we indeed confirmed headers are being send from there.

Second check was in nginx, from which the React UI is served. It looked like nginx was not forwarding the headers.

We did the following changed in ngnix.conf and it started flowing.

        location / {
            root /app/www/;
            add_header UserUPN $http_userupn;
            add_header UserDisplayName $http_userdisplayname;
        }

Second Issue
Another issue we faced was , the react UI was not able to fetch the response header, since it was loaded by nginx and no back end api calls are made as well during the load. We got were able to retrieve the headers after doing another get from the app proxy , but it was a round trip again. We finally overcame it by injecting the headers to response html meta headers using nginx using sub_filter, and made react to read the dom and query the meta headers.

        location / {
            root /app/www/;
            add_header userupn $http_userupn;
            add_header userdiplayname $http_userdisplayname;
            sub_filter '</head>' '<meta name="userupn" content="$http_userupn" />\n<meta name="userdisplayname" content="$http_userdisplayname" /></head>';
            sub_filter_once off;
        }

  React.useEffect(() => {
    // Select all meta elements in the document's head
    const metaElements = document.head.querySelectorAll('meta');

    // Loop through the meta elements and extract values
    metaElements.forEach((meta) => {
      const name = meta.getAttribute('name');
      const content = meta.getAttribute('content');

      // Check for the "userupn" and "userdisplayname" meta tags
      if (name === 'userupn') {
        setUserUpn(content || 'User Upn Not Found'); // Set the userUpn state
      } else if (name === 'userdisplayname') {
        setUserDisplayName(content || 'User'); // Set the userDisplayName state
      }
    });
  }, []);

Werkzeug Import Error - Python

Lord Jake — Wed, 04 Oct 2023 08:04:59 +0000

We got the below error all of a sudden from the already working Flask api code yesterday. Looks like there was some changes in dependent Werkzeug library.

Mentioning the version of Werkzeug in requirements.txt fixed the issue.

Werkzeug==2.3.7

References:
https://stackoverflow.com/questions/77213053/importerror-cannot-import-name-url-quote-from-werkzeug-urls

How to Enable CORS for API Management Development Portal using Terraform

Lord Jake — Thu, 27 Jul 2023 07:24:28 +0000

Today I got a task to automate the Enable CORS feature in the Portal Overview of Developer Portal in Azure API Management Services.

This can be achived either via azure_rm or az_api according to documentations. I used azure_rm to achieve this.

We would need to create a policy file with CORS definitions and refer the file to the terraform configurations.

Policy file will look as below.

<policies>  
    <inbound>    
        <cors allow-credentials="true">      
            <allowed-origins>        
                <origin>https://example-apim-19899.developer.azure-api.net</origin>      
            </allowed-origins>      
            <allowed-methods preflight-result-max-age="300">        
                <method>*</method>      
            </allowed-methods>      
            <allowed-headers>        
                <header>*</header>      
            </allowed-headers>      
            <expose-headers>        
                <header>*</header>      
            </expose-headers>    
        </cors>  
    </inbound>  
    <backend>    
        <forward-request />  
    </backend>  
    <outbound />
</policies>

Terraform code can be created as below to use the policy.

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "uksouth"
}

resource "azurerm_api_management" "example" {
  name                = "example-apim-19899"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  publisher_name      = "pub1"
  publisher_email     = "pub1@email.com"

  sku_name = "Developer_1"
}

resource "azurerm_api_management_policy" "example" {
  api_management_id = azurerm_api_management.example.id
  xml_content       = file("\\policy_files\\example.xml")
}

On running Terraform, the policies should be set and CORS should be enabled. Please make sure you give the origin in the policy file as the developer portal URL so that Azure correctly match it and it gets enabled in portal.

If you need to try using AzApi further details can be found here.
https://learn.microsoft.com/en-us/azure/templates/microsoft.apimanagement/service/policies?pivots=deployment-language-terraform

How to run self hosted agent on Azure Container Instance

Lord Jake — Wed, 26 Oct 2022 16:24:18 +0000

Azure Container Instance may be a good idea for some who don't want to run a full blown VM to run the Devops agent, which is light and probably cost effective.

1 Create a container registry via portal or CLI.

2 Create docker agent image and upload to the created container registry. The docker agent will depend on the OS to be used - I am using Ubuntu 20.04.

3 I assume you have already a running docker engine or else please install and make the service up and running.

4 mkdir ~/dockeragent
mkdir ~/dockeragent

5 Save the following content to ~/dockeragent/Dockerfile

FROM ubuntu:20.04
RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get upgrade -y

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \
    apt-transport-https \
    apt-utils \
    ca-certificates \
    curl \
    git \
    iputils-ping \
    jq \
    lsb-release \
    software-properties-common

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

# Can be 'linux-x64', 'linux-arm64', 'linux-arm', 'rhel.6-x64'.
ENV TARGETARCH=linux-x64

WORKDIR /azp

COPY ./start.sh .
RUN chmod +x start.sh

ENTRYPOINT [ "./start.sh" ]

6 Save the following content to ~/dockeragent/start.sh

#!/bin/bash
set -e

if [ -z "$AZP_URL" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

if [ -z "$AZP_TOKEN_FILE" ]; then
  if [ -z "$AZP_TOKEN" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE=/azp/.token
  echo -n $AZP_TOKEN > "$AZP_TOKEN_FILE"
fi

unset AZP_TOKEN

if [ -n "$AZP_WORK" ]; then
  mkdir -p "$AZP_WORK"
fi

export AGENT_ALLOW_RUNASROOT="1"

cleanup() {
  if [ -e config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    # If the agent has some running jobs, the configuration removal process will fail.
    # So, give it some time to finish the job.
    while true; do
      ./config.sh remove --unattended --auth PAT --token $(cat "$AZP_TOKEN_FILE") && break

      echo "Retrying in 30 seconds..."
      sleep 30
    done
  fi
}

print_header() {
  lightcyan='\033[1;36m'
  nocolor='\033[0m'
  echo -e "${lightcyan}$1${nocolor}"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE=AZP_TOKEN,AZP_TOKEN_FILE

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_PACKAGES=$(curl -LsS \
    -u user:$(cat "$AZP_TOKEN_FILE") \
    -H 'Accept:application/json;' \
    "$AZP_URL/_apis/distributedtask/packages/agent?platform=$TARGETARCH&top=1")

AZP_AGENT_PACKAGE_LATEST_URL=$(echo "$AZP_AGENT_PACKAGES" | jq -r '.value[0].downloadUrl')

if [ -z "$AZP_AGENT_PACKAGE_LATEST_URL" -o "$AZP_AGENT_PACKAGE_LATEST_URL" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent"
  echo 1>&2 "check that account '$AZP_URL' is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and extracting Azure Pipelines agent..."

curl -LsS $AZP_AGENT_PACKAGE_LATEST_URL | tar -xz & wait $!

source ./env.sh

print_header "3. Configuring Azure Pipelines agent..."

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "$AZP_URL" \
  --auth PAT \
  --token $(cat "$AZP_TOKEN_FILE") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

print_header "4. Running Azure Pipelines agent..."

trap 'cleanup; exit 0' EXIT
trap 'cleanup; exit 130' INT
trap 'cleanup; exit 143' TERM

chmod +x ./run-docker.sh

# To be aware of TERM and INT signals call run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run-docker.sh "$@" & wait $!

7 Build the docker image

docker build -t <your-acr-name>.azurecr.io/dockeragent:latest .

8 Upload the image to the already created container registry via CLI or portal;

az acr login --name <acr name> --username <adminUsername> --password <adminPassword>

docker push <your-acr-name>.azurecr.io/dockeragent:latest

9 Verify the image is in the azure container registry repository from portal

10 Create an Azure container Instance from either portal or via CLI , using the container registry attached.

NB: You need to provide the environment variables here itself. I found it not able to edit the variables once after the container instance is provisioned.

Below four are the variables required.
AZP_URL=https: your devops url
AZP_TOKEN=<Pat-Token-from-Azure-Devops>
AZP_AGENT_NAME=<Your-agent-name-which will be displayed in ADO>
AZP_POOL=<Pool-name- can be 'Default' as well>

To get the PAT Token. Navigate to Azure Devops, select Personal Access Tokens from User Settings Menu, and select New Token

Give it a name, and select Custom defined permission for Agent Pools - Read & Manage. If you are not able to see it, click on the bottom link to show all scopes
Once created copy the token, it won't be shown again.

Once created the container instance should spin up and the agent should be start listening to.

You can verify it checking in the AzureDevops Agent Pools too from Organization Settings.

I ran a sample pipeline using the container pool and it ran succesfully.

References:
https://jan-v.nl/post/2021/create-build-agent-with-azure-container-instances/

https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops&WT.mc_id=AZ-MVP-5003246#create-and-build-the-dockerfile-1

Install Azure CLI on arm64 Raspberry pi

Lord Jake — Tue, 25 Oct 2022 22:16:57 +0000

According to Microsoft documentations currently arm64 is not supported for Azure CLI.

The only work around is to install via pip.

The below worked for me, in RPi 4B, Ubuntu 20.04

Install
First make sure python3 and its related packages are installed:

Ubuntu/Debian

sudo apt install python3 python3-venv --yes

# Create a virtual environment
python3 -m venv azure-cli-env

# Update pip
azure-cli-env/bin/python -m pip install --upgrade pip

# Install azure-cli
azure-cli-env/bin/python -m pip install azure-cli

# Run any Azure CLI commands
azure-cli-env/bin/az --version

Uninstall
Delete the virtual environment:
rm -rf azure-cli-env

References:
https://github.com/Azure/azure-cli/issues/20476
https://www.frakkingsweet.com/installing-azure-cli-on-arm64/ - I am just posting this particular link for future reference, I haven't tried this.

Install Docker engine to a development Linux box - easy way

Lord Jake — Tue, 25 Oct 2022 21:21:31 +0000

This method should only be used in development boxes not in production scenarios according to docker official documentation. Also validate the script before executing. You should not run this script to upgrade your existing docker engine, it may cause issues.

curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh

I found it really easy to get up and running for testing with docker on my raspberry pi4 running on Ubuntu 20.04.

References:
https://docs.docker.com/engine/install/ubuntu/

Connect to WiFi on Ubuntu (Server version) via Terminal

Lord Jake — Tue, 25 Oct 2022 21:13:02 +0000

This tutorial assume you have an active LAN connected, since you may need to install a package.

sudo apt install network-manager

Then you can use the below interactive mode to search for wifi and connect to the same

sudo nmtui

Now I can see the Wlan0 link is UP (which was DOWN before) and it has got an IP too.

Found this really easy if you got a LAN or your ubuntu version has network-manager pre installed.

Failed pre-install: timed out waiting for the condition. Helm install error

Lord Jake — Wed, 19 Oct 2022 16:32:53 +0000

Today I was trying to install ingress controller using helm3 and got an error like below.

Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition.

I felt the above error was not giving a proper indication of what went wrong, but this could be identified by collecting the event logs and see what is actually failing under the hood.

kubectl -n <your-namespace> get events --sort-by='{.lastTimestamp}'

Which showed me there was missing images in my azure container registry which was causing ErrImagePull.

Remote Windows PSSesssion not working from Linux K8s pod

Lord Jake — Tue, 18 Oct 2022 15:11:56 +0000

Today we had an issue in which a microservice which was deployed onto a linux pod was not able to make a remote PowerShellSession (PSSession) to a Windows machine.

The base image already had PSWSMan and openssl installed, but we were getting MI_RESULT_ACCESS_DENIED error.

After long search through different posts and trial and errors, we were able to make the connection successful with the below changes in the docker image.

Add the two below packages to image

RUN apt-get install netbase -y RUN apt-get install gss-ntlmssp -y

Netbase:
Basic TCP/IP networking system
This package provides the necessary infrastructure for basic TCP/IP based networking. In particular, it supplies common name-to-number mappings in /etc/services, /etc/rpc, /etc/protocols and /etc/ethertypes.

gss-ntlmssp:
GSS-NTLMSSP is a GSSAPI mechanism plugin that implements NTLMSSP. NTLMSSP is a Microsoft Security Provider that implements various versions and falvors of the NTLM challenge-response family.

GSS-NTLMSSP, implements both NTLM and NTLMv2 and all the various security variants to the key exchange that Microsoft introduced and documented over time.

This code implements the NTLMSSP mechanism as a GSSAPI loadable mechanism and has been tested to work with MIT Kerberos' 1.11 implementation of GSSAPI.

Also the PSSession command was ran with the Negotiate authentication method

Enter-PSSession -ComputerName <IP/Hostname> -Credential <xxxxx> -Authentication Negotiate

References and further reading
https://packages.debian.org/sid/netbase
https://github.com/gssapi/gss-ntlmssp
https://packages.debian.org/sid/libs/gss-ntlmssp
https://www.bloggingforlogging.com/2020/08/21/wacky-wsman-on-linux/
https://github.com/PowerShell/PowerShell/issues/6647
https://github.com/jborean93/omi/issues/29
https://www.crowdstrike.com/cybersecurity-101/ntlm-windows-new-technology-lan-manager/

AKS node crash and Sonar rebuild

Lord Jake — Sat, 15 Oct 2022 11:30:43 +0000

Today we had a strange issue in which a pipeline in Azure devops was failing and on investigation we came to know that , it was failing due to sonar errors which in turn pointed to the sonarqube pod deployed in the cluster getting evicted and the node was under severe memory pressure with lots of evicted pods building up which further pressurized the node. The reason for severe memory pressure was not clear though, but we followed the below steps to get things back up again.

Tried deleting the evicted pods to check if the node was getting recovered.

kubectl delete pods --field-selector=status.phase=Failed --all-namespaces

Since this didn't resolve, we scaled out the cluster, using Azure portal.
Select the cluster -> Node pools -> Select the Nodepool -> Scale pool -> Scale to 2 more nodes ( we had 3 , we scale to 5)

Meanwhile the affected node was tainted with a Memory Pressure hence no pods were getting allocated to it.

Once the new nodes were started, we drained the problematic node and took it down. This is for safely evicting the pods. More on the link here

Once drained and the node was taken down. Still the sonar pod was not getting scheduled on any nodes.

We checked sonar pods describe to see the events
kubectl describe po sonar -n sonar

Pods were not able to get scheduled due to a volume node affinity conflict as below

We checked the persistent volume claim (pvc) in sonar namespace

kubectl get pvc -n sonar

Described the sonar-pvc
kubectl describe pv pvc-xxxxxx

Now the issue got narrowed down to a possibility that we don't have a node in a particular availability-zone in Azure, where the pvc is currently configured to. The sonar pods should be put on the same node to connect with the pvc (all in same availability zone node) hence the error.

So we scaled down the instances to 2 and then scaled up back to 3, and AKS created nodes in each zone which also created a node in
southeastasia-3 , which is now is agreement with the pvc node affinity section.

And this made the sonar pods to be deployed to the node with affinity and thereby bringing the sonar up resolving the pipeline issue as well.

K8s POD Auto scaling with KEDA and Azure Service Bus

Lord Jake — Sat, 15 Oct 2022 10:15:48 +0000

In one of my projects we had a requirement to scale the pods based on the queue depth of the Topic in Azure Service Bus and the search ended in KEDA - Kubernetes Event Driven Autoscaling which is a cloud native foundation, incubation project. Now let's look how we can setup Keda in our cluster.

KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication.

Keda got two key roles within the cluster, keda-operator which scales from minimum to maximum pod counts set in the ScaledObject manifest file via Kubernetes Horizontal Pod Autoscaler and keda-operator-metrics-apiserver which gets the data for the scaling decision.

Keda Installation
Keda operator and metric api-server can be installed in its own namespace and we will deploy the actual scaledobject component in the same namespaces where the pods to be scaled are.

Keda can be installed in different ways, I chose using Helm, kubectl scripts are also available in the Keda website

Install Keda using Helm
Add Helm repo
helm repo add kedacore https://kedacore.github.io/charts

Update Helm repo
helm repo update

Install Keda Helm chart
kubectl create namespace keda helm install keda kedacore/keda --namespace keda

Configuring the Azure Service Bus SAS policy
For configuring KEDA to operate on the ServiceBus, it would require management permission.

You can select the connection strings from this SAS key for updating in the KEDA yaml manifest

YAML Manifests
Scaling is made by a ScaledObject, which in turn uses the TriggerAuthentication object for the Azure Service Bus authentication.

apiVersion: v1
kind: Secret
metadata:
  namespace: poc 
  name: keda-secrets
  labels:
    app: exchange-jobs
data:
  management-connectionstring: <***Your base 64 encoded connection string from Azure SAS Key section with manage permissions****>
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  namespace: poc
  name: exchange-jobs-keda-scaler
spec:
  scaleTargetRef:
    name: exchange-jobs
  pollingInterval: 1   # Optional. Default: 30 seconds - checks for the changes
  cooldownPeriod: 20   # Optional. Default: 300 seconds - shutdown - need to find a sweet spot , Its better to keep default so that we don't interrupt any running processes
  minReplicaCount: 0
  maxReplicaCount: 50  # Optional. Default: 100
  triggers:
  - type: azure-servicebus
    metadata:
      topicName: mytopic
      subscriptionName: S1
      namespace: test-sb-keda-scaler   
      messageCount: "5" # need to find a sweet spot
    authenticationRef:
      name: keda-sbus-auth

---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  namespace: poc
  name: keda-sbus-auth
spec:
  secretTargetRef:
  - parameter: connection
    name: keda-secrets
    key: management-connectionstring

The above yaml can be applied to the namespace where the pods are present and it should work as expected.

Things to note:

The cool down lesser than 300 seconds (which is the default) will only work if you set the minReplicaCount to 0. If you don't specify a minReplicaCount Keda assumes as 0.
SB Connection string should be given as base 64 encoded.
TriggerAuthentication object parameter is connection not connectionString

If you need a SBUS topic sender and receiver for testing console app please find it here

Some useful references.
◉ Kubernetes Event-driven Autoscaling – https://aka.ms/azfr/662/01
◉ KEDA on GitHub – https://aka.ms/azfr/662/02
◉ Azure Functions on Kubernetes with KEDA – https://aka.ms/azfr/662/03
◉ Azure Friday - Azure Serverless on Kubernetes with KEDA – https://aka.ms/azfr/662/04