DEV Community: Imran Hayder

How to update Azure load balancer backend pool via a python script

Imran Hayder — Mon, 05 Oct 2020 04:34:15 +0000

Some time back ago I was working on a very interesting problem in Azure. Lets says we have two VMs added in a load-balancer. Now the ask was to do some maintenance on one of the vm while other was still in the load-balancer.

The requirements were:

take the VM out of load-balancer for maintenance
do some work on it
add it back to the load-balancer

Now there wasn't any automated way to do it via Azure so I came up with following python script. Hopefully someone out there in same predicament finds it useful :)

#!/usr/bin/env python3
import argparse
import configparser
import logging.config
import os
import sys

# Setup..
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.network import (
    NetworkManagementClient,
)
from azure.mgmt.compute import (
    ComputeManagementClient
)
import requests
from requests import Request, Session

logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')
log = logging.getLogger(__name__)

parser = argparse.ArgumentParser('RMS(one) Load Balancer update')
parser.add_argument('-s', '--stack', action='store', dest='stack', metavar='', required=True, help='Name of stack to update LB')
parser.add_argument('-a', '--action', action='store', dest='action', metavar='', required=True, help='Action for LB', choices=['create-maint', 'delete-maint'])
args = parser.parse_args()
stack = args.stack
action = args.action
base_url = 'https://management.azure.com/'

def get_token_from_client_credentials(endpoint, client_id, client_secret):
    payload = {
        'grant_type': 'client_credentials',
        'client_id': client_id,
        'client_secret': client_secret,
        'resource': 'https://management.core.windows.net/',
    }
    #TODO add back in verify for non-fiddler
    #NOTE add Verify=False when going via a proxy with fake cert / fiddler
    response = requests.post(endpoint, data=payload).json()
    return response['access_token']

def get_virtual_machine(compute_client, resource_group_name, vm_name):
    """
    :param resource_group_name: str
    :param vm_name: str
    :return: azure.mgmt.compute.VirtualMachine
    """
    virtual_machine = compute_client.virtual_machines.get(resource_group_name, vm_name)
    logging.info('using virtual machine id: %s', virtual_machine.id)
    return virtual_machine

def get_network_interface_ip_configuration(network_client, resource_group_name, network_interface_name):
    network_interface = network_client.network_interfaces.get(resource_group_name, network_interface_name)
    return network_interface
    #for ipconfig in network_interface.network_interface.ip_configurations:
    #    return ipconfig

def get_virtual_machine_network_interface(compute_client, network_client, resource_group_name, virtual_machine_name):
    virtual_machine = get_virtual_machine(compute_client, resource_group_name, virtual_machine_name)
    for profile in virtual_machine.network_profile.network_interfaces:
        print(profile.id)
        nic_uri = profile.id

    #network_interface = get_network_interface(resource_group_name)
    label = os.path.basename(os.path.normpath(nic_uri))
    logging.info('nic on vm to use is: %s', label)

    network_interface = get_network_interface_ip_configuration(network_client, resource_group_name, label)
    logging.info('nic id is: %s', network_interface.id)
    return network_interface

def build_request(vm_object, nic_object, load_balancer=None):
    """
    :param vm_object : azure.mgmt.compute.VirtualMachine
    :param nic_object : azure.mgmt.network.networkresourceprovider.NetworkInterface
    :param load_balancer : azure.mgmt.network.LoadBalancer
    :return: dict
    """
    if load_balancer is None:
        backend_pool = []
    else:
        backend_pool = [{'id' : load_balancer.backend_address_pools[0].id}]

    request = {
        'properties': {
            'virtualMachine' : {
                'id' : vm_object.id
                },
            'ipConfigurations' : [{ #may have to build by hand
                'properties' : {
                    'loadBalancerBackendAddressPools' : backend_pool,
                    'subnet' : {
                        'id' :  nic_object.ip_configurations[0].subnet.id
                        }
                    },
                'name' : nic_object.ip_configurations[0].name,
                'id' : nic_object.ip_configurations[0].id
            }]
        },
        'id' : nic_object.id,
        'name' : nic_object.name,
        'location' : vm_object.location,
        'type' : 'Microsoft.Network/networkInterfaces'
        }

    return request

def send_loadbalancer_request(payload, auth_token, resource_id, max_retries=20):
    endpoint = base_url + resource_id + '?api-version=2019-06-01'
    header = {'Authorization' : 'Bearer ' + auth_token}
    while max_retries > 0:
        session = Session()
        request = Request('PUT', endpoint, json=payload, headers=header)
        prepared = session.prepare_request(request)

        log.debug('raw body sent')
        log.debug(prepared.body)

        response = session.send(prepared)
        print(response.status_code)
        print(response.text)
        if response.status_code == 200:
            break
        elif response.status_code == 429:
            log.info('retrying an HTTP send due to 429 retryable response')
            log.info('this will be try# %s', max_retries)
        max_retries = max_retries - 1
    return response

def main():
    ini_config = configparser.ConfigParser()
    ini_config.read('~/azure.ini')
    stack_data = ini_config[stack]
    tenant_id = stack_data['tenant']
    client_id = stack_data['client_id']
    client_secret = stack_data['secret']
    sub_id = stack_data['subscription_id']
    endpoint = 'https://login.microsoftonline.com/' + tenant_id + '/oauth2/token'
    auth_token = get_token_from_client_credentials(endpoint, client_id, client_secret)
    # now the Azure management credentials
    credentials = ServicePrincipalCredentials(client_id=client_id,
                                              secret=client_secret,
                                              tenant=tenant_id)
    # now the specific compute, network resource type clients
    compute_client = ComputeManagementClient(credentials, sub_id)
    network_client = NetworkManagementClient(credentials, sub_id)
    # Resources
    resources = {"vmnames":{"dcos_vms": [f"{stack}-dcos-extpublicslave1", f"{stack}-dcos-extpublicslave2"], "maint_vm" : f"{stack}-maintpage"},
                 "vmResourceGroup": f"{stack}-dcos",
                 "netResourceGroup": f"{stack}-Network-Infrastructure",
                 "loadBalancerName": f"{stack}-dcos-extpublicslave",
                 "subnetName": f"{stack}-SubNet1",
                 "virtualNetworkName" : f"{stack}-Vnet1"}

    #TODO modify this to mach your specific settings
    vm_resource_group = resources['vmResourceGroup']
    load_balancer_name = resources['loadBalancerName']
    #TODO - end - only the "above" should need to change.
    dcos_vms_res = {}
    maint_vm_res = {}
    for dcos_vm_name in resources["vmnames"]["dcos_vms"]:
        dcos_vms_res[dcos_vm_name] = {"vm":"", "nic":""}
        dcos_vms_res[dcos_vm_name]["vm"] = compute_client.virtual_machines.get(vm_resource_group, dcos_vm_name)
        dcos_vms_res[dcos_vm_name]["nic"] = get_virtual_machine_network_interface(compute_client, network_client, vm_resource_group, dcos_vm_name)
    maint_vm_res["vm"] = compute_client.virtual_machines.get(vm_resource_group, resources["vmnames"]["maint_vm"])
    maint_vm_res["nic"] = get_virtual_machine_network_interface(compute_client, network_client, vm_resource_group, resources["vmnames"]["maint_vm"])
    #the load balancer
    load_balancer = network_client.load_balancers.get(vm_resource_group, load_balancer_name)

    # running maint on/off action
    if action == "create-maint":
        log.info("Running maintenance ON ")
        maint_vm_lb = load_balancer
        dcos_vm_lb = None
    elif action == "delete-maint":
        log.info("Running maintenance OFF ")
        maint_vm_lb = None
        dcos_vm_lb = load_balancer
    else:
        log.error("ERROR: invalid action specified")
        sys.exit(1)

    maint_lb_request = build_request(maint_vm_res["vm"], maint_vm_res["nic"], maint_vm_lb)
    send_loadbalancer_request(maint_lb_request, auth_token, maint_vm_res["nic"].id)
    for dcos_vm in dcos_vms_res:
        dcos_lb_request = build_request(dcos_vms_res[dcos_vm]["vm"], dcos_vms_res[dcos_vm]["nic"], dcos_vm_lb)
        send_loadbalancer_request(dcos_lb_request, auth_token, dcos_vms_res[dcos_vm]["nic"].id)


if __name__ == "__main__":
    main()

The script is also added on my github gist so feel free to check out.

How to run the script

Save the script as lb.py and run as:

python lb.py -s stack_name -a create-maint

The stack_name is whatever I used in azure.ini file to get the credential from . The options create-maint and delete-maint are used to switch back and forth between the two vms. tested with python 3 and python 2
Note I follow naming convention for my resource so stack has to be passed you can totally remove the inputs as you like .

Create a simple VPC Peer between Kubernetes and RDS(postgres)

Imran Hayder — Thu, 19 Mar 2020 15:46:00 +0000

Create a VPC Peering connection between EKS Kubernetes and RDS Postgres

Note: this script assumes your resources names are prefixed with name of EKS cluster so if EKS cluster name is "myEKS" then the EKS VPC name is myEKS/VPC.

please fix this script variables in Step#1 according to your naming convention or actual names of resources.
all rest of steps are same.

Set some basic information like EKS names / VPC Names

Setting variables for EKS cluster:

EKS_CLUSTER="name_of_EKS_cluster_goes_here"
# set name of VPC in which EKS exists, for me i use eksctl to create eks
# so vpc name is automatically set to name_of_eks_cluster/VPC
# you can change here to whatever your VPC name is
EKS_VPC="$EKS_CLUSTER"/VPC 
# same goes for the VPC public routing table name of EKS
EKS_PUBLIC_ROUTING_TABLE="$EKS_CLUSTER"/PublicRouteTable

and for RDS:

RDS_NAME="name_of_RDS_goes_here"
# set this variable to the name of VPC in which RDS exists 
RDS_VPC="$RDS_NAME"/VPC
# same goes for private routing table of RDS
RDS_PRIVATE_ROUTING_TABLE="$RDS_NAME"/RDSPrivateRoutingTable

RDS_DB_NAME="Name_of_RDS_instance"

Now that all variables are set, the following steps should be run as simply copy-paste

Get VPC ID of acceptor i.e. RDS

echo "getting the VPC ID and CIDR of acceptor(RDS instance)"
ACCEPT_VPC_ID=$(aws ec2 describe-vpcs --filters Name=tag:Name,Values=$RDS_VPC --query=Vpcs[0].VpcId --output text)
ACCEPT_CIDR=$(aws ec2 describe-vpcs --filters Name=tag:Name,Values=$RDS_VPC --query=Vpcs[0].CidrBlockAssociationSet[0].CidrBlock --output text)

Get VPC ID of requestor i.e. EKS

REQUEST_VPC_ID=$(aws ec2 describe-vpcs --filters Name=tag:Name,Values=$EKS_VPC --query=Vpcs[0].VpcId --output text)
REQUEST_CIDR=$(aws ec2 describe-vpcs --filters Name=tag:Name,Values=$EKS_VPC --query=Vpcs[0].CidrBlockAssociationSet[0].CidrBlock --output text)

get Public Route table ID of requestor and acceptor

REQ_ROUTE_ID=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=$EKS_PUBLIC_ROUTING_TABLE --query=RouteTables[0].RouteTableId --output text)
ACCEPT_ROUTE_ID=$(aws ec2 describe-route-tables --filters Name=tag:Name,Values=$RDS_PRIVATE_ROUTING_TABLE --query=RouteTables[0].RouteTableId --output text)

Create Peering Connection

peerVPCID=$(aws $DRY_RUN ec2 create-vpc-peering-connection --vpc-id $REQUEST_VPC_ID --peer-vpc-id $ACCEPT_VPC_ID --query VpcPeeringConnection.VpcPeeringConnectionId --output text)
aws $DRY_RUN ec2 accept-vpc-peering-connection --vpc-peering-connection-id "$peerVPCID"
aws $DRY_RUN ec2 create-tags --resources "$peerVPCID" --tags 'Key=Name,Value=eks-peer-rds'

Adding the private VPC CIDR block to our public VPC route table as destination

aws $DRY_RUN ec2 create-route --route-table-id "$REQ_ROUTE_ID" --destination-cidr-block "$ACCEPT_CIDR" --vpc-peering-connection-id "$peerVPCID"
aws $DRY_RUN ec2 create-route --route-table-id "$ACCEPT_ROUTE_ID" --destination-cidr-block "$REQUEST_CIDR" --vpc-peering-connection-id "$peerVPCID"

Add a rule that allows inbound RDS (from our Public Instance source)

RDS_VPC_SECURITY_GROUP_ID=$(aws rds describe-db-instances --db-instance-identifier $RDS_DB_NAME --query=DBInstances[0].VpcSecurityGroups[0].VpcSecurityGroupId --output text)
aws ec2 authorize-security-group-ingress --group-id ${RDS_VPC_SECURITY_GROUP_ID} --protocol tcp --port 5432 --cidr "$REQUEST_CIDR"

TESTING CONNECTIONS

Run postgresql container :

kubectl run -i --tty --rm postgresdebug --image=alpine:3.5 -- 
 restart=Never -- sh

install postgresql:
```
apk update
apk add postgresql
```

Run PSQL:

psql -h <HOST> -U <USER>
Password for user <USER>:
psql (9.6.10, server 9.6.15)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM- 
SHA384, bits: 256, compression: off)
Type "help" for help.
<DB_NAME>=

Adding cross-account access to EKS

Imran Hayder — Thu, 12 Mar 2020 18:42:08 +0000

introduction

When you want your users in IAM to access EKS cluster in another account, its very simple to do via cross account role.
This assumes you have already created the role in account B to users in account A.

steps to access EKS in second account

first make sure you have a IAM role cross-account-role created in Account B and having added trusted relationship for users in that you would like to from account A to access it.
Once thats done , make sure you have access to the EKS cluster in account B(this needs to be done in order to edit the permissions of EKS).
now edit the aws-auth configmap of that EKS cluster as:
```
kubectl edit -n kube-system configmaps aws-auth
```

add following lines under mapRoles to add the role created in step#1:

- "groups":
  - "system:masters"
  - "system:nodes"
  "rolearn": "arn:aws:iam::Account B:role/cross-account-role"

try setting the new cross-account for account B in ~/.aws/credentials :

[account-B]
role_arn = arn:aws:iam::Account B:role/cross-accountrole
region = us-west-2
source_profile = account-A

export this profile on terminal and add the EKS cluster config :

export AWS_PROFILE=account-B
aws eks update-kubeconfig --name name-of-eks-cluster-in-account-B

try running kubectl now:
```
kubectl get ns
kubectl get pods
```

Canary deployment in Gitlab using nginx-ingress

Imran Hayder — Wed, 04 Mar 2020 22:55:32 +0000

Canary deployments in gitlab AutoDevops using nginx-ingress

Gitlab AutoDevops is a great feature in Gitlab which allows us to Build,Test,Deploy our apps seamlessly to Kubernetes.

This tutorial will not walk over steps to configure kubernetes integration in Gitlab as its already well documented here.

AutoDevops Helm Chart setup for canary-deployments

This chart is the modified form of official auto-deploy-app that is intended to achieve traffic routing in canary deployment using nginx-ingress.

Assuming you have already configured your Gitlab project with AutoDevops, next is to use the modified chart that I created here -> https://gitlab.com/hayderimran7/auto-deploy-canary-chart
AutoDevops is completely customizable, so in order to use this chart instead of official chart, all you need to do is copy the files and move under chart directory in your repo.

Next, you want to enable canary stage which can be done simply in your .gitlab-ci.yml as:

include:
  - template: Auto-DevOps.gitlab-ci.yml
variables:
  CANARY_ENABLED: "true"

Traffic Routing in Auto Deploy canary deployments using Nginx-ingress

The traffic routing in canary deployment is done based on header value while using nginx-ingress and AutoDevOps.

How it works

We used canary by header value feature of nginx-ingress annotations https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary.

For that we had to modify official chart https://gitlab.com/gitlab-org/charts/auto-deploy-app to add two additional resources that are created during canary stage:

canary-ingress
canary-service

The canary-ingress checks for header canary, if set, routes to canary service backend that then forwards to canary-deployment.

There was a bug auto-deploy-app where the production service was pointing to both production deployment and production-canary deployment based on its selectors, which is raised here https://gitlab.com/gitlab-org/charts/auto-deploy-app/issues/51.

Testing in AutoDevops

Simply set CANARY_ENABLED in your .gitlab-ci.yml when using AutoDevops. Deploy the app using this chart inproductionand then tocanary`.

Now make a request to service URL as:
curl -H "canary: always" http://<service-url>
This will hit the canary-ingress that will route to canary-deployment that you can verfiy in pod logs.