DEV Community: Utkarsh Shigihalli

Use a single repository for multiple wikis in Azure DevOps

Utkarsh Shigihalli — Thu, 07 Dec 2023 00:00:00 +0000

This is a quick post to share how you can use a single Azure DevOps repository for multiple wikis in Azure DevOps. This approach will help you avoid creating multiple repositories for each wiki and manage them in a single repository.

Create a new repo for wiki

Firs lets create a separate repository for the wiki.

Create a repository

I want to create two wiki for two teams in my project. So I have created subfolders for each team in the repository and have placed sample markdown files in each folder. The idea is that each team will place their wiki content in their respective folder.

Folder structure

Publish code as a new wiki

Now let us create a new wiki in Azure DevOps. Go to the project settings and click on the Wiki tab. We will publish a new code wiki.

Publish code as wiki

Do the same for the second wiki.

Publish code as wiki

This will create two wikis in the project.

Configure the wiki

As you can see that was easy. However, at it stands now, both the wikis are open for edits from all the team members. We need to restrict the access to the wiki content based on the team. We will make use of branch policies to achieve this.

Set the branch policy

Now, when any time changes are made to the wiki under 01-team-infra/* folder and a PR is created, Infra team will automatically be included as reviewers. Similarly, for the 02-team-app/* folder, App team will be included as reviewers.

Pull request showing automatic reviewers added

That is it for this post. You can now use a single Azure DevOps repository for multiple wikis in Azure DevOps. As mentioned above, this approach will help you avoid creating multiple repositories for each wiki and manage them in a single repository.

Hope you enjoyed reading this.

.env Manager — Quick way to create Environment Variables

Utkarsh Shigihalli — Thu, 30 Jun 2022 22:40:30 +0000

Do you use .env files to manage your environment variables in your local dev environment? .env Manager is a small VSCode extension to quickly add values to .env file, either selecting a text or running the command.

https://marketplace.visualstudio.com/items?itemName=onlyutkarsh.envmanager

Checkout the quick demo.

Usage

The extension exposes a command called .env Manager: Add to .env. The command is available in VS Code "Command Palette" which you can also access using shortcut Cmd + Shift + P on Mac.

Alternatively you can also right click on any active editor and select .env Manager: Add to .env from the context menu.

If the folder does not contain .env file at the root, the extension attempts to create it before adding the line.

Make sure you backup your .env file if you have very important secrets in the .env file.

Never commit your .env file in to source control.

Trigger a Netlify build every day using GitHub Actions

Utkarsh Shigihalli — Sun, 20 Feb 2022 00:00:00 +0000

I host this blog on Netlify and source control is managed using GitHub. Often, I end up writing few blog posts together, but not necessarily want all of them published together. Jekyll allows to put future date in the post and it will not be published until that date. This lets me write blog posts on same day but end up publishing them later based on the date set for the post. However, this means that when you run jekyll build posts with future dates will be skipped and I will need to run jekyll build again on the post date to publish the blog posts. Same applies to Netlify. Curious how we can solve this by triggering scheduled deploys on Netlify?

Jekyll skipping posts with future date

Create Netlify build hook

Netlify supports build hooks, which lets you trigger new builds and deploys by making a request to a URL.

Configure build hooks

Go to Site Settings and Build Hooks section and click Add Build Hook button.

Add build hook

Give a name and select a branch to trigger the build and click Save. You will see the URL which you can use to trigger the build.

URL of the build hook

Create GitHub Actions workflow

Go to GitHub repo and create a actions yaml file under .github\workflows folder. GitHub Actions supports scheduled jobs, which lets you run a job at a specific time. See the below example. I am using cron expression to run the build every day at 04:00 AM UTC timezone.

name: nightly-netlify-build

on:
  schedule:
    - cron: "1 4 * * *"

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: trigger netlify build
        run: |
          curl -X POST -d '{}' https://api.netlify.com/build_hooks/2eae8d5c79506d0213a38be0e

Save the file and push the changes. That is it, you have created a GitHub Actions workflow which can trigger nightly build of the blog. Go to Netlify and see the Deploys tab. You will see the deploys triggered by the build hook.

Deploys triggered by the build hook

Isn’t that cool? If you liked the post, do share it in your social channels. Thanks for reading.

Policy enforced deployments for your Kubernetes resources

Utkarsh Shigihalli — Sun, 13 Feb 2022 00:00:00 +0000

As your team starts to deploy resources to Kubernetes regularly, it becomes necessary for you as a cluster administrator to maintain good standards and consistency of the Kubernetes resources. Be it, ensuring all the resources have set of labels, or ensuring you only pull images from your enterprise container registry. Gatekeeper is a well known policy enforcement tool using Open Policy Agent (OPA) - which is a opensource, Cloud Native Computing Foundation (CNCF) project.

However, Gatekeeper is installed on the cluster and thus ensures no policy is broken at deployment time. This means that any validation of policies happen only when you are trying to deploy resources to cluster. While this ensures that no resource violates the policy, you would like to know about these policies much earlier in your CI/CD pipeline. Doing policy validations much to the left of your deployment pipeline ensures your deployment going smooth when necessary.

This is where Conftest helps. Conftest relies on OPA and policies are written using Rego - thus the policies you write for Gatekeeper will be compatible with Conftest. But more importantly with Conftest, you can validate your local manifests against OPA policies locally and ensure your resources are compliant before you deploy them.

Installation

Installation is really easy if you are on Mac - For other platforms refer to the documentation

brew install conftest

Folder structure

By default, Conftest expects you to maintain your policies under policy folder at the same location as your Kubernetes resources. If you prefer a different path, you will want to pass it using CLI or set environment variable CONFTEST_POLICY.

📂 src
    📂 k8s
        📄 deployment.yml
        📄 service.yml
        📁 policy
            📄 replica.rego
            📄 labels.rego
    📂 app
        📄 main.ts
        📄 package.json

Writing Policies

As mentioned previously, policies are written in Rego. I struggled to write policies initially and constantly went back to documentation. However, once you write couple of policies, you will get a hang of it. Take a look at the simple policy to check every deployment has at least 2 replicas.

package main

deny_replicas[msg] {
    input.kind == "Deployment" # check if it is a Deployment
    input.spec.replicas < 2 # And the replicas are < 2
    msg := "Deployments must have 2 or more replicas" # show the error message and fail the test
}

input is the complete yaml document from our deployment yaml (see below) and we we are checking if kind is equal to Deployment. If its deployment, we move to next line in the constraint and check if spec.replicas is less that 2.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mynodeapi-dep
  labels:
    app: dep-k8s-nodejs-api
spec:
  replicas: 1
  selector:
    ...

Lets see how we can test our Deployment resource using Conftest.

Testing

The command to test resources using Conftest is conftest test <PATH>. Since I would like to test all resources under k8s folder, I pass folder path as below.

conftest test ./k8s

Running this you will see the output as below (ignore other errors as I have other policies). The test failed because we have set deny rule if spect.replicas < 2 and in our case our deployment yaml has replicas: 1 (see spec section in the deployment yaml above).

Conftest failing due to policy violation

Using Conftest in GitHub Actions

Making Conftest work in your Continuous Integration (CI) process is simple. For demo purposes, I am using GitHub Actions in my repo here. If you run the tests, you will see the action fails with errors - see the output

Conclusion

As you can see, Conftest lets you validate and govern your Kubernetes resources efficiently and can easily be integrated with your CI workflows. This lets your team standardise the common practices, go through PR review process before eventually deploying to the cluster. Once deployed to cluster, you can use Gatekeeper to validate as well to full proof your workloads.

Keep your workflow actions up to date using GitHub Dependabot

Utkarsh Shigihalli — Fri, 28 Jan 2022 00:00:00 +0000

GitHub Actions is great in automating your workflows. However, as you start using various actions from GitHub Marketplace in your workflow, it will soon become necessary for you to keep the actions up-to-date. Actions might contain security fixes, bug fixes etc and manually keeping track of updates or updating them when a newer version is available is a lot of hassle. This is where we can use Depndabot can help is by automatically raising PR’s whenever there is a newer version of action is available used in the workflow. In this post, we will see quick way to keep the actions up-to-date using GitHub Dependabot.

For this post, I am using my Git Config User Profiles repository. I have workflow setup which builds and releases the VS Code extension to VS Marketplace.

Create dependabot.yml file

To set up Dependabot scan, first got to .github folder in your root and create a depndabot.yml file. Then add the following content. This will ensure GitHub Dependabot raise a PR whenever there is a newer version of action is available

version: 2
updates:
  - package-ecosystem: "github-actions" # search for actions - there are other options available
    directory: "/" # search in .github/workflows under root `/`
    schedule:
      interval: "weekly" # check for action update every week

Commit the file

Commit the file created above and wait for few seconds. Based on your workflow, you will see a bunch of PR’s raised.

Dependabot Alerts as PR

If you look at the PR, you will be able to see the change and take a decision whether you want to upgrade the specific action or not. If you decide to accept the change, merge the PR and the changes on the workflow file will be made.

Commit Details

Conclusion

Isn’t it cool? This saves a lot of time, if you have a number of workflows and don’t want to keep checking the latest versions of actions. BTW, not only GitHub actions, you can use same approach to update npm, docker and many more using various package ecosystems. Do check it out!

Setting up Azure Cosmos DB Emulator on Synology NAS

Utkarsh Shigihalli — Mon, 26 Jul 2021 23:00:00 +0000

Cosmos DB Emulator is great for developing against Azure Cosmos DB in your local environment. If you want to run the emulator on Mac/Linux though, the emulator is now in preview mode at the time of writing this and uses Docker to make it available.

Recently I wanted to setup Cosmos DB Emulator on Mac, but I did not want to set it up such that I keep it running always on my Mac. Instead, I decided to use my Synology NAS to host the emulator directly on my NAS as it is built on Linux and also because it allows the emulator running and available for me all the time.

In this post we will see how to set it up on Synology NAS.

The goal is to run Cosmos DB emulator as a Docker container on Synology.

Enabling SSH on Synology

Since Cosmos DB Emulator image is hosted on Microsoft Registry (mcr.microsoft.com) which does not provide UI/discovery service, Synology’s Docker app, cannot pull images from Microsoft Registry. For this purpose we need to enable SSH on Synology so that we can SSH to Synology and run docker commands directly.

If you have firewall enabled on Synology, you must also Allow port 22.

Pulling the docker image

After enabling SSH on SYnology, you can SSH on to Synology and run the below command to pull the Docker image of Cosmos Emulator.

sudo docker pull mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator

You should see Cosmos DB Emulator image available in Syonology’s Docker app.

Starting the Container

You can now create a container using the docker image you just pulled. Before we start, as per the Microsoft documentation for the Emulator, we need to ensure we have setup few environment variables for the container. They are

AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE = <<SYNOLOGY_IP_ADDRESS>>
AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE = true
AZURE_COSMOS_EMULATOR_PARTITION_COUNT = 10

Finally, you will need to ensure you have mapped these ports for the container. Ensure you have opened these below ports on the Synology’s Firewall as well

For more information on creating container from docker image, see Synology’s documentation.

Install the certificate on your machine

To avoid errors due to certificate, you need to download the self-signed certificate for the emulator and install it on your machine. To download the certificate for the emulator run below command

curl -k https://<<SYNOLOGY_IP_ADDRESS>>:8081/_explorer/emulator.pem > emulatorcert.crt

On my Mac, I had to install it on Keychain Access App and set it as Always Trust.

Once done you should be able to browse the Cosmos Emulator UI using https://<<SYNOLOGY_IP_ADDRESS>>:8081/_explorer/index.html

That is it! Your Cosmos Emulator is now available to use all the time.

Reference

How to publish HELM 3 charts to GitHub Container Registry

Utkarsh Shigihalli — Fri, 09 Apr 2021 23:00:00 +0000

I have already written how to publish HELM chart to ACR using Azure DevOps and GitHub actions. But did you know you can also publish HELM3 charts (or any OCI compliant package) to GitHub Container Registry(GCR)? In this post we will see how to do that.

Enable improved container support

GitHub Container Registry is currently in public beta. So, the first step is to ensure that you have enabled the GitHub Container Registry for your account. If you have GitHub personal account, you can do that from Feature Preview window.

If you have a Enterprise account, you can do that going to Settings page.

GitHub documentation has step-by-step guide of enabling improved container support here

Publishing HELM 3 charts using GitHub Actions

It really takes only couple of steps to do it using GitHub Actions. Like any other action, you start by creating .github\workflow folder and create an yml file in your repository.

Excluding the name and trigger part, first step in the YAML is to define few necessary variables.

env:
  HELM_EXPERIMENTAL_OCI: 1 #enable OCI support
  HELM_VERSION_TO_INSTALL: 3.5.0 # version of HEL to install
  GCR_IMAGE: ghcr.io/$/vote-app

First variable enables the OCI support for the HELM commands we are going to run later in the YAML.

Next variable HELM_VERSION_TO_INSTALL is used later in the workflow to install specific version of the HELM. For this workflow, we need 3.5.0.

Last variable GCR_IMAGE is constructing the chart name for the publication.

The GitHub Container Registry hosts containers at ghcr.io/OWNER/IMAGE-NAME. I get the OWNER of the repository using github.repository_owner from github context.

Next steps are defining the job with steps to download the code (where we have HELM chart) and install the specific HELM tool on the runner/agent.

jobs:
  build:
    name: publish gcr
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - uses: actions/checkout@v2
        name: checkout repo

      - name: install helm
        uses: Azure/setup-helm@v1
        with:
          # Version of helm
          version: ${{ env.HELM_VERSION_TO_INSTALL }} # default is latest

The above steps setup the agent machine with the required HELM tool.

Next, we need to run few helm commands to login to GCR (GitHub Container Registry) and finally publish the chart.

We login to GCR using ${{ secrets.GITHUB_TOKEN }}

GitHub Container Registry only recently started supporting GITHUB_TOKEN. Previously you had to create a separate PAT token with specific permissions to GCR.

Next two steps in the workflow will be to save the chart and publish. We do that using helm chart save and help chart push commands as shown below.

- name: login to acr using helm
  run: |
    echo ${{ secrets.GITHUB_TOKEN }} | helm registry login ${{ env.GCR_IMAGE }} --username ${{ github.repository_owner }} --password-stdin

- name: save helm chart to local registry
  run: |
    helm chart save ${{ github.workspace }}/src/azure-vote-helm-chart/ ${{ env.GCR_IMAGE }}:${{ github.sha }}

- name: publish chart to acr
  run: |
    helm chart push ${{ env.GCR_IMAGE }}:${{ github.sha }}

That is it, if all worked successfully for you, you should see the chart in the GCR.

Publish HELM 3 charts to Azure Container Registry (ACR) using GitHub Actions

Utkarsh Shigihalli — Thu, 01 Apr 2021 00:00:00 +0000

In my previous post, we briefly covered how to publish a HELM chart to ACR using Azure DevOps. In this post we will use GitHub actions to build and publish HELM chart to ACR using GitHub Actions. We will also take a sneak peak how GitHub environments work.

Pre-requisites

I am going to assume ACR instance is setup using repository scoped tokens. Since we already covered setting up of ACR this way in the earlier post, I will not include the steps here.

Setting up secrets at GitHub

We would like store Azure Container Registry’s tokens as GitHub repository level secrets. To do that, click on Settings on the repository page and head to Secrets tab. Finally click on New repository secret and add the token name and the password. I have stored token name as ACR_PUSH_USER and token password as ACR_PUSH_TOKEN.

Add repository secrets

Creating the workflow in GitHub Actions

Publish chart to ACR

The first step is to create an yaml file under .github\workflows folder and setup a basic structure. The first things (see the yaml below) are defining name for the action, currently set to trigger via manual trigger using workflow_dispatch and define few environment variables which we are going to use later in the action.

name: ci

on: 
  workflow_dispatch:

env:
  HELM_EXPERIMENTAL_OCI: 1
  HELM_VERSION_TO_INSTALL: 3.5.0
  ACR_NAME: acrdemoutkarsh
  ACR_REPO_NAME: helmdemo/vote-app

The first environment variable conveys to ACR that we are going to publish a OCI package. Next couple of variables just define version of HELM we need on the runner, our ACR name to which we are going to publish this chart and finally to the repository we are publishing this chart to (used in below sections).

Installing Helm 3 on the agent

Now that we have all the variables defined, we need add jobs and steps to build our workflow to publish charts to ACR. We then need to install HELM tool on the agent before we can run the HELM commands. We do that using yaml below.

jobs:
  build:
    name: publish acr
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - uses: actions/checkout@v2
        name: checkout repo

      - name: install helm
        uses: Azure/setup-helm@v1
        with:
          version: ${{ env.HELM_VERSION_TO_INSTALL }}# default is latest

As you can see, we have one job named build (which will be displayed as publish acr - see screenshot below) which runs on ubuntu-latest agent. We also are targeting our deployment to an environment prod. Environments in GitHub are cool because you can have approvers, additional protection rules for environments and environment specific secrets. In the screenshot below, notice how the flow is waiting for review.

Next, we checkout the repository and using setup-helm task from Azure repo we install the specific version (3.5.0) of HELM.

Login to the ACR using Helm

Next, we need to login to ACR registry using HELM tool.

- name: login to acr using helm
  run: |
    echo $ | helm registry login $.azurecr.io --username $ --password-stdin

Save and push the chart to ACR

Next we need to save the chart directory to local cache and publish it to ACR.

- name: save helm chart to local registry
  run: |
    helm chart save $/src/azure-vote-helm-chart/ $.azurecr.io/$:latest

- name: publish chart to acr
  run: |
    helm chart push $.azurecr.io/$:latest

Run the workflow, and you will see output as below.

Go to ACR and you will see char correctly published to helmdemo/vote-app repository as declared in the env section above.

Conclusion

In this post, you saw how easily we can deploy a OCI package (helm3 chart) to ACR using GitHub actions. We also saw how GitHub environments help you approve changes to the environment. Hope you enjoyed reading this post.

Helm 3 - CI/CD with Azure DevOps using Azure Container Registry (ACR) and Azure Kubernetes Service (AKS)

Utkarsh Shigihalli — Sat, 30 Jan 2021 00:00:00 +0000

I have been bit late to Kubernetes world, but ever since I have started using it, I have been part of a great teams building awesome applications and using Helm package manager. With Helm, you package your Kubernetes application as charts, which are then stored in Helm chart repo. Helm also has a templating engine allowing you to set values in your charts dynamically allowing you to manage your applications more easily. Azure Container Registry (ACR) currently supports publishing Helm 3 charts to ACR and it is currently in preview.

In this post we will see how we can publish a sample Helm chart to ACR and also deploy the application to Azure Kubernetes Service (AKS) by consuming the published chart from ACR. We will also use ACR’s repository scoped tokens - a preview feature which offer great benefits.

We will cover following things in this post.

1) Prerequisites

ACR
- Setting up the ACR to use repository scoped tokens
- Create scope-maps
- Create a token
AKS
- Create AKS instance on Azure

For CI and CD, I am going to use Azure DevOps YAML multi-stage pipelines in this post and our pipeline will be divided as below.

2) CI Stage

Get the latest chart from GitHub
Publish to ACR

3) CD Stage

Pull the latest version of chart from ACR
Deploy the pulled to AKS
View the deployed service in Azure DevOps

The code I am using is on

1. Prerequisites

You can authenticate with ACR in multiple ways.

Using a service principal - You can create separate service principals. This gives access to whole ACR and all the repositories inside that ACR instance. More info here
Using managed identity - Using user-assigned or system-assigned managed identity for easily consuming charts on an Azure VM. More info here
Using image pull secret - For your Kubernetes (K8s) cluster (including unmanaged AKS instance), you can also define a image pull secret, which then lets K8s cluster do pull images and run the application. More info on how you can do this with AKS is here
Using a repository scoped token - This feature of ACR is currently in preview and only available on Premium container registry service tier. This allows you to define scope maps and repository specific tokens for your ACR.

Setting up the ACR to use repository scoped tokens

I am going to use Tokens to authenticate ourselves with ACR. The biggest advantage is that, as a registry owner, you can have a single instance of ACR across your organisation and provide access to certain teams only selected repositories. There are other interesting scenarios highlighted in the documentation, mainly…

Allow IoT devices with individual tokens to pull an image from a repository
Provide an external organisation with permissions to a specific repository
Limit repository access to different user groups in your organisation. For example, provide write and read access to developers who build images that target specific repositories, and read access to teams that deploy from those repositories.

If you have an existing ACR instance (which in my case I do), the first step is to upgrade your ACR instance to use Premium tier.

Then select Premium and click Save.

Creating scope maps

Once, you are on Premium tier, you will be able to to create a tokens. But before we can create tokens, let us define two scope maps. Scope maps when associated with tokens sets permissions on what can be done on ACR.

chartpull - a scope limiting read permission to helmdemo/vote-app repo. I am using content/read and metadata/read permissions.
chartpush - a scope limiting write permission inside helmdemo/vote-app repo. I am using content/write and content/read permissions.

The idea here is that, development team pushes the charts to ACR and some other team is consuming the charts/application. But you can extend this idea to let users external to your organisation to pull images only from specific repositories.

So to create scope map, go to Scope maps under Repository permissions sections and then click + Add. In my case, I am limiting these scopes only to helmdemo repo.

I follow the same process to create another scope named chartpull scope with content/read, metadata/read scopes.

Notice that in chartpush scope we also allow content/read permission. This is because, you need content/read scope along with content/write scope if you are going to push charts to ACR (for the repo defined in the scope).

You can also use Azure CLI to do the same. Use az acr token command. Refer the documentation for more information.

Creating token

Once you defined scopes, you are ready to create tokens. Go to Tokens in Repository permissions section and click + Add. Then name the token (e.g. helmdemopull) and select the scope map you created above. In my case I am creating two tokens, one for pull and another one for push.

You can repeat the process to create another token. Refreshing the screen you will see something like this below.

You will notice that these tokens (helmdemopull and helmdemopush) do not yet have passwords generated. Lets create passwords. Click on the token, click the icon under Actions column for either password1 or password2. You will also have an option to set an expiration date for the password if you planning to allow token to be valid only for specific days.

2. CI Stage

Get the latest chart from GitHub

To get the latest source from the GitHub repo, in our YAML pipeline we add a resources section and point to the repo on GitHub. We will also need to pass the endpoint parameter with the name of the GitHub service connection. More info about how to do this is here and here.

resources:
 repositories:
   - repository: helmrepo
     type: github
     name: utkarshpoc/azure-vote-helm-chart
     endpoint: github

Publish chart to ACR

Before we write steps to publish to ACR, we need to ensure we store the tokens we generated in the pipeline as variables. Here I am adding tokens as variables using the variables UI of the pipeline and marking passwords as secret.

You can also define these in Azure Pipelines Library if you intend to use these tokens in multiple pipelines.

I also have few static and non-secret variables in YAML itself so that they are source controlled using variables as below.

variables:
  acr.name: acrdemoutkarsh
  acr.repo.name: helmdemo/vote-app

The steps involved for publishing our Helm chart are simple.

Installing Helm 3 on the agent
Login to the ACR using Helm
Save and push the chart

1. Installing Helm 3 on the agent

You can do this using OOB HelmInstaller task. The YAML is as below. Here I am using latest for helmVersion, but you can stick specific to a version (3.x and above).

- task: HelmInstaller@0
  displayName: install helm
  inputs:
    helmVersion: 'latest'
    installKubectl: false

2. Login to the ACR using Helm

We can use simple script step to login to the registry.

- script: |
    helm registry login $(acr.name).azurecr.io --username $(acr.push.username) --password $(acr.push.password)
  displayName: login to acr using helm

But if you run this, at the time of writing you will get an error as below

Error: this feature has been marked as experimental and is not enabled by default. 
Please set HELM_EXPERIMENTAL_OCI=1 in your environment to use this feature

As the error states, this is because, publishing Helm charts which follow Open Container Initiative (OCI) standard is experimental and you are required to set HELM_EXPERIMENTAL_OCI to 1. We can do this by updating our variables section in the YAML. The updated yaml looks as below.

variables:
  acr.name: acrdemoutkarsh
  acr.repo.name: helmdemo/vote-app
  HELM_EXPERIMENTAL_OCI: 1

3. Save and push the chart to ACR

Next step is to save this chart locally and create an alias for the chart with our ACR registry URL. So our YAML looks as below.

- script: |
    helm chart save $(build.sourcesdirectory)/src/azure-vote-helm-chart/ $(acr.name).azurecr.io/$(acr.repo.name):latest
  displayName: save the chart and set the alias

And finally, we push the Helm chart to ACR using helm chart push command.

- script: |
    helm chart push $(acr.name).azurecr.io/$(acr.repo.name):latest
  displayName: push the chart to acr

So our CI stage is now complete.

2. CD Stage

Pull the latest version of chart from ACR

The first step is to pull the latest version of the chart (one with tag latest) to our agent machine and extract it in a folder. We can do that in pipeline as below. Notice I have added a new stage name cd and its dependent of stage ci.

- stage: cd
    displayName: CD
    dependsOn: ci
    jobs:
    - deployment: helm_publish_aks
      displayName: deploy to aks
      environment: 
        name: PROD
        resourceName: helmdemo
        resourceType: Kubernetes
      strategy:
        runOnce:
          deploy:
            steps:
            - task: HelmInstaller@0
              displayName: install helm
              inputs:
                helmVersion: 'latest'
                installKubectl: false

            - script: |
                echo "$(acr.pull.password)" | helm registry login $(acr.name).azurecr.io --username $(acr.pull.username) --password-stdin
              displayName: login to acr using helm

            - bash: |
                  helm chart pull $(acr.name).azurecr.io/$(acr.repo.name):latest
              displayName: get helm chart on agent

            - bash: |
                helm chart export $(acr.name).azurecr.io/$(acr.repo.name):latest --destination $(build.stagingdirectory)
              displayName: export the chart to folder

Because each job in Azure DevOps run in a separate agent, I have to ensure agent has Helm tool, so install Helm tool again in the first step. Also, note that I am using token with scope-map permission set only to pull charts.

Deploy the chart to AKS

Now that the latest chart is pulled from our ACR and is available on agent, only step remaining is to deploy to AKS. We can do that using HelmDeploy task.

- task: HelmDeploy@0
  displayName: deploy chart to aks
  inputs:
    connectionType: 'Azure Resource Manager'
    azureSubscription: '$(azure.service.connection)'
    azureResourceGroup: 'demos'
    kubernetesCluster: 'aksdemoutkarsh'
    namespace: 'helmdemo'
    command: 'upgrade'
    chartType: 'FilePath'
    chartPath: '$(build.stagingdirectory)/azure-vote/'
    releaseName: 'helmdemo'
    arguments: '--create-namespace --install'

View the deployed service in Azure DevOps

If you have added your Kubernetes cluster as a resource in Azure DevOps environment, you can now view deployed services directly from Azure DevOps.

You can also view deployed services in YAML and also see any ports exposed.

Browsing the exposed port, you will see application.

Conclusion

That is it! If you have read this far, thank you 🙏🏼. As we saw, with the support of tokens and OCI artifacts, ACR is one of the best in class container registry. Also, with deep integration with AKS (features like browsing services, logs, yaml), Azure DevOps brings in lots of productivity for your teams. Hope you found this post informative. If so, please share and tweet!

Quickly switch Kubernetes cluster and namespaces with kubectx and kubens

Utkarsh Shigihalli — Tue, 29 Dec 2020 00:00:00 +0000

Often when working with Kubernetes you might find yourself switching between your clusters or namespaces. I am doing this numerous times lately and I was slow switching using the regular commands. In this post, I would like to highlight two productivity utilities when working with Kubernetes which will make it simple when working with multiple clusters.

Cluster contexts and namespaces

Kubernetes stores the cluster information in ~/.kube/config file and stores in as a context information. If you are working against multiple clusters ideally you would have different config files and can reference them when running commands with kubectl.

For example, to get pods in different cluster than your default context, you use below command.

kubectl get pods --namespace dev-ns --kubeconfig ~/.kube/devClusterConfig

As you can see, typing namespace information and providing the config for every command is cumbersome.

Merging multiple config files in to one

If you already have multiple config files to connect to different clusters, you would like to merge them first. You can merge both the config file using syntax below

# Merge ~/.kube/config and ~/.kube/devClusterConfig in to new config /tmp/config
$ KUBECONFIG=~/.kube/config:~/.kube/devClusterConfig kubectl config view --flatten > /tmp/config

# Replace old config with new merged config
$ mv /tmp/config ~/.kube/config

Switching the Kubernetes clusters using kubectl commands

Kubernetes already has a concept of contexts and once you merge the config, you will be able to see the contexts information.

$ kubectl config get-contexts

You will see context information as below

CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* dev-aks-cluster dev-aks-cluster clusterUser_aksdemo_dev-aks-cluster dev-ns
          qa-aks-cluster qa-aks-cluster clusterUser_aksdemo_qa-aks-cluster

You can then switch to the different cluster (in my case its qa-aks-cluster) using below command

$ kubectl config use-context qa-aks-cluster

Switching the namespaces using kubectl commands

Now if you do not want to add --namespace <namespace-name> for every command you type, you can also set it as default namespace for the currently selected context.

kubectl config set-context --current --namespace=<insert-namespace-name-here>

Using kubectx and kubens

kubectx

kubectx is a great open-source tool to switch clusters. Just type kubectx and it lists the contexts you have. Then you can select one by typing

kubectx dev-aks-cluster

If you have fzf installed, you can also get a nice picker and you can use arrows to select one.

If you keep alternating between contexts, you can also use kubectx - to quickly go back to previous context. Isn’t it cool?

kubectx - Source: https://github.com/ahmetb/kubectx/

kubens

Similar to kubectx, kubens is another great utility from the same creator. It performs pretty much similar to kubectx but lets you switch namespaces. I regularly use this since we have couple of namespaces within our cluster which we deploy to. It has certainly saved me lots of typing.

kubens - Source: https://github.com/ahmetb/kubectx/

Conclusion

If you are into Kubernetes, both kubectx and kubectl are great productivity tools and will save you lots of time. I certainly am finding them really useful.

That’s it for this post, thanks for reading, until next time time 👋🏼

Deploying TIBCO Rendezvous on RedHat Enterprise Linux using Azure DevOps

Utkarsh Shigihalli — Mon, 28 Dec 2020 00:00:00 +0000

TIBCO Rendezvous (RDV) is one of the popular messaging product for real-time data processing. Predominantly used in Financial corporations to process real-time trading, market data and efficient information flow between control systems. In this post, we will see how we can deploy this messaging application to RedHat Enterprise Linux (RHEL) 7 (on Azure VM) using Azure DevOps.

Source: https://www.tibco.com/products/tibco-rendezvous

For this post, we are assuming that we have a RHEL VM provisioned on Azure to which we are going to deploy. Also, the installers are downloaded published to Azure Artifacts as an Universal Package.

Create SSH service connection in Azure DevOps

The first step is to ensure Azure DevOps can connect to our VM in Azure. For that, we set up a service connection in Azure DevOps. Since its Linux VM, we use SSH connection. Detailed steps of creating the SSH service connection are here

Service connection in Azure DevOps

Create deployment pipeline

To summarize, the steps for deployment are

Download and extract the installer from Azure Artifacts
Install required pre-requisites
Copy the installer to VM
Adjust the install path in TIBCO Rendezvous response file
Set the execute permission on the installer so that we can run the installer
Delete any existing instance of the RDV, so that we can trigger this pipeline again without any manual changes.
Install TIBCO Rendezvous
Add installer file to ~/.bash_profile
Set execute permissions and start the service.

1. Download and extract the installer from Azure Artifacts

We use Universal Package task from Azure DevOps.

steps:
- task: UniversalPackages@0
  displayName: download rdv package from artifacts
  inputs:
    command: 'download'
    downloadDirectory: '$(system.artifactsdirectory)'
    feedsToUse: 'internal'
    vstsFeed: 'Demo/my-artifacts'
    vstsFeedPackage: 'tibco-rdv'
    vstsPackageVersion: '8.4.5'

2. Install required pre-requisites

We then install any required pre-requisites needed on the machine. Here we are installing Java 1.8, GNU C Library. Notice that we are using SSH task so that these commands are run on the VM we are deploying to. We use the service connection created above and passing that as a parameter to this pipeline YAML. Azure DevOps has a powerful templating feature for YAML pipelines, and here I am using a template. So I am using ${{ }} syntax. More on this expression is here.

- task: SSH@0
  displayName: 'install prerequisites'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: |
        sudo yum -y install java-1.8.0-openjdk.x86_64
        sudo yum -y install glibc.x86_64
    readyTimeout: '20000'

3. Copy the installer to VM

Next we need to copy the installer files to VM. We again do that using CopyFilesOverSSH task in Azure DevOps.

- task: CopyFilesOverSSH@0
displayName: 'copy tibco rvd installer files to machine'
inputs:
  sshEndpoint: '$'
  sourceFolder: '$(system.artifactsdirectory)/out'
  contents: '**'
  targetFolder: '/data/tibco-rdv/'
  readyTimeout: '20000'

4. Adjust the install path in TIBCO Rendezvous response file

Since we are deploying TIBCO Rendezvous in our CD pipeline, we need to install using non-interactive (or silent) mode. TIBCO provides a response file (TIBCOUniversalInstaller-rv.silent) along with installer. This response file can be edited to select all the configurations (like Install path, home directory etc) for the installer. We are using simple SSH task and running few sed commands to replace the path according to our needs in the response file.

- task: SSH@0
displayName: 'adjust response silent file'
inputs:
  sshEndpoint: '$'
  runOptions: 'commands'
  commands: |
      sed -i 's/\/opt\/tibco/\/opt\/tibco\/rdv/g' /data/tibco-rdv/TIBCOUniversalInstaller-rv.silent
      sed -i 's/TIBCO-HOME/TIBCO-RDV-HOME/g' /data/tibco-rdv/TIBCOUniversalInstaller-rv.silent
  readyTimeout: '20000'

5. Set the execute permission on the installer so that we can run the installer

Before we run the installer, we need to ensure that the user running it (in our case the credentials used while creating the service connection) has execute permissions. So we set the execute permissions for the installer file.

- task: SSH@0
  displayName: 'set execute permissions'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: 'chmod a+x /data/tibco-rdv/TIBCOUniversalInstaller-lnx-x86.bin'
    readyTimeout: '20000'

6. Delete any existing instance of the RDV

Also, before we install the service, we need to remove any existing services running, This will also enable to run this pipeline again and again without manually configuring something in the VM.

- task: SSH@0
  displayName: 'delete previous instance'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: |
        pkill 'rvd64'
        sudo rm -rf /opt/tibco/rdv/*
    readyTimeout: '20000'

7. Install TIBCO Rendezvous

Next, we start the installation by executing installer file.

- task: SSH@0
  displayName: 'install tibco rdv'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: 'sudo /data/tibco-rdv/TIBCOUniversalInstaller-lnx-x86.bin -silent -V responseFile=/data/tibco-rdv/TIBCOUniversalInstaller-rv.silent -is:javahome $(dirname $(dirname $(readlink $(readlink $(which java))))) -is:log "$(build.buildnumber).log"'
    readyTimeout: '20000'

Notice we pass the modified response file to the installer so that it can without any manual interaction.

8. Add installer file to `~/.bash_profile`

Next, we need to add install path to ~/.bash_profile so that its available from anywhere.

- task: SSH@0
  displayName: 'update bash profile'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: |
        cp ~/.bash_profile ~/.bash_profile.tmp
        echo 'export PATH=$PATH:/opt/tibco/rdv/tibrv/8.4/bin' >> ~/.bash_profile
        echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/tibco/rdv/tibrv/8.4/lib' >> ~/.bash_profile
    readyTimeout: '20000'

In the above step, I am just backing up existing ~/.bash_profile file. A better way probably is to find and replace the values (to avoid duplicates) which I am planning to do in the future

9. Set execute permissions and start the service.

Final step is to start the Rendezvous service itself. Before that we need to set the execute permissions for the file. So our YAML looks as below.

- task: SSH@0
  displayName: 'set execute permissions'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: 'sudo chmod a+x /opt/tibco/rdv/tibrv/8.4/bin/rvd64'
    readyTimeout: '20000'

- task: SSH@0
  displayName: start the rvd service
  timeoutInMinutes: 2
  continueOnError: true
  inputs:
    sshEndpoint: '$'
    runOptions: 'inline'
    inline: |
        /opt/tibco/rdv/tibrv/8.4/bin/rvd64 -listen 8500
        sleep 70
        exit
    readyTimeout: '20000'

Notice in the last step, we have enabled continueOnError: true. For some reason, when you start the service, its not ending the console. There must be an option in the response file to properly start the service, but I am yet to figure it out.

Final YAML template

# template file to deploy tibco rendezvous

parameters:
  - name: sshEndPoint
    type: string
    default: ''

steps:
- task: UniversalPackages@0
  displayName: download rdv package from artifacts
  inputs:
    command: 'download'
    downloadDirectory: '$(system.artifactsdirectory)'
    feedsToUse: 'internal'
    vstsFeed: 'Demo/my-artifacts'
    vstsFeedPackage: 'tibco-rdv'
    vstsPackageVersion: '8.4.5'

- script: |
    unzip -oq $(system.artifactsdirectory)/TIB_rv_8.4.5_linux_x86.zip -d $(system.artifactsdirectory)/out
  displayName: 'extract files'

- task: SSH@0
  displayName: 'install prerequisites'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: |
        sudo yum -y install java-1.8.0-openjdk.x86_64
        sudo yum -y install glibc.x86_64
    readyTimeout: '20000'

- task: CopyFilesOverSSH@0
  displayName: 'copy tibco rvd installer files to machine'
  inputs:
    sshEndpoint: '$'
    sourceFolder: '$(system.artifactsdirectory)/out'
    contents: '**'
    targetFolder: '/data/tibco-rdv/'
    readyTimeout: '20000'

- task: SSH@0
  displayName: 'adjust response silent file'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: |
        sed -i 's/\/opt\/tibco/\/opt\/tibco\/rdv/g' /data/tibco-rdv/TIBCOUniversalInstaller-rv.silent
        sed -i 's/TIBCO-HOME/TIBCO-RDV-HOME/g' /data/tibco-rdv/TIBCOUniversalInstaller-rv.silent
    readyTimeout: '20000'                

- task: SSH@0
  displayName: 'set execute permissions'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: 'chmod a+x /data/tibco-rdv/TIBCOUniversalInstaller-lnx-x86.bin'
    readyTimeout: '20000'

- task: SSH@0
  displayName: 'delete previous instance'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: |
        pkill 'rvd64'
        sudo rm -rf /opt/tibco/rdv/*
    readyTimeout: '20000'

- task: SSH@0
  displayName: 'install tibco rdv'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: 'sudo /data/tibco-rdv/TIBCOUniversalInstaller-lnx-x86.bin -silent -V responseFile=/data/tibco-rdv/TIBCOUniversalInstaller-rv.silent -is:javahome $(dirname $(dirname $(readlink $(readlink $(which java))))) -is:log "$(build.buildnumber).log"'
    readyTimeout: '20000'        

- task: SSH@0
  displayName: 'update bash profile'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: |
        cp ~/.bash_profile ~/.bash_profile.tmp
        echo 'export PATH=$PATH:/opt/tibco/rdv/tibrv/8.4/bin' >> ~/.bash_profile
        echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/tibco/rdv/tibrv/8.4/lib' >> ~/.bash_profile
    readyTimeout: '20000'

- task: SSH@0
  displayName: 'set execute permissions'
  inputs:
    sshEndpoint: '$'
    runOptions: 'commands'
    commands: 'sudo chmod a+x /opt/tibco/rdv/tibrv/8.4/bin/rvd64'
    readyTimeout: '20000'

- task: SSH@0
  displayName: start the rvd service
  timeoutInMinutes: 2
  continueOnError: true
  inputs:
    sshEndpoint: '$'
    runOptions: 'inline'
    inline: |
        /opt/tibco/rdv/tibrv/8.4/bin/rvd64 -listen 8500
        sleep 70
        exit
    readyTimeout: '20000'

Conclusion

That’s it, you now know how we implemented Azure DevOps pipeline to deploy TIBCO Rendezvous 🥳

Publish VSCode extension to VS Marketplace using simplified ChatOps

Utkarsh Shigihalli — Mon, 10 Aug 2020 23:00:00 +0000

I have previously blogged about publishing VSCode extensions to VSMarketplace (using GitHub actions). However, that workflow relied heavily on the branching strategy and was on assumption that anything that is merged to master is ready for publishing to VSMarketplace. It worked great, but occasionally I felt the need to test the extension from master and I wanted a way to approve before I decide to publish to marketplace. In this post we will see how I used ChatOps with the help of GitHub actions and Issues to achieve the same.

How will it work?

If you are short on time, the workflow I have implemented is as below.

User commits the code in a feature branch
Commit automatically triggers CI workflow build.yml
- If trigger is from feature/* branch, it produces the artifact (.vsix file) and ends the workflow. This allows me to download and test the vsix file.
- If trigger is from master branch, it creates a GitHub issue to monitor for dispatch comments (/deploy and /reject).
Workflow dispatch.yml monitors issue comments for dispatch commands.
I create a issue comment and type text /deploy or /reject to deploy or reject a release to publish it to VS Marketplace.
If /deploy command is detected, workflow deploy.yml is triggered.
If command is instead /reject, the release is rejected and issue is marked that it can be closed.

Curious on how each step is implemented? Read on.

Triggering CI build when user commits

The goal of CI is that, with each commit, we can produce shippable .vsix (packaged extension) file as an artifact for our extension. So we additional triggers in our build.yml file.

on:
  workflow_dispatch:
    inputs:
      action:
        description: Trigger build
        required: true
        default: "ci"
  pull_request:
    branches:
      - master
  pull_request_review:
    branches:
      - master
    types:
      - submitted
  push:
    branches:
      - feature/** # match an pushes on feature/* and feature/<any sub branch>/*
      - master
    paths-ignore: # don't run when changes made to these folders
      - ".vscode/**"

workflow_dispatch trigger lets us manually trigger the workflow.

pull_request and pull_request_review triggers allow us to trigger this workflow on PRs
push lets our workflow to trigger on actual commits on master and feature/** branches.

Creating issue for dispatch events

As part of CI, once we produce artifact, we create the issue so that we can monitor issue for dispatch commands. I have a separate job create-issue in build.yml which creates the issue only if CI is running against master branch.

You can take a look at the job here

The output of the job will open an issue something like this (screenshot is from previous release).

Monitoring dispatch events in issue comments

The required issue is created and now is the time to monitor for issue comments with specific commands we defined - /dispatch and /reject.

We can do this with issue_comment trigger as below.

on:
  issue_comment:
    types:
      - created

With the above trigger our dispatch.yml workflow triggers whenever a comment is added on our issue. To dispatch the events I am using action from Peter Evans as below.

- name: dispatch command
  if: github.repository_owner == github.event.comment.user.login
  uses: peter-evans/slash-command-dispatch@v2
  with:
    token: ${{ secrets.REPO_ACCESS_TOKEN }}
    reaction-token: ${{ secrets.REPO_ACCESS_TOKEN }}
    commands: deploy, reject

As you can see, in the last line (#7), I am supporting only deploy and reject commands.

I want to dispatch events, only if user commenting on the issue is owner of the repository (which is me). This allows me to dispatch events only if I have typed the dispatch command and no on else.

Handling dispatch commands

Now, I want to deploy to marketplace, when I see /deploy, and reject the deployment when I see /reject in issue comment.

For this, I have two workflow files deploy.yml and reject.yml with trigger similar to below.

on:
  repository_dispatch:
    types:
      - deploy-command

So as soon as the user types /deploy command as a issue comment, the command is captured and dispatched by dispatch.yml workflow.

The dispatched command is captured by our deploy.yml or reject.yml which act on repository_dispatch event of either deploy or reject commands. All this work seamlessly with the help of slash-command-dispatch action and GitHub’s repository_dispatch events.

Our deploy workflow just publishes the extension to VS Marketplace and before doing this it also updates the comments to the issue which triggered the workflow.

Conclusion

There you have it! A quick and simple way to have a approval process in GitHub actions using ChatOps. I am confident that GitHub Actions will have OOB approval flows in Actions, but until that time, this is one way to achieve it. Do let me know if you know or have implemented similar things in a better way - I would love to learn about it.

DEV Community: Utkarsh Shigihalli

Use a single repository for multiple wikis in Azure DevOps

Create a new repo for wiki

Publish code as a new wiki

Configure the wiki

.env Manager — Quick way to create Environment Variables

Usage

Trigger a Netlify build every day using GitHub Actions

Create Netlify build hook

Create GitHub Actions workflow

Policy enforced deployments for your Kubernetes resources

Installation

Folder structure

Writing Policies

Testing

Using Conftest in GitHub Actions

Conclusion

Keep your workflow actions up to date using GitHub Dependabot

Create dependabot.yml file

Commit the file

Conclusion

Setting up Azure Cosmos DB Emulator on Synology NAS

Enabling SSH on Synology

Pulling the docker image

Starting the Container

Install the certificate on your machine

Reference

How to publish HELM 3 charts to GitHub Container Registry

Enable improved container support

Publishing HELM 3 charts using GitHub Actions

Publish HELM 3 charts to Azure Container Registry (ACR) using GitHub Actions

Pre-requisites

Setting up secrets at GitHub

Creating the workflow in GitHub Actions

Publish chart to ACR

Installing Helm 3 on the agent

Login to the ACR using Helm

Save and push the chart to ACR

Conclusion

Helm 3 - CI/CD with Azure DevOps using Azure Container Registry (ACR) and Azure Kubernetes Service (AKS)

1. Prerequisites

Setting up the ACR to use repository scoped tokens

Creating scope maps

Creating token

2. CI Stage

Get the latest chart from GitHub

Publish chart to ACR

1. Installing Helm 3 on the agent

2. Login to the ACR using Helm

3. Save and push the chart to ACR

2. CD Stage

Pull the latest version of chart from ACR

Deploy the chart to AKS

View the deployed service in Azure DevOps

Conclusion

Quickly switch Kubernetes cluster and namespaces with kubectx and kubens

Merging multiple config files in to one

Switching the Kubernetes clusters using kubectl commands

Switching the namespaces using kubectl commands

Using kubectx and kubens

kubectx

kubens

Conclusion

Deploying TIBCO Rendezvous on RedHat Enterprise Linux using Azure DevOps

Create SSH service connection in Azure DevOps

Create deployment pipeline

1. Download and extract the installer from Azure Artifacts

2. Install required pre-requisites

3. Copy the installer to VM

4. Adjust the install path in TIBCO Rendezvous response file

5. Set the execute permission on the installer so that we can run the installer

6. Delete any existing instance of the RDV

7. Install TIBCO Rendezvous

8. Add installer file to ~/.bash_profile

9. Set execute permissions and start the service.

Final YAML template

Conclusion

Publish VSCode extension to VS Marketplace using simplified ChatOps

How will it work?

Triggering CI build when user commits

8. Add installer file to `~/.bash_profile`