DEV Community: Janne Mattila

Debugging microservices running in Azure Kubernetes Service (AKS) using Azure Dev Spaces

Janne Mattila — Tue, 10 Dec 2019 17:21:40 +0000

Introduction

Debugging microservices can be hard. Are you going to run all those services at your own development machine, or do you mock your dependencies? Wouldn't it be nice to just debug and work on single service and use other services directly from your Azure Kubernetes Service (AKS) cluster? Luckily, Azure Dev Spaces enables you to do just that.

In this blog post I'm going to show how you can try it yourself. I'll show one demo application which I have been showing to my customers when I've wanted to explain how dev spaces works and how can you use it to improve your workflows.

I recommend that you first check the official documentation pages how to install Azure Dev Spaces on AKS and how to get client-side tooling installed to your local dev machine. With those instructions you should now have working AKS cluster with dev spaces enabled namespace that we can use in this example.

Before we jump into my demo application, I want to still go through this diagram:

Above diagram tries to illustrate more complex scenario with dev spaces. You can use dev spaces to share different microservices between your team members.
In above example services A, B, C, D, E and F are baseline
versions of the services. In this example these are running in dev spaces
enabled namespace integration.

Those baseline services are then overridden in feature specific dev space
with services feature-C, feature-D and feature-E. These are running
in child dev space integration/feature.

Developer working in this feature has created another child dev space to
override one specific service with feature-D-dev and that is running
in child dev space integration/feature/dev. This means that developer working on this individual service can debug and iterate on the code much faster.

If you think that above would be helpful also in your development, then I recommend that you try it out.

Try this yourself

This example application is tiny-node-app and you can find it from GitHub:

JanneMattila / tiny-node-app

Demonstrates how you can debug your microservices apps running Azure Kubernetes Service using Azure Dev Spaces

It's simple application and it has 3 different services web, api and db:

web is frontend service which is exposed to internet and api and db are backend services and they are not exposed to internet. Call chain is simple and it goes directly as in above diagram: web makes http request to api service which then does next request to the db service. Information then flows back and web plots the data to the simple web view.

In order to deploy this to your own Dev Spaces enabled AKS you can run following commands. Note: This means that we now deploy our team shared integration namespace directly from our development machine but in real life you should use Azure DevOps or GitHub Actions to do that. Here are the steps:

git clone https://github.com/JanneMattila/tiny-node-app.git
cd tiny-node-app

# web
cd web
npm install
azds up --detach-after Listening

# db
cd ../db
npm install
azds up --detach-after Listening

# api
cd ../api
npm install
azds up --detach-after Listening

If you missed the url of the application (it was printed when web was started) then you can get it using azds:

azds list-uris

If you open that in your browser, you should see something like this:

Notice that you can see your namespace and service name directly in the url (in my example demospace and actual exposed service web).

If you then click on that people icon next to the title you'll see this:

You can right away see that table does not contain data like it should.
So now it's time to take our debugging gloves and try dev spaces out!

Let's start debugging from our api layer because it's hidden behind the web layer and it tries to fetch data from the backend. We want to achieve this kind of debugging experience:

Without dev spaces you would have to think how to debug this since db is running inside the cluster and it's not available from your local development machine. Similarly, you would have to think that how would you flow exact same request to your api from the web service.

Let's start by opening our Visual Studio Code workspace in Visual Studio Code from our repo root:

code tiny-node-app.code-workspace

This should enable required extensions (such as Azure Dev Spaces).

Now we have the code available and now we need to ask dev spaces to do the magic for us. We start that by using Visual Studio Command palette:

Let's use the "Redirect an existing Kubernetes service to my machine".

Next we need to select service that we want to redirect to our development machine. Choose api as we want to debug that specific service:

Next we need to select mode. In this case we want to debug our api service without impacting anybody else in our team so therefore we'll choose "Clone" as our mode:

Since we chose clone it now asks which child dev space this clone should be now created (remember the diagram earlier in this blog post showing that you can use this to create multiple child dev spaces). In my demo I'll use my own developer specific namespace:

Next it will ask port to which the incoming traffic should be redirected:

Choose 3001 to be our local service port (you'll see soon why):

After the selections dev spaces prepares the connections and then shows at the toolbar that it has successfully connected to that service:

Note: Dev spaces might request that you let it modify your hosts file so that it's able to connect back to the other services such as db in our demo scenario.

Here's small snipped from my hosts file (C:\Windows\System32\drivers\etc\hosts) after dev spaces has modified it:

# Added by Azure DevSpaces
127.1.1.8 kubernetes.default.svc.cluster.local
127.1.1.8 kubernetes.default
127.1.1.8 kubernetes
# ...
127.1.1.2 db.demospace.svc.cluster.local
127.1.1.2 db.demospace
127.1.1.2 db

When we configured dev spaces, we set that application will run in port 3001. That is the default port in api application when running locally. This means that you can just run your app normally (verify that you have correct profile selected at Visual Studio Code) and dev spaces will route incoming request to that port. When you click the debug button your application is started and it's ready to receive incoming calls:

To find out our correct application url for that given child namespace, you can click on the "Dev spaces" toolbar:

It opens menu which allows you navigate to your own child namespace.

Now let's set breakpoint to our service. Open api\routes\users.js and set breakpoint to line 24. Now browse to the "people" view in your child dev space version of the app. You should now be paused in debugger in Visual Studio Code:

At this line we have already received data from our db backend which is still running inside the AKS cluster somewhere and not at our development machine! You can now freely debug and analyze the full http response from db and step through the different lines to see what's going on in our service. You can now see that the actual bug is at the line 32 which is clearly copy-paste from the above error scenario and it should really be res.json(data); instead. You can now change that line and restart debugging and see the correct output at the view:

Now when you have fixed the bug you can create pull request etc. to finalize the actual bug fix. After it gets merged it will be them automatically deployed via CI/CD to the shared integration dev space.

You can also try the shared namespace specific application while you're debugging your api service and you'll see that it's not impacted at all.

How does that routing work then? You can see all my source code in GitHub and analyze this further but one key part is the routing http header azds-route-as:

This routing header makes it possible to route the requests between different services running in dev spaces. Of course if you have other dependent services (e.g. Azure Service Bus) and you communicate via them then you need to take that into consideration when building these debugging capabilities.

I hope that I got you interested in trying out dev spaces. I really think it's big step forward. Not just in debugging front but also in pull request and code reviews when you can use dev spaces for testing out the changes implemented right away. I think really good example is code reviewing tons of HTML and CSS changes. It's so much easier to open the deployed app and side-by-side see code changes and the actual running application.

How to fail the demo (like I have done 😊)

This blog post would be too official if I wouldn't also mention how I have managed to fail my fancy dev spaces demos. So word of caution (note at the time of writing this blog post) you might get into trouble when showing this to people and you have zoomed in using Visual Studio Code because you won't find dev spaces command menu at the toolbar. I created issue about this to the dev spaces repository so hopefully in the future this won't happen.

VS Code extension and long text in toolbar causes dev spaces to disappear #249

JanneMattila posted on Nov 27, 2019

Dev spaces has quite long text in toolbar when connected:

Unfortunately this means that if you zoom even a bit in VS Code your dev spaces functionality disappears:

This means that you cannot access easily "Go to ..." functionality:

This is quite confusing for end user so maybe better option would be clip that in case it doesn't fit in long text mode (believe me I was confused when given demo and couldn't find the button I was looking for 😆).

View on GitHub

Azure PowerShell and ARM template deployment from GitHub actions

Janne Mattila — Wed, 28 Aug 2019 19:35:04 +0000

Earlier this month GitHub Actions support for CI/CD was announced. After that I have been thinking that this is something I need to learn more. So I decided to check how easy it would be to add Azure PowerShell support (which I can then use for my ARM template deployments). There was already az cli action available:

Azure / actions

Automate your GitHub workflows using Azure Actions

GitHub Actions for deploying to Azure

GitHub Actions gives you the flexibility to build an automated software development lifecycle workflow.

With GitHub Actions for Azure you can create workflows that you can set up in your repository to build, test, package, release and deploy to Azure. Learn more about all other integrations with Azure.

Get started today with a free Azure account!

To easily create GitHub CI/CD workflows targeting Azure, use our Azure starter templates to deploy your apps created with popular languages and frameworks such as .NET, Node.js, Java, PHP, Ruby or Python, in containers or running on any operating system. Also the individual Action repos have a sample workflow included in their Readme file to help you quickly get started.

Please try out the GitHub Actions for Azure and share your feedback via Twitter on @Azure. If you encounter a problem, please open an issue on…

View on GitHub

But I wanted to replicate similar setup that I've used in Azure DevOps for many years. Therefore I decided to try to create my own action (and at the same time of course learn how these actions work).

I created GitHub repository for my actions:

JanneMattila / actions

GitHub actions for Azure PowerShell and ARM template deployment

Azure PowerShell action

You can use this GitHub action for executing Azure PowerShell in your own GitHub workflows Similarly as Azure PowerShell task in Azure DevOps.

Primary use case is to call ARM template deployments using deploy.ps1 (deployment entrypoint) to make your template deployments easy but you can of course use this for any Azure automation scenarios you might on your mind.

Example

Here's small example how to execute Azure PowerShell in your own workflow:

name: Azure Deployment example
on: [push]
jobs
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1

    - uses: jannemattila/actions/azurepowershell@master
      with:
        creds: ${{ secrets.AZURE_CREDENTIALS }}
        
    - name: Azure PowerShell & ARM template deployment
      run: ./deploy/deploy.ps1 -ResourceGroupName "pwsh-dev-rg" -Location "North Europe"
      shell: pwsh

In this example you first checkout your codebase and then execute /deploy/deploy.ps1 PowerShell script…

View on GitHub

I looked the az cli repository so that I would better understand the implementation details of their action. I decided to reuse their login instructions so that you can easily jump between az cli and Azure PowerShell. After learning the setup from az cli repository I decided to use a bit simpler setup. It means that I didn't use the components that are provided in the GitHub Action Toolkit but instead I just implemented simple 45 line solution that consist only 3 files. But clearly toolkit repository is the place to go when you're doing some serious development for your actions.

When I had my Azure PowerShell action ready I then created repository for demo application:

JanneMattila / AzurePwshARMActionDemo

Demo how to use Azure PowerShell GitHub action to deploy ARM template using PowerShell.

Azure PowerShell ARM template deployment using action demo

Demo how to use Azure PowerShell GitHub action to deploy ARM template using PowerShell.

View on GitHub

It has my favorite setup from Azure infrastructure point of view:

That of course means deploy folder and deploy.ps1 and azuredeploy*.json files. This demo doesn't (yet?) have any application code in it but for example there are already actions to manage web apps so it would be easy thing to add:

Azure / appservice-actions

Enable GitHub developers to deploy to Azure WebApps using GitHub Actions

IMPORTANT NOTICE:

Actions hosted in this repo are now moved to new GitHub repositories. Please update your existing workflows with the new actions as these old actions will be ARCHIVED and will not receive any updates. Refer to https://github.com/Azure/actions for updated action repo details.
For example, the action azure/appservice-actions/webapp@master should be replaced with azure/webapps-deploy@v1 in your workflows.

Old Action	New Action
Azure WebApp (Windows / Linux WebApps)	`azure/appservice-actions/webapp@master`	`azure/webapps-deploy@v1`
Azure Web app for containers (Single / multi-container apps)	`azure/appservice-actions/webapp-container@master`	`azure/webapps-container-deploy@v1`

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment)…

View on GitHub

Those files under deploy folder are exactly the same kind of ones I have been talking and presenting last years. Next step is to create new workflow that uses my previously created Azure PowerShell action. Here's example worklow:

name: Azure Deployment
on: [push]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1

    - uses: jannemattila/actions/azurepowershell@master
      with:
        creds: ${{ secrets.AZURE_CREDENTIALS }}

    - name: Azure PowerShell & ARM template deployment
      run: ./deploy/deploy.ps1 -ResourceGroupName "pwsh-dev-rg" -Location "North Europe"
      shell: pwsh

Important part is jannemattila/actions/azurepowershell@master which means that I now reuse the action in my other repository in this workflow. So now it's suddenly super easy to user Azure PowerShell in your actions. That's the last step in my above workflow.

Remember that you need to have the Azure credentials correctly set before the workflow can even work:

Above workflow starts automatically based on event: on: [push] (in workflow definition). I can also view my worklow executions under Actions:

You can further drill-down to the details of each run:

And since I'm using ARM template deployments inside my PowerShell file I can see my deployments in Azure Portal as well:

That weird looking deployment name is coming from GITHUB_SHA environment variable which maps to the commit SHA in that repository. It means that you can fully backtrack to the actual change and investigate what has happened in detail:

[master ≡] # git checkout 933307a62df143ccfebab4aadd5925af22484d9c -b investigate-dev
Switched to a new branch 'investigate-dev'
[investigate-dev] # git log
commit 933307a62df143ccfebab4aadd5925af22484d9c (HEAD -> investigate-dev)
Author: Janne Mattila
Date:   Wed Aug 28 20:37:24 2019 +0300
...

I hope you find this post interesting!

P.S. If you thought that I managed to do this in one go... well no 😊:

To ARM, or not to ARM, that is the question

Janne Mattila — Thu, 13 Jun 2019 18:51:02 +0000

Over the last few years I have got this same question maaaany times.
It's all about how do choose your Infrastructure-as-Code (IaC) solution when you're working with Azure. These questions come in different shapes and forms but many of them can be summarized into this simple question:

"Should I use <placeholder> instead of ARM templates?"

Of course you can replace <placeholder> with different options but most commonly offered ones are Terraform and Azure CLI based solutions.

Typically people expect that I give them one simple answer like "do this or do that" but I don't think this is one of those questions that can be answered like that.

Let's start from the existing experience within your development team. Is there already people that know one of these options better than others? Is it single person or are there even more people with similar expertise? Do you plan to reuse single applications IaC assets to different applications or are you actually purchasing these different applications from different vendors and they bring their expertise always to the table and you don't want or can't be too much involved with that? Etc.

Above expertise and experience related questions can already mean that you decide to re-use existing knowledge because that enables you to move faster. I actually compare this often to programming language selection in software development: If you know e.g. Node.js well then why would you use e.g. Python in your next app if there is nothing pushing you to that direction (and of course you can swap these programming languages with any programming languages in this example).

If you don't have expertise or you are looking to evaluate these different options then I highly recommend you to try out ARM templates. There are couple of reasons which you should know before choosing one or another.

Let's first compare three different simple deployments for (as simple resource as) Application Insights and see how the resource groups look after the deployment.

Azure CLI method (example):

Terraform method (example):

ARM template method (example):

Okay so the end result looks the same. Or does it? Can you spot the difference? If you take a closer look at the ARM template deployment screenshot you should notice Deployments: 1 Succeeded (top right). This is important difference between these different deployment methods. ARM template deployments update the resource group about the deployment and thus it's capable of showing history of these deployments and details of each deployment. Read more about template deployment from here.

This might look like a small thing but if you then combine it with deployments from Azure DevOps you have now easy way to map these two together.

See "Dev" environment deployments for our demo application called "app" in Azure Portal (you can derive "Dev" directly from resource group naming serviceshortname-environment-rg):

See "Dev" environment deployment in Azure DevOps side:

And as you can see they map directly one-to-one.

This becomes extremely handy when PIIP hits the fan and you're troubleshooting this from Azure side and you need to quickly identify the actual deployment that has been done for the application. With this solution it's trivial to see what is the last deployment done and when for production environment:

This enables you to quickly jump into Azure DevOps side and see the content of the release and difference between previous and current release from source control and work item perspectives.

If you still decide to use Azure CLI script based deployment then they typically follow this pattern:

add resource x
add resource y
add resource z

etc.

Have you thought what happens if you decide to get rid of resource y? You have to get rid of it manually since the script doesn't magically remove it (unless you have code that does that). And you should not even have access rights to production to do removal (and not to even think how scary and risky that operation would be!). Terraform and ARM template deployments do manage this situation correctly. Just remember to use -Mode Complete in your ARM template deployment for the deployment mode:

$result = New-AzResourceGroupDeployment `
    -DeploymentName $deploymentName `
    -ResourceGroupName $ResourceGroupName `
    -TemplateFile $Template `
    @additionalParameters `
    -Mode Complete -Force `
    -Verbose

This means that resources that are not anymore part of your template will be removed automatically from the target resource group.

One other comment I hear a lot is that "ARM templates are static" and that "there is no place for custom logic in them". I agree and I think it's a good thing. I like my ARM templates nicely controlled via parameters but otherwise they are just plain static files. But that's also the reason why I always bundle PowerShell deploy.ps1 file next to them (and here you can swap PowerShell to e.g. Bash). I think this makes good combination between these two: PowerShell is the entrypoint for the deployment and actual infrastructure assets are described in the actual ARM templates. This also enables very nice development experience. Imagine that you get new developer to the team and they want to spin up your infrastructure to their sandbox subscription. This is what they have to do:

Login-AzAccount
Select-AzSubscription -SubscriptionName YourSubName
cd C:\YourPathToApp\deploy
.\deploy.ps1

Isn't that crazy easy? It's easy because deploy.ps1 has meaningful default values for the deployment. See example of this kind of setup from here.

Now developer doesn't have to know/remember how to set up the parameters correctly to the New-AzResourceGroupDeployment because all these are handled inside the deploy.ps1 file. And of course all the parameters that needs to be overridden per environment can be overridden by providing parameter value to it.

Above also means that you can use this same method in your deployment pipelines. So instead of this:

(screenshot is clipped since it's soooo long!)

You can just use this:

I hope that I managed to demonstrate that you should give ARM a spin if you haven't previously tried it. Or at least you now should know why I like ARM. To summarize:

Do not underestimate the power of ~~Dark side~~ ARM templates!