DEV Community: Jayendran Arumugam

My Journey with Hacktoberfest 2021 ! 🎉

Jayendran Arumugam — Fri, 15 Oct 2021 07:41:30 +0000

2020 Journey

So, this is my second Hacktoberfest participation. On 2020 I participated for the first time and it was a great experience towards Open source platforms.

With the same thrill, here I almost completed my 2021 hacktoberfest 😍 - Just waiting for standard 14 days review waiting period to get over.

2021 Story so far

Before October I did my first contribution to azure terraform

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager

Terraform Provider for Azure (Resource Manager)

The AzureRM Terraform Provider allows managing resources within Azure Resource Manager.

When using version 4.0 of the AzureRM Provider we recommend using the latest version of Terraform Core (the latest version can be found here).

Usage Example

# 1. Specify the version of the AzureRM Provider to use
terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "=3.0.1"
    }
  }
}
# 2. Configure the AzureRM Provider
provider "azurerm" {
  # The AzureRM Provider supports authenticating using via the Azure CLI, a Managed Identity
  # and a Service Principal. More information on the authentication methods supported by
  # the AzureRM Provider can be found here:
  # https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure

  # The features block allows changing the behaviour of the Azure

…

View on GitHub

which made me a special impression towards their community members and maintainers. So, I thought of using this hacktoberfest opportunity to contribute more to azure terraform.

Unfortunately my knowledge with Go language is not that good😕 But I don’t want that to stop my interest towards contribution. So, I went to their issues and start searching for the good first issues and documentation labels.

Then I started fixing the issues by raising a couple of small PRs, which was merged quickly 🎉. Then I found a great step-back 😱, unfortunately azure terraform repo is not participated in the Hacktoberfest event 😔. But the maintainers helped me to get my PR's eligible for Hacktoberfest 🤗. See the community love ♥

This motivates me to complete my final 2 PR's 😍

Here is my 4 simple PR's which fixes 4 simple existing issues

Fixing #13481 #13675

jayendranarumugam posted on Oct 09, 2021

Fixing #13481

This should be required field as per https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/services/batch/batch_pool_resource.go#L415

View on GitHub

Fix for #13380 #13643

jayendranarumugam posted on Oct 06, 2021

Fix for https://github.com/hashicorp/terraform-provider-azurerm/issues/13380

View on GitHub

Fix 10822 #13599

jayendranarumugam posted on Oct 04, 2021

Fix the case sensitive bug in docs Fix https://github.com/hashicorp/terraform-provider-azurerm/issues/10822

View on GitHub

fixing issue #13563 #13592

jayendranarumugam posted on Oct 03, 2021

Added the missing id references

Fix for https://github.com/hashicorp/terraform-provider-azurerm/issues/13563

View on GitHub

One of the interesting fact is I'm the only one who complete 4 PR's that were labeled hacktoberfest-accepted on the terraform-provider-azurerm project 😁

With all this I got the official mail of completion 🎉

Final Completion 🎉🎉

Swags 🤩

Get your limited edition Hacktoberfest 2021 Badge

Hacktoberfest 2021 is Here 🎃

Christina Gorton for The DEV Team ・ Oct 1 '21

#hacktoberfest #opensource #meta #contributorswanted

Securing Azure SignalR +Azure App Service - Part 4

Jayendran Arumugam — Sun, 25 Jul 2021 10:00:10 +0000

After a huge gap I can now officially complete this series with my final part 4. In this part I'm going to discuss about the infra and configuration part of azure signalr using terraform.

The reason for this gap between other parts and this one is because at the time of writing the earlier parts, the terraform don't have support for the Azure signalr NAC.However with the recent terraform release v2.69.0 of the Terraform Provider we got a separate resource called azurerm_signalr_service_network_acl to do so. Thanks to neil-yechenwei for his PR.

On a side note, this particular TF version v2.69.0 is also special for me. Because I did my first ever contribution 🎉 to terraform in this release.

Creating and Configuring Azure signalr using Terraform

In this post we are going to Re-creating the infra and configuration of azure signalr which I already showcased in the Part 1 and Part 2.Having said that the only difference is earlier I did from azure portal, now we are going to automate everything programmatically using Terraform. 😎

Resource Group

resource "azurerm_resource_group" "resourcegroup" {
  name     = "SecureSignalRRG"
  location = "Central US"
}

Azure SignalR

resource "azurerm_signalr_service" "securesignalr" {
  name                = "securesignalrservice1"
  location            = azurerm_resource_group.resourcegroup.location
  resource_group_name = azurerm_resource_group.resourcegroup.name

  sku {
    name     = "Standard_S1"
    capacity = 1
  }
}

VNet

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-cus"
  resource_group_name = azurerm_resource_group.resourcegroup.name
  location            = azurerm_resource_group.resourcegroup.location
  address_space       = ["10.2.0.0/16"]
}

Private Subnet

resource "azurerm_subnet" "privateendpointsubnet" {
  name                 = "private-endpoint-subnet"
  resource_group_name  = azurerm_resource_group.resourcegroup.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.2.1.0/27"]

  enforce_private_link_endpoint_network_policies = true
}

SignalR Private Endpoint

resource "azurerm_private_endpoint" "signalrprivateendpt" {
  name                = "signalrprivateEndpoint"
  resource_group_name = azurerm_resource_group.resourcegroup.name
  location            = azurerm_resource_group.resourcegroup.location
  subnet_id           = azurerm_subnet.privateendpointsubnet.id

  private_service_connection {
    name                           = "psc-signalr"
    is_manual_connection           = false
    private_connection_resource_id = azurerm_signalr_service.securesignalr.id
    subresource_names              = ["signalr"]
  }
}

Note: As of now (25-July-2021) the subresource_names for the azure signalr from official docs is having some typo where its says signalR . But it’s actually signalr which I came to know from my original question in StackOverflow.To fix this I actually raised a PR, until this PR merged please be caution on the subresource name

Azure SignalR NAC

resource "azurerm_signalr_service_network_acl" "securesignalrnac" {
  signalr_service_id = azurerm_signalr_service.securesignalr.id
  default_action     = "Deny"

  public_network {
    allowed_request_types = ["ClientConnection"]
  }

  private_endpoint {
    id                    = azurerm_private_endpoint.signalrprivateendpt.id
    allowed_request_types = ["ServerConnection","RESTAPI"]
  }
}

I hope this series helps you to understand the basics and internals of Azure SignalR. Thanks for reading 🤗

Securing Azure SignalR +Azure App Service - Part 3

Jayendran Arumugam — Wed, 02 Jun 2021 09:53:36 +0000

Hope everyone is safe and well 😊

I thought of discussing azure signalr using terraform in this part, but today I found a very interesting things from azure signalr authentication which is not yet officially documented by Microsoft. So, I'm going to discuss that in this part.

Using Managed Identity

Now a days its common and best practice to use managed identity as much as you can to avoid using the keys/connection string/password in your application or event vault. So, I also would want to use managed identity for authentication from my azure app service to the Azure SignalR resource. This has been explained so well at here

So I thought it would be super simple to implement this in my POC until I hit one road block, let’s see what is the problem and how can we solve that.

Enabling Managed Identity for my POC

If you have followed this series from the beginning, you already noticed that I was doing a POC where I am trying to securely connect my azure signalr from my azure web app. If you missed the earlier parts, no worries please check this Part 1 and Part 2 before going further into this part.

As of now my architecture will look like

And my azure signalr NAC will look like

I have securely connected my azure signal from my azure web app, but my authentication is still using the connection string with accesskey.

The typical azure signalr connection string will look like this.



Endpoint=https://<resource-name>.service.signalr.net;AccessKey=AsNsXs1UdfdfdererereNXa7VOVvPmZnHqnm2iCG6WTom31nXE=;Version=1.0;

In order to enable the managed identity, I followed the above MS doc and did like the steps like below

I need to enabled the system assigned identity for my azure web app
Assign the above system assigned identity as SignalR App Server role in my azure signalr IAM. Please note that this role is still in Preview
Update my connection by replacing the accesskey with AuthType=aad. So, my new connection string will be



Endpoint=https://<resource-name>.service.signalr.net;AuthType=aad;Version=1.0;

The problem that I hit

After enabling the managed identity when I tried to test my app I got 500 Internal Server Error for the negotiate connection

hmm.. not much useful error/information to debug further. So, I have enabled the Application Insights to see where the failure is occurring.

So the actual error is



Failed in authorizing AccessKey for 'https://xxxxx-signalr-01.service.signalr.net', will retry in 3 seconds.

The real 403 Forbidden endpoint is.



https://signalrsevice.service.signalr.net:443/api/v1/auth/accessKey?serverId=xxxx

Stack trace:



Microsoft.Azure.SignalR.Common.AzureSignalRRuntimeException:

at Microsoft.Azure.SignalR.AadAccessKey+<UpdateAccessKeyAsync>d_23.MoveNext (Microsoft.Azure.SignalR.Common, Version=1.8.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)

at Microsoft.Azure.SignalR.AccessKeySynchronizer+<UpdateAccessKeyAsync>d13.MoveNext (Microsoft.Azure.SignalR.Common, Version=1.8.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

Inner exception System.Net.Http.HttpRequestException handled at Microsoft.Azure.SignalR.AadAccessKey+<UpdateAccessKeyAsync>d_23.MoveNext:

Let us understand the error

If you read the error properly you will understand what is going wrong.

From the forbidden endpoint it seems that azure signalr is trying to call a REST API to Azure AD to get the token (for Managed identity) behind the screen

https://signalrsevice.service.signalr.net:443/api/v1/auth/accessKey?serverId=xxxx

Joining the dots / Fixing the Issue

So let’s step back and think about our NAC control configuration

In this we have allowed only Server Connection from our private endpoint, there are a couple of other connections also visible in the dropdown. such as

REST API
TRACE API

Perfect, lets enable the REST API connection also from our private endpoint. Now our NAC will look like

That’s it, I have waited for 30min, and my application is working as expected.

Interesting fact

The documentation about REST API and TRACE API is still not available public or under development phase. So, it is kind of difficult to get the relevant information for now about these two API’s.

Workflow

Permanent fix from Microsoft

I reached out to azure signalr team about this, and I got the response like

We would get rid of API calls in AAD authenticating process, which means if our customer’s use our future SDK, they will not have to enable REST API explicitly.
We will make it internally in our server connection, instead of sending a rest API. This change would take 1-2 months and the newly released SDK will be like 1.9.x. it will be documented in future release.

Overall Architecture

Securing Azure SignalR +Azure App Service - Part 2

Jayendran Arumugam — Sun, 30 May 2021 09:33:06 +0000

In the previous part, we saw the basics of azure signalr connections. In this part we are going to see how to secure the connection between azure signalr and web app.

Let us secure our Azure SignalR Service

Azure SignalR Security features

By the design itself Azure signalr service has a public endpoint that is accessible through the internet, and it currently does not support deploying directly into a virtual network and allowing IP address because of this you cannot leverage certain networking features with the offering's resources such as network security groups, route tables, or other network dependent appliances such as an Azure Firewall.

However, it allows you to create private endpoints to secure the traffic between resources in your virtual network and Azure SignalR Service.

You can also use Service tags and configure network security group rules to restrict inbound/outbound traffic to Azure SignalR Service.

Since in our POC we are using Azure Web app, which is internal to azure, so we are going to leverage the private endpoint to secure the internal connectivity and restricting the entire public internet access.

Default network access control(NAC) will looks like below

Here we are opening/allowing the Server and Client Connection to the open internet(entire public).The rest of connections like Rest API and Trace API will be denied by the Default Action setting

By the way the Trace / REST API are a new connection still in development phase, as of now there is no official docs from Microsoft yet

We are going to update this default setting as per our POC configurations.

Architecture Design

Securing connection between Azure SignalR and Web App

Our requirement is to connect the azure signalr from azure webapp. The Azure Web App can be accessible from internet. So, the way we need to configure the NAC will be like

Configure the public network rule to only allow Client Connections from the public network
For the Server connection we need to configure it privately using the private endpoint and allow the Server Connection from that private endpoint connection, i.e., we should not allow the Server connection from the public network

This step involves a series of other steps to be configured on azure app service as well, let us see that.

Configuring VNet

First we need a Virtual Network (VNet), which should be in the same location as your App services and azure Signalr. In my case it would be Central US
- vnet-cus
Create 2 subnets inside the VNet, one for our Web App and another for Private Endpoint (azure signalr)
- public-subnet
- private-endpoint-subnet

Integrating Azure App Service with VNet

Configure the VNet Integration on our Web App, i.e., Integration with vnet-cus and public-subnet

Configuring Private endpoint on Azure SignalR

From Azure signalR menu, => Private endpoint connections and create a new Private Endpoint
Configure the Private endpoint like below
- Basic (note that the Region should be same as your VNet)
- Resource (Choose your SignalR service as your Resource)
- Configuration (choose subnet as private-endpoint-subnet)
Once you created your private endpoint connection the state should be in Approved- since I'm the actually the owner of the subscription it gets auto-approved for me, if you are not seeing approved state, please reach out to the resource owner

Configuring NAC on Azure SignalR

As we now have an approved private endpoint connection, it is time to secure our Server connection via private endpoint

Testing

Well, that is it, we have secured the connectivity between our azure web app and azure signalr.

But WAIT? How can I make sure that my Web app is using the private endpoint and not the internet for the Azure SignalR connectivity?

I am glad you asked this question, let us find it out.

From my Azure Web App, I tried to nameresolve the azure signalr service.

> nameresolver signalrsevice.service.signalr.net

If we see the output , it is returning the private IP 10.2.1.4 of the signalr resource which confirms that connection to signalr is private and not public.

In the next part we will see the concepts related to signalr Infrastructure using terraform 💪

Securing Azure SignalR +Azure App Service - Part 1

Jayendran Arumugam — Mon, 24 May 2021 14:09:13 +0000

Recently I start to explore the Azure SignalR Service for one of my chat based web wpplication.

Azure SignalR Service simplifies the process of adding real-time web functionality to applications over HTTP. This real-time functionality allows the service to push content updates to connected clients, such as a single page web or mobile application. As a result, clients are updated without the need to poll the server or submit new HTTP requests for updates.

To put more simple word Azure SignalR is a PaaS helps us to update the Client (UI) without refreshing the Page - So Cool😎 Is it?

For that I need to come-up with a architecture, So I start doing a POC - "How to securely connect a Azure SignalR Service from a simple ChatBased Application" ?

You can find more asp.net sample code for AzuresignalR from the below Repo.

aspnet / AzureSignalR-samples

Code samples for Azure SignalR

Welcome to Azure SignalR Service

This repository contains documentation and code samples for Azure SignalR Service.

Azure SignalR Service is an Azure managed service that helps developers easily build web applications with real-time features.

This service supports both ASP.NET Core SignalR and ASP.NET SignalR. SignalR is an open source library that simplifies the process of adding real-time web functionality to applications.

The ASP.NET Core version is not a simple .NET Core port of the original SignalR, but a rewrite of the original version. As a result, ASP.NET Core SignalR is not backward compatible with ASP.NET SignalR (API interfaces and behaviors are different). If it is the first time you try SignalR, we recommend you to use the ASP.NET Core SignalR, it is simpler, more reliable, and easier to use.

For more information about SignalR, please visit SignalR official site.

If you're new to Azure SignalR Service…

View on GitHub

In this series of post, I am going to share the best practice / architecture on how to securely connect azuresignalr with azure app service.

Let us create a POC Environment

AzureSignalR

I choose the resource group, service name and keep the rest of another configurations to default.

The main thing to note here is the Service mode i.e, how we are going to consume/connect to the azure signalr service

Default mode- Using azure signalr for your Web application (Server based) - e.g., azure web App
Serverless mode- Using azure signalr for your Web application (Serverless based) - e.g., static web apps, function apps
Classic mode - Mixed of the above modes. This mode is for backward compatibility for those applications created before there is default and serverless mode. It is strongly recommended to not use this mode anymore

Leave the connectivity method to the public endpoint for now, we will configure this later.

Azure Web App

Create a simple webapp like below. Please keep an eye of the SKU/Size which you need at least the Standard / Premium plan that support secure network access.

Deploy the Web Chat App

Once we provisioned our web app, we deploy the ChatRoom in it.

After the deployment it’s time to update the azure signalr connection string to our app settings config variable called Azure:SignalR:ConnectionString

Finally, our site will look likes.

Different connections in Azure Signalr service

Before diving into the security, we first need to understand how the azure signalr works behind the screen.

There are 2 different connections in azure signalr (sdk) as of now(24/May/2021)

Server Connection (Not Applicable for Serverless mode)
Client Connection (Required for all modes)

Server Connection

Within your SignalR application, where you have a web server that hosts a hub (called hub server) and clients can have duplex real-time communication with the hub server

A negotiate endpoint is exposed by Azure SignalR Service SDK for each hub.
This endpoint will respond to client's negotiation requests and redirect clients to SignalR Service.
Eventually, clients will be connected to SignalR Service.

Server connection flow

From the Code

Please note that for serverless mode there is no server so we could not have a Hub server. So, this server connection will not be applicable for serverless mode.

Client Connection

There are 2 steps to establish the persistent connection.

1.Client will call the negotiate request to app server which returns a redirect response with SignalR Service's URL and access token

2.Client use the new URL and access token to make the persistent connection to SignalR

WebSocket Connection:

Client Connection flow

Overall Flow

Default mode = Server Connection + Client Connection

We will continue on how we secure our azure signalr connectivity to Azure web app in the upcoming parts.

ReadmeAloud 📢

Jayendran Arumugam — Wed, 24 Mar 2021 05:06:24 +0000

Introduction

As a developer, we often came across many open source projects in GitHub, in which Readme.md is one of the first files which we will see 🤩. It is the simplest way to understand, what the project is about, how to use it, and other related information (kind of documentation).

Here some known facts about readme from GitHub

A README is often the first item a visitor will see when visiting your repository. README files typically include information on:

What the project does

Why the project is useful

How users can get started with the project

Where users can get help with your project

Who maintains and contributes to the project

Moreover having a good readme file which will help and attract many contributors. However, it's always been challenging for any low vision or visually impaired developers and contributors. As most of the readme content is in the form of text, which is quite difficult for them to read and understand. So I developed a very simple tool called "ReadmeAloud" that can convert the raw-text from any public GitHub readme file to Speech 🎤 and also provides a way to download it in form of an mp3 file 🎵

Some of the Existing Solution

There are many solutions that already existing in the market which help to convert text into speech easily. Some of them are

Microsoft Edge Brower has an inbuilt feature called ReadAloud

Read aloud highlights each word on the webpage as it's being ?>?read. To stop listening, select the Pause button or the X to close Read aloud.

Google Chrome Extension - Read Aloud: A Text to Speech Voice Reader
Word for Microsoft 365 - Read Aloud

So you are saying that it's already existing what so special about "ReadmeAloud"?

Well, I agree with you that most of these tools were great and helpful for everyone. However, I felt there are a couple of factors that are missing like the feature to download the converted speech into a file(say mp3) and in most cases with the existing tools, either they need to pay for some license or they need to be connected with the internet to use the tool. That why I came up with my own little tool for low vision developers and focused on the most needed place for them which is Github, Readme File.

Architecuture

Here I have focused on the use-case rather than the architecture or technology that the reason you can see a very simple architecture like
-Azure front door
-Azure Webapp
-Azure Text-to-Speech Cognitive API

Source Code:

This is an open-source project.

jayendranarumugam / ReadmeAloud

"ReadmeAloud" is a simple tool that can convert the raw-text from any public GitHub readme file to Speech 🎤 and also provides a way to download it in form of an mp3 file 🎵

ReadMeAloud

A simple tool to provide an easy and efficient way to understand open-source projects for everyone especially visually impaired or sight-impaired friends

Architecutre

Read the detailed article from

Dev Community Post

View on GitHub

Workflow

The user will provide a valid Github Raw Readme.md URL e.g, https://raw.githubusercontent.com/jayendranarumugam/DemoSecrets/master/README.md in the azure front door
Once the user clicks the Search button the azure front door route the traffic to the azure web app (Blazor) securely then convert the text from the URL to Speech using Azure Cognitive API
Once the Speech is converted successfully, the audio bytes will be used to play the audio on the browser and also provide the way to download it as an mp3 file.

Code Walkthrough

This is my first time coding a server-side blazor 😁. Mostly I reused the default boilerplate blazor code, in which I modified and added some parts for my projects e.g, SpeechService. I hope I learned something about blazor now. I also put some great GitHub repos in the below References section which help me to understand blazor.

The entire logic of converting the text into speech is done using simple Azure Cognitive Speech SDK which you can find in SpeechService.cs

       public async Task<byte[]> SynthesizeAudioAsync(string text)
        {
            SpeechConfig speechConfigForAudioAsync = SpeechConfig.FromSubscription(Configuration["CognitiveAPIKey"], Configuration["CognitiveAPIRegion"]);
            speechConfigForAudioAsync.SetSpeechSynthesisOutputFormat(SpeechSynthesisOutputFormat.Audio16Khz32KBitRateMonoMp3);


            using (var synthesizer = new SpeechSynthesizer(speechConfigForAudioAsync, null as AudioConfig))
            {                                

                    using (var result = await synthesizer.SpeakTextAsync(text))
                    {
                        if (result.Reason == ResultReason.SynthesizingAudioCompleted)
                        {
                            return result.AudioData;

                        }
                        else if (result.Reason == ResultReason.Canceled)
                        {
                            var cancellation = SpeechSynthesisCancellationDetails.FromResult(result);                            
                        }
                        return null;
                    }
            }

        }

The result.AudioData is the converted audio in (byte array) byte[] format which we will use to play in the browser and download as an mp3 file using Javascript functions like downloadFromByteArray, playAudio and stopAudio which you can find in the helper.js

Demo

Improvements:

Readmealoud Is more like a small prototype where we can fit many features on top of that easily. Being said that, there are some limitations with the current design which we can improve in the future. Some of them are

The architecture itself is currently highly coupled with the server(blazor), we can make it more scalable by introducing a separate layer for the cognitive calls i.e, the azure function
Current limitation to convert only the public GitHub repo, we can improve that to private repo also by including some additional authentication.
Engish language is the by default for both text and speech conversation, we can improve that easily, since azure cognitive service has a very wide variety of supported languages
I'm not a front-end guy 🙈. So feel free to contribute to readmealoud with your creative ideas or UI/
If the Readme file is too long then the speech conversation would take more time some time it eventually timed out too. Currently, it is suited for small readme files. We can improve that by changing the architecture which we discussed above.
We can also improve the design with more accessibility for people with low vision like providing voice search capability input for GitHub repo details.

Conclusion:

The whole idea of the readmealoud is to provide an easy and efficient way to understand open-source projects for everyone especially visually impaired or sight-impaired friends. Though I showcased this for GitHub readme URL, however, we can put any valid URL for given plain text content. This is just a small idea and I hope it will reach its own audience 🤗

References:

gpeipman / BlazorDemo

Demo application for my writings about Blazor

BlazorDemo

This is my simple Blazor application that demonstrates how to build SPA on Blazor and how to communicate with ASP.NET Core backend Demo application is simple books database.

Solution contains:

Sample database BACPAC file (can be imported to MSSQL using SSMS)
Client application with Blazor UI
Basic select and CRUD operations are implemented in UI and in back-end
Displaying of delete confirmation dialog and deleting of books
Fully functioning add/edit form
Pager component and support for data paging
Dependency injection with custom service classes
Protecting Blazor application and Azure Functions based back-end using Azure AD

Azure AD example

For Azure AD there are two project in solution:

BlazorDemo.AdalClient - Blazor web application that supports Azure AD
BlazorDemo.AzureFunctionsBackend - Azure Functions project with all functions that form back-end for Blazor application

On Azure the following services are needed:

Azure AD - free one is okay
Azure SQL - instance with…

View on GitHub

khalidabuhakmeh / Farm

Blazor Farm Soundboard

Azure Synapse Analytics(workspaces): Deploy and Debug - Part 2

Jayendran Arumugam — Sat, 12 Sep 2020 14:33:36 +0000

Introduction:

In the previous post, we saw the basics architecture and understanding of the ARM template and parameters of synapse analytics workspace. In this post, we are going to see how to secure your synapse analytics workspace by giving proper permission through APIs.

Different types of APIs:

Managing Azure Synapse workspace can be possible with two different REST APIs

Management API
Data Plane API

Usually for all the azure resources we commonly use a REST API which known as Management API. In the case of synapse workspace, we have an additional one special API called Data Plane API.

Management API:

The REST APIs to create and manage Azure Synapses resources through Azure Resource Manager(ARM)
Mainly used for management operations such as create,update,delete synapse workspace.
The {api-version} should be 2019-06-01-preview
The audience claim (used for obtaining bearer token -Authorization) should be "https://management.core.windows.net" or "https://management.azure.com"
The Base API Endpoint looks like

https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}

Data Plane API:

The REST APIs to create and manage Azure Synapses resources through individual Azure synapse workspace endpoint itself.
Used for managing individual synapse workspace operations such as workspace role-assignments,managing and monitoring spark and sql jobs,dataflows,pipelines,datasets,linkedservices,triggers and notebooks.
The {api-version} should be 2019-11-01-preview or 2020-02-01-preview
The audience claim (used for obtaining bearer token -Authorization) should be "https://dev.azuresynapse.net"
The Base API Endpoint looks like

https://<workspacename>.dev.azuresynapse.net/

Right now there is no docs available for Data Plane API(preview). However you can get information from Github docs

Listing available synapse workspace - Management API

Lets see how we can call the management API to list a synapse workspace in a resource group.

Note: You can also directly try these API in the azure docs. Using Try It Option like below

However the above method use user impersonation and not client credentials (using SPN) to grab the bearer token. So lets see how we can call this API using client credentials method.

Getting Bearer token

We are going to use client_credentials way of using SPN(client_id/client_secret) to get the JWT Token.
The below is the simple curl command to invoke the Authorization API and obtain the bearer token.

Note: Make sure that your SPN have proper RBAC role for your purpose. In this below example my SPN have Contributor access to the Resource Group

curl --request POST \
  --url https://login.microsoftonline.com/<tenant-id>/oauth2/token \
  --header 'accept:  application/json' \
  --header 'content-type: multipart/form-data;' \
  --form client_id=a35373d8-c772-4ea0-9f4b-73111376354f \
  --form 'client_secret=xxxxxxx~~2Z6Es' \
  --form grant_type=client_credentials \
  --form resource=https://management.azure.com/ \
  --form scope=Microsoft.Synapse/workspaces/read

Response:

{
  "token_type": "Bearer",
  "expires_in": "3599",
  "ext_expires_in": "3599",
  "expires_on": "1599917407",
  "not_before": "1599913507",
  "resource": "https://management.azure.com/",
  "access_token": "eyJ....."
}

Insomnia Screenshot:

Calling Synapse management workspace List API

Now we got the token, lets call the actual API

curl --request GET \
  --url 'https://management.azure.com/subscriptions/<sub-id>/resourceGroups/azuresynapses/providers/Microsoft.Synapse/workspaces/?api-version=2019-06-01-preview' \
  --header 'accept:  application/json' \
  --header 'authorization: Bearer eyJ0......................' \
  --header 'content-type: application/json'

Response:

{
  "value": [
    {
      "id": "/subscriptions/<sub-id>/resourceGroups/azuresynapses/providers/Microsoft.Synapse/workspaces/azsynapse002",
      "location": "eastus",
      "name": "azsynapse002",
      "type": "Microsoft.Synapse/workspaces",
      "identity": {
        "type": "SystemAssigned",
        "principalId": "712cc76e-7dd9-4978-a259-6c2be5057d2f",
        "tenantId": "<tenant-id>"
      },
      "tags": {},
      "properties": {
        "connectivityEndpoints": {
          "web": "https://web.azuresynapse.net?workspace=%2fsubscriptions%2f<sub-id>%2fresourceGroups%2fazuresynapses%2fproviders%2fMicrosoft.Synapse%2fworkspaces%2fazsynapse002",
          "sql": "azsynapse002.sql.azuresynapse.net",
          "dev": "https://azsynapse002.dev.azuresynapse.net",
          "sqlOnDemand": "azsynapse002-ondemand.sql.azuresynapse.net"
        },
        "managedResourceGroupName": "azuresynapses",
        "privateEndpointConnections": [],
        "workspaceUID": "0fdd8032-3277-4d1b-b4c3-b069f48bd169",
        "extraProperties": {
          "IsScopeEnabled": false
        },
        "provisioningState": "Succeeded"
      }
    }
  ]
}

Insomnia Screenshot:

Synapse Workspace Roles

Before going to the Data Plane API. Let see what is the synapse workspace roles 🤔. Because, we are going to use Data Plane API to manage the workspace roles.

There are actually 3 different roles that are unique to Synapse and aren't based on Azure roles, which are

Synapse workspace admin
Synapse SQL admin
Apache Spark for Azure Synapse Analytics admin

There is an existing azure docs which has explained this in detail about the 3 different roles.

Managing workspace Role access - Data Plane API:

Now we understood the different roles in the synapse workspace. Lets see how we can manage these role access through Data Plane API.

Getting Bearer token

As usual we are going to use the SPN Authentication for getting the bearer token. Here a couple of differences are the

The resource param will be https://dev.azuresynapse.net
We no longer needed the scope param

Note: Make sure that your SPN is already a part of Workspace Admin.

curl --request POST \
  --url https://login.microsoftonline.com/<tenant-id>/oauth2/token \
  --header 'accept:  application/json' \
  --header 'content-type: multipart/form-data;' \
  --form client_id=a35373d8-c772-4ea0-9f4b-73111376354f \
  --form 'client_secret=xxxxxxx~~2Z6Es' \
  --form grant_type=client_credentials \
  --form resource=https://dev.azuresynapse.net

Response:

{
  "token_type": "Bearer",
  "expires_in": "3599",
  "ext_expires_in": "3599",
  "expires_on": "1599917407",
  "not_before": "1599913507",
  "resource": "https://dev.azuresynapse.net",
  "access_token": "eyJ....."
}

Insomnia Screenshot:

Calling Synapse Data Plane API to Add users to workspace roles

Here we are going to add a user into one of the 3 roles. In order to do so, we need to perform 2 operations

Get the role id
Add the user (object id) to that role id.

Get the role Id

In order to get the role id we have to use the below endpoint

https://<workspacename>.dev.azuresynapse.net/rbac/roles?api-version=2020-02-01-preview

Lets curl it

curl --request GET \
  --url 'https://azsynapse002.dev.azuresynapse.net/rbac/roles?api-version=2020-02-01-preview' \
  --header 'accept: application/json' \
  --header 'authorization: Bearer eyJ0' \
  --header 'content-type: application/json'

Response:

{
  "value": [
    {
      "id": "6e4bf58a-b8e1-4cc3-bbf9-d73143322b78",
      "name": "Workspace Admin",
      "isBuiltIn": true
    },
    {
      "id": "c3a6d2f1-a26f-4810-9b0f-591308d5cbf1",
      "name": "Apache Spark Admin",
      "isBuiltIn": true
    },
    {
      "id": "7af0c69a-a548-47d6-aea3-d00e69bd83aa",
      "name": "Sql Admin",
      "isBuiltIn": true
    }
  ]
}

Insomnia Screenshot:

Add Users to the Role ID

Now we got the role id for each roles (these role ids are same for all the synapse workspace globally). Let add the user using below endpoint

https://<workspacename>.dev.azuresynapse.net/rbac/roleAssignments?api-version=2020-02-01-preview

#Json body Param:

{
"roleId": "<workspace role id>",
"principalId": "<objectid of the user/group>"
}

Curl:

curl --request POST \
  --url 'https://azsynapse002.dev.azuresynapse.net/rbac/roleAssignments?api-version=2020-02-01-preview' \
  --header 'authorization: Bearer eyJ..............' \
  --header 'content-type: application/json' \
  --data '{
    "roleId": "6e4bf58a-b8e1-4cc3-bbf9-d73143322b78",
 "principalId": "fb1e7804-9542-4412-be66-e143a10e3b1a"
}'

Response:

{
  "id": "6e4bf58a-b8e1-4cc3-bbf9-d73143322b78-fb1e7804-9542-4412-be66-e143a10e3b1a",
  "roleId": "6e4bf58a-b8e1-4cc3-bbf9-d73143322b78",
  "principalId": "fb1e7804-9542-4412-be66-e143a10e3b1a"
}

Insomnia Screenshot:

Conclusion:

In this Post we just saw some cool ways to manage the synapse workspace purely using APIs. This opens-up the wide space for automation. Please keep in the mind that most of these features are in preview.So there is no such grantee that all the API endpoint/operations will be the same as now which i explained in the post.I keep my best level to update this post whenever some update needs.Here I explained one of operation like synapse role assignment for data plane.Actually Data Plane API can do much more than this.Read this github repo for getting the full power of Data Plane API Operations.

Blurry screenshots after uploaded in Dev.to Post?

Jayendran Arumugam — Sun, 23 Aug 2020 08:01:58 +0000

Not sure that I'm only facing this issue !? Whenever I upload some screenshots from my PC for my post, It was looking blurry during the preview and after the publish.

If you look at my recent post Azure Synapse Analytics(workspaces): Deploy and Debug - Part 1 most of the screenshots are blurry. I tried with many different ways to get screenshots like snipper, Snagit nothing seems to work for me.

Eg.,
Some of the screenshot like

Azure Synapse Analytics(workspaces): Deploy and Debug - Part 1

Jayendran Arumugam — Fri, 21 Aug 2020 16:05:25 +0000

Disclaimer:

This post is provided "as-is".

Information and views expressed in this post, including URL and other Internet Web site references, may change without notice. Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

Setting up the Definitions clear

Azure Synapse Analytics

Azure Synapse is an analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources-at scale

Azure Synapse Analytics Workspace (Preview)

Azure Synapse comes with a web-native Studio user experience that provides a single experience and model for management, monitoring, coding, and security called synapse analytics workspace.
(As of writing this post, Azure synapse Analytics workspace is in preview)

If you are familiar with Azure Data Platform, I can simply put synapses workspace in a single pic like below 😉

Roadmap of Azure Synapse Analytics

What we are going to see in this Post?

We can create the synapses workspace which is a public preview from azure portal easily. However currently, there are no official docs yet available that can give detailed steps for creating synapses workspace programmatically using ARM. In this post (part 1), we are going to see how we can deploy azure synapses from the ARM template using service principal, deployment architecture, the different levels of access, and conditions.

I choose this topic because right now most of the docs are still under development, I faced a lot of troubles initially, even I created few issues and PR. So this post it's just a matter of sharing my new knowledge to others 😊

Creating Synapses Workspace through ARM Template using SPN Failing during provisioning "storageRoleDeploymentResource" with BAD Request #60705

jayendranarumugam posted on Aug 12, 2020

Is there any Specific permissions needed to include while deploying a Synapses workspace through ARM with SPN Authentication?

ARM will getting failed at `storageRoleDeploymentResource' Resource provisioning state with BAD request

Also the sql admin was randomly assigned with a GUID. Do we need any special permission to include in the doc to cover the SPN Deployment ?

Doc Link: https://docs.microsoft.com/en-us/azure/synapse-analytics/security/how-to-set-up-access-control

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1149f92f-5b95-7b27-5525-db4f792d7937
Version Independent ID: 60242818-539c-6e91-7dc8-222ccc383574
Content: Secure your Synapse workspace (preview) - Azure Synapse Analytics
Content Source: articles/synapse-analytics/security/how-to-set-up-access-control.md
Service: synapse-analytics
Sub-service: security
GitHub Login: @matt1883
Microsoft Alias: mahi

View on GitHub

Synapses Creating Big Data Pool validation error: Parameter 'BigDataPoolResourceInfo.node_count' failed to meet validation requirement. #14708

jayendranarumugam posted on Aug 07, 2020

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug Running the command of creating bigdata pools for azure synapses getting the below error validation error: Parameter 'BigDataPoolResourceInfo.node_count' failed to meet validation requirement.

To Reproduce Download latest az cli and running the below command which was the example given at here

az synapse spark pool create --name testpool --workspace-name testsynapseworkspace --resource-group rg \
--spark-version 2.4 --node-count 3 --node-size Medium

Expected behavior

It should create a Big data pool

Environment summary

Windows 10

Additional context

View on GitHub

Synapses Data Plane API Audience should remove Trailing #10500

jayendranarumugam posted on Aug 18, 2020

Am not exactly sure that this is a bug in the code itself or docs need to be updated. Either way, I'm creating this issue for both the case

Data Plane APIs Synapses Audience need to remove extra /

As per the below docs

Set the Authorization header to a JSON Web Token that you obtain from Azure Active Directory. For data-plane operations, be sure to obtain a token for the resource URI / audience claim "https://dev.azuresynapse.net/", NOT "https://management.core.windows.net/" nor "https://management.azure.com/". For more information, see Acquire an access token.

From the above statement has the audience claim https://dev.azuresynapse.net/ is not working, and will give an error like

{
    "code": "InvalidTokenAuthenticationAudience",
    "message": "Token Authentication failed with SecurityTokenInvalidAudienceException - IDX10214: Audience validation failed. Audiences: '[PII is hidden]'. Did not match: validationParameters.ValidAudience: '[PII is hidden]' or validationParameters.ValidAudiences: '[PII is hidden]'."
}

But once we removed the trailing slash at the end of the URL (/) which will be https://dev.azuresynapse.net then the APIs are working.

Please make necessary fix either in the docs or the API

Reference: https://docs.microsoft.com/en-us/rest/api/synapse/?source=docs

View on GitHub

Fixed the Address/Ip Typo #12622

jayendranarumugam posted on Aug 10, 2020

Fixed the Address/Ip Typo

Description

Checklist

[x] I have read the Submitting Changes section of CONTRIBUTING.md
[x] The title of the PR is clear and informative
[x] The appropriate ChangeLog.md file(s) has been updated:
- For any service, the ChangeLog.md file can be found at src/{{SERVICE}}/{{SERVICE}}/ChangeLog.md
- A snippet outlining the change(s) made in the PR should be written under the ## Upcoming Release header -- no new version header should be added
[x] The PR does not introduce breaking changes
[x] If applicable, the changes made in the PR have proper test coverage
[x] For public API changes to cmdlets:
- [x] a cmdlet design review was approved for the changes in this repository (Microsoft internal only)
- [x] the markdown help files have been regenerated using the commands listed here

View on GitHub

Getting ARM Template from Azure Portal

We can easily grab the template for the synapses workspace template from the Azure portal itself.

Step 1:

Step 2:

After getting the ARM template will look like the below

ARM Parameters,

Architecture of Synapses Workspace:

Here I'm just giving a very simple & high-level architecture image for understanding the synapses workspace components.

As you see synapses workspace itself consist of a storage account gen2 and default on-demand SQL pool, it can be accessed by 3 different roles (NOT RBAC roles) called

Workspace Admin
SQL Admin
Spark Admin

I'll explain these roles in part 2. As of now just assume its a role needed for any user to access the workspace.

ARM Graphical Viewer

We can easily understand this ARM using VS Code + ARM Template View extension. Here the final result will look like

Great! now we understood that synapses workspace needs a storage account(gen2) and a container (gen2filessystem) in it. The ARM also has some other components like roleassignments, managedidentitysqlcontrolsettings, and firewall. which are basically for giving correct permissions for our workspace, we will look more about these in the below sections.

Analyzing Parameters from ARM

Most of the parameters are self-explanatory, however, some of them depend on some high privilege permission. Let's see those

setWorkspaceIdentityRbacOnStorageAccount : If true, this will assign the role of the workspace(MSI) as the storage blob contributor to the existing or the new storage account. This needs Microsoft.Authorization/roleAssignments/write permission which requires owner role or at-least User Access Administrator. So make sure you give owner/User Access Administrator access to your SPN if you set this true.

The below table will help you to understand this parameter based on your SPN access.

grantWorkspaceIdentityControlForSql :
Grant CONTROL to the workspace's managed identity on all SQL pools and SQL on-demand
isNewFileSystemOnly: If the storage account new/exist but when we need to create a new filesystem, use this variable to true
setSbdcRbacOnStorageAccount : If we need to enable the user, (whose object id will be provided in userObjectId) as the Storage Blob contributor to the Storage account gen2.

This is a nested task which depends on setWorkspaceIdentityRbacOnStorageAccount parameter, i.e., this will be executed only if you provide setWorkspaceIdentityRbacOnStorageAccount as true.
E.g If you provide setWorkspaceIdentityRbacOnStorageAccount as false and even if you provide setSbdcRbacOnStorageAccount as true it won't affect anything.

Conclusion

I hope you get some additional information about the synapse workspace from this post. I'll explain more about API's, security best practice in Part 2

Azure AD Managed Identity: Connecting Azure Web App and Slots with Azure SQL DB (without credentials)

Jayendran Arumugam — Wed, 17 Jun 2020 13:55:46 +0000

Introduction

Managed Identity is a great way for connecting services in Azure without having to provide credentials like username, password or even clientid or client secrets. Please note that not all azure services support managed identity. There are many great articles and blogs which discuss about in depth of managed identity and their types.You can see some of them in the See Also section.In this article we are going to see 2 main popular Azure resources to connect each other without providing any credentials in the code, they are azure app service (front-end) and azure sql database (back-end) by using System Managed Identity.

Using typical Connection String way

Usually we use the connection string to connect an azure sql database from any front-end. For instance lets say we are developing a dotnet application and want to connect to a back end (azure sql db) we will simply use the connection string like in the web.config below (which may be different for some other applications)

<connectionStrings>
<add name="DbConnection" connectionString="server=<sqlinstance>.database.windows.net;database=<dbname>;Uid=localadmin;Pwd={your_password_here};Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;" providerName="System.Data.SqlClient" />
</connectionStrings>

From the C# code we will consume the connection string from the web.config to establish the connectivity from our code to backend like

using (var connection = new SqlConnection(connectionstring))
{
connection.Open();

//Executing some sql code

}

Azure Key Vault for Connection String

It is always good to store this type of connection string in a secure place like
azure key vault secrets. However we still need to store the client id and client secret in a web.config if we aren't using the Managed Identity.See again storing a secret in a web.config, which is more like a chicken and egg problem 🐔🥚🐔

Here we need more sophisticated solution to solve this, which is the Managed Identity.

Using System Managed Identity way

Step 1: Enabling System Managed Identity in Web App

First we need to enable the System Managed Identity in our web app.

If you are using any slots you should also enable the same options in the slots as well

Step 2: Creating Managed Identity User in Azure SQL

After we enabled the System Managed Identity in Azure App, we have to create a Managed Identity User in Azure sql db. To do so we must enable the Azure Active Directory Admin, then login to the database using the Active Directory account from either SSMS or Azure Data Studio.

We have to run the below query in the corresponding database.

CREATE USER [<App Service Name>] FROM EXTERNAL PROVIDER 

CREATE USER [<App Service Name>/slots/<slotname>] FROM EXTERNAL PROVIDER -- For apps in the slots

ALTER ROLE db_datareader ADD MEMBER [<App Service Name>] -- gives permission for normal app read to database
ALTER ROLE db_datawriter ADD MEMBER [<App Service Name>] -- gives permission for normal app write to database

ALTER ROLE db_datareader ADD MEMBER [<App Service Name>/slots/<slotname>] -- gives permission for slot app  read to database
ALTER ROLE db_datawriter ADD MEMBER [<App Service Name>/slots/<slotname>] -- gives permission for slot app  write to database

By doing the above we are creating a user for our system managed identity of our app in the backend and providing required permissions like read/write a database.

Step 3: Remove the credentials from the Connection String

Finally we have to remove the credentials details like userid , password from the connection string ,like below

<connectionStrings>
<add name="DbConnection" connectionString="server=<sqlinstance>.database.windows.net;database=<dbname>;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;" providerName="System.Data.SqlClient" />
</connectionStrings>

Step 4: 1-Line Magic Code

Now we just need to add a single line of code (magic) in our application before opening the connection like below

connection.AccessToken = await (new AzureServiceTokenProvider()).GetAccessTokenAsync("https://database.windows.net/");

Sometime during the local testing if you have more than one subscription you will get error at the above line. To resolve that please use the optional parameters called tenant id for GetAccessTokenAsync method

connection.AccessToken = await (new AzureServiceTokenProvider()).GetAccessTokenAsync("https://database.windows.net/", <tenant-id>);

In order to use the above code you also need to import a package called Microsoft.Azure.Services.AppAuthentication.
You can install it from nuget like below

Install-Package Microsoft.Azure.Services.AppAuthentication -Version 1.5.0

So finally our code will look like below

using (var connection = new SqlConnection(connectionstring))
{
connection.AccessToken = await (new AzureServiceTokenProvider()).GetAccessTokenAsync("https://database.windows.net/");
connection.Open();

//Executing some sql code

}

Step 5: Testing it Locally

In order to test this from local machine we have to use
Azure CLI 2.0 or above. So install the Az Cli then run the command

az login

It will prompt you for the credentials. Once you successfully logged in you will see your subscription details like in the below pic

Now you can run your application in local.

(Note: If you have more than 1 tenant access you have to explicitly pass the tenantid in the GetAccessTokenAsync method)

After the successful run you can deploy the application in Azure.

Conclusion

Azure Managed Identity is going to remove the way of storing credentials in code even in azure key vault. In this article we saw only 2 services. But there are more and more services are coming along the way. Using this great feature we can do all the things inside Azure very safely and not leaking any credentials to others.

Serverless Prediction of a Product Feedback

Jayendran Arumugam — Sat, 23 May 2020 10:52:09 +0000

This article is part of #ServerlessSeptember. You'll find other helpful articles, detailed tutorials, and videos in this all-things-Serverless content collection. New articles from community members and cloud advocates are published every week from Monday to Thursday through September.

Find out more about how Microsoft Azure enables your Serverless functions at https://docs.microsoft.com/azure/azure-functions/.

Story behind my use-case

If you are developing any product, feedback is much more important. You have to trace each and every feedback like feature request or Bugs/Issues. Even Microsoft has a dedicated website https://feedback.azure.com/ for tracking all its feedback related to their Azure Product. For issues they have to use GitHub issues https://github.com/MicrosoftDocs/azure-docs/issues for all their open Source Projects. This would be a great way to track all the feedback on the open-source project. But how we can track Internal Project feedback efficiently!

Because ...

In this blog, we are going to use a better way to solve this problem of getting feedback about our internal projects with the help of azure server-less computing and AI 😎

Let's jump into our Project

Architecture

Let's assume that we are going to develop a new internal product called myproduct (wow what a unique name 🤣 )

I'm going to use Yammer to get all the feedback's from our internal users.

Yammer is your social layer across Microsoft 365, integrating with the apps and services you already use to stay productive. You can create and edit documents, take notes, and share resources as a group. And get back to your conversations from anywhere in Microsoft 365.

If you notice the architecture, most of the main components are server-less like azure functions and logic app. There are many advantages of using server-less is our architecture. some of them are:

Cost Efficient (Consumption model)
Less Development time (logic-app : drag/drop model)
Quick Deployments etc.,

Components used

The components in this architecture are more flexible. i.e, you can easily replace one component with the other similar one.

For e.g

You can use MS Teams or any other internal chat-based tool instead of yammer for getting the feedback.
Azure DevOps WorkItem Tracking can be replaced by github/JIRA kind of tools easily.
Cosmos DB can be replaced by the Azure Table Storage.

See how flexible this architecture is when we start using more and more server-less components 😉

Workflow

Here I'm focusing on a product that was developed for internal purposes. So our end-users are nothing but the employees in our organization. So I used Yammer as my main platform to gather all the feedback and bugs about myproduct from users.

Logic App

As soon as a user posted in yammer our logic app will be triggered. Let's see how our logic app now.

Yammer

LUIS

I'm using LUIS to predict what are the intents in the yammer post(utterances)

Intents: Bugs🐞 /Features
Utterances: Yammer post which users posted

An Utterances are input from the user that your app needs to interpret.

An Intent represents a task or action the user wants to perform. It is a purpose or goal expressed in a user's utterance.

I trained by LUIS with some user-defined Intents. You can also find my exported LUIS model in the Github

AzureDevOps WorkItems

If the predicted intent is Bug then I'll create a bug work-item in AzureDevOps else if it is Feedback then I'll create Feature work-item as simple as that.

CosmosDB

All are fine, but what happens if the intent is neither bug nor feedback?
In such a case, None Intent will be our predicted Intent. This None Intent is more important for Re-training our LUIS model. So I'm going to save these intents safely in CosmosDB

Azure Functions

OK, I've saved the None Intents in cosmos DB, now I'm going to build a static website where developers can view all the information about the none intents in near real-time without doing any refresh.

This is quite an interesting part, where I've learned how powerful the Azure-Functions are! Thanks to Anthony Chu for his wonderful blog

Image Credits: Anthony Chu

I've used Signal-R for a Broadcasting real-time updates from cosmos DB using CosmosDB Change-Feed in Azure Function.

If you see the below image I've four azure functions totally.

Let's start with OnDocumentsChanged class, which will be triggered automatically for every insert/update on the target Cosmos DB.

For every new/updated item in the cosmos DB, I'll simply adding those items as messages in the SignalRHub(SignalRHubItems)

public static class OnDocumentsChanged
    {
        [FunctionName("OnDocumentsChanged")]
        public static async Task Run([CosmosDBTrigger(
            databaseName: "MyProductDB",
            collectionName: "Items",
            ConnectionStringSetting = "AzureWebJobsCosmosDBConnectionString",
            LeaseCollectionName = "leases",
            CreateLeaseCollectionIfNotExists = true)]
             IEnumerable<object> updatedItems,
            [SignalR(HubName = "SignalRHubItems")] IAsyncCollector<SignalRMessage> signalRMessages,
            ILogger log)
        {
            foreach(var item in updatedItems)
            {
                await signalRMessages.AddAsync(new SignalRMessage
                {
                    Target = "SignalRHubUpdatedItems",
                    Arguments = new[] { item }
                });
            }
        }

The SignalRInfo class will be called from the static web page to make connections to SignalRhub to read the message from that.

public static class SignalRInfo
    {
        [FunctionName("SignalRInfo")]
        public static IActionResult Run(
            [HttpTrigger(AuthorizationLevel.Anonymous,"get","post")] HttpRequest req,
            [SignalRConnectionInfo(HubName = "SignalRHubItems")] SignalRConnectionInfo connectionInfo,
            ILogger log)
        {
            return new OkObjectResult(connectionInfo);
        }
    }

The GetItems class will also be called from the static web page to make connections to CosmosDB to get/read all the Items.

public static class GetItems
    {
        [FunctionName("GetItems")]
        public static IActionResult Run(
            [HttpTrigger(AuthorizationLevel.Anonymous,"get","post")] HttpRequest req,
            [CosmosDB("MyProductDB", "Items", ConnectionStringSetting = "AzureWebJobsCosmosDBConnectionString")]
                IEnumerable<object> updatedItems,
            ILogger log)
        {
            try{
                return new OkObjectResult(updatedItems);
            }
            catch(Exception e){
                log.LogError(e.Message.ToString());
            }
             return new OkObjectResult(updatedItems);

        }
    }

Finally StaticFileFunction class is simple Az-function to show our statics web page (index.html), inside www folder

public static class StaticFileFunction
    {
        const string staticFilesFolder = "www";
        static string defaultPage = String.IsNullOrEmpty(GetEnvironmentVariable("DEFAULT_PAGE")) ?
    "index.html" : GetEnvironmentVariable("DEFAULT_PAGE");

        [FunctionName("StaticFileFunction")]
        public static HttpResponseMessage Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            try
            {
                var filePath = GetFilePath(req, log);

                var response = new HttpResponseMessage(HttpStatusCode.OK);
                var stream = new FileStream(filePath, FileMode.Open);
                response.Content = new StreamContent(stream);
                response.Content.Headers.ContentType =new MediaTypeHeaderValue(GetMimeType(filePath));
                return response;
            }
            catch (Exception e)
            {
                string name = e.Message.ToString();
                return new HttpResponseMessage(HttpStatusCode.NotFound);
            }
        }

        private static string GetScriptPath()
            => Path.Combine(GetEnvironmentVariable("HOME"), @"site\wwwroot");

        private static string GetEnvironmentVariable(string name)
            => System.Environment.GetEnvironmentVariable(name, EnvironmentVariableTarget.Process);
        private static string GetFilePath(HttpRequest req, ILogger log)
        {
            var path = req.Query["file"];

            var staticFilesPath = Path.GetFullPath(Path.Combine(GetScriptPath(), staticFilesFolder));
            var fullPath = Path.GetFullPath(Path.Combine(staticFilesPath, path));

            if (!IsInDirectory(staticFilesPath, fullPath))
            {
                throw new ArgumentException("Invalid path");
            }

            var isDirectory = Directory.Exists(fullPath);
            if (isDirectory)
            {
                fullPath = Path.Combine(fullPath, defaultPage);
            }

            return fullPath;
        }

        private static bool IsInDirectory(string parentPath, string childPath)
        {
            var parent = new DirectoryInfo(parentPath);
            var child = new DirectoryInfo(childPath);

            var dir = child;
            do
            {
                if (dir.FullName == parent.FullName)
                {
                    return true;
                }
                dir = dir.Parent;
            } while (dir != null);

            return false;
        }

        private static string GetMimeType(string filePath)
        {
            var provider = new FileExtensionContentTypeProvider();
            string contentType;
            provider.TryGetContentType(filePath, out contentType);
            return contentType;
        }
    }

That's it! Let's test it out

Testing

Posted some posts in Yammer.

Checking AzureDevOps

It's created bugs and features correctly. However one of the posts got missed meaning it's predicted as None Intent.

Checking CosmosDB

So our post got inserted in cosmos DB correctly.

Static Web Page

Here is the live-stream

And it's Working Finally!

Source Code

This is an Open-Source Project, view the full source code in the below link and feel free to provide feedback/issues to me 😎

jayendranarumugam / AzureDevStories

Serverless Prediction of a Product Feedback (#AzureDevStories)

This Project is created as a part of Azure Dev Stories Challenge and won the First Prize 🏆

I've published a detailed article about this project in the Dev Community

Architecture Diagram

Components Used

Yammer
LUIS
Logic Apps
AzureDevOps WorkItems
Cosmos DB
Signal R
Azure Functions

Yammer

Users provide their feedback about the product. It could be many, for the demo purpose I just choose 2 topics (Bug, Feature)

LUIS

Creating Intents for Bugs and Feedbacks in the LUIS.

Logic Apps

Predicting the Intents i.e, Bug or Feedback based on the Yammer Post by the user and take necessary actions

AzureDevOps WorkItems

Create Bug/Feature if the top intent of the Post matched with LUIS

Cosmos DB

Insert the document in Cosmos DB if the top intent of the Post is None

Signal R

Serverless Signal R used to autorefresh the WebPage for the…

View on GitHub

Reference

Quantum Computing: An Introduction

Jayendran Arumugam — Fri, 21 Jun 2019 07:02:44 +0000

Introduction

We often hear that quantum computers will be the future and it will help us to solve all world problems like world hunger, natural disaster, and many complex problems. Understanding the concept of quantum and its behavior is much necessary to solve these kinds of problems. In this blog, we will understand the history and the concepts of quantum. There are many ways to understand why quantum mechanics is hard to simulate. Perhaps the simplest is to see that quantum theory can be interpreted as saying that matter, at a quantum level, is simultaneously in a host of different possible configurations (known as states) at the same time.

Quantum History

It all begins around 1672 where big scientists working on finding the behavior of light. Initially, Newton finds that light is the form of particles, later Robert Hooke found that light is a wave and not a particle, until the problem of the photoelectric effect.

Photoelectric Effect

When light hits a cable next to another cable electrons jump from one to another and it was thought it happened because waves of light made atoms vibrate until they ejected an electron. But when it was measured carefully a big contradiction was found, it’s only happened for the light of some wavelengths for others no electrons jump at all.

Einstein Invention of Quantum

In 1905, Einstein tries to solve this photoelectric problem by making a big hypothesis combining the two previous ones.

“What if light were made not of waves or particles but of both that is what if light were made of wave packets (photons)”

Light exist in both waves and particles- photons

Quantum is the smallest part of something, and nothing smaller than this of that “something” exists. Such as a cent is a quantum of American money, as is an electron of the electric charge. Photon is, in that respect, the quantum of electromagnetic field(light).

It was a revolutionary view and it opened the field of quantum physics.

He also awarded the Nobel prize for this invention.

Einstein Receiving Nobel Prize

Quantum Properties

Superposition

Quantum superposition states that an object that is not being observed exists in all possible states at once- it is in a superposition. This superposition is a combination of all the possible states that a particle could theoretically be in.

E.g., Electron in an Atom – Because until you observe the electron in the atom, it could be located anywhere outside the nucleus.

Quantum Entanglement

When any two quantum particles interact, they are entangled, and it means that their quantum states are interdependent. There’s a correlation between them, so that if one is in one particular state, then the other one has to have some other particular state across space and time.

E.g., 2 electrons have been entangled then this may create a situation where those two spins are correlated, such that if one of the electrons has a spin up, the other must-have spin down.

Quantum Entangelement of two electrons

Quantum Computing

Quantum computing is the use of quantum-mechanical phenomena such as superposition and entanglement to perform computation. A quantum computer is used to perform such computation, which can be implemented theoretically or physically.

Qubit

In quantum computing, a qubit or quantum bit is the basic unit of quantum information.

To understand Qubit theoretically, let’s introduce a vector notation for bits as follows: a bit is represented by a column vector of two elements (α,β)T, where α stands for 0 and β for 1. Now the bit 0 is represented by the vector (1,0)T and the bit 1 by (0,1)T. Just like before(classical bit), there are only two possible values(qubit in superposition).

The foundational core of quantum computing is to store information in quantum states of matter and to use quantum gate operations to compute on that information, by harnessing and learning to “program” quantum interference

A Qubit existing in a superposition

Modern Computer Vs Quantum Computer

Our modern computers made up of simple components to do simple operations like Main Memory, Arithmetic Unit, Control Unit. Looking into the depth of our Computer chips contains modules, which contain logic gates that contain transistors. A transistor is nothing but a switch that either can block or allow the way of information coming through. Here information is nothing but bits of 0’s or 1’s. Combination of several bits used to represent more complex information.

Here, more importantly, the bits are exhibit either in 0’s or 1’s which can yield many different possible data like below

2 bits = > 2⁴= > 8 Game Boy

4 bits = > 2⁷ memories= > 128 ASCII Characters

Out of 16 possible combinations 1 will be used

Now the Quantum computer a way to different that it used qubit where it can exhibit both 0 and 1 at the same time or these or a quantum superposition of 0 and 1 as long as we are unobserved. But the instance we measure it collapses into one of the definite states.

For 4 classical bits can be in one of the 2⁴ different configurations at a time, which is 16 possible combinations like (0000 to 1111) out of which we can use just one.

For 4 Qubits in superposition, however, can be all of those 16 combinations at once

All 16 possible combination exisit in superposition

Some Applications of Quantum Computer

Cryptography

Quantum computers used to create private keys for encrypting messages from one place to another. So that hackers could not secretly copy the key perfectly, due to the quantum uncertainty. They would have to break the laws of quantum physics to hack the key.

Teleportation

Teleportation of information from one location to another without physically transmitting the information (Quantum entangled property)

Problem with Quantum Computing Now

Scaling:

If we want to use quantum computers to solve real problems, they will need to explore the large space of quantum states. The number of qubits is important, but so is the error rate. By nature, qubits are fragile. They require a precise environment and state to operate correctly, and they’re highly prone to outside interference. So, if we try to scale the qubits currently error rates are also increased.

Currently, Microsoft, IBM, and other pioneers are working on creating a more stable Qubit called Topological Qubit. It can store information in a more stable form as compared to other Qubits. Topological Qubit will allow Quantum computers to scale at a much higher rate and allow us to build a quantum computer large and stable enough to solve our most challenging problems.

Microsoft Quantum Development Kit (Q#)

Microsoft Quantum Development Kit to empower a growing community with tools to unlock the quantum revolution for our tasks, problems, and solutions. The high-level programming language, Q#, was designed to address the challenges of quantum information processing; it is integrated into a software stack that enables a quantum algorithm to be compiled down to the primitive operations of a quantum computer.

Conclusion

In this blog, I just wrote an introduction to the world of quantum. I hope this blog helps you to understand the basics things of quantum. There is much more to achieve in the near future. The significance of quantum is very huge demand in the future, with the help of quantum we can solve many world problems easily and efficiently