DEV Community: c-arnab

Serverless in Azure using Static Web Apps, Functions and Cosmos DB

c-arnab — Mon, 16 Jan 2023 06:20:51 +0000

In this post, we look at Serverless development on Azure. First we look at the tools and packages necessary to develop locally. Then we create a static site using Azure Static Web Apps (SWA). Then we look at the built in authentication support provided by SWA as we create the authentication layer to ensure users can access their data securely. Then we look at how SWA supports building APIs using built in support for HTTP-triggered functions as we create and Integrate APIs to the static web site using .Net6 and C#. Then we create the data layer in Cosmos DB and integrate the same to functions built earlier. Finally we look at the CI/CD support provided where 'Github repository changes' trigger builds and deploy the solution to Azure.

Problem Statement

The people at helm at the insistence of HR Department have decided to have a Calendar system. Though the solution is supposed to have loads of features, the decision is to build the whole iteratively and a basic Proof Of Concept (POC) is to be developed first. A person at helm attended a conference where s/he heard about "serverless" which allows on-demand scaling and also brings down TCO of a solution with “pay-as-you-go” usage and so one of the requirements for the POC is to make the entire solution using serverless technologies.

Solution

Compute options in serverless services are varied in Azure. One could choose Serverless Containerized Microservices using Azure Container Apps OR Serverless Kubernetes using AKS Virtual Nodes OR Serverless functions using Azure Functions OR the newest entrant- Azure Static Web Apps.

Azure Static Web Apps supports static content hosting, APIs powered by Azure Functions, local development experience, CI/CD workflows, global availability, dynamic scale, preview environments and all this without the necessity to manage servers, creating & assigning SSL certificates, establishing reverse proxies, etc.

There are two options in Databases amongst serverless services in Azure. The relational Azure SQL Database serverless and the non relational Azure Cosmos DB.

Azure Cosmos DB is a fully managed NoSQL database which offers features such as Change Data Capture and multiple database APIs - NoSQL, MongoDB, Cassandra, Gremlin, & Table enabling one to model real world data using documents, column-family, graph, and key-value data models.

For the POC, the calendar system will enable users to authenticate themselves and view their events as well as add events to the calendar.
Static Web Apps with Cosmos DB will be used to implement these use cases. Visual Studio code will be used as IDE.

Architecture Diagram with Application Development Lifecycle

Prerequisites for local development

.net 6 sdk - https://dotnet.microsoft.com/en-us/download/dotnet/6.0
Azure Functions Core Tools v4.x - https://go.microsoft.com/fwlink/?linkid=2174087
Static Web Apps CLI - https://azure.github.io/static-web-apps-cli/
Azure Cosmos DB Emulator - https://aka.ms/cosmosdb-emulator
Install Azure Functions Extension from Visual Studio Code Extensions Tab - https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions
Install Azure Static Web Apps Extension from Visual Studio Code Extensions Tab - https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurestaticwebapps

Ensure you have .Net6 sdk (check by running command `dotnet --list-sdks`), even if you have other .Net sdks even a higher one such as .Net7. This is because Azure Functions have concept of In-process and Isolated worker process. This article will have steps supporting In-process whereas .Net7 is only supported in Isolated worker process.

Build Static Site

To get started a template in Github can be used. Go to https://github.com/login?return_to=/staticwebdev/vanilla-basic/generate and in the page add mycalendar in the Repository field and click on button Create Repository from Template to create repository.

Plain vanilla javascript template is used here. There are more templates including angular, react, vue, blazor available at https://github.com/staticwebdev

Open visual studio code and open a new terminal with bash.
Go to folder where you wish to do your development and run the following command to clone the github project to your local machine.



git clone https://github.com/<your_github_account>/mycalendar.git

In VSCode select File > Open Folder to open the cloned mycalendar repository

Delete files package.json, package-lock.json, playwright.config.ts, entire tests folder, entire .devcontainer folder and both files (playwright-onDemand.yml and playwright-scheduled.yml) in .github/workflows folder (but do not delete this folder)

Update Github Repository

At the command prompt in terminal, run the following command to update changes in workspace to staging area.



git add --all

Check the changes to be committed.



git status

Commit changes from staging to the local repository.



git commit -m "Initial commit to create base repository"

Push code to Github



git push -u origin main

Deploy to Azure

In Visual Studio Code, press F1 OR Ctrl+Shift+P to open Command Palette.
Search and Select Azure Static Web Apps:Create Static Web App...
In ensuing screens, select your subscription,
select an existing resource group or create a new one (if the screen is shown - not in below image),
decide and select on a free plan or standard plan (if the screen is shown - not in below image),
provide a name to the web app,
and then select the region to deploy.
The next screen provides a list of frontend frameworks. As the application is a vanilla javascript application, choose Custom.
Next, provide the location of application code - /src as this is where index.html resides.
Leave the API location blank for now (if the screen is shown - not in below image)
and finally provide the location of build output also /src (this is primarily useful if a framework such as angular, react, svelte, etc is used and in such cases, the build folder location goes here).

Finally, a message is shown in Azure Activity log stating that the Azure Static Web Site is created and a git pull is executed to download a workflow file from github and kept in .github/workflows folder. One of the best things about Static Web Apps is the fact that CI / CD is integrated using Github Action Workflows and anything that is now added to the workspace and then pushed to github repository will get deployed on Azure automatically.

Azure and Github credentials will have to be provided in the above process.

Check the all resources screen in Azure Portal to find the mycalendar static web app. On selecting the same, a page loads which shows the static webapp URL, the source code, action run history and the workflow which was pulled to the workspace (the last three are github links).

Confirm that the static website URL loads.

Create Application Interface

To create the calendar interface, the free and open-source DayPilot Lite library is used.

Download the library and add the daypilot-all.min.js to the src folder.

Update index.html code to the one below.



<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <script src="daypilot-all.min.js"></script>
  <title>My Calendar</title>
</head>

<body>
  <main>
    <h1>My Calendar</h1>
  </main>

  <div id="mycalendar"></div>

<script type="text/javascript">
    const mycalendar = new DayPilot.Month("mycalendar", {
        startDate: "2023-01-01",
        onTimeRangeSelected: async function (args) {

            const colors = [
                {name: "Blue", id: "#3c78d8"},
                {name: "Green", id: "#6aa84f"},
                {name: "Yellow", id: "#f1c232"},
                {name: "Red", id: "#cc0000"},
            ];

            const form = [
                {name: "Text", id: "text"},
                {name: "Start", id: "start", type: "datetime"},
                {name: "End", id: "end", type: "datetime"},
                {name: "Color", id: "barColor", options: colors}
            ];

            const data = {
                text: "Event",
                start: args.start,
                end: args.end,
                barColor: "#6aa84f"
            };

            const modal = await DayPilot.Modal.form(form, data);

            mycalendar.clearSelection();

            if (modal.canceled) {
                return;
            }

            mycalendar.events.add({
                start: modal.result.start,
                end: modal.result.end,
                id: DayPilot.guid(),
                text: modal.result.text,
                barColor: modal.result.barColor
            });
        }
    });

    mycalendar.events.list = [
    {
      "start": "2023-01-12T10:30:00",
      "end": "2023-01-12T15:30:00",
      "id": "225eb40f-5f78-b53b-0447-a885c8e92233",
      "text": "React Interview with Shirish Kumar",
      "barColor":"#cc0000"
    },
    {
      "start": "2023-01-16T12:30:00",
      "end": "2023-01-18T17:00:00",
      "id": "1f67def5-e1dd-57fc-2d39-eb7a5f8e789a",
      "text": "Kubernetes Interview with Ramesh Bhat",
      "barColor":"#3c78d8"
    },
    {
      "start": "2023-01-25T10:30:00",
      "end": "2023-01-25T16:00:00",
      "id": "aba78fd9-09d0-642e-612d-0e7e002c29f5",
      "text": "AAD Interview with Girish C",
      "barColor":"#cc0000"
    }
  ];

    mycalendar.init();


</script>
</body>

</html>

In the above code, a div with id mycalendar is added which is referenced in the javascript. DayPilot.Month ensures that the view is month based and the starting date of the calendar is stated using the startDate attribute.
onTimeRangeSelected section allows the user to add an event by either clicking on a single date or selecting multiple dates by dragging on screen. DayPilot.Modal.form provides a form with values to update and Save the event.
mycalendar.events.list adds existing event data by adding an array of data in a format the library expects.

In the bash terminal, go to mycalendar folder and run the following command.



swa start src

This command will be available if Static Web Apps CLI is installed. Testing and Debugging is one of the primary challenges with serverless as application is broken into smaller pieces and replicating the environment locally for developer is hard. Static Web Apps CLI solves this problem as we will see later. src is the folder where static content including HTML, images, javascript and stylesheets are kept.

On running the command Azure Static Web Apps emulator starts and the calendar application can be accessed at http://localhost:4280.

Confirm that events added in code can be viewed as well as new events can be added by clicking on a date.

Implement Authentication

Azure Static Web Apps has support for GitHub, Twitter, and Azure Active Directory for authentication by default. Moreover, Static Web Apps CLI provides authentication emulator to mock responses from the three providers mentioned.

To enable login using github update the HTML main content area (between tags) to the content below.



<main>
    <h1>My Calendar</h1>
    <p>
    <div id="login" style="display: flex; justify-content: end;"><a href="/.auth/login/github">Login</a></div>
</main>

After authentication, to get access to user information such as user id / email, Azure Static Web Apps provides an API endpoint which means not only do developers not have to implement and maintain any oauth related code but also the endpoint does not face serverless architecture challenges like cold start delays.

Update the <script> area with code below under mycalendar.init()



    mycalendar.init(); //Add the code below

    const app = {
      getUserInfo() {
      return fetch('/.auth/me')
        .then(response =>{
            return response.json();
          }).then(data =>{
            const { clientPrincipal } = data;
            console.log(clientPrincipal);
            if (clientPrincipal !=null){
              const userDetails= clientPrincipal.userDetails;
              return userDetails;
            }
            return null;
          })
      },
      init(){
        app.getUserInfo()
            .then(user =>{
             console.log(user); 
            })
      }
    };
    app.init();
  </script>

In the above code init function calls getUserInfo function which in turn calls the direct-access endpoint /.auth/me and from the resultant response gets the github userid (provided authenticated by github, else returns null) which gets logged in the browser console.

In the bash terminal, run the following command again.



swa start src

Load the web page in a browser in incognito mode. After clicking on the login button, the emulator mock screen comes up.

In the mock screen, add your first name in the Username field as shown in image (arnab is shown in image) and select Login.

The calendar interface shows up as Index.html is loaded. Open developer tools and go to Console. ClientPrincipal data as well as Username is shown as in the image below.

Create API and integrate with Static Site

Running logic on the browser has certain limitations namely the ability to connect to data stores / databases to persist data and then retrieve the same. That is where the necessity to run some part of the code server side comes in.
Azure Static Web Apps supports serverless API endpoints powered by Azure Functions where HTTP request triggers the function. The API route is fixed at /api. Also, Azure Static Web Apps extension for Visual Studio Code creates the Function templates in api folder by default. Also, in local development environment API will run in port 7071 and not port 4280 where the static site runs. This in normal cases will lead to Cross-Origin Resource Sharing (CORS) errors like Access to XMLHttpRequest at ""http://localhost:7071/api/events"" from origin ""http://localhost:4280"" has been blocked by CORS policy. But, Azure Static Web Apps (using Reverse Proxy) as well as the CLI (for local development scenario) takes care of this challenge as it makes the static web app and API appear to come from the same domain.

The calendar application allows users to view their existing events as well as add new events to the calendar.

For viewing existing events use case, GET method at route endpoint events will be used which means that the full api endpoint at the static site will be api/events
For adding new events use case, POST method at the same endpoint can be used.

View existing events use case

In Visual Studio Code, press F1 OR Ctrl+Shift+P to open Command Palette. Search and Select Azure Static Web Apps: Create HTTP Function.
In ensuing screens select C# as language,
add GetEvents as Function Name,
add <Your_First_Name>.MyCalendar as Namespace (Arnab.MyCalendar in my case)
and Anonymous as Access Rights (Good enough for POC scenarios but never use this setting in production).

Security is important. Do check out Azure Architecture - Serverless Functions security

A new folder in the workspace gets created named api and a Functions project gets created.

A simple piece of code is written next just sufficient to test the integration of API with static site.

In GetEvents.cs file update the contents to the code below.



using System;
using System.IO;
using System.Threading.Tasks;
using System.Collections.Generic;
using System.Text.Json;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;


namespace Arnab.MyCalendar
{
    public static class GetEvents
    {
        [FunctionName("GetEvents")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "events")] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("Get Events list");
            string username = req.Query["u"];
            log.LogInformation(name);
            string json = @"[
                        {
                        ""start"": ""2023-01-12T10:30:00"",
                        ""end"": ""2023-01-12T15:30:00"",
                        ""id"": ""225eb40f-5f78-b53b-0447-a885c8e92233"",
                        ""text"": ""React Interview with Shirish Kumar"",
                        ""barColor"":""#cc0000""
                        },
                        {
                        ""start"": ""2023-01-16T12:30:00"",
                        ""end"": ""2023-01-18T17:00:00"",
                        ""id"": ""1f67def5-e1dd-57fc-2d39-eb7a5f8e789a"",
                        ""text"": ""Kubernetes Interview with Ramesh Bhat"",
                        ""barColor"":""#3c78d8""
                        },
                        {
                        ""start"": ""2023-01-25T10:30:00"",
                        ""end"": ""2023-01-25T16:00:00"",
                        ""id"": ""aba78fd9-09d0-642e-612d-0e7e002c29f5"",
                        ""text"": ""AAD Interview with Girish C"",
                        ""barColor"":""#cc0000""
                        }
                    ]";
            List<Dictionary<string, string>> results=null;
            if (username == "arnab"){
            results =JsonSerializer.Deserialize<List<Dictionary<string, string>>>(json);
            }
            return new OkObjectResult(results);
        }
    }
}

In the above code we first specify the HTTP method (GET) and route (events).
The API accepts a querystring 'u' which contains the username value.
There is a hardcoded json string on similar lines as in index.html which is deserialized and returned provided the username is equal to a hardcoded value.

The hardcoded value here is arnab but you should update that to your first name provided you are going to use that as username in mock authentication screen. Also do remember to change the namespace to `<Your_First_Name>.MyCalendar`.

To call this API from frontend, update index.html. Block comment or remove mycalendar.events.list code, add a loadEvents function which will call the API and call this function from init as code shown below.



/* mycalendar.events.list = [
      {
        "start": "2023-01-12T10:30:00",
        "end": "2023-01-12T15:30:00",
        "id": "225eb40f-5f78-b53b-0447-a885c8e92233",
        "text": "React Training",
        "barColor":"#cc0000"
      },
      //more data below
    ]; */

    mycalendar.init();

    const app = {
      loadEvents(user) {
        console.log(user);
        var url = new URL('/api/events')
        var params = {u:user}
        url.search = new URLSearchParams(params).toString();
        //console.log(url);
        fetch(url)
          .then(response =>{
            return response.json();
          }).then(data =>{
            //console.log(data);
            mycalendar.update({
              events: data
            });
          })                       
      }, //next there will be getUserInfo()
      init(){
        app.getUserInfo()
            .then(user =>{
             console.log(user); 
             if (user !=null){
              app.loadEvents(user);  
              }
            })
      }
    };
    app.init();

In the code above getUserInfo method is called and if usename is not null, loadEvents method is called where the username is added to the API endpoint as querystring / search parameters and the return data is updated in the calendar.

To test the API and its integration with the static site, in the bash terminal, run the following command.



swa start src --api-location api

When the web site loads, login with as username in the ensuing emulator mock screen. The calendar gets updated with data but this time the data is sent from server.

In case you face any challenge / errors in running the above SWA CLI command, open a new bash terminal, ensure you are in api folder, update API urls in index.html from '`/api/events`' to '`http://localhost:7071/api/events`' and run the following command to run just the Azure function and in the first terminal run swa start src as before. This style of running is also useful in debugging as you bifurcate the running of frontend site and backend API. But, do remember to switch back the urls to its earlier form before publishing the application to Azure.



func start --cors http://localhost:4280 --port 7071

Add new events use case

Create a new HTTP Function as before and name this PostEvents.

In PostEvents.cs file update the contents to the code below.



using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;

namespace Arnab.MyCalendar
{
    public static class PostEvents
    {
        [FunctionName("PostEvents")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "events")] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("Post Event");

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            dynamic data = JsonConvert.DeserializeObject(requestBody);
            string cuser = data?.cuser;
            dynamic cevent=data?.cevent;
            log.LogInformation(cuser);
            string eguid =Guid.NewGuid().ToString(); 
            cevent.id=eguid;
            return new OkObjectResult(cevent);
        }
    }
}

In the above code, username and event data is being retrieved from request body, a Guid is created and added to the event data and the event data with Guid is returned.

To call this API from frontend, update index.html. Add an addEvents function and update the mycalendar onTimeRangeSelected function.The final script section in index.html is as code shown below



  <script type="text/javascript">
      const mycalendar = new DayPilot.Month("mycalendar", {
          startDate: "2023-01-01",
          onTimeRangeSelected: async function (args) {

              const colors = [
                  {name: "Blue", id: "#3c78d8"},
                  {name: "Green", id: "#6aa84f"},
                  {name: "Yellow", id: "#f1c232"},
                  {name: "Red", id: "#cc0000"},
              ];

              const form = [
                  {name: "Text", id: "text"},
                  {name: "Start", id: "start", type: "datetime"},
                  {name: "End", id: "end", type: "datetime"},
                  {name: "Color", id: "barColor", options: colors}
              ];

              const data = {
                  text: "Event",
                  start: args.start,
                  end: args.end,
                  barColor: "#6aa84f"
              };
              const modal = await DayPilot.Modal.form(form, data);

              mycalendar.clearSelection();

              if (modal.canceled) {
                  return;
              }
              app.getUserInfo()
              .then(user =>{
                console.log(user); 
                if (user !=null){
                  const event = {
                    start: modal.result.start,
                    end: modal.result.end,
                    text: modal.result.text,
                    barColor: modal.result.barColor
                  };
                  app.addEvents(user,event);  
                }
              })

          }
      });


    mycalendar.init();

    const app = {
      loadEvents(user) {
        console.log(user);
        var url = new URL('/api/events')
        var params = {u:user}
        url.search = new URLSearchParams(params).toString();
        //console.log(url);
        fetch(url)
          .then(response =>{
            return response.json();
          }).then(data =>{
            //console.log(data);
            mycalendar.update({
              events: data
            });
          })                       
      },
      addEvents(user,event){
        fetch('/api/events', {
                method: 'POST',
                headers: {
                    'Accept': 'application/json',
                    'Content-Type': 'application/json'
                },
                body: JSON.stringify({
                  cuser:user,
                  cevent:event,
                }),
            })
            .then(response =>{
              console.log(response);
            return response.json();
            }).then(data =>{
              console.log(data);
              mycalendar.events.add(data);
            }) 
      },
      getUserInfo() {
      return fetch('/.auth/me')
        .then(response =>{
            return response.json();
          }).then(data =>{
            const { clientPrincipal } = data;
            console.log(clientPrincipal);
            if (clientPrincipal !=null){
              console.log("inside clientprincipal not null");
              const userDetails= clientPrincipal.userDetails;
              return userDetails;
            }
            return null;
          })
      },
      init(){
        app.getUserInfo()
            .then(user =>{
             console.log(user); 
             if (user !=null){
              document.getElementById("login").style.display = "none";
              app.loadEvents(user);  
              }
            })
      }
    };
    app.init();
  </script>

The new code in onTimeRangeSelected calls getUserInfo and if username is not null sends the username and event data to addEvents function which makes the POST call to the API and updates the calendar with return data.

Once again test the API and its integration as before to confirm that the code works.

Implement Persistence layer with Cosmos DB

Run the Cosmos DB emulator. Right click and select Open Data Explorer.

Run the following command in the bash terminal to configure the connection string (available from Primary Connection String in Explorer screen) in the function project settings in the local.settings.json file.



func settings add CosmosDBConnection "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==" --connectionString

local.settings.json file should now be added to the connectionstring as below.



  "ConnectionStrings": {
    "CosmosDBConnection": {
      "ConnectionString": "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==",
      "ProviderName": "System.Data.SqlClient"
    }
  }

Emulator connection string is same always unless changed manually. Also local.settings.json is not pushed to github and the connection string there is only useful for local development. The command / process to add connection string in Azure is different and we will see so later.

In the emulator screen select Explorer and then New Container.
Add a new Database with name myCalendar, state the Container Name as eventsCollection and Partition Key /userName.

The database and Container can be viewed now in Explorer.

An easy declarative way to connect Azure services including Cosmos DB to Azure functions is using bindings. Bindings are implemented in extension packages. Run the following dotnet add package command in the terminal to install the Cosmos DB extension package.



dotnet add package Microsoft.Azure.WebJobs.Extensions.CosmosDB --version 4.0.0

Add a new file in api folder and name the same event.cs. This file will have two classes, one in the data structure format in which data arrives from frontend and the other is a structure format as the JSON data that will be persisted in Cosmos DB.



using System;
namespace Arnab.MyCalendar
{
    public class CalendarEvent
    {
        public string id { get; set; }
        public string userName{ get; set; }
        public string startsAt { get; set; }
        public string endsAt { get; set; }
    #nullable enable
        public string? eventTitle { get; set; }
        public string? barColor { get; set; }
        public DateTime eventCreateDate{ get; set; }
    }

    public class ClientPostEvent
    {
        #nullable enable
        public string? id { get; set; }
        public string? start { get; set; }
        public string? end { get; set; }
        public string? text { get; set; }
        public string? barColor { get; set; }
    }

    public class ClientData
    {
        public string cuser { get; set; }
        public ClientPostEvent cevent { get; set; }
    }
}

In the code above CalendarEvent will be used to persist data in Cosmos DB.

Update the contents of PostEvents.cs to the code below.



using System;
using System.IO;
using System.Text.Json;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;

namespace Arnab.MyCalendar
{
    public static class PostEvents
    {
        [FunctionName("PostEvents")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "events")] HttpRequest req,
            [CosmosDB(
            databaseName: "myCalendar",
            containerName: "eventsCollection",
            Connection = "CosmosDBConnection")]
            IAsyncCollector<CalendarEvent> eventsOut,ILogger log)
        {
            log.LogInformation("Post Event");

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            ClientData data = JsonSerializer.Deserialize<ClientData>(requestBody);
            string cuser = data?.cuser;
            ClientPostEvent cevent=data?.cevent;
            log.LogInformation(cuser);
            string eguid =Guid.NewGuid().ToString(); 
            cevent.id=eguid;
            CalendarEvent nevent = new CalendarEvent() { 
                id = cevent.id,
                userName= cuser,
                startsAt=cevent.start,
                endsAt=cevent.end,
                eventTitle=cevent.text,
                barColor=cevent.barColor,
                eventCreateDate=DateTime.Now
                };

            await eventsOut.AddAsync(nevent);
            return new OkObjectResult(cevent);
        }
    }
}

In the code above, the presence of bindings enable in connecting to database seamlessly with the attribute CosmosDB. Another parameter eventsOut is of type IAsyncCollector<CalendarEvent> ensuring that any instance of it calling AddAsync gets an instance of CalendarEvent persisted in the database.

Update the contents of GetEvents.cs to the code below.



using System;
using System.IO;
using System.Threading.Tasks;
using System.Collections.Generic;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;

namespace Arnab.MyCalendar
{
     public static class GetEvents
    {
        [FunctionName("GetEvents")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "events")] HttpRequest req,
            [CosmosDB(
                databaseName: "myCalendar",
                containerName: "eventsCollection",
                Connection = "CosmosDBConnection")] CosmosClient client,
            ILogger log)
        {
            log.LogInformation("Get Events list");
            string name = req.Query["u"];

            List<Dictionary<string, string>> results=new  List<Dictionary<string, string>>();
            Container myContainer = client.GetDatabase("myCalendar").GetContainer("eventsCollection");
            QueryDefinition queryDefinition = new QueryDefinition(
                "SELECT * FROM items i WHERE (i.userName = @searchterm)")
                .WithParameter("@searchterm", name);
            string continuationToken = null;
            do
            {
            FeedIterator<CalendarEvent> feedIterator = 
                myContainer.GetItemQueryIterator<CalendarEvent>(
                        queryDefinition, 
                        continuationToken: continuationToken);

                while (feedIterator.HasMoreResults)
                {
                    FeedResponse<CalendarEvent> feedResponse = await feedIterator.ReadNextAsync();
                    continuationToken = feedResponse.ContinuationToken;
                    foreach (CalendarEvent item in feedResponse)
                    {
                        results.Add(new Dictionary<string, string>(){
                                    {"start", item.startsAt},
                                    {"end", item.endsAt},
                                    {"id", item.id},
                                    {"text", item.eventTitle},
                                    {"barColor", item.barColor}
                        });
                    }
                }
            } while (continuationToken != null);

            return new OkObjectResult(results);

        }
    } 
}

In the above code, a Azure Cosmos DB binding provided CosmosClient instance, available in extension version 4.x, is used to read a list of documents. With the access of CosmosClient instance, one can do complex stuff with Cosmos DB in Static Web Applications / Functions.
Here, CosmosClient object is used to configure and execute requests against the Azure Cosmos DB service. Database is the reference to database and Container a reference to container and they both are validated serverside.
QueryDefinition helps with the query and its parameters. FeedIterator<> helps in tracking the current page of results and getting a new page of results while FeedResponse<> represents a single page of responses which is iterated over using a foreach loop.
Also notice the usage of continuationToken and .WithParameter goodness.

Though this code gets all events of a user, in production scenarios you would not wish to do that, and along with username also send start date and end date in query string to ensure limited amount of data is accessed or retrieved.

To test open a new browser in incognito / inPrivate mode, login and confirm that you are able to add new events.
To check whether the events are being persisted, check the Cosmos DB emulator explorer screen to confirm data is being persisted.

Next close the browser and open another browser again in incognito / inPrivate mode, login and confirm that you are able to view the events added earlier and persisted in Cosmos DB.

Deploy application to Azure

Provision Database

Log in to Azure Portal, Search for Cosmos DB and select the top result.
Next create Cosmos DB account by selecting Create under Azure Cosmos DB For NoSQL box.

In the next screen, select subscription, select an existing Resource Group or create a new one, add an account name, choose the nearest location and choose Serverless in capacity mode.
Backup policy can be changed as well and locally redundant backups selected (sufficient for POC). Selecting Review + Create button provisions the database.

One could also try Cosmos DB by going to cosmos.azure.com/try. Selecting the account type - Azure Cosmos DB For NoSQL would create a trial account for 30 days which would open in Azure Portal.

Once Azure Cosmos DB account page opens up, select the Data Explorer tab and a similar interface as the Data Explorer emulator opens up. Select New Container box, Add a new Database with name myCalendar, state the Container Name as eventsCollection and Partition Key /userName.
Once the database is created, select the Connect box to view the connection string (OR select Key under Settings in left bar).

Copy the same and update the local.settings.json connection string in Visual Studio Code.
Run the application and open it in a browser, add new events and confirm that they persisted in Azure Cosmos DB by viewing them in Azure Cosmos DB data explorer.

The CI / CD Magic

The workflow file in .github/workflows folder needs to be updated to let know the location of api code.
Search for api_location in Repository/Build Configurations section and update the value to /api.

Cosmos DB connection string was configured in local.settings.json file for local development. But, this file is not available in production environment.
To configure the connection string in production, go to mycalendar Static Web App page in Azure Portal, and select Configuration from the left hand bar. Click Add and in the next screen add CosmosDBConnection as Name and Azure Cosmos DB connection string as Value. Click OK and then Save.

In Visual Studio Code Terminal, open a command prompt, and from mycalendar folder run the following command to commit all code changes and push them to Github.

Though in this article / post, this step is done at the very end, in application development lifecycle, this ought to be done after each small change (after interface design, after authentication implementation, after API addition, etc )



git add -A 
git status
git commit -m "Final Code with GET, POST, Auth and Azure Cosmos DB"
git push -u origin main

And then the magic happens in the Actions tab in github repository.

I would strongly urge all to check the steps (Build Azure and Build and Deploy) to understand all that happens automatically.

Once this step completes, the application is built on Azure and is ready for users.

Run Application

Copy the URL from mycalendar Static Web Apps page in the Azure Portal and paste it in a new browser window / tab.
On clicking Login, the application does not show the mock screen any more and goes to github to authenticate.

Then there is a consent screen from Microsoft and then the Login steps are complete.
Add new events, confirm that they persisted (Azure Cosmos DB data explorer) and then open the page in another tab to check the loading of existing events.

Challenges

Deploying a modern web application ain't easy.

Even though in this article / post, vanilla javascript is used (to ensure that this article is useful to all developers with any framework skills such as angular, react, svelte, etc.), in production use cases some framework would be used. This application will have to be built and bundles generated.

While building the application all routes must be carefully configured to ensure users do not receive 404 errors.

APIs would have to be built and it is very much possible that different APIs are built using different technologies / languages.

User authentication will have to be implemented as well as APIs secured.

So, 'multiple servers' or even 'a cluster of servers' would have to be setup (to ensure reliability and availability) to host these bundles and APIs and if the bundles and APIs are hosted on different servers, CORS would have to be implemented or a Reverse Proxy configured. Also, a global CDN for the frontend bundles will be needed.

SSL will have to be configured as well as the ability to have custom domains added.

When maintainability of the application over a period of time is taken into account, a staging environment (similar to production) as well as an automated build process is necessary as well.

Earlier even if one used cloud services, s/he would have to do all these.

Azure Static Web Apps takes care of all the above challenges and ensures that the development team can focus on business requirements.

Business Benefit

The application developed can be used as a base for any scheduling application. Also, any application built on the lines mentioned in the article / post using Serverless technologies like Static Web Apps and Cosmos DB is beneficial as no necessity of server management, charges for consumed storage only, better scalability, lower latency, quick updates, IDE support (VS Code and SWA CLI), etc all lead to accelerated innovation.

Final Notes

Serverless on Azure - https://azure.microsoft.com/en-in/solutions/serverless/
Azure Static Web Apps - https://azure.microsoft.com/en-gb/products/app-service/static
Azure Cosmos DB - https://azure.microsoft.com/en-in/products/cosmos-db/
SWA Templates - https://github.com/staticwebdev
DayPilot Lite library - https://javascript.daypilot.org/download/
Source Code Repository - https://github.com/c-arnab/mycalendar

Real time analytics using .Net & Redis

c-arnab — Tue, 26 Jul 2022 11:43:00 +0000

In this post, we look at RedisTimeSeries module and build a custom web analytic tool with Redis & .NET 5 using C#

Application architecture Overview

Why use RedisTimeSeries module

Redis has been used for analytics applications since long and a search on platforms like github will provide multiple projects on similar lines.
The last couple of years has seen Redis being extended with modern data models and data processing tools (Document, Graph, Search, and Time Series).
These extensions enables development of certain kind of applications including near real time analytics applications much much easier as well as simpler.

RedisTimeSeries is a Redis module that adds a time series data structure to Redis with features such as...

High volume inserts, low latency reads

Query by start time and end-time

Aggregated queries (min, max, avg, sum, range, count, first, last, STD.P, STD.S, Var.P, Var.S, twa) for any time bucket

Configurable maximum retention period

Downsampling / compaction for automatically updated aggregated timeseries

Secondary indexing for time series entries. Each time series has labels (field value pairs) which will allows to query by labels

You will see most of these features in the demo application being developed.
A strong advise would be to study the Commands available which can be run from Redis CLI. All these commands map to the methods available in C# via NRedisTimeSeries - .Net Client for RedisTimeSeries

At a basic level, one creates a new time series using TS.CREATE and provide the retention period for each data point that goes in this time series.
You might want to delete the key prior to using TS.CREATE to ensure you do not receive an error in case the key exists.

Append a sample (data point) to the time series using TS.ADD

Access the last value using TS.GET or do aggregated queries using different methods.

Application architecture

In web analytic applications, at a high level, there are trackers /loggers, either javascript (in browser) or C#/Java/Python/NodeJS (serverside) or mobile app sdks which generate data. This data passes to a messaging layer such as kafka or pulsar or even redis and is consumed by applications to collect, validate, enrich and finally store in data stores where they would be analyzed.

If one is developing an IOT analytics application, the process would be similar but the data would in this case be generated by various sensors.

These trackers generate data from user events and sends lots of event attributes to the collector per event. For the demo application, to keep things simple, mock data is generated for a very limited number of attributes.

I call this application Redmetrix.

For the demo, this mock data is ingested by a .Net console application- Redmetrix Processor which saves the mock data in redis (in time series data structure).
The log files are in a format where every line is a valid json.
Ingestor reads the files line by line and adds data to Redis.

In the real world applications, one might have this logic preferably in azure function / aws lambda which would scale up/ down as needed.

Finally, we have a ASP.NET Core 5.0 web application using razor pages- Redmetrix Webapp which reads this data from redis and displays charts in a dashboard.

Though, in this article, we get the data in the page's codebehind file, in real world scenarios one would have a background worker which calls a service at specific intervals to get the necessary data from redis and sends the data via signalr to the clients using websockets and server side events.

Charts in Dashboard

The final article and the corresponding code in github repository will have the following charts

Total PageViews (in last min) & comparison with a min prior
Total Transactions
Total Order Value (in Rupees)
Page performance for Home page and Product page (in last min)
Conversion Funnel (in last min)
Orders by Payment Method (in last min)

Demo Infrastructure

Create Database on Redis Cloud

Select New DataBase Button.

Name the new database redmetrix and select Redis Stack radio button.

In the ensuing page where it shows the database configuration, please copy the public endpoint and password from General and Security section and keep it safe.

Generate Mock Data

One can use applications like Mockaroo to create dummy data.
Check out https://www.mockaroo.com/7d448f50 to see the one going to be used here.

The attributes used are

application.app_id - Application Id, Can be a String or even a number
application.platform - Denote whether a website or mobile app
date_time.collector_tstamp_dt - DataTime when the event reached messaging layer or was enriched
event_detail.event_type- Page Views or Form Change/Submit or Ecom Trasaction via event
event_detail.event_id - Event Id- Guid
contexts.page.type - Home/Category/Product. etc..
contexts.performance_timing.domContentLoadedEventEnd - Page load end timing
contexts.performance_timing.requestStart - Page load start timing
contexts.transaction.amount_to_pay - Transaction Amount
contexts.transaction.payment_method - Payment Method

For simplicity, all event types are Page Views but in real world applications, lions share of them are not going to be Page Views.

Also, Amount to Pay and Payment Method should be part of the event if and only if Page type is 'Success Confirmation'. But in the dummy data you would find these fields present in all events.

The Date field is randomly generated but in real world its going to be sequential. As we are not going to directly do anything with this field, this will not create any problems.

After downloading the mock data a couple of times, save them in a folder after renaming them as .txt files as though each line has json structure, the whole file does not(The file is not an array of json objects).

Developing Redmetrix

Visual Studio Code will be used for development in this article.

Processor

Create a base Console Application in the Terminal



dotnet new console -n "RedMetrixProcessor"

Move inside the RedMetrixProcessor directory



cd RedMetrixProcessor

Build and Run the application to ensure everything is working fine.



dotnet build
dotnet run

Open the folder RedMetrixProcessor in VSCode and add a New file named appsettings.json

Now copy the saved public endpoint and password from database configuration in the file as shown in code block



{
    "RedisOptions":{
        "EndPoints":"redis-14475.c280.us-central1-2.gce.cloud.redislabs.com:14475",
        "Password":"BVDwMO6Qj1tnDZalcxmfdH2mL1c1G5iA"
    }
}

The next step is to install the required nuget packages



dotnet add package Microsoft.Extensions.Hosting
dotnet add package Microsoft.Extensions.Configuration.Json

Normally console applications are not visually appealing. So, I will use a package Spectre.Console to make it colorful.

I'll not be getting into the features or the code parts of Spectre in this article. But the code is going to be available on github. Do check it out.



dotnet add package Spectre.Console

Finally, the packages required for Redis and RedisTimeSeries



dotnet add package StackExchange.Redis
dotnet add package NRedisTimeSeries

The above steps will add the below codeblock to your RedMetrixProcessor.csproj file



<ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="6.0.0" />
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
    <PackageReference Include="NRedisTimeSeries" Version="1.4.0" />
    <PackageReference Include="Spectre.Console" Version="0.44.0" />
    <PackageReference Include="StackExchange.Redis" Version="2.6.48" />
  </ItemGroup>

Add the following code to ensure that appsettings.json gets copied during build/run.



  <ItemGroup>
   <None Update="appsettings.json">
     <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
   </None>
  </ItemGroup>

Update the Program.cs code to allow access to appsettings.json and connect to Redis DB.



using System;
using System.IO;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Spectre.Console;
using NRedisTimeSeries;
using NRedisTimeSeries.DataTypes;
using StackExchange.Redis;
namespace RedMetrixProcessor
{
    class Program
    {
        public static IConfigurationRoot configuration;
        static void Main(string[] args)
        {
            ServiceCollection serviceDescriptors = new ServiceCollection();

            ConfigureServices(serviceDescriptors);
            IServiceProvider serviceProvider = serviceDescriptors.BuildServiceProvider();

            var options = new ConfigurationOptions
                {
                    EndPoints = { configuration.GetSection("RedisOptions:EndPoints").Value },
                    Password = configuration.GetSection("RedisOptions:Password").Value,
                    Ssl = false
                };
            // Multiplexer is intended to be reused
            ConnectionMultiplexer redisMultiplexer = ConnectionMultiplexer.Connect(options);

            // The database reference is a lightweight passthrough object intended to be used and discarded
            IDatabase db = redisMultiplexer.GetDatabase();
            Console.WriteLine("Hello World!");
        }
        private static void ConfigureServices(IServiceCollection serviceCollection)
        {
            configuration = new ConfigurationBuilder()
                .SetBasePath(Directory.GetParent(AppContext.BaseDirectory).FullName)
                .AddJsonFile("appsettings.json")
                .Build();

            serviceCollection.AddSingleton<IConfigurationRoot>(configuration);


        }
    }
}

Create a new file ConfigureInitializationServices.cs with two methods DeleteKeys which deletes all data from database (if any data is present) and InitializeTimeSeriesTotalPageViews which creates a time series for pageviews with a retention period of 10 mins for each data point / sample.



using System;
using System.Collections.Generic;
using Microsoft.Extensions.Configuration;
using Spectre.Console;
using NRedisTimeSeries;
using NRedisTimeSeries.DataTypes;
using NRedisTimeSeries.Commands;
using StackExchange.Redis;
namespace RedMetrixProcessor
{
    public class ConfigureInitializationServices
    {  
        private  readonly IConfigurationRoot _config;

        public ConfigureInitializationServices(IConfigurationRoot config)
        {
            _config = config;
        }

        public void DeleteKeys(IDatabase db)
        {         


                db.KeyDelete("ts_pv:t"); //TimeSeries-PageView-Total

        } 


        public void InitializeTimeSeriesTotalPageViews(IDatabase db)
        {

                db.TimeSeriesCreate("ts_pv:t", retentionTime: 600000);


        }

    }
}

The methods in github code repository is a lot more extensive with `try catch blocks` and `Spectre.Console` code.

Back in Program.cs, Update ConfigureServices method to ensure methods in ConfigureInitializationServices can be accessed from the mainmethod.



using System;
using System.IO;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Spectre.Console;
using NRedisTimeSeries;
using NRedisTimeSeries.DataTypes;
using StackExchange.Redis;
namespace RedMetrixProcessor
{
    class Program
    {
        public static IConfigurationRoot configuration;
        static void Main(string[] args)
        {
            ServiceCollection serviceDescriptors = new ServiceCollection();

            ConfigureServices(serviceDescriptors);
            IServiceProvider serviceProvider = serviceDescriptors.BuildServiceProvider();

            var options = new ConfigurationOptions
                {
                    EndPoints = { configuration.GetSection("RedisOptions:EndPoints").Value },
                    Password = configuration.GetSection("RedisOptions:Password").Value,
                    Ssl = false
                };
            // Multiplexer is intended to be reused
            ConnectionMultiplexer redisMultiplexer = ConnectionMultiplexer.Connect(options);

            // The database reference is a lightweight passthrough object intended to be used and discarded
            IDatabase db = redisMultiplexer.GetDatabase();

            AnsiConsole.Write(new FigletText("RedMetrix").LeftAligned().Color(Color.Red));
            AnsiConsole.Write(new Markup("[bold red]Copyright(C)[/] [teal]2021 Arnab Choudhuri - Xanadu[/]"));
            Console.WriteLine("");
            var rule = new Rule("[red]Welcome to RedMetrix[/]");
            AnsiConsole.Write(rule);

            var selectedoption = AnsiConsole.Prompt(
                                    new SelectionPrompt<string>()
                                        .Title("[bold yellow]Intitialize Application[/] [red]OR[/] [green]Process Data[/]?")
                                        .PageSize(5)
                                        .AddChoices(new[]
                                        {
                                            "Initialize Application", "Process Data", "Exit"
                                        }));
            if (selectedoption.ToString()=="Exit")
            {
                return;
            }else{
                if (!AnsiConsole.Confirm(selectedoption.ToString()))
                {
                    return;
                }
                else{
                     if (selectedoption.ToString()=="Initialize Application")
                     {
                        serviceProvider.GetService<ConfigureInitializationServices>().DeleteKeys(db);
                        serviceProvider.GetService<ConfigureInitializationServices>().InitializeTimeSeriesTotalPageViews(db);

                     }

                }
            }
            redisMultiplexer.Close();
            AnsiConsole.Write(new Markup("[bold yellow]Press any key to [/] [red]Exit![/]"));
            Console.ReadKey(false);
        }
        private static void ConfigureServices(IServiceCollection serviceCollection)
        {
            configuration = new ConfigurationBuilder()
                .SetBasePath(Directory.GetParent(AppContext.BaseDirectory).FullName)
                .AddJsonFile("appsettings.json")
                .Build();

            serviceCollection.AddSingleton<IConfigurationRoot>(configuration);
            serviceCollection.AddTransient<ConfigureInitializationServices>();


        }
    }
}

The application must be able to access the mock data files downloaded earlier from the development machine where this console application will run. To ensure the same, its time to add the folder location for the mock data file in appsettings.json.



{
    "RedisOptions":{
        "EndPoints":"redis-14475.c280.us-central1-2.gce.cloud.redislabs.com:14475",
        "Password":"BVDwMO6Qj1tnDZalcxmfdH2mL1c1G5iA"
    },
    "FolderPath":"D:\\redis\\article\\data"
}

Create a new file DataServices.cs which will traverse the folder for mock data file/s, read each line, and if the event type is 'Page View', add a datapoint with current timestamp and value 1(One). Do note the TSPageViews method.



using System;
using System.IO;
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.Configuration;
using Spectre.Console;
using NRedisTimeSeries;
using StackExchange.Redis;
namespace RedMetrixProcessor
{
    public class DataServices
    {
        private  readonly IConfigurationRoot _config;

        public DataServices(IConfigurationRoot config)
        {
            _config = config;
        }

        public void ProcessData(IDatabase db)
        {
            string folderPath = _config.GetSection("FolderPath").Value;
            DirectoryInfo startDir = new DirectoryInfo(folderPath);
            var files = startDir.EnumerateFiles();
            AnsiConsole.Status()
                .AutoRefresh(true)
                .Spinner(Spinner.Known.Default)
                .Start("[yellow]Initializing ...[/]", ctx =>
                {
                    foreach (var file in files)
                    {
                        ctx.Status("[bold blue]Started Processing..[/]");
                        HandleFile(file,db);
                        AnsiConsole.MarkupLine($"[grey]LOG:[/] Done[grey]...[/]");
                    }
                });

            // Done
            AnsiConsole.MarkupLine("[bold green]Finished[/]");
        }

        private static void HandleFile(FileInfo file,IDatabase db)
        {

               Console.WriteLine(file.FullName);

              using var fs = new FileStream(file.FullName, FileMode.Open, FileAccess.Read);
              using var sr = new StreamReader(fs, Encoding.UTF8);
               // int count = 0;
                //int total =0;

                string line = String.Empty;

                while ((line = sr.ReadLine()) != null)
                {
                    try{

                        using JsonDocument doc = JsonDocument.Parse(line);
                        if(doc.RootElement.GetProperty("event_detail").GetProperty("event_type").GetString()=="Page view")
                        {
                                //total++;
                                TSPageViews(db);


                        }


                    }catch(Exception ex){
                       AnsiConsole.WriteException(ex);
                     Console.WriteLine(line); 

                    }finally{

                    }

                }
           // Console.WriteLine($"{total}");
        }   

        private static void TSPageViews( IDatabase db)
        {
            db.TimeSeriesAdd("ts_pv:t", "*", 1);
        }

    }
}

Once again in Program.cs, Update ConfigureServices method to ensure methods in DataServices can be accessed from the mainmethod.



serviceCollection.AddTransient<ConfigureInitializationServices>();
serviceCollection.AddTransient<DataServices>();



if (selectedoption.ToString()=="Initialize Application")
    {
        serviceProvider.GetService<ConfigureInitializationServices>().DeleteKeys(db);
        serviceProvider.GetService<ConfigureInitializationServices>().InitializeTimeSeriesTotalPageViews(db);

    }
if (selectedoption.ToString()=="Process Data")
    {
        serviceProvider.GetService<DataServices>().ProcessData(db);
    }

Now to try what is built by building and then running the application.



dotnet build
dotnet run

Select Intitialize Application from the choices and confirm the same.

Now to check if the application did what it was supposed to do using RedisInsight.

Install and Run RedisInsight

Select the option of Add Database manually

Redis Database gets listed in Redis Insight

Select Database Alias value -redmetrix to view the Key just added.

Run the application once again and this time select Process Data

Confirm that the mock data files have been processed by running the query below to check for the number of data points/ samples in the database.



TS.RANGE ts_pv:t - + AGGREGATION sum 6000000

Later the same query will be run via .Net client in the web application.

Web Application

Create a base Razor Pages Web Application in the Terminal



dotnet new webapp -o RedMetrixWebApp

Move inside the directory and then build and run to confirm all works well.

In appsettings.json, add the RedisOptions section as is in RedMetrixProcessor.

Add the required nuget packages



dotnet add package StackExchange.Redis.Extensions.AspNetCore
dotnet add package StackExchange.Redis.Extensions.System.Text.Json
dotnet add package NRedisTimeSeries

Update the Program.cs code to allow access to appsettings.json



using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

namespace RedMetrixWebApp
{
    public class Program
    {
        private static IConfigurationRoot Configuration { get; set; }
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration((hostContext, config) =>
               {
                   config.AddJsonFile("appsettings.json");
               })
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                    webBuilder.UseUrls("http://localhost:5003");
                });


    }
}

In Startup.cs file comment the line app.UseHttpsRedirection(); as https is not going to be used.

Now RedisProvider is added to the solution in the folder Provider.

Update Startup.cs - ConfigureServices method and add line



services.AddSingleton<RedisProvider>();

Do remember to add the namespace of `RedisProvider` to `Startup.cs`

Next, its time to get data for the PageViews Widget.
This will show Total PageViews (in last min) & comparison with a min prior, so two values.

So, there is a need to create a model to hold two values.
Create a new folder named Model and in that create a file RealTimeChart.cs.



using System;
using System.Collections.Generic;
namespace RedMetrixWebApp
{
    public class RealtimeChart
    {
        public PageView PageViews { get; set; }

        public DateTime Updated { get; set; }
    }

    public class PageView   
    {
        public PageView(long pv, long prev_pv)
        {
            this.pv=pv;
            this.prev_pv=prev_pv;
        }
        public long pv { get; set; }
        public long prev_pv{ get; set; }
    }
}

Next, create a new folder named Services and in that create a file RealTimeChartService.cs.



using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Linq;
using dotnetredis.Providers;
using NRedisTimeSeries;
using NRedisTimeSeries.Commands;
using NRedisTimeSeries.DataTypes;
using StackExchange.Redis;
using Microsoft.Extensions.Logging;


namespace RedMetrixWebApp
{
    public class RealtimeChartService
    {
        private readonly ILogger<RealtimeChartService> _logger;
        private readonly RedisProvider _redisProvider;

        public RealtimeChartService(ILogger<RealtimeChartService> logger, RedisProvider redisProvider)
        {
            _logger = logger;
            _redisProvider = redisProvider;
        }

        public RealtimeChart GetChartData()
        {
            return new RealtimeChart()
            {
                PageViews=GetPageViews(),
                Updated= DateTime.Now.ToLocalTime()

            };
        }


        public PageView GetPageViews()
        {
            PageView views = new PageView(0,0);
            try{
                var db = _redisProvider.Database();
                IReadOnlyList<TimeSeriesTuple> results =  db.TimeSeriesRevRange("ts_pv:t", "-", "+", aggregation: TsAggregation.Sum, timeBucket: 60000, count:2); 

              views= new PageView(Convert.ToInt64(results[0].Val), Convert.ToInt64(results[1].Val));

            }catch(Exception ex){
                _logger.LogError(ex.Message);
            }
            return views;
        }

    }    

}

Check the method `GetPageViews` to understand how the .Net Api method `TimeSeriesRevRange` gets the data.

Update Startup.cs - ConfigureServices method and add line



services.AddTransient<RealtimeChartService>();

Finally its time to show the data received from redis DB in the web pages.
Update Index.cshtml.cs code-behind file to get data using RealtimeChartService.



using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.Extensions.Logging;
using RedMetrixWebApp;

namespace RedMetrixWebApp.Pages
{
    public class IndexModel : PageModel
    {
        private readonly ILogger<IndexModel> _logger;
        private readonly RealtimeChartService _service;

        public RealtimeChart chartdata { get; set; }

        public IndexModel(ILogger<IndexModel> logger, RealtimeChartService service)
        {
            _logger = logger;
            _service=service;
        }

        public void OnGet()
        {
           chartdata  =  _service.GetChartData();
            _logger.LogInformation("Worker running at: {Time} ", chartdata.Updated);
        }

    }
}

Finally, its time to create the the view.
The data from .cshtml.cs file can be accessed via Model in .cshtml using var chartData = @Html.Raw(Json.Serialize(Model.chartdata));

Structure of the dashboard is using gridstack.js.

Widget design is using handlebars.js

In the grid where pageviews widget will go, do note the presence of a div with id `wpageviews`. This is where handlebar templates will get injected.



@page
@model IndexModel
@{Layout = null;
}
<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="description" content="RedMetrix - RedisTimeSeries ECOM Web analytics Dashboard">
    <title>RedMetrix - A RedisTimeSeries Project</title>
    <link rel="stylesheet" href="~/lib/bootstrap/dist/css/bootstrap.min.css" />
    <link rel="stylesheet" href="~/css/gridstack.css" />

    <style>
    body {
    background: #222222;
    color: #465665;
      }
    .grid-stack-item {
        border: 0;
      }
    .grid-stack-item-content {
        background-color: white;
        text-align: center;
      }
    #blue{
        background: #0000ff;
      }
    </style>
</head>
<body>

  <div class="grid-stack" id="simple-grid"></div>
  <script src="~/lib/jquery/dist/jquery.min.js"></script>
  <script src="~/lib/bootstrap/dist/js/bootstrap.bundle.min.js"></script>
  <script src="//cdn.jsdelivr.net/npm/handlebars@latest/dist/handlebars.js"></script>
  <script src="~/js/gridstack-jq.js" ></script>
  <script type="text/javascript">
    var simple = [
      {x: 0, y: 0, w: 4, h: 2, noMove: true, noResize: true, locked: true, id:'logo', bgcolor:'#9c4274',content: '<img src="widgets/logo/logo.png" alt="logo RedMetrix" style="width:100%;margin-top: auto;"> <h2 style="color:white;"><strong> -- A RedisTimeSeries Project </strong> </h2>'},
      {x: 4, y: 0, w: 4, h: 6, noMove: true, noResize: true, locked: true, id:'pageperf', bgcolor:'#e83e8c',content: '<h5 style="color:whitesmoke">Page Performance <br/> (in ms) </h5> <div><h6 style="color:white">Home </h6><p/><canvas id ="wpageperfh"></canvas> </div><p/><div><h6 style="color:white">Product </h6></p><canvas id ="wpageperfp"></canvas> </div>'},
      {x: 8, y: 0, w: 2, h: 2,noMove: true, noResize: true, locked: true, id:'pageviews', bgcolor:'#12b0c5', content: '<div id ="wpageviews" style="width: 100%; height: 100%"></div>'},
      {x: 10, y: 0, w: 2, h: 2,noMove: true, noResize: true, locked: true, id:'totalorders', bgcolor:'#ff9618', content: '<div id ="wtotalorders" style="width: 100%; height: 100%"></div>'},
      {x: 0, y: 2, w: 4, h: 4, noMove: true, noResize: true, locked: true, id:'conversions', bgcolor:'#ec663c', content: '<h5 style="color:whitesmoke">Conversions </h5> <div><canvas id ="wconversions"></canvas> </div></p></p> <h5 style="color:whitesmoke">Conversion Rate </h5><div id="wconversionrate" style="font-size: 400%; color:lightgreen;"></div>'},
      {x: 8, y: 2, w: 4, h: 1,noMove: true, noResize: true, locked: true, id:'ordervalue', bgcolor:'#dc5945', content: '<div id ="wordervalue" style="width: 100%; height: 100%"></div>'},
      {x: 8, y: 3, w: 4, h: 3,noMove: true, noResize: true, locked: true, id:'ordbypaymeth', bgcolor:'#47bbb3', content: '<h5 style="color:whitesmoke">Orders By Payment Method </h5> <div><canvas id ="wordbypaymeth"></canvas> </div>'}
    ];

    var simpleGrid = GridStack.init({

    }, '#simple-grid');
    simpleGrid.on('added', function(e, items) {
      items.forEach(function(item) { 
        item.el.firstElementChild.style.backgroundColor=item.bgcolor;
      });
    });
    simpleGrid.load(simple);
  </script>

  <script>
  Handlebars.registerHelper('isdefined', function (value) {
    return value !== undefined;
    });
  Handlebars.registerHelper('isnumber', function (value) {
    return !isNaN(value);
    });
  Handlebars.registerHelper('showchange', function (value) {
    return value !== undefined || !isNaN(value) || value !==0;
    });

  </script>
  <script id="pageview_template" type="text/x-handlebars-template">
    <div class="entry">
      <h5 style="color:whitesmoke">{{pageview.title}}</h5>
      <div class="pageview" style="width: 100%; height: 83%">
        <div class= "views" style="font-size: 400%; color:#ff9618;">{{pageview.views}}</div>
        <div class= "views"><h5 style="color:#ff9618">Last value : {{pageview.last}}</h5></div>
        <div class="change" style="color:white"><h3>{{#if (showchange pageview.diff)}} <span>({{pageview.change}}) {{pageview.diff}}%</span>{{/if}} </h3></div>
      </div>
    </div>
  </script>
  <script>
  var pageview_source = document.getElementById("pageview_template").innerHTML;
  var pageviewTemplate = Handlebars.compile(pageview_source);

  </script> 
  <script>
    function percentage(n, total) {
    return Math.round(n * 100 / total);
  }
  function shortnumber(num) {
    if (isNaN(num)) {
      return num;
    }
    if (num >= 1000000000) {
      return (num / 1000000000).toFixed(1) + 'B';
    } else if (num >= 1000000) {
      return (num / 1000000).toFixed(1) + 'M';
    } else if (num >= 1000) {
      return (num / 1000).toFixed(1) + 'K';
    } else {
      return num;
    }
  };
  $(document).ready(function () {
  var chartData = @Html.Raw(Json.Serialize(Model.chartdata));
  console.log(chartData);
  var pageviews=0;
  var lastpv=0;
  var pageviewdiff=0;
  var pageviewchange="";
      if (!isNaN(chartData.pageViews.pv)){
            pageviews=chartData.pageViews.pv;
            if (!isNaN(chartData.pageViews.prev_pv)){
              lastpv=chartData.pageViews.prev_pv;
                var diff=0;
                if(chartData.pageViews.prev_pv>chartData.pageViews.pv){
                    diff=chartData.pageViews.prev_pv-chartData.pageViews.pv;
                    pageviewchange="-"
                }else{
                    diff=chartData.pageViews.pv-chartData.pageViews.prev_pv;
                    pageviewchange="+";
                }
                pageviewdiff=percentage(diff,pageviews);

              }
          }

        var context={
            pageview: {
            title:"Page Views",  
            views: shortnumber(pageviews),
            last:shortnumber(lastpv),
            change: pageviewchange,
            diff:pageviewdiff
            }
        };
        document.getElementById("wpageviews").innerHTML = pageviewTemplate(context);

  });      

  </script>  
</body>
</html>

And the result is

Rest of the Widgets

Total number of Orders / Transactions and Total Order Value (in Rupees)

Unlike Total page views use case where page views in last min were shown, one might also have use cases where the total value from start of business or even current financial year needs to be shown.

Processor

Add this section in the Processor appsettings.json file. This will have the values of the said data at the time when this application will run.



"InitializationOptions":{
        "TotalOrders":0,
        "TotalOrderValue":0.00
    }

In real world scenarios, you would make a Api call to your enterprise systems to get the values.

In ConfigureInitializationServices.cs, create a method that will create necessary keys and add the values from appsettings.json.



public void InitializeTimeSeriesTotalOrderNValue(IDatabase db)
        {
                db.TimeSeriesCreate("ts_o:t", retentionTime: 600000);
                ulong totalorders=0;
                // TODO: Get Data from Rest Api for total orders.
                totalorders=Convert.ToUInt64(_config.GetSection("InitializationOptions:TotalOrders").Value);
                db.TimeSeriesAdd("ts_o:t", "*", Convert.ToDouble(totalorders));  
                db.TimeSeriesCreate("ts_o:v", retentionTime: 600000);
                double totalordervalue=0;
                // TODO: Get Data from Rest Api for total order value
                totalordervalue=Convert.ToDouble(_config.GetSection("InitializationOptions:TotalOrderValue").Value);
                db.TimeSeriesAdd("ts_o:v", "*", totalordervalue);


        }

Do remember to update `DeleteKeys` method as well.

Similarly in DataServices.cs, add the following method and call the method in HandleFile method with the orderamount value when the page type is Success Confirmation.

HandleFile method not shown here. Check code repository if needed.



private static void TSOrders( double orderamount, IDatabase db)
        {
           db.TimeSeriesIncrBy("ts_o:t", 1, timestamp: "*");
           db.TimeSeriesIncrBy("ts_o:v", orderamount, timestamp: "*");
        }

The corresponding Command for CLI would be `TS.INCRBY ts_o:t 1 TIMESTAMP *`

Webapp

Update RealtimeChartService with method that uses TS.GET to get the last data in the timeseries samples.



public double GetOrderValue()
        {
            double orderamount = 0;
            try{
                var db = _redisProvider.Database();
                TimeSeriesTuple value = db.TimeSeriesGet("ts_o:v");
                orderamount = value.Val;
            }catch(Exception ex){
                _logger.LogError(ex.Message);
            }
            return orderamount;
        }

        public long GetOrders()
        {
            long orders = 0;
            try{
                var db = _redisProvider.Database();
                TimeSeriesTuple value = db.TimeSeriesGet("ts_o:t");
                orders=Convert.ToInt64(value.Val);
            }catch(Exception ex){
                _logger.LogError(ex.Message);
            }
            return orders;
        }

Page performance for Home page and Product page (in last min)

Processor

Initialize



db.TimeSeriesCreate("ts_pv:pp:h", retentionTime: 600000);
db.TimeSeriesCreate("ts_pv:pp:p", retentionTime: 600000);

and then..



private static ulong GetStartTime( JsonElement context)
        {
            string s_starttime=context.GetProperty("performance_timing").GetProperty("requestStart").GetString();
            ulong starttime=Convert.ToUInt64(s_starttime);
            return starttime;                           
        }
        private static ulong GetEndTime( JsonElement context)
        {
            string s_endtime=context.GetProperty("performance_timing").GetProperty("domContentLoadedEventEnd").GetString();
            ulong endtime=Convert.ToUInt64(s_endtime);
            return endtime;    
        }

        private static void TSPagePerformance( string pagetype,long pageperf, IDatabase db)
        {
            if (pagetype=="Home"){
                db.TimeSeriesAdd("ts_pv:pp:h", "*", pageperf);
            }else if(pagetype=="Product"){
                db.TimeSeriesAdd("ts_pv:pp:p", "*", pageperf);
            }else{}
        }

In the source code in source repository there are two additional methods commented to get start time and end time with seven performance attributes instead of two.

Webapp

Here, we will show page performance with floating bars denoting max and min whereas avg will be a line chart.
In RealtimeChart Model



public class PagePerf
    {
        public List<string> time {get;set;}
        public List<List<int>> maxmin {get;set;}
        public List<int> avg{get;set;}
    }

and in RealtimeChartService



public PagePerf GetPagePerformance(string pagetype){
            string key="";
            if (pagetype=="Home")
            {
                key="ts_pv:pp:h";
            }else if(pagetype=="Product")
            {
                key="ts_pv:pp:p";
            }else{}
            List<int> avgdata=new List<int>();
            List<string> timedata= new List<string>();
            List<List<int>> maxmindata= new List<List<int>>();
            try{
                var db = _redisProvider.Database();

                IReadOnlyList<TimeSeriesTuple> resultsmax =  db.TimeSeriesRange(key, "-", "+", aggregation: TsAggregation.Max, timeBucket: 60000, count:10);
                IReadOnlyList<TimeSeriesTuple> resultsmin =  db.TimeSeriesRange(key, "-", "+", aggregation: TsAggregation.Min, timeBucket: 60000, count:10);
                IReadOnlyList<TimeSeriesTuple> resultsavg =  db.TimeSeriesRange(key, "-", "+", aggregation: TsAggregation.Avg, timeBucket: 60000, count:10);
                maxmindata=GetMaxMinList(resultsmax,resultsmin);
                foreach (var result in resultsavg)
                {
                  avgdata.Add(Convert.ToInt32(result.Val));
                }
                timedata=GetStringTimeList(resultsavg);

            }catch(Exception ex){
                _logger.LogError(ex.Message);
            }
            return new PagePerf
            {
                time=timedata,
                maxmin=maxmindata,
                avg=avgdata
            };
        }

        private List<List<int>> GetMaxMinList(IReadOnlyList<TimeSeriesTuple> resultsmax,IReadOnlyList<TimeSeriesTuple> resultsmin)
        {
            return resultsmax.Concat(resultsmin)
                                .GroupBy(o => o.Time)
                                .Select(g => g.Select(s => (int)s.Val).ToList())
                                .ToList();
        }

        private List<string> GetStringTimeList(IReadOnlyList<TimeSeriesTuple> resultsavg)
        {
             List<string> timedata= new List<string>();
             foreach (var result in resultsavg)
                {
                   TimeStamp ts = result.Time;
                   System.DateTime dtDateTime = new DateTime(1970, 1, 1, 0, 0, 0, 0, System.DateTimeKind.Utc);
                   dtDateTime = dtDateTime.AddMilliseconds((long)ts);
                   String hourMinute = dtDateTime.ToString("HH:mm");
                   timedata.Add(hourMinute);
                }
                return timedata;
        }

Conversion Funnel (in last min)

Processor

Initialize



var label = new TimeSeriesLabel("chart", "Funnel");
                var labels = new List<TimeSeriesLabel> { label };
                db.TimeSeriesCreate("ts_fnl:pl", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_fnl:pl", "*", 0);
                db.TimeSeriesCreate("ts_fnl:pd", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_fnl:pd", "*", 0);
                db.TimeSeriesCreate("ts_fnl:ac", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_fnl:ac", "*", 0);
                db.TimeSeriesCreate("ts_fnl:vc", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_fnl:vc", "*", 0);
                db.TimeSeriesCreate("ts_fnl:co", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_fnl:co", "*", 0);
                db.TimeSeriesCreate("ts_fnl:sc", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_fnl:sc", "*", 0);

Corresponding command will be `TS.CREATE ts_fnl:pl RETENTION 600000 LABELS chart Funnel`

And then



private static void TSFunnel( string funneltype,IDatabase db)
        {
            switch (funneltype)
                {
                    case "Success":

                       db.TimeSeriesAdd("ts_fnl:sc", "*", 1);
                        break;
                    case "Checkout":

                        db.TimeSeriesAdd("ts_fnl:co", "*", 1);
                        break;
                    case "ViewCart":

                        db.TimeSeriesAdd("ts_fnl:vc", "*", 1);
                        break;
                    case "AddToCart":

                        db.TimeSeriesAdd("ts_fnl:ac", "*", 1);
                        break;
                    case "ProductDetail":

                        db.TimeSeriesAdd("ts_fnl:pd", "*", 1);
                        break;
                    case "ProductList":

                        db.TimeSeriesAdd("ts_fnl:pl", "*", 1);
                        break;
                    default:
                         AnsiConsole.MarkupLine($"[red]{funneltype}[/]");
                        break;
                }
        }

Webapp

This will be showcased as a horizontal bar chart.
In the RealtimeChart model



public class Conversion
    {
        public List<FunnelItem> FunnelItems{get;set;}
        public long TotalFunnelValue{get;set;}
       // public double ConversionRate{get;set;}
    }
    public class FunnelItem
    {
        public FunnelItem(int Order, string Item, long Value)
    {
        this.Order=Order;
        this.Item = Item;
        this.Value = Value;        
    }
       public int Order { get; set; }
       public string Item { get; set; }
       public long Value { get; set; }

    }

And, in RealtimeChartService



public Conversion GetConversions(){
            List<FunnelItem> funnelItems = new List<FunnelItem>();
            long totalFunnelValue =0;
            try{
                var db = _redisProvider.Database();
                var filter = new List<string> { "chart=Funnel" };

               var results= db.TimeSeriesMRevRange("-", "+", filter, aggregation:TsAggregation.Sum, timeBucket:600000, count: 1);

                foreach (var result in results)
                {
                    string key = result.key;
                    IReadOnlyList<TimeSeriesTuple> values = result.values;
                    funnelItems.Add(new FunnelItem(GetFunnelOrder(key),PrettyFunnelItem(key),Convert.ToInt64(values[0].Val)));
                    totalFunnelValue=totalFunnelValue+Convert.ToInt64(values[0].Val);
                }
            }catch(Exception ex){
                _logger.LogError(ex.Message);
            }
            return new Conversion
            {
                FunnelItems=funnelItems,
                TotalFunnelValue=totalFunnelValue
            };
        }

        private int GetFunnelOrder(string key){
           switch (key)
                {
                    case "ts_fnl:sc":
                        return 6;
                    case "ts_fnl:co":
                        return 5;
                    case "ts_fnl:vc":
                        return 4;
                    case "ts_fnl:ac":
                        return 3;
                    case "ts_fnl:pd":
                        return 2;
                    case "ts_fnl:pl":
                        return 1;
                    default:
                        _logger.LogInformation(key);
                    break;
                } 
            return 0;
        }
        private string PrettyFunnelItem(string key){
           switch (key)
                {
                    case "ts_fnl:sc":
                        return "Transaction Success";
                    case "ts_fnl:co":
                        return "Checkout";
                    case "ts_fnl:vc":
                        return "View Cart";
                    case "ts_fnl:ac":
                        return "Add To Cart";
                    case "ts_fnl:pd":
                        return "Product Detail";
                    case "ts_fnl:pl":
                        return "Product Listings";
                    default:
                        _logger.LogInformation(key);
                    break;
                } 
            return "";
        }

The command used is `TS.MREVRANGE - + AGGREGATION sum 600000 FILTER chart=Funnel`

Orders by Payment Method (in last min)

Processor

Initialize



var label = new TimeSeriesLabel("chart", "Ordersbypaymenttype");
                var labels = new List<TimeSeriesLabel> { label };
                db.TimeSeriesCreate("ts_o:t:cod", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_o:t:cod", "*", 0);
                db.TimeSeriesCreate("ts_o:t:dc", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_o:t:dc", "*", 0);
                db.TimeSeriesCreate("ts_o:t:cc", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_o:t:cc", "*", 0);
                db.TimeSeriesCreate("ts_o:t:nb", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_o:t:nb", "*", 0);
                db.TimeSeriesCreate("ts_o:t:ap", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_o:t:ap", "*", 0);
                db.TimeSeriesCreate("ts_o:t:gp", retentionTime: 600000, labels: labels);
                db.TimeSeriesAdd("ts_o:t:gp", "*", 0);

And then



private static void TSOrdersbypaymenttype( string paymentmethod, IDatabase db)
        {
           switch (paymentmethod)
                {
                    case "Cash On Delivery":

                       db.TimeSeriesAdd("ts_o:t:cod", "*", 1);
                        break;
                    case "Debit Card":

                        db.TimeSeriesAdd("ts_o:t:dc", "*", 1);
                        break;
                    case "Credit Card":

                        db.TimeSeriesAdd("ts_o:t:cc", "*", 1);
                        break;
                    case "Netbanking":

                        db.TimeSeriesAdd("ts_o:t:nb", "*", 1);
                        break;
                    case "Amazon Pay":

                        db.TimeSeriesAdd("ts_o:t:ap", "*", 1);
                        break;
                    case "Google Pay":

                        db.TimeSeriesAdd("ts_o:t:gp", "*", 1);
                        break;
                    default:
                         AnsiConsole.MarkupLine($"[red]{paymentmethod}[/]");
                        break;
                }
        }

Webapp

In RealtimeChart model



public class PaymentMethodOrders
    {
        public PaymentMethodOrders(string PaymentMethod, long Orders)
        {
        this.PaymentMethod = PaymentMethod;
        this.Orders = Orders;
        }
       public string PaymentMethod { get; set; }
       public long Orders { get; set; }
    }

And then in RealtimeChartService



public List<PaymentMethodOrders> GetOrdersByPaymentMethod(){
            List<PaymentMethodOrders> OrdersByPaymentMethod = new List<PaymentMethodOrders>();
            try{
                var db = _redisProvider.Database();
                var filter = new List<string> { "chart=Ordersbypaymenttype" };

               var results= db.TimeSeriesMRevRange("-", "+", filter, aggregation:TsAggregation.Sum, timeBucket:600000, count: 1);
            //      string jsonString = JsonSerializer.Serialize(results);
            //   _logger.LogInformation(jsonString);
                foreach (var result in results)
                {
                    string key = result.key;
                   IReadOnlyList<TimeSeriesTuple> values = result.values;
                    OrdersByPaymentMethod.Add(new PaymentMethodOrders(PrettyPaymentMethod(key),Convert.ToInt64(values[0].Val)));
                }
            }catch(Exception ex){
                _logger.LogError(ex.Message);
            }

            return OrdersByPaymentMethod;
        }

        private string PrettyPaymentMethod(string key){
           switch (key)
                {
                    case "ts_o:t:cod":
                        return "Cash On Delivery";
                    case "ts_o:t:dc":
                        return "Debit Card";
                    case "ts_o:t:cc":
                        return "Credit Card";
                    case "ts_o:t:nb":
                        return "Net Banking";
                    case "ts_o:t:ap":
                        return "Amazon Pay";
                    case "ts_o:t:gp":
                        return "Google Pay";
                    default:
                        _logger.LogInformation(key);
                    break;
                } 
            return "";
        }

The final view

Chart.js is used for charting.
Dashboard is refreshed using meta http-equiv="refresh"



@page
@model IndexModel
@{Layout = null;
}
<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="refresh" content="60" />
    <meta name="description" content="RedMetrix - RedisTimeSeries ECOM Web analytics Dashboard">
    <title>RedMetrix - A RedisTimeSeries Project</title>
    <link rel="stylesheet" href="~/lib/bootstrap/dist/css/bootstrap.min.css" />
    <link rel="stylesheet" href="~/css/gridstack.css" />

    <style>
    body {
    background: #222222;
    color: #465665;
      }
    .grid-stack-item {
        border: 0;
      }
    .grid-stack-item-content {
        background-color: white;
        text-align: center;
      }
    #blue{
        background: #0000ff;
      }
    </style>
</head>
<body>

  <div class="grid-stack" id="simple-grid"></div>
  <script src="~/lib/jquery/dist/jquery.min.js"></script>
  <script src="~/lib/bootstrap/dist/js/bootstrap.bundle.min.js"></script>
  <script src="//cdn.jsdelivr.net/npm/handlebars@latest/dist/handlebars.js"></script>
  <script src="//cdn.jsdelivr.net/npm/chart.js"></script>
  <script src="~/js/gridstack-jq.js" ></script>
  <script type="text/javascript">
    var simple = [
      {x: 0, y: 0, w: 4, h: 2, noMove: true, noResize: true, locked: true, id:'logo', bgcolor:'#9c4274',content: '<img src="widgets/logo/logo.png" alt="logo RedMetrix" style="width:100%;margin-top: auto;"> <h2 style="color:white;"><strong> -- A RedisTimeSeries Project </strong> </h2>'},
      {x: 4, y: 0, w: 4, h: 6, noMove: true, noResize: true, locked: true, id:'pageperf', bgcolor:'#e83e8c',content: '<h5 style="color:whitesmoke">Page Performance <br/> (in ms) </h5> <div><h6 style="color:white">Home </h6><p/><canvas id ="wpageperfh"></canvas> </div><p/><div><h6 style="color:white">Product </h6></p><canvas id ="wpageperfp"></canvas> </div>'},
      {x: 8, y: 0, w: 2, h: 2,noMove: true, noResize: true, locked: true, id:'pageviews', bgcolor:'#12b0c5', content: '<div id ="wpageviews" style="width: 100%; height: 100%"></div>'},
      {x: 10, y: 0, w: 2, h: 2,noMove: true, noResize: true, locked: true, id:'totalorders', bgcolor:'#ff9618', content: '<div id ="wtotalorders" style="width: 100%; height: 100%"></div>'},
      {x: 0, y: 2, w: 4, h: 4, noMove: true, noResize: true, locked: true, id:'conversions', bgcolor:'#ec663c', content: '<h5 style="color:whitesmoke">Conversions </h5> <div><canvas id ="wconversions"></canvas> </div></p></p> <h5 style="color:whitesmoke">Conversion Rate </h5><div id="wconversionrate" style="font-size: 400%; color:lightgreen;"></div>'},
      {x: 8, y: 2, w: 4, h: 1,noMove: true, noResize: true, locked: true, id:'ordervalue', bgcolor:'#dc5945', content: '<div id ="wordervalue" style="width: 100%; height: 100%"></div>'},
      {x: 8, y: 3, w: 4, h: 3,noMove: true, noResize: true, locked: true, id:'ordbypaymeth', bgcolor:'#47bbb3', content: '<h5 style="color:whitesmoke">Orders By Payment Method </h5> <div><canvas id ="wordbypaymeth"></canvas> </div>'}
    ];

    var simpleGrid = GridStack.init({

    }, '#simple-grid');
    simpleGrid.on('added', function(e, items) {
      items.forEach(function(item) { 
        item.el.firstElementChild.style.backgroundColor=item.bgcolor;
      });
    });
    simpleGrid.load(simple);
  </script>

  <script>
  Handlebars.registerHelper('isdefined', function (value) {
    return value !== undefined;
    });
  Handlebars.registerHelper('isnumber', function (value) {
    return !isNaN(value);
    });
  Handlebars.registerHelper('showchange', function (value) {
    return value !== undefined || !isNaN(value) || value !==0;
    });

  </script>
  <script id="pageview_template" type="text/x-handlebars-template">
    <div class="entry">
      <h5 style="color:whitesmoke">{{pageview.title}}</h5>
      <div class="pageview" style="width: 100%; height: 83%">
        <div class= "views" style="font-size: 400%; color:#ff9618;">{{pageview.views}}</div>
        <div class= "views"><h5 style="color:#ff9618">Last value : {{pageview.last}}</h5></div>
        <div class="change" style="color:white"><h3>{{#if (showchange pageview.diff)}} <span>({{pageview.change}}) {{pageview.diff}}%</span>{{/if}} </h3></div>
      </div>
    </div>
  </script>
  <script id="totalorders_template" type="text/x-handlebars-template">
    <div class="entry">
      <h5 style="color:whitesmoke">{{totalorders.title}}</h5>
      <div class="pageview" style="width: 100%; height: 83%">
        <div class= "views" style="font-size: 500%; color:#12b0c5">{{totalorders.orders}}</div>
      </div>
    </div>
  </script>
  <script id="totalsales_template" type="text/x-handlebars-template">
    <div class="entry">
      <h5 style="color:whitesmoke">{{totalordervalue.title}}</h5>
      <div class="pageview" style="width: 100%; height: 83%">
        <div class= "views"><h1 style="color:#ff9618";  font-size: 250%;>&#8377; {{totalordervalue.orderValue}}</h1></div>
      </div>
    </div>
  </script>
  <script>
  var pageview_source = document.getElementById("pageview_template").innerHTML;
  var pageviewTemplate = Handlebars.compile(pageview_source);
  var totalorders_source = document.getElementById("totalorders_template").innerHTML;
  var totalordersTemplate = Handlebars.compile(totalorders_source);
  var totalsales_source = document.getElementById("totalsales_template").innerHTML;
  var totalsalesTemplate = Handlebars.compile(totalsales_source);
  </script> 
  <script>
    function percentage(n, total) {
    return Math.round(n * 100 / total);
  }
  function shortnumber(num) {
    if (isNaN(num)) {
      return num;
    }
    if (num >= 1000000000) {
      return (num / 1000000000).toFixed(1) + 'B';
    } else if (num >= 1000000) {
      return (num / 1000000).toFixed(1) + 'M';
    } else if (num >= 1000) {
      return (num / 1000).toFixed(1) + 'K';
    } else {
      return num;
    }
  };
  var orderbypaychartdata= {
    labels: [],
    datasets: [{
      label: 'Orders',
      data: [],
      borderColor: 'rgb(255, 99, 132)',
      backgroundColor: 'rgb(255, 99, 132)',
      borderWidth: 2,
      borderRadius: 5,
      borderSkipped: false,
      order: 1
    }]
   };
var orderbypaychartconfig = {
    type: 'bar',
    data: orderbypaychartdata,
    options: {
      responsive: true,
      plugins: {
        legend: {
            display: false,
        }
      }
    }
  };

var orderbypayChart = new Chart(
  document.getElementById('wordbypaymeth'),
  orderbypaychartconfig
);

const pageperfhomechartdata = {
  labels: [],
  datasets: [
    {
      label: 'Max-Min',
      data: [],
      borderColor: 'rgb(54, 162, 235)',
      backgroundColor: 'rgb(54, 162, 235)',
      borderWidth: 2,
      borderRadius: 5,
      borderSkipped: false,
      order: 1
    },
    {
      label: 'Avg',
      data: [],
      borderColor: 'rgb(255, 205, 86)',
      backgroundColor: 'rgb(255, 205, 86)',
      type: 'line',
      order: 0
    }]
};

const pageperfhomechartconfig = {
  type: 'bar',
  data: pageperfhomechartdata,
  options: {
    responsive: true,
    plugins: {
      legend: {
        display: false,
      },
      title: {
        display: false,
        text: 'Home Page Performance'
      }
    }
  }
};

var pageperfhomeChart = new Chart(
  document.getElementById('wpageperfh'),
  pageperfhomechartconfig
);

const pageperfproductchartdata = {
  labels: [],
  datasets: [
    {
      label: 'Max-Min',
      data: [],
      borderColor: 'rgb(255, 205, 86)',
      backgroundColor: 'lightgreen',
      borderWidth: 2,
      borderRadius: 5,
      borderSkipped: false,
      order: 1
    },
    {
      label: 'Avg',
      data: [],
      borderColor: 'rgb(54, 162, 235)',
      backgroundColor: 'rgb(54, 162, 235)',
      type: 'line',
      order: 0
    }]
};

const pageperfproductchartconfig = {
  type: 'bar',
  data: pageperfproductchartdata,
  options: {
    responsive: true,
    plugins: {
      legend: {
        display: false,
      },
      title: {
        display: false,
        text: 'Product Page Performance'
      }
    }
  }
};

var pageperfproductChart = new Chart(
  document.getElementById('wpageperfp'),
  pageperfproductchartconfig
);

var conversionchartdata= {
  axis: 'y',
  labels: [],
  datasets: [{
    label: 'Value',
    data: [],
    borderColor: 'rgb(54, 162, 235)',
    backgroundColor: 'rgb(54, 162, 235)',
    borderWidth: 2,
    borderRadius: 5,
    borderSkipped: false,
    order: 1
  }]
 };
var conversionchartconfig = {
  type: 'bar',
  data: conversionchartdata,
  options: {
    indexAxis: 'y',
    responsive: true,
    plugins: {
      legend: {
          display: false,
      }
    }
  }
};

var conversionChart = new Chart(
document.getElementById('wconversions'),
conversionchartconfig
);
  $(document).ready(function () {
  var chartData = @Html.Raw(Json.Serialize(Model.chartdata));
  console.log(chartData);
  var pageviews=0;
  var lastpv=0;
  var pageviewdiff=0;
  var pageviewchange="";
  var totalorders=0;
  var totalsales=0;
      if (!isNaN(chartData.pageViews.pv)){
            pageviews=chartData.pageViews.pv;
            if (!isNaN(chartData.pageViews.prev_pv)){
              lastpv=chartData.pageViews.prev_pv;
                var diff=0;
                if(chartData.pageViews.prev_pv>chartData.pageViews.pv){
                    diff=chartData.pageViews.prev_pv-chartData.pageViews.pv;
                    pageviewchange="-"
                }else{
                    diff=chartData.pageViews.pv-chartData.pageViews.prev_pv;
                    pageviewchange="+";
                }
                pageviewdiff=percentage(diff,pageviews);

              }
          }
        if (!isNaN(chartData.orders)){
          totalorders =chartData.orders
        }
        if (!isNaN(chartData.orderValue)){
          totalsales =chartData.orderValue
        }        
        var context={
            pageview: {
            title:"Page Views",  
            views: shortnumber(pageviews),
            last:shortnumber(lastpv),
            change: pageviewchange,
            diff:pageviewdiff
            },
            totalorders: {
            title:"Transactions",  
            orders: shortnumber(totalorders)
            },
            totalordervalue: {
            title:"Total Sales",  
            orderValue: shortnumber(totalsales)
            },
        };
        document.getElementById("wpageviews").innerHTML = pageviewTemplate(context);
        document.getElementById("wtotalorders").innerHTML =totalordersTemplate(context);
        document.getElementById("wordervalue").innerHTML=totalsalesTemplate(context);
        const updatedorderbypaychart = Chart.getChart("wordbypaymeth");
        updatedorderbypaychart.data.labels=chartData.ordersByPaymentMethod.map((item) =>item.paymentMethod);
        updatedorderbypaychart.data.datasets[0].data=chartData.ordersByPaymentMethod.map((item)=>item.orders);
        updatedorderbypaychart.update();

        const updatedpageperfhomechart = Chart.getChart("wpageperfh");
        updatedpageperfhomechart.data.labels=chartData.pagePerformanceHome.time;
        updatedpageperfhomechart.data.datasets[0].data=chartData.pagePerformanceHome.maxmin;
        updatedpageperfhomechart.data.datasets[1].data=chartData.pagePerformanceHome.avg;
        updatedpageperfhomechart.update();

        const updatedpageperfproductchart = Chart.getChart("wpageperfp");
        updatedpageperfproductchart.data.labels=chartData.pagePerformanceProduct.time;
        updatedpageperfproductchart.data.datasets[0].data=chartData.pagePerformanceProduct.maxmin;
        updatedpageperfproductchart.data.datasets[1].data=chartData.pagePerformanceProduct.avg;
        updatedpageperfproductchart.update();

        var funneldata =chartData.conversions.funnelItems.sort(function (a, b) {
          return a.order - b.order;
        });
        var funneldataVal=funneldata.map((item)=>item.value);
        const updatedconversionschart = Chart.getChart("wconversions");
        updatedconversionschart.data.labels=funneldata.map((item) =>item.item);
        updatedconversionschart.data.datasets[0].data=funneldata.map((item)=>item.value);
        updatedconversionschart.update();
        document.getElementById("wconversionrate").innerHTML = percentage(funneldataVal[funneldataVal.length-1],funneldataVal[0]) + "%";
  });      

  </script>  
</body>
</html>

And the result is

RedMetrixWebApp also contains a docker file which should help in running the application easily.

Final notes

RedisTimeSeries : https://redis.io/docs/stack/timeseries/
RedisTimeSeries Commands : https://redis.io/commands/?group=timeseries
NRedisTimeSeries - .Net Client for RedisTimeSeries : https://github.com/RedisTimeSeries/NRedisTimeSeries/
Redis Cloud : https://redis.com/try-free/
Mockaroo - For mock data : https://www.mockaroo.com/7d448f50
RedisProvider : https://github.com/redis-developer/nredi2read-preview/blob/main/Providers/RedisProvider.cs
gridstack.js - For Dashboard Structure : https://github.com/gridstack/gridstack.js
handlebars.js - For widget design : https://github.com/handlebars-lang/handlebars.js
Chart.js - Charting library : https://github.com/chartjs/Chart.js
Github Code Repository - Redmetrix : https://github.com/c-arnab/Redmetrix

This post is in collaboration with Redis

.Net microservice with gRPC containerised using Azure Container Registry and deployed on Azure Kubernetes Service

c-arnab — Mon, 22 Mar 2021 01:39:15 +0000

In this post, you will learn about gRPC, protocol buffers and building high performance services in .NET 5 using C#. Then you learn to test the service locally using a tool grpcui which is similar to postman but for gRPC. Then you containerise the image and publish the same to Azure Container Registry using ACR task. Then you test the containerised application by running the image on Azure Container Instance and building a gRPC client in .NET 5 using C#. Finally, you deploy the service to Azure Kubernetes Service.

Microservices are booming.
From lower costs to better performance to less downtime to the ability to scale, microservices provide countless benefits relative to monolithic designs. Rather than using a text-based messaging protocol like Representational State Transfer (REST), leveraging a binary protocol which make serialization or deserialization easier and that is optimized for inter-service communication is ideal. gRPC, implemented in Google and open sourced is a high-performance RPC framework which is the most viable solution for building communication between internal microservices.

Develop gRPC Service

Foundational concepts of gRPC

Since a long, long time, developers have been building applications which speak with other applications - java applications speaking with java applications, .Net applications speaking with .Net applications and so on.
Then SimpleObjectAccessProtocol was born such that applications in one language could speak with applications of another language.
Along came WCF by Microsoft. Though WCF also had NETTCP binding, a fast binary encoding to speak between .Net clients and servers, the primary mechanism of communication in SOAP was XML. XML (lots and lots of it) made communication slow and there was a need for something easier for machines to understand as well as a better mode of communication within and across data centers.
And Google created their own rpc framework & binary format - protobuf. Later after HTTP/2 was created, google crated a new rpc framework, called it grpc and made it public. And we get an insanely powerful, fast, highly optimised protocol without the overhead of a soap envelope.
Unlike SOAP where different languages generated XML which would pass between datacenters, gRPC relies on Protocol Buffers language to define service contracts independently from the programming language being used to develop the applications and uses protocol buffers (protobuf) as the binary data interchange format for inter-service communication.

Create a gRPC Service using VS Code

We will be building a simple service which Checks for availability of products in inventory.
Open VS Code, Open a new terminal and run the command
dotnet new grpc -o ProductAvailabilityService

A folder with name ProductAvailabilityService will be created and templated application will be generated.
Move inside the folder ProductAvailabilityService
cd ProductAvailabilityService

Run the command
dotnet add ProductAvailabilityService.csproj package Grpc.AspNetCore.Server.Reflection
This package allows us to use Reflection and will help us test our service.

gRPC natively supports the ability to define a service contract where you can specify the methods that can be invoked remotely and the data structure of the parameters and return types. Using major programming language provided tools, a server-side skeleton and client-side code (stub) can be generated using the same Protocol Buffers file (.proto) which defines the service contract.

Delete the template generated greet.proto file in ProductAvailabilityService/Protos folder.
Then, add a new file named product-availability-service.proto in the same folder.
Here we define apis and the messages in a language-agnostic way.
Here, ProductAvailabilityCheck is the service definition which contains a method CheckProductAvailabilityRequest.
The method takes in a request ProductAvailabilityRequest and sends back a response ProductAvailabilityReply. The request and response objects along with the implementation base class will be generated automatically using the Grpc.Tools NuGet package and there is no need to implement the complexities.

// Protos/product-availability-service.proto

syntax = "proto3";

option csharp_namespace = "ProductAvailabilityService";

package ProductAvailability;

service ProductAvailabilityCheck {
  rpc CheckProductAvailabilityRequest (ProductAvailabilityRequest) returns (ProductAvailabilityReply);
}

message ProductAvailabilityRequest {
  string productId = 1;  
}

message ProductAvailabilityReply {
  bool isAvailable = 1;
}

The Nuget package knows which proto file to look for to generate the types from .csproj file.
So, we go to .csproj file and update the .proto file as shown below.

<ItemGroup>
    <Protobuf Include="Protos\product-availability-service.proto" GrpcServices="Server" />
</ItemGroup>

Delete the template generated GreeterService.cs file in Services folder.
Create a new file ProductAvailabilityCheckService.cs in services folder.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Grpc.Core;
using Microsoft.Extensions.Logging;

namespace ProductAvailabilityService
{
    public class ProductAvailabilityCheckService: ProductAvailabilityCheck.ProductAvailabilityCheckBase
    {
        private readonly ILogger<ProductAvailabilityCheckService> _logger;
        private static readonly Dictionary<string, Int32> productAvailabilityInfo = new Dictionary<string, Int32>() 
        {
            {"pid001", 1},
            {"pid002", 0},
            {"pid003", 5},
            {"pid004", 1},
            {"pid005", 0},
            {"pid006", 2}   
        };
        public ProductAvailabilityCheckService(ILogger<ProductAvailabilityCheckService> logger)
        {
            _logger = logger;
        }

        public override Task<ProductAvailabilityReply> CheckProductAvailabilityRequest(ProductAvailabilityRequest request, ServerCallContext context)
        {
            return Task.FromResult(new ProductAvailabilityReply
            {
                IsAvailable = IsProductAvailable(request.ProductId)
            });
        }

        private bool IsProductAvailable(string productId) {
            bool isAvailable = false;

            if (productAvailabilityInfo.TryGetValue(productId, out Int32 quantity))
            {
                isAvailable = (quantity > 0) ? true : false;
            }

            return isAvailable;
        }
    }
}

Here a dictionary productAvailabilityInfo acts as a dummy inventory data store for products with productid – pid001 to pid006 and available value.
The method CheckProductAvailabilityRequest defined in .proto file is implemented here. The implemented method takes in the request object along with ServerCallContext useful in authentication / authorization and returns the response object.

Open startup.cs file and update greeterservice to ProductAvailabilityCheckService.
Also, add services.AddGrpcReflection() and endpoints.MapGrpcReflectionService()

 public void ConfigureServices(IServiceCollection services)
        {
            services.AddGrpc();
            services.AddGrpcReflection();
        }

 app.UseEndpoints(endpoints =>
            {
                endpoints.MapGrpcService<ProductAvailabilityCheckService>();
                endpoints.MapGrpcReflectionService();

Finally, to setup a HTTP/2 endpoint without TLS, open program.cs and add the code below inside the method call to ConfigureWebHostDefaults to run the service in http mode in port 5000.

 .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.ConfigureKestrel(options =>
                    {
                        // Setup a HTTP/2 endpoint without TLS.
                        options.ListenLocalhost(5000, o => o.Protocols = 
                            HttpProtocols.Http2);

                    });
                    webBuilder.UseStartup<Startup>();
                });

Also add the using statement
using Microsoft.AspNetCore.Server.Kestrel.Core;

Run the service by typing in terminal
dotnet run
Service will be up and running using kestral webserver.

Test Service Locally

To test our service we use grpcui which is like Postman, but for gRPC APIs instead of REST.

To install the tool first install GO Golang Site

Then run the two commands below in powershell ( run as Administrator).
go get github.com/fullstorydev/grpcurl/...
go install github.com/fullstorydev/grpcurl/cmd/grpcurl

Next run the tool with serverhost address and server port number

grpcui -plaintext localhost:5000

gRPCui will expose a local URL to test the gRPC service.

Open the URL shown and add a productid as shown below.

The product availability will be shown as response

Test service on Azure

Next, we run the the grpc service in a container on Azure Cloud. Though we can build the container image on our workstation by installing Azure CLI and Docker engine, we will not do so but rather do so on Azure using Cloud Shell and Azure Container Registry task and push the image to Azure Container Registry.

Update program.cs to below code as we no longer wish it to listen to localhost but any ip on port 80.

.ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.ConfigureKestrel(options =>
                    {
                        // Setup a HTTP/2 endpoint without TLS.
                       // options.ListenLocalhost(5000, o => o.Protocols = 
                       //     HttpProtocols.Http2);
                        options.ListenAnyIP(80, o => o.Protocols = 
                            HttpProtocols.Http2);
                    });
                    webBuilder.UseStartup<Startup>();
                });

Dockerfile Basics

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession.

Create Dockerfile

Create a new file in the service project, name the same Dockerfile and add the below contents.

FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY "ProductAvailabilityService.csproj" .
RUN dotnet restore "ProductAvailabilityService.csproj"
COPY . .
RUN dotnet build "ProductAvailabilityService.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "ProductAvailabilityService.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "ProductAvailabilityService.dll"]

Lets understand what is happening in the dockerfile.
At first restore the packages and dependencies application requires to build. Next, build the application and then publish the same in /app/publish directory. At this stage .NET Core SDK available at Microsoft Container Registry (MCR) is used as base image.
Next, use the .NET Core Runtime as base image and copy the binaries generated in the previous stage to /app directory. Please note assembly name in ENTRYPOINT is case-sensitive.

Add source code to github

Create a .gitignore file either manually or using .gitignore Generator Visual Studio Marketplace.

Create an empty github repository at https://github.com/new

Run the following commands in terminal.

Initialise the local directory as Git repository.
git init

Add all the files in the local directory to staging, ready for commit.
git add .

Set your account’s default identity
git config --global user.email “your-emailid@your-email-domain.com”
git config --global user.name "your-username"

Commit the staged files
git commit -m "first commit"
git branch -M main

To push your local repository changes to add your GitHub repository as remote repository
git remote add origin My-GitHub-repository-URL
(I replace My-GitHub-repository-URL to https://github.com/c-arnab/grpc-article.git where you can find my entire source code)

Push local repository to the remote repository we added earlier called “origin” in its branch named master.
git push -u origin main

You can now see all your files in the GitHub repository.

Build Container Image

Select Azure Shell and then bash option

Its always advisable to create a new resource group when working on a new project or in this case code in an article.

Run the following command to create a resource group.
az group create --name grpc-container-demo --location southindia --subscription 59axxx4d-xxxx-4352-xxxx-21dd55xxxca0
Please ensure you add your own subscription id after subscription

Azure Container Registry (ACR) is an Azure-based, private registry, for container images.
Run the following command to create an Azure Container Registry (ACR). You will need to provide a unique name. The --admin-enabled flag will be useful while running the container image in container instance.
az acr create --name MyGrpcContainerRegistry --resource-group grpc-container-demo --sku Standard --admin-enabled true

Change directory to clouddrive

Clone the code with Git
git clone https://github.com/c-arnab/grpc-article.git

Change directory to grpc-article

ACR task, a function of ACR is a container image build service on the cloud.
Build container image using the az acr build command and store it in Registry.
az acr build --image product-availability-service:latest --registry MyGrpcContainerRegistry .

Run the following command to list and view image generated in Azure Container Registry
az acr repository list --name MyGrpcContainerRegistry --output table

Run image in Azure Container Instance

Azure Container Instances is a service that enables a developer to deploy containers on the Microsoft Azure public cloud without having to provision or manage any underlying infrastructure.

Once image is in ACR, its easy to test the application in container image.

Search for container registries in Azure portal and select the same.

Select the registry created and then select Access Keys under Settings
The --admin-enabled flag while creating the registry has ensured that Admin user is enabled.

Select Repositories under Services and then select the product-availability-service under Repositories.
Click on the three dots on the right side of the latest tag and select Run instance.

In the ensuing screen, Create Container Instance by providing a name and select a location near you. Then Click OK.

Once the deployment completes, you should receive a notification and clicking the same one can reach the Container group and view the Container Instance and confirm that it is running. Finally get the IP Address which will be required when we create a client for the service next.

Create a gRPC client

For this article we create a console app as a client which in turn calls CheckProductAvailabilityRequest method in the service.

Close all applications/folder in VS Code and open a new terminal.

Run command to create a console app
dotnet new console -o ProductAvailabilityClient

Open folder / application ProductAvailabilityClient

In the terminal run following commands to add necessary GRPC libraries

dotnet add ProductAvailabilityClient.csproj package Grpc.Net.Client
dotnet add ProductAvailabilityClient.csproj package Google.Protobuf
dotnet add ProductAvailabilityClient.csproj package Grpc.Tools

Create a folder Protos and drop the .proto file used in the service application in this folder.

Replace the program.cs code with the following code

using System;
using System.Threading.Tasks;
using ProductAvailabilityService;
using Grpc.Net.Client;

namespace ProductAvailabilityClient
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var channel = GrpcChannel.ForAddress("http://52.226.2.157:80");
            var client =  new ProductAvailabilityCheck.ProductAvailabilityCheckClient(channel);
            var productRequest = new ProductAvailabilityRequest { ProductId = "pid001"};
            var reply = await client.CheckProductAvailabilityRequestAsync(productRequest);

            Console.WriteLine($"{productRequest.ProductId} is {(reply.IsAvailable ? "available" : "not available")}!");
            Console.WriteLine("Press any key to exit...");
            Console.ReadKey();
        }
    }
}

Replace the ip address with the ip address of your container instance. Please note the ProductId. You can change this value to any other.

Run the console client by typing in terminal
dotnet run

The console should show whether the product id is available. Change the value of the product id and rerun the application to reconfirm that the application works.

Deploy on Azure Kubernetes Service

Foundational concepts of Kubernetes

Containers are not new. The fundamental container goes back to 1979 with Unix’s chroot that allowed to separate processes. Then came linux containers in 2008. Docker made its appearance in 2013 and gave the power of containers in the hand of developers.

And then Google came up with Kubernetes to run these containers at scale with intelligent deployment, auto repair, horizontal scaling, service discovery & load balancing, automation of deployments & rollback and key / secrets management.

An application running on Kubernetes is known as workload and are run inside a set of pods. A workload can have a single component deployed in a single pod or several that work together deployed in a single pod or multiple pods.
You can deploy container images in kubernetes either in a pod (Naked Pod) or rather than managing each Pod directly, use workload resources that manage a set of pods like Deployment and ReplicaSet.
Its advised not to use naked Pods (that is, Pods not bound to a ReplicaSet or Deployment) as Naked Pods will not be rescheduled in the event of a node failure.
Service is an abstract way to expose an application running on a set of Pods as a network service.

Azure Kubernetes Service (AKS) is a managed Kubernetes service in which the master node is managed by Azure and end-users manages worker nodes. Users can use AKS to deploy, scale, and manage Docker containers and container-based applications across a cluster of container hosts.

Create Service Principal

While creating container registry we had provided the same admin access with --admin-enabled flag to easily provide access to images to run from container instance but providing admin access is not a best practice from security perspective. While deploying to AKS we use service principal which will enable applications and services (AKS) to authenticate to container registry.

Get the full Container Registry ID
ACR_ID=$(az acr show --name MyGrpcContainerRegistry --query id --output tsv)

Create the Service principal with proper rights
SP_PASSWD=$(az ad sp create-for-rbac --name http://acr-service-principal --scopes $ACR_ID --role owner --query password --output tsv)
SP_APP_ID=$(az ad sp show --id http://acr-service-principal --query appId --output tsv)

Please copy the service principal Id and password to safe place as we will need them later.
echo "ID: $SP_APP_ID"
echo "Password: $SP_PASSWD"

Create Cluster

Create an AKS cluster using the az aks create command.
az aks create --resource-group grpc-container-demo --name mygrpcdemocluster --node-count 2 --generate-ssh-keys

IF you receive an error, check if you have VMs with your default subscription and region.
az vm list-skus -l southindia --query '[?resourceType==virtualMachines&& restrictions==[]]' -o table

If the above command does not provide any data, try the same with other locations for example try centralindia in place of southindia.

Once you get data run again the az aks create command with --location flag
az aks create --resource-group grpc-container-demo --name mygrpcdemocluster --node-count 2 --generate-ssh-keys --location centralindia

Connect to Cluster

Kubernetes command-line client, kubectl is used to manage a Kubernetes cluster which is already installed in Azure Cloud Shell.

Use the az aks get-credentials command to download credentials and configure kubectl to use the same to connect to Kubernetes cluster
az aks get-credentials --resource-group grpc-container-demo --name mygrpcdemocluster

View namespaces
kubectl get namespaces

Create a namespace to work in
kubectl create namespace mygrpcdemospace

Set created namespace to current namespace
kubectl config set-context --current --namespace=mygrpcdemospace

Confirm you are in correct namespace
kubectl config get-contexts

Deploy application

Kubernetes uses an image pull secret to store information needed to authenticate to registry. To create the pull secret for an Azure container registry, run the following command with a name (my-acr-secret), service principal id and password you created earlier along with your registry url.

kubectl create secret docker-registry my-acr-secret \
    --namespace mygrpcdemospace \
    --docker-server=mygrpccontainerregistry.azurecr.io \
    --docker-username=eddeabb7-6f26-4e72-b741-549f67e83736 \
    --docker-password=j_L.57Om6S1Pqh9-V8_iSFPSk9m7jFD054

A ReplicaSet ensures that a specified number of pod replicas are running at any given time. However, a Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods along with a lot of other useful features. Therefore, we will use Deployments instead of directly using ReplicaSets to deploy our container.

Run the following command to create a manifest yaml and submit it to our Kubernetes cluster to create a new Deployment object named "grpc-demo-deployment". Do note the image and imagePullSecrets value and change it to your values.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grpc-demo-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: grpc-demo
  template:
    metadata:
      labels:
        app: grpc-demo
        env: dev
    spec:
      containers:
      - name: grpc-demo
        image: mygrpccontainerregistry.azurecr.io/product-availability-service:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 80
      imagePullSecrets:
      - name: my-acr-secret
EOF

Check if the Deployment was created
kubectl get deployments

View the current ReplicaSets deployed
kubectl get rs

Check for the Pods brought up
kubectl get pods --show-labels

Run the following command to create a new Service object named "grpc-service", which targets TCP port 80 on any Pod with the selector app=grpc-demo label.
The selector determines the set of pods targeted by the service.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: grpc-service
spec:
  selector:
    app: grpc-demo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer
EOF

Run the following command and wait until you see an external IP assigned. Copy this ip address as we will need this to test in the next step.

watch kubectl get services

Test the service deployed on AKS

Open the client application developed earlier in VS Code and Replace the ip address with the external ip address viewed in the previous step.

Run the console client by typing in terminal
dotnet run

The console should show whether the product id is available. Change the value of the product id and rerun the application to reconfirm that the application works.

Finally..

Delete the resource group and all the resources in it.
az group delete --name grpc-container-demo

What's next

Microsft Azure also has another service named Azure Redhat Openshift.
Depending on who has implemented Kubernetes and its versions, you will have different tools for container runtime, log management, metrics & monitoring, etc. So, that led to enterprise customers to ask for a consistent version of Kubernetes which can run the same way everywhere from internal datacentres to any cloud. OpenShift is that enterprise deployment layer for Kubernetes in hybrid cloud mode.
Azure Red Hat OpenShift provides a flexible, self-service deployment of fully managed OpenShift clusters jointly operated and supported by Microsoft & Red Hat and offers an SLA of 99.95% availability.
Unlike AKS where Microsoft manages the master nodes and customer only pays for and manages the worker nodes, ARO is dedicated to a given customer and all resources, including the cluster master nodes, run in customer’s subscription.
Our demo code is a part of inventory microservice and a part of supply chain application. Such applications depending on customer preference are deployed either on customer’s internal datacentre or public cloud.
It is better for an application developer to select a deployment platform for such applications which can run in any environment (Bare metal, Virtual Machines, Public clouds) and OpenShift is ideally suited for this and Azure Redhat OpenShift is probably the best option for the same in public clouds.
So, in the next article we look at deploying a series of microservices which are part of a supply chain application on ARO with continuous integration, delivery and deployment.

DEV Community: c-arnab

Serverless in Azure using Static Web Apps, Functions and Cosmos DB

Problem Statement

Solution

Architecture Diagram with Application Development Lifecycle

Prerequisites for local development

Build Static Site

Plain vanilla javascript template is used here. There are more templates including angular, react, vue, blazor available at https://github.com/staticwebdev

Update Github Repository

Deploy to Azure

Azure and Github credentials will have to be provided in the above process.

Create Application Interface

Implement Authentication

Create API and integrate with Static Site

View existing events use case

Security is important. Do check out Azure Architecture - Serverless Functions security

A simple piece of code is written next just sufficient to test the integration of API with static site.

The hardcoded value here is arnab but you should update that to your first name provided you are going to use that as username in mock authentication screen. Also do remember to change the namespace to <Your_First_Name>.MyCalendar.

Add new events use case

Implement Persistence layer with Cosmos DB

Emulator connection string is same always unless changed manually. Also local.settings.json is not pushed to github and the connection string there is only useful for local development. The command / process to add connection string in Azure is different and we will see so later.

Though this code gets all events of a user, in production scenarios you would not wish to do that, and along with username also send start date and end date in query string to ensure limited amount of data is accessed or retrieved.

Deploy application to Azure

Provision Database

One could also try Cosmos DB by going to cosmos.azure.com/try. Selecting the account type - Azure Cosmos DB For NoSQL would create a trial account for 30 days which would open in Azure Portal.

The CI / CD Magic

Though in this article / post, this step is done at the very end, in application development lifecycle, this ought to be done after each small change (after interface design, after authentication implementation, after API addition, etc )

I would strongly urge all to check the steps (Build Azure and Build and Deploy) to understand all that happens automatically.

Run Application

Challenges

Business Benefit

Final Notes

Real time analytics using .Net & Redis

In this post, we look at RedisTimeSeries module and build a custom web analytic tool with Redis & .NET 5 using C#

Application architecture Overview

Why use RedisTimeSeries module

Application architecture

If one is developing an IOT analytics application, the process would be similar but the data would in this case be generated by various sensors.

In the real world applications, one might have this logic preferably in azure function / aws lambda which would scale up/ down as needed.

Though, in this article, we get the data in the page's codebehind file, in real world scenarios one would have a background worker which calls a service at specific intervals to get the necessary data from redis and sends the data via signalr to the clients using websockets and server side events.

Charts in Dashboard

Demo Infrastructure

Create Database on Redis Cloud

Generate Mock Data

For simplicity, all event types are Page Views but in real world applications, lions share of them are not going to be Page Views.

Also, Amount to Pay and Payment Method should be part of the event if and only if Page type is 'Success Confirmation'. But in the dummy data you would find these fields present in all events.

The Date field is randomly generated but in real world its going to be sequential. As we are not going to directly do anything with this field, this will not create any problems.

Developing Redmetrix

Processor

I'll not be getting into the features or the code parts of Spectre in this article. But the code is going to be available on github. Do check it out.

The methods in github code repository is a lot more extensive with try catch blocks and Spectre.Console code.

Web Application

Do remember to add the namespace of RedisProvider to Startup.cs

Check the method GetPageViews to understand how the .Net Api method TimeSeriesRevRange gets the data.

In the grid where pageviews widget will go, do note the presence of a div with id wpageviews. This is where handlebar templates will get injected.

Rest of the Widgets

Total number of Orders / Transactions and Total Order Value (in Rupees)

Processor

In real world scenarios, you would make a Api call to your enterprise systems to get the values.

Do remember to update DeleteKeys method as well.

HandleFile method not shown here. Check code repository if needed.

The corresponding Command for CLI would be TS.INCRBY ts_o:t 1 TIMESTAMP *

Webapp

Page performance for Home page and Product page (in last min)

Processor

In the source code in source repository there are two additional methods commented to get start time and end time with seven performance attributes instead of two.

Webapp

Conversion Funnel (in last min)

Processor

Corresponding command will be TS.CREATE ts_fnl:pl RETENTION 600000 LABELS chart Funnel

Webapp

The command used is TS.MREVRANGE - + AGGREGATION sum 600000 FILTER chart=Funnel

Orders by Payment Method (in last min)

Processor

Webapp

The final view

RedMetrixWebApp also contains a docker file which should help in running the application easily.

Final notes

This post is in collaboration with Redis

.Net microservice with gRPC containerised using Azure Container Registry and deployed on Azure Kubernetes Service

The hardcoded value here is arnab but you should update that to your first name provided you are going to use that as username in mock authentication screen. Also do remember to change the namespace to `<Your_First_Name>.MyCalendar`.

The methods in github code repository is a lot more extensive with `try catch blocks` and `Spectre.Console` code.

Do remember to add the namespace of `RedisProvider` to `Startup.cs`

Check the method `GetPageViews` to understand how the .Net Api method `TimeSeriesRevRange` gets the data.

In the grid where pageviews widget will go, do note the presence of a div with id `wpageviews`. This is where handlebar templates will get injected.

Do remember to update `DeleteKeys` method as well.

The corresponding Command for CLI would be `TS.INCRBY ts_o:t 1 TIMESTAMP *`

Corresponding command will be `TS.CREATE ts_fnl:pl RETENTION 600000 LABELS chart Funnel`

The command used is `TS.MREVRANGE - + AGGREGATION sum 600000 FILTER chart=Funnel`