DEV Community: Praveen Raghuvanshi

Making a choice for Project Management Tool

Praveen Raghuvanshi — Wed, 16 Feb 2022 18:26:12 +0000

Introduction

I was looking for a project management tool and stumbled over plethora of tools available in the market. It's difficult to make a choice considering the growth of a project that can start with couple of people to a bigger team. This is not an exhaustive article comparing a lot of tools. I just wanted to keep it simple and share my reasoning in choosing a tool. I'll be sharing the reference links that can used to get more details. Let's get started.

Tool Requirements

Below are the requirements of a tool or parameters used to make the decision considering a startup project involving couple of developers.

🧠 Learning curve
🖇️ Flexibility
💰 Pricing
📈 Scaling to big teams

Tools Considered

Lets dive into each of the parameters.

🧠 Learning curve

Asana : It's an advance tool with lot of features that cater to complex workflows. Steep learning curve.
ClickUp : It's also advanced tool with lot of features and not everyone is aware about it. Steep learning curve
Monday.com : A tool more than project management. Lots of features makes learning steep
Github : Developer friendly tool, learning curve is not steep and easy to grasp.

My choice : Github as its developer friendly.

🖇️ Flexibility

Asana : Advanced tool with lot of customization available. UI/UX is bit outdated
ClickUp: Advanced tool with good amount of customization and advanced UI/UX.
Monday.com: Advanced tool and highly customizable with easy to use UI/UX.
Github : Evolving tool with good amount of flexibility in creating boards, custom columns. Multiple board views. Easy integration with repositories

My Choice: Github due to multiple board views and customizable columns.

💰 Pricing

Asana : Free(15 members) and $10.99/month
ClickUp : Free(Unlimited members) and $5/month
Monday.com : Free(2 members) and $8/month
Github : Free(Unlimited members) and $4/month

My Choice: Github due to unlimited members in free tier.

📈 Scaling

Asana : Scaling to large teams is expensive as advanced features are available in premium and enterprise ($10.99/$24.99 per month)
ClickUp: Scaling to large teams is decently priced with good features available ($9/$19 per month)
Monday.com : Good for scaling having great features and pricing ($10/ $16 per month)
Github : Highly scalable to large teams as seen with some of the large projects hosted and tracked through it with decent pricing ($21 per month)

My Choice: ClickUp due to pricing and great premium features

Conclusion

We saw a brief comparison among different tools based on different parameters. When you are starting out on a project, all tools looks great but as the team grows, factors such as learning curve, pricing need to be evaluated. With the analysis presented I have chosen Github for my project management activity. Github has a fairly easy learning curve and comes with lot of flexibility making a good choice for developers. Overall my choice would be in the order of preference: Github > ClickUp > Asana > Monday.com. Any comments/suggestions/correction are welcome.

References

Cricket Analysis and Prediction using ML.Net(C#)

Praveen Raghuvanshi — Tue, 08 Feb 2022 23:02:35 +0000

Disclaimer: The analysis and prediction done here is for learning purpose only and should not be used for any illegal activities such as betting.

Introduction

Cricket, a game of bat and ball is one of the most popular games and played in varied formats. It's a game of numbers with each match generating a plethora of data about players and matches. This data is used by analysts and data scientists to uncover meaningful insights and forecast about matches and players performance. In this session, I'll be performing some analytics and prediction on the cricket data using Microsoft ML.Net framework and C#.

History

Invented in 1550, originated by England
First International match was played between Canada and USA in New York
First One Day International(ODI) was played in 1971
First Cricket World Cup was played in 1975
First T20 Match was played in 2003

Match Formats

One Day International(ODI) : 50 Over
Test Match : 5 days
T20 : 20 Overs
Indian Premier League : 20 Overs

IPL Statistics (2019-2020)

Participating Nations: 104
Valuation : $6.7 billion
Viewers : 370 Million

If we look at the above stats, it clearly depicts the popularity and money involved in Cricket.

Problem Statement

Analyze the cricket dataset and predict the score after 6 overs for a match.

Solution Overview

As we need to predict the continuous value, its a regression problem in the space of Machine Learning. We'll be using .Net DataFrame for Exploratory Data Analysis(EDA) and ML.Net for predicting the score on a specific ball under 6 overs.

Platform/Tool

I have used .Net interactive tool on a Jupyter Notebook to create a *.ipynb file with C# as a kernel.

Prerequisites

Dataset

Source: CricSheet > Download T20 Dataset

Exploratory Data Analysis - EDA

Summary of data analysis

No of Features: 22
Total Records: 194K
Features have a mix of date, number and string
Contains Null Values

We'll be executing command on a .Net interactive Jupyter notebook.

Lets get started

Open a command prompt or powershell and execute 'Jupyter notebook'. A new window opens in default browser.

Click New and select .Net(C#) as kernel to create a new notebook

A new notebook will be created. Give a name by clicking on Untitled and execute below command in the cell to verify installation of .net interactive.

1. Define Application wide Items

Install Nuget Packages

// ML.NET Nuget packages installation
#r "nuget:Microsoft.ML"
#r "nuget:Microsoft.ML.FastTree"    
#r "nuget:Microsoft.Data.Analysis"
#r "nuget:Daany.DataFrame, 1.0.0"
#r "nuget:CsvHelper"

Import Namespaces

using CsvHelper;
using CsvHelper.Configuration;
using Daany;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.Data.Analysis;
using Microsoft.AspNetCore.Html;
using Microsoft.DotNet.Interactive.Formatting;
using static Microsoft.DotNet.Interactive.Formatting.PocketViewTags;
using System.Net.Http;
using System.IO;
using System.IO.Compression;
using System.Globalization;

Define Constants

// File
const string DATASET_DIRECTORY = "t20s_male_csv2";
const string DATASET_FILE = "t20s_male_csv2.zip";
const string DATASET_URL = "https://cricsheet.org/downloads/";
const string DATASET_MERGED_CSV = "t20_merged.csv";
const string DATASET_ALL_CSV = "t20_all.csv";
const string DATASET_CLEANED_CSV = "t20_cleaned.csv";
const string MODEL_FILE = "model.zip";
const string SEARCH_PATTERN = "*.csv";

// DataFrame/Table
const int TOP_COUNT = 5;
const int DEFAULT_ROW_COUNT = 10;

// Columns
const string CSV_COLUMN_BALL = "ball";
const string CSV_COLUMN_SCORE = "score";
const string CSV_COLUMN_RUNS_OFF_BAT = "runs_off_bat";
const string CSV_COLUMN_EXTRAS = "extras";
const string CSV_COLUMN_TOTAL_SCORE = "total_score";
const string CSV_COLUMN_VENUE = "venue";
const string CSV_COLUMN_INNINGS = "innings";
const string CSV_COLUMN_BATTING_TEAM = "batting_team";
const string CSV_COLUMN_BOWLING_TEAM = "bowling_team";
const string CSV_COLUMN_STRIKER = "striker";
const string CSV_COLUMN_NON_STRIKER = "non_striker";
const string CSV_COLUMN_BOWLER = "bowler";

// Misc
const float OVER_THRESHOLD = 6.0f;

2. Utility Functions

Merges multiple CSV files present in specified directory

The dataset downloaded from CricSheet is a zip of multiple files pertaining to a match. In order to use it for analysis, we need to merge all the files into a single file. The below method helps merge these files

/// <summary>
/// Merges files present in a directory into a single file
/// </summary>
/// <param name="sourceFolder">Directory containing files to be merged</param>
/// <param name="destinationFile">Output single file</param>
/// <param name="searchPattern">Pattern to be searched for files within a directory such as *.csv</param>
/// <param name="excludeFiles">A predicate to exclude files</param>
public void MergeCsv(string sourceFolder, string destinationFile, string searchPattern, Predicate<string> excludeFiles)
{
    /*
     https://chris.koester.io/index.php/2017/01/27/combine-csv-files/
     C# script combines multiple csv files without duplicating headers.
     Combining 8 files with a combined total of about 9.2 million rows 
     took about 3.5 minutes on a network share and 44 seconds on an SSD.
    */

    // Specify wildcard search to match CSV files that will be combined
    string[] filePaths = Array.FindAll(Directory.GetFiles(sourceFolder, searchPattern, SearchOption.AllDirectories), excludeFiles);
    StreamWriter fileDest = new StreamWriter(destinationFile, true);

    int i;
    for (i = 0; i < filePaths.Length; i++)
    {
        string file = filePaths[i];
        string[] lines = File.ReadAllLines(file);
        if (i > 0)
        {
            lines = lines.Skip(1).ToArray(); // Skip header row for all but first file
        }
        foreach (string line in lines)
        {
            fileDest.WriteLine(line);
        }
    }
    fileDest.Close();
}

The extracted files contains files ending with 'info' which has different set of columns and we need to ignore them while merging. In order to achieve that a helper method is introduced.

/// <summary>
/// Exclude files based on a condition
/// </summary>
/// <param name="fileName">Name of the file</param>
/// <returns>True, if matched else false</returns>
public bool ExcludeFiles(string fileName)
{
    if (fileName.EndsWith("info.csv"))
    {
        return false;
    }
    return true;
}

Formatters

By default the output of DataFrame is not proper and in order to display it as a table, we need to have a custom formatter implemented as shown in next cell.

// Formats the table
Formatter.Register(typeof(Microsoft.Data.Analysis.DataFrame),(dataFrame, writer) =>
{
    var df = dataFrame as Microsoft.Data.Analysis.DataFrame;
    var headers = new List<IHtmlContent>();
    headers.Add(th(i("index")));
    headers.AddRange(df.Columns.Select(c => (IHtmlContent)th(c.Name)));
    var rows = new List<List<IHtmlContent>>();
    var take = 10;
    for (var i = 0; i < Math.Min(take, df.Rows.Count); i++)
    {
        var cells = new List<IHtmlContent>();
        cells.Add(td(i));
        foreach (var obj in df.Rows[i])
        {
            cells.Add(td(obj));
        }
        rows.Add(cells);
    }

    var t = table(
        thead(
            headers),
        tbody(
            rows.Select(
                r => tr(r))));

    writer.Write(t);
}, "text/html");

3. Load Dataset

The Dataset present in CricSheet is in compressed zip format. Internally it contains csv file that we will be using for our analysis and prediction.

3.a Clear Previous Data

The below method clears any previous data on the system

/// <summary>
/// Clear previous data present in current working directory
/// </summary>
public void ClearPreviousData()
{
    if (File.Exists(DATASET_FILE))
    {
        File.Delete(DATASET_FILE);
    }

    if (File.Exists(DATASET_MERGED_CSV))
    {
        File.Delete(DATASET_MERGED_CSV);
    }

    if (File.Exists(DATASET_CLEANED_CSV))
    {
        File.Delete(DATASET_CLEANED_CSV);
    }

    if (Directory.Exists(DATASET_DIRECTORY))
    {
        Directory.Delete(DATASET_DIRECTORY, true);
    }
}

3.b Loads dataset into a DataFrame

/// <summary>
/// Loads the dataset from specified URL
/// </summary>
/// <param name="url">Remote URL</param>
/// <param name="fileName">Name of the file</param>
/// <returns>A DataFrame</returns>
public async Task<Microsoft.Data.Analysis.DataFrame> LoadDatasetAsync(string url, string fileName)
{
    // Delete previous data
    ClearPreviousData();

    // Loads zip file from remote URL
    var remoteFilePath = Path.Combine(url, fileName);
    using (var httpClient = new HttpClient())
    {
        var contents = await httpClient.GetByteArrayAsync(remoteFilePath);
        await File.WriteAllBytesAsync(fileName, contents);
    }

    // Unzip file -> Merge CSV -> Load to DataFrame
    if (File.Exists(fileName))
    {
        var extractedDirectory = Path.Combine(Directory.GetCurrentDirectory(), DATASET_DIRECTORY);

        try
        {
            ZipFile.ExtractToDirectory(fileName, extractedDirectory);
            MergeCsv(extractedDirectory, DATASET_MERGED_CSV, SEARCH_PATTERN, ExcludeFiles);

            return Microsoft.Data.Analysis.DataFrame.LoadCsv(DATASET_MERGED_CSV);
        }
        catch (Exception e)
        {
            display(e);
            throw;
        }

    }

    return new Microsoft.Data.Analysis.DataFrame();
}

Now, we'll load the DataFrame from files.

// Load Dataset
var cricketDataFrame = await LoadDatasetAsync(DATASET_URL, DATASET_FILE);

A preview of this dataframe looks as below

3.c Filtering

As we are considering data for initial 6 overs of a match, we need to remove data for further overs.

// Filter dataset to six overs
var filterColumn = cricketDataFrame.Columns[CSV_COLUMN_BALL].ElementwiseLessThanOrEqual(OVER_THRESHOLD);
var sixOverDataFrame = cricketDataFrame.Filter(filterColumn);

4. Data Analysis

We will analyze the filtered dataframe by executing different commands

4.a Top records

sixOverDataFrame.Head(TOP_COUNT)

4.b Information about the dataset

// Information about the dataset
sixOverDataFrame.Info()

We can see there are around 95264 records present in our dataset of six overs. First row represents the datatype of each column.

4.c Description about the Dataset

// Description about the dataset
sixOverDataFrame.Description()

Min value of 0.1 and Max value of 5.9 confirms the dataset is for 6 overs only.

4.d Score and Total Score

The dataset contains various runs scored every ball such as runs_off_bat, extras, wides, legbyes etc. There is no column for runs scored per ball. We could get it by adding 'runs_off_bat' and 'extras'.

Score = runs_off_bat + extras

Similary we can get total runs per ball by having a cumulative sum of runs per ball in a inning.

Total Score = CumulativeSum(Score/inning)

We need to add a new Column(Score) to our dataframe. PrimitiveDataFrameColumn class is used to add a new column.

// Add a column for Total Score
sixOverDataFrame.Columns.Add(new PrimitiveDataFrameColumn<int>(CSV_COLUMN_SCORE, sixOverDataFrame.Rows.Count));
sixOverDataFrame.Columns[CSV_COLUMN_SCORE] = sixOverDataFrame.Columns[CSV_COLUMN_RUNS_OFF_BAT] + sixOverDataFrame.Columns[CSV_COLUMN_EXTRAS];

// Information about the dataframe
sixOverDataFrame.Info()

We could see a new column (score) has been added now.

Now, we will add Total Score which represents the total number of runs scored till that ball.

/// <summary>
/// Adds a total score column to a DataFrame
/// </summary>
/// <param name="df">DataFrame to be updated</param>
public void AddTotalScorePerBall(Microsoft.Data.Analysis.DataFrame df)
{
    // calculate total_score per ball
    df.Columns.Add(new PrimitiveDataFrameColumn<Single>(CSV_COLUMN_TOTAL_SCORE, df.Rows.Count));
    Single previousMatchId = -1;
    Single previousInning = -1;
    Single previousScore = -1;

    foreach (var dfRow in df.Rows)
    {
        var matchId = (Single)dfRow[0];
        var inning = (Single)dfRow[4];
        var score = (Single)dfRow[df.Columns.Count - 2]; // Score

        if (previousMatchId == -1) // First time
        {
            // Reset
            previousMatchId = matchId;
            previousInning = inning;

            if (previousScore == -1)
            {
                previousScore = score;
            }
        }

        if (matchId == previousMatchId && inning == previousInning)
        {
            Single newScore = previousScore + score;

            // Total Score
            dfRow[df.Columns.Count - 1] = (Single)newScore;
            previousScore = newScore;
        }
        else
        {
            // Total Score
            dfRow[df.Columns.Count - 1] = (Single)score;

            // Reset
            previousMatchId = -1;
            previousInning = -1;
            previousScore = score;
        }
    }    
}

// Adds a total score column per pall
AddTotalScorePerBall(sixOverDataFrame);

// Display dataframe
display(sixOverDataFrame)

4.e Feature Selection

There are 24 total columns in the dataset now. Not all of them are relevant for making prediction. There are different strategies available for Feature selection such as confusion matrix, seaborn plot which can be applied to select appropriate features. In order to keep things simple, I have chosen few parameters after performing different combinations. The features selected are as follows.

Input Features

venue
innings
ball
batting_team
bowling_team
striker
non_striker
bowler

Output

total_runs

// Remove extra features
sixOverDataFrame.Columns.Remove("match_id");
sixOverDataFrame.Columns.Remove("season");
sixOverDataFrame.Columns.Remove("start_date");
sixOverDataFrame.Columns.Remove("runs_off_bat");
sixOverDataFrame.Columns.Remove("extras");
sixOverDataFrame.Columns.Remove("wides");
sixOverDataFrame.Columns.Remove("noballs");
sixOverDataFrame.Columns.Remove("byes");
sixOverDataFrame.Columns.Remove("legbyes");
sixOverDataFrame.Columns.Remove("penalty");
sixOverDataFrame.Columns.Remove("player_dismissed");
sixOverDataFrame.Columns.Remove("other_wicket_type");
sixOverDataFrame.Columns.Remove("other_player_dismissed");
sixOverDataFrame.Columns.Remove("score");
sixOverDataFrame.Columns.Remove("wicket_type");

We could see, we have 9 columns only relevant to our problem.

// Information about the dataset
sixOverDataFrame.Info()

We are done with data analysis and have the dataset ready for model training and prediction which will be described in next section.

Machine Learning - Training and Prediction

In this section, we'll be leveraging the dataset prepared and use it to for making prediction by using ML.Net framework.

Note In order to make predictions, ML.Net requires a csv file that could be loaded by ML.Net API's. There is no direct way of loading a DataFrame to make predictions and DataFrame API doesn't expose any API to save the DataFrame to a csv file.

I discovered a library 'Danny' which allows creating, manipulating and saving the DataFrame to a CSV file.

Link: https://github.com/bhrnjica/daany

I'll be using this library to recreate the DataFrame, perform modifications same as above and save it as a CSV file to be used for prediction. I found it to be very flexible. As this is kind of repetitive thing, feel free to skip to Training section for model creation and training.

Data Manipulation

We'll be performing below manipulation/transformation on the dataset

Take records till 6 overs
Remove missing values
Add 'score' column representing runs per ball
Add 'total_score' column to get the total score till the current ball

Note
The csv dataset file has some values which contains comma ',' in the cell value such as venue having "Simonds Stadium, South Geelong". As a result of which, ML.Net LoadFromTextFile(...) API loads in a wrong manner and consider them to be two separate value instead of single. In order to overcome this, I have written a helper function CreateCsvAndReplaceSeparatorInCells(...) to replace comma with the specified character.

/// <summary>
/// Replace a character in a cell of csv with a defined separator
/// </summary>
/// <param name="inputFile">Name of input file</param>
/// <param name="outputFile">Name of output file</param>
/// <param name="separator">Separator such as comma</param>
/// <param name="separatorReplacement">Replacement character such as underscore</param>
private static void CreateCsvAndReplaceSeparatorInCells(string inputFile, string outputFile, char separator, char separatorReplacement)
{
    var culture = CultureInfo.InvariantCulture;
    using var reader = new StreamReader(inputFile);
    using var csvIn = new CsvReader(reader, new CsvConfiguration(culture));
    using var recordsIn = new CsvDataReader(csvIn);
    using var writer = new StreamWriter(outputFile);
    using var outCsv = new CsvWriter(writer, culture);

    // Write Header
    csvIn.ReadHeader();
    var headers = csvIn.HeaderRecord;
    foreach (var header in headers)
    {
        outCsv.WriteField(header.Replace(separator, separatorReplacement));
    }
    outCsv.NextRecord();

    // Write rows
    while (recordsIn.Read())
    {
        var columns = recordsIn.FieldCount;
        for (var index = 0; index < columns; index++)
        {
            var cellValue = recordsIn.GetString(index);
            outCsv.WriteField(cellValue.Replace(separator, separatorReplacement));
        }
        outCsv.NextRecord();
    }
}

// Replace separator in cells
CreateCsvAndReplaceSeparatorInCells(DATASET_MERGED_CSV,  DATASET_ALL_CSV, ',', '_');

// Load CSV
Daany.DataFrame cricketDaanyDataFrame = Daany.DataFrame.FromCsv(DATASET_ALL_CSV);
display(cricketDaanyDataFrame.RowCount());

Remove Missing values

// Remove missing values
var missingValues = cricketDaanyDataFrame.MissingValues();

List<string> nonNullColumns = new List<string>();
foreach (var dfColumn in cricketDaanyDataFrame.Columns)
{
    if (!missingValues.ContainsKey(dfColumn))
    {
        display(dfColumn);
        nonNullColumns.Add(dfColumn);
    }
}

cricketDaanyDataFrame = cricketDaanyDataFrame[nonNullColumns.ToArray()];
display(cricketDaanyDataFrame);

Filter dataset to include till 6 overs and discard beyond that

// Filter dataset to include till 6 overs and discard beyond that
var sixBallDaanyDataFrame = cricketDaanyDataFrame.Filter(CSV_COLUMN_BALL, OVER_THRESHOLD, FilterOperator.LessOrEqual);

Add 'score' column

Calculating Total Score

There is no direct way to calculate the total score as it has to be cumulative sum and restricted to 6 overs only in the dataset. I have written a function to calculate the Total Score

/// <summary>
/// Adds a total_score column to the DataFrame by calculating the cumulative sum
/// </summary>
/// <param name="df">DataFrame to which new column is added</param>
public void AddTotalScorePerBallDaany(Daany.DataFrame df)
{
    // https://github.com/bhrnjica/daany
    // calculate total_score per ball
    df.AddCalculatedColumn(CSV_COLUMN_TOTAL_SCORE, (r, i) =>
    {
        var response = Convert.ToSingle(r[CSV_COLUMN_SCORE]);
        return 0;
    });

    int previousMatchId = -1;
    int previousInning = -1;
    float previousScore = -1;
    int rowIndex = 0;

    foreach (var dfRow in df.GetRowEnumerator())
    {
        var matchId = (int)dfRow[0];
        var inning = (int)dfRow[4];
        var score = Convert.ToSingle(dfRow[df.Columns.Count - 2]); // Score

        if (previousMatchId == -1) // First time
        {
            // Reset
            previousMatchId = matchId;
            previousInning = inning;

            if (previousScore == -1)
            {
                previousScore = score;
            }
        }

        if (matchId == previousMatchId && inning == previousInning)
        {
            float newScore = previousScore + score;

            // Total Score
            dfRow[df.Columns.Count - 1] = newScore;

            df[rowIndex, df.Columns.Count - 1] = newScore;
            previousScore = newScore;
        }
        else
        {
            // Total Score
            dfRow[df.Columns.Count - 1] = score;
            df[rowIndex, df.Columns.Count - 1] = score;

            // Reset
            previousMatchId = -1;
            previousInning = -1;
            previousScore = score;
        }

        rowIndex++;
    }
}

// Add Total score per ball to Daany Dataframe
AddTotalScorePerBallDaany(sixBallDaanyDataFrame);
display(sixBallDaanyDataFrame);

// Filter columns
sixBallDaanyDataFrame = sixBallDaanyDataFrame[CSV_COLUMN_VENUE, CSV_COLUMN_INNINGS, CSV_COLUMN_BALL, CSV_COLUMN_BATTING_TEAM, CSV_COLUMN_BOWLING_TEAM, CSV_COLUMN_STRIKER, CSV_COLUMN_NON_STRIKER, CSV_COLUMN_BOWLER, CSV_COLUMN_TOTAL_SCORE];
display(sixBallDaanyDataFrame);

Final Dataset Columns

Venue
Innings
Ball
Batting_Team
Bowling_Team
Striker
Non_Striker
Bowler
Total_Score

Save DataFrame to CSV

// Save DataFrame to CSV
Daany.DataFrame.ToCsv(DATASET_CLEANED_CSV, sixBallDaanyDataFrame);

Training

Data Classes

We need to create few data structures to map the columns within our dataset.

Match

Match contains input features involved in prediction, its the list of filtered columns.

/// <summary>
/// Represents the input features involved in prediction
/// </summary>
public class Match
{
    /// <summary>
    /// Match venue
    /// </summary>
    [LoadColumn(0)]
    public string Venue { get; set; }

    /// <summary>
    /// Innning within a Match
    /// </summary>
    [LoadColumn(1)]
    public float Inning { get; set; }

    /// <summary>
    /// Current Ball being thrown
    /// </summary>
    [LoadColumn(2)]
    public float Ball { get; set; }

    /// <summary>
    /// Name of the batting team
    /// </summary>
    [LoadColumn(3)]
    public string BattingTeam { get; set; }

    /// <summary>
    /// Name of the bowling team
    /// </summary>
    [LoadColumn(4)] 
    public string BowlingTeam { get; set; }

    /// <summary>
    /// Batsman on strike
    /// </summary>
    [LoadColumn(5)]
    public string Striker { get; set; }

    /// <summary>
    /// Non striker batsman
    /// </summary>
    [LoadColumn(6)]
    public string NonStriker { get; set; }

    /// <summary>
    /// Current bowler
    /// </summary>
    [LoadColumn(7)]
    public string Bowler { get; set; }

    /// <summary>
    /// Total score till the current ball
    /// </summary>
    [LoadColumn(8)]
    public float TotalScore { get; set; }

    public override string ToString()
    {
        var sb = new StringBuilder();
        sb.Append($"Venue: {Venue}");
        sb.Append($"\nBatting Team: {BattingTeam}");
        sb.Append($"\nBowling Team: {BowlingTeam}");
        sb.Append($"\nInning: {Inning}");
        sb.Append($"\nBall: {Ball}");
        sb.Append($"\nStriker: {Striker}");
        sb.Append($"\nNon-Striker: {NonStriker}");
        sb.Append($"\nBowler: {Bowler}");

        return sb.ToString();
    }
}

MatchScorePrediction

/// <summary>
/// Manages the score prediction 
/// </summary>
public class MatchScorePrediction
{
    /// <summary>
    /// Total runs scored by the team at the specified ball
    /// </summary>
    [ColumnName("Score")]
    public float TotalScore { get; set; }
}

Load Dataset

Now, we'll load the dataset into MLContext of ML.Net framework for making prediction.

// Load the dataset
var mlContext = new MLContext(seed:1);
IDataView data = mlContext.Data.LoadFromTextFile<Match>(
    path:DATASET_CLEANED_CSV,
    hasHeader: true,
    separatorChar: ',');

Split Dataset

Its always advised to split the dataset into train and test for better outcome.

// Split dataset
var trainTestData = mlContext.Data.TrainTestSplit(data, 0.2); // Training/Test : 80/20

Transform Dataset

Now, we'll create Machine learning pipeline to train our model. OneHotEncoding is used for non-numerical input variables. It converts them into numeric format which an ML algorithm understands.

// Transform
var dataProcessPipeline = mlContext.Transforms.CopyColumns(outputColumnName: "Label", inputColumnName: nameof(Match.TotalScore))

    .Append(mlContext.Transforms.Categorical.OneHotEncoding("VenueEncoded", nameof(Match.Venue)))
    .Append(mlContext.Transforms.Categorical.OneHotEncoding("BattingTeamEncoded", nameof(Match.BattingTeam)))
    .Append(mlContext.Transforms.Categorical.OneHotEncoding("BowlingTeamEncoded", nameof(Match.BowlingTeam)))
    .Append(mlContext.Transforms.Categorical.OneHotEncoding("StrikerEncoded", nameof(Match.Striker)))
    .Append(mlContext.Transforms.Categorical.OneHotEncoding("NonStrikerEncoded", nameof(Match.NonStriker)))
    .Append(mlContext.Transforms.Categorical.OneHotEncoding("BowlerEncoded", nameof(Match.Bowler)))

    .Append(mlContext.Transforms.Concatenate("Features", 
        "VenueEncoded",
                        nameof(Match.Inning),
                        nameof(Match.Ball),
                        "BattingTeamEncoded",
                        "BowlingTeamEncoded",
                        "StrikerEncoded",
                        "NonStrikerEncoded",
                        "BowlerEncoded"
    ));

Train

Fit() method allows training of model on a given dataset. Here we have used FastTree regression algorithm.

// Train
var trainingPipeline = dataProcessPipeline.Append(mlContext.Regression.Trainers.FastTree(labelColumnName: "Label", featureColumnName: "Features"));
var trainedModel = trainingPipeline.Fit(trainTestData.TrainSet);

Evaluate

We'll evaluate the model on test dataset

var predictions = trainedModel.Transform(trainTestData.TestSet);
var metrics = mlContext.Regression.Evaluate(predictions, "Label", "Score");
display($"*************************************************");
display($"*       Model quality metrics evaluation         ");
display($"*------------------------------------------------");
display($"*       RSquared Score:      {metrics.RSquared:0.##}");
display($"*       Root Mean Squared Error:      {metrics.RootMeanSquaredError:#.##}");

Save Model

// Save
var savedPath = Path.Combine(Directory.GetCurrentDirectory(), MODEL_FILE);
mlContext.Model.Save(trainedModel, trainTestData.TrainSet.Schema, savedPath);
display($"The model is saved to {savedPath}");

Prediction

Lastly, we'll make prediction on unseen data using PredictionEngine from ML.Net and see how much variance is there between actual and predicted score.

// Predict
display("*********** Predict...");
var predictionEngine = mlContext.Model.CreatePredictionEngine<Match, MatchScorePrediction>(trainedModel);
var match = new Match
{
    Ball = 3.4f,
    BattingTeam = "India",
    BowlingTeam = "New Zealand",
    Bowler = "CJ Anderson",
    Inning = 1,
    Striker = "V Kohli",
    NonStriker = "Yuvraj Singh",
    Venue = "Vidarbha Cricket Association Stadium_ Jamtha"
};

// make the prediction
var prediction = predictionEngine.Predict(match);

// report the results
display($"Match Info:\n\n{match} ");
display($"\n^^^^^^ Prediction:  {prediction.TotalScore} ");
display($"**********************************************************************");
display($"Predicted score: {prediction.TotalScore:0.####}, actual score: 20");
display($"**********************************************************************");

The model has performed quite well as the predicted score is quite near to the actual score. We need to see how it perform with different set of data. We have used FastTree algorithm which seems to be good. There are lot of things we can try to improve here by choosing different algorithm, features, and large dataset etc.

Conclusion

Cricket is an interesting game with some nerve wracking moments during the match which keeps you on the toes. With this excitement and enthusiasm, a lot of data gets generated per ball that gets delivered over the field. This humungoes data is intelligently utilized by Data scients, Machine learning and AI practitioner to make insights and predictions about various aspects of the game. In this notebook, I have just scratched the possibilities that could be explored in the game of cricket. We took the dataset from web, cleaned, analyzed and finally able to make a prediction of finding a score. We have explored the usage some great libraries such as DataFrame, ML.Net and Daany.

I hope you have enjoyed the notebook.

Resources

Contact

Linktree: https://linktr.ee/praveenraghuvanshi
Twitter: @praveenraghuvan\
Github: praveenraghuvanshi\
Blog: https://praveenraghuvanshi.github.io\
Dev.To: https://dev.to/praveenraghuvanshi\
Website: https://www.praveenraghuvanshi.com\
Email: contact@praveenraghuvanshi.com

Microsoft AZ-900 Certification: A 3 week Plan

Praveen Raghuvanshi — Thu, 04 Mar 2021 18:34:08 +0000

Disclaimer: Any views or opinions represented in this blog are personal and belong solely to the blog owner and do not represent those of people, institutions or organizations that the owner may or may not be associated with in professional or personal capacity, unless explicitly stated.

Why Certification?

A lot of people have a question in their mind, whether they should go for a certification or not? We have people from both the school of thought. One in favor of certification and others not very keen in going for it. I would consider certification as a platform to explore the technology and provides an opportunity to validate your understanding. Its more than reading a book on a topic. Definitely, there is no substitute for experience but not everyone gets opportunity to explore the breadth and depth of a technology. A certification broadens your horizon that can be utilized in solving the real time problems. It helps a fresher/inexperienced person to jump start on a technology and an experienced person to explore the breadth of a technology. Certification introduced me to new topics in Azure. Certifications are beneficial and more relevant in below case.

No Knowledge < Knowledge < Knowledge + Certification

Please refer Microsoft page for more reasons and stats.

AZ-900?

If you are new to Azure or have some experience in it, this is the entry level certification. The exam doesn't assume you to be an expert in Azure and only tests your knowledge on the topic at a high level. The main objective is to give an introduction to the services available in Azure. It spans across compute, networking, governance, policies, budgeting etc. For more details please check the certification page

Exam

Some information about the exam

Pricing

Regular

The cost of exam is $99 only at the time of publication of this post.

COVID-19

For jobseekers affected by COVID-19, the exam is available at a special price of $15. Please refer certification page for eligibility

100% Discount

A discount voucher of 100% could be obtained and applied by attending two sessions of 3-hour each spanning 2 days. To register for the event please visit Microsoft Virtual Training Days and register for Microsoft Azure Virtual Training Day: Fundamentals at your convenient date. After completing the training, you'll receive an email for scheduling the exam at no cost. It can be scheduled after 5 days of completing the training.

Structure (Based on my attempt)

Passing score > 700
No of questions: 37
Time limit: 60 minutes
Type of Questions
- Scenario based
- One answer correct
- Multiple answer correct
- Match the answers
- Select an item from drop down

Scheduling

The exam can be scheduled through different platforms based on your eligibility as mentioned in certification page. I got it scheduled through Pearson VUE and it was a smooth process.

Plan

Planning for an exam is of utmost importance and you need to be committed to it. People tend to fall into vicious circle of learning which never completes and they are unable to schedule the exam. More or less they go into analysis paralysis syndrome and have a fear of never completing the exam content or ready for it. I too had the same feeling and kept postponing it months after months. The way to overcome is to analyze the preparation time and schedule it.

Here I'll describe the plan I have followed to clear the exam. Most of it is relevant irrespective of registration process(Regular/COVID-19/100% discount). I did utilize the 100% discount path for registration and attended virtual training on 22/23 October 2020. Got an email from Microsoft after couple of days describing the process of registering an exam. I did kept postponing the preparation and exam. Finally got it scheduled on Feb 1, 2021 for Feb 22, 2021 after looking at the Microsoft Learn content.

I have had prior experience of working on Azure and had some knowledge on services available in Azure. Still, there were some areas in certification curriculum which were unknown to me. I did plan for three weeks, gave the exam and cleared it. An effort of 2 hours on weekdays and 4-5 hours on weekends is required. Please find below 3 week plan.

Week 1: MS Learn Modules

First week was committed to Microsoft Learn modules. There were total 6 modules and I planned to cover at least 3 of them. First week is very important in laying the foundation and getting into rhythm. Focus on getting the concepts right instead of rushing to complete. Re-read the modules if required. Do not proceed until and unless things are clear. The modules covered are as follows and can be accessed at Microsoft Learn Path

Week 2: MS Learn and Scott Duffy

By this time, I got fair understanding of concepts in Azure and have the momentum for certification. This week was dedicated to complete remaining 3 modules. Once all modules were completed, I did a revision of all the modules at a high level. Analyzed the weak area and re-read about them. During this week, I also started to watch a Udemy course on AZ-900 by Scott Duffy which is a great course and can be considered as a video version of Microsoft Learn modules. Its a 8.5 hrs course and I did complete half of it in Week 2. The course is updated with recent changes done in the month of January, 2021. There is a practice test at the end of the course. It helps to understand the strong and weak areas with respect to Azure concepts. Take this practice test seriously and work on improving the weak areas.

Week 3: Scott Duffy and Practice tests

Completed Scott Duffy course during this week. After that took 3 test by Scott Duffy and 6 tests by Mohit Kumar. Both have good explanations and covers a good amount of questions and concepts. Do not mug up the answers, focus on the explanation provided. Re-visit any concept that is unclear.

By the end of 3rd week, a good understanding on AZ-900 question pattern gets developed and practice questions boosts confidence of giving the exam.

Exam Day

After 3 weeks of preparation, I gave the exam. The exam window opens 30 mins prior to scheduled timing in order to get things ready. The questions were scenario based and requires an understanding of the concepts. There were multiple choice questions which tries to confuse by giving near by options. There were some questions for which I had no knowledge but the percentage of such questions is very less. Finally gave the exam and once finished result(Pass/Fail) is shown immediately.

I hope you got some idea on AZ-900 and I strongly believe 3 week plan is relevant and enough to prepare.

Good luck and share your views on the above plan.

Contact

Twitter: @praveenraghuvan

Github: praveenraghuvanshi

Blog: praveenraghuvanshi.github.io

End-to-End Serverless Deep Neural Network with Azure Functions and ML.Net

Praveen Raghuvanshi — Wed, 09 Sep 2020 23:52:46 +0000

Introduction

Have you ever wondered how does a machine classify among different species of animals just by looking at a image? How does a machine able to predict whether there is a cat or dog in an image? A rule based system might work for some images, however when a new or unseen image is given to the machine, there are chances prediction may go wrong. Also, rule based system might not scale for large number of images. Deep Neural Network comes as a rescue for such kind of problems and it really scales well. Deep Neural Network simplifies and help classify the things. In this tutorial, I will cover some concepts related to Deep Neural Networks and show how to build a serverless image classification application using Microsoft Azure Functions and ML.Net framework. The implementation will be in C# language.

If you prefer video, please visit Serverless ANZ 2020.

Before I jump into code, let's revisit some of the concepts used in this tutorial.

Serverless

source: https://parkardigital.com/serverless-cloud-computing/

Serverless is an execution model where the cloud provider such as AWS, Azure or Google cloud is responsible for executing a piece of code by dynamically allocating the resources. The infrastructure is managed and maintained by the Cloud providers. The consumer is billed only for the resources consumed to run the code. It's an efficient way to reduce the overhead and cost of managing the servers. The code that is executed is in the form of a function, hence sometimes its called 'Function as a Service(Faas). It allows you to focus on writing business logic instead of spending time on managing the infrastructure.

Azure Functions

Azure Functions is a serverless framework developed by Microsoft and provides rich capabilities. It allows you to run a small piece of code(called 'Functions') without worrying about application infrastructure. It leverages event-driven capabilities and runs the code whenever a supported trigger happens. It support multiple triggers such as HTTP, Blob, Queue, Service Bus, etc. I'll be deploying the prediction part of machine learning pipeline to Azure functions which will take an image as an input and throw back a prediction.

Deep Neural Network

source: https://quantdare.com/what-is-the-difference-between-deep-learning-and-machine-learning/

Deep Learning is subset of Machine learning in Artificial Intelligence. It is capable of learning features on its own compared to machine learning where features are identified manually. A neural network, is a technology built to simulate the activity of a human brain such as pattern recognition and its done through a set of layers. When we have multiple layers present in a neural network, its called as Deep Neural network. It mainly comprise of an input layer, multiple hidden layers and a final output layer. I'll be using a pre-trained deep neural network such as ResNet18 in this tutorial. Pre-trained means its already trained on large amount of images and we'll use that learning to make predictions for new images being given to the machine learning pipeline. Its called as Transfer Learning.

source: https://quantdare.com/what-is-the-difference-between-deep-learning-and-machine-learning/

ML.Net

ML.Net is a cross-platform framework from Microsoft for developing Machine learning models in the .Net ecosystem. It allows .Net developers to solve business problems using machine learning algorithms leveraging their preferred language such as C# or F#. It's highly scalable and used within Microsoft in many of its products such as Bing, Powerpoint, etc. I'll be using image classification algorithms for this tutorial.

Application

I am going to create an application that will classify an image based on 1000 classes present in ImageNet dataset. First we'll create a local console application and thereafter move it to Azure function(Local and Cloud). The framework used for this exercise is ML.Net and language is C#.

Prerequisites

IDE - Visual Studio / Visual Studio Code
REST Client - Postman
Serverless Deployment
- Local :
- Azure function - Prediction Logic
- Blob Storage via Storage emulator - Machine learning Model
- Cloud: Azure subscription (Only for cloud deployment to Azure Function) and Azure blob storage

Cloud Architecture

Image Classification - Serverless (Azure Function) - Local

As this is a classification problem where an image is given to the App and its class is predicted. In Azure, we can achieve this using different options available such as Azure App service, Azure Functions. Azure App service uses App service plan and is more expensive compared to Azure Function which is serverless and billed based on usage only. We need an Azure subscription in order to use Azure Functions. Limited time subscription can be obtained by registring at Azure portal.

I'll be using a pre-trained MobileNet V2 model for predictions. In order to use this, donwload Mobilenet and upload it to Azure blob storage both locally(storage emulator) and cloud. I have stored it in 'serverlessdnnstorage' container and file name as mobilenetv2-7.onnx

source: https://ai.googleblog.com/2018/04/mobilenetv2-next-generation-of-on.html

Steps in creating a image classification serverless function using Azure Function

Create Azure Function project using visual studio - Http Trigger
Add Http trigger function 'ClassifyImage' to the project
Add nuget packages
- Microsoft.ML
Download MobileNet model from Blob and access it in function app
- Local - Storage emulator
- Cloud - Upload to Azure blob storage and access it from there.
Load model
Make prediction and return result as a response
Validate Azure function endpoint through Postman REST client.

I'll first create Azure function locally using template present in Visual Studio. Once everything is working fine, I'll deploy azure function to cloud.

Let's create a new project and select 'Azure Function' from the project template and add this project to the existing solution.

In the next dialog, give a name to the project 'ServerlessDNNFunction' and click create.

This will add a new project to ServerlessDNN solution. Build the solution, just to ensure there are no errors. A default function(Function1) will be added.

Rename 'Function1.cs' file to ClassifyImage.cs and ensure it gets changed in the function attribute as shown below.

namespace ServelessDNNFunction
{
    public static class ClassifyImage
    {
        [FunctionName("ClassifyImage")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            string name = req.Query["name"];

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            dynamic data = JsonConvert.DeserializeObject(requestBody);
            name = name ?? data?.name;

            string responseMessage = string.IsNullOrEmpty(name)
                ? "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response."
                : $"Hello, {name}. This HTTP triggered function executed successfully.";

            return new OkObjectResult(responseMessage);
        }
    }
}

Press F5 to run the project and it should generate an endpoint like http://localhost:7071/api/ClassifyImage

Hit the url in browser and message 'This HTTP triggered function executed successfully.
Remove 'get' from the HttpTrigger attribute of the function as we only need 'post' to upload an image.
Ensure AuthorizationLevel is Anonymous. Do not do this on Production.

In order to upload an image from postman, create a POST request with above URL and select 'Body' as binary. Select a file by clicking on 'Select File' in the Body tab

Add below code in Azure function to read and save uploaded image to temp directory

STEP-1: Upload model to Blob storage

In order to use the pre-trained model, we need to upload mobilenetv2-7.onnx (14MB) to some storage. The file can be accessed through disk or Storage Blob. In order to keep the experience same for both local and cloud, I have used storage emulator for storing/reading the model. The model can be uploaded through Microsoft Azure Storage Explorer.

Please follow steps to create a container 'serverlessdnn' and upload mobilenetv2-7.onnx

Launch Azure Storage Explorer and select 'Local & Attached' account.

Select 'serverlessdnn' and click on Upload to upload mobilenetv2-7.onnx

Model will be uploaded to blob and available to be used.

STEP-2: Load model from Blob Storage to Azure Function

Navigate back to Azure function in visual studio. In order to load model we need to have below changes

Open local.settings.json and replace with below content

{
    "IsEncrypted": false,
    "Values": {
      "AzureWebJobsStorage": "UseDevelopmentStorage=true",
      "FUNCTIONS_WORKER_RUNTIME": "dotnet",
      "CONTAINER_NAME": "serverlessdnn",
      "MODEL_FILE": "mobilenetv2-7.onnx"
      }
}

Add 'Azure.Storage.Blobs' nuget package
Add below items at the start of ClassifyImage function

// Load App Settings
var containerName = Environment.GetEnvironmentVariable("CONTAINER_NAME") ?? "serverlessdnn";
var connectionString = Environment.GetEnvironmentVariable("AzureWebJobsStorage") ?? "UseDevelopmentStorage=true";
var modelName = Environment.GetEnvironmentVariable("MODEL") ?? "mobilenetv2-7.onnx";

Add 'ReadModelFromBlob' method to read blob from container and return a stream of model.

private static Stream ReadModelFromBlob(ILogger log, string connectionString, string containerName, string modelName)
{
   try
   {
       BlobContainerClient container = new BlobContainerClient(connectionString, containerName);
       container.CreateIfNotExists(PublicAccessType.Blob);
       var blockBlob = container.GetBlockBlobClient(modelName);

       log.LogInformation($"Loading Model : {modelName}");

       var modelStream = new MemoryStream();
       blockBlob.DownloadTo(modelStream);
       return modelStream;
   }
   catch (Exception e)
   {
       log.LogError(e, e.Message);
       throw;
   }
}

We need to save this model file to temp path as ML.Net support loading of ONNX model from a path and not from the stream
Add SaveModel(...) to save the model to temp path

private static string SaveModel(ILogger log, string tempPath, string modelName, Stream modelStream)
{
    string savedModelPath = Path.Combine(tempPath, modelName);

    try
    {
         using (var fileStream = File.Create(savedModelPath))
         {
             modelStream.Seek(0, SeekOrigin.Begin);
             modelStream.CopyTo(fileStream);
         }
    }
    catch (Exception e)
    {
         log.LogError(e, e.Message);
         throw;
    }

    return savedModelPath;
}

Add below line to load the model from blob and save to temp path in ClassifyImage function

// STEP-2: Load model from Blob storage and save to temp path
string tempPath = CreateTempDirectory(log);

var modelStream = ReadModelFromBlob(log, connectionString, containerName, modelName);
var savedModelPath = SaveModel(log, tempPath, modelName, modelStream);
log.LogInformation($"Model saved to : {savedModelPath}");

STEP-3: Load ONNX model into ML.Net MLContext

Add below nuget packages
- Microsoft.ML
- Microsoft.ML.Vision
- Microsoft.ML.ImageAnalytics
In order to load the ONNX model, we need to define schema for Model input and output.

Add a new struct 'InputSettings' for holding attributes of image such as height and width. As we are working with MobileNet, it expects input image to be of size 224 x 224

public struct ImageSettings
{
     public const int imageHeight = 224;
     public const int imageWidth = 224;
}

Add a new class 'ModelInput' which holds information about the input image that will be uploaded and its class as Label. ImageType attribute allows defining the size of image.

public class ModelInput
{
    [ImageType(ImageSettings.Height, ImageSettings.Width)]
    public Bitmap ImageSource { get; set; }
}

Add a new class 'ModelOutput' which holds information about the output image that will be uploaded.

public class ModelOutput
{
    [ColumnName("mobilenetv20_output_flatten0_reshape0")]
    public float[] Score { get; set; }
}

Score attribute is important. The value 'mobilenetv20_output_flatten0_reshape0' is the name of output layer of the model. You can get it by opening mobilenetv2-7.onnx file using Netron application. As seen below input layer name is 'data' and output layer name is 'mobilenetv20_output_flatten0_reshape0'.

Make a note of input and output layer names and use it.

Now we'll create MLContext and load the ONNX model

Add 'Microsoft.ML.OnnxTransformer' nuget packages
For ML pipeline, three transformations are done
- ResizeImages: It is to ensure the size of input size is 224 x224
- ExtractPixels: Its used to extract pixel values from the data specified in the column. A pixel value varies from 0-255.
- ApplyOnnxModel: It loads the ONNX model from the specified path
Data is loaded in a lazy manner using MLContext.
In order to load the data into pipeline, Fit(...) method is called.

// STEP-3: Load ONNX model into ML.Net MLContext
var modelInputName = "data";
var modelOutputName = "mobilenetv20_output_flatten0_reshape0";

var mlContext = new MLContext(seed: 1);

var emptyData = new List<ModelInput>();
var data = mlContext.Data.LoadFromEnumerable(emptyData);

var pipeline = mlContext.Transforms.ResizeImages(resizing: ImageResizingEstimator.ResizingKind.Fill, outputColumnName: modelInputName, imageWidth: ImageSettings.Width, imageHeight: ImageSettings.Height, inputColumnName: nameof(ModelInput.ImageSource))
        .Append(mlContext.Transforms.ExtractPixels(outputColumnName: modelInputName))
.Append(mlContext.Transforms.ApplyOnnxModel(modelFile: savedModelPath, outputColumnName: modelOutputName, inputColumnName: modelInputName));

var model = pipeline.Fit(data);

STEP-4: Prediction

In order to predict, we need to supply the image uploaded to Azure function and pass it to the ML pipeline. In ML.Net PrectionEngine has Predict(...) API to perform prediction on a single input image. maxScore gives the predicted class

// STEP-4: Prediction
Bitmap testImage = (Bitmap)Image.FromStream(req.Body);

ModelInput inputData = new ModelInput()
{
    ImageSource = testImage
};

var predictionEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(model);
var prediction = predictionEngine.Predict(inputData);
var maxScore = Convert.ToInt16(prediction.Score?.Max());

STEP-5: Return Predicted value as a response to Function API

// STEP-5: Return Predicted value as a response to Function API
string responseMessage = $"Predicted: {maxScore}";
return new OkObjectResult(responseMessage);

Build the solution
ERROR In case below error is reported, add win-x64 to the ServerlessDNNFunction.csproj file

1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(4364,5): warning MSB3026: Could not copy "C:\Users\<username>\.nuget\packages\microsoft.ml.onnxruntime\1.4.0\runtimes\osx-x64\native\libonnxruntime.dylib.dSYM\Contents\Resources\DWARF\libonnxruntime.dylib" to "bin\Debug\netcoreapp3.1\bin\runtimes\osx-x64\native\libonnxruntime.dylib.dSYM\Contents\Resources\DWARF\libonnxruntime.dylib". Beginning retry 1 in 1000ms. Could not find a part of the path 'bin\Debug\netcoreapp3.1\bin\runtimes\osx-x64\native\libonnxruntime.dylib.dSYM\Contents\Resources\DWARF\libonnxruntime.dylib'.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
    <Platforms>AnyCPU</Platforms>
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Azure.Storage.Blobs" Version="12.5.1" />
    <PackageReference Include="Microsoft.ML" Version="1.5.1" />
    <PackageReference Include="Microsoft.ML.ImageAnalytics" Version="1.5.1" />
    <PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.4.0" />
    <PackageReference Include="Microsoft.ML.OnnxTransformer" Version="1.5.1" />
    <PackageReference Include="Microsoft.ML.Vision" Version="1.5.1" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.7" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

STEP-6: Validate using REST client - Postman

Run Project and hit the URL with an image of a dog from Postman as shown below

Input image

Predicted: 212 (English setter)

Predicted class is 212 which corresponds to 'English Setter', one of the dog breed. The prediction may not be very accurate as we have used MobileNet which doesn't have a very good accuracy. The reason of using it is small model size. Azure function can process small models only in consumption plan. If you have a high configuration of Azure Function, a large model could be used. Also, memory allocated in consumption plan is 1.5 GB only for the CPU.

Image Classification - Serverless (Azure Function) - Cloud

Now we are going to deploy this function app on Cloud(Azure). We need an Azure subscription for it. Deploying to azure is pretty easy through Visual Studio. Steps in deploying the Azure Function to the cloud is as follows.

Prerequisites

Azure Subscription
Azure Resource Group

Deploy to Cloud
Before we deploy function to cloud, please ensure mobilenetv2-7.onnx is saved to Azure Storage blob with the correct container name and blob name. Follow the steps mentioned for local, just replace local with Azure account.

Deploying through Visual Studio

Right click on ServerlessDNNFunction project and select Publish

Login with your Azure credentials in the next dialog. Select subscription and Resource group. I'll create a new Azure Function. Click on 'Create a new Azure Function' at the bottom.

Fill in the details of new Azure function

If above step is successful, skip next step. If its unsuccessful and gives below error, navigate to previous dialog, select 'Specific target' and select 'Windows'. Hopefully this time Azure function will be created.

Successful creation of Azure function

Once published, Azure Function app is ready to use.
Error In case below error is reported in Azure portal for the Azure function, go to Configuration -> General Settings -> Change 'Platform' to '64 Bit' and redeploy.

System.IO.FileLoadException : Could not load file or assembly 'ServerlessDNNFunction, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.

Validate using REST client - Postman

Run Project and hit the URL with an image of a dog from Postman as shown below

Input image

Predicted: 212 (English setter)

Conclusion

In this tutorial, we saw how we can leverage transfer learning to create deep neural networks and consume it within .Net ecosystem. Later we deployed the azure function app to Azure cloud and saw how easy is to create an image classification application. We saw lot of concepts such as Deep Neural networks, Serverless, Azure Functions and ML.Net being utilized and combined together to create an amazing application.

Resources

I am running an unofficial telegram group for ML.Net enthusiasts, please feel free to join it at https://t.me/joinchat/IifUJQ_PuYT757Turx-nLg

Please share your feedback/comments through below channels.

Contact

Twitter : @praveenraghuvan\
LinkedIn : https://in.linkedin.com/in/praveenraghuvanshi \
Github : https://github.com/praveenraghuvanshi \
dev.to : https://dev.to/praveenraghuvanshi

References

Composite Design Pattern: An agile story

Praveen Raghuvanshi — Sun, 19 Jul 2020 15:51:51 +0000

02-04: Composite Design Pattern

This article Composite Design Pattern: An agile story was originally posted on my blog.

Type: Structural
Scope: Object
Symptoms: Hierarchical Data

Intent

Composite design pattern is a structural design pattern which defines part-whole relationship between objects within a tree structure.
In a complex tree structure, it's a challenge to discriminate between a leaf and branch. We need to have type check before performing any operation on the tree structure and complexity grows if we have varied types of leaf and branches. This leads to higher maintenance.

A composite design pattern helps simplify above mentioned issue by introducing a common interface which needs to be implemented by leaf and branches thereby generalizing the solution to a problem.

Components

There are 3 main components in a composite pattern

1. Component: This is the abstraction/interface for all the elements in a tree. It includes both leaf and branches. The operations performed on all the elements is defined here and this is how it eliminates the need for a type checking. The client instantiates the component class and doesn't need to know what is going inside the class.

2. Leaf: It represents the leaf element in a tree and doesn't contain any child elements. It also implements all the operations defined in a component.

3. Composite: It is composed of multiple elements as child. It does implement all the operations defined in a component and mostly delegate the execution to child elements.

Implementation

Example

Domain: Agile-Scrum

Problem Statement: We need to find a way to calculate the total story points for a project

In a typical scrum project, a project contains Epics and User stories. A project may contain a user story directly also without an Epic.
We are going to calculate total story points(34) for below hierarchy/tree structure

Solutions

Solution 1: No Pattern

We have three classes(Project, Epic and UserStory). Project contains a list of Epic and User Story. Also, there are different methods for adding/removing each type(Epic/UserStory) such as AddEpic() and AddUserStory(). This makes the class fragile and for any new type introduced, we need to introduce corresponding Add/Remove methods.

 class NoComposite
    {
        public void Run()
        {
            /*
             * Project
             *      |
             *      |-----Epic-1
             *      |       |
             *      |       |----US-1(5)
             *      |       |
             *      |       |----US-2(8)
             *      |
             *      |-----Epic-2
             *      |       |
             *      |       |----US-3(3)
             *      |       |
             *      |       |----US-4(8)
             *      |       
             *      |
             *      |----US-5(2)
             *      |
             *      |----US-6(8)
             *
             * Total story points in a project: 34
             */

            // User stories
            var us1 = new UserStory(5);
            var us2 = new UserStory(8);
            var us3 = new UserStory(3);
            var us4 = new UserStory(8);
            var us5 = new UserStory(2);
            var us6 = new UserStory(8);

            // Epics
            var epic1 = new Epic();
            epic1.AddUserStory(us1);
            epic1.AddUserStory(us2);

            var epic2 = new Epic();
            epic2.AddUserStory(us3);
            epic2.AddUserStory(us4);

            // Project
            var project = new Project();
            project.AddEpic(epic1);
            project.AddEpic(epic2);
            project.AddUserStory(us5);
            project.AddUserStory(us6);

            // Calculate total sum, it should be 34
            var result = project.CalculateStoryPoints();
            Console.WriteLine($"Total number of story points in a Project is {result}");
        }

        class UserStory
        {
            private int _storyPoints;

            public UserStory(int storyPoints)
            {
                _storyPoints = storyPoints;
            }

            public int CalculateStoryPoints()
            {
                return _storyPoints;
            }
        }

        class Epic
        {
            private readonly List<UserStory> _userStories;
            private int _storyPoints;

            public Epic()
            {
                _userStories = new List<UserStory>();
            }

            public int CalculateStoryPoints()
            {
                foreach (var userStory in _userStories)
                {
                    _storyPoints += userStory.CalculateStoryPoints();
                }

                return _storyPoints;
            }

            public void AddUserStory(UserStory userStory)
            {
                _userStories.Add(userStory);
            }

            public void RemoveUserStory(UserStory userStory)
            {
                _userStories.Remove(userStory);
            }
        }

        class Project
        {
            private readonly List<UserStory> _userStories;
            private readonly List<Epic> _epics;
            private int _storyPoints;

            public Project()
            {
                _userStories = new List<UserStory>();
                _epics = new List<Epic>();
            }

            public int CalculateStoryPoints()
            {
                // Add story points from all the user stories
                foreach (var userStory in _userStories)
                {
                    _storyPoints += userStory.CalculateStoryPoints();
                }

                // Add story points from all the epics
                foreach (var epic in _epics)
                {
                    _storyPoints += epic.CalculateStoryPoints();
                }

                return _storyPoints;
            }

            public void AddUserStory(UserStory userStory)
            {
                _userStories.Add(userStory);
            }

            public void RemoveUserStory(UserStory userStory)
            {
                _userStories.Remove(userStory);
            }

            public void AddEpic(Epic epic)
            {
                _epics.Add(epic);
            }

            public void RemoveEpic(Epic epic)
            {
                _epics.Remove(epic);
            }
        }
    }

var noComposite = new NoComposite();
noComposite.Run();

Total number of story points in a Project is 34

Issues with above solution

Tight coupling between Project and Epic/UserStory
Add/Remove methods for each type(Epic/UserStory)
Open Closed Principle(OCP) violation. If a new type(Features) gets introduced, we need to modify Project class implementation and write Add/Remove methods separately.
For a new type, there will be addition of new type in CalculateStoryPoints() method.

Solution 2 : With Composite Pattern

As seen above, this is a general solution with no coupling. IScrumComponent is the interface to be implemented by all elements within a tree.
Add/Remove methods have been generalized as well. If a new type gets introduced, it just needs to be inherited from IScrumComponent and everything will work.

interface IScrumComponent
{
    int CalculateStoryPoints();
}

class WithComposite
{
    public void Run()
    {
        /*
         * Project
         *      |
         *      |-----Epic-1
         *      |       |
         *      |       |----US-1(5)
         *      |       |
         *      |       |----US-2(8)
         *      |
         *      |-----Epic-2
         *      |       |
         *      |       |----US-3(3)
         *      |       |
         *      |       |----US-4(8)
         *      |       
         *      |
         *      |----US-5(2)
         *      |
         *      |----US-6(8)
         *
         * Total story points in a project: 34
         */

        // User stories
        var us1 = new UserStory(5);
        var us2 = new UserStory(8);
        var us3 = new UserStory(3);
        var us4 = new UserStory(8);
        var us5 = new UserStory(2);
        var us6 = new UserStory(8);

        // Epics
        var epic1 = new Epic();
        epic1.AddComponent(us1);
        epic1.AddComponent(us2);

        var epic2 = new Epic();
        epic2.AddComponent(us3);
        epic2.AddComponent(us4);

        // Project
        var project = new Project();
        project.AddComponent(epic1);
        project.AddComponent(epic2);
        project.AddComponent(us5);
        project.AddComponent(us6);

        // Calculate total sum, it should be 34
        var result = project.CalculateStoryPoints();
        Console.WriteLine($"Total number of story points in a Project is {result}");
    }

    class UserStory : IScrumComponent
    {
        private int _storyPoints;

        public UserStory(int storyPoints)
        {
            _storyPoints = storyPoints;
        }

        public int CalculateStoryPoints()
        {
            return _storyPoints;
        }
    }

    class Epic : IScrumComponent
    {
        private readonly List<IScrumComponent> _components;
        private int _storyPoints;

        public Epic()
        {
            _components = new List<IScrumComponent>();
        }

        public int CalculateStoryPoints()
        {
            foreach (var component in _components)
            {
                _storyPoints += component.CalculateStoryPoints();
            }

            return _storyPoints;
        }

        public void AddComponent(IScrumComponent component)
        {
            _components.Add(component);
        }

        public void RemoveComponent(IScrumComponent component)
        {
            _components.Remove(component);
        }
    }

    class Project
    {
        private readonly List<IScrumComponent> _components;
        private int _storyPoints;

        public Project()
        {
            _components = new List<IScrumComponent>();
        }

        public int CalculateStoryPoints()
        {
            // Add story points from all the user stories
            foreach (var component in _components)
            {
                _storyPoints += component.CalculateStoryPoints();
            }

            return _storyPoints;
        }

        public void AddComponent(IScrumComponent component)
        {
            _components.Add(component);
        }

        public void RemoveComponent(IScrumComponent component)
        {
            _components.Remove(component);
        }
    }
}

var withComposite = new WithComposite();
withComposite.Run();

Total number of story points in a Project is 34

Comparison - Class Diagrams

Conclusion

Composite design pattern is very useful in simplifying the complexity involved in a hierarchial/tree structure. It allows adding new types into the system without changing the client code and this makes the system very extensible. As a result, the maintenance is reduced to a great extent.

Source: https://github.com/praveenraghuvanshi/design-patterns/tree/master/02-Structural/02-04-composite

Please share your feedback as it'll encourage me to write more.

Twitter: @praveenraghuvan

Github: praveenraghuvanshi

Blog: praveenraghuvanshi.github.io

Post not displayed on search

Praveen Raghuvanshi — Fri, 12 Jun 2020 19:04:02 +0000

Recently I published 3 posts on dev.to. While searching on some keywords present in the title, none of the my posts are listed. For E.g below are three posts titles

Now on searching with keyword such as 'COVID-19', none of the above are listed.

Is there anything I missed as its my first post. Not sure it has reached to broader audience.

Anything I could do improve the search results.

COVID-19 EDA and Prediction using .Net Dataframe and ML.Net (C#) - Prediction

Praveen Raghuvanshi — Sun, 07 Jun 2020 20:09:18 +0000

COVID-19

As per Wiki Coronavirus disease 2019 (COVID-19) is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The disease was first identified in 2019 in Wuhan, the capital of China's Hubei province, and has since spread globally, resulting in the ongoing 2019–20 coronavirus pandemic.
The virus had caused a pandemic across the globe and spreading/affecting most of the nations.
The purpose of notebook is to visualize the number of confirmed cases over time and predicting it for next 7 days using time series in ML.Net

Acknowledgement

Johns Hopkins CSSE for dataset
COVID-19 data visualization by Akshay Sb

Dataset

2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE - Time Series.

Introduction

This is Part-2 of our analysis on the COVID-19 dataset provided by Johns Hopkins CSSE. In Part-1, I did data analysis on the dataset and created some tables and plots for getting insights from it. In Part-2, I'll focus on applying machine learning for making a prediction using time-series API's provided by ML.Net framework. I'll be building a model from scratch on the number of confirmed cases and predicting for the next 7 days. Later on, I'll plot these numbers for better visualization.

ML.Net is a cross-platform framework from Microsoft for developing Machine learning models in the .Net ecosystem. It allows .Net developers to solve business problems using machine learning algorithms leveraging their preferred language such as C#/F#. It's highly scalable and used within Microsoft in many of its products such as Bing, PowerPoint, etc.

Disclaimer: This is an exercise to explore different features present in ML.Net. The actual and predicted numbers might vary due to several factors such as size and features in a dataset.

Summary

Below is the summary of steps we'll be performing

Define application level items
- Nuget packages
- Namespaces
- Constants
Utility Functions
- Formatters
Dataset and Transformations
- Actual from Johns Hopkins CSSE
- Transformed time_series_covid19_confirmed_global_transposed.csv
Data Classes
- ConfirmedData : Provides a map between columns in a dataset
- ConfirmedForecast : Holds predicted values
Data Analysis
- Visualize Data using DataFrame API
- Display Top 10 Rows - dataframe.Head(10)
- Display Last 10 Rows - dataframe.Tail(10)
- Display Dataset Statistics - dataframe.Description()
- Plot of TotalConfimed cases vs Date
Load Data - MLContext
ML Pipeline
Train Model
Prediction/Forecasting
Prediction Visualization
Prediction Analysis
Conclusion

Note : Graphs/Plots may not render in GitHub due to security reasons, however if you run this notebook locally/binder they will render.

#!about

1. Define Application wide Items

Nuget Packages

// ML.NET Nuget packages installation
#r "nuget:Microsoft.ML"
#r "nuget:Microsoft.ML.TimeSeries"
#r "nuget:Microsoft.Data.Analysis"

// Install XPlot package
#r "nuget:XPlot.Plotly"

Installed package Microsoft.ML version 1.5.0
Installed package Microsoft.Data.Analysis version 0.4.0
Installed package Microsoft.ML.TimeSeries version 1.5.0
Installed package XPlot.Plotly version 3.0.1

Namespaces

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.Data.Analysis;
using Microsoft.ML.Transforms.TimeSeries;
using Microsoft.AspNetCore.Html;
using XPlot.Plotly;

Constants

const string CONFIRMED_DATASET_FILE = "time_series_covid19_confirmed_global_transposed.csv";

// Forecast API
const int WINDOW_SIZE = 5;
const int SERIES_LENGTH = 10;
const int TRAIN_SIZE = 100;
const int HORIZON = 7;

// Dataset
const int DEFAULT_ROW_COUNT = 10;
const string TOTAL_CONFIRMED_COLUMN = "TotalConfirmed";
const string DATE_COLUMN = "Date";

2. Utility Functions

Formatters

By default the output of DataFrame is not proper and in order to display it as a table, we need to have a custom formatter implemented as shown in next cell.

Formatter<DataFrame>.Register((df, writer) =>
{
    var headers = new List<IHtmlContent>();
    headers.Add(th(i("index")));
    headers.AddRange(df.Columns.Select(c => (IHtmlContent) th(c.Name)));
    var rows = new List<List<IHtmlContent>>();
    var take = DEFAULT_ROW_COUNT;
    for (var i = 0; i < Math.Min(take, df.Rows.Count); i++)
    {
        var cells = new List<IHtmlContent>();
        cells.Add(td(i));
        foreach (var obj in df.Rows[i])
        {
            cells.Add(td(obj));
        }
        rows.Add(cells);
    }

    var t = table(
        thead(
            headers),
        tbody(
            rows.Select(
                r => tr(r))));

    writer.Write(t);
}, "text/html");

3. Dataset and Transformations

Download Dataset

Actual Dataset: Johns Hopkins CSSE
Transformed Dataset: time_series_covid19_confirmed_global_transposed.csv

I'll be using COVID-19 time series dataset from Johns Hopkins CSSE and will be performing predictions using time_series_covid19_confirmed_global.csv file.

The data present in these files have name of the countries as Rows and dates as columns which makes it difficult to map to our classes while loading data from csv. Also, it contains data per country wise. In order to keep things simple I'll work with global count of COVID-19 cases and not specific country.

I have done few transformations to the dataset as below and created transformed csv's

Sum cases from all the countries for a specific date
Just have two rows with Date and Total
Applied transformation to the csv for converting Rows into Columns and vice-versa. Refer for transformation.
Below transposed files have been saved in the current GitHub directory. There is no change in dataset. The files have data till 05-27-2020
- time_series_covid19_confirmed_global_transposed.csv : Columns - Date, TotalConfirmed

Before transformation

After transformation

4. Data Classes

Now, we need to create few data structures to map to columns within our dataset.

Confirmed cases

/// <summary>
/// Represent data for confirmed cases with a mapping to columns in a dataset
/// </summary>
public class ConfirmedData
{
    /// <summary>
    /// Date of confirmed case
    /// </summary>
    [LoadColumn(0)]
    public DateTime Date;

    /// <summary>
    /// Total no of confirmed cases on a particular date
    /// </summary>
    [LoadColumn(1)]
    public float TotalConfirmed;
}

/// <summary>
/// Prediction/Forecast for Confirmed cases
/// </summary>
internal class ConfirmedForecast
{
    /// <summary>
    /// No of predicted confirmed cases for multiple days
    /// </summary>
    public float[] Forecast { get; set; }
}

5. Data Analysis

For loading data from csv, first we need to create MLContext that acts as a starting point for creating a machine learning model in ML.Net. Few things to note

Set hasHeader as true as our dataset has header
Add separatorChar to ',' as its a csv

Visualize Data - DataFrame

var predictedDf = DataFrame.LoadCsv(CONFIRMED_DATASET_FILE);

predictedDf.Head(DEFAULT_ROW_COUNT)

index	Date	TotalConfirmed
0	1/22/2020	555
1	1/23/2020	654
2	1/24/2020	941
3	1/25/2020	1434
4	1/26/2020	2118
5	1/27/2020	2927
6	1/28/2020	5578
7	1/29/2020	6166
8	1/30/2020	8234
9	1/31/2020	9927

predictedDf.Tail(DEFAULT_ROW_COUNT)

index	Date	TotalConfirmed
0	4/28/2020	3097229
1	4/29/2020	3172287
2	4/30/2020	3256910
3	5/1/2020	3343777
4	5/2/2020	3427584
5	5/3/2020	3506729
6	5/4/2020	3583055
7	5/5/2020	3662691
8	5/6/2020	3755341
9	5/7/2020	3845718

predictedDf.Description()

index	Description	TotalConfirmed
0	Length (excluding null values)	107
1	Max	3845718
2	Min	555
3	Mean	923109.56

Number of Confirmed cases over Time

// Number of confirmed cases over time
var totalConfirmedDateColumn = predictedDf.Columns[DATE_COLUMN];
var totalConfirmedColumn = predictedDf.Columns[TOTAL_CONFIRMED_COLUMN];

var dates = new List<string>();
var totalConfirmedCases = new List<string>();
for (int index = 0; index < totalConfirmedDateColumn.Length; index++)
{
    dates.Add(totalConfirmedDateColumn[index].ToString());
    totalConfirmedCases.Add(totalConfirmedColumn[index].ToString());
}

var title = "Number of Confirmed Cases over Time";
var confirmedTimeGraph = new Graph.Scattergl()
    {
        x = dates.ToArray(),
        y = totalConfirmedCases.ToArray(),
        mode = "lines+markers"
    };



var chart = Chart.Plot(confirmedTimeGraph);
chart.WithTitle(title);
display(chart);

Analysis

Duration: 1/22/2020 through 5/27/2020
Total records: 127
Case on first day: 555
Case on last day: 5691790
No of confirmed cases was low in the beginning, there was first jump around 2/12/2020 and an exponential jump around 3/22/2020.
Cases have been increasing at an alarming rate in the past two months.

6. Load Data - MLContext

var context = new MLContext();

var data = context.Data.LoadFromTextFile<ConfirmedData>(CONFIRMED_DATASET_FILE, hasHeader: true, separatorChar: ',');

7. ML Pipeline

For creating ML Pipeline for a time-series analysis, we'll use Single Spectrum Analysis. ML.Net provides built in API for same, more details could be found at TimeSeriesCatalog.ForecastBySsa

var pipeline = context.Forecasting.ForecastBySsa(
                nameof(ConfirmedForecast.Forecast),
                nameof(ConfirmedData.TotalConfirmed),
                WINDOW_SIZE, 
                SERIES_LENGTH,
                TRAIN_SIZE,
                HORIZON);

8. Train Model

We are ready with our pipeline and ready to train the model

var model = pipeline.Fit(data);

9. Prediction/Forecasting - 7 days

Our model is trained and we need to do prediction for next 7(Horizon) days.
Time-series provides its own engine for making prediction which is similar to PredictionEngine present in ML.Net. Predicted values show an increasing trend which is in alignment with recent past values.

var forecastingEngine = model.CreateTimeSeriesEngine<ConfirmedData, ConfirmedForecast>(context);
var forecasts = forecastingEngine.Predict();
display(forecasts.Forecast.Select(x => (int) x))

index	value
0	3348756
1	3450496
2	3563966
3	3690067
4	3830294
5	3985414
6	4156340

10. Prediction Visualization

var lastDate = DateTime.Parse(dates.LastOrDefault());
var predictionStartDate = lastDate.AddDays(1);

for (int index = 0; index < HORIZON; index++)
{
    dates.Add(lastDate.AddDays(index + 1).ToShortDateString());
    totalConfirmedCases.Add(forecasts.Forecast[index].ToString());
}

var title = "Number of Confirmed Cases over Time";
var layout = new Layout.Layout();
layout.shapes = new List<Graph.Shape>
{
    new Graph.Shape
    {
        x0 = predictionStartDate.ToShortDateString(),
        x1 = predictionStartDate.ToShortDateString(),
        y0 = "0",
        y1 = "1",
        xref = 'x',
        yref = "paper",
        line = new Graph.Line() {color = "red", width = 2}
    }
};

var chart1 = Chart.Plot(
new [] 
    {
        new Graph.Scattergl()
        {
            x = dates.ToArray(),
            y = totalConfirmedCases.ToArray(),
            mode = "lines+markers"
        }
    },
    layout
);

chart1.WithTitle(title);
display(chart1);

11. Analysis

Comparing the plots before and after prediction, it seems our ML model has performed reasonably well. The red line represents the data on future date(5/8/2020). Beyond this, we predicted for 7 days. Looking at the plot, there is a sudden drop on 5/8/2020 which could be accounted due to insufficient data as we have only 127 records. However we see an increasing trend for next 7 days in alignment with previous confirmed cases. We can extend this model for predicting confirmed cases for any number of days by changing HORIZON constant value. This plot is helpful in analyzing the increased number of cases and allow authorities to take precautionary measures to keep the numbers low.

Conclusion

I hope you have enjoyed reading the notebook, and might have got some idea on the powerful framework ML.Net. ML.Net is a very fast emerging framework for .Net developers which abstracts lot of complexity present in the field of Data science and Machine Learning. The focus of Part-2 notebook is leverage ML.Net for making predictions using time-series API. The model generated can be saved as a zip file and used in different applications.

Notebook https://github.com/praveenraghuvanshi1512/covid-19

If you liked it, please like/comment at Comments. It'll encourage me to write more.

Contact

LinkedIn : https://in.linkedin.com/in/praveenraghuvanshi

Github : https://github.com/praveenraghuvanshi1512

Twitter : @praveenraghuvan

I am running an unofficial telegram group for ML.Net enthusiasts, please feel free to join it at https://t.me/joinchat/IifUJQ_PuYT757Turx-nLg

References

Be Safe

COVID-19 EDA and Prediction using .Net Dataframe and ML.Net (C#) - Exploratory Data Analysis

Praveen Raghuvanshi — Sun, 07 Jun 2020 20:08:53 +0000

COVID-19

As per Wiki Coronavirus disease 2019 (COVID-19) is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The disease was first identified in 2019 in Wuhan, the capital of China's Hubei province, and has since spread globally, resulting in the ongoing 2019–20 coronavirus pandemic.
The virus had caused a pandemic across the globe and spreading/affecting most of the nations.
The purpose of notebook is to visualize the trends of virus spread in various countries and explore features present in ML.Net such as DataFrame.

Acknowledgement

Johns Hopkins CSSE for dataset
COVID-19 data visualization by Akshay Sb

Dataset

2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE - Daily reports.

Introduction

DataFrame: DataFrame is a new type introduced in .Net. It is similar to DataFrame in Python which is used to manipulate data in notebooks. It's a collection of columns containing data similar to a table and very helpful in analyzing tabular data. It works flawlessly without creating types/classes mapped to columns in a table which we used to do with ML.Net. It has support for GroupBy, Sort, Filter which makes analysis very handy. It's a in-memory representation of structured data.

In this tutorial we'll cover below features

Load a CSV
Metadata
- Description
- Info
Display records
- Head
- Sample
Filtering
Grouping
Aggregate

For overview, please refer below links

Part-2 covers time series analysis and prediction using ML.Net

Summary

Below is the summary of steps we'll be performing

Define application level items
- Nuget packages
- Namespaces
- Constants
Utility Functions
- Formatters
Load Dataset
- Download Dataset from Johns Hopkins CSSE
- Load dataset in DataFrame
Analyze Data
- Date Range
- Display Dataset - display(dataframe)
- Display Top 5 Rows - dataframe.Head(5)
- Display Random 6 Rows - dataframe.Sample(6)
- Display Dataset Statistics - dataframe.Description()
- Display Dataset type information - dataframe.Info()
Data Cleaning
- Remove Invalid cases
Data Visualization
- Global
  - Confirmed Vs Deaths Vs Recovered
  - Top 5 Countries with Confirmed cases
  - Top 5 Countries with Death cases
  - Top 5 Countries with Recovered cases
- India
  - Confirmed Vs Deaths Vs Recovered

Note : Graphs/Plots may not render in GitHub due to security reasons, however if you run this notebook locally/binder they will render.

1. Define Application wide Items

Nuget Packages

// ML.NET Nuget packages installation
#r "nuget:Microsoft.ML"
#r "nuget:Microsoft.Data.Analysis"

// Install XPlot package
#r "nuget:XPlot.Plotly"

// CSV Helper Package for reading CSV
#r "nuget:CsvHelper"

Installed package XPlot.Plotly version 3.0.1
Installed package Microsoft.Data.Analysis version 0.4.0
Installed package CsvHelper version 15.0.5
Installed package Microsoft.ML version 1.5.0

Namespaces

using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.Data.Analysis;
using Microsoft.AspNetCore.Html;
using System.IO;
using System.Net.Http;
using System.Globalization;
using CsvHelper;
using CsvHelper.Configuration;
using XPlot.Plotly;

Constants

// Column Names
const string FIPS = "FIPS";
const string ADMIN = "Admin2";
const string STATE = "Province_State";
const string COUNTRY = "Country_Region";
const string LAST_UPDATE = "Last_Update";
const string LATITUDE = "Lat";
const string LONGITUDE = "Long_";
const string CONFIRMED = "Confirmed";
const string DEATHS = "Deaths";
const string RECOVERED = "Recovered";
const string ACTIVE = "Active";
const string COMBINED_KEY = "Combined_Key";

// File
const string DATASET_FILE = "05-27-2020";
const string FILE_EXTENSION = ".csv";
const string NEW_FILE_SUFFIX = "_new";
const char SEPARATOR = ',';
const char SEPARATOR_REPLACEMENT = '_';
const string DATASET_GITHUB_DIRECTORY = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/";

// DataFrame/Table
const int TOP_COUNT = 5;
const int DEFAULT_ROW_COUNT = 10;
const string VALUES = "Values";
const string INDIA = "India";

2. Utility Functions

Formatters

By default the output of DataFrame is not proper and in order to display it as a table, we need to have a custom formatter implemented as shown in next cell.

// Formats the table

Formatter<DataFrame>.Register((df, writer) =>
{
    var headers = new List<IHtmlContent>();
    headers.Add(th(i("index")));
    headers.AddRange(df.Columns.Select(c => (IHtmlContent) th(c.Name)));
    var rows = new List<List<IHtmlContent>>();
    var take = DEFAULT_ROW_COUNT;
    for (var i = 0; i < Math.Min(take, df.Rows.Count); i++)
    {
        var cells = new List<IHtmlContent>();
        cells.Add(td(i));
        foreach (var obj in df.Rows[i])
        {
            cells.Add(td(obj));
        }
        rows.Add(cells);
    }

    var t = table(
        thead(
            headers),
        tbody(
            rows.Select(
                r => tr(r))));

    writer.Write(t);
}, "text/html");

Copy dataset csv and replace Separator in cells

// Replace a characeter in a cell of csv with a defined separator
private void CreateCsvAndReplaceSeparatorInCells(string inputFile, string outputFile, char separator, char separatorReplacement)
{
    var culture = CultureInfo.InvariantCulture;
    using var reader = new StreamReader(inputFile);
    using var csvIn = new CsvReader(reader, new CsvConfiguration(culture));
    using var recordsIn = new CsvDataReader(csvIn);
    using var writer = new StreamWriter(outputFile);
    using var outCsv = new CsvWriter(writer, culture);

    // Write Header
    csvIn.ReadHeader();
    var headers = csvIn.Context.HeaderRecord;
    foreach (var header in headers)
    {
        outCsv.WriteField(header.Replace(separator, separatorReplacement));
    }
    outCsv.NextRecord();

    // Write rows
    while (recordsIn.Read())
    {
        var columns = recordsIn.FieldCount;
        for (var index = 0; index < columns; index++)
        {
            var cellValue = recordsIn.GetString(index);
            outCsv.WriteField(cellValue.Replace(separator, separatorReplacement));
        }
        outCsv.NextRecord();
    }
}

3. Load Dataset

Download Dataset from Johns Hopkins CSSE

We'll be using COVID-19 dataset from Johns Hopkins CSSE. The csse_covid_19_data directory has .csv file for each day and we'll be performing analysis on latest file present. Latest file present at the time of last modification of this notebook was 05-27-2020.csv. If you wish to use a different file, update DATASET_FILE constant in Constants cell above.

We'll download file to current directory.

// Download csv from github
var originalFileName = $"{DATASET_FILE}{FILE_EXTENSION}";
if (!File.Exists(originalFileName))
{
    var remoteFilePath = $"{DATASET_GITHUB_DIRECTORY}/{originalFileName}";
    display(remoteFilePath);
    var contents = new HttpClient()
        .GetStringAsync(remoteFilePath).Result;

    File.WriteAllText(originalFileName, contents);
}

https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports//05-27-2020.csv

Load dataset in DataFrame

Issue: We can load csv using LoadCsv(..) method of DataFrame. However, there is an issue of not allowing quotes and separator(comma in this case) in a cell value.
The dataset, we are using has both of them and LoadCsv fails for it.
As a workaround, we'll use CSVHelper to read the csv file and replace command separator with underscore, save the file and use it to load in DataFrame LoadCsv(..) method.

// Load and create a copy of dataset file
var newFileName = $"{DATASET_FILE}{NEW_FILE_SUFFIX}{FILE_EXTENSION}";
display(newFileName);
CreateCsvAndReplaceSeparatorInCells(originalFileName, newFileName, SEPARATOR, SEPARATOR_REPLACEMENT);

05-27-2020_new.csv

var covid19Dataframe = DataFrame.LoadCsv(newFileName);

4. Data Analysis

Data analysis is a critical activity in the field of Data science. It provides ways to uncover the hidden attributes of a dataset which can't be analyzed or predicted by simply looking at the data source. DataFrame makes the analysis simple by providing great API's such as GroupBy, Sort, Filter etc. Jupyter notebook is great tool for this kind of activity which maintains values of variables executed in a cell and providing it to other cells.

Finding the range in the records in Dataset

In DataFrame, Columns property allows access to values within a column by specifying column name. we'll use Last_Update column to get the date and sort it to get the start and end date

// Gets the data range

var dateRangeDataFrame = covid19Dataframe.Columns[LAST_UPDATE].ValueCounts();
var dataRange = dateRangeDataFrame.Columns[VALUES].Sort();
var lastElementIndex = dataRange.Length - 1;

var startDate = DateTime.Parse(dataRange[0].ToString()).ToShortDateString();
var endDate  = DateTime.Parse(dataRange[lastElementIndex].ToString()).ToShortDateString(); // Last Element

display(h4($"The data is between {startDate} and {endDate}"));

The data is between 5/28/2020 and 5/28/2020

Display 10 records

Here we have 12 columns which includes Country, State, Confirmed, Deaths, Recovered and Active cases

display(covid19Dataframe)

index	FIPS	Admin2	Province_State	Country_Region	Last_Update	Lat	Long_	Confirmed	Deaths	Active	Combined_Key
0	45001	Abbeville	South Carolina	US	2020-05-28 02:32:31	34.223335	-82.46171	35	0	35	Abbeville_ South Carolina_ US
1	22001	Acadia	Louisiana	US	2020-05-28 02:32:31	30.295065	-92.4142	397	22	375	Acadia_ Louisiana_ US
2	51001	Accomack	Virginia	US	2020-05-28 02:32:31	37.76707	-75.63235	780	12	768	Accomack_ Virginia_ US
3	16001	Ada	Idaho	US	2020-05-28 02:32:31	43.452656	-116.241554	798	22	776	Ada_ Idaho_ US
4	19001	Adair	Iowa	US	2020-05-28 02:32:31	41.330757	-94.47106	7	0	7	Adair_ Iowa_ US
5	21001	Adair	Kentucky	US	2020-05-28 02:32:31	37.1046	-85.281296	96	19	77	Adair_ Kentucky_ US
6	29001	Adair	Missouri	US	2020-05-28 02:32:31	40.190586	-92.600784	46	0	46	Adair_ Missouri_ US
7	40001	Adair	Oklahoma	US	2020-05-28 02:32:31	35.88494	-94.65859	82	3	79	Adair_ Oklahoma_ US
8	8001	Adams	Colorado	US	2020-05-28 02:32:31	39.87432	-104.33626	3006	118	2888	Adams_ Colorado_ US
9	16003	Adams	Idaho	US	2020-05-28 02:32:31	44.893337	-116.45452	3	0	3	Adams_ Idaho_ US

Display Top 5 records

covid19Dataframe.Head(5)

index	FIPS	Admin2	Province_State	Country_Region	Last_Update	Lat	Long_	Confirmed	Deaths	Active	Combined_Key
0	45001	Abbeville	South Carolina	US	2020-05-28 02:32:31	34.223335	-82.46171	35	0	35	Abbeville_ South Carolina_ US
1	22001	Acadia	Louisiana	US	2020-05-28 02:32:31	30.295065	-92.4142	397	22	375	Acadia_ Louisiana_ US
2	51001	Accomack	Virginia	US	2020-05-28 02:32:31	37.76707	-75.63235	780	12	768	Accomack_ Virginia_ US
3	16001	Ada	Idaho	US	2020-05-28 02:32:31	43.452656	-116.241554	798	22	776	Ada_ Idaho_ US
4	19001	Adair	Iowa	US	2020-05-28 02:32:31	41.330757	-94.47106	7	0	7	Adair_ Iowa_ US

Display Random 6 records

covid19Dataframe.Sample(6)

index	FIPS	Admin2	Province_State	Country_Region	Last_Update	Lat	Long_	Confirmed	Deaths	Active	Combined_Key
0	18025	Crawford	Indiana	US	2020-05-28 02:32:31	38.288143	-86.44519	23	0	23	Crawford_ Indiana_ US
1	51065	Fluvanna	Virginia	US	2020-05-28 02:32:31	37.84158	-78.27715	87	6	81	Fluvanna_ Virginia_ US
2	13241	Rabun	Georgia	US	2020-05-28 02:32:31	34.883896	-83.403046	17	1	16	Rabun_ Georgia_ US
3	12003	Baker	Florida	US	2020-05-28 02:32:31	30.3306	-82.284676	29	3	26	Baker_ Florida_ US
4	21133	Letcher	Kentucky	US	2020-05-28 02:32:31	37.123066	-82.85346	4	0	4	Letcher_ Kentucky_ US
5	12077	Liberty	Florida	US	2020-05-28 02:32:31	30.23766	-84.88293	209	0	209	Liberty_ Florida_ US

Display Dataset Statistics such as Total, Max, Min, Mean of items in a column

covid19Dataframe.Description()

index	Description	FIPS	Lat	Long_	Confirmed	Deaths	Recovered	Active
0	Length (excluding null values)	3009	3346	3346	3414	3414	3414	3414
1	Max	99999	71.7069	178.065	370680	37460	391508	229780
2	Min	0	-52.368	-164.03539	0	0	0	-364117
3	Mean	27622.38	35.3851	-78.927086	1667.1909	104.16784	688.3679	882.6784

Display Dataset type information for each column

covid19Dataframe.Info()

index	Info	FIPS	Admin2	Province_State	Country_Region	Last_Update	Lat	Long_	Confirmed	Deaths	Recovered	Active	Combined_Key
0	DataType	System.Single	System.String	System.String	System.String	System.String	System.Single	System.Single	System.Single	System.Single	System.Single	System.Single	System.String
1	Length (excluding null values)	3009	3414	3414	3414	3414	3346	3346	3414	3414	3414	3414	3414

5. Data Cleaning

Data Cleaning is another important activity in which remove the irrelevant data present in our dataset. This irrelevant data can be due missing values, invalid values or an outlier. The columns with less significance is removed for better analysis and prediction of our data. In order to keep this notebook simple, we'll use one of the techniques to remove invalid data. In this we are going to remove invalid Active cases such as the ones having negative values. The other techniques we can apply on data could be DropNull to remove rows with null values, FillNull to fill null values with other such as mean, average. We can transform DataFrame and remove some of the unnecessary columns.

Remove invalid Active cases

covid19Dataframe.Description()

index	Description	FIPS	Lat	Long_	Confirmed	Deaths	Recovered	Active
0	Length (excluding null values)	3009	3346	3346	3414	3414	3414	3414
1	Max	99999	71.7069	178.065	370680	37460	391508	229780
2	Min	0	-52.368	-164.03539	0	0	0	-364117
3	Mean	27622.38	35.3851	-78.927086	1667.1909	104.16784	688.3679	882.6784

From the above description table, we could see negative value for Active cases which seems to be incorrect as number of active cases is cases is calculated by the below formula

Active = Confirmed - Deaths - Recovered

In order to check for invalid active cases, we'll use DataFrame Filter to retrieve active column values whose value is less than 0.0

// Filter : Gets active records with negative calues

PrimitiveDataFrameColumn<bool> invalidActiveFilter = covid19Dataframe.Columns[ACTIVE].ElementwiseLessThan(0.0);
var invalidActiveDataFrame = covid19Dataframe.Filter(invalidActiveFilter);
display(invalidActiveDataFrame)

index	FIPS	Admin2	Province_State	Country_Region	Last_Update	Lat	Long_	Confirmed	Deaths	Recovered	Active	Combined_Key
0	90004	Unassigned	Arizona	US	2020-05-28 02:32:31	<null>	<null>	0	2	0	-2	Unassigned_ Arizona_ US
1	90018	Unassigned	Indiana	US	2020-05-28 02:32:31	<null>	<null>	0	159	0	-159	Unassigned_ Indiana_ US
2	90022	Unassigned	Louisiana	US	2020-05-28 02:32:31	<null>	<null>	84	105	0	-21	Unassigned_ Louisiana_ US
3	90024	Unassigned	Maryland	US	2020-05-28 02:32:31	<null>	<null>	0	67	0	-67	Unassigned_ Maryland_ US
4	90032	Unassigned	Nevada	US	2020-05-28 02:32:31	<null>	<null>	0	6	0	-6	Unassigned_ Nevada_ US
5	90033	Unassigned	New Hampshire	US	2020-05-28 02:32:31	<null>	<null>	10	51	0	-41	Unassigned_ New Hampshire_ US
6	90038	Unassigned	North Dakota	US	2020-05-28 02:32:31	<null>	<null>	0	8	0	-8	Unassigned_ North Dakota_ US
7	90056	Unassigned	Wyoming	US	2020-05-28 02:32:31	<null>	<null>	0	13	0	-13	Unassigned_ Wyoming_ US
8	<null>		C. Valenciana	Spain	2020-05-28 02:32:31	39.484	-0.7533	11089	1332	9970	-213	C. Valenciana_ Spain
9	<null>		Cantabria	Spain	2020-05-28 02:32:31	43.1828	-3.9878	2283	202	2287	-206	Cantabria_ Spain

If we take any record(index 5) and apply above formula to calculate

Active(-13) = Confirmed(10) - Deaths(51) - Recovered(0)

We could see invalid active cases.

In order to remove it, we'll apply a Filter to DataFrame to get active values greater than or equal to 0.0.

// Remove invalid active cases by applying filter

PrimitiveDataFrameColumn<bool> activeFilter = covid19Dataframe.Columns[ACTIVE].ElementwiseGreaterThanOrEqual(0.0);
covid19Dataframe = covid19Dataframe.Filter(activeFilter);
display(covid19Dataframe.Description());

index	Description	FIPS	Lat	Long_	Confirmed	Deaths	Recovered	Active
0	Length (excluding null values)	3001	3338	3338	3395	3395	3395	3395
1	Max	99999	71.7069	178.065	370680	37460	142208	229780
2	Min	0	-52.368	-164.03539	0	0	0	0
3	Mean	27564.82	35.48979	-79.35968	1664.4899	103.38468	505.34668	1055.7584

As seen above, negative active cases have been removed

6. Visualization

Visualization of data helps business owners make better decisions. The DataFrame maintains data in a tabular format. In order to prepare data for different plots, I have used DataFrame features such as Sum, GroupBy, OrderBy, OrderByDescending etc.

For visualization, I have used open source library called as XPlot.Plotly. Different plots have been used such as Bar, Pie and Line/Scatter Graph.

Global

Collect Data

//  Gets the collection of confirmed, deaths and recovered

var confirmed = covid19Dataframe.Columns[CONFIRMED];
var deaths = covid19Dataframe.Columns[DEATHS];
var recovered = covid19Dataframe.Columns[RECOVERED];

// Gets the sum of collection by using Sum method of DataFrame
var totalConfirmed = Convert.ToDouble(confirmed.Sum());
var totalDeaths = Convert.ToDouble(deaths.Sum());
var totaRecovered = Convert.ToDouble(recovered.Sum());

Confirmed Vs Deaths Vs Recovered cases

display(Chart.Plot(
    new Graph.Pie()
    {
        values = new double[]{totalConfirmed, totalDeaths, totaRecovered},
        labels = new string[] {CONFIRMED, DEATHS, RECOVERED}
    }
));

Top 5 Countries with Confirmed cases

In order to get top 5 countries data, I have used DataFrame's GroupBy, Sum, OrderByDescending methods

// The data for top 5 countries is not present in the csv file.
// In order to get that, first DataFrame's GROUPBY is used aginst the country.
// Then it was aggregated using SUM on Confirmed column.
// In the last, ORDERBYDESCENDING is used to get the top five countries.

var countryConfirmedGroup = covid19Dataframe.GroupBy(COUNTRY).Sum(CONFIRMED).OrderByDescending(CONFIRMED);
var topCountriesColumn = countryConfirmedGroup.Columns[COUNTRY];
var topConfirmedCasesByCountry = countryConfirmedGroup.Columns[CONFIRMED];

HashSet<string> countries = new HashSet<string>(TOP_COUNT);
HashSet<long> confirmedCases = new HashSet<long>(TOP_COUNT);
for(int index = 0; index < TOP_COUNT; index++)
{
    countries.Add(topCountriesColumn[index].ToString());
    confirmedCases.Add(Convert.ToInt64(topConfirmedCasesByCountry[index]));
}

var title = "Top 5 Countries : Confirmed";
var series1 = new Graph.Bar{
        x = countries.ToArray(),
        y = confirmedCases.ToArray()
    };

var chart = Chart.Plot(new []{series1});
chart.WithTitle(title);
display(chart);

Top 5 Countries with Deaths

// Get the data
var countryDeathsGroup = covid19Dataframe.GroupBy(COUNTRY).Sum(DEATHS).OrderByDescending(DEATHS);
var topCountriesColumn = countryDeathsGroup.Columns[COUNTRY];
var topDeathCasesByCountry = countryDeathsGroup.Columns[DEATHS];

HashSet<string> countries = new HashSet<string>(TOP_COUNT);
HashSet<long> deathCases = new HashSet<long>(TOP_COUNT);
for(int index = 0; index < TOP_COUNT; index++)
{
    countries.Add(topCountriesColumn[index].ToString());
    deathCases.Add(Convert.ToInt64(topDeathCasesByCountry[index]));
}

var title = "Top 5 Countries : Deaths";
var series1 = new Graph.Bar{
        x = countries.ToArray(),
        y = deathCases.ToArray()
    };

var chart = Chart.Plot(new []{series1});
chart.WithTitle(title);
display(chart);

Top 5 Countries with Recovered cases

// Get the data
var countryRecoveredGroup = covid19Dataframe.GroupBy(COUNTRY).Sum(RECOVERED).OrderByDescending(RECOVERED);
var topCountriesColumn = countryRecoveredGroup.Columns[COUNTRY];
var topRecoveredCasesByCountry = countryRecoveredGroup.Columns[RECOVERED];

HashSet<string> countries = new HashSet<string>(TOP_COUNT);
HashSet<long> recoveredCases = new HashSet<long>(TOP_COUNT);
for(int index = 0; index < TOP_COUNT; index++)
{
    countries.Add(topCountriesColumn[index].ToString());
    recoveredCases.Add(Convert.ToInt64(topRecoveredCasesByCountry[index]));
}

var title = "Top 5 Countries : Recovered";
var series1 = new Graph.Bar{
        x = countries.ToArray(),
        y = recoveredCases.ToArray()
    };

var chart = Chart.Plot(new []{series1});
chart.WithTitle(title);
display(chart);

World Confirmed Cases Map

var world = countryConfirmedGroup;
countries.Clear();
List<string> worldConfirmedCases = new List<string>();
for (int index = 0; index < world.Columns[COUNTRY].Length; index++)
{
    countries.Add(world.Columns[COUNTRY][index].ToString());
    worldConfirmedCases.Add(world.Columns[CONFIRMED][index].ToString());
}

var locations = countryConfirmedGroup.Columns[COUNTRY];

var worldConfirmedMap = Chart.Plot(
    new Graph.Choropleth()
    {
        locations = countries.ToArray(),
        z = worldConfirmedCases.ToArray(),
        locationmode = "country names",
        text = countryConfirmedGroup.Columns[COUNTRY],
        colorscale = "active",
        hoverinfo = COUNTRY,
        autocolorscale = true,

    });
worldConfirmedMap.WithTitle("World Confirmed Cases");
display(worldConfirmedMap);

India

Confirmed Vs Deaths Vs Recovered cases

Filtering on Country column with INDIA as value

// Filtering on Country column with INDIA as value

PrimitiveDataFrameColumn<bool> indiaFilter = covid19Dataframe.Columns[COUNTRY].ElementwiseEquals(INDIA);
var indiaDataFrame = covid19Dataframe.Filter(indiaFilter);

var indiaConfirmed = indiaDataFrame.Columns[CONFIRMED];
var indiaDeaths = indiaDataFrame.Columns[DEATHS];
var indiaRecovered = indiaDataFrame.Columns[RECOVERED];

var indiaTotalConfirmed = Convert.ToDouble(indiaConfirmed.Sum());
var indiaTotalDeaths = Convert.ToDouble(indiaDeaths.Sum());
var indiaTotaRecovered = Convert.ToDouble(indiaRecovered.Sum());

display(Chart.Plot(
    new Graph.Pie()
    {
        values = new double[]{indiaTotalConfirmed, indiaTotalDeaths, indiaTotaRecovered},
        labels = new string[] {CONFIRMED, DEATHS, RECOVERED}
    }
));

Conclusion

I hope you have enjoyed reading the notebook, and might have got some idea on the powerful features of DataFrame in .Net. Data science capabilities are emerging fast in the .Net ecosystem which abstracts lot of complexity present in the field. The focus of this notebook is data analysis and there is nothing present from a Machine Learning perspective such as making a prediction. In Part-2, I have done time series analysis and predictions using ML.Net

Notebook: https://github.com/praveenraghuvanshi1512/covid-19

If you liked it, please like/comment at Comments. It'll encourage me to write more.

Contact

LinkedIn : https://in.linkedin.com/in/praveenraghuvanshi

Github : https://github.com/praveenraghuvanshi1512

Twitter : @praveenraghuvan

I am running an unofficial telegram group for ML.Net enthusiasts, please feel free to join it at https://t.me/joinchat/IifUJQ_PuYT757Turx-nLg

References

Be Safe

COVID-19 EDA and Prediction using .Net Dataframe and ML.Net (C#) - Introduction

Praveen Raghuvanshi — Sun, 07 Jun 2020 20:07:57 +0000

Last week, I had the privilege of speaking at an awesome event Virtual ML.Net. The event was totally virtual and organized and participated by ML.Net enthusiasts.

ML.Net is a cross-platform framework from Microsoft for developing Machine learning models in the .Net ecosystem. It allows .Net developers to solve business problems using machine learning algorithms leveraging their preferred language such as C#/F#. It's highly scalable and used within Microsoft in many of its products such as Bing, Powerpoint, etc.

Key highlights of the event

Awesome 2-day sessions covering breadth and depth of topics around ML.Net
1-day Hands-on Workshop with multiple labs
Global Speakers covering 5 continents and 6 countries
Mix of technical sessions, labs and case studies
Sessions
- AI Ethics: Theory to adoption
- Machine learning simplified for developers with ML.Net
- Q & A with a Microsoft Data Scientist
- Transfer learning for Deep Learning: From Custom Vision to Tensorflow & ML.Net
- "You are what you eat", or how to fight data with Jupyter Notebooks for .Net
- Decision Analysis with ML.Net and Blazor
- Even smarter chat bot with ML.Net
- COVID-19 Exploratory Data Analysis using .Net Dataframe API (C#)
- Building Serverless Machine learning API's with Azure Functions, ML.Net and CosmosDB

Introduction

My presentation was on COVID-19 Exploratory Data Analysis using .Net Dataframe API (C#). It was divided into two parts as given below.

Part-1: COVID-19 Data Analysis using .Net DataFrame API
Part-2: COVID-19 Prediction using ML.Net framework

In this section, I'll mainly cover few things covered as part of my presentation such as COVID-19, DataFrame API(EDA), ML.Net(Prediction). It'll be a brief introduction to the topics, for working solution, please refer Part-1 and Part-2. Part-1 covers exploratory data analysis on COVID dataset for a particular day. Part-2 covers data analysis and prediction/forecasting for next 7 days using time-series COVID dataset. We'll leverage recently introduced support for C# in Jupyter Notebook

Acknowledgement

Dataset - Johns Hopkins CSSE
COVID-19 data visualization using Python - Akshay Sb
Organizers - Bri Achtman, Luis Quintanilla, Alexander Slotte and Jon Wood
Sponsors - Excella and Wintellect

COVID-19

As per Wiki Coronavirus disease 2019 (COVID-19) is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The disease was first identified in 2019 in Wuhan, the capital of China's Hubei province, and has since spread globally, resulting in the ongoing 2019–20 coronavirus pandemic. The virus had caused a pandemic across the globe and spreading/affecting most of the nations.

Lets analyze some statistics

Image source:https://covid19.who.int/

The above image just shows numbers for a particular date and seems to be a static data. It doesn't give any information that could help in making some decisions to bring these numbers down. It would be good, if we can get historical data or a plot to understand the trend.

Plot for number of confirmed cases and deaths as on May 28, 2020

Image source:https://covid19.who.int/

The above plot is better compared to only numbers. Through this, we could see the trend of cases over a period of 5 months. The number of cases started in the last week of January and remained low till mid-march. However, there is huge increase in the number of cases post that and it continued to increase which is quite alarming.

Similarly, different plots as shown below gives information in a way that can be analyzed and interpreted for making better decisions and take precautionary actions.

Image Source: https://slides.ourworldindata.org/2020_pandemic/2020_pandemic#/title-slide

Exploratory Data Analysis(EDA)

As per Wiki,

"In statistics, exploratory data analysis (EDA) is an approach to analyzing data sets to summarize their main characteristics, often with visual methods.

A statistical model can be used or not, but primarily EDA is for seeing what the data can tell us beyond the formal modeling or hypothesis testing task."

Its mainly analysis of data which on its own doesn't convey much about it and acts dumb. The analysis is presented in different form through various visualizations which include tables, plots such as bar, pie, scatter etc.

Let me give an analogy here. Consider you just moved to new place and looking to rent a house. In order to buy a house, you'll look for attributes such as number of bedrooms, type(Apartment or villa), locality and crime rate. If someone provides this information in a static manner(specific date), you won't be able to make a decision. You would be interested in history to make a decision. Even that information won't help much, better if its presented in a visual way of table/plots. This will help in making decision quick and relevant.

We'll be using .Net DataFrame API to perform EDA

Time Series Analysis

Number of deaths over time

Its the analysis of data organized across units of time. It helps understand past trends and plan for future. There are different types of time series analysis that can be performed such as Trends, Seasonality, Irregularity and Cyclic.

I have applied this to our COVID-19 dataset and plot a chart on the number of confirmed cases over time. I also did a prediction using ML.Net for the number of cases over next 7 days.

DataFrame

A DataFrame is a two-dimensional data structure holding information in a tabular way similar to spreadsheet. It has rows and columns, with columns having data of same type. It's mainly used for statistical analysis on the dataset and a great tool for performing exploratory data analysis by data scientists. Its present in many languages such as Python and R. For e.g, on the left side in below image, the data is represented as in spreadsheet. The same is represented on right when its loaded by Python API. The API's present in a language allows performing operations similar to spreadsheet such as filtering, grouping, sorting and many more.

.Net DataFrame

DataFrame has been a great tool of Data scientists for performing exploratory data analysis. There was nothing present in Microsoft ecosystem till December'19 when it was introduced for the first time in .Net ecosystem and allowed .Net developers to perform EDA. I have used to perform EDA on COVID-19 dataset and its present in Part-2. Some of the attributes of DataFrame are given as below

A collection of Columns
In-memory representation of structured data
Similar to Python DataFrame
Features
- Load dataset from Csv
- Row/Column Selection
- Join/Merge DataFrame
- GroupBy
- Handling Null values

While working on .Net DataFrame, I came across a similar datatype(IDataView) present in .Net ecosystem. They might look similar but have somewhat different purpose. The below table helps in differentiating them.

DataFrame	IDataView
Part of Microsoft.Data Namespace	Part of Microsoft.ML Namespace
Analogous to List	Analogous to IList
Useful for .Net developers interested in Data Science(Exploration)	Useful for .Net developers interested in Machine Learning (Prediction)
Schemaless	Schema bound
Eagerly loaded	Lazy loaded
In-memory	Stream based with forward cursor only support
Allow random access	No random access
Easy to debug	Hard to debug
Supports only CSV	Supports CSV, IEnumerable, Relational DB
Follows Apache Arrow Format	Follows Schema

ML.Net

I gave a little introduction on ML.Net in the beginning. In Part-2, it was mainly used for making prediction using time-series methodology(Single Spectrum Analysis). MLContext is used to load data and create an ML pipeline. TimeSeriesCatalog.ForecastBySsa API from ML.Net is used to predict the total number of confirmed cases in next 7 days.

Single Spectrum Analysis

SSA is a powerful framework for decomposing the time-series into trend, seasonality and noise components as well as forecasting the future values of the time-series. In principle, SSA performs spectral analysis on the input time-series where each component in the spectrum corresponds to a trend, seasonal or noise component in the time-series. For details of the Singular Spectrum Analysis (SSA), refer to this document.

Visualization

Visualization is an integral part of EDA. It allows multiple representation of data present in raw format. I have used table and plots (bar, pie, scatter) for visualization. Some of them is shown below

Confirmed Vs Deaths Vs Recovered

Prediction : Before and After

Conclusion

In this post, I have just touched on the things covered as part of my presentation. Please move on to Part-1 and thereafter Part-2 for more detailed analysis and explanation. I would recommend to have a look at other presentation of Virtual ML.Net conference and I am sure you would love them as I did. This was the first edition of Virtual ML.Net and I wish to have many more of these.

I wish a bigger and bolder next edition of Virtual ML.Net.

Code: https://github.com/praveenraghuvanshi1512/covid-19

If you liked it, please like/comment at Comments. It'll encourage me to write more.