DEV Community: Sugumar Prabhakar

S3 File Operations using Blazor WASM

Sugumar Prabhakar — Tue, 21 Mar 2023 02:11:45 +0000

In the previous chapter, we created three APIs that will list all the files available in the S3 bucket, download them, and upload a file to the S3 bucket.

In this chapter, we are going to create the Blazor web assembly front-end application that will provide functionalities to the users to see all the files from the bucket, download them and upload a new file if wanted. We are going to use the APIs we created in the previous chapter as the intermediate service that handles the user request.

Find the complete source code in my GitHub repository below.

https://github.com/sprabha1990/S3FileOperations.Blazor.NET7

We will use the empty Blazor WASM project that was already in the previous chapter.

Let's open the "S3FileOperations.Blazor" project inside the "S3FileOperations.NET7.sln" in VS2022. As a first step, we are going to incorporate bootstrap v5 into the project.

Add the following code inside the "index.html" available inside "wwwroot" folder.

<head>
    ...

    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous">
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.3.0/font/bootstrap-icons.css">

    ...
</head>
<body>
    ...
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" crossorigin="anonymous"></script>
</body>

If you have close look at the <head> tag, I'm adding the script to use bootstrap icons. If you want to use alternative icons, you can add/replace their script here.

Now, we are going to add an entity class named Files.cs. This holds the public properties of the file information that will be shown in the UI.
Let's add the below properties into the Files class.

public class Files
    {
        public string ETag { get; set; } = string.Empty;
        public string Key { get; set; } = string.Empty;
        public string BucketName { get; set; } = string.Empty;
        public long Size { get; set; } = 0;
    }

Let's add the UI HTML code in the Index.razor page available inside the Pages folder.

Workflow is that,

We are going to add an HTML table with 4 columns that will show the "eTag, fileName, bucketName, and size of the files retrieved from the S3.
We are going to add a download button adjacent to each row of the table to help users download the file easily.
We are going to add a File upload option for the user so that the user can upload any file.

Add the below code in the Index.razor file.

@page "/"
@using S3FileOperations.Blazor.Entity;
@inject HttpClient _httpClient;
@using Microsoft.AspNetCore.Components.Forms;
@using System.Net.Http.Headers;

<table class="table table-striped mt-5">
     <thead class="table-dark">
     <tr>
          <th scope="col">ETAG</th>
          <th scope="col">FileName</th>
          <th scope="col">BucketName</th>
          <th scope="col">Size (In Bytes)</th>
          <th scope="col"></th>
    </tr>
  </thead>
    <tbody class="table-light">
        @foreach(var file in FilesInS3)
        {
            <tr>
                <th scope="col">@file.ETag</th>
                <th scope="col">@file.Key</th>
                <th scope="col">@file.BucketName</th>
                <th scope="col">@file.Size</th>
                <th scope="col"><a href="**https://localhost:7097**/
api/S3/download?Key=@file.Key" class="link-success">Download</a><i class="bi bi-download ms-3 d-inline-flex align-Items-center text-danger"></i></th>
            </tr>
        }
    </tbody>
</table>

<div class="card d-inline-flex flex-row bd-highlight ms-2 mt-2 w-50 p-3">
    <div class="card-body p-2 bd-highlight">
        <InputFile OnChange="(e) => _currentFile = e.File" class="btn-light mb-2 w-100" />
        @if (_currentFile != null)
        {
            <div class="d-grid gap-3 ">
                <div class="p-2 bg-light border mt-1"><p>Name: @_currentFile.Name</p></div>
                <div class="p-2 bg-light border">Size in bytes: @_currentFile.Size</div>
                <div class="p-2 bg-light border">Last modified date: @_currentFile.LastModified.ToString()</div>
                <div class="p-2 bg-light border">Content type (not always supplied by the browser): @_currentFile.ContentType</div>
            </div>
        }
        <button type="button" class="btn btn-primary mt-3" @onclick="UploadFileAsync">Upload File <i class="bi bi-upload ms-2 d-inline-flex"></i></button>
    </div>
</div>

@code {
    public List<Files> FilesInS3 { get; set; } = new();
    public IBrowserFile? _currentFile = null;

    protected override async Task OnInitializedAsync()
    {
        FilesInS3 = await _httpClient.GetFromJsonAsync<List<Files>>("**https://localhost:7097**/api/S3/") ?? new();
    }

    public async Task UploadFileAsync()
    {
        if (_currentFile == null)
            return;

        var content = new MultipartFormDataContent();
        var fileContent = new StreamContent(_currentFile.OpenReadStream(1024000));

        content.Add(
            content: fileContent,
            name: "\"file\"",
            fileName: _currentFile.Name);

        var response = await _httpClient.PostAsync("**https://localhost:7097**/api/S3/upload", content);
        response.EnsureSuccessStatusCode();
        FilesInS3 = await _httpClient.GetFromJsonAsync<List<Files>>("https://localhost:7097/api/S3/") ?? new();
    }
}

If you have a look at the @code part above,

we are having a list of Files class instances that will be assigned at the page initialization life cycle. The value of the list of files is retrieved from the S3 via the GET API we created in the last chapter. If you see, I'm using the base address "https://localhost:7097" in the code but it will be different on your pc. You have to change it accordingly.

Next, we have the UploadFileAsync() button that will upload the file user has chosen using the POST API we created in the last chapter. We are passing the file as a Stream to that API body.

1024000 is nothing but the maximum file size I can upload. For this demo, I'm setting up 1MB as the maximum file size I can upload. But it will vary based on the requirement.

_currentFile is the object that holds information about the current file chosen by the user in the UI. This is set from the HTML Code

<InputFile **OnChange="(e) => _currentFile = e.File"** class="btn-light mb-2 w-100" />

That's it. To test the application, we need our API running first.
After running both the APIs and the Blazor app, you should see the Blazor app like the one below,

To upload a new file, the user will choose a file by pressing the "Choose File" button and then press the "UploadFile" button.

Please add your questions/suggestion in the comments.
Thanks!

S3 File Operations using .NET 7 WebAPI

Sugumar Prabhakar — Tue, 21 Mar 2023 02:11:40 +0000

In this chapter, we will create an Asp Net Core Web API on top of .NET 7 that contain 3 APIs. One will upload the file into AWS, another for downloading the file from AWS and another to list the available files.

Find the complete source code in my GitHub repository below.

https://github.com/sprabha1990/S3FileOperations.Blazor.NET7

Let's jump into creating a project via Dotnet CLI.

Assumption: .NET 7 SDK and runtime installed.

Running the below commands on the command prompt will create a visual studio solution file, an ASP Net Core Web API project, and an empty Blazor WASM project.



dotnet new sln --name S3FileOperations.NET7 --output S3FileOperations.Blazor.Net7
dotnet new webapi --name S3FileOperations.WebApi --output S3FileOperations.Blazor.Net7/Api
dotnet new blazorwasm-empty --name S3FileOperations.Blazor --output S3FileOperations.Blazor.Net7/WebApp

Now running the below commands will add the project files to the solution file.



dotnet sln S3FileOperations.Blazor.Net7\S3FileOperations.NET7.sln add S3FileOperations.Blazor.Net7/Api
dotnet sln S3FileOperations.Blazor.Net7\S3FileOperations.NET7.sln add S3FileOperations.Blazor.Net7/WebApp

Let's open the "S3FileOperations.NET7.sln" file in the VS2022. You'll find two projects in it. In this chapter, We are going to work on web APIs.

First up, let’s modify the appsettings.json of the Web API project. Make sure that your appsettings.json looks like the one below. Ensure that you are populating the Profile and Region fields with the values you configured earlier in the AWS CLI.



  "AWS": {
    "Profile": "default",
    "Region": "us-east-2"
  }

Run the following commands via Visual Studio to install the required AWS NuGet packages.

Install-Package AWSSDK.S3
Install-Package AWSSDK.Extensions.NETCore.Setup

With that done, let’s register the AWS Service into the .NET application’s Container. Open up the Program.cs and make the modifications as below. You might have to use the Amazon.S3 namespace while referencing the below changes.



builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonS3>();

Line 1 Loads the AWS Configuration from the appsettings.json into the application’s runtime.

Line 2 Adds the S3 Service into the pipeline. We will be injecting this interface into our controllers to work with Bucket and Objects!

Its time to create a new API controller under Controllers and name it S3Controller.cs. You would need to inject the IAmazonS3 interface into the constructor of the S3Controller.



    [Route("api/[controller]")]
    [ApiController]
    public class S3Controller : ControllerBase
    {
        private readonly IAmazonS3 _s3Client;

        public S3Controller(IAmazonS3 s3Client)
        {
            _s3Client = s3Client;
        }
    }

Let's create an endpoint to upload a file into S3.



 [HttpPost("upload")]
    public async Task<IActionResult> UploadFileAsync(IFormFile file)
    {

    }

This API will get the file from the user and upload the file content into S3.

Before uploading the file, we have to create the bucket in S3. Go to the S3 service in the AWS Management Console and Create a new bucket by pressing the "Create Bucket" button.

Provide a name for the bucket and press create bucket at the bottom.

Now, you can see your bucket in the bucket list.

Let's go and implement the API "UploadFileAsync" in the S3Controller class.



 [HttpPost("upload")]
        public async Task<IActionResult> UploadFileAsync(IFormFile file)
        {
            var request = new PutObjectRequest()
            {
                BucketName = "blazor-file-transfer-demo",
                Key = file.FileName,
                InputStream = file.OpenReadStream()
            };

            request.Metadata.Add("Content-Type", file.ContentType);
            await _s3Client.PutObjectAsync(request);
            return Ok($"File {file.FileName} uploaded to S3 successfully!");
        }

The above API creates a putobjectrequest with bucket name, key, and input stream. The incoming file is converted to a stream via "OpenReadStream()". The bucket name is the one which we created just now and the key name is nothing but the file name.
Calling the PutObjectAsync() API that is available in the S3 client SDK will upload the incoming file data into the S3 bucket we specified.

Now, it's time to create an API for downloading the file. As same as above, create an API called "DownloadFileAsync" inside the S3Controller.



        [HttpPost("download")]
        public async Task<IActionResult> DownloadFileAsync(string key)
        {
            var s3Object = await _s3Client.GetObjectAsync("blazor-file-transfer-demo", key);
             return File(s3Object.ResponseStream, s3Object.Headers.ContentType, key);
        }

In the above implementation, we are getting the key name (filename) from the user. We will use the key name and the bucket name to fetch the object from the S3 by Calling the GetObjectAsync() API available on the S3 Client SDK. This will provide us with the S3 object as a stream. We will return the stream as a File object to the client application.

At this point, we needed another API to fetch the list of files inside the bucket.



 [HttpGet]
        public async Task<IActionResult> GetAllFilesAsync()
        {
            var request = new ListObjectsV2Request()
            {
                BucketName = "blazor-file-transfer-demo",
            };
            var result = await _s3Client.ListObjectsV2Async(request);
            return Ok(result.S3Objects);
        }

That's it! We have created all the required APIs to list all the available files, upload a file and download a file from S3.

We'll see how to upload/download files from the blazor application with the help of these APIs in the next chapter.

Setup AWS Credentials

Sugumar Prabhakar — Tue, 21 Mar 2023 02:11:24 +0000

First, we need to create an IAM user with programmatic access.
Let's get into the AWS Management Console and search for IAM. Go to the IAM and click "Add User" button.

Provide a username and click next and next. If you want to add any tags, you can add them here and click create. Now the user will be created.

Click the user. You'll see the permission tab. For our application to access the S3 storage, we need to provide this user with S3 permission. To do that, Click on the Add permission button and
choose the "Attach Policies Directly" option. Now choose "AmazonS3FullAccess" and click Next. Now review the policy and press Add Permission. The permission should be added now to the user.

Now it's time to configure the IAM user credentials on our pc. To do that, we need to create access key and secret for the user we created just now.

Go to the "Security Credentials" inside the IAM user. Click the "Create access key" button to create keys.

Now choose the "Application running outside AWS" option and click next. Add any description tag value if you want and press "Create access key"

You can see your access key in the console now. You can download the secrets in a CSV file by clicking "Download .csv file" at the bottom.

That's it. We have created the IAM user, provided the S3 permission, and created the access keys to access S3 from our application. Let's go and configure the credentials on our pc.

To configure the AWS Credentials, the latest AWS Cli must be installed on your machine. You can download the Cli on the official AWS website here

To make sure the AWS Cli is installed on your machine, run the below command in your command prompt.

aws --version

The response should be like this below except the version number might be different.

Now run the below command,

aws configure

The command prompt will ask for the access key, secret key, default regions, and output format. You can find the access key and secret key in the CSV file you downloaded just now. Provide the same.
For the default region, I'm providing us-east-2 and "json" as the default output format. But you can provide whatever region you're gonna use for other AWS services.

S3 File Upload & Download using .NET7 Blazor WASM & WebApi

Sugumar Prabhakar — Tue, 21 Mar 2023 02:11:10 +0000

In this series, We are going to see how to create a Blazor Web Assembly application that will connect with the AWS S3 services and do file uploads and downloads into S3 buckets. An ASP NET Core WebAPI application will be acting as an intermediate backend service that will handle the request from the Blazor application and do the appropriate actions in the AWS S3.

We will use .NET 7 for this tutorial as it is the latest at this time. In the next chapter, you will find out how to set up AWS IAM users.

.NET7 Blazor Server app in AWS ECS

Sugumar Prabhakar — Thu, 09 Mar 2023 02:18:37 +0000

What is Blazor?

In simple terms, Blazor is a Single Page Application development framework provided and owned by the .NET foundation. Blazor uses the power of .NET and C# to build full-stack web apps without writing a line of JavaScript. Blazor can run anywhere such as client browsers on WASM, server-side on ASP.NET core, or in the native client apps such as mobile or desktop apps.

In this post, we are going to create a sample Blazor Server SPA application, build it as a docker image and push it to AWS and deploy it there.

What is the Blazor Server hosting model?

Blazor server application means the web application is executed on the server side instead of in a browser. ASP.NET Core apps can be configured to use Blazor Server as the hosting model. A SignalR connection using WebSockets will be created for each tab in the client browser which will be responsible for updating UI content and handling events. Blazor allows the scaling apps to handle multiple client connections.

The state on the server associated with each connected client is called a circuit. Circuits don't get tied to a specific network connection and can tolerate - temporarily - interrupted connections. In addition, clients can reconnect to the server when they want to even after going offline.

To know more about Blazor hosting models, please go through the official documentation from the Microsoft

Let's start with work!
What we gonna do here is,

Create a sample Blazor Server App in VS2022
Adding a docker file
Build the Blazor server app as an image and push it to AWS ECR.
Create Networking Infrastructure in AWS using Cloudformation.
Create ECS cluster, services, and task definitions to deploy the Blazor server app.
Test the application.

Creating Blazor Server sample app

First, open visual studio 2022 and go to create project option. In the templates, choose the Blazor Server app.

Then, choose the directories to save the project and then press next. Now it's time to choose the .NET SDK version. I have .NET7 installed, so, I'm choosing the same and press next.

That's it. The sample application is created now. Press "CTRL+F5" to run the application locally. If you see the below page in your default browser, then everything is working as expected.

Adding Dockerfile

Now it's time to build and push the application as a docker image into AWS ECR. To do that, we need to do a couple of things. First, we need to add a Dockerfile in our project folder that helps docker to build an image step by step.

I'm gonna create a folder called "buildScripts" and inside that, I create a file "Dockerfile" with the below contents on it.



FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS base
WORKDIR /app
EXPOSE 80

ENV ASPNETCORE_URLS=http://*:8000

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
WORKDIR /src
COPY ["BlazorSampleApp.csproj", "."]
RUN dotnet restore "./BlazorSampleApp.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "BlazorSampleApp.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "BlazorSampleApp.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "BlazorSampleApp.dll"]

If you have noted, I'm exposing the application inside the container in port 8000.

Build the Docker image and push it to AWS ECR

Now, I'm gonna build and push the application as a docker container image into AWS ECR. Before that, we need to configure the AWS account on our machine.

To do that, we need the latest AWS CLI installed on our PC. I do have AWS CLI V2 installed already. If you don't, get the latest version here

To verify the installation of AWS, press aws --version in the command prompt. It will tell you the correct version number if that is installed perfectly.

To configure, Open the command prompt and type aws configure and press enter. Now it will ask for Access ID, Key, default region, and output. Give the appropriate inputs and press enter so that it will be configured.

Now, we will create an ECR repository in the AWS by running the below command in the command prompt.

aws ecr create-repository --repository-name blazorserverapp

The successful response of this command will give you the full repository Uri. We will use the Uri to tag our docker image during the application build stage.

Running the below commands from the Blazor application project root directory will login your aws cli into ECR, build & tag the docker image and push it.



aws ecr get-login-password --region {your-default-region} | docker login --username AWS --password-stdin {your-aws-account-id}.dkr.ecr.{your-default-region}.amazonaws.com

docker build -t {your-aws-account-id}.dkr.ecr.{your-default-region}.amazonaws.com/blazorserverapp:latest -f buildScripts\Dockerfile .

docker push {your-aws-account-id}.dkr.ecr.{your-default-region}.amazonaws.com/blazorserverapp:latest

Create Networking Infrastructure in AWS

Our next step is to deploy the container image we pushed just now into Amazon ECS. To do that, We are gonna set up our own networks first such as VPC, Subnets, etc. We are going to create a VPC, 2 public subnets, 2 private subnets, a public load balancer, a private load balancer, and its associated resources.

We will deploy our Blazor application container image inside the public subnets which are inside the VPC. We'll use the AWS Cloudformation service to create our resources.

Add the below cloudformation template as "ecs-infra.yaml" inside the buildscripts folder



AWSTemplateFormatVersion: 2010-09-09
Description: This stack creates a VPC with 3 public subnets and private subnets.
  Then, it will create a public load balancer and private load balancer. Public LB is publicly accessable where as
  private LB will be accessible within the service that deployed inside private subnets.
  The docker containers will be deployed in Fargate ECS clusters. It will be deployed in either public subnets and associate
  with public LB listeres or in private subnets and associate with private LB

Parameters:
  VPCCIDR:
    Type: String
    Description: CIDR value for the VPC.
    Default: "10.0.0.0/16"
  PublicSubnet1CIDR:
    Type: String
    Description: CIDR value for the public subnet 1.
    Default: "10.0.0.0/24"
  PublicSubnet2CIDR:
    Type: String
    Description: CIDR value for the public subnet 2.
    Default: "10.0.1.0/24"
  PrivateSubnet1CIDR:
    Type: String
    Description: CIDR value for the private subnet 1.
    Default: "10.0.3.0/24"
  PrivateSubnet2CIDR:
    Type: String
    Description: CIDR value for the private subnet 2.
    Default: "10.0.4.0/24"
  PublicLoadBalancerName:
    Type: String
    Description: Name of the public application load balancer the blazorserverapp application will use for
  PrivateLoadBalancerName:
    Type: String
    Description: Name of the private application load balancer the blazorserverapp containers inside ecs taske will use for its communication with other containers
  ECSClusterName:
    Type: String
    Description: Name of the ecs cluster that borker 2 application uses to deploy its microservices

Resources:
  # VPC in which containers will be networked.
  # It has two public subnets, and two private subnets.
  # We distribute the subnets across the first two available subnets
  # for the region, for high availability.
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      EnableDnsSupport: true
      EnableDnsHostnames: true
      CidrBlock: !Ref VPCCIDR
      Tags:
        - Key: Name
          Value: blazorserverapp-vpc

  # Two public subnets, where containers can have public IP addresses
  PublicSubnetOne:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Select [0, Fn::GetAZs: !Ref "AWS::Region"]
      VpcId: !Ref "VPC"
      CidrBlock: !Ref PublicSubnet1CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: blazorserverapp-pub-sub1

  PublicSubnetTwo:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Select [1, Fn::GetAZs: !Ref "AWS::Region"]
      VpcId: !Ref "VPC"
      CidrBlock: !Ref PublicSubnet2CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: blazorserverapp-pub-sub2

  # Theree private subnets where containers will only have private
  # IP addresses, and will only be reachable by other members of the
  # VPC
  PrivateSubnetOne:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Select [0, Fn::GetAZs: !Ref "AWS::Region"]
      VpcId: !Ref "VPC"
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: blazorserverapp-pri-sub1

  PrivateSubnetTwo:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Select [1, Fn::GetAZs: !Ref "AWS::Region"]
      VpcId: !Ref "VPC"
      CidrBlock: !Ref PrivateSubnet2CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: blazorserverapp-pri-sub2

  # Setup networking resources for the public subnets.
  # Containers in the public subnets have public IP addresses and the routing table
  # sends network traffic via the internet gateway.
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: blazorserverapp-vpc-igw

  GatewayAttachement:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref "VPC"
      InternetGatewayId: !Ref "InternetGateway"

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref "VPC"
      Tags:
        - Key: Name
          Value: blazorserverapp-pub-route-tbl

  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: GatewayAttachement
    Properties:
      RouteTableId: !Ref "PublicRouteTable"
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: !Ref "InternetGateway"

  PublicSubnetOneRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnetOne
      RouteTableId: !Ref PublicRouteTable

  PublicSubnetTwoRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnetTwo
      RouteTableId: !Ref PublicRouteTable

  # Setup networking resources for the private subnets.
  # Containers in these subnets have only private IP addresses, and must use a NAT
  # gateway to talk to the internet. We launch two NAT gateways, one for
  # each private subnet.
  NatGatewayOneAttachment:
    Type: AWS::EC2::EIP
    DependsOn: GatewayAttachement
    Properties:
      Domain: vpc
  NatGatewayTwoAttachment:
    Type: AWS::EC2::EIP
    DependsOn: GatewayAttachement
    Properties:
      Domain: vpc
  NatGatewayOne:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGatewayOneAttachment.AllocationId
      SubnetId: !Ref PublicSubnetOne
      Tags:
        - Key: Name
          Value: blazorserverapp-nat1
  NatGatewayTwo:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGatewayTwoAttachment.AllocationId
      SubnetId: !Ref PublicSubnetTwo
      Tags:
        - Key: Name
          Value: blazorserverapp-nat2

  PrivateRouteTableOne:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref "VPC"
  PrivateRouteOne:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTableOne
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGatewayOne
  PrivateRouteTableOneAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTableOne
      SubnetId: !Ref PrivateSubnetOne
  PrivateRouteTableTwo:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref "VPC"
  PrivateRouteTwo:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTableTwo
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGatewayTwo
  PrivateRouteTableTwoAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTableTwo
      SubnetId: !Ref PrivateSubnetTwo

  PublicLoadBalancerSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: public internet access to the load balancer
      VpcId: !Ref "VPC"
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          IpProtocol: "tcp"
          Description: "Allow HTTP Request from anywhere in the internet"
          FromPort: 80
          ToPort: 80
        - CidrIp: 0.0.0.0/0
          Description: "Allow HTTPS Request from anywhere in the internet"
          IpProtocol: "tcp"
          FromPort: 443
          ToPort: 443
      Tags:
        - Key: Name
          Value: blazorserverapp-pub-lb-access-sg

  PublicLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Ref PublicLoadBalancerName
      Scheme: internet-facing
      LoadBalancerAttributes:
        - Key: idle_timeout.timeout_seconds
          Value: "30"
      Subnets:
        # The load balancer is placed into the public subnets, so that traffic
        # from the internet can reach the load balancer directly via the internet gateway
        - !Ref PublicSubnetOne
        - !Ref PublicSubnetTwo
      SecurityGroups: [!Ref "PublicLoadBalancerSG"]

  # A dummy target group is used to setup the ALB to just drop traffic
  # initially, before any real service target groups have been added.
  DummyTargetGroupPublic:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 60
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      Name: blazorserverapp-default-public
      Port: 80
      Protocol: HTTP
      UnhealthyThresholdCount: 10
      VpcId: !Ref "VPC"

  PublicLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn:
      - PublicLoadBalancer
    Properties:
      DefaultActions:
        - TargetGroupArn: !Ref "DummyTargetGroupPublic"
          Type: "forward"
      LoadBalancerArn: !Ref "PublicLoadBalancer"
      Port: 80
      Protocol: HTTP

  # An internal load balancer, this would be used for a service that is not
  # directly accessible to the public, but instead should only receive traffic
  # from your other services.
  PrivateLoadBalancerSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Restricted access to load balancer
      VpcId: !Ref "VPC"
      Tags:
        - Key: Name
          Value: blazorserverapp-pri-lb-access-sg

  PrivateLoadBalancerIngressFromECS:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Only accept traffic from a container in the fargate container security group
      GroupId: !Ref "PrivateLoadBalancerSG"
      IpProtocol: "-1"
      SourceSecurityGroupId: !Ref "FargateContainerSecurityGroup"

  PrivateLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Ref PrivateLoadBalancerName
      Scheme: internal
      LoadBalancerAttributes:
        - Key: idle_timeout.timeout_seconds
          Value: "30"
      Subnets:
        # This load balancer is put into the private subnet, so that there is no
        # route for the public to even be able to access the private load balancer.
        - !Ref PrivateSubnetOne
        - !Ref PrivateSubnetTwo
      SecurityGroups: [!Ref "PrivateLoadBalancerSG"]

  # This dummy target group is used to setup the ALB to just drop traffic
  # initially, before any real service target groups have been added.
  DummyTargetGroupPrivate:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 60
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      Name: blazorserverapp-default-private
      Port: 80
      Protocol: HTTP
      UnhealthyThresholdCount: 10
      VpcId: !Ref "VPC"

  PrivateLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn:
      - PrivateLoadBalancer
    Properties:
      DefaultActions:
        - TargetGroupArn: !Ref "DummyTargetGroupPrivate"
          Type: "forward"
      LoadBalancerArn: !Ref "PrivateLoadBalancer"
      Port: 80
      Protocol: HTTP

  # A security group for the containers we will run in Fargate.
  # Two rules, allowing network traffic from a public facing load
  # balancer, a private internal load balancer, and from other members
  # of the security group.
  #
  # Remove any of the following ingress rules that are not needed.
  FargateContainerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Access to the Fargate containers
      VpcId: !Ref "VPC"
      Tags:
        - Key: Name
          Value: blazorserverapp-ecs-container-sg

  EcsSecurityGroupIngressFromPublicALB:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Ingress from the public ALB
      GroupId: !Ref "FargateContainerSecurityGroup"
      IpProtocol: "-1"
      SourceSecurityGroupId: !Ref "PublicLoadBalancerSG"

  EcsSecurityGroupIngressFromPrivateALB:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Ingress from the private ALB
      GroupId: !Ref "FargateContainerSecurityGroup"
      IpProtocol: "-1"
      SourceSecurityGroupId: !Ref "PrivateLoadBalancerSG"

  EcsSecurityGroupIngressFromSelf:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Ingress from other containers in the same security group
      GroupId: !Ref "FargateContainerSecurityGroup"
      IpProtocol: "-1"
      SourceSecurityGroupId: !Ref "FargateContainerSecurityGroup"

  ECSCluster:
    DependsOn:
      - PublicLoadBalancer
      - PrivateLoadBalancer
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Ref ECSClusterName
      ClusterSettings:
        - Name: containerInsights
          Value: enabled

Outputs:
  InternalUrl:
    Description: The url of the internal load balancer
    Value: !Join ["", ["http://", !GetAtt "PrivateLoadBalancer.DNSName"]]
    Export:
      Name: InternalUrl
  ExternalUrl:
    Description: The url of the external load balancer
    Value: !Join ["", ["http://", !GetAtt "PublicLoadBalancer.DNSName"]]
    Export:
      Name: ExternalUrl
  VPCId:
    Description: The ID of the VPC that this stack is deployed in
    Value: !Ref "VPC"
    Export:
      Name: VPC
  PublicSubnetOne:
    Description: Public subnet one
    Value: !Ref "PublicSubnetOne"
    Export:
      Name: PublicSubnetOne
  PublicSubnetTwo:
    Description: Public subnet two
    Value: !Ref "PublicSubnetTwo"
    Export:
      Name: PublicSubnetTwo
  PrivateSubnetOne:
    Description: Private subnet one
    Value: !Ref "PrivateSubnetOne"
    Export:
      Name: PrivateSubnetOne
  PrivateSubnetTwo:
    Description: Private subnet two
    Value: !Ref "PrivateSubnetTwo"
    Export:
      Name: PrivateSubnetTwo
  FargateContainerSecurityGroup:
    Description: The security group to be used by blazorserverapp containers
    Value: !Ref FargateContainerSecurityGroup
    Export:
      Name: FargateContainerSecurityGroup
  ClusterName:
    Description: The name of the ECS cluster created for blazorserverapp applications
    Value: !Ref "ECSCluster"
    Export:
      Name: ClusterName
  PublicLoadBalancerListener:
    Description: Listener for the load balancer exposed to the public internet
    Value: !Ref PublicLoadBalancerListener
    Export:
      Name: PublicListener
  PrivateLoadBalancerListener:
    Description: Listener for the load balancer exposed internally inside the VPC
    Value: !Ref PrivateLoadBalancerListener
    Export:
      Name: PrivateListener

In a same way, Add the below configuration template as "ecs-infra.json" inside the buildscripts folder



[
  {
    "ParameterKey": "VPCCIDR",
    "ParameterValue": "11.200.13.0/24"
  },
  {
    "ParameterKey": "PublicSubnet1CIDR",
    "ParameterValue": "11.200.13.0/26"
  },
  {
    "ParameterKey": "PublicSubnet2CIDR",
    "ParameterValue": "11.200.13.64/26"
  },
  {
    "ParameterKey": "PrivateSubnet1CIDR",
    "ParameterValue": "11.200.13.128/26"
  },
  {
    "ParameterKey": "PrivateSubnet2CIDR",
    "ParameterValue": "11.200.13.192/26"
  },
  {
    "ParameterKey": "PublicLoadBalancerName",
    "ParameterValue": "blazorserverapp"
  },
  {
    "ParameterKey": "PrivateLoadBalancerName",
    "ParameterValue": "blazorserverapp-internal"
  },
  {
    "ParameterKey": "ECSClusterName",
    "ParameterValue": "blazorserverapp"
  }
]

Run the below command from the blazor application project root directory


 cloudformation create-stack --stack-name blazorserverapp-networking-infra --capabilities=CAPABILITY_NAMED_IAM --template-body file://./buildScripts/ecs-infra.yaml --parameters file://./buildScripts/ecs-infra.json

The above command will validate and create the needful networking things in the aws. You can see the status of the command in the AWS CloudFormation console. The status will be "Create Completed". If there is any failures, Make sure your IAM user have necessary access to create these resources.

Deploy Blazor app into ECS

Save the below Cloudformation template as "ecs-service.yaml" inside the buildScripts directory




AWSTemplateFormatVersion: 2010-09-09
Description: This stack will deploy the blazorserverapp on AWS Fargate,
  hosted in a private/public subnet, behind a private/public load balancer.

Parameters:
  ServiceName:
    Type: String
    Description: A name for the service
  ImageUrl:
    Type: String
    Description:
      The url of a docker image that contains the application process that
      will handle the traffic for this service
  ContainerPort:
    Type: Number
    Description: What port number the application inside the docker container is binding to
  DesiredCount:
    Type: Number
    Default: 1
    Description: How many copies of the service task to run
  ListenerRulePriority:
    Type: Number
    Default: 1
    Description: The priority of the listener rule defined in the load balancer
  LogGroupName:
    Type: String
    Description: Name of the log group where the logs from the service should be saved
  ASPNETCOREENV:
    Type: String
    Description: Value of the ASPNETCORE_ENVIRONMENT to be passed to the container
  HealthCheckURL:
    Type: String
    Description: Path of the health check address to be done by the load balancer

Resources:
  ECSTaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ecs-tasks.amazonaws.com]
            Action: ["sts:AssumeRole"]
      Path: /
      RoleName: BlazorServerAppECSTaskExecRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

  ECSTaskALBPermissionPolicy:
    Properties:
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - "elasticloadbalancing:DeregisterInstancesFromLoadBalancer"
              - "elasticloadbalancing:DeregisterTargets"
              - "elasticloadbalancing:Describe*"
              - "elasticloadbalancing:RegisterInstancesWithLoadBalancer"
              - "elasticloadbalancing:RegisterTargets"
            Resource:
              - !Join
                - ":"
                - - arn:aws:elasticloadbalancing
                  - !Ref "AWS::Region"
                  - !Ref "AWS::AccountId"
                  - !Sub "loadbalancer/app/blazorserverapp*"
      PolicyName: ECSTaskRole-ALB-PermissionPolicy
      Roles:
        - !Ref ECSTaskExecutionRole
    Type: AWS::IAM::Policy

  CloudWatchLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Ref "LogGroupName"
      RetentionInDays: 7
  # The task definition. This is a simple metadata description of what
  # container to run, and what resource requirements it has.
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref "ServiceName"
      Cpu: 256
      Memory: 512
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !GetAtt ECSTaskExecutionRole.Arn
      ContainerDefinitions:
        - Name: !Ref "ServiceName"
          Cpu: 256
          Memory: 512
          Image: !Ref "ImageUrl"
          PortMappings:
            - ContainerPort: !Ref "ContainerPort"
          Environment:
            - Name: "ASPNETCORE_ENVIRONMENT"
              Value: !Ref ASPNETCOREENV
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref "CloudWatchLogGroup"
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: !Ref "ASPNETCOREENV"

  # The service. The service is a resource which allows you to run multiple
  # copies of a type of task, and gather up their logs and metrics, as well
  # as monitor the number of running tasks and replace any that have crashed
  Service:
    Type: AWS::ECS::Service
    DependsOn: LoadBalancerRule
    Properties:
      ServiceName: !Ref "ServiceName"
      Cluster: !ImportValue ClusterName
      LaunchType: FARGATE
      DeploymentConfiguration:
        DeploymentCircuitBreaker:
          Enable: true
          Rollback: true
        MaximumPercent: 200
        MinimumHealthyPercent: 75
      DesiredCount: !Ref "DesiredCount"
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          SecurityGroups:
            - !ImportValue FargateContainerSecurityGroup
          Subnets:
            - !ImportValue PublicSubnetOne
            - !ImportValue PublicSubnetTwo
      TaskDefinition: !Ref "TaskDefinition"
      LoadBalancers:
        - ContainerName: !Ref "ServiceName"
          ContainerPort: !Ref "ContainerPort"
          TargetGroupArn: !Ref "TargetGroup"

  # A target group. This is used for keeping track of all the tasks, and
  # what IP addresses / port numbers they have. You can query it yourself,
  # to use the addresses yourself, but most often this target group is just
  # connected to an application load balancer, or network load balancer, so
  # it can automatically distribute traffic across all the targets.
  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 6
      HealthCheckPath: !Ref "HealthCheckURL"
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      TargetType: ip
      Name: !Ref "ServiceName"
      Port: !Ref "ContainerPort"
      Protocol: HTTP
      UnhealthyThresholdCount: 2
      VpcId: !ImportValue VPC

  # Create a rule on the load balancer for routing traffic to the target group
  LoadBalancerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
        - TargetGroupArn: !Ref "TargetGroup"
          Type: "forward"
      Conditions:
        - Field: path-pattern
          PathPatternConfig:
            Values:
              - /*
      ListenerArn: !ImportValue PublicListener
      Priority: !Ref ListenerRulePriority

In the same way, add the below configuration template as "ecs-service.json" file inside the buildScripts directory.




[
  {
    "ParameterKey": "ServiceName",
    "ParameterValue": "blazorserverapp"
  },
  {
    "ParameterKey": "ImageUrl",
    "ParameterValue": "{your-aws-account-id}.dkr.ecr.{your-default-region}.amazonaws.com/blazorserverapp:latest"
  },
  {
    "ParameterKey": "ContainerPort",
    "ParameterValue": "8000"
  },
  {
    "ParameterKey": "DesiredCount",
    "ParameterValue": "1"
  },
  {
    "ParameterKey": "ListenerRulePriority",
    "ParameterValue": "1"
  },
  {
    "ParameterKey": "LogGroupName",
    "ParameterValue": "blazorserverapp-service"
  },
  {
    "ParameterKey": "ASPNETCOREENV",
    "ParameterValue": "Development"
  },
  {
    "ParameterKey": "HealthCheckURL",
    "ParameterValue": "/"
  }
]

Run the below command from the blazor application project root directory. This will create ECS clusters, Service, task definitions and deploy the container image we pushed just before into the task running inside service.



 cloudformation create-stack --stack-name blazorserverapp-service-infra --capabilities=CAPABILITY_NAMED_IAM --template-body file://./buildScripts/ecs-service.yaml --parameters file://./buildScripts/ecs-service.json

Test the application.

It's time to test our application. To find the URL of our application, You need to go to AWS Cloudformation in the console. Search for "blazorserverapp-networking-infra" stack. You'll find the external Url once you go to the output tab of "blazorserverapp-networking-infra" stack. Your application should open up once you click that link.

That's it. We have successfully created and deployed a Blazor Server .NET7 app in the AWS. We can see how to set up CICD for our application in a different post.

Full source is available here
https://github.com/sprabha1990/BlazorServer.NET7

Self Signed RSA Certificate with .NET 7

Sugumar Prabhakar — Tue, 21 Feb 2023 22:01:03 +0000

Asymmetric encryption uses a public/private key pair to encrypt the data at one end and decrypt the data on the other end. Most of the times, the encryption keys are created in the form of certificates, which might have been created in advance.

But for some scenarios such as IOT, certificates should be created for each device beforehand, which should be registered on the public cloud provider before using the IoT features. It will be tricky to create certificates beforehand if we have millions of devices to work with IoT and goes for production.

In this case, we can create our own self-signed device certificates at runtime using RSA in .NET Core

I'm gonna create a console application that will create a random RSA certificate and export the certificate and its private and public keys locally.

To do that, I'm gonna create a Console App project in VS2022.

Choosing .NET 7 as a framework

Now the project has been created.
Let's create a public record model that will store the certificate instance as well as its keys.


 record CertificateModel(X509Certificate2 Certificate, string PublicKeyPEM, string PrivateKeyPEM, string PublicKeyPFX = "");

Next, I'm gonna create a helper class to create RSA certificates and store its keys in a CertificateModel instance




internal class RSAHelper
{
    /// <summary>
    ///     Create a random RSA certificate for asymentric encryption
    /// </summary>
    /// <param name="subjectName">Subject name for the certificate. Ex, "cn=test"</param>
    /// <param name="hashAlgorithmName">Algorithm mode for creating hash keys</param>
    /// <param name="signaturePadding">Padding mode for the certificate signature</param>
    /// <returns>Certificate model</returns>
    public CertificateModel CreateRSACertificate(string subjectName
        , HashAlgorithmName hashAlgorithmName
        , RSASignaturePadding signaturePadding)
    {
        using var rsa = RSA.Create();
        var certRequest = new CertificateRequest(subjectName, rsa, hashAlgorithmName, signaturePadding);
        var certificate = certRequest.CreateSelfSigned(DateTimeOffset.Now, DateTimeOffset.Now.AddDays(1));

        var privateKey = Convert.ToBase64String(rsa.ExportRSAPrivateKey(), Base64FormattingOptions.InsertLineBreaks);
        var publicKey = certificate.ExportCertificatePem();
        var publicKeyPfx = Convert.ToBase64String(certificate.Export(X509ContentType.Pfx), Base64FormattingOptions.InsertLineBreaks);

        return new CertificateModel(certificate, publicKey, privateKey, publicKeyPfx);
    }
}

That's it. The method "CreateRSACertificate" will create random RSA Keys and create a self-signed X509Certificate2 instance using the RSA keys which we created. It exports the private key and public key in a PEM format and stores it in the CertificateModel instance which will be returned to the caller method.

Now, we have to call RSAHelper.CreateRSACertificate() instance from the Program.cs class.



RSAHelper rsaHelperInstance = new RSAHelper();

Console.Write("Enter the subject name of the certificate : ");
var subjectName = $"CN={Console.ReadLine()?.ToUpper()}";

CertificateModel certificate = rsaHelperInstance.CreateRSACertificate(subjectName: subjectName
    , hashAlgorithmName: HashAlgorithmName.SHA256
    , signaturePadding: RSASignaturePadding.Pkcs1);

Console.WriteLine("Certificate is ready !!");
Console.Write("Do you want to save it locally? Press y or n : ");

char choice = Console.ReadKey().KeyChar;

if (choice == 'y')
{
    if (!Directory.Exists("Certificates"))
        Directory.CreateDirectory("Certificates");

    File.WriteAllText("Certificates/certificate.pem", certificate.PublicKeyPEM);
    File.WriteAllText("Certificates/certificate.private.pem", certificate.PrivateKeyPEM);
    File.WriteAllText("Certificates/certificate.pfx", certificate.PublicKeyPFX);

    Console.WriteLine();
    string filesLocation = Path.Combine(Directory.GetParent(Assembly.GetExecutingAssembly().Location)?.FullName ?? "", "Certificates");
    Console.WriteLine($"Files are saved in {filesLocation}");

    Process.Start(fileName: "explorer.exe", arguments: filesLocation);
}

Here, I'm getting the subject name for the certificate from the user and passing it to the CreateRSACertificate() method which returns the CertificateModel Instance. Based on the user's interest, I'm saving the certificates and keys in a local file.

We can also save the certificates in X509Stores for appropriate usage.

Full source code is available here "RSACertificateGenerator.NET7"
_

Data streaming in .NET 6 using Kafka

Sugumar Prabhakar — Fri, 05 Nov 2021 02:11:14 +0000

Data streaming is nothing but a flow of data from one end to another. Normally, there will be a producer who sends some event information to consumers via message queue/brokers. Here, we are going to create a data streaming framework using .NET core 6 which acts as producer and consumers while Kafka will act as message broker.

Before starting to create the producer and consumers, we need to create a Kafka broker. I'm using docker and docker-compose to create a Kafka in my localhost.

Here is the docker-compose.yml file.

version: "3"

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - kafka_net
  kafka:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    restart: on-failure
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    networks:
      - kafka_net
networks:
  kafka_net:
    driver: "bridge"

Running this script with 'docker-compose -d up' will create the Kafka and ZooKeeper service in my local host.

This post is not about the docker, so I'm moving to the actual part.

Kafka Producer

Kafka producer is a .NET core 6 console application with C#10.
We need the Nuget package Confluent.Kafka to use Kafka in .NET client application. Hence, we are installing this via Nuget package manager.

As one of the improvements in C#10, we can use 'global using' throughout the application. We are now creating the global.cs file with default.

global using System.Text;
global using static System.Console;
global using System.Text.Json;
global using Confluent.Kafka;

Next, we are going to create a Message class. This is the data class which will be serialized and published to the Kafka in this example. This is a simple class containing 3 public properties namely Id, Data and Timestamp.

namespace KafkaProducer;

public class Message
{
    public string Id { get; set; } = Guid.NewGuid().ToString();
    public string? Data { get; set; }
    public DateTime Timestamp { get; set; } = DateTime.UtcNow;

    public override string ToString()
    {
        return JsonSerializer.Serialize(this);
    }
}

The ID is an auto generated GUID and the timestamp will be the current time when the Message object is created.

Now, we are going to create a Custom Serializer class. The reason for this class is that Kafka implicitly supports some predefined datatypes to serialize and deserialize in it which are,

Null
Ignore
Int32
Int64
Utf8
Single
Double
ByteArray

However, if we need to publish/consume different data from these, we need to tell Kafka which serializer/deserializer object it needs to use.

namespace KafkaProducer
{
    public class CustomValueSerializer<T> : **ISerializer<T>**
    {
        public byte[] Serialize(T data, SerializationContext context)
        {
            return Encoding.UTF8.GetBytes(JsonSerializer.Serialize(data, typeof(T)));
        }
    }
}

As you can see, the CustomValueSerializer is implementing the ISerialize interface from Kafka which has a single method implemented called "Serialize". This will be used by Kafka while publishing this message.

Now, we are going to create a producer class which publishes the message to Kafka asynchronously.

namespace KafkaProducer;
public class Producer<T>
{

    readonly string? _host;
    readonly int _port;
    readonly string? _topic;

    public Producer()
    {
        _host = "localhost";
        _port = 9092;
        _topic = "producer_logs";
    }

    ProducerConfig GetProducerConfig()
    {
        return new ProducerConfig
        {
            BootstrapServers = $"{_host}:{_port}"
        };
    }

    public async Task ProduceAsync(T data)
    {
        using (var producer = new ProducerBuilder<Null, T>(GetProducerConfig())
                                             .SetValueSerializer(new CustomValueSerializer<T>())
                                             .Build())
        {
            await producer.ProduceAsync(_topic, new Message<Null, T> { Value = data });
            WriteLine($"{data} published");
        }
    }
}

This class has a public method "ProduceAsync" which gets the message information and publishes it to Kafka. If you notice, while creating the ProducerBuilder object, we are letting Kafka know that the value serializer for this producer is "CustomValueSerializer" by using an api .SetValueSerializer(new CustomValueSerializer())

Now, from the program.cs, we are creating an instance of producer class and invoke the produceasync method to publish our data to Kafka.

namespace KafkaProducer;

public class Program
{
    public static async Task Main(string[] args)
    {
        var producer = new Producer<Message>();

        for(int i=0; i< 25 ; i++)
        {
            await producer.ProduceAsync(new Message
            {
                Data = $"Pushing Data {i} !!",
            });

            await Task.Delay(1000);
        }

        WriteLine("Publish Success!");
        ReadKey();
    }
}

If you run this, the application will publish 25 times to Kafka.

Kafka Consumer

Kafka consumer is again a .NET core 6 console application with C#10. We need the same Nuget package Confluent.Kafka here too.
After installing that, we will create a global.cs file to add global usings like what we did for producer.

Once global file is added, we will create a Message class.

namespace KafkaConsumer;

public class Message
{
    public string? Id { get; set; }
    public string? Data { get; set; }
    public DateTime Timestamp { get; set; }

    public override string ToString()
    {
        return JsonSerializer.Serialize(this);
    }
}

Next, we are going to create Custom serializer class that will be used by Kafka while serializing the published message.

namespace KafkaConsumer
{
    public class CustomValueDeserializer<T> : IDeserializer<T>
    {
        public T Deserialize(ReadOnlySpan<byte> data, bool isNull, SerializationContext context)
        {
            return JsonSerializer.Deserialize<T>(data);
        }
    }
}

Now, we are going to create a generic Consumer class to start consuming the Kafka topic.

namespace KafkaConsumer;
public class Consumer<T>
{
    readonly string? _host;
    readonly int _port;
    readonly string? _topic;

    public Consumer()
    {
        _host = "localhost";
        _port = 9092;
        _topic = "producer_logs";
    }

    ConsumerConfig GetConsumerConfig()
    {
        return new ConsumerConfig
        {
            BootstrapServers = $"{_host}:{_port}",
            GroupId = "foo",
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
    }

    public async Task ConsumeAsync()
    {
        using (var consumer = new ConsumerBuilder<Ignore, T>(GetConsumerConfig())
            .SetValueDeserializer(new CustomValueDeserializer<T>())                        
            .Build())
        {
            consumer.Subscribe(_topic);

            WriteLine($"Subscribed to {_topic}");

            await Task.Run(() =>
            {
                while (true)
                {
                    var consumeResult = consumer.Consume(default(CancellationToken));

                    if(consumeResult?.Message?.Value is Message result)
                    {
                        WriteLine($"Data Received - {result}");
                    }
                }
            });

            consumer.Close();
        }
    }
}

As you can see, this class contains a method "ConsumeAsync" which creates a consumer object and starts consuming from Kafka using "consumer.consume()" method. This is not an async method, so if you call this method in a main thread, it will be blocked until you receive a message here which is why I have consumed it in a different task. Once a message is received, we are printing it here.

Now the main program looks like,

namespace KafkaConsumer;

public class Program
{
    public static async Task Main(string[] args)
    {
        var consumer = new Consumer<Message>();
        await consumer.ConsumeAsync();
    }
}

When you run both the producer and consumer now,

The original source is available in my github repo

How to choose a collection in C#

Sugumar Prabhakar — Thu, 14 Oct 2021 22:10:11 +0000

When I started my career with .Net Framework 4.0, I always felt challenging when it comes to choose the right collection for my development. I always end up with either Generic Dictionary or List.

But this article from the Microsoft can be a life-saver [https://docs.microsoft.com/en-us/dotnet/standard/collections/?WT.mc_id=-blog-scottha]