DEV Community: Kim CH

.NET: Integration Test for uploading csv/excel file via HttpClient

Kim CH — Fri, 07 Feb 2025 07:41:20 +0000

Suppose that we have 2 APIs to import the list of codes - one uses a CSV template, and the other uses an Excel template. Please note that saving data isn't included; I'm just showing how to test posting CSV/Excel files.

The following packages have been used

CsvHelper for parsing the CSV file
ExcelMapper for working with Excel file
Shouldly for asserting the results

[ApiController]
public class ImportProductsController : ControllerBase
{

    [HttpPost("/products/import-csv")]
    [ProducesResponseType<List<ImportProductModel>>(StatusCodes.Status200OK)]
    public IActionResult DoImportCsv(IFormFile file)
    {
        using var stream = file.OpenReadStream();
        using var reader = new StreamReader(stream);
        using var csv = new CsvReader(reader, CultureInfo.InvariantCulture);
        csv.Context.RegisterClassMap<ClassMapImportProductModel>();
        var records = csv.GetRecords<ImportProductModel>().ToList();
        return Ok(records);
    }

    [HttpPost("/products/import-excel")]
    [ProducesResponseType<List<ImportProductModel>>(StatusCodes.Status200OK)]
    public IActionResult DoImportExcel([FromForm] IFormFile file)
    {
        var records = new ExcelMapper(file.OpenReadStream()) { HeaderRow = true }.Fetch<ImportProductModel>();
        return Ok(records.ToList());
    }
}

Test import CSV

[Fact]
public async Task TestImportByCsv()
{
    // ==================================
    // Step-01: Generate the CSV stream
    // ==================================
    var sb = new StringBuilder().AppendLine("Product Code,Product Name");

    var numberOfProducts = 10;
    for (int i = 0; i < numberOfProducts; i++)
    {
        sb.AppendLine(string.Join(",", [
            this._faker.Random.AlphaNumeric(10).ToUpper(),
                this._faker.Commerce.ProductName()
            ]));
    }

    var csvBytes = Encoding.UTF8.GetBytes(sb.ToString());
    var csvStream = new MemoryStream(csvBytes);

    // ==================================
    // Step-02: Prepare the FormData
    // ==================================
    using var formData = new MultipartFormDataContent();
    formData.Add(new StreamContent(csvStream), name: "file", fileName: "abc.csv");

    // ==================================
    // Step-03: Execute the POST request
    // ==================================
    var httpClient = this._factory.CreateClient();
    var response = await httpClient.PostAsync("/products/import-csv", formData);
    response.StatusCode.ShouldBe(HttpStatusCode.OK);

    var products = await this.ParseResponse<List<ImportProductModel>>(response);

    products.ShouldNotBeEmpty();
    products.Count.ShouldBe(numberOfProducts);
    products.ForEach(p =>
    {
        p.Code.ShouldNotBeNullOrWhiteSpace();
        p.Name.ShouldNotBeNullOrWhiteSpace();
    });
}

Test import Excel file

[Fact]
public async Task TestImportViaExcel()
{
    // ==================================
    // Step-01: Generate the Excel stream
    // ==================================
    var workBook = new XSSFWorkbook();
    var sheet = workBook.CreateSheet("Sheet1");
    var headerRow = sheet.CreateRow(0);
    headerRow.CreateCell(0).SetCellValue("Product Code");
    headerRow.CreateCell(1).SetCellValue("Product Name");

    const int numberOfProducts = 10;
    for (int i = 1; i <= numberOfProducts; i++)
    {
        var dataRow = sheet.CreateRow(i);
        dataRow.CreateCell(0).SetCellValue(this._faker.Random.AlphaNumeric(10).ToUpper());
        dataRow.CreateCell(1).SetCellValue(this._faker.Commerce.ProductName());
    }

    var excelStream = new MemoryStream();
    workBook.Write(excelStream, leaveOpen: true);

    // ==================================
    // Step-02: Prepare the FormData
    // ==================================
    using var formData = new MultipartFormDataContent();
    formData.Add(content: new StreamContent(excelStream),
                name: "file",
                fileName: "abc.excel");

    // ==================================
    // Step-03: Execute the POST request
    // ==================================
    var httpClient = this._factory.CreateClient();
    var response = await httpClient.PostAsync("/products/import-excel", formData);
    response.StatusCode.Should().Be(HttpStatusCode.OK);

    var products = await this.ParseResponse<List<ImportProductModel>>(response);
    products.ShouldNotBeEmpty();
    products.Count.ShouldBe(numberOfProducts);
    products.ForEach(p =>
    {
        p.Code.ShouldNotBeNullOrWhiteSpace();
        p.Name.ShouldNotBeNullOrWhiteSpace();
    });
}

Thanks for reading!!!
The full example can be found at my repo on GitHub - setup-dotnet-test-projects

Simplify observability .NET with OpenTelemetry Collector

Kim CH — Tue, 25 Apr 2023 05:26:45 +0000

[Updated 20 July, 2024]

I've updated to .NET 8 & changes regarding otel collector

Overview

The Observability concept become the standard of almost system in recently. It helps team to troubleshoot what's happening inside the system. There are 3 pillars of Observability - Traces, Metrics and Logs.

OpenTelemetry is the most straightforward way to collect Traces and Metrics. Then export to Zipkin or Jaeger for tracings; Prometheus
Serilog is also the most straightforward way to collect logs and then export to Seq or ElasticSearch

Because of the various exporting ways, we have to consider one of these options when implementing

Support all type of exporting then toggle via settings. For example, only export to Zipkin if it's enabled
Or, just only export to Zipkin or Jaeger

Only one answer for the concerns

❓ Concerns

Is there any way that just only one export for multiple consumers? Or,
Is there any way that just only one export but change consumer without changing the code?

🌟 Only one answer

And luckily, there is an elegant way to accomplish them - that is OpenTelemetry Collector (OTEL Collector in short)

Objectives

Usability: Reasonable default configuration, supports popular protocols, runs and collects out of the box.

Performance: Highly stable and performant under varying loads and configurations.

Observability: An exemplar of an observable service.

Extensibility: Customizable without touching the core code.

Unification: Single codebase, deployable as an agent or collector with support for traces, metrics, and logs (future).

An image more than thousand words

💻 Let our hand dirty

👉 The below steps are just the showcase of using OTEL Collector within .NET 8. The full implementation can be found at - .NET with OpenTelemetry Collector

👉 In which, we'll export the telemetry signals from application to OTEL Collector then they'll be exported to - Zipkin or Jaeger for tracings; Prometheus; and Loki for logs

Nuget packages Directory.Packages.props




<Project>
    <PropertyGroup>
        <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    </PropertyGroup>
    <ItemGroup>
        <PackageVersion Include="Grpc.AspNetCore" Version="2.64.0" />
        <PackageVersion Include="Microsoft.AspNetCore.OpenApi" Version="8.0.7" />
        <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.0" />
        <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.9.0" />
        <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.9.0" />
        <PackageVersion Include="Serilog" Version="4.0.0" />
        <PackageVersion Include="Serilog.AspNetCore" Version="8.0.1" />
        <PackageVersion Include="Serilog.Sinks.OpenTelemetry" Version="3.0.0" />
        <PackageVersion Include="Serilog.Sinks.PeriodicBatching" Version="5.0.0" />
        <PackageVersion Include="Swashbuckle.AspNetCore" Version="6.6.2" />
    </ItemGroup>
</Project>

Register OpenTelemetry, typically from `Program.cs`




var builder = WebApplication.CreateBuilder(args);

builder.Host.AddSerilog();

builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(observabilityOptions.ServiceName))
    .AddMetrics(observabilityOptions)
    .AddTracing(observabilityOptions);

Configure Tracings



private static OpenTelemetryBuilder AddTracing(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
    if (!observabilityOptions.EnabledTracing) return builder;

    builder.WithTracing(tracing =>
    {
        tracing
            .SetErrorStatusOnException()
            .SetSampler(new AlwaysOnSampler())
            .AddAspNetCoreInstrumentation(options =>
            {
                options.RecordException = true;
            });

        /* Add more instrument here: MassTransit, NgSql ... */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        tracing
            .AddOtlpExporter(_ =>
            {
                _.Endpoint = observabilityOptions.CollectorUri;
                _.ExportProcessorType = ExportProcessorType.Batch;
                _.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
            });
    });

    return builder;
}

Configure for Metrics



private static OpenTelemetryBuilder AddMetrics(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
    builder.WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation();

        /* Add more instrument here */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        metrics
            .AddOtlpExporter(_ =>
            {
                _.Endpoint = observabilityOptions.CollectorUri;
                _.ExportProcessorType = ExportProcessorType.Batch;
                _.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
            });
    });

    return builder;
}

Configure Logs



private static WebApplicationBuilder AddSerilog(this WebApplicationBuilder builder, ObservabilityOptions observabilityOptions)
{
    var services = builder.Services;
    var configuration = builder.Configuration;

    services.AddSerilog((sp, serilog) =>
    {
        serilog
            .ReadFrom.Configuration(configuration, new ConfigurationReaderOptions
            {
                SectionName = $"{nameof(ObservabilityOptions)}:{nameof(Serilog)}"
            })
            .ReadFrom.Services(sp)
            .Enrich.FromLogContext()
            .Enrich.WithProperty("ApplicationName", observabilityOptions.ServiceName)
            .WriteTo.Console();

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        serilog
            .WriteTo.OpenTelemetry(c =>
            {
                c.Endpoint = observabilityOptions.CollectorUrl;
                c.Protocol = OtlpProtocol.Grpc;
                c.IncludedData = IncludedData.TraceIdField | IncludedData.SpanIdField | IncludedData.SourceContextAttribute;
                c.ResourceAttributes = new Dictionary<string, object>
                                                {
                                                    {"service.name", observabilityOptions.ServiceName},
                                                    {"index", 10},
                                                    {"flag", true},
                                                    {"value", 3.14}
                                                };
            });
    });

    return builder;
}

The interesting here

1️⃣ - Refer to docker-compose.observability.yaml

2️⃣ - Refer to otel-collector.yaml to configure OTEL Collector



receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s

  resource:
    attributes:
      - action: insert
        key: loki.resource.labels
        value: service.name, service.namespace
      - action: insert
        key: loki.format
        value: json

exporters:
  debug:
    verbosity: normal

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: test-space
    resource_to_telemetry_conversion:
      enabled: true
    enable_open_metrics: true

  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"
    format: proto

  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    default_labels_enabled:
      exporter: false
      job: true


extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [debug, otlp/jaeger, zipkin]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, prometheus]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, loki]

Let's 👍 OTEL Collector and take fully implementation at - .NET with OpenTelemetry Collector

Cheers!!! 🍻

Move docker-desktop-data distro out of System Drive

Kim CH — Wed, 03 Jun 2020 10:05:43 +0000

By default, Docker Desktop for Window will create 2 distros below

docker-desktop
docker-desktop-data

If we access the path %LOCALAPPDATA%/Docker/wsl; we can see 2 folders; and inside it is vhdx file. For detail,

data/ext4.vhdx which is consumed by docker-desktop-data
distro/ext4.vhdx which is consumed by docker-desktop

In which, docker-desktop-data is used to store images and so on. Therefore, its size will be increased in the future, consequently, our System Drive will be out of space. Below are step-by-step to move docker-desktop-data out of System Drive, for example, E:\docker-desktop\data.

Step 1: Stop Docker

Step 2: Export, unregister then import distro

1- Shutdown all WSL distros

wsl --shutdown

2- Export docker-desktop-data to tar file

wsl --export docker-desktop-data E:\docker-desktop\docker-desktop-data.tar

3- Unregister current docker-desktop-data distro

wsl --unregister docker-desktop-data

4- Import docker-desktop-data distro from tar file

wsl --import docker-desktop-data E:\docker-desktop\data E:\docker-desktop\docker-desktop-data.tar --version 2

Notes:

In this step, we may meet the error of cannot create a specific network. Just re-run the import command.

Step 3: Start Docker

Resources

Introducing the Docker Desktop WSL 2 Backend: we can get more information about docker-desktop, docker-desktop-data and even the new architecture of Docker Desktop
Moving Docker wsl files
Where are Docker volumes located when running Windows and WSL?

Setup Development Environment for Angular 7 with Windows Subsystem for Linux (WSL)

Kim CH — Wed, 20 May 2020 20:08:05 +0000

[Update on 03 June 2020]

Since Microsoft just released the update which is windows 10 build 2004, and I just convert to WSL 2 without any issue.

Here are the steps to upgrade

Download and install WSL2 Linux Kernel
Update to WSL 2

[End of Update]

Recently, I was assigned to Angular 7 project, the first thing I thought was how to set up my development environment without impact to existing work, they could be the version of NodeJs, npm packages (i.e. node-gyp), python and so on... Luckily, we can run Linux natively on Windows 10, it's known as Window Subsystem for Linux - WSL in short.

Note:

Although Microsoft has WSL 2, but in this post, I'm going to install WSL 1 since I'm not running Windows Insider version.
This post assumes that we has already Window Terminal installed.

Let's start with opening Window Terminal as Administrator

Enable Windows Subsystem for Linux

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

Install Ubuntu-18.04-LTS distro

The simplest way is we can install via Window Store but we may face the problem of drive system got full since it's installed at %LOCALAPPDATA%\Ubuntu-18.04-LTS\rootfs as default. In order to avoid this problem, we may want to install on Non-System drive, for instance, E:\_wsl\Ubuntu-18.04-LTS drive. Then follow the below steps

Create the folder E:\_wsl\Ubuntu-18.04-LTS
Download Ubuntu-18.04-LTS distro, the downloaded file may be CanonicalGroupLimited.Ubuntu18.04onWindows_1804.2018.817.0_x64__79rhkp1fndgsc.Appx, then copy to E:\_wsl\Ubuntu-18.04-LTS
Change to E:\_wsl\Ubuntu-18.04-LTS

   cd E:\_wsl\Ubuntu-18.04-LTS

Change to zip file

  move .\CanonicalGroupLimited.Ubuntu18.04onWindows_1804.2018.817.0_x64__79rhkp1fndgsc.Appx .\Ubuntu-18.04-LTS.zip

Extract Ubuntu-18.04-LTS.zip

  Expand-Archive .\Ubuntu-18.04-LTS.zip

Then we execute ubuntu1804.exe inside the folder Ubuntu-18.04-LTS, in this step, we have to provide username and password. If there is no issue, we can see as below

  .\ubuntu1804.exe

Install node version manager (aka nvm)

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash

export NVM_DIR="$HOME/.nvm"

[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"

[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"

nvm --version

Install nodejs

In this step, I'm going to install nodejs v11.15.0

nvm install 11.15.0

nvm use 11.15.0

Install node-gyp

sudo apt-get update -y

sudo apt-get install -y node-gyp

Getting started example Angular 7

Install angular cli

npm install -g @angular/cli@7.1.4

Create project via angular cli

cd /mnt/d/

mkdir learning-angular7

cd learning-anguar7

ng new my-app

Launch my-app

cd my-app

ng serve --open

DEV Community: Kim CH

.NET: Integration Test for uploading csv/excel file via HttpClient

Test import CSV

Test import Excel file

Simplify observability .NET with OpenTelemetry Collector

[Updated 20 July, 2024]

Overview

Only one answer for the concerns

💻 Let our hand dirty

Nuget packages Directory.Packages.props

Register OpenTelemetry, typically from Program.cs

Configure Tracings

Configure for Metrics

Configure Logs

The interesting here

Move docker-desktop-data distro out of System Drive

Resources

Setup Development Environment for Angular 7 with Windows Subsystem for Linux (WSL)

Enable Windows Subsystem for Linux

Install Ubuntu-18.04-LTS distro

Install node version manager (aka nvm)

Install nodejs

Install node-gyp

Getting started example Angular 7

Install angular cli

Create project via angular cli

Launch my-app

Resources

Register OpenTelemetry, typically from `Program.cs`