DEV Community: Ali MSELMI

Create a Windows Server virtual machine with PowerShell

Ali MSELMI — Sun, 07 Jun 2020 07:42:22 +0000

In today blog post we are going to create and use an Azure VM with powershell, This blog post also shows you the steps to:

Connect to the virtual machine with a remote client.
Clean up resources.

Install and import AzureRM

In order to use and execute the below commands we need to install and import the AzureRM powershell module if we don't already have it, if you already have it you can skip this section.

PS: You need elevated privileges to install modules from PowerShell Gallery


Install-Module -Name AzureRM -AllowClobber

By default, the PowerShell gallery is not configured as a trusted repository for PowerShellGet. The first time you use PSGallery, the following message is displayed:


Untrusted repository

You are installing the modules from an untrusted repository. If you trust this repository, change
its InstallationPolicy value by running the Set-PSRepository cmdlet.

Are you sure you want to install the modules from 'PSGallery'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend [?] Help (default is "N"):

So, don't worry about this message.

Once installed, you can import AzureRM by running the following command:


Import-Module AzureRM

Finnaly, to finish with this section we need to connect to Azure account, simply execute this command and you we be prompted:


# Connect to Azure with an interactive dialog for sign-in
Connect-AzureRmAccount

Create a resource group

A resource group is a logical container in which Azure resources are deployed and managed.
From your SDK, run the following code block to create a resource group:


# Create variables to store the location and resource group names.
$location = "francecentral"
$ResourceGroupName = "resource-group-1"

New-AzureRmResourceGroup -Name $ResourceGroupName -Location $location

Create storage resources

Create a storage account to store the output of boot diagnostics.


# Create variables to store the storage account name and the storage account SKU information
$StorageAccountName = "msstorage01"
$SkuName = "Standard_LRS"

# Create a new storage account
$StorageAccount = New-AzureRMStorageAccount `
  -Location $location `
  -ResourceGroupName $ResourceGroupName `
  -Type $SkuName `
  -Name $StorageAccountName

Set-AzureRmCurrentStorageAccount `
  -StorageAccountName $storageAccountName `
  -ResourceGroupName $resourceGroupName

Create networking resources

Create a VNet (virtual network), subnet, and a public IP address. Creating these Azure resources help us providing the network connectivity to our VM.


# Create a the subnet configuration
$subnetConfig = New-AzureRmVirtualNetworkSubnetConfig `
  -Name subNet-1 `
  -AddressPrefix 192.168.1.0/24

Create the virtual network


$vnet = New-AzureRmVirtualNetwork `
  -ResourceGroupName $ResourceGroupName `
  -Location $location `
  -Name vNet-1 `
  -AddressPrefix 192.168.0.0/16 `
  -Subnet $subnetConfig

Create a public IP address and specify a DNS name


$publicip = New-AzureRmPublicIpAddress `
  -ResourceGroupName $ResourceGroupName `
  -Location $location `
  -AllocationMethod Static `
  -IdleTimeoutInMinutes 4 `
  -Name "mypublicdns$(Get-Random)"

Create a NSG (network security group ) and a NSG rule

The NSG secures our VM by using inbound and outbound rules.

Now, we need to create an inbound rule for port 3389 to allow incoming RDP (Remote Desktop) connections and an inbound rule for port 80 to let our VM receive the incoming web traffic.

Create an inbound NSG rule for the 3389 port


# Create an inbound NSG rule for the 3389 port
# This rule will allow us to connect to the VM via an RDP connection
$nsgrdprule = New-AzureRmNetworkSecurityRuleConfig `
  -Name nsg-rdp-rule `
  -Protocol Tcp `
  -Direction Inbound `
  -Priority 1000 `
  -SourceAddressPrefix * `
  -SourcePortRange * `
  -DestinationAddressPrefix * `
  -DestinationPortRange 3389 `
  -Access Allow

Create an inbound network security group rule for port 80


# This rule will allow the VM to receive incoming web connections via the port 80
$nsgwebrule = New-AzureRmNetworkSecurityRuleConfig `
  -Name nsg-inbound-www-rule `
  -Protocol Tcp `
  -Direction Inbound `
  -Priority 1001 `
  -SourceAddressPrefix * `
  -SourcePortRange * `
  -DestinationAddressPrefix * `
  -DestinationPortRange 80 `
  -Access Allow

Create the NSG (network security group)


# This will wrap up previously created rules (nsg-web-rule and nsg-rdp-rule) within an NSG
$nsg = New-AzureRmNetworkSecurityGroup `
  -ResourceGroupName $ResourceGroupName `
  -Location $location `
  -Name nsg-1 `
  -SecurityRules $nsgrdprule,$nsgwebrule 

# This command will create a VNC (virtual network card) and associate it with public IP address and NSG
$nic = New-AzureRmNetworkInterface `
  -Name nic-1 `
  -ResourceGroupName $ResourceGroupName `
  -Location $location `
  -SubnetId $vnet.Subnets[0].Id `
  -PublicIpAddressId $publicip.Id `
  -NetworkSecurityGroupId $nsg.Id

# Define a credential object to store the username and password for the VM
$UserName='ali.mselmi'
$Password='P@ssword123'| ConvertTo-SecureString -Force -AsPlainText
$Credential=New-Object PSCredential($UserName,$Password)

Create the VM configuration object


$VmName = "VirtualMachinelatest"
$VmSize = "Standard_A1"
$VirtualMachine = New-AzureRmVMConfig `
  -VMName $VmName `
  -VMSize $VmSize

$VirtualMachine = Set-AzureRmVMOperatingSystem `
  -VM $VirtualMachine `
  -Windows `
  -ComputerName "MainComputer" `
  -Credential $Credential -ProvisionVMAgent

$VirtualMachine = Set-AzureRmVMSourceImage `
  -VM $VirtualMachine `
  -PublisherName "MicrosoftWindowsServer" `
  -Offer "WindowsServer" `
  -Skus "2016-Datacenter" `
  -Version "latest"

# Sets the operating system disk properties on a VM.
$VirtualMachine = Set-AzureRmVMOSDisk `
  -VM $VirtualMachine `
  -CreateOption FromImage | `
  Set-AzureRmVMBootDiagnostics -ResourceGroupName $ResourceGroupName `
  -StorageAccountName $StorageAccountName -Enable |`
  Add-AzureRmVMNetworkInterface -Id $nic.Id

Create the VM

Finnaly we can create the VM deployment configuration.



# Create the VM.
New-AzureRmVM `
  -ResourceGroupName $ResourceGroupName `
  -Location $location `
  -VM $VirtualMachine

We can check the VM creation via the Azure portal:

Connect to the VM

To do a remote access into the VM that we created in the previous step, we need its public IP address also previously set up.

To do that, we just need to run the following command and get the public IP address:


Get-AzureRmPublicIpAddress `
  -ResourceGroupName $ResourceGroupName | Select IpAddress

Now, we can do a remote desktop access after creating a Remote Desktop session with the VM, just you need replace the IP address with the publicIPAddress of your VM.
When prompted, you can login by using your credentials used when creating the VM.


mstsc /v publicIpAddress

Connect to the Azure VM with an RDP, SSH or Bastion

You can connect to the VM via RDP, SSH or Bastion, you just need to click to the created VM via Azure portal and then click connect.

Delete the VM

We can use the following command to remove the resource group that contains the VM and its related resources:


Remove-AzureRmResourceGroup `
  -Name $ResourceGroupName

Final words...

The advantage of scripting the entire VM creation process is that often we don't need to create a single VM, but to create several, creating a script for the process gives us the flexibility to customize the VMs creation at large-scale.

Structured logging with Serilog and Seq and ElasticSearch under Docker

Ali MSELMI — Mon, 20 Apr 2020 18:26:23 +0000

This blog post demonstrates Structured Logging with Serilog, Seq, ElasticSearch and kibana under Docker containers.
This post is a follow up on the beginner post I wrote on Serilog.

If you are a newbie on Serilog then I will recommend that you give a quick read to official Serilog page and this blog post.

Example application

The complete below example shows serilog structured logging in a containerized web application with microservices style using docker, with events sent to the Seq and elastic search as well as a date-stamped rolling log file with the use of available below sinks:

Serilog.Sinks.File
Serilog.Sinks.Http
Serilog.Sinks.Seq
Serilog.Sinks.ElasticSearch

Create a new empty solution

To start, we need create a new solution using Visual studio or your favorite IDE.

Create a new web Application project

Create new Web application project like below:

Install the core Serilog package and the File, Seq and ElasticSearch sinks

In Visual Studio, open the Package Manager Console and type:

Install-Package Autofac
Install-Package Autofac.Extensions.DependencyInjection
Install-Package Microsoft.AspNetCore.Mvc.NewtonsoftJson
Install-Package Microsoft.Extensions.DependencyInjection
Install-Package Microsoft.Extensions.Hosting
Install-Package Microsoft.Extensions.Logging
Install-Package Microsoft.Extensions.Options
Install-Package Newtonsoft.Json
Install-Package Serilog
Install-Package Serilog.AspNetCore
Install-Package Serilog.Extensions.Hosting
Install-Package Serilog.Extensions.Logging
Install-Package Serilog.Settings.AppSettings
Install-Package Serilog.Settings.Configuration
Install-Package Serilog.Sinks.ElasticSearch
Install-Package Serilog.Sinks.File
Install-Package Serilog.Sinks.Http
Install-Package Serilog.Sinks.Seq

Add the following code to Program.cs

Create GetConfigurationmethod



private static IConfiguration GetConfiguration()
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.Development.json", optional: true)
        .AddEnvironmentVariables();

    return builder.Build();
}

Create CreateSerilogLoggermethod



private static ILogger CreateSerilogLogger(IConfiguration configuration)
{
    var seqServerUrl = configuration["Serilog:SeqServerUrl"];
    return new LoggerConfiguration()
        .MinimumLevel.Verbose()
        .Enrich.WithProperty("ApplicationContext", AppName)
        .Enrich.FromLogContext()
        .WriteTo.File("catalog.api.log.txt", rollingInterval: RollingInterval.Day)
        .WriteTo.Elasticsearch().WriteTo.Elasticsearch(ConfigureElasticSink(configuration, "Development"))
        .WriteTo.Seq(string.IsNullOrWhiteSpace(seqServerUrl) ? "http://seq" : seqServerUrl)
        .CreateLogger();
}

Create ConfigureElasticSinkmethod



private static ElasticsearchSinkOptions ConfigureElasticSink(IConfiguration configuration, string environment)
{
    return new ElasticsearchSinkOptions(new Uri(configuration["Serilog:ElasticConfiguration"]))
    {
        BufferCleanPayload = (failingEvent, statuscode, exception) =>
        {
            dynamic e = JObject.Parse(failingEvent);
            return JsonConvert.SerializeObject(new Dictionary<string, object>()
                {
                    { "@timestamp",e["@timestamp"]},
                    { "level","Error"},
                    { "message","Error: "+e.message},
                    { "messageTemplate",e.messageTemplate},
                    { "failingStatusCode", statuscode},
                    { "failingException", exception}
                });
        },
        MinimumLogEventLevel = LogEventLevel.Verbose,
        AutoRegisterTemplate = true,
        AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
        CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage: true),
        IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}",
        EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | 
                               EmitEventFailureHandling.WriteToFailureSink |
                               EmitEventFailureHandling.RaiseCallback
    };
}

Create GetDefinedPorthelper method



private static int GetDefinedPort(IConfiguration config)
{
    var port = config.GetValue("PORT", 80);
    return port;
}

Create CreateHostBuildermethod



private static IWebHost CreateHostBuilder(IConfiguration configuration, string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .UseConfiguration(configuration)
        .CaptureStartupErrors(false)
        .ConfigureKestrel(options =>
        {
            var httpPort = GetDefinedPort(configuration);
            options.Listen(IPAddress.Any, httpPort, listenOptions =>
            {
                listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
            });
        })
        .UseStartup<Startup>()
        .UseContentRoot(Directory.GetCurrentDirectory())
        .UseSerilog()
        .Build();

Create Main method



public static readonly string Namespace = typeof(Program).Namespace;
public static readonly string AppName = Namespace.Substring(Namespace.LastIndexOf('.', Namespace.LastIndexOf('.') - 1) + 1);

public static int Main(string[] args)
{
    var configuration = GetConfiguration();
    Log.Logger = CreateSerilogLogger(configuration);
    try
    {
        Log.Information("Configuring web host ({ApplicationContext})", AppName);
        var host = CreateHostBuilder(configuration, args);
        Log.Information("Starting web host ({ApplicationContext})", AppName);
        host.Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Program terminated unexpectedly ({ApplicationContext})!", AppName);
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}

Add the following code to Startup.cs

Inject IConfigurationinto the constructure



public Startup(IConfiguration configuration)
{
    Configuration = configuration;
}
public IConfiguration Configuration { get; }

Create ConfigureServices method



public IServiceProvider ConfigureServices(IServiceCollection services)
{
    services.AddCustomMVC(Configuration);

    var container = new ContainerBuilder();
    container.Populate(services);

    return new AutofacServiceProvider(container.Build());
}

Create Configuremethod



public void Configure(IApplicationBuilder app, IWebHostEnvironment env, ILoggerFactory loggerFactory)
{
    var pathBase = Configuration["PATH_BASE"];

    if (!string.IsNullOrEmpty(pathBase))
    {
        loggerFactory.CreateLogger<Startup>().LogDebug("Using PATH BASE '{pathBase}'", pathBase);
        app.UsePathBase(pathBase);
    }

    app.UseCors("CorsPolicy");
    app.UseRouting();
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapDefaultControllerRoute();
        endpoints.MapControllers();
    });
}

Create the CustomExtensionMethodsextension method



public static class CustomExtensionMethods
{
    public static IServiceCollection AddCustomMVC(this IServiceCollection services, IConfiguration configuration)
    {
        services.AddControllers(options =>
        {

        }).AddNewtonsoftJson();

        services.AddCors(options =>
        {
            options.AddPolicy("CorsPolicy",
                builder => builder
                .SetIsOriginAllowed((host) => true)
                .AllowAnyMethod()
                .AllowAnyHeader()
                .AllowCredentials());
        });

        return services;
    }
}

Add appsettings.json / appsettings.Development.json

It's very important to use the container name as ElasticSearchurl and not http://localhost:9200



{
  "Serilog": {
    "SeqServerUrl": "http://seq",
    "LogstashgUrl": "http://locahost:8080",
    "ElasticConfiguration": "http://elasticsearch:9200",
    "MinimumLevel": {
          "Default": "Debug",
          "Override": {
            "Microsoft": "Debug",
            "CatalogAPI": "Debug",
            "MYSHOP": "Debug",
            "System": "Warning"
          }
        }
      }
  }

Add Docker support to our solution

Add Docker compose project to our solution

Add Dockerfile to the project



FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/core/sdk:aspnet:3.1-buster AS build
WORKDIR /src

COPY "Services/Catalog/Catalog.API/CatalogAPI.csproj" "Services/Catalog/Catalog.API/CatalogAPI.csproj"
COPY "docker-compose.dcproj" "docker-compose.dcproj"

COPY "NuGet.config" "NuGet.config"

RUN dotnet restore "MyShop.sln"

COPY . .
WORKDIR /src/Services/Catalog/Catalog.API
RUN dotnet publish --no-restore -c Release -o /app

FROM build AS publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "CatalogAPI.dll"]

Add docker-compose.yml to the docker project



version: '3.4'

services:

  seq:
    image: datalust/seq:latest

  catalog.api:
    image: ${REGISTRY:-myshop}/catalogapi
    build:
      context: .
      dockerfile: Services/Catalog/CatalogAPI/Dockerfile

networks:
  elastic:
    driver: bridge

volumes:
  elasticsearchdata:
    external: true

Add Docker docker-compose.overrides.yml to the docker project



version: '3.4'

services:

  seq:
    environment:
      - ACCEPT_EULA=Y
    ports:
      - "5340:80"

  elasticsearch:
    build:
      context: elk/elasticsearch/
    volumes:
      - ./elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"

  logstash:
    build:
      context: elk/logstash/
    volumes:
      - ./elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./elk/logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "8080:8080"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: elk/kibana/
    volumes:
      - ./elk/kibana/config/:/usr/share/kibana/config:ro
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  catalog.api:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://0.0.0.0:80
      - PORT=80
      - PATH_BASE=/catalog-api
    ports:
      - "4201:80"

Create Configuration for ElasticSearchand Kibana

Create the folder structure

Create "elk" folder in the root folder of the solution with following structure

Create elasticsearch configuration

Under elk/elasticsearch create new Dockerfileand new config folder like bellow:

Add the following code to the Dockerfile



# https://github.com/elastic/elasticsearch-docker
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2

# Add your elasticsearch plugins setup here
# Example: RUN elasticsearch-plugin install analysis-icu

Create new elasticsearch.yml under the config folder





  
  
  Default Elasticsearch configuration from elasticsearch-docker.


  
  
  from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml


#

cluster.name: "docker-cluster"

network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP

# set to 1 to allow single node clusters

# Details: https://github.com/elastic/elasticsearch/pull/17288

discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks

## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html

#

discovery.type: single-node

Create kibana configuration

Under elk/kibana create new Dockerfileand new config folder like bellow:

Add the following code to the Dockerfile



# https://github.com/elastic/kibana-docker

FROM docker.elastic.co/kibana/kibana-oss:7.6.2

# Add your kibana plugins setup here

# Example: RUN kibana-plugin install <name|url>

Create new kibana.yml under the config folder

## Default Kibana configuration from kibana-docker.

## from https://github.com/elastic/kibana-docker/blob/master/build/kibana/config/kibana.yml

#

server.name: kibana

server.host: "0"

elasticsearch.hosts: "http://elasticsearch:9200"

Conclusion

We have seen how to configure .net core web application to log in Elasticsearch, Kibanaand Sequsing Serilog. This solution offers many advantages and I invite you to discover the range of possibilities on the Elastic and Seq websites.
The sources are here feel free to clone the solution and play with it.

Logging with Serilog

Ali MSELMI — Mon, 20 Apr 2020 09:41:00 +0000

This blog post illustrates structured logging and the goal of using such an approach compared to the traditional way of logging (log4net, NLog),
Another follow-up article demonstrating the use of Serilog with Elasticsearch acting as a log repository and running Docker is here, if you are a Serilog novice, this blog article is for you.

Why Serilog and structured logging?

Structured logging and Serilog are different. We can store the object as it is or finely-tuned to record just the right information for optimal diagnostics depending in our use case and what we need to log, then deconstructing the object using the Serilog framework on your behalf does not need to record all of the object’s fields of interest, get them into desired string format, and then store them in log instructions.

Serilog includes many integrations that cover many cases such as:

Logging to console
Logging to file
Logging to data base
Logging to Message queues
Logging to logstash and elasticsearch
Serilog also controls the way events are captured and enriched

The use of structured logging also has other advantages:

We can store our logs in databases like ElasticSearch which offer quite impressive search functions on this type of data.
We can build KPIs around stored logs using ElasticSearch
We can create a dashboards with nice widgets, pie, graphics.
We can store our logs directly in Seq, then we can benefit from the ability to search and filter the logs using SQL like language
Great documentation and community
C# based configuration

How it works ?

Install the core Serilog package and the File sinks
In Visual Studio, open the Package Manager Console and type:

In the Main method we need to initialize Serilog and for this purpose we need to grab the configuration and configure Serilog inside the CreateSerilogLogger.

var configuration = GetConfiguration();
Log.Logger = CreateSerilogLogger(configuration);

Grab the the configuration from appsettings.json and build the Configuration object:

private static IConfiguration GetConfiguration()
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.Development.json", optional: true)
        .AddEnvironmentVariables();
   return builder.Build();
        }

Configure Serilog like below:

private static ILogger CreateSerilogLogger(IConfiguration configuration)
{
     return new LoggerConfiguration()
          .MinimumLevel.Verbose()
          .Enrich.WithProperty("ApplicationContext", AppName)
          .Enrich.FromLogContext()
          .WriteTo.File("log.txt", rollingInterval: RollingInterval.Day)
          .CreateLogger();
}

Add some configuration in the appsettings.json:

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Debug",
      "Override": {
        "Microsoft": "Debug",
        "CatalogAPI": "Debug",
        "MYSHOP": "Debug",
        "System": "Warning"
      }
    }
  }
}

And finally the body of the Main method should look like this:

public static int Main(string[] args)
{
    var configuration = GetConfiguration();

    Log.Logger = CreateSerilogLogger(configuration);

    try
    {
       Log.Information("Configuring web host ({ApplicationContext})", AppName);
       var host = CreateHostBuilder(configuration, args);

       Log.Information("Starting web host ({ApplicationContext})", AppName);
       host.Run();

       return 0;
    }
    catch (Exception ex)
    {
       Log.Fatal(ex, "Program terminated unexpectedly ({ApplicationContext})!", AppName);
       return 1;
    }
    finally
    {
       Log.CloseAndFlush();
    }
}

Structured logging

The main benefit from Serilog is the structured logging feature:

The traditionnal way of logging messages into any support (file, database) was always in a unstructured way

But with Serilog we can store my information in a structured way.

In the below example we tell Serilog to replace {FirstName} with Ali and {LastName} with MSELMI, this produces not only a nice log message (as any logging framework does), but also tells Serilog to structure the data in key / value pairs, FirstName/LastName keys and theirs values.

The outcome of this feature is to store structured data into nosql databases like mongodb or Elasticsearch and we can filter, export data, create dashboards and even create useful KPIs for example.

Log.Information("Hello {FirstName} {LasttName}", "Ali", "MSELMI");

As you can see, Seq captured the events coming from Serilog in structured manner.

And now we can filter the logs using SQL-like language using structured data.

Conclusion

Log4net has served as default logging choice for the past two decades, but Serilog offered a new way of logging (Structured logging, enrichments) and C# based configuration.

In most cases, I would recommend you to use Serilog logging framework and to continue to use log4net only when you are working on legacy system