DEV Community: Bazen

The Memphis .NET SDK: My Journey from Contributor to Maintainer

Bazen — Sat, 20 May 2023 06:06:56 +0000

Hello! My name is Bazen, and I'm a software engineer who enjoys working on exciting projects and, of course, coffee. One of my favorite frameworks is .NET and the C# programming language. I want to share my journey from contributing to maintaining the .NET SDK for Memphis. Memphis is an open-source project, and in this article, I will start by explaining what Memphis is, my contribution journey, and, last but not least, the different ways others can contribute to Memphis and other open-source projects.

What is Memphis?

Memphis is a next-generation message broker. It is a simple, robust, and durable cloud-native message broker. It provides cost-effective, fast, and reliable development of modern queue-based systems. Memphis scales exceptionally well with large volumes of streamed and enriched data. It is designed with development experience in mind and requires significantly less development time for data-oriented developers and data engineers than other solutions. Memphis is transparent and provides visibility on the activities taking place, which is helpful when troubleshooting. Memphis is feature rich and contains several features not mentioned here. I recommend checking it out on GitHub and The official Memphis website to learn more about it.

My Contribution

I was working on a company project involving working with high-volume data. I was researching different event streaming platforms such as Kafka, RabbitMq, Amazon SQS, and more to find a suitable solution, and that is how I stumbled across Memphis. At first glance, I was impressed by the observability of the platform, which motivated me to set it up locally and try it out. I played around with the Go SDK and quite liked the platform, which inspired me to learn more about it on GitHub.

At the time, Memphis already had SDKs in multiple languages, such as Go, TypeScript, and Python. However, the Memphis .NET SDK was in an early stage. Recognizing the platform's potential, I took it upon myself to reach out to the Memphis team and asked how I could contribute to the .NET SDK. Not long after that, I received a response from Yaniv Ben Hemo, the Co-Founder and CEO at Memphis. He was very welcoming and pointed me to the issues I could work on.

I worked on one of the issues and made my first pull request to the Memphis .NET SDK, and I was pleased by the encouraging feedback from the Memphis community.

Since then, I have contributed actively to the .NET SDK of the Memphis project. In addition, I am an active participant in the community, helping to resolve issues, guiding new contributors, and ensuring that the .NET SDK for Memphis remains up-to-date and fully functional.

What I Learned

As a Software Engineer, contributing to open-source projects and building in public is very important, and it provides the opportunity to learn about a new subject in detail. For example, my understanding of message brokers and their technical complexity was limited before Memphis. Some interesting things I learned from Memphis are schema verse and dead letter station.

Schema verse is a schema store and schema management layer on top of the Memphis broker. It's a neat way to specify and enforce the data structure, and this is useful to validate input data and increase its quality. The schema can be defined from the Memphis UI and using code, and one schema can be applied to multiple stations. The supported formats are ProtoBuf, Avro, JSON, and GraphQL. You can learn more about schema verse at Memphis.

The other concept is the dead letter station(DLS), which is where faulty messages end up. A message can be considered faulty for many reasons, for example, if it doesn’t satisfy the enforced schema, if a consumer fails to process it properly, and so on. In these scenarios, instead of dropping these messages, Memphis stores them in the dead letter station(DLS) along with a log that explains why the message is in DLS. This is helpful when troubleshooting and increases the application’s resilience. You can learn more about DLS at Memphis.

Memphis has many features that make a developer's life easier and an exciting project to learn from and contribute to.

How To Contribute?

A good starting point for anyone who would like to contribute is to

Check out the different Memphis repositories and look for issues labeled “good first issue.”
You can then apply the proposed changes
Create the pull request.

These pull requisites are likely to comprise small changes, and it is unlikely for there to be multiple back and forth; at the same time, this will give you a good feel of the contribution workflow. To learn more about the contribution guide, please check out the How to Contribute? guide.

Memphis Community

The Memphis Community is composed of very kind and skillful people. In addition to GitHub, they are very active on social media platforms such as LinkedIn, Discord, and Twitter. Since day-1, people like Yaniv Ben Hemo, Idan Asulin, and Avital Trifsik have been very welcoming and understanding. They were kind enough to send me their cool swag pack as a token of their appreciation. Moreover, they helped me understand and learn different technical aspects of the event processing platform. At first, the thought of contributing could seem daunting, but speaking from experience, I assure you that you will be relieved of your worries with a community like Memphis.

Conclusion

Memphis owes much of its success to the tireless efforts of its contributors, and I am proud to be a part of this very knowledgeable and vibrant community of developers. I have been privileged to make valuable contributions to this next-generation event-processing platform. As Memphis continues to grow and evolve, there is no doubt that I will continue to contribute to the project. So, if you're a developer looking to get involved in a dynamic and exciting project, please join the Memphis community and contribute to help shape the future of event processing.

Using npm Workspaces With ReactJS and .NET

Bazen — Fri, 07 Jan 2022 16:41:40 +0000

This article explains how to leverage the existing .NET SPA template to work with npm workspaces. explanation on what npm workspaces are is not addressed in this article. for any one who is new to npm workspaces its recommended to check npm official documentation. npm workspaces is a nice way of organizing code but at the time being in order to use workspaces in .NET some customization are required, which will be explained in the following sections of this article.

Content

Creating .NET project
Setting up SPA
Modifying .NET Project
Configuring Publish Profiles

Creating .NET project

.NET project with react can be created by running the following command

dotnet new react -n SampleApp

Setting up SPA

Once the SampleApp project is created by default it will contain ClientApp directory, which is where the SPA(in this case React App) resides. as the default SPA template doesn't fit the required scenario delete everything inside ClientApp directory.

To setup workspaces open terminal inside the ClientApp directory first run the following command

npm init -y

Running this command will generate package.json file which will contain the workspace information. for this example I want to create four workspaces named

@clientapp/table : contains React app that displays information in tabular format
@clientapp/card : contains React app that displays information in card
@clientapp/config : contains shared configurations(eg. tsconfig)
@clientapp/core : contains shared components and functionalities

The ClientApp will now look like the following

Now package.json inside ClientApp have to be updated to configure the workspaces as shown bellow:

{
  "name": "@clientapp/root",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "start:table": "npm run start -w @clientapp/table",
    "start:card": "npm run start -w @clientapp/card",
    "build:table": "npm run build -w @clientapp/table",
    "build:card": "npm run build -w @clientapp/card"
  },
  "workspaces": [
    "workspaces/*/**"
  ]
}

To create the two applications inside ClientApp\workspaces\apps directory run the following commands consecutively

@clientapp/table

npx create-react-app table --template typescript

updated name field inside ClientApp\workspaces\apps\table\package.json to

"name": "@clientapp/table"

@clientapp/card

npx create-react-app card --template typescript

updated name field inside ClientApp\workspaces\apps\card\package.json to

"name": "@clientapp/card"

changes for both apps

By default in both @clientapp/table & @clientapp/card we will not be able to use the typescript libraries from other workspaces. in order to support typescript I will use craco instead of react-scripts. the changes in this section must be applied in both @clientapp/table & @clientapp/card.

Install craco as dev dependency

 npm install craco --save-dev

Create file name craco.config.js

const path = require("path");
const { getLoader, loaderByName } = require("craco");

const packages = [];
/**
 * add the typescript workspaces this project is dependent up on
 */
packages.push(path.join(__dirname, "../../libs/core"));

module.exports = {
  webpack: {
    configure: (webpackConfig,  { env, paths }) => {
      /**
       * Overriding the output directory of build to fit with default configuration of .NET wrapper
       */
      paths.appBuild = webpackConfig.output.path = path.resolve('../../../build');
      const { isFound, match } = getLoader(webpackConfig, loaderByName("babel-loader"));
      if (isFound) {
        const include = Array.isArray(match.loader.include)
          ? match.loader.include
          : [match.loader.include];

        match.loader.include = include.concat(packages);
      }
      return webpackConfig;
    },
  },
};

Update the scrpts section inside package.json of both @clientapp/table & @clientapp/card as shown below:

{
  ...
  "scripts": {
    "start": "craco start",
    "build": "craco build",
    "test": "craco test",
    "eject": "craco eject"
  },
  ...
}

@clientapp/core

From ClientApp\workspaces\libs open terminal and run the following command

npx create-react-app core --template typescript

updated name field inside ClientApp\workspaces\apps\card\package.json to

"name": "@clientapp/core"

Since @clientapp/core is not dependent on another workspace there is no need to configure craco.

From all application delete node_modules directory

To install the @clientapp/core workspace into @clientapp/table & @clientapp/card run the following commands from ClientApp directory

npm install @clientapp/core -w @clientapp/table

npm install @clientapp/core -w @clientapp/card

To install the dependency packages run npm install from ClientApp directory.

At this point the SPA workspace configuration is completed & can be tested by running either of the following commands

npm run start:table

npm run start:card

Modifying .NET Project

For development update Configure method inside Startup.cs by replacing

spa.UseReactDevelopmentServer(npmScript: "start");

spa.UseReactDevelopmentServer(npmScript: "run start:table");

To start @clientapp/table. & replace it by

spa.UseReactDevelopmentServer(npmScript: "run start:card");

To start @clientapp/card

For publish update SampleApp.csproj by replacing

<Target Name="PublishRunWebpack" AfterTargets="ComputeFilesToPublish">
  <!-- As part of publishing, ensure the JS resources are freshly built in production mode -->
  <Exec WorkingDirectory="$(SpaRoot)" Command="npm install" />
  <Exec WorkingDirectory="$(SpaRoot)" Command="npm run build" />
  <!-- Include the newly-built files in the publish output -->
  <ItemGroup>
    <DistFiles Include="$(SpaRoot)build\**" />
    <ResolvedFileToPublish Include="@(DistFiles->'%(FullPath)')" Exclude="@(ResolvedFileToPublish)">
      <RelativePath>%(DistFiles.Identity)</RelativePath>
      <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
      <ExcludeFromSingleFile>true</ExcludeFromSingleFile>
    </ResolvedFileToPublish>
  </ItemGroup>
</Target>

<Target Name="PublishRunWebpack" AfterTargets="ComputeFilesToPublish">
  <Error Condition="'$(SpaBuildScript)' == ''" Text="Spa build script is not specified." />
  <!-- As part of publishing, ensure the JS resources are freshly built in production mode -->
  <Exec WorkingDirectory="$(SpaRoot)" Command="npm install" />
  <Exec WorkingDirectory="$(SpaRoot)" Command="$(SpaBuildScript)" />
  <!-- Include the newly-built files in the publish output -->
  <ItemGroup>
    <DistFiles Include="$(SpaRoot)build\**" />
    <ResolvedFileToPublish Include="@(DistFiles->'%(FullPath)')" Exclude="@(ResolvedFileToPublish)">
      <RelativePath>%(DistFiles.Identity)</RelativePath>
      <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
      <ExcludeFromSingleFile>true</ExcludeFromSingleFile>
    </ResolvedFileToPublish>
  </ItemGroup>
</Target>

Add Two publish profiles one for @clientapp/card & one for @clientapp/table

CardAppProfile.pubxml

<?xml version="1.0" encoding="utf-8"?>
<!--
https://go.microsoft.com/fwlink/?LinkID=208121. 
-->
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <DeleteExistingFiles>False</DeleteExistingFiles>
    <ExcludeApp_Data>False</ExcludeApp_Data>
    <LaunchSiteAfterPublish>True</LaunchSiteAfterPublish>
    <LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <PublishProvider>FileSystem</PublishProvider>
    <PublishUrl>bin\Release\net5.0\publish\</PublishUrl>
    <WebPublishMethod>FileSystem</WebPublishMethod>
    <SpaBuildScript>npm run build:card</SpaBuildScript>
  </PropertyGroup>
</Project>

TableAppProfile.pubxml

<?xml version="1.0" encoding="utf-8"?>
<!--
https://go.microsoft.com/fwlink/?LinkID=208121. 
-->
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <DeleteExistingFiles>False</DeleteExistingFiles>
    <ExcludeApp_Data>False</ExcludeApp_Data>
    <LaunchSiteAfterPublish>True</LaunchSiteAfterPublish>
    <LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
    <LastUsedPlatform>Any CPU</LastUsedPlatform>
    <PublishProvider>FileSystem</PublishProvider>
    <PublishUrl>bin\Release\net5.0\publish\</PublishUrl>
    <WebPublishMethod>FileSystem</WebPublishMethod>
    <SpaBuildScript>npm run build:table</SpaBuildScript>
  </PropertyGroup>
</Project>

After adding these publish profiles, @cilentapp/table can be published by running the following command for

dotnet pubilsh /p:PublishProfile="Properties\PublishProfiles\TableAppProfile.pubxml"

And for @cilentapp/card

dotnet pubilsh /p:PublishProfile="Properties\PublishProfiles\CardAppProfile.pubxml"

That is one way of using npm workspaces with .NET, full source code can be found on GitHub.

Thanks for reading, Happy coding!

Understanding Docker, Docker Compose & Swarm

Bazen — Tue, 22 Jun 2021 11:39:57 +0000

The intention behind this article is to provide high level understanding of what they are, how to use them and explain the commonly used syntax & semantics. the article might be a bit long but the reason I preferred not make it into series is because there is no deep dive into each and every aspect of container solution.

Points of discussion in this article

Docker
- Docker CLI
- Container
  - Interacting with containers
- Image
  - Creating an Image
- Persistent data
  - Bind Mount
  - Volumes
- Network
Docker Compose
Swarm

If particular section is not area of your interest feel free to jump between sections. having said that lets get to it.

1. Docker

Docker at its core is a platform used to run and deploy applications in container, in addition to this docker also comprises a lot of features which you will come to see as we go further. The reason for running applications in a container is to facilitates the entire life cycle of software development and management(which includes development, testing, deployment, update & maintenance), which docker does in a very elegant way.

Docker CLI

The commands used in this article are from Windows machine & if you are using Mac/Linux the commands might be different(you might need to run the commands as sudo).Initially the docker CLI commands were structured as following

docker [OPTIONS] COMMAND

And as the product become bigger and bigger they also introduced Management commands to make the CLI more organized. The command structure for Management commands is as following

docker [MANAGEMENT COMMAND] [SUB COMMAND] [SUB COMMAND OPTIONS]

For every command there is --help option which provides all the necessary information's about the command and its corresponding options.

Example-1

docker container --help

will have the following output.

The --help command is quite useful because with the extensive feature sets of docker it can be difficult to memorize all the commands.

Container

Are running instances of docker image & it is possible to run multiple container from the same docker image. Images are explained in the Image section for now lets take a look at a simple container command.

Example-2

 docker container run --name nginx-server --publish 80:80 nginx

or alternatively

 docker container run --name nginx-server --publish 80:80 --detach nginx

After running this command if there is an existing image of nginx docker will run a container from that image however if there is no image of nginx docker will pull the image first and then start the container. lets take a look at the command

container : Management command used to execute all container related commands.
--name : sets the name of the container.

It is not possible for multiple containers to have the same name because the container name will be used for DNS resolution(which is explained in detail in the Network section ).

--publish : Maps the containers running port into host port. in this case we check this by visiting http://localhost:80 from browser. The port mapping pattern is as following

... --publish [HOST PORT]:[CONTAINER PORT] ...

It is not possible to use the same host port to run multiple containers.

--detach : starts the container in background and prints only the container id
nginx : name of an image from which we want to run our container

Another example

Example-3

docker container run --name db-server --env MYSQL_ROOT_PASSWORD=my-sample-password --detach mysql

Some images require environment variable to run(such as database server) and the --env option is used to pass environment variable from CLI.

Other common container commands

docker container ls : lists all running containers
docker container ls -a : lists all containers that are running and that are not
docker container stop nginx-server : stops the container named nginx-server
docker container start nginx-server : starts the container named nginx-server
docker container logs nginx-server : prints the logs from running container named nginx-server
docker container rm nginx-server : removes container named nginx-server

As you can see above the commands are self explanatory and strait forward. The intention of showing those basic commands is to demonstrate & be familiar with the commands and usages. There is no need to memorize any of this as you can get those and many more sub-commands of the container command which are not listed above by simply using the --help command.

Interacting with containers

There are two commands we can use to interact with containers.

1. Using `run` command

When starting a container using the run command it is possible to add -it option and have the container start in interactive mode.

Example-4

docker container run -it --name nginx-server-it --publish 8080:80 nginx bash

Will give us the bash console inside the nginx-server-it container to interact with. The output is shown below

With this method once we exit from the interactive window (in this case bash terminal) the container will be stopped automatically.

2. Using `exec` command

exec command is used to interact with running container.

Example-5

docker container exec -it nginx-server-it bash

This command will have the same result to the previous(... run -it) command. the difference is once we exit from the interactive window the container will keep running.

Image

Are simply the binaries, libraries and source code that make up the application. and images are used as a template to construct a container. images are constructed in a layered fashion and Union file system is used under the hood to achieve this. Union file system is not covered in this article but for those of you who wants to know more about it Deep Dive into Docker Internals — Union Filesystem is good article. the reason for creating images in layered fashion is mainly to:

Speedup the startup time
Avoid storing duplicate files

Creating an Image

Images can be created either from an existing image or from Dockerfile. lets take a look at each method.

Creating image from existing image The tag command can be used to create from another image as shown below

Example-6

 docker image tag nginx sample/my-nginx

The output will look like

Though there is newly created image the Id attribute for both images is the same. That's because the layers used to construct both images are identical therefor the layers are shared between the two images(This has to do with the Union file system).

Creating image from Dockerfile

Dockerfile comprises all the commands that user can use from the CLI. here are quick overview of the commands and description

FROM : sets the base image for subsequent instruction

WORKDIR : Defines the working directory of the image
ENV : Environment variables expected to start a container similar to the mysql example above.
RUN : To run commands inside the container
COPY : Copies files from source to target directory
EXPOSE : Exposed container port. This port can then be mapped to the host port using the --publish command as shown in the example above.
CMD : Final command that runs when starting container.

These are the most commonly used commands, for extensive list of all available commands official documentation is a great place.

It is recommended to put the commands for frequently changing items(such as source code) at the bottom and commands for items that doesn't change as often at the top.as the order of command entries define the way the image is constructed.
In every Dockerfile there must be FROM & CMD commands. and to clarify if there is no CMD command in a Dockerfile that means CMD is defined in the base image & it is inherited via the FROM command.

Example-7

create file named Dockerfile with the following content

FROM nginx
WORKDIR /usr/share/nginx/html
COPY . .

create index.html file with the following content

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<body>
    <h1>Docker is awsome!</h1>
</body>
</html>

from the root directory run the following command

docker image build --tag sample/my-nginx-2 .

Great! you created your own image which you can check by running the ls command. to make things more interesting lets run the image created and see the the changes made.

 docker container run --rm -p 80:80 sample/my-nginx-2

When visiting localhost from browser you should get something like this

Persistent data

Containers by design are short lived and immutable(redeploy not change). this design puts the need for persistent data as separate concern, and there are two solution provided by Docker for persistent data namely Bind Mount and Volumes.

Bind Mount

Is a way of mounting host machines file system in to a container. while this is not often used in production it is better way for local testing and development. lets take a look at example by running the following command inside root directory of Example-7

docker container run --rm --publish 8080:80 --volume ${pwd}:/usr/share/nginx/html/ nginx

While the container is running if the index.html file is updated in the host machine those changes will be reflected from the running container as well. for demonstration update the index.html as shown below

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<body>
    <h1>[Content Updated] Docker is awsome!</h1>
</body>
</html>

after these update you can see the changes from browser as shown below

TIP for referencing current directory when Bind Mapping from CLI
${pwd} - PowerShell
$(pwd) - Mac/Linux
%cd% - Command Prompt

Volumes

Similar to Bind Mount volumes are another means of persisting data. The distinction from that of bind mount is that volumes are managed by Docker Bind Mount on the other hand are reliant on specifying the destination path on the host machine.

The first place to specify volume is in the Dockerfile by using the VOLUME command. the Dockerfile of databases (such as postgres, mysql) are setup to include volume.

Example-8

docker container run --rm --env POSTGRES_PASSWORD=test-password postgres

Will start postgres container, then we can check for the volumes using the following command

docker volume ls

To make things even more interesting we can inspect the volume as well

docker volume inspect [VOLUME_NAME]

The output will look like this

As you might have noticed the problem with this is the name of the volume is not readable. this can be fixed by naming the volume using the following command

Example-9

docker container run --rm --env POSTGRES_PASSWORD=test-password --volume my-db-volume:/var/lib/postgresql/data postgres

if we now inspect the volume it will look as follows

Which is more readable.

NOTE the syntax difference between Bind Mount & Named Volume Is that the host path of Bind Mount is preceded by /(on Mac/Linux) or by //(on windows).

Network

Docker network is basically a way of making the running containers accessible from the outside world(it could be the internet or local network) and enables containers communicate to each other. of-course there is a lot more to it but I will not go deep in this article.
Creating virtual network is as easy as running create command as shown below

Example-10

docker network create my-network

We can run multiple containers in side this container by running the following command

docker container run --rm --network my-network --name nginx1 nginx

docker container run --rm --network my-network --name nginx2 nginx

NOTE Running containers in the same virtual network can communicate to each other by using the container name as a domain name.

Since both containers (nginx1, nginx2) are on the same virtual network it is possible to use the container name as domain name and to communicate between containers. from Example-5 we can use the exec command to open bash & run Curl and see if we can reach the other container

docker container exec -it nginx1 bash

curl nginx2

The result is shown below

NOTE Bridge(or Docker0) is the default docker virtual network & it doesn't support the usage of container name as a domain name even if the container are running in the same virtual network.

The next section is about Docker compose, but before that cheers on reading this far, most of the important fundamentals are covered. as you will come to see docker compose is only a more organized way of doing the previously mentioned points. lets get to it...

Docker Compose

Is a simplified way of managing multiple containers at the same time. docker-compose works fine both in production and local development. docker-compose comprises yml configuration file and CLI tool to manage containers using the configuration file.
configuration file syntax overview

version:[] 
services: 
   service_name1:
         .
         .
   service_name2:
         . 
         . 
   service_nameN:
volumes:
   named_volume1:
   named_volume2:
        .
        .
   named_volumeN:
networks:
   network1:
   netwprk2:
      .
      .
   networkN:

For better understanding of the general syntax defined above lets take a look at an example. for this example create file called docker-compose.yml in the same directory as Example-7

Example-11

version: "3.4"                                       # Specifies the docker compose file format
services: 
    nginx-simple:                                    # Service name
        image: nginx                                 # Image name & docker checks if image is available locally if not image will be pulled first 
        ports:                                       # Expose ports to make services accssible from outside. the port mapping is read as map host's port 80 to container port 80 
          - 80:80                                    # host_port:container_port
    nginx-bindmount:
        image: nginx
        volumes:                                     # defines the volume for the container   
          - .:/usr/share/nginx/html                  # BindMapping of the files in the current directory(in host) into containers path (which is /usr/share/nginx/html). the "." indicates current host directory  
        ports: 
          - 5005:80
        depends_on:                                  # defines the priority of the services. in this case nginx-bindmount is dependent on nginx-simple therefor nginx-simple has to start first
          - nginx-simple                             # list of services that nginx-bindmount is dependent up on
    nginx-custom-image:                  
        build:                                       # Tells docker we want to build the image from local Dockerfile
          context: .                                 # Defines the directory of the Dockerfile in the host machine. "." indicates current host dircetory 
          dockerfile: Dockerfile                     # name of the Dockerfile
        ports: 
          - 5006:80
    nginx-named-volume:                              
        image: mysql
        volumes: 
          - db-volume:/var/lib/mysql                 # defines named volume called db-volume for nginx-named-volume service to store persistent data
        environment:                                 # passes list of environment variables into nginx-named-volume service
          - MYSQL_ROOT_PASSWORD=my-password          # enviroment variable variable_key=value, in this case MYSQL_ROOT_PASSWORD is variable key & my-password is the corresponding value   

volumes:                                             # defines list the named volumes
   db-volume:                                        # entrie of named volume

I hope the comments on the compose file are explanatory enough for each entries. now by running

docker compose up

Will start all containers & setup all volumes, networks. yeah! it is that simple. & to clean up everything(container, volume & networks) use the following command

docker compose down

docker compose also contains start & stop commands to start and stop services with out cleaning up everything.

NOTE: docker compose doesn't attach the services with docker's default virtual network(called bridge) rather it creates it own network and runs the services in that network. that way the services in the same docker-compose configuration file can communicate with each other by using the Service name as domain name.

Swarm

From the previous section we have seen how to setup and manage image as well as containers and finally using docker compose to manage multiple containers and images. while that all good it doesn't address a couple of problem that surface when using docker, such as

Easily scaling up and down
Restarting container if it fails
Deployment with zero down time
Secure communication between containers
Manage which container runs where
Storing sensitive information & making it accessible only to specific container

That's where Swarm mode comes to the picture and address the problems stated above. Swarm is docker built in clustering solution.

Manager Nodes: are nodes with proper authorization and configuration to manage swarm.
Workers Nodes: are nodes with responsibility to carry out a task and no authority to administer the swarm.
Raft Database: Is a database the resides on every manager nodes, It comprises manager node's configuration that guarantee's their authority.

By default Swarm mode is not enabled. to enable swarm mode run the following command.

docker swarm init

There is multiple ways to setup the swarm, but for demonstrating some of the basics I will use Play with Docker.

Example-12

Create multiple instances in the docker lab and to initialize the swarm run the following command

docker swarm init --advertise-addr 192.168.0.8

The IP Address has to be public IP so other instances can join to that swarm(in my case it happens to be 192.168.0.8).

Once swarm is created other instances can join using the command generated after creating the swarm

docker swarm join --token [TOKEN] [IP_ADDRESS]


docker swarm join --token SWMTKN-1-64cek9j9nfuysu5v76ydgqxn16te1tfwcyv1beqks023dh1yz2-0y6lj91cx03iwbu53btmsmdm3 192.168.0.8:2377

After joining the two instances into the swarm we can see the nodes in the swarm

docker node ls

With in the swarm we can now create multiple services with out worrying about the container distribution because that will be handled by the swarm.

docker service create --publish 80:80 --replicas 3 nginx

This will create three containers of nginx and distribute the containers into the instances with in the swarm. to see active services list

docker service ls

To see more information's of particular service run

docker service ps [SERVICE_NAME | SERVICE_ID]

As shown above there is 3 replicas of nginx and they are running on different instances. by updating the number of replicas we can easily scale up and scale down. even more intersting if a container fails in the swarm for some reason it will then be recreated automatically.

And if we see the list of containers again

This is only the tip of the iceberg, there is a lot more you can do with swarm(such as using secrets using docker-compose with swarm) which could not be explained in one article. but I hope this is enough to give you an idea of what Swarm is & its importance.

Thanks for reading. Cheers!

Serving DocFX Static Site From .NET With Authentication

Bazen — Wed, 16 Jun 2021 12:58:56 +0000

DocFX is a static site generator from source code files and markdown. Its primary use is for documentation. However, it is essential to mention that it is flexible and is applicable for many different purposes. Some use it for blogging and profile sites. It is really up to you to define what to use it for. For more information on DocFX you can visit their official website.

Why Serve It From .NET?

There are different advantages to serving the static site generated by DocFX from .NET since it allows adding different functionalities on top of the static site. One practical advantage is adding authentication to access the static site. The reason for adding the authentication could be that you have internal documentation within your company and want it to be accessible only by authorized persons. This scenario is only to demonstrate a practical use case, & the use case can be different depending on your scenario.

Pre-request

IdentityServer4: For authentication
DocFX: For generating static site
Add DocFX to environment variable for smooth build process

I used IdentityServer4 for authentication in this article, however this implementation works fine with other methods of authentications as well.

Setting up IdentityServer4

Create IdentityServer4 project



dotnet new is4inmem -n Identity

The templates for IdentityServer4 projects is available on IdentityServer's Github repo. After creating the identity project we don't need to make further change.

Setting up Static site server

Create ASP.NET Api project



dotnet new webapi -n StaticSiteServer

Add reference to Microsoft.AspNetCore.Authentication.OpenIdConnect NuGet package.

There are a couple of changes to be made here and there when setting up the authentication. Not to go off topic, I will not cover that in this article. However, the source code can be found in my Github repository.

Setting up static site

In the same directory as StaticSiteServer create static site project:



docfx init -q -o Docs

Setting up the build process

Let's update StaticSiteServer.csproj to define the root directory of the DocFX file system:



<PropertyGroup>
      .
      .
      .
    <DocFXRoot>Docs\</DocFXRoot>
</PropertyGroup>

Next, let's update the StaticSiteServer.csproj to generate static site on both during build & publish.



<ItemGroup>
  <Content Remove="$(DocFXRoot)**" />
  <None Remove="$(DocFXRoot)**" />
  <None Include="$(DocFXRoot)**" Exclude="$(DocFXRoot)_site\**" />
</ItemGroup>
<Target Name="DebugEnsureDocFXEnv" BeforeTargets="Build" Condition=" '$(Configuration)' == 'Debug' And !Exists('$(DocFXRoot)node_modules') ">
  <Exec Command="docfx --version" ContinueOnError="true">
    <Output TaskParameter="ExitCode" PropertyName="ErrorCode"/>
  </Exec>
  <Error Condition="'$(ErrorCode)' != '0'" Text="DocFX is required to build and run this project. To continue, please install DocFX from https://github.com/dotnet/docfx/releases, and then restart your command prompt or IDE." />
  <Message Importance="high" Text="Generate static sites from Markdown and code files. This may take several minutes..." />
  <Exec WorkingDirectory="$(DocFXRoot)" Command="docfx build"/>
</Target>
<Target Name="PublishRunWebpack" AfterTargets="ComputeFilesToPublish">
  <Exec WorkingDirectory="$(DocFXRoot)" Command="docfx build"/>
  <ItemGroup>
    <DistFiles Include="$(DocFXRoot)_site\**" />
    <ResolvedFileToPublish Include="@(DistFiles->'%(FullPath)')" Exclude="@(ResolvedFileToPublish)">
      <RelativePath>%(DistFiles.Identity)</RelativePath>
      <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
      <ExcludeFromSingleFile>true</ExcludeFromSingleFile>
    </ResolvedFileToPublish>
  </ItemGroup>
</Target>

Now, with these changes, the build will generate the static site. However, the StaticSiteSeserved web API is not serving the static site. To server the static site, let's update the ConfigureServices method in the Startup.cs file. Moreover, we will add authorization:



public void ConfigureServices(IServiceCollection services) {
    .
    .
    .
    services.AddAuthorization(options => {
        options.FallbackPolicy = new
        AuthorizationPolicyBuilder()
            .RequireAuthenticatedUser()
            .Build();
    });
    .
    .
    .
}

Again in the Startup.cs file let's update the Configure method to serve static files.



public void Configure(IApplicationBuilder app, IWebHostEnvironment env) {
    .
    .
    .
    app.UseFileServer(new FileServerOptions {
        FileProvider = new
        PhysicalFileProvider(
            Path.Combine(env.ContentRootPath,
                "Docs", "_site")),
    });
    .
    .
    .
}

NOTE app.UseFileServer* must be added after app.UseAuthentication() and app.UseAuthorization().

That's it. When accessing the StaticSiteServer, the user is redirected for authentication. After authenticating, the user can access the static site, as shown below:

1. StaticSiteServer

2. Authentication page

3. Documentation static site

4. Documentation static site

Complete source code can be found on Github

Thanks for reading, cheers!

Conditional Rendering In React With No Library

Bazen — Thu, 29 Apr 2021 18:27:13 +0000

If you ever worked with react it is likely that at some point you implemented some conditional rendering. such as the following

export const Component:FC=()=>{
   const [isLoading, setIsLoading] = useState<boolean>();
   //
   // Some logic
   //
   return (
     <Fragment>
      { isLoading? <LoadingComponent/> : <Content/>}
     </Fragment/>
   )
}

While this is fine.it is limited to two values(true & false) which is not always the case. moreover it complicates the component code as our component grows. lets see at other way of implementing this.

interface ConditionalRendererProps{
  activeState: string
  mapping: { [key: string]:React.ReactNode}
}

export const ConditionalRenderer:FC<ConditionalRendererProps>=(props)=>{
  return(
    <Fragment>
      { props.mapping[props.activeState] }
    </Fragment>
  )
}

The idea behind this implementation is to use ConditionalRenderer as a wrapper component and pass a mapping and activeState as a props.

mapping as its name indicates it contains state to component mapping. meaning what component corresponds to a given state.
activeState is selected state from the mapping props

While this could seem over complicating the implementation. it is actually more flexible that the first implementation and makes our code more cleaner. well how? okay to answer that lets take a look at another more common scenario where we need conditional rendering. when we have task that needs sometime to complete(e.g. when making API request). in this scenario there is more than two states for now to keep things simple lets agree we have four states namely initial, processing, complete & error & we want to render different components depending on the active state

export enum StateTypes{
  Init='INIT',
  Error='ERROR',
  Success='SUCCESS',
  Processing='PROCESSING',
}

The StateTypes enum defines all possible states, next lets define generic wrapper component for components containing asynchronous actions

interface StateMachineWrapperProps{
    asyncTask: Function
    component: React.ComponentType<any>
}
export const StateMachineWrapper:FC<StateMachineWrapperProps> =(props) =>{
  const machine = useAsync<string>(props.asyncTask)
  return (
    <ConditionalRenderer
        activeState={machine.currentState}
        mapping={{
          [StateTypes.Init]:<Fragment/>,
          [StateTypes.Processing]: <p>{machine.message}</p>,
          [StateTypes.Error]: <p>{machine.message}</p>,
          [StateTypes.Success]: <props.component {...machine.payload}/>
        }}
    />
  )
}

StateMachineWrapper renders components that comprises async actions.
It is highly likely that we have multiple components that communicate to an external API or perform some other async task and for every component we can use the StateMachineWrapper component and separate the side effect from our core component. lets see the usage...

function longRunningTask(){
      return new Promise((resolve, reject)=>{
          setTimeout(()=>{
              resolve({data: 'Async task completed'})
          },1000)
      });
}

To keep things simple the long running task does nothing practical but it is easy to modify the implementation according to your use case. finally lets take a look at the core component...

interface ContentProps{
  data: string
}
export const Content:FC<ContentProps>=(props)=>{
  return (
    <div>
      <h3>Content</h3>
      <p>{ props.data }</p>
    </div>
  )
}
export const ContentWrapper:FC =()=>{
  return (
      <StateMachineWrapper 
        asyncTask={longRunningTask}
        component={Content}/>
  )
}

the Content component is pure component and is decoupled with the side effect(or longRunningTask). as you can see the initial effort pays off finally because the components that mimic the StateMachineWrapper are pure components and the concerns are separated. this is one use case to demonstrate the ease of implementing conditional rendering in such a way.

Github gist can be found here

Thank you for reading, cheers!

IdentityServer4,ASP.NET Identity for Authentication & Authorization with ReactJS client

Bazen — Sat, 20 Feb 2021 21:05:03 +0000

Very recently I had a scenario where I have to implement two web apps. for both apps I had to implement authentication and authorization. I spent quite sometime researching on what is the better way to implement this & I hope this saves your time for anyone interested. anyway after doing my research I decided to use the following technologies...

1) IdentityServer4 :- for authentication & authorization
2) ASP.NET Identity :- User information storage
3) .NET API :- API protected by IdentityServer4
4) React :- React & Typescript Client App that is going to consume API

Lets start coding...

Step 1: Identity Server

We can either create empty project and do all the work by our self or we can use the one of the IdentityServer4 templates.to keep things simple I'm going to use one of the templates by running the following commands.

dotnet new -i identityserver4.templates

To see the installed templates run

dotnet new -l

There are couple of template options to choose from.in this case we want to use ASP.NET Identity as the user data storage so we will run this command:

dotnet new is4aspid

Now that we have our project ready its time to edit some of the code. find Config.cs file which contains the identity configuration. the first this we are going to do is to add the Api resource:

public static IEnumerable < ApiResource > ApiResources => new ApiResource[] {
    new ApiResource("sample.api", "Sample Api") {
        Scopes = new string[] {
            ProtectedApiScopes.ScannerApiScope.Name
        }
    }
};

Next step is to add our Client SPA app to the clients list:

public static IEnumerable < Client > Clients => new Client[] 
{
    new Client {
        ClientId = "sample.spa",
            ClientName = "Sample SPA",
            AllowedGrantTypes = GrantTypes.Implicit,
            AllowAccessTokensViaBrowser = true,
            RequireConsent = false,
            AccessTokenLifetime = 120,
            RedirectUris = {
                "http://localhost:3000"
            },
            PostLogoutRedirectUris = {
                "http://localhost:3000"
            },
            AllowedCorsOrigins = {
                "http://localhost:3000"
            },
            AllowedScopes = {
                IdentityServerConstants.StandardScopes.OpenId,
                IdentityServerConstants.StandardScopes.Profile,
            },
    };
}

This is enough change for the identity configuration, now we have to add our configuration to IServiceCollection in StartUp.cs as follows:

services.AddIdentityServer()
    .AddDeveloperSigningCredential()
    .AddInMemoryPersistedGrants()             
    .AddInMemoryIdentityResources(Config.GetIdentityResources())
    .AddInMemoryApiResources(Config.GetApiResources())
    .AddInMemoryClients(Config.GetClients())
    .AddAspNetIdentity<ApplicationUser>();

In production its better to store your identity server configuration in the database, but lets keep things simple for now.we are done configuring the identity server, next step is to create & setup the .NET api project.project can be created by running the following command:

dotnet new webapi

After creating the project we have to add reference to the IdentityServer4.AccessTokenValidation package.we can then add configuration in the StartUp.cs file by adding the following code

 services.AddAuthentication("Bearer")
     .AddIdentityServerAuthentication(options => {
         options.Authority = "http://localhost:5000"; //Identity Server Uri
         options.RequireHttpsMetadata = false;
         options.ApiName = "sample.api";
     });

That's it now we can simply guard any end point by adding [Authorize] attribute on top of it:

[Authorize]
[HttpGet]
public IActionResult Get()
{
   return Ok(new { Message= "You are authenticated" })
}

Next step is to create & setup our react app by running the following commands:

npx create-react-app sample-spa --template typescript

yarn create react-app my-app --template typescript

After creating the react app. we will add the best oidc library called oidc-react which is the best oidc library I have seen by far.

npm install oidc-react

Create oidc-config.ts file in the src directory of sample-spa project and add the following configuration

export const customOidcConfig: AuthProviderProps = {
  clientId: 'sample.spa',
  automaticSilentRenew: true,
  redirectUri: 'http://localhost:3000/',
  responseType: 'token id_token',
  scope:"openid profile",
  authority: 'http://localhost:5000',
  onBeforeSignIn:()=>{
     /**
      * This method gets executed before signin redirecting begins
      * it can be used to store Uri
      */
  }
  onSignIn:async (user: User | null)=>{
      console.log('PATH',window.location.pathname)
        if(user){
            console.group('[ LOGIN: SUCCESS ]', user);
         };
         window.location.hash = '';
    }
    else{
       console.error('[ LOGIN: ERRNO ]')
    }
  },
  onSignOut:async (options?: AuthProviderSignOutProps) =>{
      console.log('[ SignOutOpts ]', options);
  }
};

Next step is to initiate login using the configuration above. find App.tsx file and update it using the following code

<AuthProvider {...customOidcConfig}>
   <AuthCheck/>          
</AuthProvider>

This will automatically initiate the login process. we can also check if the user is logged in by using useAuth hook.

export const AuthCheck: FC =() =>{
   const authState = useAuth();
   return (
     <Fragment>
        { 
          authState && 
          authState.userData && 
          <p>Authenticated</p> 
        }
        {
          (!authState ||
          !authState.userData) && 
          <p>Not Authenticated</p>
        }
     </Fragment>
   )
}

Now we are done. I hope you enjoyed this.
Thanks for reading

Happy Coding!!!