DEV Community: Douglas Minnaar

GraphQL For .NET

Douglas Minnaar — Sun, 03 Jul 2022 00:41:00 +0000

I have recently created a project on Github that demonstrates how to build a GraphQL API through the use of a number of examples. I created the project in order to help serve as a guide for .NET developers that are new to GraphQL and would like to learn how to get started. As a final step to this project, I also demonstrate how to create the required infrastructure in Microsoft Azure to host the GraphQL API.

Key Takeaways

Project Overview
Learn how to build a .NET 6 GraphQL API from scratch
Learn how to define GraphQL queries, and mutations
Learn how to provide validation and error handling
Add EntityFramework support both in-memory and SQLite database
Add support for query projection, filtering, sorting, and paging
Provision Azure App Service using Azure CLI, Azure, Powershell, and Azure Bicep

Project Overview

This project explores how to build and deploy a GraphQL API in a number of easy to follow examples.
Each example introduces a new concept that is clear, small, and easy to understand
All examples are built using .NET 6 (LTS) and ChilliCream Hot Chocolate (Open-Source GraphQL Server for Microsoft .NET platform)
The GraphQL API is deployed to Azure App Service
The project demonstrates how to provision an Azure App Service using Azure CLI, Azure Powershell, and Azure Bicep

All examples are based on a fictitious company called MyView, that provides a service to view games and game ratings. It also allows reviewers to post and delete game reviews. Therefore, in the examples that follow, I demonstrate how to build a GraphQL API that provides the following capabilities:

List games
List games with associated reviews
Find a game by id
Post a game review
Delete a game review

The details of the provided examples are as follows:

📘 Example 1
- Create empty API project
- Add GraphQL Server support through the use of the ChilliCream Hot Chocolate nuget package
- Create a games query
- Create a games mutation
- Add metadata to support schema documentation

📘 Example 2
- Add global error handling
- Add input validation

📘 Example 3
- Add EntityFramework support with an in-memory database
- Add support for query filtering, sorting, and paging
- Add projection support
- Add query type extensions to allow splitting queries into multiple classes

📘 Example 4
- Change project to use SQlite instead of an in-memory database

Lastly, I provide the instructions on how to provision an Azure App Service and deploy the GraphQL API to it. The details are as follows:

Provision Azure App Service using Azure CLI
Provision Azure App Service using Azure Powershell
Provision Azure App Service using Azure Bicep
Deploy GraphQL API

📘 Find all the infrastructure as code here

Required Setup and Tools

It is recommended that the following tools and frameworks are installed before trying to run the examples:

.NET 6 SDK

All code in this example is developed with C# 10 using the latest cross-platform .NET 6 framework.

See the .NET 6 SDK official website for details on how to download and setup for your development environment.

Visual Studio Code

Find more information on Visual Studio Code with relevant C# and .NET extensions.

Bicep Installation and Docs

Everything you need to install and configure your windows, linux, and macos environment

Official Bicep Visual Studio Extension

Provides Bicep language support

jq is a lightweight and flexible command-line JSON processor

Azure Tools

A package of all the Visual Studio Code extensions that you will need to work with Azure

Azure CLI Tools

Tools for developing and running commands of the Azure CLI

Example 1

Example 1 demonstrates how to get started in terms of creating and running a GraphQL Server.

📘 The full example can be found on Github.

Step 1 - Create project


# create empty solution
dotnet new sln --name MyView

# create basic webapi project using minimal api's
dotnet new webapi --no-https --no-openapi --framework net6.0 --use-minimal-apis --name MyView.Api

# add webapi project to solution
dotnet sln ./MyView.sln add ./MyView.Api

Step 2 - Add GraphQL Capability to Web API

In this section, we turn the web application into a GraphQL API by installing the Chillicream Hot Chocolate GraphQL nuget package called HotChocolate.AspNetCore. This package contains the Hot Chocolate GraphQL query execution engine and query validation.


# From the /MyView.Api project folder, type the following commands:

# Add HotChocolate packages
dotnet add package HotChocolate.AspNetCore --version 12.11.1

Step 3 - Add Types

We are building a GraphQL API to allow querying and managing game reviews. Therefore, to get started we need to create a Game and GameReview type. A few things to note about the types we create:

I append the suffix "Dto" to indicate that the type is a Data Transfer Object. I use this to make it explicitly clear as to the intent of the type.
The GraphQLName attribute is used to rename the type for public consumption. The types will be exposed as Game and GameReview as opposed to GameDto and GameReviewDto
The GraphQLDescription attribute is used to provide a description of the type that is used by the GraphQL server to provide more detailed schema information
The types are defined as a record type, but can be declared as classes. I have chosen to use the record type as it allows me to define data contracts that are immutable and support value based equality comparison (should I require it).


[GraphQLDescription("Basic game information")]
[GraphQLName("Game")]
public sealed record GameDto
{
   public GameDto()
   {
      Reviews = new List<GameReviewDto>();
   }

   [GraphQLDescription("A unique game identifier")]
   public Guid GameId { get; set; }

   [GraphQLDescription("A brief description of game")]
   public string Summary { get; set; } = string.Empty;

   [GraphQLDescription("The name of the game")]
   public string Title { get; set; } = string.Empty;

   [GraphQLDescription("The date that game was released")]
   public DateTime ReleasedOn { get; set; }

   [GraphQLDescription("A list of game reviews")]
   public ICollection<GameReviewDto> Reviews { get; set; } = new List<GameReviewDto>();
}

// GameReviewDto.cs

[GraphQLDescription("Game review information")]
[GraphQLName("GameReview")]
public sealed record GameReviewDto
{
   [GraphQLDescription("Game identifier")]
   public Guid GameId { get; set; }

   [GraphQLDescription("Reviewer identifier")]
   public Guid ReviewerId { get; set; }

   [GraphQLDescription("Game rating")]
   public int Rating { get; set; }

   [GraphQLDescription("A brief description of game experience")]
   public string Summary { get; set; } = string.Empty;
}

Step 4 - Create First Query

One or more queries must be defined in order to support querying data.

For our first example we will create a query to support retrieving of game related (games, reviews) data. A few things to note are as follows:

At this stage we do not not have a database configured and will instead use in-memory data to demonstrate fetching of data.
I use the GraphQLDescription attribute to provide GraphQL schema documentation


// In this example, we use GameData (in-memory list) to provide sample dummy game related data

[GraphQLDescription("Query games")]
public sealed class GamesQuery
{
   [GraphQLDescription("Get list of games")]
   public IEnumerable<GameDto> GetGames() => GameData.Games;

   [GraphQLDescription("Find game by id")]
   public GameDto? FindGameById(Guid gameId) =>
      GameData.Games.FirstOrDefault(game => game.GameId == gameId);
}

Step 5 - Create First Mutation

We will add 2 operations. One to create, and one to remove a game review. A few things to note are as follows:

At this stage we do not not have a database configured and will instead use in-memory data to demonstrate saving of data.
The GraphQLDescription attribute is used to provide GraphQL schema documentation


[GraphQLDescription("Manage games")]
public sealed class GamesMutation
{
   [GraphQLDescription("Submit a game review")]
   public GameReviewDto SubmitGameReview(GameReviewDto gameReview)
   {
      var game = GameData
         .Games
         .FirstOrDefault(game => game.GameId == gameReview.GameId)
         ?? throw new Exception("Game not found");

      var gameReviewFromDb = game.Reviews.FirstOrDefault(review =>
         review.GameId == gameReview.GameId && review.ReviewerId == gameReview.ReviewerId);

      if (gameReviewFromDb is null)
      {
         game.Reviews.Add(gameReview);
      }
      else
      {
         gameReviewFromDb.Rating = gameReview.Rating;
         gameReviewFromDb.Summary = gameReview.Summary;
      }

      return gameReview;
   }

   [GraphQLDescription("Remove a game review")]
   public GameReviewDto DeleteGameReview(Guid gameId, Guid reviewerId)
   {
      var game = GameData
         .Games
         .FirstOrDefault(game => game.GameId == gameId)
         ?? throw new Exception("Game not found");

      var gameReviewFromDb = game
         .Reviews
         .FirstOrDefault(review => review.GameId == gameId && review.ReviewerId == reviewerId)
         ?? throw new Exception("Game review not found");

      game.Reviews.Remove(gameReviewFromDb);

      return gameReviewFromDb;
   }
}

Step 6 - Configure API with GraphQL services

We need to configure the API to use the ChilliCream Hotchocolate GraphQL services and middleware.


builder
   .Services
   .AddGraphQLServer() // Adds a GraphQL server configuration to the DI
   .AddMutationType<GamesMutation>() // Add GraphQL root mutation type
   .AddQueryType<GamesQuery>() // Add GraphQL root query type
   .ModifyRequestOptions(options =>
   {
      // allow exceptions to be included in response when in development
      options.IncludeExceptionDetails = builder.Environment.IsDevelopment();
   });

Step 7 - Map GraphQL Endpoint


app.MapGraphQL("/"); // Adds a GraphQL endpoint to the endpoint configurations

Step 8 - Run GraphQL API

Run the Web API project by typing the following command:


# From the /MyView.Api project folder, type the following commands:

dotnet run

Schema Information

Selecting the Browse Schema options allows one to view the schema information for queries, mutations, and objects as can be seen by the following screen shots.

Objects Schema

Queries Schema

Mutations Schema

Write GraphQL Queries


# List all games and associated reviews:

query listGames {
  games {
    gameId
    title
    releasedOn
    summary
    reviews {
      reviewerId
      rating
      summary
    }
  }
}


# Find a game (with reviews) by game id

# Add the following JSON snippet to the variables section:

{
  "gameId": "8f7e254f-a6ce-4f13-a44c-8f102a17f2f5"
}

# Write query to find a game (with reviews)

query findGameById ($gameId: UUID!) {
  findGameById (gameId: $gameId) {
    gameId
    title
    reviews {
      reviewerId
      rating
      summary
    }
  }
}

Write Mutations


# Submit a game review

# Define the following JSON in the 'Variables' section:

{
  "gameReview": {
    "gameId": "8f7e254f-a6ce-4f13-a44c-8f102a17f2f5",
    "reviewerId": "025fefe4-e2de-4765-9af3-f5dcd6d47458",
    "rating": 75,
    "summary": "Enim quidem enim. Eius aut velit voluptas."
  },
}

# Write a mutation to submit a game review

mutation submitGameReview($gameReview: GameReviewInput!) {
  submitGameReview(gameReview: $gameReview) {
    gameId
    reviewerId
    rating
    summary
  }
}

# Write a mutation to DELETE a game review

# Define the following JSON in the 'variables' section:

{
    "gameId": "8f7e254f-a6ce-4f13-a44c-8f102a17f2f5",
    "reviewerId": "025fefe4-e2de-4765-9af3-f5dcd6d47458"
}

# Write mutation to delete game review

mutation deleteGameReview($gameId: UUID!, $reviewerId: UUID!) {
  deleteGameReview(gameId: $gameId, reviewerId: $reviewerId) {
    gameId
    reviewerId
    rating
    summary
  }
}

Example 2

Example 2 demonstrates the following concepts:

Implement input validations (like validating a review that is submitted)
Implement global error handling

📘 The full example can be found on Github.

Implement Input Validation

Currently, we have no validation in place when submitting a GameReview. In this section, we are going to provide input validation through the use of the [Fluent Validation] nuget package.

Step 1 - Add Fluent Validation packages


dotnet add package FluentValidation.AspNetCore --version 11.1.1
dotnet add package FluentValidation.DependencyInjectionExtensions --version 11.1.0

Step 2 - Create Validator

We need to define a class that will handle validation for the GameReview type. To do this, we create a GameReviewValidator as follows:


public sealed class GameReviewValidator : AbstractValidator<GameReviewDto>
{
   public GameReviewValidator()
   {
      RuleFor(e => e.GameId)
        .Must(gameId => GameData.Games.Any(game => game.GameId == gameId))
        .WithMessage(e => $"A game having id '{e.GameId}' does not exist");

      RuleFor(e => e.Rating)
        .LessThanOrEqualTo(100)
        .GreaterThanOrEqualTo(0);

      RuleFor(e => e.Summary)
        .NotNull()
        .NotEmpty()
        .MinimumLength(20)
        .MaximumLength(500);
   }
}

Step 3 - Use GameReviewValidator

The GameReviewValidator is used in the GamesMutation class to validate the GameReviewDto.
We use constructor dependency injection to provide the GameReviewValidator
The code _validator.ValidateAndThrow(gameReview); will execute the validation rules defined for GameReviewDto and throw a validation exception if there are any validation failures.


[GraphQLDescription("Manage games")]
public sealed class GamesMutation
{
   private readonly AbstractValidator<GameReviewDto> _validator;

   public GamesMutation(AbstractValidator<GameReviewDto> validator)
   {
      _validator = validator ?? throw new ArgumentNullException(nameof(validator));
   }

   [GraphQLDescription("Submit a game review")]
   public GameReviewDto SubmitGameReview(GameReviewDto gameReview)
   {
      // use fluent validator
      _validator.ValidateAndThrow(gameReview);

      .
      .
      .
   }
}

Step 4 - Add Validation Service


// configure fluent validation for GameReviewDto
builder.Services.AddTransient<AbstractValidator<GameReviewDto>, GameReviewValidator>();

Configure Global Error Handling

We are throwing exceptions in a number of areas. Those exceptions will be allowed to bubble all the way to the client if allowed to. The following list highlights the areas where we are throwing exceptions along with the GraphQL response resultiing from the exception:

Validation


  public GameReviewDto SubmitGameReview(GameReviewDto gameReview)
  {
      // use fluent validator
      _validator.ValidateAndThrow(gameReview);
      .
      .
      .
  }


  {
    "errors": [
        {
            "message": "Unexpected Execution Error",
            "locations": [
                {
                "line": 2,
                "column": 3
                }
            ],
            "path": [
                "submitGameReview"
            ],
            "extensions": {
                "message": "Validation failed: \r\n -- Rating: 'Rating' must be greater than or equal to '0'. Severity: Error\r\n -- Summary: 'Summary' must not be empty. Severity: Error\r\n -- Summary: The length of 'Summary' must be at least 20 characters. You entered 0 characters. Severity: Error",
                "stackTrace": "   at FluentValidation.AbstractValidator`..."
                }
        }
    ]
  }

Delete Game


  public GameReviewDto DeleteGameReview(Guid gameId, Guid reviewerId)
  {
      var game = GameData
         .Games
         .FirstOrDefault(game => game.GameId == gameId)
         ?? throw GameNotFoundException.WithGameId(gameId);
         .
         .
         .
  }


  {
    "errors": [
        {
            "message": "Unexpected Execution Error",
            "locations": [
                {
                "line": 11,
                "column": 3
                }
            ],
            "path": [
                "deleteGameReview"
            ],
            "extensions": {
                "message": "Exception of type 'MyView.Api.Games.GameNotFoundException' was thrown.",
                "stackTrace": "   at MyView.Api.Games.GamesMutation.DeleteGameReview ..."
            }
        }    
    ]
  }

Step 1 - Create Custom Error Filter

We create a new class called ServerErrorFilter that inherits from the IErrorFilter interface as follows:


public sealed class ServerErrorFilter : IErrorFilter
{
   private readonly ILogger _logger;
   private readonly IWebHostEnvironment _environment;

   public ServerErrorFilter(ILogger logger, IWebHostEnvironment environment)
   {
      _logger = logger ?? throw new ArgumentNullException(nameof(logger));
      _environment = environment ?? throw new ArgumentNullException(nameof(environment));
   }

   public IError OnError(IError error)
   {
      _logger.LogError(error.Exception, error.Message);

      if (_environment.IsDevelopment())
         return error;

      return ErrorBuilder
         .New()
         .SetMessage("An unexpected server fault occurred")
         .SetCode(ServerErrorCode.ServerFault)
         .SetPath(error.Path)
         .Build();
   }
}

Step 2 - Add ValidationException Handler


public sealed class ServerErrorFilter : IErrorFilter
{
    .
    .
    .

    public IError OnError(IError error)
    {
        if (error.Exception is ValidationException validationException)
        {
            _logger.LogError(validationException, "There is a validation error");

            var errorBuilder = ErrorBuilder
                .New()
                .SetMessage("There is a validation error")
                .SetCode(ServerErrorCode.BadUserInput)
                .SetPath(error.Path);

            foreach (var validationFailure in validationException.Errors)
            {
                errorBuilder.SetExtension(
                    $"{ServerErrorCode.BadUserInput}_{validationFailure.PropertyName.ToUpper()}",
                    validationFailure.ErrorMessage);
            }

            return errorBuilder.Build();
        }

        .
        .
        .
    }
}

Step 3 - Add GameNotFoundException Handler


public sealed class ServerErrorFilter : IErrorFilter
{
    .
    .
    .

    public IError OnError(IError error)
    {
        .
        .
        .

        if (error.Exception is GameNotFoundException gameNotFoundException)
        {
            _logger.LogError(gameNotFoundException, "Game not found");

            return ErrorBuilder
                .New()
                .SetMessage($"A game having id '{gameNotFoundException.GameId} could not found")
                .SetCode(ServerErrorCode.ResourceNotFound)
                .SetPath(error.Path)
                .SetExtension($"{ServerErrorCode.ResourceNotFound}_GAME_ID", gameNotFoundException.GameId)
                .Build();
        }

        .
        .
        .
    }
}

Step 4 - Add GameReviewNotFoundException Handler


public sealed class ServerErrorFilter : IErrorFilter
{
    .
    .
    .

    public IError OnError(IError error)
    {
        .
        .
        .

        if (error.Exception is GameReviewNotFoundException gameReviewNotFoundException)
        {
            _logger.LogError(gameReviewNotFoundException, "Game review not found");

            return ErrorBuilder
                .New()
                .SetMessage($"A game review having game id '{gameReviewNotFoundException.GameId}' and reviewer id '{gameReviewNotFoundException.ReviewerId}' could not found")
                .SetCode(ServerErrorCode.ResourceNotFound)
                .SetPath(error.Path)
                .SetExtension($"{ServerErrorCode.ResourceNotFound}_GAME_ID", gameReviewNotFoundException.GameId)
                .SetExtension($"{ServerErrorCode.ResourceNotFound}_REVIEWER_ID", gameReviewNotFoundException.ReviewerId)
                .Build();
        }

        .
        .
        .
    }
}

Step 5 - Configure GraphQL Service to support Error Filter


builder
   .Services
   .AddGraphQLServer()
   // Add global error handling
   .AddErrorFilter(provider =>
      {
         return new ServerErrorFilter(
            provider.GetRequiredService<ILogger<ServerErrorFilter>>(),
            builder.Environment);
      })
      .
      .
      .
   .ModifyRequestOptions(options =>
   {
      options.IncludeExceptionDetails = builder.Environment.IsDevelopment();
   });

Step 6 - Test Error Handling

Validation


  {
    "errors": [
        {
            "message": "There is a validation error",
            "path": [
                "submitGameReview"
            ],
            "extensions": {
                "code": "BAD_USER_INPUT",
                "BAD_USER_INPUT_RATING": "'Rating' must be greater than or equal to '0'.",
                "BAD_USER_INPUT_SUMMARY": "The length of 'Summary' must be at least 20 characters. You entered 0 characters."
            }
        }
    ]  
  }

GameNotFoundException


  {
    "errors": [
        {
            "message": "A game having id '8f7e254f-a6ce-4f13-a44c-8f102a17f2f4 could not found",
            "path": [
                "deleteGameReview"
            ],
            "extensions": {
                "code": "RESOURCE_NOT_FOUND",
                "RESOURCE_NOT_FOUND_GAME_ID": "8f7e254f-a6ce-4f13-a44c-8f102a17f2f4"
            }
        }
    ]
  }

GameReviewNotFoundException


  {
      "errors": [
          {
            "message": "A game review having game id '8f7e254f-a6ce-4f13-a44c-8f102a17f2f5' and reviewer id '8b019864-4af2-4606-88b5-13e5eb62ff4e' could not found",
            "path": [
                "deleteGameReview"
            ],
            "extensions": {
                "code": "RESOURCE_NOT_FOUND",
                "RESOURCE_NOT_FOUND_GAME_ID": "8f7e254f-a6ce-4f13-a44c-8f102a17f2f5",
                "RESOURCE_NOT_FOUND_REVIEWER_ID": "8b019864-4af2-4606-88b5-13e5eb62ff4e"
            }
          }
      ]
  }

Example 3

For Example 3, we extend Example 2 to cover the following topics:

Add EntityFramework support with in-memory database
Add support for query projection, filtering, sorting, and paging
Add query type extensions to allow splitting queries into multiple classes
Add database seeding (seed with fake data)

Add EntityFramework Support

Add required nuget packages
Create the following entities that will be used to help represent the data stored in our database
- Game
- GameReview
- Reviewer
Create a context that will serve as our Repository/UnitOfWork called AppDbContext
Configure data services

Add EntityFramework Packages


# add required EntityFramework packages
dotnet add package Microsoft.EntityFrameworkCore.Design --version 6.0.6
dotnet add package Microsoft.EntityFrameworkCore.InMemory --version 6.0.6

# add HotChocolate package providing EntityFramework support
dotnet add package HotChocolate.Data.EntityFramework --version 12.11.1

Define Entities


public class Game
{
   public Guid Id { get; set; }
   public string Title { get; set; } = string.Empty;
   public string Summary { get; set; } = string.Empty;
   public DateTime ReleasedOn { get; set; }
   public ICollection<GameReview> Reviews { get; set; } = new HashSet<GameReview>();
}

public sealed class GameReview
{
   public Guid GameId { get; set; }
   public Game? Game { get; set; }
   public Guid ReviewerId { get; set; }
   public Reviewer? Reviewer { get; set; }
   public int Rating { get; set; }
   public string Summary { get; set; } = string.Empty;
   public DateTime ReviewedOn { get; set; }
}

public class Reviewer
{
   public Guid Id { get; set; }
   public string Name { get; set; } = string.Empty;
   public string Email { get; set; } = string.Empty;
   public string Username { get; set; } = string.Empty;
   public string Picture { get; set; } = string.Empty;
   public ICollection<GameReview> GameReviews { get; set; } = new HashSet<GameReview>();
}

Create AppDbContext


public sealed class AppDbContext : DbContext
{
   public AppDbContext(DbContextOptions<AppDbContext> options) : base(options)
   {
   }

   public DbSet<Game> Games { get; set; } = null!;
   public DbSet<GameReview> Reviews { get; set; } = null!;
   public DbSet<Reviewer> Reviewers { get; set; } = null!;

   protected override void OnModelCreating(ModelBuilder modelBuilder)
   {
      modelBuilder.Entity<Game>(game =>
      {
         ...
      });

      modelBuilder.Entity<Reviewer>(reviewer =>
      {
         ...
      });

      modelBuilder.Entity<GameReview>(gameReview =>
      {
         ...
      });

      base.OnModelCreating(modelBuilder);
   }
}

Configure Data Services


var builder = WebApplication.CreateBuilder(args);

// configure in-memory database
builder
   .Services
   .AddDbContextFactory<AppDbContext>(options =>
   {
      options.UseInMemoryDatabase("myview");
      options.EnableDetailedErrors(builder.Environment.IsDevelopment());
      options.EnableSensitiveDataLogging(builder.Environment.IsDevelopment());
   });

builder
   .Services
   .AddScoped<AppDbContext>(provider => provider
      .GetRequiredService<IDbContextFactory<AppDbContext>>()
      .CreateDbContext());

Update GamesQuery


   [GraphQLDescription("Get list of games")]
   public IQueryable<GameDto> GetGames([Service] AppDbContext context)
   {
      return context
         .Games
         .AsNoTracking()
         .TagWith($"{nameof(GamesQuery)}::{nameof(GetGames)}")
         .OrderByDescending(game => game.ReleasedOn)
         .Include(game => game.Reviews)
         .Select(game => new GameDto
         {
            GameId = game.Id,
            Reviews = game.Reviews.Select(review => new GameReviewDto
            {
               GameId = review.GameId,
               Rating = review.Rating,
               ReviewerId = review.ReviewerId,
               Summary = review.Summary
            }),
            ReleasedOn = game.ReleasedOn,
            Summary = game.Summary,
            Title = game.Title
         });
   }

   [GraphQLDescription("Find game by id")]
   public async Task<GameDto?> FindGameById([Service] AppDbContext context, Guid gameId)
   {
      var game = await context
         .Games
         .AsNoTracking()
         .TagWith($"{nameof(GamesQuery)}::{nameof(FindGameById)}")
         .Include(game => game.Reviews)
         .FirstOrDefaultAsync(game => game.Id == gameId);

      if (game is null) return null;

      return new GameDto
      {
         GameId = game.Id,
         Reviews = game.Reviews.Select(review => new GameReviewDto
         {
            GameId = review.GameId,
            Rating = review.Rating,
            ReviewerId = review.ReviewerId,
            Summary = review.Summary
         }),
         ReleasedOn = game.ReleasedOn,
         Summary = game.Summary,
         Title = game.Title
      };
   }

Update GamesMutation


   [GraphQLDescription("Submit a game review")]
   public async Task<GameReviewDto> SubmitGameReview(
        [Service] AppDbContext context,
        GameReviewDto gameReview)
   {
      // use fluent validator
      await _validator.ValidateAndThrowAsync(gameReview);

      var game = await context
         .Games
         .FirstOrDefaultAsync(game => game.Id == gameReview.GameId)
         ?? throw GameNotFoundException.WithGameId(gameReview.GameId);

      var reviewer = await context
         .Reviewers
         .FirstOrDefaultAsync(reviewer => reviewer.Id == gameReview.ReviewerId)
         ?? throw ReviewerNotFoundException.WithReviewerId(gameReview.ReviewerId);

      var gameReviewFromDb = await context
         .Reviews
         .FirstOrDefaultAsync(review => 
            review.GameId == gameReview.GameId 
            && review.ReviewerId == gameReview.ReviewerId);

      if (gameReviewFromDb is null)
      {
         context.Reviews.Add(new GameReview
         {
            GameId = gameReview.GameId,
            Rating = gameReview.Rating,
            ReviewedOn = DateTime.UtcNow,
            ReviewerId = gameReview.ReviewerId,
            Summary = gameReview.Summary
         });
      }
      else
      {
         gameReviewFromDb.Rating = gameReview.Rating;
         gameReviewFromDb.Summary = gameReview.Summary;
      }

      await context.SaveChangesAsync();

      return gameReview;
   }


   [GraphQLDescription("Remove a game review")]
   public async Task<GameReviewDto> DeleteGameReview(
      [Service] AppDbContext context,
      Guid gameId,
      Guid reviewerId)
   {
      var gameReviewFromDb = await context
         .Reviews
         .FirstOrDefaultAsync(review => review.GameId == gameId && review.ReviewerId == reviewerId)
         ?? throw GameReviewNotFoundException.WithGameReviewId(gameId, reviewerId);

      context.Reviews.Remove(gameReviewFromDb);

      return new GameReviewDto
      {
         GameId = gameReviewFromDb.GameId,
         Rating = gameReviewFromDb.Rating,
         ReviewerId = gameReviewFromDb.ReviewerId,
         Summary = gameReviewFromDb.Summary
      };
   }

Add Query Projection, Paging, Filtering and Sorting Support

There are 2 steps required to enable projections, paging, filtering and sorting.

Step 1 - Add Attributes To Queries

Add the following attributes to the method performing queries. The ordering of the attributes is important and should be defined in the following order

UsePaging
UseProjection
UseFiltering
UseSorting


[GraphQLDescription("Get list of games")]
[UsePaging]
[UseProjection]
[UseFiltering]
[UseSorting]
public IQueryable<GameDto> GetGames([Service] AppDbContext context)
{
    ...
}

[GraphQLDescription("Find game by id")]
[UseProjection]
public async Task<GameDto?> FindGameById([Service] AppDbContext context, Guid gameId)
{
    ...
}

Step 2 - Configure GraphQL Service


builder
   .Services
   .AddGraphQLServer()
   .
   .
   .   
   // The .AddTypeExtension allows having queries defined in multiple files
   // whilst still having a single root query
   .AddQueryType() // Add GraphQL root query type
      .AddTypeExtension<GamesQuery>()
      .AddTypeExtension<ReviewerQuery>()
    // Add Projection, Filtering and Sorting support. The ordering matters.    
   .AddProjections()
   .AddFiltering()
   .AddSorting()
   ...
   );

Add Database Seeding

Define Seeder class to generate fake data


internal static class Seeder
{
   public static async Task SeedDatabase(WebApplication app)
   {
      await using (var serviceScope = app.Services.CreateAsyncScope())
      {
         var context = serviceScope
            .ServiceProvider
            .GetRequiredService<AppDbContext>();

         await context.Database.EnsureCreatedAsync();

         if (!await context.Reviewers.AnyAsync())
         {
            var reviewers = JsonSerializer.Deserialize<IEnumerable<Reviewer>>(
               _reviewersText,
               new JsonSerializerOptions { PropertyNameCaseInsensitive = true })!;

            context.AddRange(reviewers);
         }

         if (!await context.Games.AnyAsync())
         {
            var games = JsonSerializer.Deserialize<IEnumerable<Game>>(
               _gamesText,
               new JsonSerializerOptions { PropertyNameCaseInsensitive = true })!;

            context.Games.AddRange(games);
         }

         await context.SaveChangesAsync();
      }
   }

   private static readonly string _gamesText = @"
      [
         .
         .
         .
      ]";

   private static readonly string _reviewersText = @"
      [
         .
         .
         .
      ]";
}

Configure when to Seed database


// Program.cs

app.MapGraphQL("/");

// seed database after all middleware is setup
await Seeder.SeedDatabase(app);

Example 4

For Example 4, we extend Example 3 to use a SQLite database.

Configure SQLite

Add Packages


# add required EntityFramework packages
dotnet add package Microsoft.EntityFrameworkCore.Design --version 6.0.6
dotnet add package Microsoft.EntityFrameworkCore.SQLite --version 6.0.6

# add HotChocolate package providing EntityFramework support
dotnet add package HotChocolate.Data.EntityFramework --version 12.11.1

Configure AppDbContext

Update the OnModelCreating method to provide all the required entity-to-table mappings and relationships.


public sealed class AppDbContext : DbContext
{
   .
   .
   .

   protected override void OnModelCreating(ModelBuilder modelBuilder)
   {
      // configure Game
      modelBuilder.Entity<Game>(game =>
      {
         // configure table
         game.ToTable(nameof(Game).ToLower());

         // configure properties
         game.Property(e => e.Id).HasColumnName("id");
         game.Property(e => e.ReleasedOn).HasColumnName("released_on").IsRequired();
         game.Property(e => e.Summary).HasColumnName("summary").IsRequired();
         game.Property(e => e.Title).HasColumnName("title").IsRequired();

         // configure primary key
         game.HasKey(e => e.Id).HasName("pk_game_id");

         // configure game relationship
         game.HasMany(e => e.Reviews).WithOne(e => e.Game);
      });


      // configure Reviewer
      modelBuilder.Entity<Reviewer>(reviewer =>
      {
         // configure table
         reviewer.ToTable(nameof(Reviewer).ToLower());

         // configure properties
         reviewer.Property(e => e.Id).HasColumnName("id");
         reviewer.Property(e => e.Email).HasColumnName("email").IsRequired();
         reviewer.Property(e => e.Name).HasColumnName("name").IsRequired();
         reviewer.Property(e => e.Picture).HasColumnName("picture").IsRequired();
         reviewer.Property(e => e.Username).HasColumnName("username").IsRequired();

         // configure primary key
         reviewer.HasKey(e => e.Id).HasName("pk_reviewer_id");

         // configure reviewer relationship
         reviewer.HasMany(e => e.GameReviews).WithOne(e => e.Reviewer);
      });


      // configure GameReview
      modelBuilder.Entity<GameReview>(gameReview =>
      {
         // configure table
         gameReview.ToTable("game_review");

         // configure properties
         gameReview.Property(e => e.GameId).HasColumnName("game_id").IsRequired();
         gameReview.Property(e => e.ReviewerId).HasColumnName("reviewer_id").IsRequired();
         gameReview.Property(e => e.Rating).HasColumnName("rating").IsRequired();
         gameReview.Property(e => e.ReviewedOn).HasColumnName("reviewed_on").IsRequired();
         gameReview.Property(e => e.Summary).HasColumnName("summary").HasDefaultValue("");

         // configure primary key
         gameReview
            .HasKey(e => new { e.GameId, e.ReviewerId })
            .HasName("pk_gamereview_gameidreviewerid");

         // configure game relationship
         gameReview
            .HasOne(e => e.Game)
            .WithMany(e => e.Reviews)
            .HasConstraintName("fk_gamereview_gameid");

         gameReview
            .HasIndex(e => e.GameId)
            .HasDatabaseName("ix_gamereview_gameid");

         // configure reviewer relationship
         gameReview
            .HasOne(e => e.Reviewer)
            .WithMany(e => e.GameReviews)
            .HasConstraintName("fk_gamereview_reviewerid");

         gameReview
            .HasIndex(e => e.ReviewerId)
            .HasDatabaseName("ix_gamereview_reviewerid");
      });

      base.OnModelCreating(modelBuilder);
   }
}

Configure Data Services


builder
   .Services
   .AddDbContextFactory<AppDbContext>(options =>
   {
      options.UseSqlite("Data Source=myview.db");
      options.EnableDetailedErrors(builder.Environment.IsDevelopment());
      options.EnableSensitiveDataLogging(builder.Environment.IsDevelopment());
   });

builder
   .Services
   .AddScoped<AppDbContext>(provider => provider
      .GetRequiredService<IDbContextFactory<AppDbContext>>()
      .CreateDbContext());

Create Database Migrations

Before running the application, we need to create the database migrations in order to create/update the SQLite database. After the migrations and database update are complete, a SQLite database called myview.db will be created in the root of the project folder.


# navigate to API project folder
cd ./MyView.Api

# create the database migrations
dotnet ef migrations add "CreateInitialDatabaseSchema"

Create and Update SQLite Database


# run the migrations to create database
dotnet ef database update

Deploy GraphQL API

In this section, we will cover the following primary topics:

Provision Azure App Service
Deploy GraphQL API to Azure App Service

Provision Azure App Service

We will be deploying the 'MyView API' into [Azure App Service]. However, before we can deploy our API, we first need to provision our Azure App Service. This section demonstrates how to provision an Azure App Service using 3 different techniques and are listed as follows:

Azure CLI
Azure Powershell
Azure Bicep

Regardless of the chosen technique, there are 3 general steps that need to be completed in order to provision the Azure App Service.

Create a resource group
Create an App Service Plan
Create an App Service

Once all the required resources are created, we will be ready to deploy the 'MyView API' to Azure.

Create Azure App Service Using Azure CLI

All the commands that are required to create the Azure App Service using the Azure CLI can be found in the iac/azcli/deploy.azcli file that is part of the example github repository


$location = "australiaeast"

# STEP 1 - create resource group
$rgName = "myview-rg"
az group create `
    --name $rgName `
    --location $location

# STEP 2 - create appservice plan
$aspName = "myview-asp"
$appSku = "F1"
az appservice plan create `
    --name $aspName `
    --resource-group $rgName `
    --sku $appSku `
    --is-linux

# STEP 3 - create webapp
$appName = "myview-webapp-api"
$webapp = az webapp create `
    --name $appName `
    --resource-group $rgName `
    --plan $aspName `
    --runtime '"DOTNETCORE|6.0"'

# STEP 4 - cleanup
az group delete --resource-group $rgName -y

Create Azure App Service Using Azure Powershell

All the commands that are required to create the Azure App Service using the Azure Powershell can be found in the /azpwsh/deploy.azcli file that is part of the example github repository.


$location = "australiaeast"

# STEP 1 - create resource group
$rgName = "myview-rg"
New-AzResourceGroup -Name $rgName -Location $location

# STEP 2 - create appservice plan
$aspName = "myview-asp"
$appSku = "F1"
New-AzAppServicePlan `
    -Name $aspName `
    -Location $location `
    -ResourceGroupName $rgName `
    -Tier $appSku `
    -Linux

# STEP 3 - create webapp
$appName = "myview-webapp-api"
New-AzWebApp `
    -Name $appName `
    -Location $location `
    -AppServicePlan $aspName `
    -ResourceGroupName $rgName

# STEP 4 - configure webapp

## At this point, the webapp is not using .NET v6 as is required.
## This can be verified by the following commands
az webapp config show `
    --resource-group $rgName `
    --name $appName `
    --query netFrameworkVersion

Get-AzWebApp `
    -Name $appName `
    -ResourceGroupName $rgName `
    | Select-Object -ExpandProperty SiteConfig `
    | Select-Object -Property NetFrameworkVersion

Get-AzWebApp `
    -Name $appName `
    -ResourceGroupName $rgName `
    | ConvertTo-Json `
    | jq ".SiteConfig.NetFrameworkVersion"

## lets configure the webapp with the correct version of .NET
Set-AzWebApp `
    -Name $appName `
    -ResourceGroupName $rgName `
    -AppServicePlan $aspName `
    -NetFrameworkVersion 'v6.0'

$apiVersion = "2020-06-01"
$config = Get-AzResource `
    -ResourceGroupName $rgName `
    -ResourceType Microsoft.Web/sites/config `
    -ResourceName $appName/web `
    -ApiVersion $apiVersion
$config.Properties.linuxFxVersion = "DOTNETCORE|6.0"
$config | Set-AzResource -ApiVersion $apiVersion -Force

# cleanup
Remove-AzResourceGroup -Name $rgName -Force

Create Azure App Service Using Azure Bicep

In this example, I demonstrate both a basic and more advanced option (uses modules) to deploy bicep templates.

Azure Bicep - Basic

All the commands and templates that are required to create the Azure App Service using Azure Bicep can be found in the iac/bicep/basic folder that is part of the example github repository


# file: iac/bicep/basic/deploy.azcli

$location = "australiaeast"

# STEP 1 - create resource group
$rgName = "myview-rg"
az group create --name $rgName --location $location

# STEP 2 - deploy template
$aspName = "myview-asp"
$appName = "myview-webapp-api"
$deploymentName = "myview-deployment"

az deployment group create `
    --name $deploymentName `
    --resource-group $rgName `
    --template-file ./main.bicep `
    --parameters appName=$appName `
    --parameters aspName=$aspName

# cleanup
az group delete --resource-group $rgName -y


// file: iac/bicep/basic/main.bicep

// parameters
@description('The name of app service')
param appName string

@description('The name of app service plan')
param aspName string

@description('The location of all resources')
param location string = resourceGroup().location

@description('The Runtime stack of current web app')
param linuxFxVersion string = 'DOTNETCORE|6.0'

@allowed([
  'F1'
  'B1'
])
param skuName string = 'F1'

// variables

resource asp 'Microsoft.Web/serverfarms@2021-02-01' = {
  name: aspName
  location: location
  kind: 'linux'
  sku: {
    name: skuName
  }
  properties: {
    reserved: true
  }
}

resource app 'Microsoft.Web/sites@2021-02-01' = {
  name: appName
  location: location
  properties: {
    serverFarmId: asp.id
    httpsOnly: true
    siteConfig: {
      linuxFxVersion: linuxFxVersion
    }
  }
}

Azure Bicep - Advanced (with modules)

All the commands and templates that are required to create the Azure App Service using Azure Bicep can be found in the iac/bicep/advanced folder that is part of the example github repository


# file: iac/bicep/advanced/deploy.azcli

$location = "australiaeast"


# STEP 1 - deploy template
$deploymentName = "myview-deployment"

az deployment sub create `
    --name $deploymentName `
    --location $location `
    --template-file ./main.bicep `
    --parameters location=$location


# STEP 2 - get outputs from deployment
az deployment sub show --name $deploymentName --query "properties.outputs"
$hostName = $(az deployment sub show --name $deploymentName --query "properties.outputs.defaultHostName.value")
$rgName = $(az deployment sub show --name $deploymentName --query "properties.outputs.resourceGroupName.value")
echo $hostName, $rgName


# STEP 3 - cleanup
az group delete --resource-group $rgName -y


// file: iac/bicep/advanced/modules/app-service.bicep

// parameters
@description('The name of app service')
param appName string

@description('The name of app service plan')
param aspName string

@description('The location of all resources')
param location string = resourceGroup().location

@description('The Runtime stack of current web app')
param linuxFxVersion string = 'DOTNETCORE|6.0'

@allowed([
  'F1'
  'B1'
])
param skuName string = 'F1'


// define resources
resource asp 'Microsoft.Web/serverfarms@2021-02-01' = {
  name: aspName
  location: location
  kind: 'linux'
  sku: {
    name: skuName
  }
  properties: {
    reserved: true
  }
}

resource app 'Microsoft.Web/sites@2021-02-01' = {
  name: appName
  location: location
  properties: {
    serverFarmId: asp.id
    httpsOnly: true
    siteConfig: {
      linuxFxVersion: linuxFxVersion
    }
  }
}


// define outputs
output defaultHostName string = app.properties.defaultHostName


// file: iac-src/bicep/advanced/main.bicep

// define scope
targetScope = 'subscription'


// define parameters
@description('The location of all resources')
param location string = deployment().location


// define variables
@description('The name of resource group')
var rgName = 'myview-rg'

@description('The name of resource group')
var aspName = 'myview-asp'

@description('The name of app')
var appName = 'myview-webapp-api'


// define resources
resource resourceGroup 'Microsoft.Resources/resourceGroups@2021-04-01' = {
  name: rgName
  location: location
}


// define modules
module appServiceModule 'modules/app-service.bicep' = {
  scope: resourceGroup
  name: 'myview-module'
  params: {
    appName: appName
    aspName: aspName
    location: location
  }
}


// define outputs
output defaultHostName string = appServiceModule.outputs.defaultHostName
output resourceGroupName string = rgName

Deploy GraphQL API to Azure App Service

Publish GraphQL API Locally


dotnet publish `
  --configuration Release `
  --framework net6.0 `
  --output ./release `
  ./MyView.Api

Deploy GraphQL API


cd ./release

az webapp up `
  --plan myview-asp `
  --name myview-webapp-api `
  --resource-group myview-rg `
  --os-type linux `
  --runtime "DOTNETCORE:6.0"

Where To Next?

I have provided a number of examples that show how to build a GraphQL Server using ChilliCream Hot Chocolate GraphQL Server. If you would like to learn more, please view the following learning resources:

HotChocolate Templates

There are also a number of Hot Chocolate templates that can be installed using the dotnet CLI tool.

Install HotChocolate Templates:


  # install Hot Chocolate GraphQL server templates (includes Azure function template)
  dotnet new -i HotChocolate.Templates

  # install template that allows you to create a GraphQL Star Wars Demo
  dotnet new -i HotChocolate.Templates.StarWars

List HotChocolate Templates


  # list HotChocolate templates
  dotnet new --list HotChocolate


  Template Name                        Short Name   Language  Tags
  -----------------------------------  -----------  --------  ------------------------------
  HotChocolate GraphQL Function        graphql-azf  [C#]      Web/GraphQL/Azure
  HotChocolate GraphQL Server          graphql      [C#]      Web/GraphQL
  HotChocolate GraphQL Star Wars Demo  starwars     [C#]      ChilliCream/HotChocolate/Demos

Create HotChocolate project using templates


  # create ASP.NET GraphQL Server
  dotnet new graphql --name MyGraphQLDemo

  # create graphql server using Azure Function
  dotnet new graphql-azf --name MyGraphQLAzfDemo

  # create starwars GraphQL demo
  mkdir StarWars
  cd StarWars
  dotnet new starwars

8 C#.NET Projects

Douglas Minnaar — Sun, 02 May 2021 00:09:26 +0000

I've recently been busy upgrading and creating a number of projects written in C# using .NET5 Framework. The projects range in difficulty from intermediate to advanced and cover a broad range of topics including:

using C# with database tech like Postgres and MongoDB
API examples for REST, GraphQL, and gRPC
AWS examples covering services like DynamoDb, Cognito, S3, SNS, and SQS
working with message brokers like Rabbit MQ
setting up your development stack using Docker and Docker-Compose
ASP.NET and ASP.NET Identity
.NET Entity Framework Core
working with Redgate Flyway to manage database versioning and migrations
architecture examples like modular monolith, onion architecture, clean architecture etc

There are 8 projects in total and can be summarised as follows:

NoteWorx ASP.NET
Identity Example
Algorithms and Data Structures
RabbitMQ for .NET Developers
AWS for .NET Developers
MFlix .NET
Chinook .NET
Ranker .NET (REST API Guide)

NoteWorx ASP.NET

NoteWorx ASP.NET is a basic note management web application that has been built using ASP.NET Core, ASP.NET Core Identity, Entity Framework Core, and Postgresql.

Highlights:

.NET5
ASP.NET application using Identity
Shows how to use Gulp as an asset manager for javascript, css, and fonts etc
Uses docker, docker-compose to manage stack (Postgres etc)

Project Page:

Find the project with detailed README here

Identity Example

Identity Example is a project that shows how to use, customise, and configure Identity on Asp.Net Core with different backend stores (postgresql, mssql, sqlite). If you've ever wondered about how to take complete control over the way the Identity database is generated, then this is a good project to use as a guide.

Highlights:

.NET5
Focus on ASP.NET Identity customization
Includes sample API project that demonstrates how to use ASP.NET Identity and Jwt Bearer Authentication. The API project also demonstrates how to implement refresh tokens.
Uses docker, docker-compose to manage stack for Postgres, MSSQL
Uses powershel and bash scripts to automate spinning up and down of various stacks

Project Page:

Find the project with detailed README here

Algorithms and Data Structures

Algorithms and Data Structures is a compilation of various algorithms and data structures that have been implemented using C# and Javascript.

Highlights:

.NET5 and Javascript
Covers Bubble Sort, Shell Sort, Selection Sort, Insertion Sort, Merge Sort, Quick Sort, Singly Linked List, Doubly Linked List
Accompanied by detailed README explaining various algorithms and data structures

Project Page:

Find the project with detailed README here

RabbitMQ for .NET Developers

RabbitMQ for .NET Developers is a composition of RabbitMQ examples for C# .NET developers. This project has a specific focus on demonstrating how to use RabbitMQ with C#.NET 5. This is achieved by providing a collection of practical examples (written in C#.NET 5) that highlight a number of messaging patterns.

Highlights:

.NET5
Demonstrates the following message patterns:
Accompanied by detailed README files for each example demonstrating message pattern.
Accompanied by a "getting started" guide that will help you learn a little more about RabbitMQ
Manage entire RabbitMQ stack using Docker and Docker-Compose

Project Page:

Find the project with detailed README here

AWS for .NET Developers

AWS for .NET Developers is a collection of independent .NET projects written in C# .NET 5 that demonstrate how to integrate with various AWS services using the AWS SDK for dotnet.

Highlights:

.NET5
Demonstrates the following AWS services:
Accompanied by detailed README files for each AWS service example demonstrating how to use the .NET AWS SDK.
Demonstrates how to use .NET Secret Manager by storing AWS config as "secrets"

Project Page:

Find the project with detailed README here

MFlix .NET

MFlix .NET is a showcase of how to build API's (REST, gRPC, GraphQL) using C# .NET. I have created a contrived use-case that allows me to demonstrate how to build different API's for different needs. The high-level requirements cater to 3 types of developers as illustrated below:

Furthermore, the project is based on the following high-level architectural diagram that highlights

the type of API's being built
the way in which different API's will be accessed by different types of applications
the scope of API's (public or private)

Highlights:

.NET5
Error handling
Logging
Caching
Security
Dependency Injection
Configuration
How to do paging, filtering, and sorting via the API
Developing with MongoDB and C#
Demonstrates how to seed a MongoDB database as part of docker-compose stack
Preparing a local development environment using Docker and Docker-Compose
Working with MongoDB database
Working with SEQ as a log server
Accompanied by detailed README that explains the project in more detail and also how to get started.

Project Page:

Find the project with detailed README here

Chinook .NET

Chinook .NET is project designed to illustrate how to build a Modular Monolith using the Onion Architecture as a guide. The motivation behind this project is to demonstrate an alternative to starting off with Microservices straight away.

The high-level architecture for this Modular Monolith is illustrated as follows:

There are 3 primary modules, where each module has it's own API. Each module should be constrained (though not a hard constraint for practical reasons) to a single schema within a shared database. Each module is based on the Onion Architecture with influences from Clean Architecture. Therefore, each module can be viewed in terms of the following illustration:

Highlights:

.NET5
Working example of Modular Monolith
Working example of Onion Architecture
Demonstrates how to use MediatR
Demonstrates how to use Redgate Flyway to manage database versioning and migrations
Demonstrates how to setup a local development stack using Docker and Docker-Compose
Accompanied by detailed README that goes into more detail about the project and how to get started

Project Page:

Find the project with detailed README here

Ranker .NET (REST API Guide)

Ranker .NET (REST API Guide) is a guide with the goal of laying down foundational knowledge that is required when speaking about building REST API's. I decided to backup the theory with a practical example written in C#.NET5.

Highlights:

REST Constraints
HATEOAS (Hypermedia As The Engine Of Application State)
Richardson Maturity Model
REST in Practice (Some practical guidelines)
Example project (Written in C# .NET 5) called Ranker
Demonstrates how to setup a local development stack using Docker and Docker-Compose
Accompanied by detailed README that goes into more detail about the project and how to get started. The README also forms the basis for the REST API Guide

Project Page:

Find the project with detailed README here

Conclusion

These projects are very much a labour of love. I create these projects so that I have something to point people to when I am explaining topics. I will continue to build on these projects and improve as I find more time to commit. But I would also like to share knowlege about my other love, namely, javascript/typescript. Therefore, I will most likely be building some React frontends for some of these API projects in the future. I'm also keen on demonstrating how to build a Microservices project by implementing services in both Typescript and C#. Also, people have been asking me if I'd consider producing some online videos about content relating to the aforementioned projects. I'm considering the idea, although my biggest concern is being able to make the time 🤷‍♂️

AWS For .NET

Douglas Minnaar — Wed, 14 Apr 2021 09:48:48 +0000

AWS for .NET Developers

I have created a github project that provides a collection of independent .NET projects written in C# .NET 5 that demonstrate how to integrate with various AWS services using the AWS SDK for dotnet.

I summarise the project in the following sections. However, if you prefer going straight to the code then check out the following links:

AWS Services And Projects Summary

So far, projects have been created for the following AWS Services:

Amazon S3 - An AWS service that provides authentication, authorization, and user management for your web and mobile apps.
- S3 ConsoleApp - Console Application
Amazon SNS - Amazon Simple Notification Service (Amazon SNS) is a web service that coordinates and manages the delivery or sending of messages to subscribing endpoints or clients.
- SNS ConsoleApp - Console Application
Amazon SQS - Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
- SQS ConsoleApp - Console Application
Amazon Cognito - Amazon Simple Notification Service (Amazon SNS) is a web service that coordinates and manages the delivery or sending of messages to subscribing endpoints or clients.
- Cognito Mvc - ASP.NET Core MVC Application
- Cognito Api - ASP.NET Core WebAPI Application
Amazon DynamoDb - Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability.
- DynamoDb ConsoleApp - Console Application

Projects

Cognito Mvc - This project demonstrates how to integrate an ASP.NET Core MVC web application with Amazon Cognito using the Amazon SDK.

The following features have been implemented:
- Signup for a new account
- Confirm Signup (using confirmation code)
- Sign into account
- Sign out of account

Cognito Api - This project demonstrates how to integrate a .NET Core Web Api application with Amazon Cognito using the Amazon SDK.

This Web Api application provides the following endpoints:

Signups
- POST api/signups - Allows an api consumer to signup for a new account
Tokens
- POST api/tokens - Provides JWT access token for valid account credentials
Values
- GET api/values - An unauthorized endpoint that returns a list of values
- GET api/values/123 - An authorized endpoint (Requires JWT access token) that returns a single value
- S3 ConsoleApp - This project demonstrates how to integrate a .NET Core Console application with Amazon S3 using the Amazon SDK.

The following features have been implemented:

List all S3 buckets
Create new S3 bucket
Delete S3 bucket

DynamoDb ConsoleApp - This project demonstrates how to integrate a .NET Core console application with DynamoDb using the Amazon SDK. This application provides the functionality required to both manage DynamoDb tables, and manage the data stored in a DynamoDB table

The following features have been implemented:
- Manage Tables
- Create table
- List and find tables
- Wait for tables to be described (eventually consistent)
- Delete table
- Manage Book Table Data
- Automatically creates Book table
- Add books to Book table
- Update books in Book table
- Delete books from Book table
- List and find books in Book table
SNS ConsoleApp - This project demonstrates how to integrate a .NET Core console application with SNS using the Amazon SDK. This application provides the functionality required to manage SNS topcs, subscriptions, and publications.

The following features have been implemented:
- Manage Topics
- Create topic
- List and find topics
- Delete topic
- Manage Subscriptions
- Create an email subscription
- Cancel a subscription
- List subscriptions
- Manage Publications
- An example showing how to publish a 'Game Ranking' to a 'game-ranking' topic

SQS ConsoleApp - This project demonstrates how to integrate a .NET Core console application with SQS using the Amazon SDK.

The following features have been implemented:
- Manage Queues
- Create queue
- List queues
- Delete queue
- Get queue url
- Manage Game Ranking Queue
- Enqueue game ranking to queue
- Dequeue game rankings from queue

What's Next

I have plans to extend this project by implementing the following features:

Improve existing examples (both code and documentation)
Add additional examples for the current services that I have provisioned for (S3, DynamoDB, SQS, SNS, Cognito)
Add examples for new services

That's all folks!

MFlix - A C# .NET 5 Project

Douglas Minnaar — Tue, 06 Apr 2021 09:01:26 +0000

Intro

I have created a project that demonstrates how to build 3 API's using 3 different API tehnologies, namely GraphQL, gRPC, and REST (HTTP). I have also chosen to use MongoDB as my database.

You can find the project on github. The project has been built and tested on both Linux and Windows 10 environments using Visual Studio Code and .NET 5.

I provide more detail about the project in the project README.

The summary of the project thus far is as follows:

The following diagram provides a high-level view of what I am building.

Initially, I thought about creating this project as part of a series of blog posts. However, I decided to instead build the application first. MFlix is by no means complete, but now is a good time to start sharing the work being done due to a number of interesting aspects that I am working on. In the coming months, I will continue to build out MFlix. Also, if there is interest, I would like to provide a series of blog posts where I discuss some of the more interesting aspects of the project.

About MFlix

I chose MFlix as the name for this project because I am using one of the sample databases provided by MongoDB called MFlix. The MFlix database is composed of collections of movie related data. I provide more detail in the MongoDB section where I explain how to get a copy of the MFlix database. Therefore, because of the database name, and the fact that MFlix is a short and catchy name, I decided to go with MFlix.

The primary purpose of MFlix is to demonstrate how to build gRPC, REST (http), and GraphQL API's using C# and the .NET Framework. In order to do this, I have created a contrived use-case that will allow me to demonstrate how to build different API's for different needs. Each API demonstrates how to address the following requirements:

Error handling
Logging
Caching
Security
Dependency Injection
Configuration
How to do paging, filtering, and sorting via the API

The secondary purpose of MFlix is to demonstrate how to work with different technologies, frameworks, and libraries using the C# .NET Framework. The different topics that will be demonstrated are as follows:

Developing with MongoDB and C#
Preparing a local development environment using Docker and Docker-Compose
Working with MongoDB database
Working with SEQ as a log server

Lastly, I cover a useful set of tools that can be used to test the various API's. Some of the tools covered are as follows:

HTTP API
- Postman
- Insomnia
gRPC API
- gRPCurl
- gRPCUI
- BloomRPC
- Insomnia
GraphQL API

Outro

This was a quick introduction to MFlix. I consider what I have done thus far as my "MVP (Minimum Viable Product)". Depending on the feedback, I may follow up with a series of blog posts and maybe a few videos on topics relating to technology and frameworks used within the project. If you like what you see so far, please star the MFlix project.

Jukenook Db

Douglas Minnaar — Sat, 25 Jul 2020 10:46:56 +0000

I have published a new project on my Github account called Jukenook Db.

Jukenook Db is a reference project that demonstrates how to:

create a PostgreSQL environment using Docker and Docker-Compose
configure database versioning and migration using Flyway

About

Jukenook Db is based on the Chinook Database Project. The Chinook Database Project is a sample database that is available to be used with a number of database engines such as Postgres, SQL Server, and MySQL for example. The Chinook data model represents a digital media store, including tables for artists, albums, media tracks, invoices and customers. To see the full description and supported databases, please visit the Chinook Database Project. All credit goes to the Chris Woodruff Chinook Database Project.

The reason for creating this project is due to a continuous need for a consistent database structure that I can use when experimenting with different application development ideas. I searched for a data model that is simple, yet has the potential to evolve into something more sophisticated over time. After investigating a number of different options, I decided on the Chinook Database Project. I have taken the original Chinook Database Project and changed it to work specifically with a Postgres database engine using Flyway migrations. I will also continue to evolve Jukenook Db as I experiment with different ideas.

Why The Name

Firstly, I chose not to keep the original name 'Chinook (Chinook Database Project)'. This is primarily due to reasons involving the evolution of the database model over time.

Secondly, Because the primary database concept is focused around music, I decided to take inspiration from a musical Jukebox. Therefore, I combined the naming of Jukebox and Chinook to form the new name 'Jukenook'. A 'nook' also happens to be a corner or recess, and I think that makes a good place to locate a Jukebox 🤔

Purpose

Primary Objective

The primary objective for this project is to provide a Postgres database that can be easily deployed into one's local development environment.

Secondary Objectives

Show how to use Flyway to manage database versioning and migration
Show how to setup a Postgres database using Docker and Docker-Compose
- Defining docker-compose file to create local Postgres environment using the Postgres 12 and pgAdmin
- Defining docker-compose files that provide an easy way to work with Flyway using Flyway Docker Image

Supported Platforms

This project has been successfully tested on the following platforms:

Ubuntu 20.04 - Ubuntu is an open source software operating system that runs from the desktop, to the cloud, to all your internet connected things.
Windows 10 Professional (2004) - Latest version of Windows 10 operating system. For this particular platform I used Powershell Command Line to execute project tasks.
Windows Subsystem For Linux 2 - WSL 2 is based on a new architecture that provides full Linux binary application compatibility and improved performance. WSL 2 is powered by a real Linux kernel in a lightweight virtual machine that boots in under two seconds. Specifically, the Ubuntu 20.04 distribution was used.

General Tools

The following list of tools are required in order to run this project correctly:

Docker - Docker is a computer program that performs operating-system-level virtualization also known as containerization. In other words it allows one to containerize applications.
Docker-Compose - Compose is a tool for defining and running multi-container Docker applications.
Flyway - Version control for your database by making database migrations repeatable and easy
Postgres - PostgreSQL, also known as Postgres, is a free and open-source relational database management system emphasizing extensibility and technical standards compliance. It is designed to handle a range of workloads, from single machines to data warehouses or Web services with many concurrent users.
pgAdmin - Open Source administration and development platform for PostgreSQL
Visual Studio Code - Visual Studio Code is a source code editor developed by Microsoft for Windows, Linux and macOS. It includes support for debugging, embedded Git control, syntax highlighting, intelligent code completion, snippets, and code refactoring.
Npm - A package manager for the Javascript programming language. In this project it is used as a task runner to simplify running various infrastructural tasks

Getting Started

In order to successfully run the Jukenook database project, you will require the following tools and frameworks to be installed:

For this project, every database definition will be described using a number of database migration scripts. The database migration scripts are defined as per the Flyway database versioning and migration tool guidelines. Flyway is an open-source database migration tool that favors simplicity and convention over configuration. For more information, please see the following links:

Get the repository

git clone https://github.com/drminnaar/jukenook-db.git

Manage Database

I have created docker-compose files to help with the management of bringing up and taking down a PostgreSQL database environment. I also provide docker-compose files to help manage the database migrations and versioning. For example:

docker-compose.yml - Defines the PostgreSQL and pgAdmin services
docker-compose-dev-flyway:info.yml - Runs Flyway migration to obtain information about migrations
docker-compose-dev-flyway:migrate.yml - Runs Flyway migration to do actual database migrations
docker-compose-dev-flyway:validate.yml - - Runs Flyway migration to validate migrations

In order to make it simpler to manage the environment, I have created an NPM package file wherein I have defined a number of tasks. For example:

"scripts": {
    "dev-db:up": "docker-compose up -d",
    "dev-db:down": "docker-compose down --volumes && docker-compose -f ./flyway/docker-compose-dev-flyway:info.yml -f ./flyway/docker-compose-dev-flyway:validate.yml -f ./flyway/docker-compose-dev-flyway:migrate.yml down",
    "dev-db:migrate": "docker-compose -f ./flyway/docker-compose-dev-flyway:migrate.yml up",
    "dev-db:info": "docker-compose -f ./flyway/docker-compose-dev-flyway:info.yml up",
    "dev-db:validate": "docker-compose -f ./flyway/docker-compose-dev-flyway:validate.yml up",
    "start": "npm run dev-db:down && npm run dev-db:up"
  }

I generally prefer using NPM as a task runner unless I have more advanced requirements.

Spinning Up

This step will create an empty Jukenook database running in a docker container. The docker-compose.yml file is used to:

create a Postgres12 (running on Alpine Linux) database in a docker container
load pgAdmin application via docker container. The pgAdmin application is accessible at http://localhost:8080

$ npm run dev-up

> jukenook-db@1.0.0 dev-db:up /home/dmin/projects/jukenook/jukenook-db
> docker-compose up -d

Creating network "jukenook_net" with the default driver
Creating jukenook_db    ... done
Creating jukenook_admin ... done

Tearing Down

This step will:

stop and remove all related docker containers
remove all related docker volumes
remove all related docker networks

$ npm run dev-down

> jukenook-db@1.0.0 dev-db:down /home/dmin/projects/jukenook/jukenook-db
> docker-compose down --volumes && docker-compose -f ./flyway/docker-compose-dev-flyway:info.yml -f ./flyway/docker-compose-dev-flyway:validate.yml -f ./flyway/docker-compose-dev-flyway:migrate.yml down

Stopping jukenook_admin ... done
Stopping jukenook_db    ... done
Removing jukenook_admin ... done
Removing jukenook_db    ... done
Removing network jukenook_net
Network jukenook_net is external, skipping

Manage Database Migrations

For the following commands, the Flyway database versioning and migration tool is used.

Get Database Migrations Info

This command will return a list of migrations with relevant information like the migration state.

$ npm run dev-db:info

> jukenook-db@1.0.0 dev-db:info /home/dmin/projects/jukenook/jukenook-db
> docker-compose -f ./flyway/docker-compose-dev-flyway:info.yml up

Creating jukenook_flyway ... done
Attaching to jukenook_flyway
Flyway Community Edition 6.4.4 by Redgate
Database: jdbc:postgresql://jukenook_db/jukenook (PostgreSQL 12.3)
Schema version: << Empty Schema >>
+------------+---------+------------------------------+------+--------------+---------+
| Category   | Version | Description                  | Type | Installed On | State   |
+------------+---------+------------------------------+------+--------------+---------+
| Versioned  | 1.1     | Create operations schema     | SQL  |              | Pending |
| Versioned  | 1.2     | Create employee table        | SQL  |              | Pending |
| Versioned  | 1.3     | Create catalog schema        | SQL  |              | Pending |
| Versioned  | 1.4     | Create genre table           | SQL  |              | Pending |
| Versioned  | 1.5     | Create artist table          | SQL  |              | Pending |
| Versioned  | 1.6     | Create album table           | SQL  |              | Pending |
| Versioned  | 1.7     | Create media type table      | SQL  |              | Pending |
| Versioned  | 1.8     | Create track table           | SQL  |              | Pending |
| Versioned  | 1.9     | Create playlist table        | SQL  |              | Pending |
| Versioned  | 1.10    | Create composition table     | SQL  |              | Pending |
| Versioned  | 1.11    | Create sales schema          | SQL  |              | Pending |
| Versioned  | 1.12    | Create customer table        | SQL  |              | Pending |
| Versioned  | 1.13    | Create invoice table         | SQL  |              | Pending |
| Versioned  | 1.14    | Create invoice line table    | SQL  |              | Pending |
| Repeatable |         | 001 Install extensions       | SQL  |              | Pending |
| Repeatable |         | 002 Insert genre data        | SQL  |              | Pending |
| Repeatable |         | 003 Insert media type data   | SQL  |              | Pending |
| Repeatable |         | 004 Insert artist data       | SQL  |              | Pending |
| Repeatable |         | 005 Insert album data        | SQL  |              | Pending |
| Repeatable |         | 006 Insert track data        | SQL  |              | Pending |
| Repeatable |         | 007 Insert employee data     | SQL  |              | Pending |
| Repeatable |         | 008 Insert customer data     | SQL  |              | Pending |
| Repeatable |         | 009 Insert invoice data      | SQL  |              | Pending |
| Repeatable |         | 010 Insert invoice line data | SQL  |              | Pending |
+------------+---------+------------------------------+------+--------------+---------+

Validate Database Migrations

This command will validate pending migrations and return information relating to the validity thereof.

$ npm run dev-db:validate

> jukenook-db@1.0.0 dev-db:validate /home/dmin/projects/jukenook/jukenook-db
> docker-compose -f ./flyway/docker-compose-dev-flyway:validate.yml up

Recreating jukenook_flyway ... done
Attaching to jukenook_flyway
Flyway Community Edition 6.4.4 by Redgate
Database: jdbc:postgresql://jukenook_db/jukenook (PostgreSQL 12.3)
ERROR: Validate failed:
Detected resolved migration not applied to database: 1.1
Detected resolved migration not applied to database: 1.2
Detected resolved migration not applied to database: 1.3
Detected resolved migration not applied to database: 1.4
Detected resolved migration not applied to database: 1.5
Detected resolved migration not applied to database: 1.6
Detected resolved migration not applied to database: 1.7
Detected resolved migration not applied to database: 1.8
Detected resolved migration not applied to database: 1.9
Detected resolved migration not applied to database: 1.10
Detected resolved migration not applied to database: 1.11
Detected resolved migration not applied to database: 1.12
Detected resolved migration not applied to database: 1.13
Detected resolved migration not applied to database: 1.14
Detected resolved repeatable migration not applied to database: 001 Install extensions
Detected resolved repeatable migration not applied to database: 002 Insert genre data
Detected resolved repeatable migration not applied to database: 003 Insert media type data
Detected resolved repeatable migration not applied to database: 004 Insert artist data
Detected resolved repeatable migration not applied to database: 005 Insert album data
Detected resolved repeatable migration not applied to database: 006 Insert track data
Detected resolved repeatable migration not applied to database: 007 Insert employee data
Detected resolved repeatable migration not applied to database: 008 Insert customer data
Detected resolved repeatable migration not applied to database: 009 Insert invoice data
Detected resolved repeatable migration not applied to database: 010 Insert invoice line data

Run Database Migrations

This command will run the actual migrations on the database.

$ npm run dev-db:migrate

> jukenook-db@1.0.0 dev-db:migrate /home/dmin/projects/jukenook/jukenook-db
> docker-compose -f ./flyway/docker-compose-dev-flyway:migrate.yml up

Recreating jukenook_flyway ... done
Attaching to jukenook_flyway
Flyway Community Edition 6.4.4 by Redgate
Database: jdbc:postgresql://jukenook_db/jukenook (PostgreSQL 12.3)
Successfully validated 24 migrations (execution time 00:00.264s)
Creating Schema History table "public"."flyway_schema_history" ...
Current version of schema "public": << Empty Schema >>
Migrating schema "public" to version 1.1 - Create operations schema
Migrating schema "public" to version 1.2 - Create employee table
Migrating schema "public" to version 1.3 - Create catalog schema
Migrating schema "public" to version 1.4 - Create genre table
Migrating schema "public" to version 1.5 - Create artist table
Migrating schema "public" to version 1.6 - Create album table
Migrating schema "public" to version 1.7 - Create media type table
Migrating schema "public" to version 1.8 - Create track table
Migrating schema "public" to version 1.9 - Create playlist table
Migrating schema "public" to version 1.10 - Create composition table
Migrating schema "public" to version 1.11 - Create sales schema
Migrating schema "public" to version 1.12 - Create customer table
Migrating schema "public" to version 1.13 - Create invoice table
Migrating schema "public" to version 1.14 - Create invoice line table
Migrating schema "public" with repeatable migration 001 Install extensions
Migrating schema "public" with repeatable migration 002 Insert genre data
Migrating schema "public" with repeatable migration 003 Insert media type data
Migrating schema "public" with repeatable migration 004 Insert artist data
Migrating schema "public" with repeatable migration 005 Insert album data
Migrating schema "public" with repeatable migration 006 Insert track data
Migrating schema "public" with repeatable migration 007 Insert employee data
Migrating schema "public" with repeatable migration 008 Insert customer data
Migrating schema "public" with repeatable migration 009 Insert invoice data
Migrating schema "public" with repeatable migration 010 Insert invoice line data
Successfully applied 24 migrations to schema "public" (execution time 00:02.983s)

Connect Using pgAdmin

As part of the database setup and configuration in the docker-compose file. The web based PosrgreSQL database management tool is also run within a container.

Step 1 - Open pgAdmin Tool

Open browser and enter the address 'http://localhost:8080' to access pgAdmin
For username, enter 'admin@example.com'
For password, enter 'password'
Logon

You should be logged in with the following view open

Step 2 - Create Server Group

Create a server group called 'jukenook'

Step 3 - Create new server

Create a server called 'jukenook-docker'
Select the 'jukenook' server group

Step 4 - Provide Connection Details

Select 'Connection' tab
For host, because we are using docker to run pgAdmin, we need to obtain the IP Address of the Postgres docker container. Therefore, open up a terminal and type the following commands:

  docker container inspect -f '{{ .NetworkSettings.Networks.jukenook_net.IPAddress }}' jukenook_db

  -- Result
  172.18.0.2 (you may recieve a different IP Address)

Enter the connection details
- Host - see above to obtain IPAddress
- Username - specified in docker-compose.yml file
- Password - specified in docker-compose.yml file
- Port - leave as the default 5432
- Role - enter 'admin'. The admin role is created as part of database initialization script 'create-jukenook-db (defined in entrypoint-scripts folder)'.
Select 'Save' connection

After saving connection, you should see a view similar to the following:

Step 5 - Execute a Query

Create a new query window by selecting the 'Query Tool' button.
Enter a query and execute

Entrypoint Scripts

You may be wondering where I create the database and configure users and roles. I make use of entrypoint scripts to perform onceoff database initialization. For example, I currently have a single entrypoint script called 'create-jukenook-db' that looks as follows:

#!/bin/bash
set -e

psql -v ON_ERROR_STOP=1 --username "postgres" --dbname "postgres" <<-EOSQL
    -- create database
    CREATE DATABASE jukenook;

    -- restrict public access
    REVOKE CREATE ON SCHEMA public FROM PUBLIC;
    REVOKE ALL ON DATABASE jukenook FROM PUBLIC;

    -- create admin role
    CREATE ROLE admin;
    GRANT ALL PRIVILEGES ON DATABASE jukenook TO admin;

    -- create readonly role
    CREATE ROLE readonly;
    GRANT CONNECT ON DATABASE jukenook TO readonly;

    -- create readwrite role
    CREATE ROLE readwrite;
    GRANT CONNECT ON DATABASE jukenook TO readwrite;

    -- create admin users
    CREATE USER jukenookboss WITH PASSWORD 'password';
    GRANT admin TO jukenookboss;

    -- create reader user
    CREATE USER reader WITH PASSWORD 'password';
    GRANT readonly TO reader;

    -- create writer user
    CREATE USER writer WITH PASSWORD 'password';
    GRANT readwrite TO writer;
EOSQL

The entrypoint scripts can be found in the 'entrypoint-scripts' folder as follows:

There is also an entry added to the volumes section of the Postgres service in the docker-compose.yml file:

version: '3.5'
services:
  jukenook_db:
    .
    .
    .
    volumes:
      - ./entrypoint-scripts:/docker-entrypoint-initdb.d/
    .
    .
    .

Schema

The database schema is composed of 4 schemas:

music_catalog (yellow)
sales (green)
operations (cyan)
public (orange)

gRPC Guide

Douglas Minnaar — Sun, 09 Feb 2020 09:04:59 +0000

tl;dr

In this guide, I aim to provide the salient points concerning gRPC and Protocol Buffers. If you know what gRPC and Protocol Buffers are, then this guide is not for you. This is a guide for the beginner or the mildly interested.

gRPC is a framework for building API's and is an alternative to JSON and XML based services. gRPC has the following attributes:

a free and open-source framework that was originally developed by Google but is now part of the Cloud Native Computing Foundation (CNCF)
a cross-platform and language agnostic
built on HTTP/2 and supports unary and streaming communication

Protocol Buffers serialize structured data and are used by default by gRPC. Protocol Buffers have the following attributes:

are language agnostic
help structure information for serialization
facilitates code generation
results in small payloads and efficient serialization due to data being binary

Most of the concepts in this guide are language agnostic. However, for now, I only provide examples in C# using .NET Core 3.1. The examples have been tested on both Linux and Windows. All examples have been created using Visual Studio Code with the C# for Visual Studio Code extension. Find the examples here,

Introduction

In a nutshell, gRPC can be summed up as follows,

A high performance, open-source universal RPC framework

-- https://grpc.io

Before delving into gRPC, I'd like to take a step back and understand the problem that gRPC solves. For that I need to discuss the theory of communication fundamentals. Although this is a technical article, I won't be discussing communication theory as applied to computing systems. Instead, I will discuss an example of 2 people speaking.

Let’s imagine 2 people having a discussion in the same room. Both people speak the same language and the communication medium is essentially the same shared space in which the 2 people are speaking. For now we will keep it simple and make the assumption that the communication is unidirectional (communication happens in one way at a time).

The Sender sends a message (request) to the Receiver and waits for a response
The Receiver receives the message (request) and returns a response as another message

Because the 2 people are in the same room speaking the same language, the communication is very simple. There is no need for anything special to enable successful communication. For clarity, the following attributes can be observed for this type of communication.

Same language
Same space (non-distributed)
Same medium

The following diagram further illustrates how the communication occurs for this scenario.

Now imagine that we have 2 people that are in separate rooms, in different countries, and speak different languages. The following attributes contrast the difference to the example used above:

Different language
Different space (distributed)
Different medium

For this example, for successful communication to occur, we will need a mechanism to enable distributed communication. The mechanism will need to define some form of protocol (agreed upon rules) and will also need to do some form of encoding and decoding of messages sent between the Sender and the Receiver. This can be seen as illustrated in the diagram below.

Now imagine that we simply replace the 2 people with 2 computing systems. A Client and a Server. The Client may be a NodeJS Console Application written in Javascript. The Server could be running a C# .NET application. We have 2 separate systems that are running in different data centers and connected via a network. In other words we have a distributed computing system. In order for the Client and the Server to communicate, a mechanism with a protocol will need to be used. Protocol Buffers (Protobuf) and gRPC are just such a mechanism. The diagram below illustrates how our basic communication model can be translated into gRPC clients and server. The clients and server can be completely distributed and running in different data centres in different locations throughout the world.

There have been other solutions that solved this problem too. We are still using some of these at the time of this writing. Http services for example (REST) is still hugely popular. gRPC is simply an alternative to enable communication between distributed services and systems. It's important to remember that although there are different approaches to solving a common problem, the way in which we use these different approaches can be quite different. For example, building and using a Http service is entirely different from building and using a gRPC service.

There is some really good online documentation that describes gRPC and Protocol Buffers.

gRPC - The official gRPC website
Protocol Buffers - The official Google developer guide

Because the online documentation is so good, I won't be repeating some of the great work that already been done. Instead, I'm going to focus on the salient points of gRPC and Protocol Buffers. I will also be providing some examples that illustrate how to get started with gRPC. All examples will be provided using gRPC on .NET Core. For more specific documentation relating to building and consuming gRPC services on .NET Core, please see the following great online documentation:

Protocol Buffers

What?

Protocol Buffers are a mechanism for serializing structured data. They are used by gRPC as the default serialization mechanism.

For more information:

Protocol Buffers - The official Google developer guide
Working with Protocol Buffers - Protocol buffers explained on gRPC website

How?

Because Protocol Buffers are used for serializing structured data, you need to define the structure of the information that you want serialized. We do this by defining protocol buffer message types in a '.proto' file.

For example:

message User {
    string firstName = 1;
    string lastName = 2;
    string email = 3;

    enum AddressType {
        HOME = 0;
        POSTAL = 1;
        WORK = 2;
    }

    message Address {
        string line1 = 1;
        string line2 = 2;
        string region = 3;
        string city = 4;
        string suburb = 5;
        string code = 6;
        AddressType type = 7;        
    }

    repeated Address addresses = 4;
}

Practice Time

1. Setup protoc compiler

The most basic way to compile the 'meetings.proto' file from above is to use the 'protoc' compiler. The protoc compiler is available for a number of different platforms. See the protobuf release page.

If, like me, you're a developer on Debian/Ubuntu and/or Windows, you will find the following online resources useful to install the protobuf compiler.

# For Windows using Chocolatey

  - Install Chocolatey (https://chocolatey.org/install)

  - Install protoc (https://chocolatey.org/packages/protoc) 

    choco install protoc

# For Debian Stretch using apt

  - Install protoc (https://packages.debian.org/stretch/protobuf-compiler)

    sudo apt install protobuf-compiler

# For Ubuntu Bionic using apt

  - Install protoc (https://launchpad.net/ubuntu/bionic/+package/protobuf-compiler)

    sudo apt install protobuf-compiler

2. Create 'meetings.proto' file

syntax = "proto3";

package examples;

message User {
    string firstName = 1;
    string lastName = 2;
    string email = 3;

    enum AddressType {
        HOME = 0;
        POSTAL = 1;
        WORK = 2;
    }

    message Address {
        string line1 = 1;
        string line2 = 2;
        string region = 3;
        string city = 4;
        string suburb = 5;
        string code = 6;
        AddressType type = 7;        
    }

    repeated Address addresses = 4;
}

message Meeting {
    repeated User users = 1;
}

3. Compile 'meetings.proto' file

# For Python
protoc -I . --python_out=. ./meetings.proto

# For Node/Javascript
protoc -I . --js_out=. ./meetings.proto

# For C#
protoc -I . --csharp_out=. ./meetings.proto

The final step is to use the generated code with your programming language of choice. See more tutorials here:

gRPC

gRPC is a free open-source RPC (Remote Procedure Call) framework for building distributed services. According to Wikipedia, RPC can be summarized as follows:

In distributed computing, a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction. That is, the programmer writes essentially the same code whether the subroutine is local to the executing program, or remote. This is a form of client–server interaction (caller is client, executor is server), typically implemented via a request–response message-passing system.

-- Wikipedia

gRPC uses Protocol Buffers for communications. Furthermore, gRPC is summarized as follows:

Designed by Google
Opensourced in 2015 and now a Cloud Native Computing Foundation incubating project
Ample language support
- C++
- Java
- Python
- GO
- C#
- Node.js
- Android Java
- Dart
A client can execute a method on a server application (on a different machine) as if it was a local object
A gRPC service is defined by an interface composed of services (methods with parameters and return types) and messages (properties or data with specified types)
The interface is implemented by the server and runs as a service that accepts calls from remote clients
The client uses a stub (exact representation of the interface used by the server) to make remote calls to the server
Servers and clients are platform and language agnostic. This means that one can have a server implemented in Java, but the server can be used by clients that have been implemented in other languages/platforms like Python, Node, C# etc.
Uses Protocol Buffers by default. The protocol buffers can be used as both Interface Definition Language and the underlying message interchange format.

Why Should You Care About gRPC?

I'm going to share a very opinionated perspective (my opinion) on why you should learn about gRPC.

1. I think that gRPC is a natural fit for building API's. Because gRPC can help make connecting, executing and debugging distributed systems as easy as making local function calls, it doesn't feel like you're doing anything different to your usual code flow. And it's this ease of use and simplicity that I think makes it feel more natural and easy to use.

2. gRPC is renowned for it's efficiency (serialization), speed and low latency. Therefore I think it is well suited to building Microservices.

3. For discussion purposes, I list 3 types of API's that are typically developed:

Internal - This means that your API's are accessible from anywhere within a service boundary of your choosing. In other words, internal API's are meant for internal consumption on your private network only.
Partner - Partner API's are sort of internal and sort of public. But they are intended as integration points between your applications and services with your Partners applications and services. Typically these API's would be locked down using techniques like VPN and/or IP whitelisting.
Public - Anyone on the public internet can access the API

With the 3 types of API's in mind, I think that initially gRPC is ideal for internal API's. As more companies and developers get hooked on gRPC, I think that the mass adoption will begin to drive the development of gRPC services for Partner and Public API's too.

4. Having the skills to build gRPC services will serve you well into the future. I think that the popularity of gRPC will continue to grow. Below, I provide a google trend on gRPC for the past 5 years (gRPC was open-sourced in 2015). Note the steady upward trend.

View gRPC trend here

Tools

Most developers that work with REST or HTTP services are familiar with the excellent [Postman] tool. Postman is the ideal client tool for interacting with HTTP services. Unfortunately, Postman does not offer support for gRPC .... yet. Luckily for all of us, there is one gRPC client tool worth mentioning. And that tool is bloomRPC.

bloomRPC

GUI Client for gRPC Services

Features

Native gRPC calls
Unary Calls and Server Side Streaming Support
Client side and Bi-directional Streaming
Automatic Input recognition
Multi tabs operations
Metadata support
Persistent Workspace
Request Cancellation

Please checkout the bloomRPC repository and while you're at it, please give them a star. It is well deserving :)

Other

.NET Core 3.1 SDK - .NET Core is a cross-platform version of .NET, for building apps that run on Linux, macOS, and Windows
Visual Studio Code - Visual Studio Code is a source-code editor developed by Microsoft for Windows, Linux and macOS
C# for Visual Studio Code - C# for Visual Studio Code (powered by OmniSharp)
vscode-proto3 && vscode-proto3-ext - Protobuf 3 support for Visual Studio Code

Examples

Prerequisites

Tested On

Messaging Example

This example is a simple messaging application that demonstrates how to create a .NET Core gRPC Client and gRPC Server.

By the end of this example, you will have a gRPC Client Console application written in C# that is able to send and receive messages to and from a gRPC Server Console application written in C#. The following concepts will be demonstrated:

Create a gRPC Client (C# Console Application)
Create a gRPC Server (C# Console Application)
Send messages from gRPC Client to gRPC Server

1. Create Solution

mkdir grpc-messaging
cd grpc-messaging
dotnet new sln -n Messaging

2. Create 'messaging.proto' file

# for linux
touch messaging.proto

# for powershell
New-Item -Name messaging.proto

Define 'messaging.proto' file

syntax = "proto3";

package messaging;

option csharp_namespace = "Messaging";

message MessageRequest {
    string message = 1;
}

message MessageResponse {
    string message = 1;
}

service Messenger {
    rpc Message (MessageRequest) returns (MessageResponse);
}

3. Create gRPC Server

3.1 Create the Server Console application

dotnet new console -n Messaging.ServerApp
dotnet sln Messaging.sln add Messaging.ServerApp

3.2 Add packages required to create gRPC Server

cd Messaging.ServerApp
dotnet add package gRPC
dotnet add package gRPC.Tools
dotnet add package Google.Protobuf

dotnet list package

Project 'Messaging.ServerApp' has the following package references
   [netcoreapp3.1]:
   Top-level Package      Requested   Resolved
   > Google.Protobuf      3.11.3      3.11.3
   > gRPC                 2.27.0      2.27.0
   > gRPC.Tools           2.27.0      2.27.0

3.3 Create a 'Services' folder

mkdir Services

3.4 Edit Messaging.ServerApp.csproj file

We need to add the following 'ItemGroup' to Messaging.ServerApp.csproj to enable the appropriate code generation for our gRPC server.

<ItemGroup>
    <Protobuf Include="../*.proto" GrpcServices="Server" OutputDir="%(RelativePath)Services" CompileOutputs="false" />
</ItemGroup>

- Include="../*.proto" - Include all '.proto' files for code generation
- GrpcServices="Server" - Only generate code relevant to server
- OutputDir="%(RelativePath)Services" - path for generated files
- CompileOutputs="false" - prevents compiling generated files into assembly

The resulting Messaging.ServerApp.csproj file should look as follows:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Protobuf Include="../*.proto" GrpcServices="Server" OutputDir="%(RelativePath)Services" CompileOutputs="false" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Google.Protobuf" Version="3.11.3" />
    <PackageReference Include="gRPC" Version="2.27.0" />
    <PackageReference Include="gRPC.Tools" Version="2.27.0">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
  </ItemGroup>

</Project>

3.5 Build Solution

# Build from the root of solution
cd ..
dotnet build

After the build, you should see a solution structure as follows. Take note of the 2 generate files in the 'Services' folder.

Messaging.cs
MessagingGrpc.cs

3.6 Create 'MessengerService'

Create a Service that implements the generated gRPC 'MessengerBase' code

using System.Threading.Tasks;
using Grpc.Core;

namespace Messaging.ServerApp
{
    public sealed class MessengerService : Messenger.MessengerBase
    {
        public override Task<MessageResponse> Message(MessageRequest request, ServerCallContext context)
        {
            return Task.FromResult(new MessageResponse
            {
                Message = $"This is your friendly gRPC Server. Received message: '{request.Message}'"
            });
        }
    }
}

3.7 Update 'Program.cs' to run server

using System;
using System.IO;
using System.Threading.Tasks;
using Grpc.Core;

namespace Messaging.ServerApp
{
    internal sealed class Program
    {
        internal static async Task Main(string[] args)
        {
            const int Port = 50050;

            Server server = null;

            try
            {
                server = new Server
                {
                    Services = { Messenger.BindService(new MessengerService()) },
                    Ports = { new ServerPort("localhost", Port, ServerCredentials.Insecure ) }
                };

                server.Start();

                Console.WriteLine($"Messenger server listening on port {Port}");
                Console.WriteLine("Press any key to stop the server...");
                Console.ReadKey();
            }
            catch(IOException exception)
            {
                Console.WriteLine($"Server failed to start: {exception.Message}");
                throw;
            }
            finally
            {
                if (server != null)
                    await server.ShutdownAsync();
            }
        }
    }
}

3.8 Run the gRPC Server

dotnet run -p Messaging.ServerApp

# You should see the following response
Messenger server listening on port 50050
Press any key to stop the server...

4. Create gRPC Client

4.1 Create the Client Console application

# From the root of solution

dotnet new console -n Messaging.ClientApp
dotnet sln Messaging.sln add Messaging.ClientApp

4.2 Add packages required to create gRPC Client

cd Messaging.ClientApp
dotnet add package gRPC
dotnet add package gRPC.Tools
dotnet add package Google.Protobuf

dotnet list package

Project 'Messaging.ClientApp' has the following package references
   [netcoreapp3.1]:
   Top-level Package      Requested   Resolved
   > Google.Protobuf      3.11.3      3.11.3
   > gRPC                 2.27.0      2.27.0
   > gRPC.Tools           2.27.0      2.27.0

4.3 Create a 'Services' folder

mkdir Services

4.4 Edit Messaging.ClientApp.csproj file

We need to add the following 'ItemGroup' to Messaging.ClientApp.csproj to enable the appropriate code generation for our gRPC Client.

<ItemGroup>
    <Protobuf Include="../*.proto" GrpcServices="Client" OutputDir="%(RelativePath)Services" CompileOutputs="false" />
</ItemGroup>

- Include="../*.proto" - Include all '.proto' files for code generation
- GrpcServices="Client" - Only generate code relevant to Client
- OutputDir="%(RelativePath)Services" - path for generated files
- CompileOutputs="false" - prevents compiling generated files into assembly

The resulting Messaging.ClientApp.csproj file should look as follows:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Protobuf Include="../*.proto" GrpcServices="Client" OutputDir="%(RelativePath)Services" CompileOutputs="false" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Google.Protobuf" Version="3.11.3" />
    <PackageReference Include="gRPC" Version="2.27.0" />
    <PackageReference Include="gRPC.Tools" Version="2.27.0">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
  </ItemGroup>

</Project>

4.5 Build Solution

# Build from the root of solution
cd ..
dotnet build

After the build, you should see a solution structure as follows. Take note of the 2 generate files in the 'Services' folder.

Messaging.cs
MessagingGrpc.cs

4.6 Update 'Program.cs' to run Client

using Grpc.Core;
using Messaging;
using System;
using System.Threading.Tasks;

namespace Messaging.ClientApp
{
    internal sealed class Program
    {
        internal static async Task Main(string[] args)
        {
            Channel channel = new Channel("127.0.0.1:50050", ChannelCredentials.Insecure);

            var client = new Messenger.MessengerClient(channel);
            var reply = client.Message(new MessageRequest { Message = "These are not the droids you are looking for ..." });
            Console.WriteLine("Message: " + reply.Message);

            await channel.ShutdownAsync();

            Console.WriteLine("Press any key to exit...");
            Console.ReadKey();
        }
    }
}

4.7 Run the gRPC Client

dotnet run -p Messaging.ClientApp

# You should see the following response
Message: This is your friendly gRPC Server. Received message: 'These are not the droids you are looking for ...'
Press any key to exit...

Solution

The full solution can be found here.

Calculator Example

Now it's your turn. Create a gRPC Calculator service that provides the following arithmetic operations:

Add
Subtract
Multiply
Divide

Solution

The full solution can be found here.

Proto File

syntax = "proto3";
package calculators;
option csharp_namespace = "Calculators";

message OperationRequest {
    repeated double operands = 1;
}

message OperationResponse {
    double result = 1;
}

message DivisionRequest {
    double dividend = 1;
    double divisor = 2;
}

message DivisionResponse {
    double result = 1;
}

service Calculator {
    rpc Add (OperationRequest) returns (OperationResponse);
    rpc Subtract (OperationRequest) returns (OperationResponse);
    rpc Multiply (OperationRequest) returns (OperationResponse);
    rpc Divide (DivisionRequest) returns (DivisionResponse);
}

gRPC Calculator Server

/// ########################
/// CalculatorService
/// ########################

using Grpc.Core;
using System.Linq;
using System.Threading.Tasks;

namespace Calculators.ServerApp
{
    public sealed class CalculatorService : Calculator.CalculatorBase
    {
        public override Task<OperationResponse> Add(OperationRequest request, ServerCallContext context)
        {
            return Task.FromResult(new OperationResponse
            {
                Result = request.Operands.Sum()
            });
        }

        public override Task<DivisionResponse> Divide(DivisionRequest request, ServerCallContext context)
        {
            return Task.FromResult(new DivisionResponse
            {
                Result = request.Dividend / request.Divisor
            });
        }

        public override Task<OperationResponse> Multiply(OperationRequest request, ServerCallContext context)
        {
            return Task.FromResult(new OperationResponse
            {
                Result = request.Operands.Aggregate((acc, operand) => acc *= operand)
            });
        }

        public override Task<OperationResponse> Subtract(OperationRequest request, ServerCallContext context)
        {
            return Task.FromResult(new OperationResponse
            {
                Result = request.Operands.Aggregate((acc, operand) => acc -= operand)
            });
        }
    }
}

/// ########################
/// Program
/// ########################

using System;
using System.IO;
using System.Threading.Tasks;
using Grpc.Core;

namespace Calculators.ServerApp
{
    internal sealed class Program
    {
        internal static async Task Main(string[] args)
        {
            const int Port = 50050;

            Server server = null;

            try
            {
                server = new Server
                {
                    Services = { Calculator.BindService(new CalculatorService()) },
                    Ports = { new ServerPort("localhost", Port, ServerCredentials.Insecure) }
                };

                server.Start();

                Console.WriteLine($"Calculator server listening on port {Port}");
                Console.WriteLine("Press any key to stop the server...");
                Console.ReadKey();
            }
            catch (IOException exception)
            {
                Console.WriteLine($"Server failed to start: {exception.Message}");
                throw;
            }
            finally
            {
                if (server != null)
                    await server.ShutdownAsync();
            }
        }
    }
}

gRPC Calculator Client

/// ########################
/// Program
/// ########################

using System;
using System.Threading.Tasks;
using Grpc.Core;

namespace Calculators.ClientApp
{
    internal sealed class Program
    {
        internal static async Task Main(string[] args)
        {
            Channel channel = new Channel("127.0.0.1:50050", ChannelCredentials.Insecure);

            var client = new Calculator.CalculatorClient(channel);
            var operands = new double[5] { 1D, 2D, 3D, 4D, 5D };

            var addRequest = new OperationRequest();
            addRequest.Operands.AddRange(operands);

            var subtractRequest = new OperationRequest();
            subtractRequest.Operands.AddRange(operands);

            var multiplicationRequest = new OperationRequest();
            multiplicationRequest.Operands.AddRange(operands);

            var divisionRequest = new DivisionRequest
            {
                Dividend = 45,
                Divisor = 5
            };

            var sum = await client.AddAsync(addRequest);
            var difference = await client.SubtractAsync(subtractRequest);
            var product = await client.MultiplyAsync(multiplicationRequest);
            var division = await client.DivideAsync(divisionRequest);

            Console.WriteLine($"Sum: {sum.Result}, Difference: {difference.Result}, Product: {product.Result}, Division: {division.Result}");

            await channel.ShutdownAsync();

            Console.WriteLine("Press any key to exit...");
            Console.ReadKey();
        }
    }
}

Where To From Here?

I've only introduced the gRPC and Protocol Buffer fundamentals in this guide. I'd recommend doing some online digging for information relating to your platform/language of choice. For example, I only spoke about Unary communication in this guide. Be sure to do some research on other alternatives like:

Server streaming
Client streaming
Bi-directional streaming

I'm looking forward to feedback from the community to see if there is any way that I can help developers learn more about gRPC. I will then most likely explore a number of follow-up topics for future posts.

gRPC for the win 🏆

REST API Guide

Douglas Minnaar — Sun, 02 Feb 2020 07:23:26 +0000

tl;dr

This is a guide with the goal of laying down foundational knowledge that is required when speaking about building REST API's. The following topics are covered:

REST Constraints
Richardson Maturity Model
REST in Practice (Some practical guidelines)
Example project (Written in C# using .Net Core 3.1) called Ranker

The main points that I would like to summarize with regards to REST are listed as follows:

REST IS an architectural style used to describe web architecture
REST IS protocol agnostic
REST IS about web architecture (REST != API)
REST IS NOT a design pattern
REST IS NOT a standard. However standards can be used to implement REST.

1. REST Fundamentals

This sections covers REST essentials. The goal of this section is to make the reader comfortable with the notion of REST. It is also intended to provide the minimum required theory to start talking about REST and building HTTP services that incorporate a REST architectural style.

Introduction

REST (REpresentational State Transfer) is an architectural style that was defined by Roy Thomas Fielding in his PhD dissertation "Architectural Styles and the Design of Network-based Software Architectures".

According to Fielding,

The name “Representational State Transfer” is intended to evoke an image of how a well-designed Web application behaves: a network of web pages (a virtual state-machine), where the user progresses through the application by selecting links (state transitions), resulting in the next page (representing the next state of the application) being transferred to the user and rendered for their use.

(Fielding, 2000) pg 109

Why REST?

If you're someone that builds HTTP services for distributed systems, then understanding and applying REST principles will help you build services that are more:

scalable
reliable
flexible
portable

By building services based on REST principles, one is effectively building services that are more web friendly. This is because REST is an architectural style that describes web architecture.

REST Architectural Constraints

Fielding describes REST as a hybrid style that is derived from several network-based architectural styles (Chapter 3) combined with a number of additional constraints. In this section, the six architectural constraints as applied to REST are discussed. The key takeaway is that these constraints encourage design that will result in applications that easily scale, are faster, and more reliable.

The 6 architectural REST constraints are as follows:

1. Client-Server

A guiding principle of Client-Server is the separation of concerns. It's all about achieving high cohesion and loose coupling in order to improve portability and flexibility. It also allows systems to evolve independently of each other. As can be seen by the diagram below, a Client sends a request, and a Server receives the request.

2. Statelessness

A Server must not store any state during communications. All information required to understand a request must be contained within the Request. Therefore, every Request should be able to execute on its own and be self-contained. Also, a Client must maintain it's own state. The benefit of this approach is as follows:

Visibility - Everything required to understand the Request is within the Request. This makes monitoring a request easier.
Reliability - Recovering from failures is easier because the Server does not need to track/rollback/commit state because all the state is essentially captured within the message. If a Request fails, it can be as simple as resending the Request.
Scalability - Because there is no need to manage state and resources between requests, and because all Requests are isolated,scalability is improved and simplified.
Aligned with web architecture (the internet is designed this way)

A disadvantage of this approach is that it decreases network efficiency because the Requests need to contain all the information required for that interaction. The more information, the larger the Request size, and therefore the more bandwidth is used. This will have a negative effect on latency as well.

3. Cache

The primary reason for the Cache constraint is to improve network efficiency. As noted above in the Stateless constraint, the size of Requests can decrease network efficiency due to the need for more bandwidth. Through caching, it is possible to reduce and sometimes remove the need for a Client to interact with the Server. In other words it's possible to reduce and/or eliminate the need for Requests. Therefore, the Cache constraint states that a Server must include additional data in the response to indicate to the client whether the Request is cacheable and for how long. A network Client can then decide the appropriate action based on provided cache information in Response.

Caching can improve performance. However, it comes with a number of disadvantages that impact the reliability of the system. For example:

Data Integrity - Response data could be inaccurate due to stale or expired data
Complexity - The implementation and use of caching mechanisms is renowned for it's complexity in the Computer Science world

4. Uniform Interface

At the core of this constraint is the principle of generality which is closely related to the principle of anticipation. It stems from the fact that it is impossible to build the exact required interface for all network clients of a server service. Therefore, by providing a generic interface, one is able to provide a simplified interface with higher visibility that is able to satisfy the requirements of more clients. A disadvantage of this approach is that because the interface is so general, one is not able to satisfy specific client requirements. In other words, providing a generic interface can lead to a sub-optimal interface for many clients.

There are four additional constraints that form part of the Uniform Interface and are listed as follows:

Identification of resources

A key abstraction of REST is a resource. According to Fielding (Resources and Resource Identifiers), a resource is any information that can be named. Furthermore, I personally like to think of a resource as a "Noun".

Noun - a word (other than a pronoun) used to identify any of a class of people, places, or things ( common noun ), or to name a particular one of these ( proper noun ).

It is also better to think of a single resource as a collection of resources. For example, if we were to provide an API to allow a Client to submit or retrieve a "rating", one would typically identify the resource as follows:
```
 GET /ratings
```
Generally, there should only be a single way to access a resource. But this is more a guideline than a rule.
Manipulation of resources through representations

This constraint states that the client should hold the representation of a resource that has enough information to create, modify or delete a resource. It's important that the representation of a resource is decoupled from the way the resource is identified. A resource can be represented in multiple formats or representations such as JSON, XML, HTML, PNG etc. A client should be able to specify the desired representation of a resource for any interaction with the server. Therefore, a Client can specify to receive a resource in JSON format, but send the resource as input in XML format.

For example:

For the retrieval of an Employees resource, we use XML format
by specifying a "Accept: application/xml" header.
```
 GET /ratings
 Accept: application/xml

 <ratings>
   <rating>
     <id>7337</id>
     <userId>98765</userId>
     <movieId>12345</movieId>
     <score>6</score>
   </rating>
 </ratings>
```
For the creation of an Employees resource, we use
JSON format by specifying a "Content-Type: application/json" header
```
 POST /ratings
 Content-Type: application/json
 {
     "userId": 98765,
     "movieId": 12345,
     "score": 6
 }     
```
Should a specific format not be supported, it is important for the Server to provide an appropriate response to indicate that a specific format is not supported. For example:
- Return a 406 Not Acceptable status code to indicate that the client specified a request with an Accept header format that the Server is unable to fulfill. [See here for more information]
- Return a 415 Unsupported Media Type when a response is specified in an unsupported content type. [See here for more information]
Self descriptive messages

Self descriptive messages enable intermediary communication by allowing intermediary components to transform the content of the message. In other words, the semantics of the message are exposed to the intermediaries. The implication of this constraint is that interactions are stateless, standard methods and media types are used to expose the semantics of message, and responses indicate cacheability.
Hypermedia as the engine of application state (HATEOAS)

A key concept about HATEOAS is that it implies that a Response sent from a Server should include information that informs the Client on how to interact with the Server.

The advantages of HATEOAS are as follows:
- Improves discoverability of resources through published set of links (provided with response)
- Indicates to Clients what actions can be taken next. In other words, without HATEOAS, a Client only has access to the data but no idea about what actions may be taken with that data

5. Layered System

The key principle of this constraint is that the Client cannot make any assumptions that it is communicating directly with the Server. This constraint relates to the Client-Server constraint (discussed above) in such a way that Client and Server are decoupled. Therefore the Client makes no assumptions about any kind of hidden dependencies and this enables us to insert components and entire sub-systems between the Client and the Server. This allows one to add load balancers, DNS, caching servers and security (authentication and authorization) between Client and Server without disrupting the interaction.

Layering allows one to evolve and improve ones architecture to improve scalability and reliability ones system.

6. Code On Demand

This is an optional constraint. The key concept about this constraint is that when a Client makes a request to a resource on a Server, it will receive the resource as well as the code to execute against that resource. The Client knows nothing about the composition of the code and only needs to know how to execute it. Javascript is an example of where this is done.

Richardson Maturity Model

The Richardson Maturity Model is a heuristic maturity model that can be used to better understand how mature a service is in terms of the REST architectural style.

Level 0

Services at this level are described as having a single URI, and using a single HTTP verb (usually POST). This is very characteristic of most Web Services (WS-*) in that this services would have a single URI accepting an HTTP POST request having an XML payload.

Level 1

Services at this level are described as having many URIs with a single HTTP verb. The primary difference between Level 0 and Level 1 is that Level 1 services expose multiple logical resources as opposed to a single resource.

Level 2

Services at this level are described as having many URI-addressable resources. Each addressable resource supports both multiple HTTP verbs and HTTP status codes.

Level 3

Services at this level are like Level 2 services that additionally support Hypermedia As The Engine Of Application State (HATEOAS). Therefore, representations of a resource will also contain links to other resources (the actions that can be performed relating to current resource).

When thinking about the RMM applies to your API, please refrain from thinking in terms of having a Level 2 or Level 3 REST API. According to this model, an API cannot be called a REST API unless it at least satisfies a Level 3 of the RMM. Therefore, it would be better to think of ones API as an HTTP API that satisfies a Level 1,2, or 3 on the RMM.

2. REST in Practice

I have developed a simple Http Api to demonstrate some of the concepts that I discussed in Part 1 of this guide.

Ranker

A REST API guide with and example project written in C# using .Net Core 3.1

I've also started another project that I plan to use to demonstrate various technology concepts like REST API's.

Chinook

A playground for demonstrating concepts such as architecture, design, dotnet core, typescript, react, database and docker

2. Defining A Contract

In this example, we are going to define contracts for 3 types of resources:

Users
Movies
Ratings

There are 5 important aspects to defining a contract:

Naming a resource
Http methods used to interact with resource
Status codes used to describe the state of an interaction
Content Negotiation
Be consistent

2.1 Naming Guidelines

Resources should have names that are represented by nouns and not actions (behaviors)

  # Incorrect naming

  /getUsers
  /getUserById/{userId}

  # Correct Naming

  /users
  /users/{userId}

Resources should be named using plural form

  # Incorrect naming

  /user
  /movie
  /rating

  # Correct naming

  /users
  /movies
  /ratings

Mapping RPC style methods to resources

The naming guidelines seem to suit naming resources very well. However, what happens when one needs to name something that is more a behavior than a resource? For example, let's say we want to compute the average rating for a movie. How would we structure our naming?

  /movies/{movieId}/averageRating

I don't think there is 100% consensus on what the correct naming strategy is for a scenario such as this one. However, when faced with defining a contract for something that feels more about behavior than resources, I like to define contracts based on the outcomes of those behaviors. Therefore, for the example above:

  /averageMovieRatings
  /averageMovieRatings/{movieId}

But what if we try to define a contract for a calculator? This is clearly an example of where defining a contract around a behavior is very difficult and "unnatural" to REST. The reason why it feels unnatural is because REST is an architectural style for describing web architecture. So if you imagined every endpoint as a webpage, then clearly the behaviors for a calculator don't map very well. My suggestion is to use an alternative technology like gRPC if you are building API's that are more about behavior than resources.

Represent hierarchy

  /users/{userId}
  /users/{userId}/ratings
  /users/{userId}/ratings/{ratingId}

  /movies/{movieId}
  /movies/{movieId}/ratings
  /movies/{movieId}/ratings/{ratingId}

Filtering, searching and sorting are not part of naming

For filtering:

  # Incorrect
  /users/firstName/{firstName}

  # Correct
  /users?firstName={firstName}

For searching:

  # Incorrect
  /users/search/{query}

  # Correct
  /users?q={query}

For ordering:

  # Incorrect
  /users/orderBy/{firstName}

  # Correct
  /users?order={firstName}

2.2 Http Methods

Http Method	Request Body	Uri	Response
GET	-	/users	List of users
GET	-	/users/{userId}	Single user
POST	Single user	/users	Single user
PUT	Single user	/users/{userId}	-
PATCH	Json Patch Document for user	/users/{userId}	-
DELETE	-	/users/{userId}	-
HEAD	-	/users	-
HEAD	-	/users/{userId}	-
OPTIONS	-	/users	-

2.3 Status Codes

In this section, a list of commonly used status codes is provided. Status codes help convey meaning in client/server interactions. They also help achieve consistency in terms of defining a contract.

Level 200 - Success

200 Ok - Request succeeded
201 Create - Request succeeded and resource created
204 No Content - Request succeeded and there is no additional content to send in response body

Level 300 - Redirection Responses

301 Moved Permanently - The URL of requested resourced has changed permanently. The new URL is provided in response
302 Found - Indicates that the URI of requested resource changed, and can therefore use the same URI for future requests
304 Not Modified - Used for caching. Indicates that the resource has not changed and that the same cached version can be used

Level 400 - Client Mistake

400 Bad Request - The request could not be understood by the server due to malformed syntax. The client should not repeat the request without modifications
401 Unauthorized - Request failed due to authentication failure
403 Forbidden - Request failed due to authorization failure
404 Not Found - The requested resource could not be found
405 Method Not Allowed - The request method is understood by server but not supported. In other words, the server doesn't have an endpoint supporting requested method.
406 Not Acceptable - When a request is specified in an unsupported content type using the Accept header
409 Conflict - Indicates a conflict in terms of requested resource state. For a POST, it could mean that a resource already exists. For a PUT, it could mean that the state of resource changed thereby making current request data stale.
415 Unsupported Media Type - When a response is specified in an unsupported content type
422 Unprocessable Entity - Indicates the the request was correct and understood by server, but the data contained within request is invalid.

Level 500 - Server Mistake

500 Internal Server Error - Indicates that something went wrong on the server that prevent the server from fulfilling the request.
503 Service Unavailable - Indicates that the server is functional but not able to deliver requested resource. This is usually a result of a server being overloaded, server is under maintenance, or a client side issue relating to DNS server (DND server could be unavailable).
504 Gateway Timeout - Indicates that a proxy server did not receive a timely response from the origin (upstream) server.

2.4 Content Negotiation

Implies the type of representation (Media Type) that will be used for request and response. The Media Type is specified in header of request. Two popular Media Type formats that are used with Http Api's are:

application/json
application/xml

Typically, I would support at least the two aforementioned formats. For any media type format that is not supported, the Api should return a 406 Not Acceptable status code.

Examples:

# Send POST request to create a a new user.
# The request will use 'application/json' as input, but XML in return (application/xml)

POST /users
Accept: application/xml
Content-Type: application/json

{
  "firstName": "Bob",
  "lastName": "TheBuilder"
}

# The response is returned as XML

<User>
  <Id>112233</Id>
  <FirstName>Bob</FirstName>
  <LastName>TheBuilder</LastName>
</User>

3. Example Project

To illustrate some of the topics that have been discussed, I created an example project called Ranker.

Ranker is an API that has been designed by using REST as a guide. In terms of the Richardson Maturity Model, I have implemented all endpoints to be at least a Level 2. However, I have implemented some endpoints to a Level 3 (with HATEOAS). Conceptually, Ranker provides the following features:

Interface to manage Users (with HATEOAS)
Interface to manage Movies
Interface to manage Ratings

In the following sections I provide more detail about the project and how to get started.

Architecture

Although the focus of this example project is to illustrate an implementation of REST, I decided to provide a basic architecture to also illustrate a good separation of concerns so that the Api layer (Controllers) are kept very clean.

I've chosen a architecture based on the Onion Architecture. Below, I provide 2 different views of what equates to exactly the same architecture.

Layered Architecture

Onion Architecture

Primary Responsibility: Provides a distributed interface that gives access to application features

This API has been implemented as a number of HTTP services based on REST guidelines. The API itself is based on an MVC (Model, View, and Controllers) architecture. The Controllers are essentially the public facing API contract.

Infrastructure

Primary Responsibility: Provide the core of the system an interface to the "world".

This layer is all about defining and configuring external dependencies such as:

database access
proxies to other API's
logging
monitoring
dependency injection
- Application

Primary Responsibility: Application logic.

This layer is typically where you would find "Application "Services".

Domain

Primary Responsibility: Enterprise domain logic.

All domain logic relating to domain models and domain services are handled in this layer.

API Contract

The API has been implemented with the Open Api Specification (OAS). Once you have the API up and running, you can browse to the following Url to get access to the OAS Swagger Document.

http://localhost:5000

The Swagger document will look something like below:

Pagination

For this project, any endpoint returning a collection of items has been implemented with paging. Use the following query parameters to control paging:

page - the page number
limit - the number of items per page

Pagination has been implemented in two ways for this example project.

Pagination in Header

  GET http://localhost:5000/v1/movies?page=2&limit=5

  Header: X-Pagination

  {
      "CurrentPageNumber": 2,
      "ItemCount": 9742,
      "PageSize": 5,
      "PageCount": 1949,
      "FirstPageUrl": "http://localhost:5000/v1/movies?page=1&limit=5",
      "LastPageUrl": "http://localhost:5000/v1/movies?page=1949&limit=5",
      "NextPageUrl": "http://localhost:5000/v1/movies?page=3&limit=5",
      "PreviousPageUrl": "http://localhost:5000/v1/movies?page=1&limit=5",
      "CurrentPageUrl": "http://localhost:5000/v1/movies?page=2&limit=5"
  }

Pagination as links (HATEOAS)

  GET http://localhost:5000/v1/users?page=1&limit=1

  {
    .
    .
    .
    "links": [
        {
            "href": "http://localhost:5000/v1/users?page=1&limit=1",
            "method": "GET",
            "rel": "current-page"
        },
        {
            "href": "http://localhost:5000/v1/users?page=2&limit=1",
            "method": "GET",
            "rel": "next-page"
        },
        {
            "href": "",
            "method": "GET",
            "rel": "previous-page"
        },
        {
            "href": "http://localhost:5000/v1/users?page=1&limit=1",
            "method": "GET",
            "rel": "first-page"
        },
        {
            "href": "http://localhost:5000/v1/users?page=610&limit=1",
            "method": "GET",
            "rel": "last-page"
        }
    ]
  }

Filtering

Where practical, I've tried to provide a filter per resource property. I've implemented filtering using 3 techniques:

1. Basic

   // filter users by last name and age

   GET http://localhost:5000/v1/users?last-name=doe&gender=male

2. Range

For numeric resource (and date) properties, I've implemented range filters as follows:

   // Possible input for age could be:
   // age=gt:30
   // age=gte:30
   // age=eq:30
   // age=lt:30
   // age=lte:30

   GET http://localhost:5000/v1/users?age=gte:30

3. Multiple (comma separated values)

   // get a list of movies for the genres animation and sci-fi

   GET http://localhost:5000/v1/movies?genres=animation,sci-fi

Ordering

I've chosen to keep ordering parameters very succinct. Therefore, ordering for a collection of resources may be executed in the following ways:

Order by a single resource property in ascending order

  // order by last name ascending

  GET http://localhost:5000/v1/users?order=last-name

Order by a single resource property in descending order

  // order by age descending

  GET http://localhost:5000/v1/users?order=-age

Order by multiple resource properties using mixed sort orders

Notice that we use comma separated values for the order.

  // order by last-name ascending then by age descending

  GET http://localhost:5000/v1/users?order=last-name,-age

Caching

I have implemented some basic client side caching behavior.

For example:

The following endpoints use response caching where the cache expires after 10 seconds.

GET http://localhost:5000/v1/users

GET http://localhost:5000/v1/movies
GET http://localhost:5000/v1/movies/{movieId}

GET http://localhost:5000/v1/ratings
GET http://localhost:5000/v1/ratings/{ratingId}

The following endpoint uses caching with an ETag.

GET http://localhost:5000/v1/users/{userId}

HATEOAS

The following endpoints have been implemented to return links as part of response.

// Get links available from root

GET http://localhost:5000/v1

[
    {
        "href": "http://localhost:5000/v1",
        "method": "GET",
        "rel": "self"
    },
    {
        "href": "http://localhost:5000/v1/movies",
        "method": "GET",
        "rel": "movies"
    },
    {
        "href": "http://localhost:5000/v1/movies",
        "method": "POST",
        "rel": "create-movie"
    },
    {
        "href": "http://localhost:5000/v1/ratings",
        "method": "GET",
        "rel": "ratings"
    },
    {
        "href": "http://localhost:5000/v1/ratings",
        "method": "POST",
        "rel": "create-rating"
    },
    {
        "href": "http://localhost:5000/v1/users",
        "method": "GET",
        "rel": "users"
    },
    {
        "href": "http://localhost:5000/v1/users",
        "method": "POST",
        "rel": "create-user"
    }
]

// Get as single user, including a list of navigational links

GET http://localhost:5000/v1/users

{
    "userId": 10,
    "age": 30,
    "firstName": "Durham",
    "lastName": "Franks",
    "gender": "male",
    "email": "durhamfranks@kog.com",
    "links": [
        {
            "href": "http://localhost:5000/v1/users/10",
            "method": "DELETE",
            "rel": "delete-user"
        },
        {
            "href": "http://localhost:5000/v1/users/10",
            "method": "GET",
            "rel": "self"
        },
        {
            "href": "http://localhost:5000/v1/users?Page=1&Limit=10",
            "method": "GET",
            "rel": "users"
        },
        {
            "href": "http://localhost:5000/v1/users",
            "method": "OPTIONS",
            "rel": "options"
        },
        {
            "href": "http://localhost:5000/v1/users/10",
            "method": "PATCH",
            "rel": "patch-user"
        },
        {
            "href": "http://localhost:5000/v1/users",
            "method": "POST",
            "rel": "create-user"
        },
        {
            "href": "http://localhost:5000/v1/users/10",
            "method": "PUT",
            "rel": "update-user"
        },
        {
            "href": "http://localhost:5000/v1/users/10/ratings",
            "method": "GET",
            "rel": "ratings"
        }
    ]
}

And for a collection of users (with links), we can use the request below. Please take note of the paging information that is returned as part of response

// Get list of users (with links), and paging links

GET http://localhost:5000/v1/users

{
    "users": [
        {
            "userId": 23,
            "age": 40,
            "firstName": "Michele",
            "lastName": "Jacobs",
            "gender": "female",
            "email": "michelejacobs@kineticut.com",
            "links": [
                {
                    "href": "http://localhost:5000/v1/users/23",
                    "method": "DELETE",
                    "rel": "delete-user"
                },
                {
                    "href": "http://localhost:5000/v1/users/23",
                    "method": "GET",
                    "rel": "self"
                },
                {
                    "href": "http://localhost:5000/v1/users?Page=1&Limit=10",
                    "method": "GET",
                    "rel": "users"
                },
                {
                    "href": "http://localhost:5000/v1/users",
                    "method": "OPTIONS",
                    "rel": "options"
                },
                {
                    "href": "http://localhost:5000/v1/users/23",
                    "method": "PATCH",
                    "rel": "patch-user"
                },
                {
                    "href": "http://localhost:5000/v1/users",
                    "method": "POST",
                    "rel": "create-user"
                },
                {
                    "href": "http://localhost:5000/v1/users/23",
                    "method": "PUT",
                    "rel": "update-user"
                },
                {
                    "href": "http://localhost:5000/v1/users/23/ratings",
                    "method": "GET",
                    "rel": "ratings"
                }
            ]
        },
        {
            "userId": 33,
            "age": 40,
            "firstName": "Barnett",
            "lastName": "Griffith",
            "gender": "male",
            "email": "barnettgriffith@corpulse.com",
            "links": [
                {
                    "href": "http://localhost:5000/v1/users/33",
                    "method": "DELETE",
                    "rel": "delete-user"
                },
                {
                    "href": "http://localhost:5000/v1/users/33",
                    "method": "GET",
                    "rel": "self"
                },
                {
                    "href": "http://localhost:5000/v1/users?Page=1&Limit=10",
                    "method": "GET",
                    "rel": "users"
                },
                {
                    "href": "http://localhost:5000/v1/users",
                    "method": "OPTIONS",
                    "rel": "options"
                },
                {
                    "href": "http://localhost:5000/v1/users/33",
                    "method": "PATCH",
                    "rel": "patch-user"
                },
                {
                    "href": "http://localhost:5000/v1/users",
                    "method": "POST",
                    "rel": "create-user"
                },
                {
                    "href": "http://localhost:5000/v1/users/33",
                    "method": "PUT",
                    "rel": "update-user"
                },
                {
                    "href": "http://localhost:5000/v1/users/33/ratings",
                    "method": "GET",
                    "rel": "ratings"
                }
            ]
        }
    ],
    "links": [
        {
            "href": "http://localhost:5000/v1/users?order=-age&page=1&limit=2",
            "method": "GET",
            "rel": "current-page"
        },
        {
            "href": "http://localhost:5000/v1/users?order=-age&page=2&limit=2",
            "method": "GET",
            "rel": "next-page"
        },
        {
            "href": "",
            "method": "GET",
            "rel": "previous-page"
        },
        {
            "href": "http://localhost:5000/v1/users?order=-age&page=1&limit=2",
            "method": "GET",
            "rel": "first-page"
        },
        {
            "href": "http://localhost:5000/v1/users?order=-age&page=305&limit=2",
            "method": "GET",
            "rel": "last-page"
        }
    ]
}

4. Technology Used

OS

I have developed and tested Ranker on the following Operating Systems.

Ubuntu 18.04 LTS

Ubuntu is an open source software operating system that runs from the desktop, to the cloud, to all your internet connected things.

Windows 10 Professional

In addition to developing Ranker on Windows 10, I have also tried and tested Ranker using Windows Subsystem For Linux. Specifically, I have used [WSL-Ubuntu]. See more about WSL below.

Windows Subsystem For Linux

The Windows Subsystem for Linux lets developers run a GNU/Linux environment -- including most command-line tools, utilities, and applications -- directly on Windows, unmodified, without the overhead of a virtual machine.
Windows Subsystem For Linux 2

NOTE: I have not tested Ranker on WSL2 yet. I mention it here because I want to be clear that I've only tested on WSL.

WSL 2 is a new version of the architecture in WSL that changes how Linux distros interact with Windows. WSL 2 has the primary goals of increasing file system performance and adding full system call compatibility. Each Linux distro can run as a WSL 1, or a WSL 2 distro and can be switched between at any time. WSL 2 is a major overhaul of the underlying architecture and uses virtualization technology and a Linux kernel to enable its new features.

Code

Visual Studio Code

Visual Studio Code is a source code editor developed by Microsoft for Windows, Linux and macOS. It includes support for debugging, embedded Git control, syntax highlighting, intelligent code completion, snippets, and code refactoring.

Visual Studio Community Edition

A fully-featured, extensible, FREE IDE for creating modern applications for Android, iOS, Windows, as well as web applications and cloud services.

Database

Kept things simple and only used an in-memory database

5. Getting Started

Before getting started, the following frameworks must be installed on your machine:

Dotnet Core 3.1

Get The Code

Clone 'ranker' repository from GitHub

# using https
git clone https://github.com/drminnaar/ranker.git

# or using ssh
git clone git@github.com:drminnaar/ranker.git

Build The Code

# change to project root
cd ./ranker

# build solution
dotnet build

Running the API

Run the API from the command line as follows:

# change to project root
cd ./ranker/Ranker.Api

# To run 'Ranker Api' (http://localhost:5000)
dotnet watch run

Open Postman Collection

I have provided a postman collection for the Ranker API. Please find the Postman collection _'Ranker.postman_collection'_at the root of the solution.

NoteWorx - Project 6

Douglas Minnaar — Mon, 17 Sep 2018 07:31:39 +0000

Overview

Over the past few months, I have been building a simple note taking application in 6 different ways. The goal of the project was to build something simple, but build it in different ways, and using different technologies. Up until now, the first 5 NoteWorx projects were all built using Javascript and Javascript frameworks with a MongoDB database. For the sixth NoteWorx project, I decided to use the cross-platform dotnetcore framework, C#, and a PostgreSQL database.

This project is a reference project and will illustrate the following concepts:

Build end-to-end asp.net core web application using C#, ASP.NET Core Identity, and Entity Framework Core.
Create a Cake build script to manage project build.
Use Docker and Docker-Compose to manage containers used to host web application and postgres database.
How to use NPM and Gulp to manage client side assets (css, javascript, etc).
Use PostgreSQL as a database for all note and identity data.

The source code for NoteWorx (Project 6) can be found here

The previous NoteWorx projects are listed as follows:

noteworx-cli-fs

A basic note application that uses a CLI (Command Line Interface) frontend to capture and manage notes, and a file system to store notes

noteworx-cli-mongodb

A basic note application that uses a CLI (Command Line Interface) frontend to capture and manage notes, and mongodb to store notes.

noteworx-cli-mongoose

A basic note application that uses a CLI (Command Line Interface) frontend to capture and manage notes, Mongoose ODM to manage MongoDB interaction, and mongodb to store notes.

noteworx-cli-couchbase

A basic note application that uses a CLI (Command Line Interface) frontend to capture and manage notes, and couchbase as a data store.

noteworx-cli-express-mongodb

A basic note application that uses a CLI (Command Line Interface) frontend to capture and manage notes, an express note management API built using Express, and Mongodb to store notes.

noteworx-expressui-mongodb

A basic note application that uses an Express frontend to capture and manage notes, and mongodb to store notes.

noteworx-react-mongodb

A basic note application that uses React frontend to capture and manage notes, an api written in ExpressJS, and mongodb to store notes.

What is NoteWorx?

NoteWorx is a basic note management web application. This version of NoteWorx (Project 6), has been built using the latest ASP.NET Core 2.1, ASP.NET Core Identity, Entity Framework Core, and Postgresql.

NoteWorx provides the following features:

Add a note
Edit a note
Remove a note
List all notes
Find note by title or description

Developed With

The goal of this version of NoteWorx is to illustrate how one can build an end-to-end dotnetcore application entirely on a Linux OS. Therefore, the entire application was built using Visual Studio Code on Ubuntu 18.04. The following list is a summary of the primary tools, languages and frameworks used to build the application:

.NET Core - .NET Core is a free and open-source managed software framework for Linux, Windows and macOS.
C# - A multi-paradigm programming language encompassing strong typing, imperative, declarative, functional, generic, object-oriented (class-based), and component-oriented programming disciplines.
ASP.NET Core - ASP.NET Core is a free, cross-platform, and open-source web framework
ASP.NET Core Identity - ASP.NET Core Identity is a membership system that adds login functionality to ASP.NET Core apps.
Entity Framework Core - EF Core is an object-relational mapper (O/RM) that enables .NET developers to work with a database using .NET objects. It eliminates the need for most of the data-access code that developers usually need to write.
Postgresql - Is an object-relational database management system with an emphasis on extensibility and standards compliance
Docker - Used to host Postgresql database
Docker-Compose - Compose is a tool for defining and running multi-container Docker applications.
Cake - Cake (C# Make) is a cross-platform build automation system with a C# DSL for tasks such as compiling code, copying files and folders, running unit tests, compressing files and building NuGet packages.
Bootstrap 4 - Build responsive, mobile-first projects
Gulp - Gulp is a toolkit for automating painful or time-consuming tasks in your development workflow, so you can stop messing around and build something.

Notable Nuget Packages

In addition to the aforementioned tools I want to make a special mention of 2 Nuget packages that I found very useful.

OdeToCode.UseNodeModules - ASP.NET Core middleware to serve files from the node_modules directory in the root of the project. This package enabled me to reference the 'node_modules' folder from within my Razor views. This is great for development. I have 'Gulp' tasks that copy the required packages from the 'node_modules' folder into the 'wwwroot' folder for production.
Humanizer.Core - Humanizer meets all your .NET needs for manipulating and displaying strings, enums, dates, times, timespans, numbers and quantities. This package enabled me to display date and times in a more "human friendly" manner. For example instead of display a date from 1 week ago in a date format, Humanizer takes the date and returns "1 Week ago" instead. Humanizer can be used for a whole lot more.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

The following software is required to be installed on your system:

Dotnet Core

Version 2.1.402 is required.

Run the following command from the terminal to verify version of dotnet:

  dotnet --version

NodeJS

The following version of Node and Npm are required:

Node 8.x
Npm 3.x

Type the following commands in the terminal to verify your node and npm versions

  node -v
  npm -v

Docker

Version 18.06 is required.

Run the following command from the terminal to verify version of Docker:

  docker -v

Install

Follow the following steps to get development environment running.

Clone 'noteworx-aspnetcore' repository from GitHub

   git clone https://github.com/drminnaar/noteworx-aspnetcore.git

or using ssh

   git clone git@github.com:drminnaar/noteworx-aspnetcore.git

Install node modules

   cd noteworx-aspnetcore/src/NoteWorx.Web
   npm install

Build

There are 2 build options:

Build using dotnet CLI tool

  cd noteworx-aspnetcore
  dotnet build

Build using Cake build script

  # linux terminal
  cd noteworx-aspnetcore
  ./build.sh

In both cases, Gulp processes the "gulpfile.js" file. The gulp process prepares client assets to be included in wwwroot folder.

Run

The NoteWorx application is configured to seed it's database with test data. Therefore, to sign into the NoteWorx application, one can use any of the users that are specified in the '/NoteWorx.Web/Infrastructure/SeedFiles/users.json' file. The password for all users is 'P@ssword123!'.

Run NoteWorx Demo In Container Using Docker

If you would like to simply see the application running in demo mode, type the following command in the terminal at the root of the solution.

cd noteworx-aspnetcore
docker-compose -f ./docker-compose-demo.yml build
docker-compose -f ./docker-compose-demo.yml up

The above command will build required image based on Dockerfile, setup a postgres database in a container, and setup pgAdmin (postges db tool) in a container.

To view the demo, type the following in your browser:

# To view the NoteWorx demo application
http://localhost:8080

# To open the database admin tool (pgAdmin)
http://localhost:8081

When you're done running the demo, type the following command to stop and cleanup containers.

docker-compose -f ./docker-compose-demo.yml down

Run NoteWorx From Source

Before running NoteWorx from source, you need to have a running Postgres database instance. For that you can type the following command to start a Postgres container and a pgAdmin container.

cd noteworx-aspnetcore
docker-compose -f ./docker-compose-data.yml -up

Next, type the following to verify that the Postgres database has started in a container

# You should see a container having the name 'db-dev' by typing the following
docker container ls

You will need to ensure you have the correct IP Address configured in the connection-string setting of the 'appsettings.Development.json' file. Therefore, type the following command to determine the IP Address of postgres container (db-dev).

# use the following putput IP address in connection string
docker container inspect | grep -w IPAddress

Next, run the NoteWorx web applicatio

cd noteworx-aspnetcore/src/NoteWorx.Web
dotnet run

After the application starts, open the web browser and type the following address:

http://localhost:5000

When you're done running the application, type the following command to stop and cleanup containers.

docker-compose -f ./docker-compose-data.yml down

Other

The online JSON data generator www.json-generator.com was used to generate all data used to seed the NoteWorx application.

Templates

The following templates were used to generate data.

User Template

[
  '{{repeat(10)}}',
  {
    picture: 'http://placehold.it/32x32',
    age: '{{integer(20, 40)}}',
    name: '{{firstName()}} {{surname()}}',
    gender: '{{gender()}}',
    company: '{{company().toUpperCase()}}',
    email: '{{email()}}',
    phone: '+1 {{phone()}}',
    address: '{{integer(100, 999)}} {{street()}}, {{city()}}, {{state()}}, {{integer(100, 10000)}}',
    about: '{{lorem(1, "paragraphs")}}'
  }
]

Tag Template

[
  '{{repeat(100)}}',
  {
    value: '{{lorem(1, "words")}}'
  }
]

Note Template

[
  '{{repeat(50)}}',
  {
    title: '{{lorem(random(3,4,5), "words")}}',
    description: '{{lorem(1, "paragraph")}}',
    createdAt: '{{date(new Date(2018, 0, 1), new Date(), "ISODateTimeTZ")}}',
    modifiedAt: function() {
      var createdDate = new Date(this.createdAt);
      var modifiedDate = new Date(createdDate);
      modifiedDate.setDate(createdDate.getDate() + Math.floor(Math.random() * Math.floor(10)));
      return modifiedDate;
    }
  }
]

Conclusion

Next I will be building NoteWorx using an Angular frontend that connects to an ASP.NET Core API using JWT tokens.

Flyway - Getting Started

Douglas Minnaar — Mon, 04 Jun 2018 20:30:51 +0000

Overview

This guide is inspired by an article that I read a few years ago that describes the concept of evolutionary database design. The article can be found here. In the article, reference was made to a number of database migration tools. Flyway was one of the tools mentioned. As it turns out, it is Flyway that I adopted in the end. Why Flyway? Because it is powerful yet easy to learn and use.

In this guide, I am going to explain how to use Flyway by providing a working example using tools such as Docker, Docker-Compose, PostgreSQL, pgAdmin, and of course Flyway.

According to the official Flyway documentation, Flyway is an open-source database migration tool that strongly favors simplicity and convention over configuration.

The key concept in the above definition is "database migration". Within the context of discussing Flyway, a "database migration" is a group of one or more database changes. In order to keep track of migrations, Flyway creates a table (per schema) that maintains a history of all migrations for a specific schema.

Flyway supports 2 forms of migration namely versioned and repeatable migrations.

Versioned
- Has a version number, checksum, and description. The version number ensures that migrations are unique. The checksum helps prevent accidental changes from being migrated. The description provides metadata about the migration.
- Versioned migrations are applied in order and only once
- An "undo migration" having the same version number can be provided
Repeatable
- Has a description and checksum but a version number is not required.
- Repeatable migrations can be re-applied everytime their checksum changes.
- Always applied after versioned migrations.

Migrations can be written in Java and SQL. However, in this guide I will be focusing on the SQL migrations. SQL migrations offer the most flexibility due to them being agnostic of any particular development platform. I will also be focusing on versioned migrations.

In this getting started guide, the simplest scenario is to begin with a new database. This guide uses Postgres to illustrate migrations. To make it simpler to view the changes in the database, pgAdmin4 will be used to connect to and inspect database changes. All migrations that will be created in this guide are available in this repo.

Technology Used

The technology used to produce this guide is summarised as follows:

Visual Studio Code

Docker

Docker is a computer program that performs operating-system-level virtualization also known as containerization. It is developed by Docker, Inc.

Docker-Compose

Compose is a tool for defining and running multi-container Docker applications.

PostgreSQL

PostgreSQL is an object-relational database management system.

pgAdmin4

Open Source administration and development platform for PostgreSQL

Flyway

Flyway is an open source database migration tool.

Prerequisites

In order to follow this guide, a basic understanding of the following technologies is required:

Git is a free and open source distributed version control system. Although not strictly mandatory, it is suggested to have Git installed on your local machine to make it simpler to retrieve the accompanying example repository. However, a zip file may also be downloaded.

Docker

For this guide, I primarily use software hosted within Docker containers. Although I provide all the instructions to follow this guide, it is still recommended that one familiarise oneself with Docker. I have written a Docker guide that can be found here. As part of the Docker guide, there is also a getting started section.

In addition to Docker related technologies, the tool docker-compose will also be used to run the containers. More specifically, docker-compose is a tool for defining and running multi-container Docker applications.

Postgres

PostgreSQL is a free and open source object-relational database management system. In this guide, PostgreSQL will be run from a Docker container. For more information on PostgreSQL on Docker, please visit the official PostgreSQL Docker registry.

pgAdmin

The tool pgAdmin, is an open source administration and development platform for PostgreSQL. In this guide, pgAdmin will be run from a Docker container. For more information on pgAdmin on Docker, please visit the official pgAdmin Docker registry.

Environment Setup

At this point you should have Docker and Docker-Compose installed. This can be verified by running the following commands from the command line.

To verify Docker:

  docker version
  docker info
  docker run hello-world

To verify Docker-Compose:

  docker-compose --version

If all is well, the above commands should have run flawlessly.

Next up, let's get the repository for this guide.

Get Repository

There are 3 ways to get the repository for this guide:

Clone Repo Using HTTPS

   git clone https://github.com/drminnaar/flyway.git

Clone Repo Using SSH

   git clone git@github.com:drminnaar/flyway.git

Download Zip File

   wget https://github.com/drminnaar/flyway/archive/master.zip
   unzip ./master.zip

Initialise Environment

Navigate to the 'flyway' directory using the command line.
Once inside the flyway directory, you will notice a file called 'docker-compose.yml'. This file contains all the instructions required to initialise our environment with all the required containerised software. In our case, the docker-compose.yml files holds the instructions to run postgresql and pgadmin containers.
Type the following command to initialise environment to run Flyway code migrations:

  docker-compose up

Type the following command to verify that there are 2 containers running. One container will be our PostgreSQL server. The second container will be our pgAdmin web application.

  docker-compose ps

The above command should display the running containers as specified in the docker-compose file.

Create Database

The first thing to be aware of when creating a migration, is that migrations do not create databases. Migrations only apply within the context of a database and do not create the database itself. Therefore, for my demonstration I will create an empty database from scratch and then create migrations for that database.

In this example, I create a database called "heroes". It is a database that stores data related to, you guessed it, heroes.

At this point, you should have a running PostgreSQL container instance. To verify this, run the following command:

  docker-compose ps

List available databases by running the following command:

  docker exec -it $(docker container ls -qf name=pg-dev) psql -U postgres -c '\l'

Currently, there is no heroes database.

Type the following command to create a heroes database:

  docker exec -it $(docker container ls -qf name=pg-dev) psql -U postgres -c 'CREATE DATABASE heroes OWNER postgres'

List available databases by running the following command:

  docker exec -it $(docker container ls -qf name=pg-dev) psql -U postgres -c '\l'

Create Migrations

For clarity sake, please take note that a migration is nothing more than a SQL file consisting of various SQL operations to be performed on the database.

Understanding The Migrations

The heroes database now exists. We are now ready to run our migrations. Please take note of the migrations folder that is part of the repo for this example. The migrations folder consists of 7 migrations that are briefly described as follows:

V1_1_Create_hero_schema.sql - Creates a new _hero_data schema

  CREATE SCHEMA hero_data AUTHORIZATION postgres;

V1_2_Create_hero_table.sql - Create a new _hero table in the hero_data schema

  CREATE TABLE hero_data.hero
  (
      id BIGSERIAL NOT NULL,
      name VARCHAR(250) NOT NULL,
      description TEXT NOT NULL,
      debut_year INT NOT NULL,
      appearances INT NOT NULL,
      special_powers INT NOT NULL,
      cunning INT NOT NULL,
      strength INT NOT NULL,
      technology INT NOT NULL,
      created_at TIMESTAMPTZ NOT NULL,
      updated_at TIMESTAMPTZ NOT NULL
  );

  ALTER TABLE hero_data.hero ADD CONSTRAINT pk_hero_id PRIMARY KEY (id);

V1_3_Add_Destroyer_hero.sql - Inserts our first hero into _hero table

  INSERT INTO hero_data.hero (
      name,
      description,
      debut_year,
      appearances,
      special_powers,
      cunning,
      strength,
      technology,
      created_at,
      updated_at) VALUES (
      'Destroyer',
      'Created by Odin, locked in temple, brought to life by Loki',
      1965,
      137,
      15,
      1,
      19,
      80,
      now(),
      now());

V1_4_Create_user_schema.sql - Create a _user_data schema

  CREATE SCHEMA user_data AUTHORIZATION postgres;

V1_5_Create_user_table.sql - Create a new _user table in the user_data schema

  CREATE TABLE user_data.user
  (
      id BIGSERIAL NOT NULL,
      first_name VARCHAR(250) NOT NULL,
      last_name VARCHAR(250) NOT NULL,
      email VARCHAR(250) NOT NULL,
      alias VARCHAR(250) NOT NULL,
      created_at TIMESTAMPTZ NOT NULL,
      updated_at TIMESTAMPTZ NOT NULL
  );

  ALTER TABLE user_data.user ADD CONSTRAINT pk_user_id PRIMARY KEY (id);

V1_6_Add_unique_hero_name_contraint.sql - Alter _hero table by adding a unique name constraint

  ALTER TABLE hero_data.hero ADD CONSTRAINT uk_hero_name UNIQUE (name);

V1_7_Add_unique_user_email_constraint.sql - Alter _user table by adding a unique email constraint

  ALTER TABLE user_data.user ADD CONSTRAINT uk_user_email UNIQUE (email);

You will have noticed the strange naming convention. The way we name a migrations is as follows:

According to the official Flyway documentation, the file name consists of the following parts:

Prefix: V for versioned migrations, U for undo migrations, R for repeatable migrations
Version: Underscores (automatically replaced by dots at runtime) separate as many parts as you like (Not for repeatable migrations)
Separator: __ (two underscores)
Description: Underscores (automatically replaced by spaces at runtime) separate the words

Run Migrations

Finally we get to run our migrations. To run the migrations, we will execute the Flyway Docker container.

Before running the migration, we need to obtain the IP address of the postgres container as follows:

docker container inspect -f "{{ .NetworkSettings.Networks.flyway_skynet.IPAddress}}" flyway_pg-dev_1

We plug the obtained IP address from above into the command below. In my case, my IP address is 172.18.0.2

docker run --rm --network docker_skynet -v $PWD/migrations:/flyway/sql boxfuse/flyway -url=jdbc:postgresql://172.18.0.2:5432/heroes -user=postgres -password=password migrate

You should see an output similar to the following output:

As can be seen from output above, all 7 migrations ran successfully.

Run the following command to see a list of tables in the heroes database:

docker exec -it $(docker container ls -qf name=pg-dev) psql -U postgres -d heroes -c "SELECT table_schema, table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema NOT IN ('pg_catalog', 'information_schema')"

You should see a list of tables as follows:

table_schema	table_name
public	flyway_schema_history
hero_data	hero
user_data	user

The database table flyway_schema_history contains all the records for the database migrations that took place.

Lastly, log into pgAdmin to view the flyway_schema_history table.

Navigate to http://localhost:8080 in your browser

email/username: iamhero@heroes.com
password: password

If you're wondering where the pgadmin credentials come from, you can find them specified in the docker-compose.yml file. They're passed in as environment variables.

Once logged in, you can connect to the PostgreSQL server by adding a connection as follows:

Open the flyway_schema_history table that is located in the public schema of the heroes database.

Conclusion

This has been a basic introduction into using the tool Flyway for performing database migrations. Although basic, it doesn't really get much more complicated than this.

I did not cover how to use Flyway with an existing database, however, this is also supported by using the baseline command with Flyway. I will cover this in more detail in a next part to this guide.

Choosing a Programming Language

Douglas Minnaar — Mon, 19 Mar 2018 07:23:45 +0000

Summary

I recently came accross 2 videos on the BigThink website that addresses the topic 'Five Programming Languages that Everyone Should Know'. The original videos can be found via the following links:

In the first video, Larry Wall (computer programmer responsible for creating Perl), begins by providing a very valid point. The point that he makes is that answering this question is very much influenced by context as well as the times. What may be good a few years ago may not necessarily hold true for current or future times. He then goes on to provide what he currently thinks are 5 programming languages that everyone should know and they are listed as follows:

Javascript
Java
Haskell
C
Perl

In the second video, Bjarne Stroustrup (computer programmer who designed and implemented the computer programming language C++), begins by making a very strong statement. Bjarne says:

"Nobody should call themselves a professional if they only knew one language.”

For the rest of the video, Bjarne gets straight to the point and nominates the following computer programming languages as the 5 that he thinks you should know:

C++
Java
Python
Functional Programming Language (doesn't specify anything particular)
High Performance Computing Programming Lanaguage (doesn't specify anything particular)

If you would like to have your say, I have created a survey where you can nominate your top 5 programming languages. This survey is for fun and I allow anonymous access. You will also see the current results straight after making your picks. If you're interested, please access the survey here.

In the next section I provide some detail into what "knowing" a programming language means.

EDIT (2018-03-21) - Fixed inaccurate explanation of side effects as seen in functional programming section

"Knowing" a Programming Language

I feel that it is important to provide some context and insight into what I perceive "knowing" a programming language entails. It is an opinion, therefore my thoughts on the matter should not be taken as 100% correct and applicable to all programming languages.

Below I provide a list of concepts that I consider to be the pillars of "knowing" a programming language:

Nuts and Bolts

By "Nuts and Bolts", I'm referring to the core features and mechanics that make the programming language work. These features have nothing to do with using libraries and frameworks. It is the language in it's purest form. More specifically, I am referring to things like compilation, type systems, memory managment, constructs, loops, evaluation mechanisms, variables, scope and closure to name a few. The "Nuts and Bolts" are the building blocks of a programming language and one cannot effectively use the language unless one has a firm grasp in terms of understanding how to use and apply those building blocks. This is where everyone starts when learning a new language. It's also the easiest part in my opinion. For some, they may feel that once they know the "Nuts and Bolts", they know the language. But for me this is merely the beginning and but the tip of a very large iceberg.

Libraries

Most programming languages come packaged with a core set of libraries. These libraries are reusable bits of code that can be shared, adopted, extended, and applied to various programming problems. Typically, there are hundreds, actually thousands of libraries to choose from. This is especially more prevalent as a result of third party and open source development. It is impossible to learn about all libraries relating to a programming language. Though it is good practice to at least make oneself familiar with as many as possible. This takes a lot of time and commitment.

Frameworks

I've often heard the terms "libraries" and "frameworks" used interchangeably. However, from my perspective, frameworks imply something far bigger than a simple library. Frameworks are usually created to help simplify development that may span multiple layers of an application architecture. Where libraries are easy to add, remove, or replace. Frameworks form an inherent part of ones architecture and cannot be easily removed or changed. Frameworks are big, and they require even more investment of time to fully understand and apply them. And often, just when you've achieved a good understanding of a framework, a new one comes along to replace it and one has to start all over again.

Design

So much about effective programming is about effective design. And the design of a programming language often determines how we design systems using that programming language. A good design is the key difference between a system that "works" over the long term and one that ultimately fails. A design that works in one programming language may not necessarily work for another programming language. Once again, it is only through time and experience that one is able to determine what designs work and what designs don't work at all. Also, different design patterns will emerge for different programming languages. This has a lot to do with the fact that some programming languages are imperative by nature and others are functional for example. As I've mentioned before, each programming language will have short-comings. For example, much to the dismay of many, the GO programming language currently does not support "generics". Therefore, one would need to use a different design than if one were programming in C#/Java (that do support generics). There is a constant tradeoff when designing programs for the real-world and it's different for each and every programming language.

Application and Practice

Each language will come with it's own set of good and bad practices. It is important to know about both. Each language will also come with subtle nuances and "traps" that one should be aware of. It's this practical side of working with a programming language that truly determines whether you know a programming language or not. It is knowledge only acquired by working continuously with the programming language and applying it in real-world scenarios that one can acquire a good sense of "knowing" what the programming language is all about.

As one can see (and I'm sure many are aware of), learning a programming language is difficult. And for me it only get's harder with time as one is required to write ever increasingly sophisticated programs. But there's an interesting "little" problem that exists when learning a new programming language. And that "little" problem is time. You cannot shortcut time and therefore you cannot shortcut experience. To "know" a programming language takes a huge investment in time to gain sufficient experience in becoming adept at that programming language. I have approximately 15 years programming experience where I have programmed in C++, Java, C#, Javascript, and SQL. I can honestly say that I feel that I still don't "know" these languages. But that's what programming is all about for me. To continuously learn and improve.

In the next section I am going to provide some insight into the programming languages that I think you should know. However, judging by what I have written in this section, you may have realised my subtle hint in that I don't think it is possible to "know" 5 programming languages. But I do think we should try because it makes us better thinkers and therefore better programmers.

The 6 Programming Languages I Think You Should Know

After watching the "Big Think" videos, it got me thinking. What do I consider as the top 5 programming lanaguages that someone (by someone I mean anyone practicing or interested in programming) should know. After thinking about it I decided that there are actually 6 that I would like to recommend.

Before answering the million dollar question, I would like to provide some of my working background as it does influence my choices. For the past 15 years, I have worked in 3 primary types of industry. Namely, financial services (banking and insurance), medical services industry, and the online gaming (gambling) and sports-betting industry (where I still work). The types of applications that I have been responsible for designing and building have always involved working within 3 primary areas of a system and they are listed as follows:

Databases

One cannot build applications without some form of long-term persistent data store. If experience has taught me anything, it's that applications come and go but data is forever. It's the one software asset that tends to outlive everything else within a system.

Frontend

Where there is data, there is a need to easily view, store, retrieve, and manage that data. Therefore, I have found that there is often a "one-to-many" relationship between databases and frontend applications. By that I mean that for each database there may be many applications built to interact with that database. The types of frontend applications that I have had to build involved using many different types of languages. For example, typically I would use one set of languages to build native applications and another set of languages to build web applications.

Backend

Very seldom have I been required to create simple CRUD (Create, Read, Update, and Delete) applications. There have always been business rules that govern the flows of different processes within a system. Some business rules are very complex and lead to the creation of many different kinds of domain models that involve integrating to databases and services (like API's for example). Often, I have been required to expose domain models through API's. Once again there are certain programming languages that work better at this level than others.

Based on my background and experience, I now provide what I feel are important programming lanaguages to know or learn. My top 6 programming lanaguages are listed as follows in no particular order:

1. Javascript

It feels like Javascript is everywhere these days, and it is. No longer being limited to clientside development, Javascript has matured into a language that can be used server side too. One only needs to read the job boards, and various blogs to realise that Javascript is very much here to stay. It always makes me think of Atwoods Law that states:

"any application that can be written in Javascript, will eventually be written in Javascript"

The fact of the matter is that there are very few jobs out there that do not benefit from someone having some Javascript knowledge. It is a very simple language to learn, however, it get's more complicated the more you learn. The Javascript language is full of subtle nuances that can trick even the most experienced Javascript developer.

For me, knowing Javascript goes hand-in-hand with knowing a language that can easily transpile into Javascript. The reason why you would want to learn such a language is if you are building a large javascript codebase and want to improve maintainability. My preference is Typescript which is a superset of Javascript. There are also other Javascript alternatives out there like:

Coffeescript - CoffeeScript is a programming language that transcompiles to JavaScript. It adds syntactic sugar inspired by Ruby, Python and Haskell in an effort to enhance JavaScript's brevity and readability.
Elm - Elm is a domain-specific programming language for declaratively creating web browser-based graphical user interfaces. Elm is purely functional, and is developed with emphasis on usability, performance, and robustness.

There is also a "type checker" called Flow that can be used with Javascript. Though I'm not seeing adoption anything close to Typescript at the moment.

2. Object Oriented Programming (OOP) Language

At university, I majored in distributed computing using Java as my primary programming language. Although, I also done a fair amount of programming using C++ and a lesser known programming language called Eiffel. However, Java was always my favourite. Learning Java certainly enabled me to easily adapt to other programming languages. The only thing that I can say I hated about the Java programming language was "checked exceptions". And no, there is nothing you can do to convince me otherwise, despite any technical merit they may have.

It is the C# programming language that I ended up spending most of my time with. I have been developing in C# as my primary language for more than 10 years now. The biggest drawback to me was that C# was never open-source or cross-platform. But thankfully that has all changed in recent years, and I am happily developing C# applications on a Linux OS. It's strange, but of late, the Javascript language has changed so much, that when I am writing Javascript, it often feels like I'm writing C# code. More recently for example, when the async/await feature was added to Javascript, I felt right at home because I had been writing multi-threaded asynchronous programs for years in C# using a very similar pattern. I must admit though, I am not a fan of introducing "class oriented" concepts into the Javascript language.

So do I think you should learn C#, Java, or C++ etc? My personal feeling is that it doesn't matter all that much, but pick one. Like Bjarne Stroustrup mentions in the video from above. These are a cluster of languages and knowing one helps you to know them all. As you can see from my personal experience, it started with Java and C++ but ended up mostly programming in Javascript and C#.

3. Python

Python has been around for a long time. Sometimes referred to as a scripting language, I will take it one step further and call it a general purpose programming language. Lately, circa 2017/2018, it has been gathering massive adoption. One cannot talk about data science without Python being mentioned in the discussion. The other one is of course R but for now my first choice is Python. I am mostly a Python hobbyist programmer and have not used it in any production environment. That doesn't mean that I think of it as a hobbysit language. On the contrary. Once again, one only needs to visit a job board or some trending stats to realise that Python is a clear favourite of many. With the rise of Machine Learning and Artificial Intelligence, it seems that Python has become a popular choice in those domains too.

4. SQL (Structure Query Language)

Yes, SQL is a language. More specifically SQL is a fourth-generation language (4GL) that's closer to human language. I sometimes wonder if SQL is not the most omnipresent programming language in the world. SQL is everywhere, and in all my working experience, I have not once NOT had to work with SQL. Whether it was MSSQL, Postgres, MySQL, or Oracle, SQL existed in some shape or form. Of course SQL is not as sexy and shiny as the new NoSQL technologies out there, but SQL is here to stay for a long time yet. Interestingly, Couchbase, a NoSql database, uses the N1QL (pronounced "nickel") query language. The interesting thing about N1QL is that it adopts nearly full SQL ANSI-92 compliance. Even some tools like NoSqlBooster allow one to write SQL queries over a MongoDB database.

5. BASH

BASH is a scripting language and it is Turing-Complete. Like my prior choices in programming languages to know, the reason BASH makes it onto my list is for reasons of it's massive adoption and community. For me, the difference between being a King or a God of Linux is ones ability to program in BASH. That and being able to work with VIM ;)

Furthermore, Linux is by far the most used OS in server computing. Therefore, I think it is a good idea to master as much as possible about Linux as you're most likely bound to work with it sooner or later. Learning BASH scripts is part of mastering Linux. Knowing BASH will also help you to write repeatable and automated tasks that can be shared and used my many. Therefore, in addition to being a valuable sysadmin tool, I think it is a valuable devops tool too.

When writing about Linux, I'm always reminded by the following picture:

6. Functional Programming Language

I have no experience developing software using a purely (or close to pure) functional programming language. Therefore, I cannot make any recommendation based from experience. However, in the absence of experience, I turned to data to help drive my decision. For the past year I have been keeping a close eye on functional programming languages. The 3 functional programming languages that always seem to attract the most attention (in my experience) are as follows:

Haskell - Haskell is a standardized, general-purpose purely functional programming language, with non-strict semantics and strong static typing.
Scala - Scala is a general-purpose programming language providing support for functional programming and a strong static type system. Designed to be concise, many of Scala's design decisions aimed to address criticisms of Java.
F# - F# is a strongly typed, multi-paradigm programming language that encompasses functional, imperative, and object-oriented programming methods.

I suspect that functional programming is going to become THE programming skill to have in the near future. The reason for me saying so is due to the increased demand to write concurrent and parallel programs. That demand is driven by the changing physical architecture of micro processors. In a nutshell, as programmers we have only had to worry about writing programs that work with a single core processor. Until a few years ago that is. If we needed more speed, we would increase the speed of the processor. In recent years, the processor industry reached the limits of providing higher processing speed. As a result, the processor architecture changed to provide 2, 4 and more cores per processor to help run programs faster. But programs didn't run faster because nobody had programming languages that could effectively utilise the additional cores to write truly concurrent programs without side-effects (more on that later). Nobody except those using functional programming languages that is. It turns out that functional programming languages are very good for multi-core, multi-threaded, and distributed system domains. One of the biggest advantages that one will often hear being advertised as a reason to use functional programming is to eliminate "side-effects". A "side-effect" is when you have some code that is accessing or changing the state of some data (like a variable) that is outside the scope of that code. The problems associated with side-effects are exacerbated in the presence of multiple threads. Another important concept to understand as it applies to parallel computing is Amdahl's Law. To explain Amdahl's Law would require an entire dedicated article, and I won't be providing that here. I would however like to direct you to an interesting explanation of Amdahl's Law that can be found here. I should mention that programming languages like GO, Rust, Java and C# now have concurrency (the ability to write concurrent programs) baked into the language too and feel certain that this will only improve with time.

I encourage anyone that is new or experienced with programming to invest time in learning a functional programming languages. If anything it will help one to think better about how to correctly write programs without side-effects. Also, functional paradigms are making their way into many popular languages like Java and C# too.

It is Scala that holds the most interest to me in terms of functional programming languages. For me it is a rising star in the world of functional programming.

That concludes my musings in terms of the programming languages that I think you should learn. In the next section I conclude this article and provide a few more programming languages that I feel are also important, and anyone of them could easily have taken the place of the one's I have already mentioned.

Conclusion

In my experience, there is no programming language to rule them all. I have mostly picked up languages out of necessity to get particular jobs done in the easiest way possible. The programming languages that I discussed are by no means the most popular languages. But they are languages that have massive communities, massive amounts of documentation and learning materials, and massive amounts of jobs.

In closing I would like to mention a few programming languages that I would love to try. Anyone of them could easily have taken the place of the programming languages I discussed in previous sections. They are listed as follows:

Rust - Rust is a systems programming language sponsored by Mozilla Research, which describes it as a "safe, concurrent, practical language", supporting functional and imperative-procedural paradigms
GO - Go is a programming language created at Google in 2009. It is a compiled, statically typed language in the tradition of Algol and C, with garbage collection, limited structural typing, memory safety features and CSP-style concurrent programming features added.
Kotlin - Kotlin is a statically-typed programming language that runs on the Java virtual machine and also can be compiled to JavaScript source code or use the LLVM compiler infrastructure.

I suspect that I will be exploring Scala or Kotlin next as a potential "next skill".

Don't forget that if you would like to have your say, you can access the survey that I am running for fun at this link.

MongoDB Guide - Getting Started

Douglas Minnaar — Tue, 13 Mar 2018 08:07:29 +0000

Summary

In this guide, I am going to cover the essentials in terms of getting up and running with MongoDB. The best way to start learning new technology is to jump straight in and start using it. Therefore, after a brief introduction to MongoDB, the following sections are all about setting up your own MongoDB instance using 4 different hosting methods. Next, I introduce 5 tools that you can use to work with your MongoDB databases. The next section illustrates methods on creating your own database and collection with data. In this section I also provide a number of examples that illustrate some basic queries to query your data.

This guide is summarized as follows:

What Is MongoDB
MongoDB Setup
- Install And Host Locally
- Install And Host Using Docker
- Host With MongoDB Atlas
- Host With MLab
MongoDB Tools
- MongoDB Shell
- MongoDb Compass
- NoSqlBooster
- Robo 3T
- Visual Studio Code
Basic MongoDB Commands
Conclusion

Lastly, this guide does not cover the reasons to use MongoDB (perhaps a guide for another time). However, I would like to encourage readers to spend some time researching and understanding the rationale between choosing a NoSql database over a Sql database. MongoDB is not a replacement for SQL databases. It is merely an alternative that may (or may not) align better with ones specific requirements.

The following question posted on Quora may be a good starting point:

What are some reasons to use traditional RDBMS over NoSQL?

From my personal experience and perspective, I will mention that one should think very carefully before committing to a NoSql database if your data is required to be agnostic. In this case I have found Sql databases to be a far better choice due to the simplicity of modelling data and exposing it in many different ways for many different applications.

The original guide can be found on my Github repository. Like my Docker Guide, I will be updating and adding more content over time to this repository.

What Is MongoDB?

MongoDB is a database. More specifically, it is an open source document-oriented database that has been designed for scalability and simplicity for both developers and sysadmins. Traditional relational database management systems (RDBMS) like MSSQL, Oracle, MySQL, and PostGreSQL store data in tables having a static schema composed of rows and columns. However, MongoDB stores it's data in JSON-like documents having dynamic schemas.

MongoDB Setup

I will discuss 4 options to get up and running with MongoDB:

Install and host locally
Install and host in Docker
Register for and use MongoDB Atlas (Database As A Service)
Register for and use MLab (Database As A Service)

Install And Host Locally

Installing MongoDB is relatively straight forward. There are currently 3 platform (Windows, Linux, OSX) releases available and can be found here

For more specific installation instructions, please see the following links:

Install And Host Using Docker

If you have never used Docker before, then this option is less suitable for you as it implies having some Docker knowledge and experience. However, if you are still interested in pursuing this option, then I recommend viewing my Docker Guide as a starting point. I also have a Docker Cheatsheet that you may find useful.

The Docker image that will be used, is the official Docker MongoDB image.

Run MongoDB

Run MongoDB Using Named Volume

For the following examples I map the Docker MongoDB port of 27017 to a local port of 37017. The reason for this is because I have a local instance on MongoDB running that is already using the 27017 port.

To run a new MongoDB container, execute the following command from the CLI:

docker run --rm --name mongo-dev -p 37017:27017 -v mongo-dev-db:/data/db -d mongo

CLI Command	Description
--rm	remove container when stopped
--name mongo-dev	give container a custom name
-p	map container published port to local port
-v mongo-dev-db/data/db	map the container volume 'data/db' to a custom name 'mongo-dev-db'
-d mongo	run mongo container as a daemon in the background

Run MongoDB Using Bind Mount

cd
mkdir -p mongodb/data/db
docker run --rm --name mongo-dev -p 37017:27017 -v ~/mongodb/data/db:/data/db -d mongo

CLI Command	Description
--rm	remove container when stopped
--name mongo-dev	give container a custom name
-p	map container published port to local port
-v ~/mongodb/data/db/data/db	map the container volume 'data/db' to a bind mount '~/mongodb/data/db'
-d mongo	run mongo container as a daemon in the background

Access MongoDB

Access MongoDB From Local Machine

Type the following command to access MongoDB instance hosted within docker container:

   mongo --host localhost --port 37017

Access MongoDB Via Docker Interactive TTY

There are 2 steps to accessing the MongoDB shell.

Firstly, access the MongoDB container shell by executing the following command:

   docker exec -it mongo-dev bash

This will open an interactive shell (bash) on the MongoDB container.

Secondly, once inside the container shell, access the MongoDB shell by executing the following command:

   mongo localhost

Host With MongoDB Atlas

MongoDB Atlas is a Data as a Service (DaaS) offering, and is hosted in the cloud. There is no installation of MongoDB required and a free tier is available.

To get started, signup for free by registering for a free tier account here. The free tier entitles you to 512MB storage.

Please review the [MongoDB Atlas Documentation] for more information.

Once you have registered and setup your MongoDB instance on Atlas, you will be presented with a dashboard resembling the following image:

Host With MLab

MLab is a Data as a Service (DaaS) offering, and is hosted in the cloud. There is no installation of MongoDB required and a free tier is available.

To get started, signup for free account here. The free tier entitles you to 500MB storage.

Please review the MLab Documentation for more information.

Once you have registered and setup your MongoDB instance on MLab, you will be presented with a dashboard resembling the following image:

MongoDB Tools

The tooling for MongoDB has improved a lot of late and there are many options. I have worked with the following tools and I find them all quite good. It's really a matter of personal preference but I find that I tend to use them together.

MongoDB shell

The MongoDB shell is the default way of interacting with MongoDB databases.

Connect to MongoDB

  mongo localhost

List Available Databases

  show dbs

Create Database

  use message_db

Create Collection

  db.messages.insert({ message: 'hello mongodb' })

List Collections

  show collections

MongoDB Compass

MongoDB Compass is a Graphical User Interface (GUI) tool that allows one to explore your MongoDB data.

It has a free addition available
It is cross platform and is available for Linux, Windows, and Mac
It supports the following primary features:
- Run ad hoc queries
- Perform CRUD operations on data
- View and optimize query performance

In the following screenshot, I am connected to MongoDB running on my local machine, using MongoDB Compass.

For more information on MongoDB Compass, go here

NoSqlBooster

NoSqlBooster is a Graphical User Interface (GUI) that provides an easy to use interface to work with your MongoDB database.

It has a free addition available
It is cross platform and is available for Linux, Windows, and Mac
It provides the following features:
- Shell extensions
- Fluent Query API for MongoDB
- Query MongoDB with SQL
- Use Node Modules in your script

In the following screenshot, I am connected to MongoDB running on Docker, using NoSqlBooster.

For more information, please visit the official NoSqlBooster website.

Robo 3T

Robo 3T, is a free GUI tool that can be used to explore your MongoDB databases.

It is free to use.
It is cross platform and is available for Linux, Windows, and Mac
It provides a MongoDB GUI with embedded shell

In the following screenshot, I am connected to a local instance of MongoDB using Robo 3T:

Robo 3T can be downloaded here

For more information, go here.

Visual Studio Code

Using the Azure CosmosDB Extension, one can connect to MongoDB databases in addition to Azure CosmosDB databases.

For more information, see the following links:

Install Azure CosmosDB Extension

Launch VS Code
Launch Quick Open (ctrl+P) from within VS Code
Paste the following command and press enter

  ext install ms-vscode.vscode-azureextensionpack

Configure User Settings

Once you have the extension installed, you will need to ensure that you have a path to Mongo configured in your user settings. Press Ctrl+, to open your user settings. Therefore, because I am using a Linux based OS, I had to add the following setting to the user settings json:

"mongo.shell.path": "/var/lib/mongodb",

Connect To Mongo On localhost

To connect to MongoDB instance, follow the following steps:

Expand Azure Cosmos DB extension panel located in explorer
Select the option to 'Attach Database Account'
Select MongoDB
Enter the connection details to MongoDB instance

  mongodb://localhost:27017

The steps are shown below:

Run Commands Using Mongo ScrapBooks

To use the 'ScrapBook Feature', follow the following steps:

Attach to your MongoDB instance as seen above
Connect to database of your choice
Select 'New MongoDB ScrapBook' option to open scrapbook editor
Enter MongoDB command
Press Ctrl+Shft+' to execute

The steps are shown below:

At the time of this writing, there seems to be an issue with Mongo ScrapBooks

Basic MongoDB Commands

Once you have your MongoDB instance running or hosted, open a mongo shell to your MongoDB server. For the purposes of this demonstration, I will be using my local installation 'localhost'.

Connect to MongoDB instance

  mongo localhost

Create a 'zips_db' database

  use zips_db

Create 'zips' collection

  db.zips.insertMany([
     { "city" : "AGAWAM", "loc" : [ -72.622739, 42.070206 ], "pop" : 5338, "state" : "MA", "_id" : "01001" },
     { "city" : "CUSHMAN", "loc" : [ -72.51564999999999, 42.377017 ], "pop" : 36963, "state" : "MA", "_id" : "01002" },
     { "city" : "BARRE", "loc" : [ -72.10835400000001, 42.409698 ], "pop" : 4546, "state" : "MA", "_id" : "01005" },
     { "city" : "BELCHERTOWN", "loc" : [ -72.41095300000001, 42.275103 ], "pop" : 10579, "state" : "MA", "_id" : "01007" },
     { "city" : "BLANDFORD", "loc" : [ -72.936114, 42.182949 ], "pop" : 240, "state" : "MA", "_id" : "01008" },
     { "city" : "BRIMFIELD", "loc" : [ -72.188455, 42.116543 ], "pop" : 706, "state" : "MA", "_id" : "01010" },
     { "city" : "CHESTER", "loc" : [ -72.988761, 42.279421 ], "pop" : 688, "state" : "MA", "_id" : "01011" },
     { "city" : "CHESTERFIELD", "loc" : [ -72.833309, 42.38167 ], "pop"  177, "state" : "MA", "_id" : "01012" },
     { "city" : "CHICOPEE", "loc" : [ -72.607962, 42.162046 ], "pop" : 3396, "state" : "MA", "_id" : "01013" },
     { "city" : "CHICOPEE", "loc" : [ -72.576142, 42.176443 ], "pop" : 1495, "state" : "MA", "_id" : "01020" }
   ])

Search Queries
- Get total collection count

  db.zips.count()

Get all zip documents

  db.zips.find().pretty()

Get 5 zip documents

  db.zips.find().limit(5).pretty()

Get zip having a city name of 'BLANDFORD'

  db.zips.find({"city": "BLANDFORD"})

Get zip having a city name of 'BLANDFORD', ignore case

  db.zips.find({"city": { "$regex": /blandford/i } })

Get zip having a city names of 'BLANDFORD' and 'BRIMFIELD'

  db.zips.find({"city": { "$in": ["BLANDFORD", "BRIMFIELD"] } }).pretty()

Get zip having a city names of 'BLANDFORD' and 'BRIMFIELD', ignore case

  db.zips.find({"city": { "$in": [ /blandford/i, /brimfield/i] } }).pretty ()

Get zip documents having a population greater than or equal to 30000

  db.zips.find({"pop": { "$gte": 30000}}).pretty()

Get all zip documents and use projection to only display city and population

  db.zips.find({}, { "city": 1, "pop": 1, "_id": 0 })

Get all zip documents and use projection to only display city and population. Sort documents by city name in descending order

  db.zips.find({}, { "city": 1, "pop": 1, "_id": 0 }).sort({"city": -1})

Delete operations
- Delete all cities called 'CHICOPEE'. Ignore case

  db.zips.remove({ "city": { "$regex": /chicopee/i }})

Update operations
- Update the city of 'BLANDFORD' by setting its population to 10

  db.zips.updateOne({"city": "BLANDFORD"}, { "$set": { "pop": 10}})

Conclusion

This has been a quick introduction to getting started with MongoDB. In this guide we learned about 4 different ways to host your MongoDB instance. We also learned about 5 different tools that can be used to manage your MongoDB databases. Lastly, we explored some basic queries that can be used to create a database, create a collection, insert data, update data, and fetch data.

MongoDB is an exciting and relatively new database that has a massive community and knowledgebase. I encourage all those that are interested to get involved by trying out the following learning resources:

MongoDB Docs - The official MongoDB documentation
MongoDB University - Free Online Classes on MongoDB from MongoDB

The following MongoDB book is my personal favourite:

If you're interested in certification, please see the MongoDB Professional
Certification Program.

React Redux Starter

Douglas Minnaar — Mon, 26 Feb 2018 09:05:07 +0000

Summary

Recently, I updated my react-starter project by adding the following notable changes:

accommodate multiple build configurations (dev and prod)
allow one to specify addons (bundleanalyzer, bundlebuddy) as part of NPM scripts
added webpack-dashboard as an optional NPM run script

I was undecided between whether or not to introduce more advanced libraries like Redux and React-Router for example. In the end I decided to create a new starter project called react-redux-starter that serves as more of an advanced starter project for those wanting to work with Redux and React-Router in addition to React. Therefore, in this post I will describe what is available in the react-redux-starter project.

Project Overview

The react-redux-starter project is a basic template that consists of the essential elements that are required to start building a Single Page Application (SPA) using React, React Router, Redux, Bootstrap 4, Sass, and Webpack.

Although I am calling this project a template of sorts, it is also a mini application in that it demonstrates a full vertical slice of the architecture. What this means, is that there is an example of Redux and React-Router in action. I provide a very simple component that fetches a list of zip codes. I kept the example simple by providing data via a 'Zip Code JSON file'. This simple component does however illustrate the use of Redux Actions, a Redux Container, and a Redux Store.

The following diagram illustrates how data is fetched using Redux and a Zip Code Service:

Features

The template consists of:

a typical project layout structure
a Babel setup and configuration
a Webpack setup and configuration
an ESLint setup and configuration
a SCSS setup and configuration
a sample React component to display a list of zip codes
a Redux setup to handle zip codes state
a React Router setup to show basic navigation

Additionally, the template provides a development and production webpack configuration.

The template also allows one to include specific plugins as part of build. Please see here for more detail

Showcase

The following animated gifs illustrate what the application looks like.

Desktop

Mobile

Developed With

Visual Studio Code - A source code editor developed by Microsoft for Windows, Linux and macOS. It includes support for debugging, embedded Git control, syntax highlighting, intelligent code completion, snippets, and code refactoring
Node.js - Javascript runtime
React - A javascript library for building user interfaces
React Router - Declarative routing for React
Redux - Redux is a predictable state container for JavaScript apps.
Redux-Promise-Middleware - Redux middleware for promises, async functions and conditional optimistic updates
Redux-Thunk - Thunk middleware for Redux
Babel - A transpiler for javascript
Webpack - A module bundler
SCSS - A css metalanguage
Bootstrap 4 - Bootstrap is an open source toolkit for developing with HTML, CSS, and JS

Getting Started

Find the react-redux-starter project here.

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

The following software is required to be installed on your system:

Node 8.x
Npm 3.x

Type the following commands in the terminal to verify your node and npm versions

  node -v
  npm -v

Install

Follow the following steps to get development environment running.

Clone 'react-redux-starter' repository from GitHub

  git clone https://github.com/drminnaar/react-redux-starter.git

OR USING SSH

  git clone git@github.com:drminnaar/react-redux-starter.git

Install node modules

   cd react-redux-starter
   npm install

Build

Build Application

dev	prod
npm run build:dev	npm run build:prod

Build Application And Watch For Changes

dev	prod
npm run build:dev:watch	npm run build:prod:watch

Build Application With BundleAnalayzer Plugin Included

dev	prod
npm run build:dev:bundleanalyze	npm run build:prod:bundleanalyze

After running the above command, a browser window will open displaying an interactive graph resembling the following image:

Build Application With BundleBuddy Plugin Included

dev	prod
npm run build:dev:bundlebuddy	npm run build:prod:bundlebuddy

Run ESlint

Lint Project Using ESLint

  npm run lint

Lint Project Using ESLint, and autofix

  npm run lint:fix

Run

Run Start

This will run the 'serve:dev' npm task

npm start

Run Dev Server

npm run serve:dev

Run Dev Server With Dashboard

npm run serve:dev:dashboard

The above command will display a dashboard view in your console resembling the following image:

Run Prod Server

This command will build application using production settings and start the application using live-server

npm run serve:prod

Conclusion

It's always difficult providing a starting point that's simple but has enough detail to illustrate a few important concepts. I hope this project helps achieve that goal. I will be updating and tweaking over time to try and align the project as an ideal starting point to learning how to build React applications using Redux, Webpack and React Router.