DEV Community: Jack Lewis

Setting up a simple testing project with C#

Jack Lewis — Fri, 26 May 2023 21:31:24 +0000

NOTE: This article is an attempt to showcase how to set up testing in C#, and doesn't represent some "standard" practices observed when unit testing (such as TDD). This is so the article can more easily explain the process in a way that flows easily.

Preface

Testing is extremely important in any programming language. However, there's so many ways to do it, so I thought I'd write a short article on how I set up testing in C#.

Getting Started

To start with you need to choose what type of project you want to work with. This could be a website using MVC, or a service or something like a desktop application using WPF. For the case of this demo, I'm going to use a console application as it's reasonably straightforward. You can do this by going Visual Studio and choosing a project type you want. In this case I want to use .Net 6, so I'm picking the one which doesn't say (.Net Framework):

After naming the project, you get an option of using the "new" style of writing a C# program using Top-level statements, or going with the more classic version of choosing a project. For the purposes of this demo, I'm going with the classic way of building Program.cs as I tend to find it a little confusing to show how C# works to a beginner. So for this I'm clicking the checkbox Do not use top level statements:

Just for context, if you chose to use top level statements it would look like this:

Console.WriteLine("Hello, World!");

By contrast, this is an example of what it looks like without:

namespace SimpleTestingApp
{
    internal class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello, World!");
        }
    }
}

At this point, we're going to throw in our first test project to be able to cover this program. We can do that by right-clicking on the solution and going to Add > New Project

At this point you're going to see a familiar screen asking you to select a project. Here we're looking for a test project. By default, Visual Studio gives you access to 3 different testing frameworks based on your choice of project. These are MSTest, XUnit and NUnit. Ultimately, all 3 of these testing accomplish the same thing, and I've worked with all of them at various points in my career. The difference is mainly in exact syntax and documentation. Although, it's generally considered that MSTest is a little "older" than NUnit or XUnit, so I tend to see it less now. For the purposes of this demo, I'm going to go with NUnit:

At this point we just hit next, and we get an option to name the project. In C#, the naming convention is usually <project name>.Test as our test projects are designed to be directly related to the project we want to test. In this case, as the project I want to test is called SimpleTestingApp, my test project is SimpleTestingApp.Test.

At this point a project should be created a default test file and global using file containing NUnit:

Now at this stage we could start writing tests, but in reality we normally abstract code out of Program.cs as much as possible because it's static and difficult to test. While we could just create some extra files with the SimpleTestingApp project, it's far more likely to create a separate project that contains most of the code. In this case I'm going to create a SimpleBankingApp project. Given I want to unit test all of my code, what I tend to do is try and group my test projects with my "real" code. To do this, I right-click on the solution in Solution explorer and go to add > New solution folder and name the folder SimpleBankingApp.

This is directory I'll create my new projects in, which in this case is a class library and a test project. Once you've done this, your solution explorer should look something like this:

As you can see, Visual Studio automatically changes the icon of the project based on what it is.

Project to test

So at this point I'm just going to write some code as this section isn't that important, it's testing that we're interested in. For context, this is what my application looks like:

and some of the more important classes:

public interface IInterestCalculator
{
    decimal CalculateInterest(IAccount account);
}

public class InterestCalculator : IInterestCalculator
{
    private readonly IRateCalculator _standardRateCalculator;

    public InterestCalculator(IRateCalculator standardRateCalculator)
    {
        _standardRateCalculator = standardRateCalculator;
    }

    public decimal CalculateInterest(IAccount account)
    {
        return account.Age * account.Balance * _standardRateCalculator.StandardRate;
    }
}

public interface IRateCalculator
{
    decimal StandardRate { get; }
}

public class StandardRateCalculator : IRateCalculator
{
    public StandardRateCalculator()
    {
        StandardRate = .032M;
    }

    public decimal StandardRate { get; }
}

public class Customer
{
    public string? Name { get; set; }

    public string? TelephoneNumber { get; set; }

    public List<IAccount> Accounts { get; set; } = new List<IAccount>();
}

public interface IAccount
{
    decimal Balance { get; set; }

    int Age { get; set; }
}

This is a really straightforward application which I'm going to use as the base for unit testing. So I'm going to move on to my testing project for setup.

Test project setup

First up, I'm using dependency injection to pass in interfaces to the InterestCalculator class. On top of being good practice depend on abstractions, this means my project is perfect for mocking which is the process used to isolate complex pieces of code. Additionally, one of the primary purposes of mocking is to allow you to run tests without needing to rely on third parties. For example, when doing web requests or database calls.

In terms of mocking there are several frameworks you can use, but I've mainly relied on Moq and NSubstitute. Within this demo, I'm going to use NSubstitute as I've found it a little easier to use.

To do this we're just going install it into the test project using the Visual Studio NuGet package manager. All you need to do is right-click on SimpleBankingApp.Test and select "Manage NuGet packages" and then search for NSubstitute:

Writing tests

We're now ready to write some tests. To start with, as it's slightly more complicated than anything else, I'm going to write some tests for the InterestCalculator class. To do this, I'm going to just create a folder called Unit and then a class within that folder called InterestCalculatorTest in my test project. I'll also delete the generate file UnitTest1.cs:

I can now move onto setting up my tests. This is straightforward in this case, as I have very little to set up. Pretty much all I want to do is mock out IRateCalculator for use throughout the class and as I only need one version of this throughout, I thend to set them up in the constructor:

public class InterestCalculatorTest
{
    IRateCalculator _rateCalculator;

    public InterestCalculatorTest()
    {
        // mocking the rate calculator
        _rateCalculator = Substitute.For<IRateCalculator>();

        // telling the rate calculator to return 0.05;
        _rateCalculator.StandardRate.Returns(0.05m);
    }
}

If this was to change on a per test basis, I could either construct this object in the test itself, or use a setup method that's run before every test. For example:

[SetUp]
public void Setup()
{
    // mocking the rate calculator
    _rateCalculator = Substitute.For<IRateCalculator>();

    // telling the rate calculator to return 0.05;
    _rateCalculator.StandardRate.Returns(0.05m);
}

While there's a lot of setup to get to this point, I'm now ready to write my first test. In this case, I'm going to check that I can calculate an interest. In order to do that I'll just create a standard void method. The hardest part now, is how to name a test. There are lots of different ways of doing this, but I'm a fan of a 3 part test name, which is separated by underscores. They contain the following:

The method you're testing
The expected outcome
What you're doing to get that outcome.

In this case, my test name is going to be CalculateInterest_MakesAnInterestCalculation_WhenCalledWithStandardValue. I know this is a bit wordy for the name of a method, but the idea is that it's very clear about what this test is doing.

Once this is done, I instantly add the following to my test:

public void CalculateInterest_MakesAnInterestCalculation_WhenCalledWithStandardValue()
{
    // Arrange

    // Act

    // Assert
}

This is to always remind me of the 3 parts I need to do when unit testing, and indicates to a developer coming after me what each section of the code is doing. This because large can b e some of the most confusing pieces code to figure because you can need to do a lot of setup.

Anyway, the parts of a test are as follows:

Arrange
1. Anything that needs to be done to set a test up for a run
Act
1. Where you actually run the method to test
Assert
1. This is where you check that the response from the Act step matches what you expect to happen.

I can then fill out the test to be something like this:

[Test]
public void CalculateInterest_MakesAnInterestCalculation_WhenCalledWithStandardValue()
{
    // Arrange
    // I'm setting up an account for the test, so this is arrange
    var account = new Account()
    {
        Age = 1,
        Balance = 1000
    };

    // I'm constructing the interest calcualtor, which is arrange
    var interestCalculator = new InterestCalculator(_rateCalculator);

    // I know what the calculation is, so I'm making sure my calculations will match
    // This is usally frowned upon due to creating strong links between code and test
    var expectedInterest = account.Balance * account.Age * _rateCalculator.StandardRate;

    // Act
    // Running the method I want to run is act
    var interest = interestCalculator.CalculateInterest(account);

    // Assert
    // Asserting that the interest will be greater than 0
    Assert.That(interest, Is.EqualTo(expectedInterest));
}

NOTE: the [Test] decorator is the way that the code flags to Visual Studio that this is a test.

I can now check my test is running from the test explorer. You can get this window by going to View > Test Explorer. Then you just need to press the green button at the top to run a test:

We're now on to the point that can be a little difficult for developers as it requires a different mindset to standard development. We need to work out how to break our code and make sure it can handle those cases. The first one I can think of, is what happens if somebody has a negative account value? Well, I'd probably ask a question in Teams to see what other people thought, and in this case the code needs to throw an error.

So I'll go back to modify the CalculateInterest method so that it throws an error on negative balance:

public decimal CalculateInterest(IAccount account)
{

    if (account.Balance < 0)
    {
        throw new ArgumentException("you can't calculate negative balances");
    }

    return account.Age * account.Balance * _standardRateCalculator.StandardRate;
}

I can now write a test for this. As this is a test related to the previous one I've worked on, I'll copy the last test and modify it to work for me:

[Test]
public void CalculateInterest_ThrowsAnArgumentException_WhenCalledWithNegativeBalance()
{
    // Arrange
    var account = new Account()
    {
        Age = 1,
        Balance = -1000
    };

    var interestCalculator = new InterestCalculator(_rateCalculator);

    // Act and Assert
    Assert.Throws<ArgumentException>(() => interestCalculator.CalculateInterest(account));
}

As you can see, I've combined the Act and Assert sections in the test. This is because the method throws an exception, which would stop the code execution in normal circumstances. Therefore, the testing framework needs to catch the error, and we can do this with Assert.Throws, but this does mean we can't separate the Act and Assert.

The next test to talk about, is that if this was a bank, we would have multiple customers, who can hold multiple accounts that we need to calculate the interest for. You could manually construct these objects if you want, but I'm going to use a package called AutoFixture to make life easier.

How AutoFixture works is that it takes in an object and fills out every field with dummy data. The big benefit of this, is that if your object changes in the future, you don't need to modify all your tests to take in the new values, AutoFixture will automatically fill them out for you. You can get AutoFixture in the same way you added NSubstitute to the project.

I can then use AutoFixture in the following manner:

[Test]
public void CalculateInterest_CalculatesInterestInUnderTenSeconds_WhenCalledWithTenCustomersWithTenAccountsEach()
{
    // Arrange
    Fixture fixture = new Fixture();

    // This allows us to use fixtures on interfaces, by telling AutoFixture to use a concrete implementation
    fixture.Customizations.Add(
        new TypeRelay(
            typeof(IAccount),
            typeof(Account)));

    var customers = fixture.CreateMany<Customer>(10);
    foreach (var customer in customers)
    {
        customer.Accounts = fixture.CreateMany<IAccount>(10).ToList();
    }

    var interestCalculator = new InterestCalculator(_rateCalculator);

    // setting up a stopwatch to check how long this will take
    Stopwatch stopwatch = new Stopwatch();

    // the maximum time we're giving this method to run
    var maxTime = TimeSpan.FromSeconds(10).TotalMilliseconds;

    // Act
    stopwatch.Start();

    foreach (var customer in customers)
    {
        foreach (var account in customer.Accounts)
        {
            interestCalculator.CalculateInterest(account);
        }
    }

    stopwatch.Stop();

    // Assert
    Assert.That(stopwatch.ElapsedMilliseconds < maxTime);
}

This is a fairly advanced implementation of using AutoFixture as I'm using an interface, and the test isn't great as it's pretty slow, but I hope it helps show how powerful it can be when you just need some fake data.

Other things of note

You might have noticed when you were looking in NuGet, there was a package called coverlet installed into the project:

This is a really great tool we can use to implement a paid feature into the community edition of Visual Studio, which is the ability to get test coverage. It's also the package I use when building CICD pipelines in order to show test coverage in ADO and SonarQube together.

To use this locally, you can run coverlet by going to the package manager console and running the following command:

dotnet test --collect:"XPlat Code Coverage"

In order to view the results in a human-readable format, you will need to install the NuGet package Report Generator. Unfortunately, this isn't quite as easy as just installing the NuGet package.

The easiest thing to do is find where the coverage report is generated to, which in my case was a folder called TestResults which was at the top level of test application:

You then need to run the following 4 commands from within that folder:

dotnet tool install -g dotnet-reportgenerator-globaltool

dotnet tool install dotnet-reportgenerator-globaltool --tool-path tools

dotnet new tool-manifest
dotnet tool install dotnet-reportgenerator-globaltool

Once you've done this, you should see a .config folder containing a JSON file and a tools folder which has a copy of report generator in it. This indicates you're ready to run report generator with this command:

 dotnet reportgenerator "-reports:.\coverage.cobertura.xml" "-targetdir:.\report"

You should see a folder generated called report that contains a number of files. All you need to do to see the report now, is find the file ending in .html and opening it:

How to setup a simple static website using Svelte (with login)

Jack Lewis — Fri, 19 May 2023 11:57:51 +0000

Preface

Svelte is an up and coming JavaScript framework being developed by Rich Harris (and the Svelte team). It's gained acclaim for being very simple to use, reducing boilerplate and generally being a nice platform to develop new web apps on. This article is an attempt to explain how you can leverage the power of Svelte to create modern static sites that can be hosted from simple file storage like S3 in AWS.

While hosting the application is outside the scope of this article, I do have another here which explains how to do the hosting within it.

It needs to be understood, that converting Svelte to run in a static context will; mean that a lot of the features touted by the Svelte team will be disabled. However, I still feel it's worth it, as writing in Svelte is still a really nice experience.

Benefits

The main benefit of this approach is cheap hosting. You don't need to run a docker container or a machine in your cloud service provider of choice, which reduces the cost of the website as you're only paying for serving files. Additionally, many cloud service providers have products that allow you to leverage a content delivery network easily. Finally, static applications are fast because there's no need communicate backwards and forwards with a server, it just downloads some files to the browser.

Drawbacks

Essentially, all server based code in a Svelte will not work using static hosting. Given Svelte is split into code that runs everywhere and code that runs in the server, this can be a little annoying. You'll need to remove all .ts/.js files that contain the server keyword (i.e.: +page.server.ts or +server.js) as these files won't work. Additionally, some features ofSvelte won't work such as dynamic routing and server side rendering.

Tutorial

Prerequisites

You'll need an up-to-date version of your favourite node based package manager. I'm using npm, but pnpm and yarn both work.

Installation

to start with we need to create aSvelte app from the terminal:

npm createSvelte@latest

It will ask you a few questions about how you want to set up your project. For the cases of this demo, I went with a skeleton project and with TypeScript. You select options (such as prettier) using the space bar.

You then need to run the installation:

npm install

Just for context, you can run npm build at this stage to see what the built output looks like without convertingSvelte to run as static. You can see it just generates a large number of files in the .svelte-kit folder. You can run from this directory using npm run preview:

Converting to static

It's really quite straight forward to convert aSvelte app to run in a static context. Svelte does this using adapters. By default, Svelte uses the auto adapter which works best for general cases. There are a few different adapters, and you choose which one based on what you're doing. For example, there are adapters based on providers (such as Netlify or Vercel) however, the one we're interested in, is adapter-static which makes Svelte run like a static site generator.

In order to get this to work, all you need to do is to install the adapter-static package:

npm i -D @sveltejs/adapter-static

Then go to svelte.config.js at the top level, and change the line talking about adapter-auto to say adapter-static. I.e.:

import adapter from '@sveltejs/adapter-auto';

import adapter from '@sveltejs/adapter-static';

NOTE: there's a lot of additional settings you can set for this, but for the purposes of this guide, we're not going change any of the defaults.

At this point you can try and build the application, and it will instantly spit out a warning about dynamic routes in the src/routes folder. It gives you a bunch of options on how to fix this, but the simplest is to create a file called layout.{ts|js} in src/routes and then place the following into it:

export const prerender = true;

This forces all files in the src/routes folder to be prerendered, which is what we need for a static site.

At this point, if you run npm run build you should see a build folder created that contains an index.html, which is the entry point for the static site:

UI library

Usually, one of the first things I do on creating a new web app is to throw a UI library in to help style components. There are several UI libraries that can be used by Svelte, but in this case I went with daisyUI because it's a fairly popular UI library which includes tailwind. To install daisyUI, you first need to install tailwind. There's a few different ways to do this (such as this guide), but the easiest way I've found is the following command, which also adds PostCSS and AutoPrefixer:

npxSvelte-add@latest tailwindcss

You should then be able to install daisyUI:

npm install daisyui

The final step for daisyUI is to then add a plugin for daisyUI into the tailwind.config.cjs folder. It should look something like this:

/** @type {import('tailwindcss').Config}*/
const config = {
    content: ['./src/**/*.{html,js,svelte,ts}'],

    theme: {
        extend: {}
    },

    plugins: [require("daisyui")],
};

module.exports = config;

You can tell that daisyUI is installed correctly in 2 ways, the first is to add a styled component, like a button:

<button class="btn">Button</button>

or alternatively, you can run npm run build or npm run dev and the output tells you that daisyUI is installed:

Animations

While Svelte has a pretty good built-in library of animations, if I do need some more custom animations, I don't actually use a library, but I do use Animista to pick out something which works for me.

Repository Setup

I talk about this in more detail in my article here, but I'm a big fan of adding pre-commit into my git repositories to help out.

Login

Whether it's a good idea or not, it's definitely possible to add login functionality to a static site. This can be done through JSON web tokens and the OAuth Authorization code flow. How this works is that we offload authorization to a third party, that then returns a token that verifies a user is logged. That token can then be used to authenticate with an API somewhere. This can be a little difficult to get your head around, but this essentially what happens when you click the "log in with Google/Facebook/GitHub" button on a website and is known as "Single Sign On" or "SSO".

We're specifically using the authorization code flow as this is the one needed for when the entire website is served to the browser i.e. when using a static website.

If you do want to add login functionality to a static website, you need to make sure that all data related to a user comes in from an authenticated backing API. For example, API gateway in AWS. The reason for this is that any data you serve to a website can be modified by the client, meaning that somebody could "force" a login by just changing the JavaScript locally. You need to stop this by making something like an OAuth token required on every call to the backend, as this is verifying that a user has access to those resources.

Why?

This can be a little counterintuitive to do as if you're doing this, why don't you just use server side features of Svelte and avoid a separate API all together, which would likely be faster than offloading the calls to the browser. The answer to this is mainly in cases where you have a public API being used by others, as well as yourself or that you're using the same resources with multiple OAuth flows, such as the client credential flow for machine to machine calls. As such, this can be useful on admin portals for application support.

How to add

Just about every cloud provider has their own implementation of a JavaScript library that can offload the complexity of writing your own implementation. For example, AWS has Amplify, GCP has Firebase and Azure has Identity. You can also use a more generic library like Auth.js. However, I'm going to go with Auth0 in this guide as I've not used this service before, and it's a slightly more "neutral" option than going for a service offered directly by a cloud provider. Additionally, Auth0 will allow you to get started without needing to add a credit card, which makes it useful for demonstrations.

Picking the best service for your needs is outside the scope of this article, but safe to say a lot of considerations to keep in mind. This can be everything from a cloud provider offering a tighter integration with their own products to the number of monthly free users you're allowed (called MAU's) to the number of SSO providers offered for integration.

Setup

To start with you need to sign up for an account with Auth0 and add the Auth0 single page app SDK to your Svelte application:

npm install @auth0/auth0-spa-js

After this, you need to go into the front page of Auth0, and click "create application" and select "Single Page Application":

There's a large number settings you can use, but that's outside this guide. However, a nice feature of the documentation is that if you're logged in, the documentation will be updated to use details for your account:

You then need to set callback and logout URL's. Svelte usually sets the startup port as 5173, but pick whatever you need:

I've put localhost in at the moment as I'm just running this locally. Obviously if you're actually running this, just set the callback and logout URL's to the valid paths there.

At this point all you need to do to set up the login functionality is to construct the auth0 object:

import { Auth0Client } from '@auth0/auth0-spa-js';

const auth0 = new Auth0Client({
    domain: '<domain>',
    clientId: '<client id>'
});

and then you can start the login process by setting up a button which calls Auth0:

async function login() {
    await auth0.loginWithRedirect({
        authorizationParams: {
            redirect_uri: 'http://localhost:5173/'
            }
    })
}

<button on:click={login} class="btn">Login</button>

If you press the button, it should take you to the Auth0 login page where you can now sign up a user. It also handles email verification and SSO. Once you do this, it should redirect you back to your application. You can tell it's working at this point if you can see that you have token in the URL as a query parameter. For example:

http://localhost:5173/?code=tlcjskafaUOODPicz_P83g6YrwgiFxOO7PXv0gNZAKDsC&state=ekdCQzFRZjllTmJyOTUwcU5XWjNqV0RQN04wZC11alZ0UHB1Q0hPNElYTQ%3D%3D

In order to handle this, we need to call the auth0.handleRedirectCallback() function on page load. This can be done using onMount in Auth0:

import { onMount } from 'svelte';

onMount(async () => {
    await auth0.handleRedirectCallback()
    .then(async () => {
        const user = await auth0.getUser();
        console.log(user);
    })
    .catch(() => {
        console.log("not authenticated");
    });
})

Logout functionality is just as straight forward:

async function logout() {
    await auth0.logout();
}

For context, here's a full minimal implementation of this:

<script lang="ts">
    import { Auth0Client, User } from '@auth0/auth0-spa-js';
    import { onMount } from 'svelte';

    const auth0 = new Auth0Client({
        domain: '<auth0 domain>',
        clientId: '<auth0 client id>'
    });

    let user: User | undefined;

    async function login() {
        await auth0.loginWithRedirect({
            authorizationParams: {
            redirect_uri: 'http://localhost:5173/'
            }
        })
    }

    async function logout() {
        await auth0.logout();
    }

    onMount(async () => {
        await auth0.handleRedirectCallback()
        .then(async () => {
            user = await auth0.getUser();
        })
        .catch(() => {
            console.log("not authenticated");
        });
    })

</script>

<button on:click={login} class="btn">Login</button>
<button on:click={logout} class="btn">Logout</button>

{#if user != undefined}
    <p>hello {user.nickname}</p>
{/if}

The final thing to talk about, is that if you want to get the token to use in an authorization header for retrieving data through an API, you can get this using either auth0.getTokenSilently() or auth0.getTokenWithPopup() depending on your needs.

While it's outside the scope of this article, Auth0 provides documentation about how to integrate with all the major cloud providers. For example, the documentation for how to implement Auth0 with AWS API Gateway is found here.

A quick tour of collections in C#

Jack Lewis — Fri, 12 May 2023 19:32:25 +0000

Preface

Collections in C# are really quite useful when used correctly, but they tend to be used far less often than lists despite fulfilling a similar purpose. There are a lot of versions of a collection in C# and this article aims to display why you should use collections in your code, as well as how to use them. It's also important to understand that by Collection I mean, a specific type of object, rather than the System.Collection namespace or a "collection" which can include essentially anything that can be described as "a collection of objects". I've also included a few of the less common objects in C# that are called a collection in the name of the class.

What is a collection?

It's really quite straightforward, in that a collection is a list of things and most of them implement IList. However, there are some nuances in this. For example, methods in a collection are marked as virtual, meaning they can be overridden for custom implementations. What this means is that additional functionality (or custom handling) is much easier to add to a collection over a list. On the other hand, lists have more functionality related to sorting and searching for data. What this all adds up to, is that if you think you may require a custom implementation of a list (for example, to fire events when items are added), use a collection. In fact, it's generally recommended providing collections to users rather than lists, but in reality this rarely happens as lists work in 99.99% of use cases. Also, in my experience it's far more likely to need sort and search functionality than any other operation.

Types of collection

Once again, I'm going to go through some collection types in C# as well as the differences between them.

Collection

This is the "default" class for a collection. MSDN does describe this as a "base class" but it's not an abstract class and should not be confused for CollectionBase.

This collection can be declared in a very similar fashion to a list:

Collection<string> myCollection = new Collection<string>();

Adding, removing and inserting data in a collection is quite straightforward:

myCollection.Add("item 1");
myCollection.Insert(1, "item 2");

myCollection.RemoveAt(1);
myCollection.Remove("item 1");

myCollection.Clear();

While finding data in a collection isn't as extensive as a list, there's still a lot of ways you can find data in a collection:

// index based
string itemOne = myCollection[0];
string itemTwo = myCollection.ElementAt(1);
IEnumerable<string> firstTwo =  myCollection.Take(2);
IEnumerable<string> lastTwo = myCollection.TakeLast(2);

// item based
string anotherItemTwo = myCollection.First(i => i == "item 2");
string? nullItem = myCollection.FirstOrDefault(i => i == "not in the list");
IEnumerable<string> items = myCollection.Where(x => x == "item 1");
IEnumerable<string> amotherMultipleitems = myCollection.TakeWhile(x => x == "item 1");
IEnumerable<string> anotherItemList = myCollection.Intersect(new List<string>() { "item 2" });

// returning indexes
int index = myCollection.IndexOf("item 1");


// boolean based
myCollection.Contains("item 1");
bool isEqualTo = myCollection[1] == "item 2";

While there aren't any methods to sort the collection, there are functions that can be used to order the collection:

IOrderedEnumerable<string> orderedEnumerable = myCollection.OrderByDescending(x => x == "item 2"); // item 2, item 1
IOrderedEnumerable<string> orderedEnumerableTwo = myCollection.OrderBy(x => x == "item 1"); // item 2, item 1

Finally, collections provide a lot of functions to cast to other object types. For example, the classic ToList(),ToString() and ToArray() but also to various specialized objects, such as ToSortedSet().

The Collection class lives in the System.Collections.ObjectModel namespace.

Keyed Collection

Frankly a keyed collection should probably be discussed in my article about dictionaries as they are pretty closely related. There are some differences though. For example, a keyed collection guarantees order whereas a Dictionary does not. Additionally, a keyed collection is actually an abstract class, meaning that it needs to be inherited from to work:

// This is a pretty useless class as it just sets the key to the value.
public class ConcreteKeyedCollection : KeyedCollection<string, string>
{
    protected override string GetKeyForItem(string item)
    {
        return item;
    }
}

Once this done, you can declare a keyed list like this:

KeyedCollection<string, string> myKeyedCollection = new ConcreteKeyedCollection();

While I did say this acts like a dictionary internally, you still add items without a key and then this key is set based on the data in the abstract class:

myKeyedCollection.Add("test");

What's interesting about the keyed collection, is that it contains both a collection and a dictionary. This means you have access to data retrieval in both the fashion listed above and methods found on dictionaries, like TryGetValue(). This can be seen when we take a look at the QuickWatch window:

Obviously, this does mean you get a slight performance hit compared to just using a straight dictionary (as you're maintaining a list as well), there is also the increased code complexity due to having to create concrete implementation of the class. Therefore, unless you specifically need an ordered dictionary, I would just stick with an ordinary dictionary.

The keyed collection abstract class lives in the System.Collections.ObjectModel namespace.

Observable Collection

While the collection might be the "default" class, the one I most often see used, is the observable collection. This is especially true within C# GUI programs such as WPF. The reason for this, is that when items are added to this type of collection, an event is fired. This makes it extremely useful for two-way data binding as compared to a list, which does not tell an observer when the collection has changed.

Observable collections can be declared like this:

ObservableCollection<string> myObservableCollection = new ObservableCollection<string>();

As I mentioned above, the main difference between a standard collection and an observable collection, is an event called CollectionChanged which can be used in the same way as a standard event:

public void SomeMethod()
{
    myObservableCollection.CollectionChanged += OnCollectionChanged;
}

private void OnCollectionChanged(object sender, NotifyCollectionChangedEventArgs e)
{
    throw new NotImplementedException();
}

The observable collection class can also be found in the System.Collections.ObjectModel namespace.

Read Only Collection

As in my articles on dictionaries and lists, a read only collection is used when you don't want a user to be able to modify a collection after initialization. Unlike lists and dictionaries though, a read-only collection is not an interface and is instead a concrete implementation:

List<string> myStringList = new List<string>()
{
    "item 1",
    "item 2"
};

ReadOnlyCollection<string> myReadOnlyCollection = new ReadOnlyCollection<string>(myStringList);

Finally, the read-only collection class can also be found in the System.Collections.ObjectModel namespace.

Blocking Collection

Within collections, this is the thread-safe version. It's essentially a C# implementation of the producer-consumer pattern. In addition to blocking, there is also bounding functionality, where you can set the maximum size of the collection:

BlockingCollection<string> myBlockingCollection = new BlockingCollection<string>();

// only allows 2 items before being blocked
BlockingCollection<string> myBoundedBlockingCollection = new BlockingCollection<string>(2);

you can add and remove items in a few different ways that are different from standard:

myBoundedBlockingCollection.Add("item 1");
myBoundedBlockingCollection.Add("item 2");
myBoundedBlockingCollection.Add("item 3"); // this will block the thread until an item is removed
myBoundedBlockingCollection.TryAdd("item 3"); // this continues execution even if bounded

// removing items
myBoundedBlockingCollection.Take(); // this is FIFO and has several overloads
myBoundedBlockingCollection.Take();
myBoundedBlockingCollection.Take(); // blocks the thread until something added
myBoundedBlockingCollection.TryTake(out string? item, TimeSpan.FromSeconds(1)); // the second parameter is a blocking timeout for if the collection is empty

Most notably, addition/removal operations accept cancellation tokens for additional use under threaded workloads.

In addition to this, blocking collections provide a few additional properties:

int capacity = myBoundedBlockingCollection.BoundedCapacity; // this would be 2

bool addingCompleted = myBoundedBlockingCollection.IsAddingCompleted;  // false
myBoundedBlockingCollection.CompleteAdding();
bool addingCompletedAfterCompleteAdding = myBoundedBlockingCollection.IsAddingCompleted; // true

bool fullyCompleted = myBoundedBlockingCollection.IsCompleted; // only true when `CompleteAdding` has been called and the collection is empty

Finally, if you're working with multiple blocking collections in multithreaded workflow, the class itself has some static methods that can be helpful:

List<BlockingCollection<string>> myBlockingCollectionList = new List<BlockingCollection<string>>()
{
    myBlockingCollection,
    myBoundedBlockingCollection
};

// just picks one of the blocking collections to add/remove the item
BlockingCollection<string>.AddToAny(myBlockingCollectionList.ToArray(), "an item");
BlockingCollection<string>.TakeFromAny(myBlockingCollectionList.ToArray(), out string? item);

// there are also `Try` versions of both these methods

The blocking collection lives in the System.Collections.Concurrent namespace

Name Value Collection

The name value collection doesn't work the same way as the other collections in this article and is more like a combination of a dictionary, collection and tuple. It's primarily used for holding things like headers for web requests. Additionally, a name value collection only accepts strings.

It can be declared like the following:

NameValueCollection myNameValueCollection = new NameValueCollection();

Moreover, what makes the name value collection different from a dictionary, is that the keys do not have to be unique. The best way I've found to think about this, is that while addition works by adding a single string to both the key and value, internally it adds the value to an array:

myNameValueCollection.Add("item 1", "item 1");
myNameValueCollection.Add("item 2", "item 2");
myNameValueCollection.Add("item 1", "item 3");

string[]? strings = myNameValueCollection.GetValues(0); // contains "item 1" and "item 3"
string? commaSeparated = myNameValueCollection["item 1"]; // interestingly, this returns a comma separated list of values

The name value collection lives in the System.Collections.Specialized namespace

Visual Basic Collection

Generally I'd consider this collection a forerunner to the collection and dictionary in C# that has been carried forward (I'm pretty sure it came from Visual Basic). It can be constructed like the following, of which I've included the namespace to avoid confusion with the collection referenced above:

Microsoft.VisualBasic.Collection myVisualBasicCollection = new Microsoft.VisualBasic.Collection();

I wouldn't recommend this collection in ~~modern~~ C# at all, because it's a holdover from Visual Basic, and it's not generic, but instead uses object. This means it's not type-safe. For example this is perfectly valid:

myVisualBasicCollection.Add("1");
myVisualBasicCollection.Add(1);

Also, this is technically a key-value collection where you can choose not to add a key. This is another reason I would not to use this, but you can add a key in the following way:

myVisualBasicCollection.Add("value", "key");

Given this is a much older class, there is also a lot fewer methods that can be used to interact with objects held within one.

As mentioned above, this collection can be found in the Microsoft.VisualBasic namespace.

Turning Git into a large file backup solution

Jack Lewis — Fri, 05 May 2023 19:21:00 +0000

NOTE: this code, while working, is very much proof of concept and as such has lots of bugs. I'm also not planning to fix these as this is a pretty stupid idea. However, I thought it was interesting enough to share.

Background

This whole thing came about from a few realizations:

GitHub allows a maximum of 100MB per file before they block uploads
Based on this answer from Stack Overflow, the "hard limit" for the size of a repository in GitHub is 100GB, but really should be kept under 5GB (and preferably 1GB) - it's a little difficult to find information on this though
GitHub lets you have an unlimited number of repositories
There's an endpoint that lets you set up a GitHub repository programmatically
zip and tar files both have the ability to easily be split apart

So, pretty much from all of this, can I build something that can I take in a large file, split it apart into manageable chunks, upload it via Git and then reassemble everything when I need to restore from backup?

You talk about GitHub, why are you using Git?

This comes down to the simple premise that GitHub would probably not be too happy with me trying this, so better not to do it at all.

TL;DR - yes I can

Repository here:

jlewis92 / gitBackup

gitBackup

GitBackup contains the following commands:

-r, --restore    Restore files.

-b, --backup     Backup files.

--help           Display this help screen.

--version        Display version information.

In addition to the command line arguments, there is an appSettings.json that can be set in the executable directory to further modify how the program works. This is what this file looks like with all the defaults:

{
  "AppSettings": {
    "FilesToBackupLocation": "<current folder>"
    "BackupLocation": "<current folder>\\backup",
    "RestoreLocation": "<current folder>\\backup",
    "FilesToRestoreLocation": "<current folder>\\restoreTestFolder",
    "RecursiveFileBackup": false,
    "GitSettings": {
      "RepositoryNamingConvention": "gitBackup-",
      "RepositorySizeInKilobytes": 800000,
      "GitRemoteLocation": "", // this value will need to be set at least
      "OverwriteOnAdd": true,
      "Username": null,
      "Email": null,

…

View on GitHub

Video in action here:

Why?

Honestly, it seemed like a fun challenge and that's pretty much it. Really, this is a terrible idea to choose as a backup strategy. It's incredibly slow (that video above is over 1 minute of waiting for files to transfer locally) and if you know anything about Git, binary files (such as zip files) are terrible thing use Git to store. This is because Git works by analysing differences in files and saving them. If you're using binaries, this means that git has to store a stupid amount of extra data to make it work. Additionally, because of how I wrote this software, you could have up to 4 copies of the same data on disk at the same time, which is pretty bad. Finally, it needs to be said that relying on a company to not just delete all your files if you do actually do this, it's probably a bad idea to even attempt to store anything in this manner.

I know this section is more why not to do this, which is why I really have no excuse for doing this, other than to say I could.

Tooling choice

I'm primarily a C# developer, so I'm going to use C# and given I'm going to be interacting with the Git CLI, the easiest project to choose is a console application. Additionally, I decided to go with zip over TAR as I'm using windows. I'll also need a database to store where the files live in GitHub and given I'm already storing everything in files, and need to move them around, a file based database was my best option, and from these SQLite before, so this was my tool of choice.

Flowcharts

At this point I needed to visualize the flow of the application in my head. I decided I needed 2 commands to start with, backup and restore. Given there's a fair amount of complexity on how exactly each command will work, here are some flowcharts explaining it:

Backup

Restore

Database Structure

As I said, I needed a simple database to keep track of where everything lives within the repositories and because I'm going to want to back that file up, the best choice for me was to use SQLite. I decided to go with 3 tables:

Manifest Entries: holds details related to each file.
Repository Entries: holds details related to each repository being tracked
Compressed File Entries: holds details related to compressed file, as well as the manifest entry and repository entry that is attached to the compressed file.

This fits together something like this:

Code Structure

I depended heavily on interfaces to build up this code base. It's probably simplest to just show off the class diagram for this code below. If you're interested in exploring the code itself, it's linked above.

Packages

Within the code itself, I made use of a few packages to make life easier on myself. The main ones chosen are listed here:

CommandLineParser
- Creates better syntax structures for command line arguments, such as -b for --backup
DotNetZip
- Popular library for working with zip files in .Net
- This library was chosen as it has support for splitting files
LibGit2Sharp
- It's a library for working with Git in C#
Entity Framework Core
- Common C# ORM used to link objects in code to the SQLite database
- One of the nice things about this implementation, is that I can call EnsureCreated() to create a database if it doesn't already exist
Serilog
- Excellent logging tool
TestableIO.System.IO.Abstractions
- This library creates abstract version of System.IO which makes them far more testable

Challenges

There were a few issues I did run into when building this code out - thought I'd throw these out there in case anybody runs into something similar

Working with filesystems

I had to work heavily with the file system on this project. Everything from the zip library outputting directly into files, to moving things around for the git repositories. While the code itself doesn't do anything fancy, System.IO is pretty untestable in C# and I wanted to write some tests for this. As a result, I needed some way to test the file system. I settled on using the TestableIO library mentioned above which allows you to use dependency injection on an interface class, rather than writing my own wrapper.

SQLite Write-Ahead Logging

If you're not sure what write-ahead logging is, there's a full explanation here. Essentially, it's a way of implementing ATOMIC rollback in SQLite. However, how this works is that new commits to the database are held in a separate file, and then occasionally added back to the database in a process known as "checkpointing". Unfortunately, because I'm moving the database file around shortly after writing, the additional time taken waiting for checkpointing to occur meant that I'd be uploading a version of the database that doesn't contain the latest changes. While I did attempt to modify when the checkpointing occurred, I decided to avoid problems entirely and turned this feature off.

Running the code

GitBackup contains the following commands:

-r, --restore    Restore files.

-b, --backup     Backup files.

--help           Display this help screen.

--version        Display version information.

{
  "AppSettings": {
    "FilesToBackupLocation": "<current folder>",
    "BackupLocation": "<current folder>\\backup",
    "RestoreLocation": "<current folder>\\backup",
    "FilesToRestoreLocation": "<current folder>\\restoreTestFolder",
    "RecursiveFileBackup": false,
    "GitSettings": {
      "RepositoryNamingConvention": "gitBackup-",
      "RepositorySizeInKilobytes": 800000,
      "GitRemoteLocation": "", // this value will need to be set at least
      "OverwriteOnAdd": true,
      "Username": null,
      "Email": null,
      "Password": null
    },
    "FileCompressionSettings": {
      "FileSizeInKilobytes": 80000
    }
  }
}

In the future

So now I've written this and established that it's pretty useless, I'm not likely to come back to this project. However, if I was, there's a few improvements that could be made quickly.

Additional logging
- I've added some, but it pretty poor
Better error handling
- this was built quickly and as such error handling is pretty rudimentary
More tests
- I've written a few, but I'd need more to write a fully robust test suite
Multithreading
- Granted the majority of the time this program executes is waiting for Git, but writing in the ability to pack/unpack zip files and upload git repositories in a multithreaded fashion would likely speed up the execution of this code quickly

A quick tour of lists in C#

Jack Lewis — Fri, 28 Apr 2023 20:29:47 +0000

Preface

Some of you might have seen my previous article around dictionaries in C#. Seemed to be fairly popular, so I thought I'd write a similar around lists in C#. AS in dictionaries, C# provides a lot of versions of a simple list. I've not seen previous articles comparing and contrasting the different versions of a list and where to use them.

What's a list?

Quite simply, a list is a data structure in C# used to hold list of objects of the same type. They are also generic, meaning the type in the list is declared at initialization. Lists are also specifically an ordered data structure, meaning that the order you add items to the list is the order of the items in that list, including duplicate values. Finally, lists are "designed" to be throwaway. This is because it can be difficult to extend a list in the same way a collection can be. In fact, while collections are outside the scope of this article, it's generally considered better practice to return a collection from public methods, but in reality a list works fine 999 times out of 1000.

There is more information on lists found here.

Types of list

As in the previous article, I'll run through some of the features I use most often in the "default" list and move onto other implementations afterwards.

Note: I'm using nullable strings (i.e.: string?) throughout this article to make it more clear.

List

Lists can be declared like the following example using a list of strings:

List<string> stringList = new List<string>();

In general, data can be added, inserted or removed from the list like the following:

There are a lot of methods that can be used to find things in lists. Here's a few of the more common ones.

// The simplest way to find an item
// returns null if it's not in the list
string? foundString = stringList.Find(s => s == "some string");
string? nullString = stringList.Find(s => s == "this is not in the list");

// If you want more than one, you can use .Where:
IEnumerable<string> multipleFound = stringList.Where(s => s.Contains("string"));
// As a list:
List<string> multipleFoundAsList = stringList.Where(s => s.Contains("string")).ToList();
// Returns an empty list if it finds nothing
List<string> noneFound = stringList.Where(s => s.Contains("nothing here")).ToList();

// There are also methods like select, which checks whether the item matches
// and then returns a bool based on the match.
IEnumerable<bool> test = stringList.Select(x => x == "first string");

// you can find by index like this:
string firstItem = stringList[0];
//string throwsArgumentOutOfRangeException = stringList[4];

// You cans earch for first or last directly as well
string anotherFirst = stringList.First();
// If you don't want it to throw an error for an empty list
string? lastItem = stringList.LastOrDefault();

// There are also several ways to go in the opposite direction:
int firstIndex = stringList.IndexOf(firstItem);
int anotherWayTofindIndex = stringList.FindIndex(s => s == firstItem);
// If you want to find the last occurrence in the list:
int lastIndex = stringList.LastIndexOf(firstItem);
int anotherWayTofindLastIndex = stringList.FindLastIndex(s => s == firstItem);

// There are also binary searches, but these should ONLY be used on sorted lists
int firstIndex = stringList.BinarySearch("first string");

You can sort lists with the Sort method and it's overloads.

Finally, there are options around copying lists:

List<string> stringList = new List<string>()
{
    "first string",
    "second string",
    "third string"
};

string[] stringArray = new string[3];

// Copy as an array
stringList.CopyTo(stringArray);

// copy as a list
List<string> newStringList = stringList.GetRange(0, stringArray.Length);

The general list is available via the System.Collections.Generic namespace.

Read-only List

A read-only list, is a list that cannot have items added to it after initialization. It's usually used when data has to be passed to a calling method, but you don't want the calling code to have access to change the list.

As in the read-only dictionary, a read-only list can be declared like the following:

IReadOnlyList<string> newList = new List<string>();

You can also create a read-only collection from a list like the following:

IReadOnlyCollection<string> readOnlyCollection = stringList.AsReadOnly();

Linked List

At the top level, a linked list works the same way as a generic list. However, it also contains additional methods to place items relative to a position in the list. As a result, if you need to retrieve or add items relative to an item in the list, they work better. Otherwise, it's better to use the normal List<T> class.

Examples of some additional methods in a linked list are:

// Adding
LinkedList<string> linkedStringList = new LinkedList<string>(new List<string>()
{
    "second item"
});

linkedStringList.AddFirst("first item");
linkedStringList.AddLast("last item");

LinkedListNode<string>? linkedListNode = linkedStringList.First;
linkedStringList.AddAfter(linkedListNode, "actually second item");
linkedStringList.AddBefore(linkedStringList.First, "actually first item");

// Removing
linkedStringList.RemoveFirst();
linkedStringList.RemoveLast();

Linked lists are also available via the System.Collections.Generic namespace.

Immutable List

An immutable list is a list that returns a new version of the list whenever an item is added or removed. This means that the immutable list is thread-safe provided you use lock functionality in C#. However, because the list creates an entirely new list every time items are added or removed, it can provide a performance hit over using a standard list.

Immutable lists are a little strange, in that you don't create them the same way as a normal list as they don't have a public constructor:

ImmutableList<string> immutableStringList = ImmutableList<string>.Empty;

As I said earlier, add and remove actually create new versions of the immutable list:

ImmutableList<string> newImmutableStringList = immutableStringList.Add("first");

Sorted List

A sorted list is a list that uses the IEqualityComparer to sort the list. It only supports key-value pairs and can be declared like the following:

SortedList<string, string> sortedStringList = new SortedList<string, string>();

In the same way that a Dictionary works, you cannot add duplicate keys to the list. However, where it differs is that it sorts lists based on the value. Here's an example of this in action:

Sorted lists also live in the System.Collections.Generic namespace.

Stacks and Queues

Really, there's enough variations of stacks and heaps that they require their own article. However, hey still are technically lists as they maintain the order of a collection, just in slightly different ways. It's best to say that stacks and heaps in C# closely implement the stack and heap data structures.

Stack

While it's outside the scope of this article, a stack is a first-in-last-out data structure.

If you're used to JavaScript, the syntax should be fairly familiar:

Stack<string> stackStringList = new Stack<string>();

// Add
stackStringList.Push("first");
stackStringList.Push("second");
stackStringList.Push("third");

// Check
string third = stackStringList.Peek();
stackStringList.TryPeek(out string? alsoThird);

// Remove
string thisIsthird = stackStringList.Pop();
stackStringList.TryPop(out string? second);

Queue

Queues are essentially the opposite of a stack, in that they are first-in-first-out.

You can use them like the following:

Queue<string> queueStringList = new Queue<string>();

queueStringList.Enqueue("first");
queueStringList.Enqueue("second");
queueStringList.Enqueue("third");

string first = queueStringList.Dequeue();

Both the default stack and queue live in the System.Collection.Generics namespace.

Array

An array is also an ordered list of objects. However, the big difference is that you need to declare how long an array is at initialization:

string[] arrayOfFiveEmptyString = new string[5];

string[] arrayOfThree = { "one", "two", "three" };

// changing the array to a length of five
Array.Resize(ref arrayOfThree, 5);

Arrays are older data structures in C#, and while they are absolutely used within C# (such as a byte array), it's usually preferable to use a list due to the increased flexibility as arrays throw errors if you try to assign values outside the array's length.

Arrays are fundamental to the operation of C# and as such live in the System namespace.

Array List

An array list is the forerunner to a modern list from the times before C# had generics. As such an array list accepts a list of objects, rather than a generic. This means that you do not benefit from type safety and as such, should be used with caution.

For example, the following code does not throw errors:

ArrayList arrayList = new ArrayList();

arrayList.Add("1");
arrayList.Add(1);

Array lists live within the System.Collections namespace

Tag List

Tag lists are a little strange, in that they are structs and like the sorted list, accept key-value pairs. However, essentially they're used when you need to optimize for memory usage as they avoid allocating memory when you have 8 or fewer items in the collection. They also accept objects in the value field, rather than generics, meaning that you don't have the advantage of type safety when using them.

TagList tagList = new TagList();
tagList.Add("tag", "1");
tagList.Add("tagTwo", 1);

var tag = tagList.First(t => (string?)t.Value == "1");
// throws an exception
var exceptionTag = tagList.First(t => (int?)t.Value == 1);

var tagTwo = tagList.Last(t => (int?)t.Value == 1);

Tag lists actually live within the System.Diagnostics namespace

A quick tour of dictionaries in C#

Jack Lewis — Fri, 21 Apr 2023 19:10:53 +0000

Preface

Dictionaries in C# are one of the more useful data structures you can use to store and retrieve data quickly and most programming languages have some form of dictionary built into them. However, there are a lot more versions of dictionary in C# and I haven't an article bringing them all together to discuss where you'd use one over the other, so hopefully this might help you out.

What's a dictionary?

Put simply, a dictionary is a list of key-value pairs. It's mainly used when you need to store large or complex objects within the code while also optimizing for quick retrieval of data. It does this by allowing you to pull out a key (which has to be unique) that is linked to the larger object (the value). However, while this was original intent, dictionaries are regularly used for 'simpler' retrieval such as pairs of strings as they are also useful guaranteeing uniqueness in a collection. Dictionaries are also (usually) part of the generic namespace in C# because the type of the key and value is declared at initialization.

Dictionary types

As I mentioned earlier there's a fair number of types of dictionary in C# so let's go through them now.

Dictionary

This is the "default" dictionary. It's pretty decent in most scenarios where you need one and is also the 'oldest' form of dictionary in C#. Given this is the default, I'm going to run through some common features of a dictionary within C#.

You can declare this dictionary like this:

Dictionary<int, string> myDictionary = new Dictionary<int, string>();

This form of dictionary allows you to add and remove entries at will:

myDictionary.Add(1, "stuff");

myDictionary.TryAdd(1, "stuff");

// You only remove via the key
myDictionary.Remove(1);

myDictionary.Clear();

you can find items in a dictionary in a few different ways (though I'd probably avoid linq):

myDictionary.ContainsKey(1);
string value = myDictionary[1];

// This is actually coming from the IReadOnlyDictionary interface
myDictionary.GetValueOrDefault(1);

myDictionary.TryGetValue(1, out string? value);

myDictionary.Where(m => m.Key == 1).First();

You probably noticed there's a few methods available are prefixed by Try. This is because the default of a dictionary is to throw an ArgumentException if the method does not return positively. As a result, C# provides some methods that allow you to safely retrieve values without having to resort to try/catch blocks which can be slow.

Interestingly, because dictionaries are generic, you can actually chain them together like this:

Dictionary<int, Dictionary<string, string>> myChainedDictionary = new Dictionary<int, Dictionary<string, string>>();

This is probably something I wouldn't recommend outside some very specific situations, but it's possible.

Finally, a dictionary is not thread-safe by default, but you can introduce this feature using locks.

Read-Only Dictionary

A read only dictionary works the same way under the hood as a normal dictionary only that values can only be added during initialization. What this means, is that there are no add and remove methods available.

There are actually 2 ways a read only dictionary can be declared, although one of them is via an interface:

IReadOnlyDictionary<int, string> myReadOnlyDictionary = new Dictionary<int, string>()
{
    { 1, "stuff" }
};

// This lives in System.Collections.ObjectModel
ReadOnlyDictionary<int, string> myReadonlyDictionary = new ReadOnlyDictionary<int, string>(myDictionary);

There is a small difference between these methods as the ReadOnlyDicitonary has a TryAdd method, but attempting to call it will cause a NotSupportedException to be thrown.

The main reason to have these methods available is when you want to make data in a dictionary available to callers, but don't want to allow a caller to modify the collection.

Immutable Dictionary

An immutable dictionary works in a very similar manner to the standard dictionary in C# but whenever you add or remove an item from the dictionary, it creates a copy of the dictionary, adds the item and then returns the new collection to you. This means that the underlying object stays the same and therefore an immutable dictionary is thread-safe compared to previous entries in this list. However, because new dictionaries are created every time an immutable dictionary is updated, there is a performance impact that isn't found on a standard dictionary.

This dictionary type also lives in the System.Collections.Immutable namespace.

Sorted Dictionary

The sorted dictionary is really quite straightforward, it will sort the dictionary based on the value of the key.

I've written a DotNetFiddle showing this in practice below:

Sorted dictionaries are really only useful when you need the data to be sorted (an example I've seen is using DateTime as a key and needing to retrieve anything between 2 dates). This is because a sorted dictionary is much slower than a standard dictionary due to the increased overhead sorting has when adding and fetching items. There are more details on this found here.

Concurrent Dictionary

There's a really good in-depth article found here around how a concurrent dictionary works, but essentially it's a dictionary that's designed to run safely across multiple threads (as described by the "concurrent" part). However, it works very differently from an immutable dictionary and does not return a new dictionary every timer a value is added. The concurrent dictionary was added fairly recently to C# and it's what I'd recommend if you need to access to a shared dictionary.

Frozen dictionaries do have a really useful method called AddOrUpdate which will attempt to add a value to the dictionary, if the key already exists, it will update the value. This method uses function within itself to choose whether to update the value, or keep it the same

Here's an example of this:

ConcurrentDictionary<int, string> concurrentDictionary = new ConcurrentDictionary<int, string>();

concurrentDictionary.AddOrUpdate(1, "value", (key, currentValue) => currentValue);
concurrentDictionary.AddOrUpdate(1, "someValue", (key, currentValue) => "valueTwo");
concurrentDictionary.AddOrUpdate(2, "someValue", (key, currentValue) => "valueTwo");
concurrentDictionary.AddOrUpdate(3, "value", (key, currentValue) => "valueTwo");
concurrentDictionary.AddOrUpdate(3, "someValue", (key, currentValue) => currentValue);

// this results in the following dictionary:
// key: 1, value: valueTwo
// key: 2, value: someValue
// key: 3, value: value

The concurrent dictionary lives in the System.Collections.Concurrent namespace

Frozen Dictionary

As of writing this article, frozen dictionaries are brand new to C#. They share a lot of similarities between a read-only dictionary, in that once a frozen dictionary has been initialized, new values cannot be added to them. However, there's a major difference between the two in that frozen dictionaries are heavily optimized for read speed as opposed to the read-only dictionary, which is closer to a wrapper on the normal dictionary class.

What this means, is that frozen dictionaries take longer to initialize than a regular dictionary, but retrieving values is much faster. There's a comment here on the GitHub repository for .Net showing that a frozen implementation of hash set was over twice as quick to get a value, as compared to a regular hash set. As a consequence, the use case for a frozen dictionary is for long-lived collections of data that doesn't change.

The frozen dictionary lives in the System.Collections.Frozen namespace.

Hash Table

OK, a hash set isn't technically a dictionary, but it performs a function similar enough that worth mentioning in this article as they are essentially, a collection of key-value pairs that require unique keys.

The first difference between a hash table and a dictionary is that they aren't generic, they instead accept objects. The difference is subtle when you're new to C#, but basically, generics are quicker and also signpost errors better than when you're using objects.

This means a hash table can be declared like this:

Hashtable myHashTable = new Hashtable();

Frankly, while hash tables exist in C#, I would be cautious of using them given how dangerous they can be around typing. For example:

myHashTable.Add(1, "stuff");

// this line will throw an error at runtime
// myHashTable.Add(1, "stuffTwo");

// this won't
myHashTable.Add("1", "stuffTwo");

// this results in the following hash table:
// key: 1, value: stuff
// key: "1", value: stuffTwo

As you can see, you can use multiple similar keys with a different type in a hash table, and most crucially, it doesn't complain when you get the type wrong. This can become a serious issue when you start objects throughout the code, which can be less obvious on if the types match.

The hash table can be found in the System.Collections namespace.

List of Tuples

This is most definitely not a dictionary, but it still stores 2 (or more) values in a list. This should be used when all values are of equal importance, and you actually want the duplicates in the list. To be absolutely clear, a tuple is not a key-value pair, it's better described as value-value pair and can in fact be several values (i.e.: value-value-value-value etc). A contrived example of using a list of tuples could be stock tracking system where all you care about is what's been sold and how much. In this case you'd want to know that you'd had 10 socks leave the warehouse on several occasions, rather than that somebody had removed 10 socks from the warehouse on at least 1 occasion that a dictionary would give you.

A list of tuples can also use named values which makes it easier to keep track of what the values mean, as compared to a dictionary which is always named key and value. For example:

List<(int number, string item)> myTupleList = new List<(int, string)>();

myTupleList.Add((10, "socks"));

int numberOfFirstItem = myTupleList[0].number;

I also find tuples useful when you want to return more than 1 object from a method, without having to resort to the out parameter in C#.

7 JavaScript libraries to help express yourself on DEV

Jack Lewis — Fri, 14 Apr 2023 18:01:22 +0000

I've written a few articles on DEV now and one of the interesting things you can do on DEV is to use liquid tags to embed websites into a post. There's quite a few of them that are related to programming sandboxes such as CodePen which you can use to embed programming directly into an article on DEV. As I've explained in a previous article anything which pushes you to write code is a great thing to help keep your skills up. As a result, I quite like the idea of embedding programming into my articles as it's a bit of a sneaky way to push me to write more code. Additionally, having a complete app that you can view and play with is really useful for understanding exactly how I did something.

I'm limiting this to JavaScript/TypeScript currently as this is the most common language used to write front ends and therefore has a lot of libraries written with the intention of displaying information visually. However, if you're learning a specific language, there's usually a liquid tag associated with. For example, C# has liquid tags support for DotNetFiddle.

Mermaid.js

Here's a codepen I've written for a previous article around estimation:

This isn't the prettiest example of the kind of output you can get from Mermaid, but it at least shows how you can build a flowchart out using a code based solution. Flowcharts aren't the only thing you can do with Mermaid and there's a full list in the link above, but you can also write UML like class, state and sequence diagrams and also pie and Gantt charts. While I've found writing these diagrams does take longer than using something like draw.io (which is also written in JavaScript) it does entirely do away with manual formatting of a diagram, which can be the hardest thing to do when writing a diagram.

Carbon

OK, this isn't a tool to directly write code and also is probably a worse option on dev as you can't copy and paste the code directly. However, for simply showing a beautiful screenshot of code quickly, this is an excellent tool. Carbon supports a wide variety of languages, can directly pull out an image from Unsplash to use as the background and has direct integration editors, including vscode. Additionally, Carbon has tools to autoformat code, which is more difficult to do if you're directly writing code into the markdown.

Chart.js

Simple chart:

I think Chart.js is pretty self-explanatory. It's used to make charts. The above is a codepen based on the number of referrals I've received for one of my articles. Just in this simple use case, it's easier to see the relative values of all the different places I'm getting viewers from, which is much easier to understand than the list that the default view gives.

Charts.js is one of many charting libraries on GitHub, but it's currently the most popular with over 60k stars. It supports 8 different chart types, animations, stacking graphs and as it's JavaScript based, Charts.js can be used to generate charts from code.

Two.js

I want to point out, the above CodePen is made by the Two.js team, not me (and it's a lot more impressive than what I could do). This is a 2D drawing library supporting WebGL, canvas2d and SVG. Two.js can be used to make interactive code by binding to input devices via JavaScript. There are a lot of examples of this happening found here.

MathJax

This is another one that might not be that useful as Katex is supported through liquid tags. You can see an example of this same expression using the liquid tags here. MathJax is essentially a library to support writing mathematical expressions through code using LaTeX, MathML, and AsciiMath notation, meaning it's a bit more modular than Katex. Either of these tools will work though. Additionally, if you're using a CodePen you can set the formatting how you want, rather than relying on the default in DEV.

Simple motion canvas animation:

MotionCanvas is a way of describing animations via code for use in vide recording. Granted, the primary use case of MotionCanvas is for video editing software and outputs individual frames rather than a full video and can't be built in a codepen. However, it's not difficult to use FFmpeg to get a WebM. There's a full video on how powerful MotionCanvas can be here:

summary
If you want to generate a WebM using FFmpeg, the following command being run from the output directory to do this:

ffmpeg -framerate 60 -i %6d.png -c:v libvpx-vp9 -lossless 1 -pix_fmt yuva420p output.webm

Cytoscape.js

CodePen showing things on my desk:

Cytoscape is another graphing library, but specifically built around networks. It's pretty straightforward to show complex relationships between objects. For example, this impressive diagram showing railways in Tokyo that also integrates travelling salesmen problem-solving. Cytoscape is particularly useful as it helps to visualize objects in a vector database, which given it feels like everything is moving towards AI recently, it's probably nice to know. The above CodePen also integrates dagre to simplify layout.

Tips and tricks on how to keep motivated while writing code

Jack Lewis — Fri, 07 Apr 2023 19:25:35 +0000

Preface

I've been programming professionally for nearly a decade now (and casually for a lot longer than that!), and while I'm still learning new things every day, there are certain things I've learnt over the past 10 years that I keep finding have helped me keep writing code and learning new things. For me, software development is one of those careers where you need to keep learning as what was a great choice even last year, can quickly become a terrible choice today. As a result, anything which pushes me to do some work in my spare time is helpful to me personally and professionally. Finally, in my experience the best way to learn something is to use it as opposed to just reading about it, as it teaches you the pitfalls and idiosyncrasies that online guides can sometimes miss. I should note, that these are things that I've found help me, and this isn't an exhaustive list.

'Fun' Money

Really, the first few tips come down to equipment choice that I'll discuss later. However, the thing which I will always advocate is keep a small pot of money every month to the side that will purely go towards something technical. This doesn't need to be used up every month, or be a large amount of money, just something that is affordable to you every month. The reason for this is that if you get a new toy, you want to use it, and I usually see a marked increase in time spent after purchasing something like this. Related to this though, I'd avoid subscription services as these can quickly get out of hand. The sort of stuff I look at is some stuff on humble bundle, or a new interesting piece of hardware that involves me learning to use it. Smart home products are great for this as well.

Interesting Personal Project

This seems kind of obvious, but actually working on a project as opposed to a contrived coding challenge pushes me to work harder in my spare time. Reasons for this working are that you're working towards a concrete goal that can take multiple days to get to a final product, as opposed to coding challenges which usually have multiple defined end points that can make you want to stop after completing a certain number of them. Additionally, a personal project usually mixes up several techniques, packages and languages which means that a single project can help you to learn multiple technologies at once in a way that can be quicker than learning each of them individually as one of the harder tasks in software development is integration work.

By interesting, I mean pick something that you're excited to work on as opposed to purely wanting to learn that tech, or for commercial purposes. This is because in my opinion, the hardest part about not working on the code, but building the motivation to work on the project. It's no good having the coolest personal project you can think of, if you just aren't motivated to work on it. It doesn't matter that somebody else has already built the same thing, what matters is that it gets you excited to work on the project.

Keep them short

When choosing to work on a personal project, it's tempting to talk about an all singing all dancing project that will do everything better than anybody else has ever done before. I would avoid doing this, as it leads to pretty heavy demotivation when you realize how long it will take to actually write the project. Just take a look at the roots of some of today's largest technology companies, the first product is vastly different from the slick and complex websites and software we see from them today. For me, I would look at a project which will take anywhere from 2 weeks to 2 months to complete as a perfect project, and definitely less than 6 months.

Mix the known with the unknown

One of the major reasons for writing a personal project is to learn new things. However, as I said earlier the hardest part is keeping up motivation. By mixing the known with the unknown, you get a good mix of learning new things and the ability to switch context when you get frustrated with something not working, which gives you time to step back and reevaluate whether what you're trying to do is the best way to do it.

Time management

I've previously written an article about how to estimate. Some of the things I've discussed there are pretty relevant to just writing stuff in general. Specifically I mean timeboxing. This where you say to yourself "I have 3 hours now, in that time I have to get feature X to do Y". If you fail, no big deal but just by fixing in your head a concrete goal, I find it helps to actually complete the goal, as well as giving you a minor dopamine rush when you do what you said you would.

Be realistic

This is pretty related to keeping a project short, but you're (probably) a single developer working on a project, don't expect to be able to write a 100 000 line project by yourself without burning out.

Keyboard and Mouse

These tools are the primary way you interact with a computer, so having a satisfying way of interfacing with a computer pushes you to write more. As a result, I'd spend a fair portion of my budget on a really nice mechanical keyboard as the feel of typing and the sound makes me want to type more, meaning that I spend less time on taking breaks when I should be working and also lets me work longer as the ergonomics of a good keyboard make typing less tiring. Only thing I'd say though, is be mindful of if you have to work around other people as a loud keyboard can be extremely distracting. In terms of a mouse, it's exactly the same as I find it just uncomfortable to work with a small, cheap mouse as my hand would start to cramp up with extended use.

Music

This is heavily personal preference and I know there are a lot of people who like to work in silence, but I pretty much always have music running when I'm writing. The reason for this is I find it easy tune out, and it masks the far more annoying sounds I can hear nearby, such as road noise or somebody talking loudly into a phone. Compared to this, music is a vast improvement. I also use a smart speaker so that if I want to change songs because I don't like the current one, I can just shout at it, rather than having to stop what I'm doing and press a button to skip.

Good (maybe not great) Computer

This might be a slightly controversial take, but you don't need the latest and greatest hardware to code effectively, and it might actually be a good idea to choose something a bit more cost-efficient and spend the saved money on something else related to software development. However, while it might be a little redundant to say today, I would say that an SSD is mandatory just for the reduced boot time.

Second Monitor

Research has shown that using a second monitor can boost productivity by over 40% which I think is probably good enough to leave there. However, from personal experience writing code on one monitor, and having a webpage running on the second monitor makes it so much easier to reference what I'm doing, without constantly having to tab between windows.

Technical Writing

This is another "from personal experience" point, but having an audience reading about what I'm doing absolutely is pushing me to spend more of my time working on projects so that I have something interesting to push out. As a result, writing up what you've done can help make you complete a project, as well as allowing you to brush up on a very important skill that can be left by the wayside for being boring - documentation. Also, knowing that somebody might actually (god forbid) look at the code I'm writing makes me spend more time on writing the code well, because making silly mistakes is embarrassing.

Get Inspired

One of the things that I find helps me to keep up my personal work is to look the interesting stuff other people are doing. The best place I've found for this is the GitHub trending page. This helps as some of the most interesting project ideas I've had, is to look at something somebody else has made (or thought of), and then realize I could apply something similar to a totally different problem.

Work with someone else's project

By this I mean, actually use something somebody else has written. You can just look at any one of the awesome lists on GitHub and I can pretty much guarantee you will find an interesting project to test out. These can turn in to full on projects themselves which can teach you about concepts related to software development, but don't directly impact them. For example, I have a really long article describing how I built my home network. I did very little coding, but it taught me a ton about networking I didn't know before.

Relation to other interests

A great way to keep you interested in continuing to write code, is to relate it to other interests you have. This helps as you have increased domain knowledge over something you've never done before which can mean that you need to do less research and reduce the possibility of hitting a dead end that you didn't realize was there before you'd spent several hours working on the dead end.

Avoid Becoming Your Job

I really enjoy coding, and I've worked a lot on fascinating projects professionally, but there's just something about writing code for myself that's so much more interesting to me. This is most likely not because of the coding itself, but all the extra bits and pieces you have to deal with when writing a commercial product that are frankly, not that interesting.

Take Breaks

It can be tempting to spend every waking moment working on writing code when you get interested in what you're writing. In my opinion, this can be detrimental as it leads to burn out pretty quickly which means you stop work. Additionally, taking a break allows you to take a step back from the project, and check your assumptions are correct. One of the ways to avoid this, is to set regular breaks even when you don't want to stop. I've found the best way to force myself to do this, is to actually leave the house and go for walk or out for a meal or something.

Collaborate

If you have other people depending on the work you're doing, this can push you to spend time working on the project and give you more motivation to continue as there is somebody else relying on you're output. Working with other people also gives you the benefit of having a "rubber duck" you can bounce ideas off of and also gain confidence that what you're writing is actually the correct way of doing things when you agree. Additionally, when you work with other people, varying interests mean that some the sections of writing code that you find boring, will be picked up by other developers as being interesting to them and vice versa. For example, you might like working with object relational mappers and your friend enjoys building CI/CD pipelines. There are limits to this of course (I think you'd struggle to find many people interested in writing DTO's) but can still help.

Failure Happens

I think this is something very easy to say, but hard to put into practice. However, it still needs saying. I have entire folders of full of failed projects, and I think most developers would say something similar. What's important though, is that you pick yourself up after a failed project, coding challenge or feature and keep writing. This doesn't have to be immediately, taking a break after something fails is a good idea, and it's natural to feel unmotivated afterwards. Although, some of the things that can help afterwards are to use up the fun money (if you have any), work on something which isn't code based or go on a short holiday, and eventually you will find yourself interested in doing some coding again.

Most Productive Time

One of the things that I find helps, is to figure out when you're most productive and plan to work on projects around that time. For example, I'm a bit of a night owl and tend to find I'm most productive after dinner, so the majority of time, I tend to work on projects between 7pm and 10pm, which gives me enough time to "cool down" afterwards and not spend half the night thinking about what I could do next.

Final words

I know a lot of these tips are pretty straightforward, but hopefully one or two of these tips are new to you and help you keep up motivation and help you to keep writing code, which as I said earlier, is the best way to keep learning new things and stay up to date on the latest in software development.

Great programs to help out any software developer

Jack Lewis — Fri, 31 Mar 2023 16:25:52 +0000

Background

Whenever you decide to develop software for the first time it can be a little daunting to know where to get started on actually developing software. There is a dizzying array of tools and programs which can be used to develop pretty much anything, while also being told all you need is notepad. This is a list of some of the tools I use (and some alternatives) to help out when I'm developing software.

Caveats

It does need to be said before I write anything down that this list is highly subjective and if you have a tool which works great for you, then it's better than whatever I'm recommending. Additionally, I'm going to keep these tools Windows specific, as that tends to be what I used to develop against.

Software

Some of these choices are going to be extremely obvious, but they're still worth repeating as I'm trying to write this article in a beginner-friendly manner.

Git

Starting off with one of the most recognizable tools in software development, Git is used to help track changes to software and helps to avoid losing code. It was initially developed by Linus Torvalds for supporting Linux and has since grown into a massively popular tool used just about everywhere.

Git can be a little difficult to get started with as it's a command line tool, but documents around its use can be found here.

Git GUI tools

Keeping on the Git theme, Git itself can be confusing to learn when you're starting out and one of the ways I found to simplify learning Git was to use a GUI to start with. There's a lot of them, all of which can help speed up software development if you use them right. For myself, when I was starting out I used SourceTree, before moving on to GitHub Desktop, but I don't use either of these any more because a separate client tended to actually slow me down. Right now, I use TortoiseGit because of its integration directly into the context menu in File Explorer and also the specific Git integrations that are built into the various tools I use to write code.

Package managers

Windows does have the ability to download packages similarly to Linux and these tools are called package managers. There's 2 which tend to be most common in Windows WinGet and Chocolatey. There's a fair amount of history between these two package managers, but I'm currently using Chocolatey a lot more often. The point of these programs is to vastly simplify the installation process for common software and also quickly share and download large numbers of packages and simplify the update process. Just be aware that Chocolatey (usually) needs to be run from an elevated command shell to work correctly. A fair number of the programs I'm going to talk about today can be downloaded via Chocolatey.

If you really don't want to mess around with a terminal, but have a new machine you want to install some of the most common tools to, you can use Ninite to build out a combined package as well. However, this isn't really a package manager and more of a quick way to install software once using a GUI tool.

Password managers

As a developer, we have a lot of passwords and secrets that need to be remembered but only used very occasionally. It should come as no surprise that a password manager of some description is needed to remember these passwords for you. While most modern browser's come equipped with password manager features, they are heavily focussed on storing user passwords and nothing else and while you can force them to store more, they aren't built for it. If you're looking for an all-in-one management solution, then unless you can self-host something like VaultWarden or are okay dealing with the idiosyncrasies of something like the free version of NordPass, this is going to cost a monthly fee. The largest player in the space is probably LastPass. This has upsides (lots more devs working on supporting the app) and downsides (most targeted for hacking), so really if you want to go this route, you'll need to do some research.

Alternatively, if you don't mind missing out on some features (such as a built-in MFA authenticator), I use KeePassXC. I use this particular password manager as it's cross-platform, relatively easy to share passwords across computers (using Google Drive) and has a fairly comprehensive feature set. I did previously use KeePass, but stopped as that software is windows only meaning I couldn't use it on my phone.

Authenticator

If you're using passwords to authenticate, you should also be enabling MFA to improve your security. For this, you need an authenticator. This can be hardware, using something like a YubiKey, or software based using an authenticator app. As mentioned earlier, if you're using a tool like LastPass, there are some pretty great authenticators included. However, if you want a free version, there's the Microsoft Authenticator that works on iOS and Android. If you do choose to use the Microsoft Authenticator, make sure you turn on the cloud backup feature as this will save you a lot of pain if you lose or break your phone.

Text Editor

For me, a text editor is a pretty important part of my workflow. I'm always copying text around, or needing to quickly format something and text editors can do this faster than a full-blown IDE or code editor. When I'm evaluating a text editor, I'm looking for something that I can access via a file explorer context menu, and is really lightweight. From this, I've found 3 options that work in this role. The first is notepad, that's built into Windows. I'll use this when I just need to write some quick notes, but it's not what I'll reach for if there's anything more in-depth I need to do (such as formatting JSON and reading scripts). The next is vim, which I pretty much installed for evaluation when I was working on some Linux stuff. I rarely use it nowadays as it's fairly unintuitive, but will work in this role. The option I use the most however, is notepad++. The reason is that it's a lot more powerful than notepad and easier to use than vim.

IDE and Code Editors

There was a distinction between an IDE and a code editor in the past where an IDE was considered powerful, but slow, with features like debugging and linting tools. Whereas a code editor was less capable, but lightweight and designed to work with multiple languages. However, I do think the lines have blurred heavily from computers getting more capable, and the release of tools like NeoVim and Visual Studio Code that can do everything that an IDE can do and more traditional IDEs bringing in greater support for other languages, like Visual Studio working with Python. My advice on this, is that I use the free language specific tool if there is one, such as Visual Studio for C# and Eclipse for Java. I should note this is for personal development projects, in an organization I normally use a paid IDE, such as one of the many developed by JetBrains (like IntelliJ). If I don't have a specific tool, or it's JavaScript based, I'll use Visual Studio Code.

Browser

I know this sounds extremely obvious, but the choice of browser can actually be really important as (while this has gotten a lot better recently) different web browser's can render content differently (who else remembers the dreaded this website only runs on Internet Explorer). What this means is that if you want your website to look like what the vast majority of your user's experience it's a good idea to track something like the most popular browsers. For example, the w3schools website shows that Google Chrome has a market share of nearly 80% across 60 million visits.

This is one of the reasons I use Google Chrome as my primary browser, but I am hoping that Arc comes to Windows, which is also based on Chromium.

Log Monitor

There are two sides to this one, a centralized log management tool and a more general log monitoring tool that works like a modified text editor. These are both really useful for different things.

First up is a centralized log management tool such as Splunk or elastic. The reason for the existence of these tools is pretty straightforward: if you have 100000 instances, and a 0.1% occurrence on a per instance basis, there are 100 examples of that issue in production. While you might laugh about ever running 100000 instances of your software, this absolutely does happen in industry and there are many issues with a higher occurrence than 0.1%. One of the ways you can track this is with a centralized log management tool, which will be able to tell you how common an issue is. Additionally, these systems are useful for real-time log monitoring which can pinpoint very quickly when there is a major issue occurring. While I wouldn't say it's the best idea to try and have one of these tools ready to go on your desktop as there are loads of them, understanding one of them can give you a deeper insight on how to manage and collate log files. If you are still interested, I have a (short) tutorial on how to use OpenSearch found here that can also be used for centralized log management.

The second part to this, is a log monitoring tool. These are useful for analysis rather than monitoring. In my experience, once an error has been found, the most difficult errors to diagnose actually start throwing up signs in a log long before an actual error is shown, meaning that you need to hunt through a log file for some sort of error. While you can upload entire log files to a centralized log management tool, this can cost a lot of money based on the amount of data that you're putting into one. For example, I've personally witnessed an ElasticSearch cluster dealing with well over a terabyte of log files per week and this was several years ago. You can probably imagine how expensive that was just to store it in AWS! All this adds up to needing some form of tool to look through a log file. While you can just use a text editor like Notepad++, a dedicated log monitor can vastly speed up and simplify the diagnosis process. There are a few tools that can do this, such as BareTail (there's also a chocolatey package of the free version found here) or KLOGG, which is free and open source. The reason you would use a log monitor over a text editor is that they are optimized for opening massive log files quickly and provide a clear display of data, regex based highlighting tools (useful for support) and speed. They also work much better over a slow network than a text editor in my experience.

Terminal

By terminal, I don't mean specific software related to a scripting language, PowerShell or Command Prompt. I really mean one of the much newer combination terminal programs like the aptly named Terminal by Microsoft or Tabby. The benefit of these programs is that they allow you to run multiple terminals via a tabbed interface. They both are pretty similar but with some differences. For example, Terminal has access to the command palette found in vscode and Tabby has plugins that can be installed as well as a password vault built-in for secrets. Both have the ability to customize the terminal appearance, run shells as an administrator and have customized shortcuts to simplify terminal use.

Sysinternals

Probably less useful for a web developer, but these tools can be a lifesaver when trying to diagnose complex issues in a Windows environment. This is a collection of tools developed to help show what is happening under the hood. They can be downloaded here.

Really, there's far too many tools in this pack to go into any real detail, but can roughly be split into monitoring tools and helpers. Some of the highlights within the monitoring tools are process explorer (a more powerful version of task manager), syslog and process monitor (both used to monitor logs in Windows - very useful for diagnosing issues with services!). Essentially, if there's some information you would like to know about what's happening in Windows, there's probably a tool in Sysinternals that can help out. On the helper side, there are tools like Autorun (that provides a detailed interface into running things on startup), Autologon (simplifies the automatic logon process setup in Windows) and ZoomIt (A series of tools used to make performing presentations easier).

Screenshot and recording tool

While using the Windows default prntscr button will get you perfectly adequate screenshots (especially when using alt + prntscr) there's always a time when I want something a bit better. There's even one already built into Windows! You can access the snipping tool using Windows + shift + s which gives you the ability to select what part of a screen you want to screenshot.

However, the tool I personally use is ShareX. The benefits of this tool is everything that the snipping tool can do and the ability to quickly record videos for demonstrations. A video of how something works makes it so much easier for someone who is not so involved in the technical side to understand how something works and is easier for the developer as it reduces the need for time-consuming and stressful demonstrations. It's also useful for test teams as it instantly shows how to reproduce a bug without playing email bingo over how to get a bug to manifest.

Docker

I think Docker should really be self-explanatory. It sometimes feels like I can't go a single day without hearing Somebody talking about Docker. However, it's rightfully popular as it massively simplifies the process of running an application and offloads the complicated bit of managing a server. A full explanation of Docker is outside the scope of this article (and you can probably find one if you spend 5 minutes browsing DEV), but in short it works by running applications in a container, which provides separation from the underlying infrastructure and helps to guarantee that the application will start the same way every time because they contain only the things needed to run an application. There's also a strong focus on programmatically building working containers so that updating and deploying the application can be done from a button press, rather than a manual upgrade process.

Windows subsystem for Linux

You can't really talk about Docker without also mentioning the tool it uses to run Linux images on Windows.

The Windows Subsystem for Linux (usually shortened to wsl2) is a how you can get a Linux operating system running on top of Windows directly from the Windows store. It vastly simplifies the process if you need to work on something Linux, but don't have access to another machine. This is an alternative solution to using a full-blown virtual machine like VirtualBox and on top of being heavily integrated into Windows, is a lighter weight (and therefore quicker) option.

PDF Reader

While the vast majority of information found on a system is written on the internet, it's still pretty common to encounter a PDF file for documentation. This goes triple of you're working directly with a supplier to integrate some closed source hardware or software. At work, I do just use the in-built tools in chrome to view a PDF as it's easy, and I don't need to justify having it installed on my machine to a sysadmin. However, at home I use Foxit because it massively reduces the buffer time to load a PDF and has a lot of features in the free version that aren't replicated in Adobe Acrobat.

Ending notes

Once again, I do need to reiterate that this is all personal preference, and frankly there's a lot of tools out there that are better than what I use, but in general I've found that everything I've used has helped me personally to be more productive.

Building a dev.to analytics dashboard using OpenSearch

Jack Lewis — Fri, 24 Mar 2023 17:31:39 +0000

Background

So I was doing some looking through bits and pieces you can get access to within your DEV account and I noticed there was a rather interesting tab in my dashboard for analytics. While I'm not a data scientist by any stretch of the imagination, it still pretty cool to see how you're doing in DEV.to. So taking a look, there's some pretty cool features showing things like readers, new followers and reactions. For reference, this is what mine looks like as I'm writing this article:

You can also get similar data out of an individual article.

From looking at this I can see that I clearly did well at the beginning this month, and it's tapered off pretty quickly (maybe need to look in to writing something a bit more popular?). I can see as well, that I've got 2 spikes in my reader graph that are related to an article I wrote around my setup for posting to DEV.to and also how to convert your local git repository to another remote programmatically. Interestingly, my second popular article didn't have any corresponding engagement in DEV.to, which means that it was probably popular via another website, rather than directly from DEV.to. Looking at the traffic summary for that second article, I can see this is true and that it's most popular on Twitter for some reason?

Anyway, getting a bit off track here, but after the first few minutes of looking at the analytics you can get out, there really isn't that much information I can get out, and it's also pretty general i.e.: some of the things I'm interested in finding out are probably pretty niche. So I spent some time thinking of if there was a way I could get some more information out that I wanted to know. Turns out, there is!

Forem API

The first step to figuring out if there's anything I can do to get more information, is to find out if there are any data sources that are accessible to me as a user. Fortunately for me, DEV is built on an open source software platform known as Forem. Luckily for me, it seems like somebody has also wanted to be able to programmatically grab data out of DEV and there's already an API that I can use to grab data from. Being honest, if this wasn't here I likely would have stopped at this point, as while it would be possible to scrape data directly from the DEV, it would have been a pain that I didn't want to deal with so this API makes everything possible.

While the API itself is incredibly useful, while writing this article there is currently 2 versions of the API which contain different endpoints:

This basically means I'm going to have to do some extra work later as both version contain useful stuff and I currently can't use the v1 API for everything. However, as I'll talk about later I've got some ways to make this really easy to implement later.

OpenSearch

Now I know I've got some data I could use, I now need to find a platform that I can use to analyse the data coming from the Forem API. I did consider some other pieces of software, such as Google BigQuery (with looker studio) and ElasticSearch (with Kibana), I ultimately went with OpenSearch which is essentially a forked version of ElasticSearch maintained by AWS. The main reasons are that I could host it locally for free (unlike BigQuery). I do have some prior experience with both elastic (back when it was called ELK) and OpenSearch, but my work with OpenSearch was far more recent, so I decided to go with that.

OpenSearch also provides code libraries that allow you to directly interact with the OpenSearch database from code, which given I'm writing something new that's specifically supposed to interact with just OpenSearch and an API, makes it a much more straightforward to implement. This is instead of going down the more 'traditional' route of analysing log files (which is the 'L' in ELK).

OpenSearch really consists of 2 parts, the NoSQL database OpenSearch, and the data visualization tool known as OpenSearch Dashboard. As a NoSQL database, OpenSearch doesn't really have the concept of "tables" but uses indexes instead, which work pretty similarly. As with a fair number of other NoSQL databases, the schema for the data is figured out based on the data itself, rather than setup beforehand. This does mean you can get into a mess if your data doesn't use standard data formats. Fortunately, as Forem is using OpenAPI, this shouldn't be an issue.

Installation

OpenSearch has an AWS managed service (of course) but the way I use it is via docker. If you're using windows (like me) you can use Docker Desktop to run the containers. OpenSearch provides 3 different ways to run the docker container, these are running a single node via docker and a pair of docker-compose files. I decided not to run in single node mode as I also wanted to run OpenSearch dashboard, which both docker-compose files allow you to do automatically. In terms of the docker compose versions, there is a production version and a dev version, the difference being that the dev version has the security plugin disabled. Given I'm just running locally and the security plugin takes some extra setup, I just went with the dev docker-compose file.

The Docker compose itself, will create a pair of OpenSearch nodes in a cluster and a OpenSearch dashboard instance to view the data:

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # Disables security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node2 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligibile to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # Disables security plugin
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data # Creates volume called opensearch-data2 and mounts it to the container
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200","http://opensearch-node2:9200"]'
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

NOTE: As I said earlier, this is NOT a production setup, this is a dev setup compose file, so should not be used outside a dev environment as you need the extra security.

If this is the first time you've run a OpenSearch cluster in Docker Desktop you might notice that the containers crash out and complain about something like vm.max_map_count is less than 262144. This is because the cluster needs more resources than the Docker receives by default. While you can just write into the WSL subsystem to fix it each time, I like to set up the config file, so I don't get annoyed by having to do some command line setup after every restart. In order to fix this, you need to create a file in C:\Users\<user running docker> called .wslconfig. The full file path would be something like C:\Users\jack.lewis\.wslconfig, and then you then need to add the following to this file:

[wsl2]
memory=4GB      # up's the memory from 2GB to 4GB
processors=4    # up's the number of processors to 4
kernelCommandLine="sysctl.vm.max_map_count=262144"   # up's the max map count to the minimum required by OpenSearch

At this point, if you used the config I linked above, the containers should start and look like the following:

From this, what you need to know is that http://localhost:9200 is the address you need to use to push data into OpenSearch and http://localhost:5601 can be opened in the browser to look at OpenSearch dashboard.

If you just want to explore the things OpenSearch can do, at this point you can open up dashboard, press the home button on the left-hand side, press add sample data and it should give you a choice to add some sample data to play around with:

Finding data to analyse

Immediately, I was very happy to note that Forem provides an OpenAPI spec document for both versions of the API, which is going to make it really easy to explore.

All you need to do is download the spec, and then drag the file into Postman and you instantly have a quick way to explore all the endpoints in the Forem API:

NOTE: if you want to actually return data, you need to set an API key at the top level folder (where it says DEV API (beta) for V0 and Forem API V1 for V1) and it's under variables. You can get the DEV API key from here and going to the point where it talks about DEV Community API Keys.

I think I might have also had to change the V1 base URL from https://dev.to/api to https://dev.to.

I think I really need to say though that the most common work I do day to day is building and integrating with API's. From this I know how difficult it is to get an OpenAPI spec right, and while the one from Forem does have some minor issues, on the whole I've had to do less messing around with it than pretty anything else I've integrated previously - which is great!

At this point, I can start exploring for data that would be useful and when I'm looking to integrate something into OpenSearch, I'm looking for 2 things:

WHAT happened
WHEN it happened

These, to me are the most important thing as OpenSearch is a time based analytics dashboard (when it happened) and the most common thing I really do in it is count how often something occurs (what happened).

From this, I'm drawn to the user's published article's endpoint in V1 and the follower's endpoint in V0. There's a lot more than this which could be useful, but at this point I'm pretty much just building a proof of concept, so these were the 2 endpoints I found that were most immediately useful to me (and honestly, the easiest to try and pull some useful stuff out of).

Integration

At this point, I now need to glue everything together. To do this, I need a program which can pull down data from Forem and then push it into the OpenSearch cluster.

If you just want to take a look at the final solution, it lives here:

jlewis92 / ForemAnalyticsGatherer

Retrieves analytics for forem

ForemAnalyticsGatherer

Retrieves analytics for forem

The command line arguments supported are as follows:

 -a, --articleGatherer     (Default: true) Whether the article gatherer is enabled

  -f, --followerGatherer    (Default: true) Whether the follower gatherer is enabled

  -f, --GatherData          (Default: once per day) How often to gather data in TimeSpan format

  -k, --ApiKey              The API key used to connect to the Forem API

  -n, --NodeList            A list of opensearch nodes

  -b, --BasePath            (Default: https://dev.to) The base path of the Forem site

  --help                    Display this help screen.

  --version                 Display version information.

View on GitHub

Programming language choice

It really wasn't much of a choice in this case as C# is my main language and I do enjoy working with it. Additionally, while .Net 7 is out, after being burnt by .Net Core 3.0 announcing end-of-life a few days before it happened, I pretty much always work with the long term support version if I can help it, so I went with .Net 6.

Forem integration

While I could have written out a full API to talk to the Forem API, this would have taken a long time and I had an OpenAPI spec document, so I used OpenAPI Generator which can pretty much instantly generate a full API integration, just from the OpenAPI spec. While I'm using C# and .Net 6, this tool has a pretty large list of supported languages, so if you wanted to code something similar, you could absolutely use your language of choice, rather than C#.

While there's a lot of different ways to install OpenAPI Generator, the easiest is probably NPM, where you just need to use the command npm install @openapitools/openapi-generator-cli -g and you should be able to start using the generator.

For me, all I needed to do was run the generator twice, once for V0 and again for V1 using the following command:

 openapi-generator-cli generate -i <location of the OpenAPI file on disc> -g csharp-netcore -o <output location> --additional-properties=targetFramework=net6.0,apiName=ForemVersion<Zero or One>,packageName=ForemVersion<Zero or One>

breaking this down:

 openapi-generator-cli generate # generate an API
 -i # The location of the Forem API OpenAPI yaml
 -g csharp-netcore # The name of the generator - .net 5+ lives in this one as well
 -o # where you want to save the output
  --additional-properties= # properties that are generator specific
  targetFramework=net6.0, # This where I set .Net 6
  apiName=ForemVersion<Zero or One> # This is the name of the default class generated by the API - it would normally be DefaultAPI
  ,packageName=ForemVersion<Zero or One> #  This is the name of the C# project that will be generated

After you run this command, go to the output directory, then into src and you should see 2 C# projects:

As you can see, it generates 2 C# projects that you can import into a C# solution. Rather helpfully, the generator also generates a load of tests for the generated code. All I needed to do at this point, was import the generated projects into my own via the solution explorer in Visual Studio and I had a ready built integration into Forem:

For those who use C#, the generator uses the RestSharp library by default, as opposed to HttpClient, but it's possible to change this if you really want to.

There were some bits and pieces I did need to update in the integration though, as building code this way will do exactly what swagger definition tells it to do, for example, if a value on a Request or Response is set as Required and you either don't set the value, or the value comes back as null, the API will throw an error. When I was doing this, there were a few values in the Articles Response that are marked as required, but I didn't get them back, so did need to modify the generated code very slightly to remove the IsRequired attribute from a few Articles model.

OpenSearch integration

Note: The next few bits are heavily C# based, so if you're not that interested in C# you can probably skip to the point where I start talking about OpenSearch again.

While ElasticSearch was originally built to ingest log files, you can also use a package to directly integrate OpenSearch into the code itself, which is what I did with this project. If you're interested in trying this out, you can find a list of supported clients here.

Integrating OpenSearch into an object-oriented language is extremely simple because all you need to do is pass the objects you want to index into OpenSearch, and then the rest is pretty much handled for you. For reference, here is pretty much all the code I use to push articles into OpenSearch (after setup):

/// <summary>
/// Indexes articles into OpenSearch
/// </summary>
/// <param name="articles">The articles to index</param>
/// <returns>A response based on whether the upload succeeded</returns>
public async Task<BulkResponse> IndexArticleData(List<ForemVersionOne.Model.ArticleIndex> articles)
{
    var response = await _openSearchClient.IndexManyAsync(articles, "articles");

    return response;
}

While I could spend some time throwing some additional error handling etc. This code is proof of concept, so I'm not too bothered. It should also be noted, that the object I'm passing in to this method is taken from the generated API code discussed above.

Breaking down what I'm doing in OpenSearch is as follows:

await // I'm using async methods to push the data
_openSearchClient // calling an OpenSearch client I setup in this class
.IndexManyAsync // I'm using index many, inbstead of index as I just want to push in a load of articles, then forget about it
(articles, // The list of articles I want to push into OpenSearch
"articles"); // The index in OpenSearch I'm pushing the data to

Re-indexing articles that I've previously indexed does not cause a new version of the article to be indexed, but does update the old article. While I've not dug into the OpenSearch code, I'm assuming this means there's something which is tracking identifiers attached to the data being pushed in (like this endpoint does have).

Pulling everything together

Now I've got code that can handle both pulling data down from Forem and then taking that data and pushing it into OpenSearch, I now just need to pull everything together and provide an interface that is easy to use.

For pulling everything together, it's not that special, I'm just using a standard C# library project that takes in an AppSettings object for settings. I did decide that I wanted the ability to toggle the collection of data from each endpoint, so I did split out the code along these lines. Also, given the data is paginated, I do loop through until I can get all the data for use. I understand this is not the most "efficient" method of doing this as I'm retrieving data I've previously indexed, but that might be something I look at in the future. If you're interested, the code for how this looks, the article's endpoint is here.

Settings

In terms of access, I decided the easiest thing to do was to create a console app project that can link into the analytics library easily and be very flexible. For example, as it's just an executable it can be run directly, via a service or via scheduled task with minimal setup. This is all facilitated via a timer event being fired off after a set amount of time (default to once a day) so that you don't need to keep running the data collection task. I also added the ability to run as a Linux docker container because Visual Studio makes this only a couple of button presses to add:

In order to work, the application does need some settings passed in, most notably there is an AppSettings file built into the project. This controls settings for the ForemGatherer library itself and by default looks like this:

{
  "AppSettings": {
    "BasePath": "https://dev.to",
    "NodeList": [
      "http://localhost:9200/"
    ],
    "ApiKey": "" // While the API key can be set in this file, I don't reccomend it
  }
}

The project is set up to pull settings into this file via several methods:

directly into the file - support is also there for using NETCORE_ENVIRONMENT i.e.: appsettings.dev.json
environment variables
user secrets
command line arguments
- These override all other methods

The command line arguments supported are as follows:

  -a, --articleGatherer     (Default: true) Whether the article gatherer is enabled

  -f, --followerGatherer    (Default: true) Whether the follower gatherer is enabled

  -f, --GatherData          (Default: once per day) How often to gather data in TimeSpan format

  -k, --ApiKey              The API key used to connect to the Forem API

  -n, --NodeList            A list of opensearch nodes

  -b, --BasePath            (Default: https://dev.to) The base path of the Forem site

  --help                    Display this help screen.

  --version                 Display version information.

As I said, I can also run this as a Docker container, and I've got some vague ideas to link it into a docker compose so that I can run everything together:

All you need to do is run the code and verify that you're getting some data in OpenSearch. You can do this by going to the following address in a web browser http://localhost:9200/_cat/indices?v and you should see something like the following:

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .opensearch-observability       fnli1mGtSJCzRUsMXXQJYg   1   1          0            0       416b           208b
green  open   followers                       2bSUsOgGSBOi3rJyuq3NPw   1   1         52            0     73.6kb         36.8kb
green  open   .opendistro-reports-definitions SqwXfm_BRMmyTk305ktGaQ   1   1          0            0       416b           208b
green  open   articles                        2LNsx-bOS6GizJvoyRAfLQ   1   1          8            0    112.8kb         56.4kb
green  open   .opendistro-reports-instances   40Wf6ofGStSqqJW7Z3-jew   1   1          0            0       416b           208b
green  open   .kibana_1                       bc9P7hL8SJ68jDutyPajFg   1   1         23            9     95.4kb         47.7kb

OpenSearch Dashboards

Once in OpenSearch, I need to add the data to the dashboard from OpenSearch. To do this, open the hamburger menu to the left and go to Management > Stack Management > Index Patterns and this should give you the following page:

Index Patterns are used to tell OpenSearch Dashboard what data you're going to use. These can incorporate as many or as few indexes as you want, but given you also need to set a time field, which is different between Followers and Articles, which meant I just created 2 index patterns. For articles, you do get a few choices on the next screen, which is setting a time field, but I'm most interested in published articles, so that's what I chose. This has the knock on impact of essentially removing my draft articles, which is also what I wanted to do.

Now I've got data into the OpenSearch Dashboard, I need to analyse the data. To start with, I used the Discover tool to see if there was interesting, which I did find pretty quickly, such as DEV using Cloudinary for image storage and that I've been fairly consistent in releasing articles:

One of the nicest things about OpenSearch is that I can set how long the data range I want to view is:

This is pretty helpful, as it lets me set the exact length of time I'm interested in, as opposed to the stats page, where the only option after monthly is infinite which is pretty difficult to see what's going on, each day as I've now been on the sight (slightly) more than a month.

I'm not going to go through how I made every individual visualization, but if you're interested I've dropped a copy of the dashboard I built in the repository for this project. Here are some pictures that I've taken of the dashboard:

It definitely bears repeating, I'm not a data scientist, so I'm sure you could figure out some better things to look at (as well as not constantly changing case in the names of the visualizations). I do think this does give a pretty good idea of the things I can gather for helping me see how I'm doing on DEV.

Next steps

It's pretty clear the API is still heavily under development by the Forem team, evidenced by the "Tags" endpoint disappearing while I was writing this article (to be fair it wasn't THAT useful anyway) and that there's a fairly new pull request for removing V0 endpoints I think it's likely that access to the OpenAPI docs (and followers and articles) will stay around. I'm thinking I could pull out some more data based around the endpoints I do have, but I do need to do some thinking about. Also, given the API is under active development, and there's a fair amount of data DEV has access to in the analytics console that I don't have access to via the API, it would be nice to extend my dashboard to include this if it does get updated.

A developer's guide to estimation

Jack Lewis — Fri, 17 Mar 2023 16:48:15 +0000

Background

I should probably start by saying I really dislike estimating. Most of the time it's a bit like licking your finger and sticking it into the air and the everywhere else it's even worse! Between having to (usually) quantify numerous unknowns that can slow you down, you have to try and plan in whether the co-worker helping you out isn't going to (rightly) go on sick leave just before that major dependency comes in. When you finally do give an estimate, to the people you are giving the estimate to, it's not an estimate, it's a guarantee and when it doesn't happen for a multitude of reasons, you get a pretty uncomfortable meeting where you have to try and explain why you didn't do what you said you would. This feeling is so entrenched into software development that there's a pretty common 'joke' about it:

person: Hey we've got this really cool feature we need adding! All you need to do is add a button to the form
developer: 3 weeks.
person: but it's just adding a button?
developer: ah sure, it'll take 4 weeks.

However, none of this really matters, because the business needs this information to try and forecast future work, and if this doesn't happen, the company can go out of business. So as much as I dislike doing estimates, it's still a key skill for any developer to have.

How to estimate

So we've established that we all need to know how to estimate, but there's a slight problem in that how does a developer actually estimate. I'm sure most of us with a little experience have just thrown a number out and been reasonably accurate in that prediction without really thinking about it. This is of course, utterly unhelpful when you're trying to learn how to estimate, and you might read articles such as this one from Time Camp which go through a fair amount of the individual techniques you can use to estimate. However, in my opinion, the real answer to this we use a mixture of techniques that can give you an estimate in under 30 seconds (really 10) - any longer, and you're probably overthinking it, or the task is too large, and you need to break it down. Having said that, here is how I personally estimate tasks.

Estimation can really be broken in to three parts for me. Getting an actual estimate, converting how long I think it will take into a realistic estimate and then finally, how to convey that estimate to the business. I've broken this down below.

Getting the estimate

When I think about how long something is going to take, I'm essentially running through questions like in the below flowchart to try and draw together an accurate estimate:

NOTE: The above isn't everything, just representative of the questions you should be asking yourself before you give an estimate. For example, if the code I'm writing is using an interesting technology, I'd probably slightly reduce the time, just because I'd put extra work into it.

Converting to a real estimate

There's a few methodologies you can use to do this, but I'm going to discuss 2 of the more common ones.

The three point estimate

Frankly, I don't use this that often any more, but when I was starting out and had no idea what to do, doing a three point estimate helps you frame the problem in an easy to digest way.

NOTE: Three point estimation is the endpoint of estimation i.e.: once you've gone through the other steps in trying to figure out how long the task will take

While there are several ways to get to a three point estimation, we're most interested in the quickest way of getting there, so we're going to use the simplest.

Essentially, you ask yourself these 3 questions:

How long will it take if it's simpler than I expected
How long will it take if everything goes wrong
How long is it going to take realistically, based on what I currently know

You then divide the 3 estimates by 3 and then dived them by 3. For example: I get a task to add a button to form and based on what I know, optimistically it will take me an hour to add, realistically it will take 4 hours, and pessimistically it'll take me 2 days.

This gives me the following:

\over 3} = 7

Now as this is close to 8 hours, I'd then just round up to a full 8 hour day as that's close enough.

Time plus

This is a really simple way of estimating, and in my experience it's the most popular way of doing this activity. I really wouldn't recommend this for a beginner as it's easy to get wrong, but essentially you take how long you think it's going to take, add 20% and then round up to the nearest of 4 hours, 1 day, 1.5 days, 2 days, 3 days, 4 days and 1 week. Any more and the task should probably be split apart.

i.e.:

\times 1.2 = 9.6

I then take the 9.6 estimate and then round up to 1.5 days.

This is (usually) far less accurate than a three point estimate, but can get you there most of the time. In my opinion, despite simpler calculation, this is actually more complicated to do as it relies on already having a full understanding of the solution before you can give an accurate estimate and as I said earlier, better not to rely on if you're new to estimating.

Conveying the estimate

Conveying can really be broken down into 2 parts. What to do when you're giving an estimate, and what to do while you're working on the task.

In the meeting

When you're actually in the meeting, and getting asked for an estimate here are some tips:

Take a few seconds to consider before answering
- Even if you already know what you're going to be asked to do, it always worth spending a few seconds considering the answer in your head as this is the last chance before the estimate is fixed in people's minds. On multiple occasions has helped me to remember that one little thing I forgot to consider, allowing me to make a more accurate estimate.
Be honest
- I know that occasionally, you will get pressured to give shorter estimate than you really expect it to take. If this is happening to you, trust me it's going to be far less awkward to explain now, than when you have to tell the same people pressuring you it's going to take longer than you said
Be honest with yourself
- If you're throwing out estimates that you think may be possible, but have historically taken you a lot longer (as well as constantly having to explain to the project manager why it's taking longer), it's probably a good idea to take a look at you're estimation process.
Don't panic
- If you're new to the process of giving estimates, and the other people around you know that if you get it wrong it's not such a big deal, provided you're getting closer every time. We've all been where you, are, and it's not an easy thing to do when you're just getting started
Explain you're reasoning
- What I mean by this is, is that just give a short sentence showing you understand the task and set expectations around difficulties you may have. For example, if I know that I've got dependencies on another team, I'd probably say something like "I can get this done by end of day tomorrow, provided I get dependency X in the next couple of hours".

After the estimate

Once you've given your estimate, there are some things you can do to help try and get the task done on time and also mitigate issues if you do get it wrong:

Timeboxing
- There are a lot of guides on how to this (such as this one by Clockify), but in a nutshell it's the practice of allocating a set time in your day when you try and complete the task.
Communicate
- If you've said a task is going to take a week, and every day you've said everything's going well up until the final day when you suddenly say that you're going to need another week, you are pretty much asking for angry questions to be asked. It's always better to make sure the project manager understands how you're doing and in my mind, is a major reason for the daily stand-up. If you're team isn't doing a daily stand-up for some reason, then you probably have access to an instant messaging tool, and you can use that instead.
Don't worry if you get it wrong
- We've all been in this position at some point, the best thing to do is to not get too worried over it, and let somebody know that you might need some help and why you're having issues. What will get you in trouble is not the underestimation, it's when you try to hide it.

The AWS Cloud Resume Challenge

Jack Lewis — Fri, 10 Mar 2023 16:15:38 +0000

NOTE: I did write this article a month ago, but waited to release it to see how much it would cost - turns out it was about 60p

Background

Back in January I'd made a promise to myself to spend at least a little bit of time doing something technical every week, whether that be writing software, building out hardware or just generally reading something to try and keep my skills up. To start with I was going to try and complete code kata's in a few languages I wasn't too familiar with, such as python. However, knowing myself I do just find writing code with no reason other than to write code to be not that interesting to me, so I was looking for something a bit different. As a result I'd been looking around the internet for something technical to do and came across the AWS purity test which I did as a bit of fun as I work with AWS pretty heavily in my day job. One of the questions on the test was talking about completion of the cloud resume challenge and while I'm not in the market for a new job, it sounded like a really interesting project to tie all my skills together and I thought might as well try it. 3 weeks later I've got the following:

Impressive right? No not really, but to be fair it's a resume so it's supposed to be simple and most of the interesting stuff in this project is pretty much all behind the scenes.

If you're interested in viewing the website itself it's here

Slightly more impressive, I think is this diagram of the end product:

What's the challenge?

Put simply, it's a series of 16 steps to build and host a static website in your cloud of choice, then pull some data in from a database. You also need set up a fully automated deployment of both the backend and frontend using infrastructure as code. Once you've done all this, you then need to post a write-up about it somewhere online with tips and tricks and your experiences of doing the challenge. Additionally, there's a discord you can use if you have questions, but I intentionally avoided this as I wanted to see if I could do this on my own. If you're thinking of doing the challenge, it might be best to stop here if you want to avoid some spoilers, otherwise let's go through it!

Final outcome

Thought I'd put this at the top so it doesn't get lost, but there are essentially 4 outcomes for this project:

Timeline

This project was completed on a vague timescale of the below. I wasn't really working on this full time at the start, just a few hours a week but I did a fair amount the last week of this project

week 1: HTML/CSS and cloud front hosting
week 2: JavaScript, database, API and python
week 3: everything else

Dev environment

I pretty much used vscode to write everything as I've used it pretty heavily previously. In terms of some of the extensions I'm using they are:

Live Server
- Allows you to quickly run and host a simple HTML website and reloads on changes automatically

Certification

The guide recommends you have a cloud practitioner's certification before starting the challenge. I don't personally have one of these, mainly because I've been working with AWS in industry anyway so most of the things it's talking about I have a vague idea on anyway. However, if you're coming into this with no experience whatsoever it's probably a decent way of understanding how AWS works without fumbling blindly

HTML

It should be said that the frontend wasn't really the focus of this project so there's not really much to talk about here, I just wrote a simple template in HTML which I could use to put data in. Given this was supposed to be simple, I'm just directly writing text into the file but if I was actually doing this in a production environment that would definitely not be the case.

CSS

It has been a fair few years since I've written purely in HTML/CSS without something like bootstrap or tailwind to make life easier for me, but I think I did an ok job, some of the slightly more interesting things I did were using media queries to set styling based on screen width so it would be a bit more useful for mobiles:

 @media only screen and (min-width: 1000px) {
    .resume-body-skills-wrapper {
        column-width: 250px;
        resize: none;
    }
 }

This then does the following with the skills section on the resume:

One of the interesting thing's I did find was that you can use css to do stuff when certain tags are sat next to each other, like the following:

span+span {
    margin-left: 50px;
    /* adds in 50 px spacing when there's a double span tag */
}

Both

Generally, I've worked much more heavily WPF when building frontend's which uses XAML. I think as an observation between the 2, I've found that it's probably easier to place your content where you want it to go with XAML (I tend to not find my self googling how to centre content anyway!) but actually styling elements within XAML is much harder.

AWS setup

If you've never setup an AWS account before, it's pretty straight forward, just go here and give AWS your credit card number. While outside the scope of this article, it's pretty much mandatory for me that you complete all of the security recommendations in IAM before starting as well as enabling MFA everywhere. You should also download the AWS CLI and setup credentials using aws configure:

$ aws configure
AWS Access Key ID [None]: key ID
AWS Secret Access Key [None]: access key
Default region name [None]: region
Default output format [None]: json or yaml

If you've also setup MFA everywhere you might also need to use the sts get-session-token CLI function as well. AWS recommends you use the toolkit instead but I've tended to find it didn't work very well when I was using it for work last year. You should also setup billing alerts on your account, because unexpected bills are terrible

Static Website

This is really easy to do, you just go to S3 give it a name and ignore all the warnings it gives you when you untick block all public access:

At this point, all you need to do is upload your website files, then go to properties -> static website permissions and tell it what the index document of your website is. At this point, all you need to do is select the link S3 gives you and you should see your website on a random http url. At this point you're done! Well, not really, if all you wanted to do was host your resume, that's it unfortunately we're not even halfway through if you want to complete the actual challenge. S3 pricing is pretty negligible for this project as I'm storing under 1MB and AWS charges about 2 cents per GB per month

DNS

Technically, HTTPS comes before this step, but CloudFront offers a lot of automation if you have this up beforehand. The guide recommends using using Route 53 but I'm not a massive fan as the interface is pretty old school, and even though I've set my account into GBP, it's still showing me prices in dollars. Finally, it also doesn't have the .dev top level domain that I want, possibly because Google operates it:

I specifically wanted .dev as it matches what I do and also comes with added bonus of forcing HTTPS, meaning if I want this to work it needs to have certificates setup properly. For this domain to buy it off Google Domains and set it to auto renew

Only real change to this is that I had to use google domains and then transfer it to Route 53. This is the point where you're going to incur a small monthly fee as pricing for Route 53 charges 50p per month per hosted zone, then a further 40p for every 1 million requests.

HTTPS

Really, this section should be called the CloudFront section of the guide as it's basically what you're going to do here. I wasn't a massive fan of this section as pretty much the website link that the AWS resume challenge for this basically does everything for you and when I'm trying to learn, I'd much prefer this to be a bit more separated out.

CloudFormation

To start with , you're pretty much going to go to the link in the AWS resume challenge and then choose stack 3 and filling out the questions it asks you. This wasn't a great experience for figuring out how it works as the CloudFormation template is pretty obtuse, given it calls out to other scripts within it, and the CloudFormation designer UI is pretty cluttered and difficult to see what's going on:

As you can see, the entire section of the template showing how the ACM stuff works is just being pulled in from another template. Also, if you drag the designer around, your location in the template gets reset to the top which is really annoying. However, I did like that the template made it super easy to set the subdomain to "resume" instead of "www" as I'm planning to host a more modern website on that subdomain at some point in the future.

CloudFront

Once you've gotten you're template done and waited the 20 minutes for everything to get setup for you, all you need to do is throw your website into the bucket the template generated and watch as your site is served! Well, with a caveat - because CloudFront is using edge locations to serve your content as close to the geographical location of the caller as it can, it basically has to distribute these to the servers around the world and cache them. As a result, it can take up to 2 days for content to be displayed throughout the world. Once you've realised this, you can take advantage of CloudFront invalidation requests to force the cache to refresh. The pricing for this is that you get 1000 free invalidation requests per month, and if you want to refresh the entire website you can use /* or for individual files you can use /<file name>.<file name type> and list the specific ones you want. You also need to pay for edge computing, but currently my bill is sat at £0.

As a sidenote, the CloudFormation template does generate logs into an S3 bucket without rotation. For now I've left them on as it's pretty small amounts of data, but I've still set a lifecycle retention policy on the logs folder just in case. You can also set alerts around stuff like number alerts over a time period or data the rate for upload and download among other things. You can also get some metrics around the website, such as usage:

Or viewers:

NOTE: one of the big errors I had around the CloudFormation template was around how it basically disables JavaScript from running by setting the Content Security Policy to deny everything in the AWS Role that it generates, as I'd missed this when I first went through it, it caused me a lot issues later on.

JavaScript

This part of the challenge is to create a visitor counter that's updated via JavaScript

I think this is the first time I've worked fully with vanilla JavaScript before as I normally use TypeScript or some sort of framework. However, there's really not much to speak about on it as it's less than 10 lines long. All it's doing is getting some text from an API and then setting a visitor counter on the footer to that value. I can say I much prefer using frameworks which let me set values using {braces} rather than having to use the document query selector.

DynamoDb

I've used dynamo pretty heavily before so I'm pretty much using this to brush up on this again. For myself, while you can write tables straight into AWS, I find it's much easier to visualise and work on tables using an offline tool called NoSql Workbench. If you want to use DynamoDb offline is to just set the toggle for DDB Local server and publish a table into a local connection and it's now setup for you:

Once this is setup, all you need to do in order to use DynamoDb local is attach an endpoint url when you create a client for DynamoDb. If you haven't changed the default port for DynamoDb, it will look something like this in Python - endpoint_url="http://localhost:8000".

However, the best thing about NoSql Workbench is it's ability to do your work for you by generating code directly using the operation builder:

The expression above basically increments the VisitorCount column by 1 and if I hit the generate code button, it basically gives me the code to do this in Python, JavaScript or Java.

I quite enjoy working with DynamoDb and always have done as I find it's really quick to get going in development. There is a fairly large learning curve if you're coming SQL databases as the single table design and replication of data can be a little mind-bending. There are some pitfalls to be aware of, for example retrieving ALL the data out of a table is considered a bad idea because of how much it can cost.

Lambda

This section comes after the API in the cloud resume challenge document, but I find it's usually easier to write the code that retrieves data from the database, before you build out the API calls. In this case, I'm writing an AWS lambda expression in Python. I'm using Python as I've previously written lambda expressions in C# and I thought the best thing to do for this point was to work in a language I've only had a little experience of.

I did spend more time than I was expecting on this section of the project though...

I'm pretty happy with Python as a language as it's easy to see what's going and at least in small scripts the lack of compilation speeds up development. However, I think I do still prefer C# to write Lambda expressions in and just general development. This is almost entirely due to not the language itself and more the tooling surrounding the language in Windows (I'm aware this is different in Linux) as I'm feeling like I'm constantly fighting it during initial setup. For example, while sure nowadays if you type Python into PowerShell, it just opens up a Windows store window which allows you to get hold of Pip and Python quickly, what it doesn't do, is setup the scripts folder on your PATH, which led to issues when I was using the Python lambda local library to test the AWS lambda expressions. Then of course there's messing around with venvs in vscode.

However, I did manage to eventually get everything up and running, using some slightly modified code from NoSql workbench which would allow it to return the data in the same call to update the database. I think the code is pretty good as it runs in under a second on AWS Lambda. Given lambda pricing gives you over 800 hours free per month at 128Mb, I think this is fine.

Here's the full request for updating and returning a value for the visitor counter in DynamoDb:

def create_update_item_input(table_name: str):
    return {
        "TableName": table_name,
        "Key": {
            "PK": {"S":"resumeProject"}
        },
        "UpdateExpression": "SET #visitorCount = #visitorCount + :increment",
        "ExpressionAttributeNames": {"#visitorCount":"VisitorCount"},
        "ExpressionAttributeValues": {":increment": {"N":"1"}},
        "ReturnValues":"UPDATED_NEW"
    }

If you want, here's a direct link to the full source code

Tests

Once I got this all done I moved onto writing some tests. I just decided to keep simple and use the unittest library that's built into Python. I then found that there's a pretty great mocking library for the AWS boto3 client called moto that can be used to write proper tests for Python. Compared to C# this is great, as my only option there appears to be writing wrapper classes that call the function. To be honest though, this is where I spent a lot of my time as I ended encountering a bug in moto that I ended up making an issue on GitHub about. Fortunately, the people running the moto repository responded quickly and it's been fixed - all I need to do now is update my tests to work with the fixes!

API

The API is built using API gateway that has a lambda proxy integration to the lambda expression I'd built in the previous step and returns data to the client like this:

I decided to just return a string because it was easy that way. If I was to extend this and have more than a single value returned to the client I'd change this to JSON as it's much nicer handing data between server and client. However, this was such a small part of the project I just decided not to do it. I understand it's not a great idea though as my website just returns whatever the result of the fetch is, I'm just lazy and had spent far too much time on this already. I should also say I'm a bit iffy about leaving a lambda gateway open as I usually stick them behind a Cognito pool or oauth authoriser whenever I've used an API from API gateway in the past.

Just as a sidenote however, if you're doing a lambda proxy in the gateway, the response returned by the lambda must be in the format as follows:

{
    "statusCode": "200",
    "body": "body",
    "headers": "headers"
}

Once I got this all hooked up, I just updated the JavaScript to pull in from the backend and I can now see my visitor counter is updating:

Although no it wasn't initially, as I needed add CORS headers to the request to get it to display content. Technically, I could have just hosted it on an endpoint within jacklewis.dev, but I wanted to try adding headers directly instead. I'll talk about how I did that in the next step.

Infrastructure as Code

For this I used Terraform for the backend as I've used it a little bit before. It's a pretty great tool for describing resources in AWS and has the benefit of being cross-platform as opposed to the AWS SAM model first pointed out the the cloud resume challenge website. I should probably note as well, that most of the stuff I did build directly via the console to check my understanding, before translating it into Terraform. I like to style Terraform by having not much in the main.tf file and then linking each file to it's purpose. As a result the structure for the Terraform ended up like this:

├── api-gateway.tf
├── dynamo.tf
├── github-runner.tf
├── iam.tf
├── lambda.tf
├── main.tf
└── provider.tf

The Terraform is setup to use variables for everything which I don't people to have access to, such as my account id. These values are injected into the Terraform .env file that I get to run locally using the Terminal.env extension to make it easier to pull these values in when I want to run the Terraform.

I'm going to talk through each of these files individually (outside of main and provider) as they all have interesting features

API Gateway

Within this .tf file I've got configuration entirely related to the single endpoint I've got in my gateway. However, there is actually 2 endpoints i the repository, a POST which actually returns the data and an OPTIONS request which is used for CORS. Unfortunately, the way to write Terraform for gateway is pretty verbose as there's quite a lot of steps to enable everything together. I've split it all out though with headers to make it a little easier what each step is - for reference, it's basically having to setup all of these stages:

This particular request is taken from an example here if you want to see how this was setup within the AWS console.

DynamoDb Terraform

The interesting part for this is that I was using Terraform import commands to import existing resources in AWS into Terraform, so I don't need to recreate them and then delete the old one. Otherwise, the setup for this is pretty straightforward

GitHub Runner Terraform

This was interesting in that AWS and GitHub have recently integrated the ability to use OIDC to run build jobs, which is a lot safer than just setting a secret access token as a secret on GitHub and hoping nobody steals it in the several years you don't touch it. How this works is that GitHub and AWS negotiate for a short-lived access token that can run stuff in your account via an AWS role you define. I did purposely leave out defining a role for this account as I wanted to control access more directly via the AWS console as it's quite important.

IAM Terraform

This is a pretty standard (and small) section which basically gives the AWS lambda expression access to the DynamoDb table and write logs about the output.

Lambda Terraform

I've previously has lambda expressions written which are uploaded to S3 because keep track of versions was important, in this implementation though, I've got it to zip up and directly upload into a lambda expression. However, what can do is set a source code hash like this:

source_code_hash = data.archive_file.zip_the_python_code.output_base64sha256

Which means that the lambda will only be uploaded if there's been actual changes to the file. I've also set a retention policy for AWS CloudWatch as it doesn't have this by default.

Overall

I did quite enjoy writing the Terraform, it's interesting to work on something like this and really, there's nothing quite like just entering terraform apply -auto-approve and just watching everything get update for you.

CI-CD Backend

The backend can be found here

So to start with I just setup a private GitHub repository (that I've since made public) and stick a Terraform and Python .gitignore file into it and the MIT licence. After that I added some Git pre-commits to do stuff like auto-format the Terraform scripts and check if I've left any secrets in plain-text on the repository. I did get a false positive on the GitHub thumbprint for the OIDC token which is why it has # pragma: allowlist secret written next to it.

Once I'd done this I committed everything I had and it was time to work on the GitHub Actions project. As the Terraform is using a load environment variables to run, I set this up as secrets on Actions so that they couldn't be seen in logs:

After this I wrote a pipeline which does the following:

Install and cache dependencies
Checkout the code
Run the python tests
Setup the OIDC tokens for GitHub
Run a bunch of Terraform formatting tools and checks to make sure everything's good
Apply the changes if the build is running on the main branch

I also got the build to output some nice content about Terraform in a pull request:

I should probably say, I work on builds pretty regularly so this stuff isn't that new to me

CI-CD Frontend

The frontend can be found here

The CI-CD for the frontend was simpler than the backend as it didn't have a load of Terraform to run, and it really just needs to sync with an S3 bucket and run an invalidation to force the changes to propagate. Given there's a limit to the number of invalidation requests you can do before being charged, I did think about adding a rolling monthly limit, but ultimately decided there wasn't much point as I've used less than 20 of them out 1000 I get free every month. I also added secrets needed by the github runner to get an OIDC token.

Blog post

Well, this is what you're reading (if you managed to get to the end of my apparent essay on this topic)

I did decide though, that I'd also automate my ability to add blog posts to dev.to using GitHub. I decided to be meta about it and published a blog post about how I did this here for practice.

What's next

I'm probably going to take a short break from this, but I did really enjoy it so I likely do more of this. Interestingly, one of my next projects has just arrived so I'll probably working on that soon - think it's good idea to write a post about it when I finish, as writing these articles was a good way to fix it into my mind. I've also got a few other more further into the future projects - I'd like to write a homepage which is more reactive (pun possibly intended?) that I'd like to do at some point.

In terms of this project it's pretty much done just need to keep everything updated, write the proper Python tests with the updated library and likely run through and edit these posts.