DEV Community: Rex

JavaScript or typescript script for automation

Rex — Mon, 09 Dec 2024 23:24:25 +0000

Instead of writing shell scripts for automation, write them in Typescript or JavaScript and run them with node:

Way easier to write and maintain and AI will happily do it for you with higher correctness than the shell scripts
Its platform agnostic! Node runs the same way on windows/Linux/mac
Javascript automation script can do anything you can think of.

Personally I hate writing shell scripts because of weird syntax, hard to reuse, zero testability, mostly I think I am not smart enough for it and I am more than ok with it. For me, simplicity wins.

Testing from outside with docket containers with zero mocks is a game changer

Rex — Mon, 09 Dec 2024 22:57:16 +0000

Being able to test the whole web application as a black box from outside with Playwright is a project saver. Its like having tons of QAs checking everything all the time. Docker is one the best thing happened to the software industry!

Testing the .net webapi backend(ef and postgres) from the outside using Testcontainers library and .net’s WebApplicationFactory also give us very high degree of confidence. Game changer.

Zero mocks means less code to maintain. Tests ignorant of implementation details are also remarkable easy to write and maintain, and they bring freedom to refactoring which lead to easy to read maintainable code.

Win-win-win🎉🎉🎉

Apply Domain Driven Design with EF 9

Rex — Sat, 23 Nov 2024 23:19:53 +0000

I am having so much fun with .NET 9 and EF Core 9. EF Core has very good support for Domain Driven Design. We can use owned entities to model value objects, and they are mapped in the database sensibly. EF Core has good default rules, but we can configure them to the teeth.

Collection of value objects are mapped to separate tables and they are always eager loaded with the aggregate. This causes potential performance issues. For read-only queries, we can use AsNonTracking to improve performance, and for commands we can use AsSplitQuery.

However, these operators are not without downsides. I ended up using regular entities for collections of value objects and making them inaccessible from the DBContext to force the use of aggregates. Those entity models can implement ValueObject behaviors, Id being one of the properties for equality.

I am happy with the middle ground achieved above where I can apply DDD and enjoy the best performance, with a bit of a hack.

Moving on, I also implemented audit trails to record all transactions for the capability of being able to roll back transactions. Then I found that the single owned entity in the domain is being treated as a separate entity by EF Core, therefore they use one entry in the audit log.

I do not like the look of the audit log with an implementation detail like owned entities jamming up the audit trail. So with regret, I unpacked them all and use regular properties instead for cleaner audit trails.

Takeaway: EF Core 9 supports value objects in many ways. However, I ended up not using any of them:

Eager loading of collections of value objects causes performance issue
the non-collection value objects are treated as stealth entities, polluting audit trails.

What a missing favicon can do..

Rex — Mon, 11 Nov 2024 23:02:58 +0000

Just fixed a bug, the remix app rerun the root loader on every page. Turns out we have a link loading favicon and the icon file is missing, adding it fixes the problem.

My guess is the browser tries to reload the missing favicon which is a navigation action, causing a reloading the app.

How to authenticate and register GitHub Runners hosted in Azure Container Apps with GitHub App via Azure Key Vault

Rex — Wed, 13 Sep 2023 16:57:28 +0000

Recently I have been working on designing and automating the deployment of GitHub Runners hosted by an Azure container Apps via Pulumi. One of the challenges is to authenticate with GitHub without the use of a long lived PAT token.

The more secure solution is to authenticate with GitHub App’s private key. The challenge is how to handle the private key in the form of a PEM securely.

Azure Key Vault offers a way:

The PEM file is uploaded manually to the Key Vault as a Key
Once in the Key Vault, the PEM file is not downloadable or visible to anyone
The Key in the Vault can used to sign JWT tokens with the Key

The next challenge is how to access the Key Vault securely from the runners running inside the Container Apps.

Azure Managed Identity comes to the rescue.

During the bootstrap of the runners, we can sign in with Azure Cli with the Managed Identity of the Container App via DefaultAzureCredential, which has the RBAC role of CryptoUser permission, and request the Key Vault to sign a short-lived JWT token. Then we can request an access token from the installation of the GitHub App and finally use the access token to register our runner.

The big Gotcha is that only the System Managed Identity can be used this way, not the User Assigned Identity.

One thing to do before Git Rebase to other branch

Rex — Fri, 12 Aug 2022 12:55:00 +0000

When rebasing, Git will iterate each commits since the divergence.

If there is a conflict in a commit, it stops and wait for the conflict to be resolved. After resolving the conflict, we can “Git rebase —continue” to resume the process.

It can be confusing to see stale changes and resolving conflict with stale changes again and again is not fun.

The one thing to do before rebasing to other branch is: squash or fix up commits into one commit so that when you resolve conflict, it is against the latest changes and you only have to do it once.

To squash the commits: “Git rebase -i HEAD~5”, replace the number 5 to the number of commits you intend to squash.

Reduce Complexity with Nullable Reference Types

Rex — Wed, 11 May 2022 12:53:23 +0000

Introduction

C# 8.0 introduced the feature Nullable Reference Types(NRT), differentiating reference types and nullable reference types and giving us the ability to explicitly mark a reference type as nullable, offering compile-time static null-state analysis. (Nullable reference types | Microsoft Docs),

I spoke to families and friends about NRT and I have three questions:

We are very confident about our TDD-produced codebase, and with extensive integration tests, all the use-cases are guarded by our tests, what benefits does the NRT offer?
We have extensive validation rules, why should we care about NRT?
With NRT, I am often forced to write very fat constructors, I hate it because it reduces readability and readability is the number one priority! Why is NRT more important than readability?

In this blog post, I attempt to answer the above questions.

The short answer has two parts:

Requirement: NRT requires us to be mindful and explicit about whether a property can be null or not. Without this mind shift, NRT will only confuse everyone.
Benefit: Assuming we are explicit about all reference types’ nullability, the static null-state analysis offered by the compiler will effectively increase the readability of our codebase and avoid throwing NullReferenceException at runtime without us paying much attention. It also promotes better encapsulation and simpler code. As a result, improves our development experience.

TDD and Nullability

TDD is about testing Input/output-based business requirements, focusing on the business functionalities, and avoiding testing implementation details at all costs.

TDD should pick up some cases and result in adding validation rules to guard against user inputs. However, TDD does not prevent NullReferenceException in our implementation details because they do not test implementation details.

Yes, if we write the correct integration tests, this will be picked up, but there is a big if here.

With NRT enabled and properties correctly marked as nullable, we can easily iron out most of NullReferenceException compile time.

Validation Rules and NRT

Validation rules are necessary to guard against public APIs and user inputs, we will always need to have them with or without NRT because we have no control over actions that happened outside.

However, we can control ourselves not to violate the NRT rules set for ourselves, we can rely on the compiler to eliminate the NullReferenceException. So writing validation rules to guard the internals for NullReferenceException is cluttering our codebase and increasing the complexity unnecessarily.

Readability and Better Encapsulation

When a class object has many required properties, with NRT enabled, the compiler would suggest generating a constructor to initialize the required properties. We would have a very fat constructor with way too many parameters, and the readability would be greatly reduced.

We should enforce the required properties in the constructor and ensure they are always valid(not null). It reflects my favorite definition out there for Encapsulation: An encapsulated object is an object that guarantees always stay in a valid state, a black box.

But readability should always be on the top of the priorities! That’s what record type is for, together with named parameters, we can enjoy the same readability.

With the Record type, we remove the need for a fat constructor by using the record primary constructor. For example, the below class on on the top becomes a record type that followed.

public class Person
{
    public Person(string firstName, string lastName)
    {
        FirstName = firstName;
        LastName = lastName;
    }

    public string FirstName { get; init; }
    public string LastName { get; init; }
    public string? FavoriteBook { get; set; }
}

public record Person(string FirstName , string LastName){
    public string? FavoriteBook { get; set; }
}

And with the use of named parameters shown below, we can enjoy the same level of readability compared to the syntax we use and love as shown below without NRT and constructor enforcement. The beauty is that order doesn’t matter with the named parameters.

//The world without NRT and constructor:
var rex = new Person{
  FirstName = "Rex"
  LastName = "Ye",
  FavoriteBook = "The Godfather"
};

// With named paramters (not much different than the code without NRT and constructor followed)
var rex = new Person(
  FirstName: "Rex"
  LastName: "Ye"
){
  FavoriteBook = "The Godfather";
};

C# 11 may release a “required” keyboard. With it, constructors are no longer needed, it also enable us to use object initialiser.

Conclusion: NRT Reduces Complexity

When we are explicit about the nullability of our reference types, we are effectively telling the compiler what can be null and what cannot be. Most of the time, with the later versions of c#, the compiler will help us by issuing warnings about nullability and help us eliminate most of runtime NullReferenceExceptions.

With NRT, assuming that we do not violate the NRT rules intentionally(for example giving null! to the required parameters), we can safely avoid checking for nulls or writing validation rules for our internals, reducing the complexity of our codebase.

With NRT enabled everywhere and warnings dealt with, we would see a positive ripple effect on our development experience.

We still need to put the null checks and validation rules for our public APIs because we cannot control what happens outside of our codebase.

Limitations

There are times the compiler issues false positives. When we are sure that a property cannot be null, we can safely use ! to suppress the warnings. Please use ! operators with care.
There are a few Known Pitfalls we need to keep in mind.

A Naïve Construction of a Suffix Array for an auto-complete function with considerable input size via TDD workflow

Rex — Tue, 01 Mar 2022 12:27:25 +0000

What is a suffix array

Wikipedia definition:

In computer science, a suffix array is a sorted array of all suffixes of a string. It is a data structure used in, among others, full text indices, data compression algorithms, and the field of bibliometrics.

Suffix Array is a space efficient alternative to Suffix Tree. The construction of a suffix array can be O(n) with an advanced algorithm. It can handle extremely large input, and it makes searching substrings very efficient.

Example

The steps to build a suffix array for the word Banana are:

find a list of all the suffixes. They are:

Suffix	index
banana$	1
anana$	2
nana$	3
ana$	4
na$	5
a$	6
$	7

sort the suffixes. The array of the new sorted indexes is the suffix array for Banana: [7, 6, 4, 2, 1, 5, 3].

Suffix	index
$	7
a$	6
ana$	4
anana$	2
banana$	1
na$	5
nana$	3

Because we can reconstruct all suffixes with the original string and its suffix array, there is no need to store any suffixes; therefore, it is very space-efficient compared to the suffix tree.

Typically, we add a $ sign to the end of the word to separate words i.e. to prevent a suffix of the previous word from being treated as prefixes to the following word. For example, if we are tasked to build a suffix array for Love Banana, we would concatenate the two words into a single string Love$Banana$ and then build the suffix array. The $ sign prevents the program to treat ove as a prefix of banana. Note that we can choose any non-alphabet characters as long as it is smaller than any alphabet.

Auto-complete

Input: An array of terms and a search text
Output: a list of terms containing the given search text

Approach

build a suffix array(naive approach: O(n^2logn), there are ways to build a suffix array at O(n))

concatenate all terms, with $ at the end of each term, to prevent suffix being treated as a prefix to other terms
get all suffixes from the concatenated string
sort the suffixes, remove suffixes starting with $
build a suffix array from the sorted suffixes

binary search the suffix array to find suffixes matching the search text

start in the middle, compare search text with the middle suffix
if smaller, search in the left half
if larger, search in the right half
if found a match to the search text
- expand and get all adjacent matches
return all matches

Code

Tests

Tests result from the above planning in the approach section and TDD workflow. The tests can also serve as documentation for the implementations.

Implementations

My ultimate goal is to write simple and easy to understand code i.e. require very low cognitive load to read. If you are interested, you can read this awesome blog post by David Whitney Writing good code by understanding cognitive load.

If you feel you need to work out mentally on how something works in the code, I have failed.

Clean Architecture vs Vertical Slice Architecture

Rex — Tue, 01 Feb 2022 22:36:40 +0000

I was asked about clean architecture and vertical slice architecture during an interview, which caught me off guard. I use Clean Architecture, DDD and MediatR for my backend, but I had not heard about vertical slice architecture before. We learn a lot from interviews, we definitely should do more of these.

I was curious about vertical slice architecture, so I looked up. Ironically, it turns out that my application layer in Clean Architecture is all about Vertical Slice Architecture(VSA). The use of MediatR enforces VSA. MediatR and VSA have the same father, the famous Jimmy Bogard.

MediatR is the library to help encapsulate each HTTP request into a single handler that takes a request and returns a response. The handler is a black box and does the necessary thing to produce a response, just like a pure function(strictly speaking, only queries).

The result is vertically slicing the application into handlers. The main benefit of this approach is that requests(use cases) are isolated, input/output based and self-contained units like Lego pieces. It makes the code base simple, flat and easy to maintain.

In my application(epicERP.app), I use both Clean Architecture and Vertical Slice Architecture.

Clean Architecture:
- I have Domain and Application layer hosting all the business logic.
- The Presentation Layer is a web API layer, but the controllers have no logic. All they are doing are calling handlers inside Application Layer.
- The infrastructure depends on the Application Layer via interface Inversion of Control.
- Domain-Driven Design principle is implemented, i.e. domain logic is pushed down to the domain layer.
- I do not have any services or a service layer, which is the main difference from traditional onion layered architecture.
Vertical Slice Architecture: my Application Layer consists of self-contained commands and queries; they usually inject a Unit of Work that contains repositories. Through the repository, the handlers have access to the database.

Implementing Clean Architecture, Vertical Slice Architecture and DDD together made my application extremely easy to maintain because use cases are translated into independent Lego pieces. Most of them are simple and stupid.

If you are curious about Clean Architecture and Vertical Slice Architecture, I recommend these videos:
Vertical Slice Architecture, Jimmy Bogard's presentation back in 2018. The Pipeline Behaviours mentioned in this video are very interesting.
Clean Architecture, by Jason Taylor.

TDD and Publish-Subscribe pattern implementation via TDD

Rex — Tue, 25 Jan 2022 13:12:55 +0000

My understanding of TDD

TDD is a practice that follows red-green-refactor circles:

Red: write a minimal and straightforward test first, one at a time. The test fails hence the name 'red', because the code is not written yet.
Green: write just enough code to get the test, be it a shortcut that doesn't make any business sense, for example, hard coding a string. The aim here is to get the test to pass i.e. turning green.
Refactor: refactor both tests and code and keep the test green
Repeat with more tests, one at a time, as simple as possible, edging closer to the business requirements

TDD requires an accurate understanding of business requirements. TDD is the process of constantly expressing business requirements in the form of tests, one small step at a time.

TDD is the ultimate weapon for highly maintainable software. However, from what I learned so far, it is not practical to apply TDD to all codes or you will get burned and give up.

The most effective way is to separate the business logic from other noises to become an input/output based structure. TDD the business logic.

Testing is hard. Sticking to testing is all about forcing ourselves to design our code in such a way that testing is not hard anymore. It is the design that is hard.

What is publish-subscribe pattern

Definition from Wikipedia:

publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead categorize published messages into classes without knowledge of which subscribers, if any, there may be.

Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.

The essence of the pattern is the decoupling of publishers and subscribers.

This blog post will show an attempt to implement a topic-based publish-subscribe pattern using Typescript and Jest.

Behaviours

publishers can send messages with a topic and payload
subscribers can subscribe to any topics and provide a call back function
subscribers' call back function gets called with the payload if any publishers publish any of the subscribed topics

Constrains

Publishers and subscribers do not know the existence of each other

Code

Testing a mui Auto Complete Adaptor Component integrated with React Hook Form

Rex — Sat, 22 Jan 2022 11:59:54 +0000

Subject Under Test

A mui auto-complete adaptor component integrated with React Hook Form's form context. It uses the Controller component from React Hook Form(RHF) and configures mui's Autocomplete to handle validations and more. I use this component instead of mui's Autocomplete in all my forms.

Behaviours

It registers the Autocomplete component with RHF's form context
It inherits all the behaviours from mui's' Autocomplete and accepts all Autocomplete props.
It has three modes: edit, locked, and read-only. In edit mode, it renders a functional Autocomplete component. In locked mode and read-only mode, it renders a TextField component.
It supports single and multiple selection modes
It can limit the number of options it renders with optionLimit prop.
It has a build-in required validation rule
It accepts validation rules from rules prop.
It can be locked(non-editable in edit mode), and it would show a lock icon with a tooltip explaining why it is locked.

Variants - a design decision

The component has three variants:

EpicAutocomplete, a regular auto-complete. It has the same API surface as mui's Autocomplete component plus binding props for RHF.
EpicAutocompleteWithManger, an auto-complete with an additional option for managing options, extends EpicAutocomplete and has extra required props: manager and managerLabel.
EpicAutocompleteWithUrl, an auto-complete with a link icon button for users to navigate to the detail page of the selected entity. It extends EpicAutocomplete and has an extra prop: url.

Before writing the tests for this component, I did not have the above variants, and all the functions were crammed into a single component, which was a leaky implementation.

Many instances don't need the manager or URL but carry the baggage with them.
URL require react-router, and it adds unnecessary dependencies to users that don't use it
The component internals are too complicated.

For the new structure, I moved the implementations into a base component called EpicAutocompleteBase.

It is private and not exposed to external libraries. It cannot be used in forms directly.
All three variants are composed of it.

With the above design, the API for the variants are more precise and lean, and they don't carry unnecessary dependencies with them.

I sincerely invite suggestions from my dear readers for a better approach.

Notes

Each variant has its separate tests. Their tests are focused on their public APIs (props)
No direct tests for EpicAutocompleteBase because three variants are testing it.

Code

Testing a mui Date Picker Adaptor Component Integrated with React Hook Form

Rex — Fri, 21 Jan 2022 22:01:35 +0000

Subject Under Test

A date picker component integrating mui's date picker with React Hook Form's form context. It uses the Controller component from React Hook Form(RHF) and configures mui's DatePicker to handle validations and more. I use this component instead of mui's DatePicker in all my forms.

Behaviours

It inherits all the behaviours from DatePicker of mui and accepts all DatePicker props as-is.
It takes name, formContext and defaultValue required props and registers the DatePicker to the form context of RHF
It has two modes: edit mode and read-only mode. In read-only mode, it is disabled, has no date picker icon button and is rendered as a standard(underline) TextField. In edit mode, it is rendered as outlined TextField.
It builds in the required validation rule and takes a required prop.
It builds in a validation rule for invalid date input
It accepts validation rules and enforces them.
It takes an optional onChange prop. It will update the value and trigger the given onChange method on change.
It has a default mask and date format and can be changed with props.
It defaults to size small, full width and shrink label.
It set time to end of the day.
It takes a style prop for styling the underlying TextField.

Code

Notes

matchMedia is mocked so that the date picker can be rendered in desktop mode with the date picker icon button
TestComponent sets up a React Hook Form environment and shows how the SUT can be used.
FormForTesting is a testing utility component for testing React Hook Form form components.
The tests are grouped into three categories: appearance, behaviours and validations.