DEV Community: Bernhard Häussermann

Creating a weak dictionary in .NET

Bernhard Häussermann — Thu, 10 Nov 2022 04:14:15 +0000

I once found myself in need of a dictionary for temporarily holding references to some objects. This dictionary was to hold weak references to its values so that said values would become eligible for garbage collection once they were no longer needed.

Such a WeakDictionary would be useful when:

you need a temporary handle to objects having a known key, and
your code isn't aware of when these objects fall out of use (i.e. you can't remove the entries explicitly).

The most obvious solution to this scenario would be to use a Dictionary<KeyType, WeakReference> object. The WeakReference will ensure that the dictionary doesn't contribute to the reference count of the object being referenced, allowing the object to be reclaimed by garbage collection even if it still has an entry in the dictionary.

However, if our code isn't aware of when an entry can be safely removed, dictionary entries will accumulate over time. We would like for the dictionary to automatically remove any "stale" entries.

In the spirit of test-driven development here's an NUnit test that shows how we would expect the WeakDictionary to behave.

Due to the way the garbage collector is implemented in .NET Core, there are a few hoops we need to jump through in order to ensure the test works. See this StackOverflow answer. In particular, compiler optimizations need to be switched on in the test project, and the debugger must not be attached.

While searching for a solution, I quickly came across this StackOverflow post. Whereas there were a lot of proposed answers, none of them were really what I was looking for. Each answer either:

implemented a dictionary with weak references to the keys as opposed to the values, or
relied on an explicit "purge" operation that needed to be called from time to time in order to clean up old references. I was looking for an implementation that would remove the entries of garbage collected objects immediately.

Thankfully, though, the answers that were there provided me with the necessary ingredients that enabled me to create a dictionary that would work the intended way.

A key ingredient is the ConditionalWeakTable. It is a dictionary-like data structure that holds weak references to its keys and removes an entry when the key is garbage collected. This is exactly what we need, except we need that behavior on the value as opposed to the key.

Using this ConditionalWeakTable internally is a good start to our implementation of the WeakDictionary as it may enable us to automatically remove dictionary entries when a value gets garbage collected: just add values of the WeakDictionary as keys to the ConditionalWeakTable and use it to properly maintain the entries in a separate dictionary.

But how do we become aware of when an entry is removed from the conditionalWeakTable in order to remove the corresponding entry from the internalDictionary?

Think about what happens when a key object of the WeakDictionary is garbage collected:

The corresponding entry from conditionalWeakTable is removed.
This causes the reference count of the corresponding value in the conditionalWeakTable (typed as object) to be decremented, causing it to be garbage collected as well (provided there are no other references to it).

Together with the fact that when an object is garbage collected, its finalizer (historically known as a destructor) will be invoked, this provides us with a way to act on this event by using a class such as the following:

We make this class be the value type of conditionalWeakTable and bring everything together in our class' Add() method as follows:

This way when a value object is garbage collected, it will immediately be removed from internalDictionary.

All of the remaining methods of the IDictionary<> interface are implemented to simply relay the method call to internalDictionary.

We run the test and, sure enough, it passes!

This gives us a useful data structure to use when we need key-based access to objects whose life-time we don't control.

The complete implementation is available in the GitHub repo.
The library is available as a NuGet package.

How to display an external link symbol next to links

Bernhard Häussermann — Fri, 05 Aug 2022 14:56:52 +0000

One day I was working on my personal portfolio site, adding external links towards some of the web apps that I had built. I was about to add some external link symbols to them when all of a sudden I realized... I'm not really sure how 😶.

It's a convention of the web that you display the symbol next to any link that navigates to a different web site (i.e. a different domain than the one you are on).

On some web sites the link will open in a new tab, but on many it won't. It can be argued that it's not for the web page to decide that the link should open in a new tab since the user can always intentionally do this by either middle-clicking the link or right-clicking and selecting "Open Link in New Tab".

I turned to Google, looking for a "best-practice" way of displaying an external link symbol. Given how common the symbol is, one would expect there to be a standardized way of displaying it, right?

It turns out there isn't 😶. I came across this StackOverflow post which asks exactly this question and contains a wide array of answers ranging from using a Unicode character, to using an icon library, to using an image. And yet, none of these answers completely satisfied me. I will discuss each type of method that was suggested (I'm linking each section to the relevant answer of that post) and why I had decided against it before I present the solution that I did end up using. Hopefully you will find that solution useful too.

Unicode character

Using a Unicode character to represent the external link symbol would be the easiest solution by far. One suggestion is to use the North East Arrow Unicode character ↗. It looks pretty similar to the external link symbol, but alas, where's the box?!

As of this writing, there are 144 697 different Unicode characters and one would think that there would be one to represent an external link! It turns out that this had been proposed, but the proposal was rejected. In my understanding the reason for the rejection was that the symbol does not represent "plain text" in the sense that it doesn't constitute the informational content on the page. To justify this practically, if you would copy and paste some text off of a web page, you wouldn't want the external link symbols to come along with the text. Ok, fair enough...

Icon library

It turns out that some icon libraries, such as Font Awesome 5, do feature an external link symbol (see the Solid Icons cheat sheet).

It looks nice and you can easily adjust the symbol's size and color. This would probably have been my solution of choice if my project had an icon library included. However, it didn't, and I wasn't going to add an entire icon library to my project for just this one symbol.

Raster image

The answer featuring the image presents the following CSS code:

I like the fact that it's expressed as a CSS rule because that way we aren't duplicating the code for the symbol, and it removes clutter from the HTML code.

However, I don't like using raster images for things like icons. They don't remain crisp when viewed on a screen that is scaled (think of Retina displays), and if the user should increase the browser's zoom level, the image will also become blurred. There's also no easy way to change the image's color or to resize it without distorting it.

SVG image

The answer with the SVG image presents the following CSS code:

This CSS rule comes very close to being my preferred solution. The only nit-pick I have is that the SVG code is completely shrouded as a base-64 string. I prefer including the SVG code as a URL-encoded XML string for transparency.

My preferred solution

I ended up using the following CSS rule:

Which could be used as:

Granted, the fact that the SVG code has to be URL-encoded makes parts of it a bit cryptic, but at least you can still read it for the most part. You could even easily change the color should you want to (which is specified here as stroke='#666').

Ultimately, in a project that is already using an icon library I would first check to see if it includes an external link symbol and use that if it does. Otherwise I would use the aforementioned CSS rule. In both cases we are using vector graphics, so there are no issues related to resizing, and the color can easily be changed if desired.

Say "no" to silent failures in your methods

Bernhard Häussermann — Wed, 18 May 2022 18:48:41 +0000

One of my personal pet peeves when working on code is encountering logic that checks for some unexpected state and when that unexpected situation applies... does nothing. This is a classic case of sweeping mistakes under the carpet. Consider the following C# code:

In this example we check the state of the order and if it is not in the expected state for processing, we do nothing. The problem here is that if the ProcessOrder() method gets called on an order that is not in the right state for processing, this is most certainly a bug in the code that is calling the method. By ignoring the unexpected state, the ProcessOrder() method is hiding the issue, making the underlying bug much harder to locate.

This is because the code calling ProcessOrder() will assume that after the method had been called, the order had been successfully processed. The if-condition in ProcessOrder() makes it so that this assumption is not true so that the calling code might proceed interacting with an unprocessed order as if it were a processed order. This would cause potential havoc, diverting attention away from the source of the problem.

The core problem with ProcessOrder() is that there are situations where the method would successfully return despite not having done what it was expected to do, i.e. processing the order.

The Solution

My personal rule of thumb for preventing this kind of situation is that a method should always do what it claims to do in its name (in this case processing the order), or otherwise throw an exception. If the method returns without an exception, this must imply that the operation was completed successfully. This is the central point of this post, so I will state it explicitly:

A method must return without throwing an exception if and only if it successfully did what it claims to do in its name. In any other case an exception must be thrown to indicate that what the method was expected to do could not be done.

Therefore a better implementation of the method above should look like:

The Try-Parse Pattern

There can be operations where it is not unusual that the operation can't be performed for one reason or another. That is, the reason for the operation not being able to be performed is not due to an error or a programming bug, but merely a valid possible scenario. Since exceptions are supposed to be used for exceptional cases only, such a situation calls for a different approach. Consider the following method signature:

The caller should expect that if the method successfully returns, it would provide a Route leading from the provided source to the destination. If there is no such route, then based on the method's name, it should throw an exception because it wasn't able to "calculate the route". But what if the non-existence of a route is a common and valid use-case? We could make the method return null if no route could be calculated, but this would amount to a false impression from the perspective of the caller that the route was calculated when that's not the case.

This kind of scenario is often a good candidate for the try-parse pattern. Based on this pattern the method signature above would change into:

Now, if no route is found, the method would return false to indicate the fact. When it does, it's not "lying" about having performed its task because it did, in fact, try to calculate the route like its name states.

The Microsoft documentation about the try-parse pattern presents this as a much more performant alternative to throwing exceptions where exceptions would be thrown for common scenarios. Another benefit of using this pattern is that it makes it clear to the method caller that a certain case needs to be handled explicitly.

Making a habit of ensuring all methods always do what their names say (or throwing an exception otherwise), makes it easier to reason about the code and helps make the source of programming bugs be more obvious when they occur.

String Literals in JavaScript: Should I Use Double-quotes or Single-quotes?

Bernhard Häussermann — Mon, 09 May 2022 05:20:02 +0000

Most JavaScript (and TypeScript) developers know that the language allows expressing string literals using either double quotes (") or single quotes ('). Given that both ways of expressing strings are semantically equivalent, this leads to the question of which one is the “right” way.

Generally, when it comes to coding style, it is advisable to conform to the convention of the particular language you are dealing with (for example, using camelCase for methods in JavaScript, but PascalCase in C#).

Unfortunately, it appears that there is no official convention for string quotes when it comes to JavaScript. So, the next best thing we could do in deciding on the best style of quotes to use is to try and find out which convention is more common.

Looking at some popular JavaScript projects on GitHub, I noted that react, moment, and express all use single quotes for strings. However, another popular project, tslib, uses double quotes 🙄. So, while it would seem that single quotes is more common, this metric is not entirely conclusive.

The deciding factor that convinced me personally applies to web development using a JavaScript framework that uses HTML-templates, such as Angular or Vue.js. In particular, these frameworks allow us to use JavaScript expressions for HTML attribute values. Take for example the following:

Since HTML itself also happens to allow either double quotes or single quotes for delimiting attribute values, this kind of brings us back to the same debate. In the case of HTML attributes, however, I will always stick to double-quotes, since using single-quotes for attribute values is very uncommon and frankly just seems wrong.

So, given that we use double quotes for HTML attributes, the only way for us to use double quotes in a JavaScript expression would be like so:

which for obvious reasons is pretty much out of the question. We are therefore basically forced to use single quotes for any string literal inside of a JavaScript expression.

When it comes to coding conventions, applying a convention consistently is more important than using the “best” convention. If we are then going to use single quotes in JavaScript within HTML, we must do the same everywhere else for consistency’s sake. For this reason my personal final choice for JavaScript literals is to use single quotes.

Coding style tooling

Regardless of whether you are decided on using single quotes or double quotes in JavaScript, the more important issue is applying your convention consistently throughout the code base. ESLint is one tool that will help a great deal in enforcing a consistent coding style. You supply it with a config file specifying a list of rules describing your coding conventions, and it will indicate where your code is not conforming to the specified rules. See the following links to get started:

ESLint Getting Started Guide
TypeScript ESLint: tooling for adding TypeScript-support to ESLint
ESLint Integrations: locate the plugin for adding ESLint support to your favourite IDE

Add the rule quotes: ['error', 'single'] to your .eslintrc.js to make ESLint enforce single quotes.

If you happen to have a lot of style transgressions in your code, know that some rule violations can be automatically fixed by ESLint (including strings literals using the wrong type of quote). Just run the eslint command including the --fix option, like so (adjust the --ext option based on whether your code is in JavaScript or TypeScript):

eslint -c .eslintrc.js --fix --ext .js ./

I can also highly recommend Prettier for maintaining a consistent coding style and fixing up existing source code.

Which one of the quote-styles you settle on using in your JavaScript / TypeScript code is not nearly as important as actually committing to that and applying the rule consistently. Consistent application of coding conventions is not to be underestimated since it can significantly reduce the cognitive overhead when reading the code. The use of tooling such as ESLint and Prettier can help a lot in ensuring the coding style remains consistent even with many developers contributing to the code base.

Adding dark mode to your web app

Bernhard Häussermann — Fri, 11 Feb 2022 04:37:57 +0000

Dark mode in applications and web sites is a feature that seems to be becoming increasingly popular and ubiquitous. Thankfully, adding basic dark mode support to an existing web app that uses CSS is a fairly straightforward process. If you want to get started quickly, you've come to the right place! For a more in-depth and comprehensive guide, see the Complete Guide to Dark Mode on the Web.

The Demo App

We will be adding dark mode support to a very simple web page that is backed by the following HTML code:

Here is the CSS used to style the "item" elements:

Here is what the page looks like in the browser:

To test how well we are doing in dark mode out of the box, I first ensure that my browser's theme is set to the system theme (or "auto"), meaning it will display everything in light or dark mode based on the operating system setting (this should be the default). I then change the operating system setting to dark mode (assuming it was in light mode to begin with). The browser's window border changes accordingly, but the web page does not. We've got some work to do.

Adding basic dark mode support

The quickest first step we can take is to tell the browser that our page supports both "light" and "dark" colours by adding the "color-scheme" meta tag to our page's <head> element:

Refreshing the page reveals that we've already achieved a result:

The elements that didn't have colours assigned to them in the CSS now take on the browser's default colours for the dark theme. This wasn't happening before because in the absence of this tag, the browser will assume that the page supports light mode only. Other benefits to specifying the "color-scheme" meta tag are:

unstyled controls (e.g. text boxes, check boxes) will also have their appearance changed to correspond to the selected theme, and
even if all colours are specified in CSS, having this tag set will prevent a white background with black text flashing while the CSS is downloaded and the dark colours finally applied.

Adding the "color-scheme" meta tag was a quick win, but there's clearly more work left to do: we need to define the dark mode colours for the elements that do have their colours set via CSS. Colours in our CSS code can easily be overridden for dark mode using the prefers-color-scheme media query:

We're looking much better now!

A quick way to find "dark mode" colours is to simply invert all the regular colours. This can easily be done by opening the Calculator, switching to Programmer mode and selecting hexadecimal. Then simply calculate FFFFFF - xxxxxx, where xxxxxx represents the regular colour in six hexadecimal digits. This provides a good starting point from which to tweak the colours further.

Making things managable

On anything but the smallest of applications, CSS code like the above will prove to be a lot of work to maintain with future styling changes. Moving all the colours into CSS variables and setting these variables based on the current theme will result in more tidy CSS code that is much easier to maintain:

By keeping the declaration of all the variables at the top of the file, the colours can now all be changed in one place.

Dealing with images

If your web app features images, they may not fit in well with the dark colours. In this case you may want to create a "dark" version of each image to use instead. If the image is specified in HTML, then switching between the "light" and "dark" version can all be done in HTML by replacing:

with

Allowing colour scheme selection via in-app setting

There may be scenarios where a user would want to override the colour scheme in your app, regardless of their operating system setting. In this case you may want to offer options for "light", "dark", and "auto" (which would use the operating system setting).

At this point CSS alone won't cut it; we're going to have to use some JavaScript to achieve this kind of functionality. One way would be to have the JavaScript code add or remove some special class (say dark) to the top-most HTML element and set up the CSS rules to react to that:

The JavaScript code would have to watch the prefers-color-schema media value in case the user has "auto" selected. The underlying code may look something like this:

Conclusion

Accessibility and improved battery life are just some of the reasons why dark mode has become a popular alternative colour scheme on most of the operating systems we use today; not to mention, it's hip and trendy 😎. Thankfully, implementing dark mode in web applications almost couldn't be easier, so why wait? 😉

Adding TypeScript-support to your Node.js project

Bernhard Häussermann — Mon, 15 Nov 2021 11:18:08 +0000

Many Node.js projects will reach a level of complexity where we'll often find ourselves wondering what a certain object's structure looks like. We'll also have the need for early warnings about errors due to trying to access fields that don't exist or assuming different types from what they really are. These are good indications that you would derive a lot of benefit from using a type-checking system such as TypeScript. This article will look into how to make the transition from JavaScript to TypeScript in an existing project.

TypeScript-support for Node.js is provided by the typescript NPM package. The best way to implement this package will depend on your project's build tooling. If you have webpack set up, then the easiest way will be by using the ts-loader package (see the "Setting up ts-loader for webpack" section below for this ts-loader + typescript setup). However, if you don't have a module bundler configured, then the simplest way to add TypeScript will be through the tsc ("TypeScript compiler") command, which is included in the typescript package. This tsc setup is described next.

Setting up TypeScript compiler (tsc)

tsc is the official TypeScript transpiler which converts your TypeScript source files into JavaScript files that can be executed by Node.js or the browser. This section assumes you have a working Node.js project where you run your main js-file directly using node.

The first order of business is adding the typescript package to your project as a dev-dependency:

npm install --save-dev typescript

The next step is to create a config file for typescript. A good starting point is to generate the file using the command npx tsc --init. Add the following properties to the compilerOptions property in the generated file:

"noImplicitAny": true — Disallows the use of the any type - a common anti-pattern in TypeScript.
"allowJs": true — Allows us to have JavaScript (.js) files amongst the TypeScript (.ts) files. When we need to migrate an existing JavaScript project to TypeScript, this allows us to systematically convert files from JavaScript to TypeScript one at a time. Once the conversion is complete, this flag can be removed.
"outDir": "dist" — The folder where the transpiled JavaScript files will be placed.
"rootDir": "src" — The location of your TypeScript / JavaScript source code.

After adding these properties the tsconfig.json looks as follows:

And just like that, the code is ready to be compiled! Just run the command npx tsc and see the output files appear in the dist folder.

Before trying to run the compiled code, keep in mind that tsc outputs CommonJS-style JavaScript. This means that if your source code is written as ES modules, you need to change the "type" property in your package.json from "module" to "commonjs" in order to run the compiled code (tsc will still interpret your source code as ES modules). At this point the "main" .js file in the dist folder should run successfully via the node command: node dist/my-app.js

Source maps

A side-effect of running the compiled code instead of running the source code directly is that stack traces of errors will refer to the line numbers inside the compiled code instead of in the source code, which is not very helpful. Luckily we can have tsc generate source map files which map each line in the compiled code to the corresponding line in the source code. These can be used to make our application report the correct line numbers in error stack traces.

Getting tsc to generate the source map files is an easy matter of adding the "sourceMap": true property to the "compilerOptions" in tsconfig.json:

Run npx tsc again and note that in the dist folder a .js.map file is created for each compiled .js file. However, we still need to make these mappings be interpreted at run-time. To do this, add the source-map-support package as a run-time dependency. We also add its types declaration package for TypeScript as a dev-dependency:

npm install --save source-map-support
npm install --save-dev @types/source-map-support

And activate it by adding the following to your main source file:

Compile and run the application. Error stack traces will now refer to the lines in the source code.

Adapting script commands

Creating script commands (in package.json) for compiling and running the application is pretty simple:

"build": "tsc",
"run": "node dist/my-app.js",

For a streamlined developer experience we would want to have a command that will listen for source file changes and then recompile and restart the application whenever they occur.

The tsc command conveniently has a --watch flag we can use to recompile. Then we can use the nodemon package to restart the application whenever we detect file changes in the dist folder (due to the recompilation). Hence we can have the following two scripts:

"build:watch": "tsc --watch",
"run:watch": "nodemon dist/my-app.js --watch dist",

But we need these two commands to run at the same time. This can be achieved using the npm-run-all package.

Add the required packages as dev-dependencies:

npm install --save-dev nodemon npm-run-all

Final list of scripts:

"build": "tsc",
"run": "node dist/my-app.js",
"build:watch": "tsc --watch",
"run:watch": "nodemon dist/my-app.js --watch dist",
"start": "npm-run-all --parallel build:watch run:watch"

Run npm start to compile and run the application. Whenever you make a change to a source file, the application will automatically recompile and then restart.

Setting up ts-loader for webpack

If your application already has build tooling set up via webpack, the easiest way to add TypeScript support is by using the ts-loader package.

This time around, add the ts-loader and typescript packages as dev-dependencies:

npm install --save-dev ts-loader typescript

The same tsconfig.json configuration file as above can be used in this case:

Note that in this case, since webpack is already configured to process JavaScript source files, there is no need for including the "allowJs": true flag here, unless you would like both JavaScript and TypeScript files to be processed by ts-loader. If this is the case, make sure to include the extension js in the "test" property of the rule added to webpack.config.js below.

In webpack.config.js add a rule telling webpack to invoke ts-loader for all TypeScript files:

At this point the application should build and run fine. We are now ready to start converting .js files to .ts files.

Migrating existing JavaScript files to TypeScript

At this point we should have a tooling setup capable of compiling a combination of JavaScript and TypeScript files. This means we can systematically convert JavaScript files to TypeScript one at a time, compiling and testing the application along the way by renaming a .js file to .ts and fixing the compiler errors as they come up. Once all .js files in the project have been converted, the "allowJs": true flag in the tsconfig.json can be removed.

Here are some general notes to observe during this conversion:

use strict

Any 'use strict' directives in existing JavaScript files can be removed since the "strict": true setting in the tsconfig.json causes 'use strict' to be generated into the compiled JavaScript files automatically.

Extending the Error class

If you have defined any sub-classes of Error, note that there is a known bug in TypeScript whereby testing for an instance of this error using instanceof will not work.

See this StackOverflow post for a work-around. If you have multiple sub-classes of Error, I would recommend applying the work-around to a common "base" error class (eg. class ErrorBase extends Error) and have all other error classes extend this class.

Alternatively, if your code need not support running on IE 11, you should be able to safely change the compiler target from ES5 to ES6 by changing the "target" property in tsconfig.json to "es6" (see the ES6 compatibility chart). This way tsc will generate all classes as actual ES-classes in the target code, effectively circumventing the bug and obviating the need for the work-around.

Conclusion

There are many benefits to TypeScript that make it worthwhile to take the time to set it up for new projects, and even to convert from JavaScript in existing projects. Making the necessary changes to an existing project's build tooling is generally quite simple, and for new projects there is no need to add a module bundler just to be able to use TypeScript, thanks to tsc.

I have applied this migration to a project from my other articles. Feel free to view the version with the tsc setup or the version with the webpack / ts-loader setup on GitHub.

Adding a Swagger UI page to your Express REST API

Bernhard Häussermann — Thu, 21 Oct 2021 04:55:38 +0000

This article is a continuation of a series about how to build a REST API in Node.js. In the preceding article of this series we added request and response validation that is based on an OpenAPI spec of the API. We used the swagger-jsdoc package to generate this OpenAPI spec from comments in our code that annotate the REST operations.

Consumers of our API won't know how to make requests to it without some form of documentation. The /swagger.json route which we added returns the OpenAPI spec as a JSON string which serves this purpose. However, this JSON is not very easy to interpret for humans. In this article we will show how the OpenAPI spec can be used to render a Swagger UI page which serves as a more visual and interactive documentation for our API.

Add the swagger-ui-express package as a run-time dependency.



npm install --save swagger-ui-express

Then in server.js we register Swagger UI at the route /swagger just after the statement that registers the /swagger.json route:

Loading localhost:3000/swagger in the browser will now show the Swagger UI page for our API. This will show all endpoints and operations that are available and even allow to try out the calls from the browser!

While we now have our API spec presented in a nice GUI, consumers of our API may still need access to the OpenAPI JSON definition to import into their client-side tooling. We can add a link to the already-defined /swagger.json route to the Swagger UI page by changing the previous app.use() statement to the following:

This will make Swagger UI read the API spec from the specified route and it will display a link to it on the page.

The code for the API developed in this series of articles is available on GitHub here.

Adding request and response validation to your Express REST API

Bernhard Häussermann — Wed, 01 Sep 2021 04:51:09 +0000

This article is a continuation of a series about how to build a REST API in Node.js. In the first article of this series, we created a rudimentary REST API for managing a list of employees. But that API doesn't perform any validation on the requests that are received. Therefore, nothing stops the consumer from making a request with incorrect query parameters or a malformed body. So we could for example send the following request:

POST localhost:3000/employees
{
  "surname": "Symonds",
  "firstname": "Andrew"
}

The API would happily add the specified object to the list of employees, even though it has the wrong names for the properties ("surname" instead of "lastName" and "firstname" instead of "firstName").

The express-openapi-validator package addresses this issue by validating requests and responses based on a provided OpenAPI spec. Having our responses validated and not just the requests will be a good idea as that will ensure the API code always responds in the expected schema.

express-openapi-validator provides a function that takes an OpenAPI specification as a parameter and returns a request handler that we can add to Express. The most straightforward way to use it would be to write our OpenAPI spec in a JSON or YAML file and read it from this file to pass into express-openapi-validator. But this would mean that whenever we make changes to the API in future, the corresponding part of the OpenAPI spec would have to be updated in this separate file. It would be much easier to keep the OpenAPI spec up to date if it were defined along with the relevant code.

The swagger-jsdoc package enables us to do this. It looks for JSDoc comments in code (annotated with an @openapi tag) to generate the OpenAPI specification.

Add the express-openapi-validator and swagger-jsdoc packages as run-time dependencies:

npm install --save express-openapi-validator swagger-jsdoc

Let's add the JSDoc comments for defining the POST /employees operation. This involves an employee structure, which we can define as a schema object on the EmployeesController class:

We can then reference this schema object in a comment that we add just before the POST /employees operation:

We use swaggerJsDoc to generate the OpenAPI specification by telling which source files to look for the JSDoc comments via the apis property. We then register the generated OpenAPI spec by passing it into openApiValidator. We add all of this just before the registerRoutes() call:

We can also set a route to serve the OpenAPI spec from localhost:3000/swagger.json as an easy way to access the generated OpenAPI spec. This can help with troubleshooting by ensuring the spec is generated as expected and allows our API consumers to access the spec easily.

The following statement needs to be added before the app.use() call which registers the validation. If it's added after the app.use() a request to localhost:3000/swagger.json will be validated and consequently return a 404 response since the /swagger.json route is not defined in our app's OpenAPI spec.

Now, if we make the POST request as above, we get the following 400-code response:

The following post will show how we can leverage the generated OpenAPI spec to add an interactive Swagger documentation page to our API.

The code for the API developed in this series of articles is available on GitHub here.

How to implement async operations in your Express REST API

Bernhard Häussermann — Mon, 16 Aug 2021 07:37:36 +0000

This article is a continuation of a series about how to build a REST API in Node.js. In the first article of this series, we built a rudimentary REST API for managing a list of employees. However, instead of using a real database, the EmployeesService simply maintained the list of employees in memory. Had it implemented actual calls to a database, the methods would likely have been asynchronous, returning promises rather than performing the work synchronously. In this post we will discover how to properly add support for async functions.

Let's change the EmployeesService.get() method to be asynchronous and see what happens. By adding the following await statement we simulate the database query taking 1 second before it returns the results:

Now, if we call GET localhost:3000/employees/1 the response we get is the empty object {}. Keep in mind that in JavaScript an async method effectively returns a Promise that resolves when the method completes. It appears that in employees-controller.js where we call res.json()...

Express just stringified the Promise object passed into res.json() rather than awaiting it and using the resolved value. We need to pass the resolved value into res.json() instead:

Testing the call using our REST client, this seems to work. However, when we request an employee that doesn't exist, something weird happens. The request hangs and eventually times out.

This is because the request-handler function passed into the get() method above is now async. The get() method handles the case where the passed function throws an error by making the request respond with an error code. Now that the method is async, what happens instead is that the function returns successfully returning a Promise which eventually gets rejected, and get() doesn't handle this as expected.

We can make this route handle errors correctly by using the third next parameter passed into the request-handler function like so:

Running the request for an employee that doesn't exist now returns the expected response.

Suppose now that all of our request-handler functions were async (and in a real-world application, most of them will be). We would have to wrap every operation's code into a try-catch block, and pass the error into the next function to prevent a request hanging forever should an error occur. This is far from ideal. Thankfully, the express-async-handler provides a much more convenient way of dealing with errors.

First, use npm to add the package as a run-time dependency:

npm install --save express-async-handler

And use it in EmployeesController like so:

Of course, if we had not had to deal with the particular case of the 404, then the try-catch block would not have been needed since the asyncHandler() deals with any errors that might happen.

As long as we remember to pass each async operation's function into asyncHandler(), we can rest assured that whenever an error occurs within the operation, it will still return the correct error code and message and not just hang until it times out.

Future posts will show how we can add additional middleware such as for adding request / response validation and Swagger documentation thanks to Node.js packages available for this.

The code for the API developed in this series of articles is available on GitHub here.

Building a Node.js REST API

Bernhard Häussermann — Wed, 04 Aug 2021 05:29:20 +0000

This article is the first of a series outlining the steps to build a REST API from scratch that runs in Node.js using the Express web application framework. In this article, we will show how to set up the project. The following articles will build on this by adding features such as request / response validation and a Swagger UI page for online documentation.

Project setup

The configuration needed for following along in your own project is minimal. All that is required to get started is a package.json file generated using npm init.

Ensure that "type": "module" is set within the package.json file. This declares our package as an ES 6 module so we can use import syntax to import packages in our source code.

Add a basic web server

Add the Express package as a run-time dependency:

npm install --save express

Then create a new file server.js in a folder named src with the following contents:

And just like that we have a working web endpoint listening at port 3000!

In the code above, we have defined a single route which we can use to test that the service is running.

Testing that our endpoint works is easy as:

Run node src/server.js
Using your favourite REST API testing tool (I recommend Postman), request GET localhost:3000/greeting

We should get a 200-response containing some text as a JSON string.

Adding automatic restarts

Running our server script as above means that whenever a change is made to a source file, we need to manually stop and start the program for the changes to take effect. This is easy to fix, thanks to a simple tool called nodemon. We can easily add a script that will restart our application whenever a source file is changed.

First, we add nodemon as a development dependency to the project:

npm install --save-dev nodemon

We then define the following set of scripts in package.json:

"scripts": {
  "run": "node src/server.js",
  "run:watch": "nodemon src/server.js --watch src",
  "start": "npm run run:watch"
},

The run script will run the API without automatic restarts as before if we execute npm run run

The run:watch script will run the API, restarting it whenever any file inside the src folder changes.

The start script will simply run the run:watch script but can be executed merely as npm start

Structuring the code based on REST resources

Most REST APIs have their routes arranged based on a number of resources. We will define employees as a REST resource with CRUD (create, retrieve, update, delete) operations. Keeping with REST conventions, we will define the following routes:

GET /employees: Return the list of all employees.
GET /employees/{employee-id}: Gets the single employee having the ID {employee-id}. Return a 404 (Not Found) response code if no employee with the specified ID was found.
POST /employees: Add a new employee entry.
PUT /employees/{employee-id}: Update the details of the employee having the ID {employee-id}.
DELETE /employees/{employee-id}: Delete the employee having the ID {employee-id}.

If we keep on defining all our routes and the code implementing them directly in server.js, the code will quickly become unmanageable. To help keep the code organized, I recommend defining each REST resource's routes in one file and implementing them in another. We call the file defining the routes the "controller" and the file containing the implementation the "service".

Implementing the employees resource leads to the following folder structure:

src
   controllers
      employees-controller.js
   services
      employees-service.js
   server.js

Here is a simple implementation of employees-service.js.

Whereas in a typical application the objects would be persisted in some kind of a database, we store the list of employees in memory for simplicity's sake.

The EmployeeNotFoundError class is defined in a file named employee-not-found-error.js as:

Note that EmployeesService does not contain any logic that relates to REST notions like query parameters, response statuses etc. The EmployeesService is concerned solely with the details of how employees are persisted. This is in adherence to the Single-responsibility principle. It also makes the class easier to test using some testing framework.

The EmployeesController class deals with the REST-related specifics and hooks up the REST routes to their respective implementations in the employees service:

Note the block-comment before the registerRoutes() method. This is a JSDoc comment that specifies descriptions to use when generating documentation using JSDoc. However, in this case, we add the block-comment only to inform our IDE of the expected types of the method's parameters. Visual Studio Code, for instance, has built-in support for JSDoc and will interpret the type-declarations of the app and controller parameters inside the block-comment to inform its IntelliSense and code completion functionality.

We define the ExpressError class to represent a REST error which is to be handled by a generic error route handler function in server.js:

Finally, we make the following changes to server.js:

To register the routes, we now simply call registerRoutes() passing in the Express application and a new instance of EmployeesService.
We also add a route handler for returning the proper response when an error is thrown.
To parse the request body of the POST and PUT operations as JSON payloads, we add the statement app.use(express.json())

We can now use our favourite REST client to test the different routes to verify the behaviour is as expected:

Get all employees
```
GET localhost:3000/employees
```
Get employee 1
```
GET localhost:3000/employees/1
```
Get employee 2 (doesn't exist)
```
GET localhost:3000/employees/2
```

Update employee 1's first name

PUT localhost:3000/employees/1
{
  "firstName": "André"
}

Add a new employee

POST localhost:3000/employees
{
  "lastName": "King",
  "firstName": "Robert",
  "title": "Sales Representative"
}

Delete employee
```
DELETE localhost:3000/employees/2
```

In conclusion

Getting a REST API off the ground using Node.js and Express is relatively straightforward to do, and by defining separate controller and service classes for each type of API resource, we keep the REST-specific details separate from the underlying implementation details of each operation.

Future posts will show how we can quickly add middleware such as request / response validation and Swagger documentation thanks to Node.js packages that are available for this.

The code for the API developed in this article is available on GitHub here.