DEV Community: Alexey Yakovlev

How to Cache Variables Between Restarts in Your Node.js Application

Alexey Yakovlev — Tue, 27 May 2025 15:14:21 +0000

Hot reloading is a staple of Node.js development: when a file changes, your application restarts, re-running all setup logic. For small projects, this is barely noticeable. But for large monoliths, especially those with heavy startup routines, the wait can be frustrating.

This was exactly my situation. My application used webpack-dev-middleware to compile a massive monolithic project with thousands of modules. Each restart triggered a full Webpack build, which, thanks to legacy loaders, could take up to 70 seconds. On top of that, one server module fetched a 60MB JSON configuration from an external source, and parsing this data took another 90 seconds. Every minor change meant waiting a minute and a half before I could see the results—completely killing my development feedback loop.

Why Not Just Optimize?

I considered several approaches:

Load configuration on-demand:

Not feasible. The configuration was deeply coupled and used everywhere, so lazy loading wasn’t practical.
Decouple configuration-dependent modules:

This would only help for modules that didn’t need the config, but most did.
Optimize load times:

We were already transitioning to Vite, but the bottleneck was the unavoidable JSON.parse on a huge file.
Cache heavy operations between reloads:

This seemed the most promising: minimal effort, maximum impact.

But how do you cache variables between restarts?

Webpack HMR and why it did not work for me

Webpack implements its own module system, enabling Hot Module Replacement (HMR) in both browser and Node.js environments. In development, you can use module.hot to keep state between reloads:

const expensiveVariable = module.hot?.data?.expensiveVariable ?? createExpensiveVariable();
if (module.hot) {
  module.hot.dispose((data) => {
    data. expensiveVariable = expensiveVariable;
  });
}

However, this requires compiling your app with Webpack, which is slow and complex for large backends. I deemed this completely unnecessary for my plain JavaScript backend. Other tools like Babel or SWC don’t provide HMR for Node.js.

The CommonJS Trick: Keeping Variables Between Restarts

Since my app was written in CommonJS, I could leverage Node’s require.cache API. This cache tracks all loaded modules. By clearing the cache for everything except a special “keeper” module, I could persist variables across reloads. This is not possible with ES Modules, as they don’t expose a public cache API.

The Hot-Reload Entrypoint

To make this work, I needed a new entrypoint that:

Watches source files for changes
Clears the require cache for changed files
Restarts the application in the same process

Killing and respawning a child process wouldn’t work, since each process has its own memory and cache. Instead, the entrypoint must control the server and be able to close and restart it in-place.

Here’s a simple implementation:

const fs = require('fs');
const http = require('http');
const { createRequire } = require('module');
const chokidar = require('chokidar');

const requireApp = createRequire(__filename);
let server = null;

function startServer() {
  if (server) server.close();
  const app = requireApp('./app.js');
  server = http.createServer(app).listen(3000);
}

chokidar.watch('./src').on('change', (file) => {
  Object.keys(require.cache).forEach((key) => {
    if (key.includes('/src/')) delete require.cache[key];
  });
  startServer();
});

startServer();

You could optimize further by only clearing changed modules and their dependents, but this is more complex and not always faster.

The Variable Cache Module

To persist variables, I created a special module that is never cleared from the cache. It simply exports keep and kept functions:

// keeper.js
const store = new Map();

function keep(name, value) {
  store.set(name, value);
  return value;
}

function kept(name, defaultValue) {
  return store.has(name) ? store.get(name) : defaultValue;
}

module.exports = { keep, kept };

In the app, it is used like this:

const { keep, kept } = require('./keeper');

// On startup, restore or create the expensive resource
let expensiveResource = kept('expensiveResource', null);
if (!expensiveResource) {
  expensiveResource = createExpensiveResource();
  keep('expensiveResource', expensiveResource);
}

Now, when the app reloads, the resource is kept alive!

Forcibly Closing Connections

One last challenge: some middlewares (like webpack-dev-middleware or webpack-hot-middleware) keep connections open, preventing the server from closing. To fix this, the entrypoint must forcibly close all open sockets:

const sockets = new Set();

server.on('connection', (socket) => {
  sockets.add(socket);
  socket.on('close', () => sockets.delete(socket));
});

function closeServer() {
  for (const socket of sockets) {
    socket.destroy();
  }
  server.close();
}

Probably those middlewares should handle close calls themselves and destroy active connections, but they did not.

The Result

With this setup, I can now restart my server almost instantly after a file change, instead of waiting 90 seconds or more. The feedback loop is tight, and development isn't painful anymore.

Introducing: `hot-keeper`

I’ve packaged this approach into a reusable CLI utility: hot-keeper.

Install it from npm:

pnpm add hot-keeper

Use it as your entrypoint:

pnpm hot-keeper app.js

And in your app, use the cache helpers:

const { keep, kept } = require('hot-keeper');

let expensive = kept('expensive', null);
if (!expensive) {
  expensive = createExpensive();
  keep('expensive', expensive);
}

For further configuration and more examples, see the README.

Now you can enjoy fast reloads regardless of setup routines in your Node.js CommonJS projects!

Webpack On-Demand Compilation: How to Speed up Dev mode of Legacy Webpack Applications

Alexey Yakovlev — Mon, 20 Nov 2023 14:00:50 +0000

Disclaimer: Due to the extensible and configurable nature of Webpack this article will likely not supply you with a ready-made solution to introduce on-demand compilation to Webpack. But it will present a few examples and give a general idea of how your implementation could look like.

If you are not interested in the supplementary introduction, you can skip right to the implementation part.

Introduction

According to the State of JS Survey, Webpack remains the most used Build Tool in the JavaScript Ecosystem as of the end of 2022. However, comments from respondents suggest that while Webpack is crucial to their build pipelines, its performance and general developer experience are among the most disliked aspects of using it. Some users also express frustration with the lack of relevant examples available online - which I am trying to fix right now. Still, for many, switching to a different build tool is not an option.

I have personally experienced this situation myself. In the project I work on, we have a large monolithic application consisting of 12,700 modules, all compiled using a single Webpack configuration. Unfortunately, the application, especially crucial parts like Server Side Rendering, is tightly coupled to the way Webpack works, making a complete rewrite too expensive at the moment.

Despite these challenges, the application serves a large business that needs to continue growing. However, local development of new features is slow and cumbersome, with most developers taking a coffee break while waiting for the application to boot up. Hot reload is also slow, to the point I would no longer consider it "hot". While we have implemented newer transpilers and minifiers like ESBuild and SWC to replace the outdated Babel and Terser, the performance improvements have not been significant enough to eliminate these issues.

At some point, I couldn't tolerate this any longer and started to explore other ways to improve performance. Given that our application consists of multiple entrypoints or "bundles," it made sense to attempt to compile only the necessary entrypoints when demanded by the user (developer). This concept is known as on-demand compilation. However, a search for "Webpack on-demand compilation" on the web returned very few relevant results.

But amidst the list of largely irrelevant results, I stumbled upon a hidden gem - a Merge Request to the internal GitLab Repo introducing the so-called "incremental compiler." Don't let the name deceive you - it is actually about on-demand compilation. Incremental compilation is a different technique that allows the compiler to rebuild only the code that has changed. Webpack already incorporates this technique, as every hot reload does not take the same amount of time as a complete build. We can use this Merge Request as inspiration for our own on-demand compilation implementation.

How to introduce on-demand compilation to Webpack

Before we dive deeper, let's provide a small recap for those who skipped the introduction.

Incremental compilation is a technique that enables the compiler to only recompile the parts of the code that have changed during rebuilds. This feature is supported by default in Webpack's watch mode, though the term is not explicitly mentioned in the documentation.

On-demand compilation is a technique that allows the compiler to only compile those parts of the code that are needed for certain functionality to work. In Webpack it is not supported by default. To implement on-demand compilation, multiple entrypoints (webpack.entry) are recommended. This allows us to specify which parts of the code need to be compiled. It should be noted that using a single entrypoint makes it difficult, but not impossible, to determine the necessary code and configure Webpack accordingly.

Our goal is to introduce on-demand compilation into our Webpack configuration. We will achieve this by demanding entrypoints that correspond to different pages within our application. Therefore, when working locally, you will only need to wait for the compilation of the pages that you are currently working on during the session. To illustrate, a sample Webpack configuration could resemble the following:

// Pages or larger parts of the application
const entries = ['index', 'catalog', 'profile', 'settings', 'search'];

const IS_PRODUCTION = process.env.NODE_ENV === 'production';

/**
 * Takes the entries and reduces them into key-value object with path to entrypoint
 * { "index": ["src/pages/index/client"]}
 */
function generateEntries() {
    if (IS_PRODUCTION) {
        return Object.values(entries).reduce((generatedEntries, entry) => {
            generatedEntries[entry] = path.join(__dirname, `./src/pages/${entry}/client`);
            return generatedEntries;
        }, {});
    }

    return Object.values(entries).reduce((generatedEntries, entry) => {
        generatedEntries[entry] = [
            'webpack-hot-middleware/client',
            path.join(__dirname, `./src/pages/${entry}/client`),
        ];
        return generatedEntries;
    }, {});
}

const config = {
    mode: IS_PRODUCTION ? 'production' : 'development',
    entry: generateEntries,
    output: {
        // every entrypoint converted into a separate bundle
        path: path.join(__dirname, './build'),
        filename: '[name].build.js',
        chunkFilename: '[name].bundle.js',
    },
    resolve: {
        plugins: [new TsconfigPathsPlugin({})],
        extensions: ['.js', '.jsx', '.ts', '.tsx', '.json'],
    },
    module: {
        rules: [
            {
                test: /\.(js|ts|jsx|tsx)$/,
                loader: 'esbuild-loader',
                options: {
                    loader: 'tsx',
                    target: 'es2015',
                },
            },
            {
                test: /\.css$/,
                sideEffects: true,
                use: [
                    'style-loader',
                    'css-loader',
                ],
            },
        ],
    },
    optimization: {
        minimize: IS_PRODUCTION,
    },
    plugins: [],
    target: 'web',
    devtool: false,
};

module.exports = config;

The sample configuration might or might not be valid - this is irrelevant to the topic.

Another crucial component in the puzzle is webpack-dev-middleware. Whether you have your own Node.js server or you're using webpack-dev-server for local development, both setups utilize webpack-dev-middleware.

webpack-dev-middleware provides access to an invalidate callback. This callback is responsible for invalidating the current Webpack configuration and initiating recompilation with the updated one. Consequently, the generateEntries function, defined in the configuration, is invoked again, allowing us to modify its results.

So the general idea is the following: use the development server to intercept requests and identify which pages need to be compiled. Store the list of compiled pages in a Set and invalidate webpack-dev-middleware. Hook into the generateEntries function and return only the entries that are present in the Set.

It is a great idea to incapsulate the logic for storing and filtering demanded entrypoints in a separate class which could look like this:

/**
 * Singleton which stores demanded entrypointsa nd filters those
 */
class WebpackOnDemandCompiler {
    #demandedEntries = new Set([]);

    static #instance = null;

    /**
     * @returns {WebpackOnDemandCompiler}
     */
    static getInstance() {
        if (!this.#instance) {
            this.#instance = new WebpackOnDemandCompiler();
        }
        return this.#instance;
    }

    /**
     * Should be invoked before returning value to `webpack.entries`
     * @typedef {string | string[]} Paths
     * @typedef {Record<string, Paths>} Entries
     * @param {Entries} entries
     */
    filterEntries(entries) {
        const filteredEntries = Object.entries(entries).reduce((acc, [page, paths]) => {
            if (this.#demandedEntries.has(page)) {
                acc[page] = paths;
            } else {
                // in some configurations integration with dev server will not work
                // unless entry is present from start
                // the file is empty so it does not add overhead
                acc[page] = "./empty-file.js"
            }

            return acc;
        }, {});


        return filteredEntries;
    }

    /**
     * Should be invoked when a new entry is requested
     * (e.g. `res.render` call or specific requests to `webpack-dev-server`)
     */
    handleEntry(entry, devMiddleware) {
        if (!this.#demandedEntries.has(entry)) {
            this.#demandedEntries.add(entry);

            devMiddleware.invalidate();
        }
    }
}

module.exports = WebpackOnDemandCompiler;

It makes sense for such a class to be a singleton - so that the state is shared across integrations with dev server and inside Webpack configuration.

Modified generateEntries should look like this:

const WebpackOnDemandCompiler = require("./webpack-on-demand-compiler");

const entries = ["index", "catalog", "profile", "settings", "search"];

function generateEntries() {
  if (IS_PRODUCTION) {
    return Object.values(entries).reduce((generatedEntries, entry) => {
      generatedEntries[entry] = path.join(
        __dirname,
        `./src/pages/${entry}/client`
      );
      return generatedEntries;
    }, {});
  }

  const allEntries = Object.values(entries).reduce(
    (generatedEntries, entry) => {
      generatedEntries[entry] = [
        "webpack-hot-middleware/client",
        path.join(__dirname, `./src/pages/${entry}/client`),
      ];
      return generatedEntries;
    },
    {}
  );

  const onDemandCompiler = WebpackOnDemandCompiler.getInstance();

  return onDemandCompiler.filterEntries(allEntries);
}

The only thing left is to integrate with the "demand" of the pages.

Integrating with `webpack-dev-server`

When integrating with webpack-dev-server you can use this Merge Request as inspiration.

Use devServer Webpack config property and its before hook. In the hook add a middleware that filters out requests made for entrypoints and invokes WebpackOnDemandCompiler.handleEntry providing it with the entry and server.middleware instance. The implementation could look like this:

const WebpackOnDemandCompiler = require("./webpack-on-demand-compiler");

const entries = ["index", "catalog", "profile", "settings", "search"];

const config = {
  // ...
  devServer: {
    before(app, server) {
      app.use((req, res, next) => {
        const fileName = path.basename(req.url);
        // {entryName}.build.js
        const entryName = fileName.split('.')[0];

        /**
         * Here you should filter only page entries
         * 
         *
         * Be sure to filter out hot update files that are for example named "{entryName}.[hash].hot-update.js"
         */
        if (entries.some(entry => entry === entryName) && fileName.endsWith('.build.js')) {
          const onDemandCompiler = WebpackOnDemandCompiler.getInstance();
          // pass `webpack-dev-middleware` instance
          onDemandCompiler.handleEntry(entryName, server.middleware);
        }

        next();
      });
    }
  }
};

With this code in place you should be good to go!

Integrating with a custom server

With a custom server we have to make a few assumptions on how the server actually works. In this text I will assume that the server uses webpack-dev-middleware and res.render callback to render pages (or something similar - depending on your library or framework).

While a central location like render function yields an easy integration with on-demand compilation. You could use a less straightforward approach with, for example, endpoints being assigned a page and triggering on-demand compilation when those endpoints are invoked.

Given the assumption mentioned earlier, res.render becomes perfect place to intercept requests made to pages and process them using WebpackOnDemandCompilation:

const onDemandCompiler = WebpackOnDemandCompiler.getInstance();

// Make sure to only apply this middleware in dev mode and after `webpack-dev-middleware`
const onDemandCompilerMiddleware = (req, res, next) => {
  const baseRender = res.render;

  res.render = (entry, ...args) => {
    onDemandCompiler.handleEntry(entry, res.locals.webpack.devMiddleware);
    return baseRender(entry, ...args);
  };
  next();
};

Once you have implemented this code, you should notice a significant decrease in startup times (initially, only empty files will be compiled). When you request a new page, it will still take some time for compilation, but it will likely be much faster compared to performing a complete build during startup. Additionally, hot reload should also be quicker now!

It is important to keep in mind that as you navigate through your application, you will be requesting more and more pages. At a certain point, this can lead to a decrease in performance and a return to the pre on-demand compilation levels. To address this issue, you can implement a limit on the number of entrypoints that can be compiled at once. When this limit is reached, the least recently requested entrypoint should be removed from the list. By doing this, you can ensure that your development mode experience remains fast and efficient, regardless of the length of your session.

Conclusion

Due to the extensibility of Webpack very little information on the web about Webpack is actually relevant to a specific case. I tried to keep the guide as configuration-agnostic as possible - and I sure hope it was helpful in your specific case!

If you have any corrections or suggestions you can leave them in the comments. And I am also very interested in your both success and failure stories on speeding up Webpack and migrating from it.

How to create a Keycloak plugin

Alexey Yakovlev — Mon, 31 Oct 2022 12:27:57 +0000

Introduction

Keycloak is an open-source Identity and Access Management solution. I find it very useful for small to large teams that need to implement their own authentication system but do not have the capacity to develop a secure service themselves. It is written in Java and offers an SPI (Service Provider Interface). This means that it is easily extendable with custom implementations of existing classes and new additions via plugins.

So how do you create a plugin? In this tutorial I would like to give an example of how to develop and test a Keycloak plugin.

I should note that, while every users needs are different, it is highly likely that the plugin you want to develop already exists in this brilliant repo by user thomasdarimont. Make sure to take a look and at least get an inspiration.

In this tutorial we will be developing a plugin that would authenticate users based on a link sent to their email. At the time of writing there are no examples like this present in the forementioned repository. If you are looking for the complete project, visit my GitHub - https://github.com/yakovlev-alexey/keycloak-email-link-auth. Commit history roughly follows this tutorial.

Introduction
Table of Contents
Initialize the project
Create a test bench
Implement a custom Authenticator
- Custom Action Token
- Authenticator implementation
- Custom resources for plugins
- Notify Keycloak of new Providers
Configure test bench
Authenticator configuration
Next steps
Conclusion

Initialize the project

First let's create a Maven project. I prefer doing that with a shell command but you may be using your favourite IDE for the same result.



mvn archetype:generate -DgroupId=dev.yakovlev_alexey -DartifactId=keycloak-email-link-auth-plugin -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

Make sure to enter your own groupId and artifactId.

This would generate a structure like this



.
├── pom.xml
└── src
    ├── main
    │   └── java
    │       └── dev
    │           └── yakovlev_alexey
    │               └── App.java
    └── test
        └── java
            └── dev
                └── yakovlev_alexey
                    └── AppTest.java

At this time we are not going to create unit tests although it is entirely possible. So let's remove the test folder and App.java file. Now we need to install required dependencies to develop our plugin. Make the following modifications to your pom.xml



    <properties>
        <!-- other properties -->
        <keycloak.version>19.0.3</keycloak.version>
    </properties>

    <dependencies>
        <!-- generated by maven - may be removed if you do not plan to unit test -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.11</version>
            <scope>test</scope>
        </dependency>

        <!-- HERE GO KEYCLAOK DEPENDENCIES -->
        <dependency>
            <groupId>org.keycloak</groupId>
            <artifactId>keycloak-server-spi-private</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.keycloak</groupId>
            <artifactId>keycloak-server-spi</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.keycloak</groupId>
            <artifactId>keycloak-services</artifactId>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.keycloak</groupId>
                <artifactId>keycloak-parent</artifactId>
                <version>${keycloak.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

All keycloak dependencies are specified as provided. According to documentation this means that Maven will expect the runtime (JDK or in our case Keycloak) to provide those dependencies. This is exactly what we need. Still we need a centralised way to manage Keycloak dependency versions. For this we use dependencyManagement property. Make sure to run mvn install to update your dependencies.

Now we are ready to develop our plugin though it would make sense to first create a test bench where we would be able to quickly test our changes locally without modifiying your actual Keycloak instance or even deploying it.

Create a test bench

Let's create a test bench using Docker and docker-compose. I recommend using docker-compose from the start since it's highly likely that your plugin is going to have some external dependency like an SMTP server. Using docker-compose would allow you to later effortlessly add other services to create a truly isolated environment. I use the following docker-compose.yaml:



# ./docker/docker-compose.yaml
version: "3.2"

services:
    keycloak:
        build:
            context: ./
            dockerfile: Dockerfile
            args:
                - KEYCLOAK_IMAGE=${KEYCLOAK_IMAGE}
        environment:
            KEYCLOAK_ADMIN: admin
            KEYCLOAK_ADMIN_PASSWORD: admin
            DB_VENDOR: h2
        volumes:
            - ./h2:/opt/keycloak/data/h2
        ports:
            - "8024:8080"

And the following Dockerfile:



# ./docker/Dockerfile
ARG KEYCLOAK_IMAGE

FROM $KEYCLOAK_IMAGE

USER root
COPY plugins/*.jar /opt/keycloak/providers/
USER 1000

ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start-dev"]

In order for docker-compose to work we need a few environment variables. Specify them in .env file.



<!-- ./docker/.env -->
COMPOSE_PROJECT_NAME=keycloak
KEYCLOAK_IMAGE=quay.io/keycloak/keycloak:19.0.3

Create plugins folder inside docker directory. This is where jar files with compiled plugins will go.



docker
├── .env
├── Dockerfile
├── docker-compose.yaml
├── h2
└── plugins
    └── .gitkeep

Make sure to ignore h2 folder and plugins contents in your VCS. Your .gitignore might look something like this:



# ./.gitignore
target
docker/h2

docker/plugins/*
!docker/plugins/.gitkeep

Finally you can run docker-compose up --build keycloak in your docker folder to start your test instance of Keycloak.

Implement a custom Authenticator

Now back to our goal. We want to authenticate users based on the link we send to their email. To do that we implement a custom Authenticator. Authenticator is basically a step in the process of authenticating a user. It may be anything from a form that requires input from user to complex redirect.

In order to get an idea how to create an authenticator (or any other class you might need that leverages SPI for that matter) I recommend taking a look at Keycloak source code hosted on GitHub - https://github.com/keycloak/keycloak. As you might expect Keycloak has a very large codebase therefore it would be easier to use GitHub search to find the class you might need. In our case you may find authenticators directory interesting - https://github.com/keycloak/keycloak/tree/main/services/src/main/java/org/keycloak/authentication/authenticators. I will not go into detail about existing Keycloak authenticators and their implementations and rather start scaffolding our own authenticator.

Another thing I recommend is to follow the same folder structure as Keycloak. Therefore our plugin should look something like this:



src
└── main
    └── java
        └── dev
            └── yakovlev_alexey
                └── keycloak
                    └── authentication
                        └── authenticators
                            └── browser
                                └── EmailLinkAuthenticator.java

Create class EmailLinkAuthenticator and implement Authenticator interface from org.keycloak.authentication.Authenticator. After stubbing required methods your code could look like this:



package dev.yakovlev_alexey.keycloak.authentication.authenticators.browser;

import org.keycloak.authentication.AuthenticationFlowContext;
import org.keycloak.authentication.Authenticator;
import org.keycloak.models.KeycloakSession;
import org.keycloak.models.RealmModel;
import org.keycloak.models.UserModel;

public class EmailLinkAuthenticator implements Authenticator {

    @Override
    public void close() {
        // TODO Auto-generated method stub
    }

    @Override
    public void action(AuthenticationFlowContext context) {
        // TODO Auto-generated method stub
    }

    @Override
    public void authenticate(AuthenticationFlowContext context) {
        // TODO Auto-generated method stub
    }

    @Override
    public boolean configuredFor(KeycloakSession session, RealmModel realm, UserModel user) {
        // TODO Auto-generated method stub
        return false;
    }

    @Override
    public boolean requiresUser() {
        // TODO Auto-generated method stub
        return false;
    }

    @Override
    public void setRequiredActions(KeycloakSession session, RealmModel realm, UserModel user) {
        // TODO Auto-generated method stub
    }

}

There is a bunch of methods but we will need only some of them. Important ones are authenticate which is called when user enters this Authenticator in the Authentication flow and action which is called when user submits the form.

To start with we may specify that this Authenticator requires a user. This basically means that at the point of reaching this authenticator we should already have an idea who the user may want to authenticate as. In simpler terms prior to reaching our authenticator the user should enter their username. In order to show that to Keycloak just return true from requiresUser method.

configuredFor method allows us to tell Keycloak whether we are able to authenticate the user in a certain context. Context being the realm with its settings, Keycloak session and the user being authenticated. In our case we only need the user to have an email. So let's return user.getEmail() != null from configuredFor.

Now it is time to implement the actual logic for this authenticator. We need it to do a few things:

When the authenticator gets called the first time, send an email with a link and then show a "email sent" page
Page should have a button to resend the email (in case delivery fails)
Page should have a button to submit the verification (e.g. I confirmed the verification in a different browser/device and want to continue in this tab)

The closest authenticator to ours is VerifyEmail required action. While it is not really an authenticator it is very similar. Looking at the implementation we see that in order to send an email it creates a token and encodes it in an action token URL.



private Response sendVerifyEmail(KeycloakSession session, LoginFormsProvider forms, UserModel user, AuthenticationSessionModel authSession, EventBuilder event) throws UriBuilderException, IllegalArgumentException {
    RealmModel realm = session.getContext().getRealm();
    UriInfo uriInfo = session.getContext().getUri();

    int validityInSecs = realm.getActionTokenGeneratedByUserLifespan(VerifyEmailActionToken.TOKEN_TYPE);
    int absoluteExpirationInSecs = Time.currentTime() + validityInSecs;

    String authSessionEncodedId = AuthenticationSessionCompoundId.fromAuthSession(authSession).getEncodedId();
    VerifyEmailActionToken token = new VerifyEmailActionToken(user.getId(), absoluteExpirationInSecs, authSessionEncodedId, user.getEmail(), authSession.getClient().getClientId());
    UriBuilder builder = Urls.actionTokenBuilder(uriInfo.getBaseUri(), token.serialize(session, realm, uriInfo),
            authSession.getClient().getClientId(), authSession.getTabId());
    String link = builder.build(realm.getName()).toString();
    long expirationInMinutes = TimeUnit.SECONDS.toMinutes(validityInSecs);

    try {
        session
            .getProvider(EmailTemplateProvider.class)
            .setAuthenticationSession(authSession)
            .setRealm(realm)
            .setUser(user)
            .sendVerifyEmail(link, expirationInMinutes);
        event.success();
    } catch (EmailException e) {
        logger.error("Failed to send verification email", e);
        event.error(Errors.EMAIL_SEND_FAILED);
    }

    return forms.createResponse(UserModel.RequiredAction.VERIFY_EMAIL);
}

Custom Action Token

I think it makes sense to copy the implementation except our parameters would be slightly different to account for different contexts in authenticators and requried actions. But to properly use this implementation we need an action token of our own. Action token is a class that represents a JWT. It extends DefaultActionToken class and has a corresponding ActionTokenHandler class. Tokens can be encoded into a URL to which action token handler will respond. Let's create our own action token by copying VerifyEmail one.

EmailLinkActionToken should be located in keycloak.authentication.actiontoken.emaillink package. Your folder structure should look like the following one:



src
└── main
    └── java
        └── dev
            └── yakovlev_alexey
                └── keycloak
                    └── authentication
                        ├── actiontoken
                        │   └── emaillink
                        │       ├── EmailLinkActionToken.java
                        │       └── EmailLinkActionTokenHandler.java
                        └── authenticators
                            └── browser
                                └── EmailLinkAuthenticator.java

Let's implement EmailLinkActionToken by copying VerifyEmailActionToken and replacing the name and removing originalAuthenticationSessionId:



package dev.yakovlev_alexey.keycloak.authentication.actiontoken.emaillink;

import org.keycloak.authentication.actiontoken.DefaultActionToken;

public class EmailLinkActionToken extends DefaultActionToken {
    public static final String TOKEN_TYPE = "email-link";

    public EmailLinkActionToken(String userId, int absoluteExpirationInSecs, String compoundAuthenticationSessionId,
            String email, String clientId) {
        super(userId, TOKEN_TYPE, absoluteExpirationInSecs, null, compoundAuthenticationSessionId);
        this.issuedFor = clientId;

        setEmail(email);
    }

    private EmailLinkActionToken() {
    }
}

As you might notice I removed originalAuthenticationSessionId. It is used to create a new authenticaton session by reissuing the token and using its link as the submit action to confirm sign up in a different browser.

Next let's implement the handler for it. This time just copying will not work: contexts are very different. First let's settle on an implementation where visiting the link immediately gives the user consent (unlike the VerifyEmail one where manually clicking a button is required). After visiting the link an info page will be shown. To achieve this behaviour we will roughly follow the same procedure as in VerifyEmailActionTokenHandler except I will try to comment important parts since the code is not as easy to understand.



package dev.yakovlev_alexey.keycloak.authentication.actiontoken.emaillink;

import org.keycloak.authentication.actiontoken.AbstractActionTokenHandler;
import org.keycloak.TokenVerifier.Predicate;
import org.keycloak.authentication.actiontoken.*;
import org.keycloak.events.*;
import org.keycloak.forms.login.LoginFormsProvider;
import org.keycloak.models.ClientModel;
import org.keycloak.models.KeycloakSession;
import org.keycloak.models.RealmModel;
import org.keycloak.models.UserModel;
import org.keycloak.services.managers.AuthenticationManager;
import org.keycloak.services.managers.AuthenticationSessionManager;
import org.keycloak.services.messages.Messages;
import org.keycloak.sessions.AuthenticationSessionCompoundId;
import org.keycloak.sessions.AuthenticationSessionModel;

import dev.yakovlev_alexey.keycloak.authentication.authenticators.browser.EmailLinkAuthenticator;

import java.util.Collections;

import javax.ws.rs.core.Response;

public class EmailLinkActionTokenHandler extends AbstractActionTokenHandler<EmailLinkActionToken> {

    public EmailLinkActionTokenHandler() {
        super(
                EmailLinkActionToken.TOKEN_TYPE,
                EmailLinkActionToken.class,
                Messages.STALE_VERIFY_EMAIL_LINK,
                EventType.VERIFY_EMAIL,
                Errors.INVALID_TOKEN);
    }

    @Override
    public Predicate<? super EmailLinkActionToken>[] getVerifiers(
            ActionTokenContext<EmailLinkActionToken> tokenContext) {
        // this is different to VerifyEmailActionTokenHandler implementation because
        // since its implementation a helper was added
        return TokenUtils.predicates(verifyEmail(tokenContext));
    }

    @Override
    public Response handleToken(EmailLinkActionToken token, ActionTokenContext<EmailLinkActionToken> tokenContext) {
        AuthenticationSessionModel authSession = tokenContext.getAuthenticationSession();
        UserModel user = authSession.getAuthenticatedUser();
        KeycloakSession session = tokenContext.getSession();
        EventBuilder event = tokenContext.getEvent();
        RealmModel realm = tokenContext.getRealm();

        event.event(EventType.VERIFY_EMAIL)
                .detail(Details.EMAIL, user.getEmail())
                .success();

        // verify user email as we know it is valid as this entry point would never have
        // gotten here
        user.setEmailVerified(true);

        // fresh auth session means that the link was open in a different browser window
        // or device
        if (!tokenContext.isAuthenticationSessionFresh()) {
            // link was opened in the same browser session (session is not fresh) - save the
            // user a click and continue authentication in the new (current) tab
            // previous tab will be thrown away
            String nextAction = AuthenticationManager.nextRequiredAction(tokenContext.getSession(), authSession,
                    tokenContext.getRequest(), tokenContext.getEvent());
            return AuthenticationManager.redirectToRequiredActions(tokenContext.getSession(), tokenContext.getRealm(),
                    authSession, tokenContext.getUriInfo(), nextAction);
        }

        AuthenticationSessionCompoundId compoundId = AuthenticationSessionCompoundId
                .encoded(token.getCompoundAuthenticationSessionId());

        AuthenticationSessionManager asm = new AuthenticationSessionManager(session);
        asm.removeAuthenticationSession(realm, authSession, true);

        ClientModel originalClient = realm.getClientById(compoundId.getClientUUID());
        // find the original authentication session
        // (where the tab is waiting to confirm)
        authSession = asm.getAuthenticationSessionByIdAndClient(realm, compoundId.getRootSessionId(),
                originalClient, compoundId.getTabId());

        if (authSession != null) {
            authSession.setAuthNote(EmailLinkAuthenticator.EMAIL_LINK_VERIFIED, user.getEmail());
        } else {
            // if no session was found in the same instance it might still be in the same
            // cluster if you have multiple replicas of Keycloak
            session.authenticationSessions().updateNonlocalSessionAuthNotes(
                    compoundId,
                    Collections.singletonMap(EmailLinkAuthenticator.EMAIL_LINK_VERIFIED,
                            token.getEmail()));
        }

        // show success page
        return session.getProvider(LoginFormsProvider.class)
                .setAuthenticationSession(authSession)
                .setSuccess(Messages.EMAIL_VERIFIED, token.getEmail())
                .createInfoPage();
    }

    // we do not really want users to authenticate using the same link multiple times
    @Override
    public boolean canUseTokenRepeatedly(EmailLinkActionToken token,
            ActionTokenContext<EmailLinkActionToken> tokenContext) {
        return false;
    }
}

To practice a little yourself try to reimplement this action token and handler to require additional confirmation (like it works with VerifyEmailActionToken).

Authenticator implementation

With action token completed we may continue implementing EmailLinkAuthenticator. We will use VerifyEmail.sendVerifyEmailEvent as reference but once again since context is very different we will change a lot of things.



    private static final Logger logger = Logger.getLogger(EmailLinkAuthenticator.class);

    protected void sendVerifyEmail(KeycloakSession session, AuthenticationFlowContext context, UserModel user)
            throws UriBuilderException, IllegalArgumentException {
        AuthenticationSessionModel authSession = context.getAuthenticationSession();
        RealmModel realm = session.getContext().getRealm();

        // use the same lifespan as other tokens by getting from realm configuration
        int validityInSecs = realm.getActionTokenGeneratedByUserLifespan(EmailLinkActionToken.TOKEN_TYPE);
        long expirationInMinutes = TimeUnit.SECONDS.toMinutes(validityInSecs);

        String link = buildEmailLink(session, context, user, validityInSecs);

        // event is used to achieve better observability over what happens in Keycloak
        EventBuilder event = getSendVerifyEmailEvent(context, user);

        Map<String, Object> attributes = getMessageAttributes(user, realm.getDisplayName(), link, expirationInMinutes);

        try {
            session.getProvider(EmailTemplateProvider.class)
                    .setRealm(realm)
                    .setUser(user)
                    .setAuthenticationSession(authSession)
                    // hard-code some of the variables - we will return here later
                    .send("emailLinkSubject", "email-link-email.ftl", attributes);

            event.success();
        } catch (EmailException e) {
            logger.error("Failed to send verification email", e);
            event.error(Errors.EMAIL_SEND_FAILED);
        }

        showEmailSentPage(context, user);
    }

    /**
     * Generates an action token link by encoding `EmailLinkActionToken` with user
     * and session data
     */
    protected String buildEmailLink(KeycloakSession session, AuthenticationFlowContext context, UserModel user,
            int validityInSecs) {
        AuthenticationSessionModel authSession = context.getAuthenticationSession();
        RealmModel realm = session.getContext().getRealm();
        UriInfo uriInfo = session.getContext().getUri();

        int absoluteExpirationInSecs = Time.currentTime() + validityInSecs;

        String authSessionEncodedId = AuthenticationSessionCompoundId.fromAuthSession(authSession).getEncodedId();
        EmailLinkActionToken token = new EmailLinkActionToken(user.getId(), absoluteExpirationInSecs,
                authSessionEncodedId, user.getEmail(), authSession.getClient().getClientId());
        UriBuilder builder = Urls.actionTokenBuilder(uriInfo.getBaseUri(), token.serialize(session, realm, uriInfo),
                authSession.getClient().getClientId(), authSession.getTabId());
        String link = builder.build(realm.getName()).toString();

        return link;
    }

    /**
     * Creates a Map with context required to render email message
     */
    protected Map<String, Object> getMessageAttributes(UserModel user, String realmName, String link,
            long expirationInMinutes) {
        Map<String, Object> attributes = new HashMap<>();

        attributes.put("user", user);
        attributes.put("realmName", realmName);
        attributes.put("link", link);
        attributes.put("expirationInMinutes", expirationInMinutes);

        return attributes;
    }

    /**
     * Creates a builder for `SEND_VERIFY_EMAIL` event
     */
    protected EventBuilder getSendVerifyEmailEvent(AuthenticationFlowContext context, UserModel user) {
        AuthenticationSessionModel authSession = context.getAuthenticationSession();

        EventBuilder event = context.getEvent().clone().event(EventType.SEND_VERIFY_EMAIL)
                .user(user)
                .detail(Details.USERNAME, user.getUsername())
                .detail(Details.EMAIL, user.getEmail())
                .detail(Details.CODE_ID, authSession.getParentSession().getId())
                .removeDetail(Details.AUTH_METHOD)
                .removeDetail(Details.AUTH_TYPE);

        return event;
    }

    /**
     * Displays email link form
     */
    protected void showEmailSentPage(AuthenticationFlowContext context, UserModel user) {
        String accessCode = context.generateAccessCode();
        URI action = context.getActionUrl(accessCode);

        Response challenge = context.form()
                .setStatus(Response.Status.OK)
                .setActionUri(action)
                .setExecution(context.getExecution().getId())
                .createForm("email-link-form.ftl");

        context.forceChallenge(challenge);
    }

This implementation does a few things:

Creates an event that will allow you to collect data on usage and debug if needed
Generates a link to an action token
Sends an email with a few attributes including the user and the link
Shows a form that allows you to resend the email or confirm that you verified your authentication via link

I took the liberty to hard-code a few variables temporarily - we will return to replace them with actual constants or configruation parameters. This includes "emailLinkSubject" being the message id for email subject and email and form templates: "email-link-email.ftl", "email-link-form.ftl". Messages and templates are stored in resources directory in Keycloak. We will put them there later.

Now let's implement the most important methods: action and authenticate.



    @Override
    public void authenticate(AuthenticationFlowContext context) {
        // the method gets called when first reaching this authenticator and after page
        // refreshes
        AuthenticationSessionModel authSession = context.getAuthenticationSession();
        KeycloakSession session = context.getSession();
        RealmModel realm = context.getRealm();
        UserModel user = context.getUser();

        // cant really do anything without smtp server
        if (realm.getSmtpConfig().isEmpty()) {
            ServicesLogger.LOGGER.smtpNotConfigured();
            context.attempted();
            return;
        }

        // if email was verified allow the user to continue
        if (Objects.equals(authSession.getAuthNote(EMAIL_LINK_VERIFIED), user.getEmail())) {
            context.success();
            return;
        }

        // do not allow resending e-mail by simple page refresh
        if (!Objects.equals(authSession.getAuthNote(Constants.VERIFY_EMAIL_KEY), user.getEmail())) {
            authSession.setAuthNote(Constants.VERIFY_EMAIL_KEY, user.getEmail());
            sendVerifyEmail(session, context, user);
        } else {
            showEmailSentPage(context, user);
        }
    }

    @Override
    public void action(AuthenticationFlowContext context) {
        // this method gets called when user submits the form
        AuthenticationSessionModel authSession = context.getAuthenticationSession();
        KeycloakSession session = context.getSession();
        UserModel user = context.getUser();

        // if link was already open continue authentication
        if (Objects.equals(authSession.getAuthNote(EMAIL_LINK_VERIFIED), user.getEmail())) {
            context.success();
            return;
        }

        MultivaluedMap<String, String> formData = context.getHttpRequest().getDecodedFormParameters();
        String action = formData.getFirst("submitAction");

        // if the form was submitted with an action of `resend` resend the email
        // otherwise just show the same page
        if (action != null && action.equals("resend")) {
            sendVerifyEmail(session, context, user);
        } else {
            showEmailSentPage(context, user);
        }
    }

The last thing we want to do with our authenticator class is create a factory for it. Keycloak uses factories to instantiate providers. For our action token handler factory was implemented in the base class. So it is entirely possible to implement both the provider and the factory in the same class. In this case we will implement it in a different class EmailLinkAuthenticatorFactory.



package dev.yakovlev_alexey.keycloak.authentication.authenticators.browser;

import org.keycloak.Config;
import org.keycloak.authentication.Authenticator;
import org.keycloak.authentication.AuthenticatorFactory;
import org.keycloak.models.AuthenticationExecutionModel;
import org.keycloak.models.KeycloakSession;
import org.keycloak.models.KeycloakSessionFactory;
import org.keycloak.provider.ProviderConfigProperty;

import java.util.List;

public class EmailLinkAuthenticatorFactory implements AuthenticatorFactory {
    public static final EmailLinkAuthenticator SINGLETON = new EmailLinkAuthenticator();

    @Override
    public String getId() {
        return "email-link-authenticator";
    }

    @Override
    public String getDisplayType() {
        return "Email Link Authentication";
    }

    @Override
    public String getHelpText() {
        return "Authenticates the user with a link sent to their email";
    }

    @Override
    public String getReferenceCategory() {
        return null;
    }

    @Override
    public boolean isConfigurable() {
        return false;
    }

    @Override
    public boolean isUserSetupAllowed() {
        return false;
    }

    @Override
    public AuthenticationExecutionModel.Requirement[] getRequirementChoices() {
        return new AuthenticationExecutionModel.Requirement[] {
                AuthenticationExecutionModel.Requirement.REQUIRED,
                AuthenticationExecutionModel.Requirement.DISABLED,
        };
    }

    @Override
    public List<ProviderConfigProperty> getConfigProperties() {
        return null;
    }

    @Override
    public void init(Config.Scope config) {
    }

    @Override
    public void postInit(KeycloakSessionFactory factory) {
    }

    @Override
    public void close() {
    }

    @Override
    public Authenticator create(KeycloakSession session) {
        // a common pattern in Keycloak codebase is to use singletons for factories
        return SINGLETON;
    }
}

At the moment we do not allow any configuration for our authenticator. Also our authenticator is either required or disabled. This means that it is not possible to bypass this authenticator by not having an email. However if this is what you want you may make this authenticator optional. You would also need to call context.attempted() in authenticate method in the authenticator class.

Another option to allow emailless users to skip this authenticator is to create another conditional authenticator - however this is out of scope for this tutorial.

Custom resources for plugins

We now have all the code we need for desired functionality to work. However we still miss the form and messages to display. To add resources to our plugins let's create resources directory in src/main. In it create a subdirectory theme-resources. It is used by Kecyloak to import templates and messages.

Messages should be stored in messages folder in a .properties file named like messages_{language}.properties.

Templates similarly are stored in templates folder. Email message templates are split into html and text subfolders. Some email clients may not render HTML - text version will be used then. Form templates should be stored at the root of templates.

Other resources are stored similarly stored in css, js, img directories. Read more about in official docs.



src
└── main
    ├── java
    └── resources
        └── theme-resources
            ├── messages
            │   └── messages_en.properties
            └── templates
                ├── html
                │ └── email-link-email.ftl
                ├── text
                │ └── email-link-email.ftl
                └── email-link-form.ftl

Keycloak uses FreeMaker to store and render templates. Read more about how Keycloak manages its themes in the official documentation.

To implement the form in email-link-form.ftl for our plugin we will use the following template:



<#import "template.ftl" as layout>
<@layout.registrationLayout; section>
    <#if section = "header">
        ${msg("emailLinkTitle")}
    <#elseif section = "form">
        <form id="kc-register-form" action="${url.loginAction}" method="post">
            <div class="${properties.kcFormGroupClass!}">
                <button type="submit" class="${properties.kcButtonClass!} ${properties.kcButtonDefaultClass!} ${properties.kcButtonBlockClass!} ${properties.kcButtonLargeClass!}" name="submitAction" id="confirm" value="confirm">${msg("emailLinkCheck")}</button>
                <button type="submit" class="${properties.kcButtonClass!} ${properties.kcButtonDefaultClass!} ${properties.kcButtonBlockClass!} ${properties.kcButtonLargeClass!}" name="submitAction" id="resend" value="resend">${msg("emailLinkResend")}</button>
            </div>
        </form>
    </#if>
</@layout.registrationLayout>

msg function allows you to place localized strings from resources/messages. They will be substituted by Keycloak during rendering. Otherwise the syntax is pure FreeMaker.

You can see our submit buttons have a name and a value. When you submit an HTML form via a button with a value this value is appended to form data sent to the server. This is what we were using in action handler to make sure we only resend email when user actually asks for it by clicking the button.

As for emails templates look a little different. HTML version in email-link-email.ftl look like this:



<html>
<body>
${kcSanitize(msg("emailLinkEmailBodyHtml",link, expirationInMinutes, realmName))?no_esc}
</body>
</html>

Here we just use a message for all contents of our email. We also pass variables that would be substitued into the message, more on that later. no_esc allows us to render HTML as HTML and not as text. And kcSanitize is needed to ensure no dangerous markup ends up in the resulting document.

The text version of the same email look like this:



<#ftl output_format="plainText">
${msg("emailLinkEmailBody",link, expirationInMinutes, realmName)}

Here we explicitely tell that we want a plain text document and not an HTML file. Otherwise it is basically the same except we do not want HTML here therefore do not call neither kcSanitize nor no_esc.

At this point we have all the templates except we do not have any of the messages our templates (and Providers) use. Our messages_en.properties file should look something like this:



emailLinkSubject=Your authentication link

emailLinkResend=Resend email
emailLinkCheck=I followed the link

emailLinkEmailBody=Your authentication link is {0}.\nIt will be valid for {1} minutes. Follow it to authenticate in {2} and then return to the original browser tab.
emailLinkEmailBodyHtml=<a href="{0}">Click here to authenticate</a>. The link will be valid for {1} minutes. Follow it to authenticate in {2} and then return to the original browser tab.

You can see its just a file containing all the messages we need. If you specify any of the messages that already exist in Keycloak, they will not get replaced. To replace existing messages use themes. Some messages may contain HTML markup. But as I already said they need to be properly sanitized and unescape to be used.

Notify Keycloak of new Providers

At this point we have the code and the resources for our plugin. However if we build it, put into Keycloak plugins folder and run Keycloak nothing will happen. Keycloak will not know what to do with our code. To tell Keycloak that we want it to inject certain classes we have to add certain meta information to our built jar file. For this use resources/META-INF directory.



src
└── main
    ├── java
    └── resources
        ├── theme-resources
        └── META-INF
            └── services
                ├── org.keycloak.authentication.actiontoken.ActionTokenHandlerFactory
                └── org.keycloak.authentication.AuthenticatorFactory

Here we created 2 files in services subdirectory. Each file in this directory should be named after a base interface or class that our own classes implement or extend.

In each file we need to specify all classes of our own that derivate from the filename class on separate lines.



# org.keycloak.authentication.actiontoken.ActionTokenHandlerFactory
dev.yakovlev_alexey.keycloak.authentication.actiontoken.emaillink.EmailLinkActionTokenHandler

# org.keycloak.authentication.AuthenticatorFactory
dev.yakovlev_alexey.keycloak.authentication.authenticators.browser.EmailLinkAuthenticatorFactory

Finally we can build our project by running mvn clean package. A file with the name of keycloak-email-link-auth-plugin-1.0-SNAPSHOT.jar should be created in target folder. Copy this file to docker/plugins directory.

Configure test bench

We are almost ready to test our plugin. However in order to properly do this we would need an SMTP server since our plugin sends emails. You could use some real SMTP server. But that would likely cost money and you will have to use a real email. It is much easier to use a mail trap service like MailHog.

Add it as service to docker-compose.yaml



version: "3.2"

services:
    keycloak:
        build:
            context: ./
            dockerfile: Dockerfile
            args:
                - KEYCLOAK_IMAGE=${KEYCLOAK_IMAGE}
        environment:
            KEYCLOAK_ADMIN: admin
            KEYCLOAK_ADMIN_PASSWORD: admin
            DB_VENDOR: h2
        volumes:
            - ./h2:/opt/keycloak/data/h2
        ports:
            - "8024:8080"

    mailhog:
        image: mailhog/mailhog
        ports:
            - 1025:1025
            - 8025:8025

Port 1025 is used for SMTP and 8025 for web UI/API.

Now you can run Docker using docker-compose up --build keycloak. Visit http://localhost:8024 and enter administration console. Username and password should both be admin as stated in the yaml.

To use our authenticator you need to configure an authenticator flow that uses it. Flows are sequences of authenticators. By default browser authentication flow is used. Go ahead and copy it as browser-email in Authentication tab.

Replace Username Password Form with Username Form. Since we want to authenticate based on the link sent to the email we do not really want password from our users. Though it is also possible to keep both password and emails to implement a sort of 2 factor authentication.

After Username Form add our Email Link Authentication and make it Required.

Next configure SMTP server in Realm Settings -> Email. Specify any From value. Host for the connection should be mailhog and port 1025. No SSL or authentication is used.

After you saved Email configuration run Test Connection. Email should be sent successfully and you should see a test message in MailHog web UI at http://localhost:8025/.

Make sure your admin user has a valid email (not necessarily one that you have access to) in Users tab.

Finally enter Clients, find account-console and in Advanced tab configure Authentication flow overrides. Browser flow should be overriden with browser-email. This will allow you to easily test your plugin by entering http://localhost:8024/realms/master/account/ and not break the admin panel if your changes do not work.

Everything is set up and you can now visit http://localhost:8024/realms/master/account/ and enter admin as your username. You should see the form we implemented earlier and an email should be visible in MailHog UI at http://localhost:8025/.

Visit the link from the message.

You should see the link has been verified.

Return to the original tab and voila! You should be authenticated now.

Authenticator configuration

You may remember that we hardcoded some of the variables that should either be configurable or at least stored separately. Let's go back and fix that.

Create a folder to store all utility classes for our authenticator.



src
└── main
    └── java
        └── dev
            └── yakovlev_alexey
                └── keycloak
                    └── authentication
                    │ ├── actiontoken
                    │ └── authenticators
                    └── emaillink
                        ├── ConfigurationProperties.java
                        ├── Constants.java
                        └── Messages.java

ConfigurationProperties.java is a class with constants to enable configuration for our authenticator. Constants.java stores generic variable that do not need to be configurable. And Messages.java contains all message string keys our plugin needs. Some developers put there only the keys that Java code needs but I prefer to use Messages class as a list for all strings (even if they are only used in templates) so that it is easy to add localized versions (e.g. messages_de.properties file).

Let's implement ConfigurationProperties class:



package dev.yakovlev_alexey.keycloak.emaillink;

import org.keycloak.provider.ProviderConfigProperty;

import java.util.Arrays;
import java.util.List;

import static org.keycloak.provider.ProviderConfigProperty.*;

public final class ConfigurationProperties {
    public static final String EMAIL_TEMPLATE = "EMAIL_TEMPLATE";
    public static final String PAGE_TEMPLATE = "PAGE_TEMPLATE";

    public static final String RESEND_ACTION = "RESEND_ACTION";

    public static final List<ProviderConfigProperty> PROPERTIES = Arrays.asList(
            new ProviderConfigProperty(EMAIL_TEMPLATE,
                    "FTL email template name",
                    "Will be used as the template for emails with the link",
                    STRING_TYPE, "email-link-email.ftl"),
            new ProviderConfigProperty(PAGE_TEMPLATE,
                    "FTL page template name",
                    "Will be used as the template for email link page",
                    STRING_TYPE, "email-link-form.ftl"),
            new ProviderConfigProperty(RESEND_ACTION,
                    "Resend Email Link action",
                    "Action which corresponds to user manually asking to resend email with link",
                    STRING_TYPE, "resend"));

    private ConfigurationProperties() {
    }
}

And Constants class:



package dev.yakovlev_alexey.keycloak.emaillink;

public final class Constants {
    public static final String EMAIL_LINK_SUBMIT_ACTION_KEY = "submitAction";

    private Constants() {
    }
}

And finally Messages class:



package dev.yakovlev_alexey.keycloak.emaillink;

public final class Messages {
    public static final String EMAIL_LINK_SUBJECT = "emailLinkSubject";
    public static final String EMAIL_LINK_STALE = "emailLinkStale";

    public static final String EMAIL_LINK_SUCCESS = "emailLinkSuccess";

    public static final String EMAIL_LINK_TITLE = "emailLinkTitle";
    public static final String EMAIL_LINK_RESEND = "emailLinkResend";
    public static final String EMAIL_LINK_CHECK = "emailLinkCheck";

    public static final String EMAIL_LINK_EMAIL_BODY = "emailLinkEmailBody";
    public static final String EMAIL_LINK_EMAIL_BODY_HTML = "emailLinkEmailBodyHtml";

    private Messages() {
    }
}

A few changes in EmailLinkAuthenticatorFactory are needed to make it configurable:



    @Override
    public boolean isConfigurable() {
        return true;
    }

    @Override
    public List<ProviderConfigProperty> getConfigProperties() {
        return ConfigurationProperties.PROPERTIES;
    }

Now update EmailLinkAuthenticator class to leverage new constants:



    // ***

    @Override
    public void action(AuthenticationFlowContext context) {
        // this method gets called when user submits the form
        AuthenticationSessionModel authSession = context.getAuthenticationSession();
        AuthenticatorConfigModel config = context.getAuthenticatorConfig();
        KeycloakSession session = context.getSession();
        UserModel user = context.getUser();

        // if link was already open continue authentication
        if (Objects.equals(authSession.getAuthNote(EMAIL_LINK_VERIFIED), user.getEmail())) {
            context.success();
            return;
        }

        MultivaluedMap<String, String> formData = context.getHttpRequest().getDecodedFormParameters();
        String action = formData
                .getFirst(dev.yakovlev_alexey.keycloak.emaillink.Constants.EMAIL_LINK_SUBMIT_ACTION_KEY);

        // if the form was submitted with an action of `resend` resend the email
        // otherwise just show the same page
        if (action != null && action.equals(config.getConfig().get(ConfigurationProperties.RESEND_ACTION))) {
            sendVerifyEmail(session, context, user);
        } else {
            showEmailSentPage(context, user);
        }
    }

    // ***

    private void sendVerifyEmail(KeycloakSession session, AuthenticationFlowContext context, UserModel user)
            throws UriBuilderException, IllegalArgumentException {
        AuthenticationSessionModel authSession = context.getAuthenticationSession();
        AuthenticatorConfigModel config = context.getAuthenticatorConfig();
        RealmModel realm = session.getContext().getRealm();

        // use the same lifespan as other tokens by getting from realm configuration
        int validityInSecs = realm.getActionTokenGeneratedByUserLifespan(EmailLinkActionToken.TOKEN_TYPE);
        long expirationInMinutes = TimeUnit.SECONDS.toMinutes(validityInSecs);

        String link = buildEmailLink(session, context, user, validityInSecs);

        // event is used to achieve better observability over what happens in Keycloak
        EventBuilder event = getSendVerifyEmailEvent(context, user);

        Map<String, Object> attributes = getMessageAttributes(user, realm.getDisplayName(), link, expirationInMinutes);

        try {
            session.getProvider(EmailTemplateProvider.class)
                    .setRealm(realm)
                    .setUser(user)
                    .setAuthenticationSession(authSession)
                    // hard-code some of the variables - we will return here later
                    .send(config.getConfig().get(Messages.EMAIL_LINK_SUBJECT),
                            config.getConfig().get((ConfigurationProperties.EMAIL_TEMPLATE)), attributes);

            event.success();
        } catch (EmailException e) {
            logger.error("Failed to send verification email", e);
            event.error(Errors.EMAIL_SEND_FAILED);
        }

        showEmailSentPage(context, user);
    }


    // ***

    /**
     * Displays email link form
     */
    protected void showEmailSentPage(AuthenticationFlowContext context, UserModel user) {
        AuthenticatorConfigModel config = context.getAuthenticatorConfig();
        String accessCode = context.generateAccessCode();
        URI action = context.getActionUrl(accessCode);

        Response challenge = context.form()
                .setStatus(Response.Status.OK)
                .setActionUri(action)
                .setExecution(context.getExecution().getId())
                .createForm(config.getConfig().get(ConfigurationProperties.PAGE_TEMPLATE));

        context.forceChallenge(challenge);
    }

You can access authenticator configuration via context.getAuthenticatorConfig(). Constants overlaps with Keycloak Constants class used in the same file therefore to avoid name clashes full package name is specified here.

Now when you enter Authentication tab and edit browser-email flow you should see a gear button beside Email Likn Authentication that will open settings for this authenticator.

In Keycloak 19 there is a bug in interface that does not allow opening settings for authenticators. Refer to this issue.

Next steps

To make sure you get a good grasp of how to develop Keycloak plugins I recommend you do a few "homework" tasks yourself. There are some things that could be improved about this plugin that I already mentioned.

Implement a configuration option that would enable a confirmation screen when you open the link. This screen should have a "confirm" button. Only after clicking this button you may continue in your original tab. If link is open in the same browser window user should be immideately authenticated as it works now.
Create a separate plugin that implements a condition authenticator HasEmailCondition. You may use ConditionalUserAttributeValue as reference. Use both plugins in a test bench to configure a flow where users with password set up are authenticated via password and others are required to follow email link.

Conclusion

I hope this tutorial gave you an idea how to create Keycloak plugins. There is much more to it, many internal SPIs could be implemented to achieve different scenarios. My goal was to focus on the basics: plugin composition, resources imports and getting inspiration from Keycloak source code. If you have any questions, ideas or suggestions make sure to leave them in GitHub issues.

Find the complete code on my GitHub - https://github.com/yakovlev-alexey/keycloak-email-link-auth.

Creating an interactive SVG map on the web with D3

Alexey Yakovlev — Thu, 06 Oct 2022 21:01:43 +0000

I was recently approached with a task I was unfamiliar with. It involved painting a SVG map on a webpage with clickable points of interest. The problem was that the user should be able to zoom and pan across the map. I found the information on the internet on this topic rather scarce. Hence I am here, telling you how I solved my problem. If you are the type of person looking for the complete code, see Conclusion.

For the tutorial I recommend using CodePen where the complete solution is hosted or a similar service since we are going to make a very small iterative project. We are not going to be using any frameworks or languages other than Babel to transpile the code.

Basic zooming and panning

First lets add basic HTML layout

<div class="container">
  <svg id="map" width="100%" height="100%" style="background-color:red">
    <image id="image" href="https://upload.wikimedia.org/wikipedia/commons/1/1e/Alexander_III_conquest_from_Issos_to_Babylon-fr.svg">
  </svg>
</div>

And style it a litte

* {
  margin: 0;
  padding: 0;
}

.container {
  width: 100%;
  height: calc(100vh);

  overflow: hidden;
}

What we see is a map of Alexander III conquest from Issos to Babylon in French. However we see only a part of the map and on larger screens on the side we get a large portion of red SVG background sticking out. What we want is an experience where the user is able to pan and zoom across the map. However they should not be allowed to get out of the map bounds.

In order to achieve this we need to use D3 library. It includes a lot of utility methods and modules that help you create charts, tables, maps and other interactive experiences on the web using JavaScript and HTML. Many higher level libraries use it as the groundwork. I will not go into detail about library API but rather leave relevant links where possible.

import * as d3 from "https://cdn.skypack.dev/d3@7.6.1";

// create a D3 selection of the SVG itself - https://github.com/d3/d3-selection
const svg = d3.select("#map");
// create a D3 selection of the image element
const image = svg.selectChild("#image");

// create and configure an instance of zoom behaviour - https://github.com/d3/d3-zoom
const zoom = d3.zoom().on("zoom", zoomed);

// apply configured zoom behaviour to our svg
svg.call(zoom);

function zoomed(event) {
  const { transform } = event;
  // apply calculated transform to the image
  image.attr("transform", transform.toString());
}

Now when we run the code we should be able to pan across the image. But at the moment we are not bound to the image limits and may go outside of the bounds infinitely. To fix this we should use extent configuration for the zoom behaviour.

Scrolling to zoom inside the CodePen iframe seems to not be working on dev.to. Please open the CodePen in a different tab to see it properly.

const { width, height } = image.node().getBoundingClientRect();
const { width: svgWidth, height: svgHeight } = svg
  .node()
  .getBoundingClientRect();

const zoom = d3
  .zoom()
  // scale extent is how much you can zoom into or out of the image - https://github.com/d3/d3-zoom#zoom_scaleExtent
  .scaleExtent([1, 8])
  // extent is mostly used to calculate things and make them smooth during zooming and panning - https://github.com/d3/d3-zoom#zoom_extent
  // by default it is the viewbox or width and height of the nearest SVG ancestor - this works for us
  .extent([
    [0, 0],
    [svgWidth, svgHeight],
  ])
  // translate extent is optional and is used to bound the viewport to the image - https://github.com/d3/d3-zoom#zoom_translateExtent
  .translateExtent([
    [0, 0],
    [width, height],
  ])
  .on("zoom", zoomed);

Now we are unable to pan outside the image. However we still see red background on the sides. This is clearly due to the fact that we are zoomed out too far. To fix this we need to make our scaleExtent more accurate.

You may experiment and see that if we remove scaleExtent we would be able to zoom out far enough to fit the whole image in the viewport. In this case translateExtent would not allow us to pan at all since the actual extent of the translate is already reached.

// minimum scale is the largest ratio between the container dimension and the image dimension
const minScale = Math.max(svgWidth / width, svgHeight / height);

const zoom = d3
  .zoom()
  // scale extent is how much you can zoom into or out of the image
  .scaleExtent([minScale, 8])
  .extent([
    [0, 0],
    [svgWidth, svgHeight],
  ])
  .translateExtent([
    [0, 0],
    [width, height],
  ])
  .on("zoom", zoomed);

// apply calculated default scale
zoom.scaleTo(svg, minScale);

At this point it may become clear that I chose a fairly poor image as an example since it too small to really justify the ability to zoom. But you get the point nevertheless!

Congrats! We got what we wanted to achieve. However there is still a thing we need check. What if we want to zoom and pan across multiple elements? Like an image with a few circles over it?

Groups and interactive elements

Let's add a red circle over our image.

<div class="container">
  <svg id="map" width="100%" height="100%" style="background-color:red">
    <image
      id="image"
      href="https://upload.wikimedia.org/wikipedia/commons/1/1e/Alexander_III_conquest_from_Issos_to_Babylon-fr.svg"
    />
    <circle fill="red" cx="50" cy="50" r="25" />
  </svg>
</div>

Now when we zoom and pan the image the circle always stays at the same place. In our case this circle may mark a place of a significant battle and we want it to be on the map. To fix this we should wrap our image with the circle inside a group g.

<div class="container">
  <svg id="map" width="100%" height="100%" style="background-color:red">
    <!-- id="image" was moved to the group so that our JS code still works without changes -->
    <g id="image">
      <image
        href="https://upload.wikimedia.org/wikipedia/commons/1/1e/Alexander_III_conquest_from_Issos_to_Babylon-fr.svg"
      />
      <circle fill="red" cx="50" cy="50" r="25" />
    </g>
  </svg>
</div>

Now when we pan and zoom the image the circle stays at the same place on the map. Brilliant!

Window resizes

Still, there is a bug left. When you resize the window our scaling and panning brakes! To fix this we need to listen to resize events on window and update our extent properties. this is how our code would look like:

import * as d3 from "https://cdn.skypack.dev/d3@7.6.1";

// create a D3 selection of the SVG itself
const svg = d3.select("#map");
// create a D3 selection of the image element
const image = svg.selectChild("#image");

// calculate image dimensions once - they should not change
const { width, height } = image.node().getBoundingClientRect();

// ===========================================

const zoom = d3.zoom().on("zoom", zoomed);

function updateExtents() {
  const { width: svgWidth, height: svgHeight } = svg
    .node()
    .getBoundingClientRect();

  const minScale = Math.max(svgWidth / width, svgHeight / height);

  zoom
    // scale extent is how much you can zoom into or out of the image
    .scaleExtent([minScale, 8])
    // extent is mostly used to calculate things and make them smooth during zooming and panning
    // by default it is the viewbox or width and height of the nearest SVG ancestor - this works for us
    .extent([
      [0, 0],
      [svgWidth, svgHeight],
    ])
    // translate extent is optional and is used to bound the viewport to the image
    .translateExtent([
      [0, 0],
      [width, height],
    ]);

  // apply calculated default scale
  zoom.scaleTo(svg, minScale);
}

// ===========================================

// apply configured zoom behaviour to our svg
svg.call(zoom);

updateExtents();
window.addEventListener("resize", updateExtents);

function zoomed(event) {
  const { transform } = event;
  // apply calculated transform to the image
  image.attr("transform", transform.toString());
}

Now resize your window and see that after a resize our translate and scale bounds still work properly.

Minimap

I think it would be quite interesting to implement a read-only minimap using the knowledge we just gained. Let's duplicate our image into a preview group outside the image (so that id does not scale or pan).

<div class="container">
  <svg id="map" width="100%" height="100%" style="background-color:red">
    <g id="image">
      <image
        href="https://upload.wikimedia.org/wikipedia/commons/1/1e/Alexander_III_conquest_from_Issos_to_Babylon-fr.svg"
      />
      <circle fill="red" cx="50" cy="50" r="25" />
    </g>

    <g id="preview" transform="translate(20,20) scale(0.1)">
      <image
        href="https://upload.wikimedia.org/wikipedia/commons/1/1e/Alexander_III_conquest_from_Issos_to_Babylon-fr.svg"
      />
      <circle fill="red" cx="50" cy="50" r="25" />
      <!-- here the minimap viewport will go -->
    </g>
  </svg>
</div>

Now we need to update the code to inject a viewport rectangle into the preview box.

I recommend you try think of a solution yourself before you go ahead!

const preview = svg.select("#preview");

const rect = preview
  .append("rect")
  .style("fill-opacity", "0")
  .style("stroke", "red")
  .style("stroke-width", "3px");

function updateViewport() {
  const { width: svgWidth, height: svgHeight } = svg
    .node()
    .getBoundingClientRect();

  rect.attr("width", svgWidth).attr("height", svgHeight);
}

// ===========================================

// apply configured zoom behaviour to our svg
svg.call(zoom);

updateExtents();
updateViewport();

window.addEventListener("resize", updateExtents);
window.addEventListener("resize", updateViewport);

Do not forget to update the width and height of the viewport on window resizes!

Next update the zoomed function to transform both the image and the viewport. It is important to note that the transform we apply in this function is destined for the image. So it does not scale the viewport but the content. The trick is to invert the scale since the amount that the viewport scales is inversely proportial to how the content scales.

function zoomed(event) {
  const { transform } = event;
  // apply calculated transform to the image
  image.attr("transform", transform.toString());

  const scale = 1 / transform.k;

  const inverseTransform = new d3.ZoomTransform(
    scale,
    -transform.x * scale,
    -transform.y * scale
  );

  rect.attr("transform", inverseTransform.toString());
}

Profit! Now you have a minimap for your SVG map. Still we can go event further! How about allowing the user to click on a location on the minimap to jump to it?

Clickable minimap

To achieve that add a listener to click event on preview group. When clicked, calculate the position from the top left corner of the minimap. Then divide it by the scale of the minimap to get the position on the actual map. Finally, use this position to move the viewport.

Since we use the translateTo method it would not allow the user to clip out of bounds using the minimap since this method respects extent properties.

preview.on("click", handleMinimapClick);

const previewRect = preview.node().getBoundingClientRect();

// this scale is taken from HTML transform property above
const PREVIEW_SCALE = 0.1;

function handleMinimapClick(event) {
  const dx = event.clientX - previewRect.x,
    dy = event.clientY - previewRect.y;

  const position = [dx / PREVIEW_SCALE, dy / PREVIEW_SCALE];

  zoom.translateTo(svg, ...position);
}

Check it out! And you can still use the minimap pan around the actual map. If you do not want that call image.call(zoom) instead of svg.call(zoom). Then zoom will only listen to clicking and dragging over the image but not the minimap.

Conclusion

Hope you found this tutorial useful! If you did, your like on the post would make my day, and if you did not I would appreciate your comment. I should note that you can use any SVG markup as the image in this tutorial. So it is entirely and easily possible to have different points of interest on the image that you might make interactive with popups and etc.

Find the code on CodePen - https://codepen.io/yakovlev-alexey/pen/MWGqKER

Do not read DDD Distilled by Vaughn Vernon

Alexey Yakovlev — Sun, 13 Feb 2022 14:50:36 +0000

Domain-Driven Design (DDD) is a software design approach that focuses on modelling software to match a domain according to input from that domain's experts. It emerged in late 90s and started to gain traction in the first decade of the 21st century. Lately many software development companies strive to implement it in their processes. But how should you as a developer get acquainted with it?

Eric Evans "Domain-Driven Design: Tackling Complexity in the Heart of Software" 2003 book is considered to be the book about DDD (also known as the blue book). However it is quite repulsive with its big price and reputation for being cumbersome to read. So how could you as a developer get a grasp of what DDD is without investing a lot of time?

You may be tempted to read the green book "Domain-Driven Design Distilled" by Vaughn Vernon. It is advertised as an accessible guide to the main principles of DDD. I fell for the trap, read it and found a lot of thing wrong with it. And I am really keen to tell about it and suggest a better alternative.

Why you should not read Domain-Driven Design Distilled

You would think that a small "distilled" version of a book about DDD should focus on creating a picture of what Domain-Driven Design is. But in fact I found myself puzzled about what it actually is after finishing this book.

The picture I got was a middle managers look on DDD: it's another practice that involves moving stickers on the wall until developers get at least some understanding of what the business wants. The book went into very little detail explaining the building blocks of a software system designed in domain-driven manner. And what it seems like no detail on how actual domain concepts should map into the software.

Instead I found myself reading multiple paragraphs on what colours and sizes your stickers should be which is bizarre to me. And there were only what felt like a few words on what aggregates and its elements are. Do not get me wrong: explaining how to employ event stormings to start using DDD is important but it should not focus on sticker colours this heavily.

Another thing in the book I found counterintuitive and unhelpful is the path it takes to explain DDD. It starts from scalability with multiple subdomains which you might not even have in your domain. And for the first 50 or so pages you see words like aggregates, events, entities that you can only guess the meaning of. I believe that those concepts are crucial in the understanding of scalability mechanisms in DDD and should always come before those.

Still the book is not useless. Perhaps if you already have some understanding of many DDD concepts and only need to bind them together this book will be largely helpful. And obviously it is good at what it really is: selling Domain-Driven Design to managers. But for practicing developers with a goal to understand how to use and implement Domain-Driven Design it is certainly a bad choice.

What you should read instead

There is actually a somewhat hidden gem in the world of books about DDD. "Domain-Driven Design Quickly" by Abel Avram and Floyd Marinescu is a free (if we are talking about the PDF/epub version) book that brilliantly summarises Eric Evans' blue book.

It follows a predictable path of explaining DDD with concrete examples which builds a real picture of a project build with DDD. It also gives the understanding of hexagonal architecture that usually comes with DDD.

To me this book felt like the actual "distilled" version of DDD. Still it only paints a picture with the theory on DDD. I felt like I still required more practical examples to really finish digesting all the information I recieved. And there is no better way to do this than to look at a real project template built with DDD and to read a few articles with different opinions and points of view on implementing DDD.

For this there is an absolutely brilliant repository by Sairyss on GitHub. domain-driven-hexagon properly implements hexagonal architecture with some of more advanced practices in TypeScript and Nest.js. Most aspects of DDD are explained in a simple language with an example in the code and links to relevant articles. Despite the fact that it is written in TypeScript you will still largely benefit from the information displayed in the repository and most likely interpret code examples well enough even if you are not familiar with JavaScript.

So if you as a developer find yourself interested in Domain-Driven Design but not invested enough to read Eric Evans' blue book you should start with "Domain-Driven Design Quickly" by Abel Avram and Floyd Marinescu and then see some practical examples in domain-driven-hexagon to really build an understanding of DDD. I hope this article saved you from investing your precious time into an unworthy book and paved a better path of getting to know DDD.

Creating a Miro-like UI overlay layout using CSS

Alexey Yakovlev — Fri, 28 Jan 2022 09:10:38 +0000

When creating highly interactive websites like Miro (or Google Maps, Drawio, etc) with a drawing board or some other interactive experience occupying most of the screen developers are challenged with implementing responsive overlays. Such overlays add necessary information and tools to interact with the product.

The most obvious way to implement such an overlay would be to use a combination of absolute or fixed positioning - which could work. But it would involve a lot of magic numbers with positioning and would not be very responsive - adding a new item would involve a lot of changes and perhaps even some JavaScript statements.

But there are other ways to implement an overlay like this without any JavaScript and with as little of absolute positioning as possible.

I will be using codepen.io to prototype this layout. I find this application very useful when discovering different CSS layout implementation options. Find the pen with this tutorial here.

Test bench

First let's create a bench where we may try out different ways of implementing such a layout and test whether our implementation works properly.

Some text and a button should suffice. We should also make sure we can scroll the container both ways.

<div class="wrapper">
  <div class="overlay">
  </div>

  <div class="content">
    <p> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
    </p>

    <button>
      I understand
    </button>
  </div>
</div>

And add some CSS to make it a little nicer.

* {
  margin: 0;
  padding: 0;
}

.wrapper {
  position: relative;

  width: 100%;
  min-width: 350vw;
  min-height: 350vh;
}

.content {
  max-width: 500px;
  padding: 20px;

  margin-top: 150px;
  margin-left: 500px;
}

.content > :not(:first-child) {
  margin-top: 10px;
}

button {
  cursor: pointer;

  padding: 8px;

  box-shadow: none;

  border: none;
  border-radius: 4px;
  background: #dedede;

  color: black;
}

button:hover {
  box-shadow: 0 0 4px #000;
}

We will use div.overlay as the container for all our overlays. At the moment it is an empty div - we will fill it with content now.

Implementing the overlay

Looking at the overlay of Miro we may imagine the container for all the UI items as a vertical flexbox. Informational messages may be appended at the top and other items would adapt accordingly. Let's make it in a similar way.

  <div class="overlay">
    <header class="overlay-row">
      <div class="widget" style="width: 320px">
        <button>
          Exit or something
        </button>
      </div>

      <div class="widget">
        <button>
          Open knowledge base or something
        </button>
      </div>
    </header>

    <div class="overlay-row toolbar-row">
      <div class="widget" style="height: 240px; width: 32px">

      </div>
    </div>

    <footer class="overlay-row">
      <div class="widget" style="width: 320px">
        <button>
          Change view or something
        </button>
      </div>

      <div class="widget">
        <button>
          Go fullscreen or something
        </button>
      </div>
    </footer>
  </div>

And some CSS to position everything.

.overlay {
  position: fixed;
  top: 0;
  left: 0;

  display: flex;
  flex-flow: column nowrap;

  width: 100vw;
  height: 100vh;
  padding: 20px;
}

.overlay-row {
  display: flex;
  flex-flow: row nowrap;
  align-items: center;
  justify-content: space-between;

  width: 100%;
}

.widget {
  display: flex;
  flex-flow: row nowrap;
  align-items: center;

  padding: 4px;

  background: #fff;
  border-radius: 4px;
  box-shadow: 0 0 2px rgba(0, 0, 0, 0.5);
}

.toolbar-row {
  flex: 1;
}

The widget class will serve as the base container for every piece of overlay UI - we may expand it later. UI elements are placed inside rows with specific purposes.

The UI looks fine - it is positioned well and you can click UI buttons. However when it comes to the content it gets worse. We can't interact with the button inside the content and we can't select any text. Obviously the div.overlay we fixed over the div.wrapper blocks us from accessing underlying elements.

To fix this we will use a clever hack. We will move the overlay outside the visible area of the screen using transform. This will clear the underlying elements - we will once again be able to click the content button and select text. Though UI widgets would not be visible. widget class will also recieve a transform property to translate back to the original position.

.overlay {
  position: fixed;
  top: 0;
  left: 0;

  display: flex;
  flex-flow: column nowrap;

  width: 100vw;
  height: 100vh;
  padding: 20px;

  transform: translateX(-100vw);
}

.widget {
  display: flex;
  flex-flow: row nowrap;
  align-items: center;

  padding: 4px;

  background: #fff;
  border-radius: 4px;
  box-shadow: 0 0 2px rgba(0, 0, 0, 0.5);

  transform: translateX(100vw);
}

You may wonder: why don't we apply transform: translateX(100vw) to overlay-row class? The answer is simple - rows still would obstruct the user from clicking the content since they also occupy the whole screen with their boxes.

Now we still see the UI widgets where we should see them and can still click the content. Great success!

But what about scrolling?

Also works as intended! Turns out it is this simple to develop an overlay like Miro's.

It is important to note however that it would still likely require some JavaScript to collapse some of the widgets or buttons inside them for smaller screens. But this topic is out of scope for this post.

At this point it should be visible that the code for an overlay like this is far simpler and shorter compared to absolutely positioning every widget. But how easy is it to add new elements?

Extending the overlay

Let's imagine our Product Owner approaches us asking to add a banner at the top of the screen with some very useful information that he believes every user should see when entering your application. Being professional developers we discussed everything with them and started implementing this banner.

  <div class="overlay">
    <div class="banner">
      Some very useful information that every user should see clearly
    </div>

    <div class="overlay-content">
      <header class="overlay-row">
        <div class="widget" style="width: 320px">
          <button>
            Exit or something
          </button>
        </div>

        <div class="widget">
          <button>
            Open knowledge base or something
          </button>
        </div>
      </header>

      <div class="overlay-row toolbar-row">
        <div class="widget" style="height: 240px; width: 32px">

        </div>
      </div>

      <footer class="overlay-row">
        <div class="widget" style="width: 320px">
          <button>
            Change view or something
          </button>
        </div>

        <div class="widget">
          <button>
            Go fullscreen or something
          </button>
        </div>
      </footer>
    </div>

.overlay-content {
  display: flex;
  flex-flow: column nowrap;
  flex: 1;

  padding: 20px;
}

.banner {
  padding: 4px 8px;

  font-size: 18px;

  background: rgba(66, 98, 255, 1);

  transform: translateX(100vw);

  color: white;
}

As you can see wee added a wrapper for all the widgets so that it is easier to add more outside banners. The layout will automatically adapt accordingly.

It is also extremely easy to hide the banner: just set display property to none and the widgets would snap back to place.

Another advantage you may get with this layout is when you embed it in a container - you may easily change div.overlay to use sticky positioning. This will allow some external static elements on the page.

Conclusion

Hopefully you found this post interesting and useful. I remind you that the complete tutorial is available here.

This overlay implementation is very different to what you usually find websites using. But it offers very predictable adaptivity using just CSS and with very little magical numbers and unintuitive styles. You may also use grid layouts for even more intricate widget locations.

Perhaps you have something to say about this implementation. Maybe you do not agree with me and have reasons why you think having every widget absolutely positioned is less challenging? Feel free to discuss below.

How I improved a Node web application perfomance multifold

Alexey Yakovlev — Fri, 21 Jan 2022 11:39:58 +0000

This post is a continuation of Nest.js + Next.js series and it fixes a big issue I left in the code in the second article. While I will be using the project from those articles this article is not specific neither to Nest.js nor to Next.js. The goal of this article is to tell a story how I managed to improve a Node application perfomance multifold and share my experience on how to profile complex perfomance issues and make sure they do not return.

Table of Contents
Introduction
Emulating environment
- Containerizing application
- Generating load
- Narrowing down the issue
Profiling a remote Node.js application
- Attaching Node.js inspector
- Reading profiles
- Benching improvements
Lessons learned

Introduction

I was assigned to refactor bits of server-side rendering pipelines in a project using Nest.js + Next.js combo. Bridge implementation is very similar to the one I left in the articles mentioned earlier. I was still discovering all the quirks with this technology and it was the time I discovered that undefined values can not be serialized by Next.js. To fix this issue I introduced a sanitizing function. Inspired by JavaScripts functional code style and immutability the function was recursive and did not modify any values, only copied where needed.

This solution worked fine when the project was small. But the application was growing and we started transferring rather large objects with a lot of properties and deep nesting. Later we discovered a huge increase in FCP times in our user monitorings.

As the person with the largest amount of experience with nest-next in the team I went on a quest to investigate and find the source of issues. At this point I obviously forgot about the potential ineffectiveness of a function I wrote earlier. I also did not pay enough attention to the amount of data we started transferring.

I will show how I approached and overcome this challenge and what lessons I learned.

We will start by emulating a deployed environment. Developers strive to make different environments including the local one as similar as possible. Unfortunately often it is not possible. To make sure that I make proper inspections I had to analyze a deployed environment and not a local one. And so we will do in this article. Kind of.

Emulating environment

To emulate a remote server with SSH connection we will develop a Dockerfile with connection instructions. Then we will synthesize load in the application from Nest.js + Next.js series. With the load in place we may start inspecting our application to narrow down what causes the issue. Use this revision to work along or check complete code on my GitHub - https://github.com/yakovlev-alexey/nest-next-example.

Containerizing application

In order to run application in an isolated environment I will be using Docker. We will need to run both Node.js app and SSH server in the background. The base image would be node:12.22-alpine. Alpine Linux images have certain prerequisites in order to properly boot background processes. About those prerequisites read a recent post I made on how to setup SSH server on an Alpine container.

FROM node:12.22-alpine

ARG ssh_pub_key

RUN mkdir -p /root/.ssh \
    && chmod 0700 /root/.ssh \
    && passwd -u root \
    && echo "$ssh_pub_key" > /root/.ssh/authorized_keys \
    && apk add openrc openssh \
    && ssh-keygen -A \
    && sed -i 's/AllowTcpForwarding no/AllowTcpForwarding yes/g' /etc/ssh/sshd_config \
    && echo -e "PasswordAuthentication no" >> /etc/ssh/sshd_config \
    && mkdir -p /run/openrc \
    && touch /run/openrc/softlevel

WORKDIR /app

COPY . .

RUN yarn && yarn build

ENTRYPOINT ["sh", "-c", "rc-status; rc-service sshd start; yarn start"]

Something that was not present in the post I mentioned is the line configuring AllowTcpForwarding. This option allows us to transfer any data by binding ports using -L client option. It is required for Node.js inspector to communicate with a remote server. Otherwise this Dockerfile is the same and I will neither go into details of its workings nor develop a proper caching solution at this time.

Add shortcuts to frequently used commands to scripts section in package.json. We will map port 3000 as is and port 7655 to internal container port 22 - the default one for SSH server. For authorization a local key will be used and injected into the container.

"start:docker": "docker run --rm -p 3000:3000 -p 7655:22 -it $(docker build --build-arg ssh_pub_key=\"$(cat ~/.ssh/id_rsa.pub)\" -q .)"
# unwrapped command
docker run --rm \
    -p 3000:3000 -p 7655:22 \
    -it $(docker build \
        --build-arg ssh_pub_key="$(cat ~/.ssh/id_rsa.pub)" \
        -q . \
     )

To establish a connection we will use the following script.

"start:ssh": "ssh -L 9221:localhost:9229 root@localhost -p 7655"

Unfortunately each new built image will have a new host key since they are generated during the build. Just rm ~/.ssh/known_hosts locally or remove the host key from this file. StrictHostKeyChecking=no option will not help since it still prohibits port binding.

Generating load

Before we start adding stub data to generate load let's examine our current perfomance. Use Chrome Dev Tools embedded Lighthouse for this. Start the app using start:docker (it may take some time to build the image at first) and visit localhost:3000. After conducting both mobile and desktop tests I got the following results for mobile.

And for desktop.

Seems very much fine at the moment.

Now let's imagine our application puts some large amounts of data into the client payload in order to display the page. It could be anything from actual data to simple translations. For this example I will be stubbing a large object of translations - as close to a real life scenario as possible.

// ./src/server/translations.middleware.ts
import { NextFunction, Request, Response } from 'express';

const createStubTranslation = () => {
  return {
    en: {
      value: 'Lorem ipsum',
    },
    gb: {
      value: 'Lorem ipsum',
    },
    meta: {
      description: 'Dolore sit ame',
      properties: {
        name: 'string',
        age: 'number',
      },
    },
  };
};

const TRANSLATIONS = Array.from(Array(5000)).reduce((acc, _item, index) => {
  acc[index] = createStubTranslation();
  return acc;
}, {});

export const translationsMiddleware = (
  req: Request,
  _res: Response,
  next: NextFunction,
) => {
  req.translations = TRANSLATIONS;

  next();
};

This basic middleware will inject a translations property with a large amount of nested properties. To make sure it properly works extend the type for Express Request.

// ./@types/express.d.ts
import 'express';

declare module 'express' {
  interface Request {
    translations: any;
  }
}

And add this middleware to AppModule.

import { MiddlewareConsumer, Module } from '@nestjs/common';
import { translationsMiddleware } from './translations.middleware';

@Module({})
export class AppModule {
  public configure(consumer: MiddlewareConsumer) {
    consumer.apply(translationsMiddleware).forRoutes('*');
  }
  // ...
}

Finally update extractAppData to filter translations property.

// ./src/client/ssr/extractAppData.ts
import { Request } from 'express';
import { GetServerSidePropsContext } from 'src/shared/types/next';
import { AppData } from 'src/shared/types/app-data';
import { filterUnserializable } from './filterUnserializable';
import { StaticQuery } from './buildServerSideProps';

const extractAppData = (
  ctx: GetServerSidePropsContext<Partial<StaticQuery>>,
) => {
  const { features, basePath } = ctx.query.config || {};

  const { translations } = ctx.req as Request;

  return filterUnserializable({
    features,
    basePath,
    translations,
  }) as Partial<AppData>;
};

export { extractAppData };

Narrowing down the issue

The load is in place - time to check Lighthouse. Reboot start:docker and generate another report.

Seems fine, right? But it does not align well with what you actually see when you refresh the page - loading takes well over a second - and that is on a desktop (assuming you are on one) for an almost empty page.

Well, if you scroll Lighthouse report a little lower you will see a few "opportunities". It turns out it indeed takes more than a second to recieve the first byte from the server. However it is not exactly clear from the Lighthouse score. It can be very deceiving and that is one of the reasons why you should not consider a primary metric for perfomance.

Another tool we may employ to narrow down a bottleneck is request timings. In the network section of Dev Tools select any request and open "Timing" tab. For the initial request you will see a similar picture. Large TTFB always indicates issues with server-side request handling.

You may narrow down issue location by checking other requests which do not require server rendering a page. They will have little to no TTFB. Clearly something is wrong with server-side rendering.

When a server handles the request it typically first executes all the middlewares, then interceptors, actual handlers and filters if needed. Server-side rendering may be stalled by blocking requests run synchronously one after another. In large projects it is really challenging to pinpoint what causes issues analytically.

Therefore we should do a perfomance inspection using profiling. But the code is run on a remote server, how can we debug and profile a remote Node.js application?

Profiling a remote Node.js application

I will be largely following the official doc on Node.js debugging.

Attaching Node.js inspector

First thing we should do is establish a remote SSH connection to our server. We already configured SSH in the Docker container and may use start:ssh script to connect.

If you are having issues connecting try troubleshooting recommendations from my recent post on Alpine Docker images SSH I mentioned earlier.

Upon successful connection you should find the PID for the Node.js application. Use ps command for this.

$ yarn start:ssh
# <...>
$ ps
PID   USER     TIME  COMMAND
    1 root      0:01 node /opt/yarn-v1.22.17/bin/yarn.js start
  127 root      0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
  154 root      0:18 /usr/local/bin/node dist/server/main.js
  176 root      0:00 sshd: root@pts/1
  178 root      0:00 -ash
  181 root      0:00 ps

You would see a similar result. What we need is main.js process - PID 154. Currently there is not inspector attached to it. To open a Node.js process to inspector attachments send a SIGUSR1 signal to it using kill -s SIGUSR1 154.

Now that Node.js process is ready to be inspected you should open dedicated Dev Tools for Node. Open chrome://inspect in your Chromium based browser. You would see a page like this.

Click "Open dedicated Dev Tools for Node". Here you should add a new remote connection. Remember SSH connection script? -L 9221:localhost:9229 binds internal Docker container port 9229 (the default one used after sending SIGUSR1 signal) to localhost:9221. Therefore new connection network address would localhost:9221.

If you have done everything correctly you should see a new remote target on chrome://inspect.

At the same time Dev Tools should now show some messages in the console and server file system in "Sources". There you may actually add breakpoints and debug your code remotely. But what we are interested in at the moment is "Profiling" tab.

Reading profiles

Click "Start" and refresh localhost:3000 in your browser. After successful refresh and complete page load return to Dev Tools and stop the recording. You should now see "Profile 1" in the list on the left.

At the top we would see functions that took the longest to execute. That would be _defineProperty and _objectSpread generated by tsc for new language features to work with ES5. Still we may be using those in a variety of places. Let's sort not by "Self time" (time that took to execute statements in the function itself and not the internal function calls) but by "Total time". And we indeed see a clear winner.

filterUnserializable takes an unreasonable amount of time to finish. _defineProperty and _objectSpread functions already tell us that it is due to inefficient spreads and property assigning. Clearly we should refactor this function to make it faster.

Benching improvements

In order to be sure that we introduce a quicker solution we should employ a tool like jsbench.me. First we add tested data at "Setup JS" stage. Then add test cases with different function implementations. Check this bench with suggested implementation and run it yourself for results.

Clearly the new function is much better. With smaller payloads old implementation will show slightly better results but still worse than the new mutable one.

When refactoring such an important function you surely would want to first add unit tests to make sure that it has the same behaviour. At this time we will not do that but make sure you do so when refactoring production applications. The new implementation also fixes a bug with invalid array serialization.

Let's implement the improvement.

// ./src/client/ssr/filterUnserializable.ts
const filterUnserializable = (
  obj: Record<string, unknown>,
  filteredValues: unknown[] = [undefined],
) => {
  Object.keys(obj).forEach((key) => {
    if (filteredValues.includes(obj[key])) {
      delete obj[key];
      return;
    }

    if (obj[key] === Object(obj[key]) && !Array.isArray(obj[key])) {
      filterUnserializable(obj[key] as Record<string, unknown>, filteredValues);
    }
  });

  return obj;
};

export { filterUnserializable };

Now the application is using a much more effective way of filtering unserializable data and can grow much larger without perfomance hickups. Let's make sure that improvements indeed took place, restart Docker container and record another profile.

When rebooting container and attempting to connect via SSH you will likely see error messages like channel 3: open failed: connect failed: Connection refused. Node.js Dev Tools make attempts to connect though there is no servers listening on internal Docker container port 9229. Run kill -s SIGUSR1 <APP_PID> again to allow inspector attachments. Or temporarily close/delete connection in Dev Tools.

Times of 100ms are much more reasonable for large objects like this. Obviously you could make more improvements like making sure that objects like translations are valid before they reach SSR or caching validated objects. You may also assume that there are no nested undefined properties. For the purpose of this article existing improvements will suffice.

Lessons learned

Now that we profiled and fixed the issue in an emulated environment we can compile a list of things a developer or a team should do to avoid situations when you need to conduct profiling on your application.

First I will list preemptive solutions and then how you may improve perfomance when issues arise.

1) Collect real usage data using Real user monitoring. Some services like Vercel provide RUM out of the box. Collect data like FCP, TTI and other metrics.

2) Setup alerts over spikes in RUM data. This way when a release happens to introduce a significant slowdown you will be alerted immediately. If you are against alerts or have technical difficulties implementing them develop a culture of frequent monitoring for this data. Regularly assess it a few days after a release happens.

3) Perform regular load testing. It will also succeed in showing drops in responsiveness of a service under load.

4) Do not overuse immutability. While it does help you write predictable code it also hurts perfomance when overused. When in doubt create a quick bench on jsbench to make sure you make the right decision.

5) Parallelize requests and stream data where possible. If a page requires multiple pieces of information attempt requesting them simultaneously in parallel.

6) Prevalidate data and use as little of it as possible. Do not pull unnecessary entities. Split data required for individual pages. Use cache where applicable.

7) For critical parts that are inaccessible by RUM setup manual monitoring by tracking time to execute with tools like console.time() or process.hrtime.

And when challenged with improving perfomance do:

1) Before making any improvements create a baseline for perfomance. This may be some metric or a combination of those. It should work toward improving actual user experience.

2) Do not rely on assumptions. Conduct perfomance tests and record profiles. Use real data on an environment as close to the production one as possible. Use this data to choose next steps.

3) When introducing an improvement make sure that it behaves the same way using unit tests. And make sure it does what you want it to do - perform better. Once again employ jsbench.

I hope you found this article helpful in learning remote Node.js profiling and general perfomance care. Perhaps you have some rules of your own you follow to make sure your app stays perfomant? Or a story of a big perfomance hickup? Feel free to discuss in the comments.

Running SSH in an Alpine Docker Container

Alexey Yakovlev — Fri, 14 Jan 2022 09:31:30 +0000

When deploying your web application you will likely be using Docker for containerization. Many base Docker images like Node or Python are running Alpine Linux. It is a great Linux distro that is secure and extremely lightweight.

However many Docker images provide Alpine without its OpenRC init system. This causes a few issues when trying to run sidecar services with your primary process in a Docker Container.

Table of Contents
Case study
Configuring SSH
- Configuring user
- Installing SSH
- Running sshd
Troubleshooting
Wrapping up

Case study

Let's imagine that we have a remote server hosted somewhere. We are using this server for deploying our production application. To run the application in an isolated environment we are continerizing a Node.js/Python/nginx or any other app. To administrate processes we would want to have direct access to production environment - docker container in this case. SSH is the way to go here, but how should we setup it?

We could SSH into the remote server and then use docker exec but that would not be a particularly secure or elegant solution. Perhaps we should forward SSH connection to the Docker container itself?

Binding ports is fairly easy - we will bind not only port 443 (or any other port you might use for your use case) but also port 22. Run command would like something like docker run -p 443:<docker_app_port> -p 22:22 <container_id>.

The more challenging part would setting up the actual SSH inside the container. We will take a simple Node.js Dockerfile as a base.



FROM node:12.22-alpine

# added code goes here

WORKDIR /app

COPY . .

RUN yarn

RUN yarn build

CMD ["yarn", "start"]

Configuring SSH

To reproduce the connection to a remote server we would be running this Docker image locally and connecting using localhost.

Configuring user

Firstly, in order to be able to login as root (or any other user) we would have to unlock the user and add authorized SSH keys (unless you would want to use text passwords to login which is very insecure).



# create user SSH configuration
RUN mkdir -p /root/.ssh \
    # only this user should be able to read this folder (it may contain private keys)
    && chmod 0700 /root/.ssh \
    # unlock the user
    && passwd -u root

Now the root user is unlocked to be logged into and there is a folder for SSH configuration. However at the moment anyone can login as root since there is not password and password auth is enabled by default. Our goal would be to disable password auth and to add our public key as an authorized one for this user.

You may also use a different user for this purpose - just replace /root with the path to the desired users home directory and replace username.



# supply your pub key via `--build-arg ssh_pub_key="$(cat ~/.ssh/id_rsa.pub)"` when running `docker build`
ARG ssh_pub_key

RUN echo "$ssh_pub_key" > /root/.ssh/authorized_keys

Installing SSH

Now in order to disable password auth via SSH we have to first install SSH. I will be using OpenSSH for this but other implementations should work similarly.



RUN apk add openrc openssh \
    && ssh-keygen -A \
    && echo -e "PasswordAuthentication no" >> /etc/ssh/sshd_config

Stripped Alpine Docker images like the Node one do not provide OpenRC by default so we should install it ourselves. Then generate host SSH keys so clients may authorize our container as an SSH host. Finally, append PasswordAuthentication no to the end of sshd_config to disable password auth via SSH.

For real applications you would invest in pre-generating host keys so that keys do not update every time a new container is built. However this and managing keys more elegantly is out of scope for this post.

At this point we may start the sshd service.



RUN rc-status \
    # touch softlevel because system was initialized without openrc
    && touch /run/openrc/softlevel \
    && rc-service sshd start

However when you run the container you would see something like kex_exchange_identification error. When docker exec into the container and execute rc-status you would see that sshd service is crashed.

Running sshd

Service is crashed because Alpine Docker images allow only a single process to be launched. It is actually a good concept that facilitates using microservices and creating docker compositions. However in this particular case there is not way to run SSH in a different container. To avoid that the actual ENTRYPOINT or CMD command should boot multiple processes.



FROM node:12.22-alpine

ARG ssh_pub_key

RUN mkdir -p /root/.ssh \
    && chmod 0700 /root/.ssh \
    && passwd -u root \
    && echo "$ssh_pub_key" > /root/.ssh/authorized_keys \
    && apk add openrc openssh \
    && ssh-keygen -A \
    && echo -e "PasswordAuthentication no" >> /etc/ssh/sshd_config \
    && mkdir -p /run/openrc \
    && touch /run/openrc/softlevel

WORKDIR /app

COPY . .

RUN yarn

RUN yarn build

ENTRYPOINT ["sh", "-c", "rc-status; rc-service sshd start; yarn start"]

In this final Dockerfile I combined all previous RUN commands into a single one to reduce the amount of layers. Instead of running rc-status && rc-service sshd start in RUN we do that in ENTRYPOINT inside sh -c. This way Docker container will execute only a single process sh -c that would spawn childs.

As a side effect this Node.js application will not recieve SIGINT/SIGTERM signals (or any other signals from Docker) when stopping Docker container.

Run built container using docker run -p 7655:22 <container_id>. In a different terminal instance run ssh root@localhost -p 7655. Voila - you successfully SSHed into a Docker Container. Using SSH for your production app would be the same except you would be using its IP instead of localhost and a valid port.

To properly build and run container without an app around it replace yarn start with node and remove yarn/yarn build directives.

Troubleshooting

When any difficulties with running SSH may arise first try to docker exec into the container and check rc-status. sshd service should be started. If it has crashed or is not present you may not be starting it properly in ENTRYPOINT or have incorrect configuration in sshd_config. You may also be missing SSH host keys or use an already bound port.

To further troubleshoot you can run the container with docker run -p 7655:22 7656:10222 <container_id>. Then docker exec <container_id> and run $(which sshd) -Ddp 10222. This would create another instance of sshd that would listen on a different port (10222) with verbose logging. From there you should see informational messages on why the process might crash.

If the process does not crash attempt connecting. Run ssh root@localhost -p 7656 on the Docker host machine. From this point you will see additional logs in your previous terminal window with details on why the connection has been refused.

Wrapping up

While this tutorial is pretty specific to running SSH in an Alpine Docker container, you may reuse this knowledge to run SSH in other Linux Docker distros. Or you may have better luck configuring other sidecar services inside an Alpine Docker container. In any case I hope you found what you were looking for in this post or learnt something new.

Advanced Practices for Nest.js + Next.js Projects

Alexey Yakovlev — Fri, 07 Jan 2022 11:22:38 +0000

This is the second and final part in a series of articles about combining nest.js and NEXT.js. In the first part we created a project and discovered a few basic SSR techniques for this stack. At this point the project can already be used as a base for developing a real website. However there are a few more things we can do: enable Hot Module Replacement for nest-next, discover more reusable SSR techniques and learn of subdirectory deployments.

This post is a translation of the original articles by me on Habr. The translation had no input from experienced tech writers or editors. So any feedback towards correcting any mistakes is greatly appreciated.

Table of Contents
Introduction
CDN deployment
HMR and NEXT.js instance caching
- Hot Reload in nest.js
- Caching NEXT.js instance
More SSR techniques
- Creating a configuration
- AOP for GSSP
- Accessing application context
- Client-side navigation
Subdirectory deployment
- Development proxy
- Adding basePath
- Proxy and basePath
- Fetch wrapper
- Proxy without rewrites
Conclusion

Introduction

I should remind you that the final application is available at my GitHub - https://github.com/yakovlev-alexey/nest-next-example - commit history mostly follows the flow of the article. I assume that you are already finished with the first part of series and have the project at the relevant revision (commit 5139ad554f).

NEXT.js and nest.js are very similar in naming so I will be always typing NEXT.js (the React framework) in uppercase and nest.js (the MVC backend framework) in lowercase to differenciate between them more easily.

CDN deployment

Let's start simple. NEXT.js supports CDN (Content Delivery Network) deployments out of the box. Simply put your path to statics as assetPrefix in your next.config.js when building for production. Then upload ./.next/static to your CDN of choice. It is important to note that assetPrefix also has to be set in the config when launching production nest.js server.

HMR and NEXT.js instance caching

At the moment the application works really fast in dev mode. Reboots should not take more than 3 seconds and pages load up instantly. However this might not be always that way - large projects tend to have significantly crippled build procedures. It is especially noticable for large frontends. Every change to nest.js server would reboot it and create a new NEXT.js instance that would start recompiling your client (almost) from scratch.

This greatly degrades developer experience significantly increasing feedback cycles when developing complex solutions. Let's fix this issue by implementing Hot Module Replacement in nest.js and caching NEXT.js instances between reloads.

Hot Reload in nest.js

We will first follow official documentation on Hot Reload in nest.js. I will not duplicate most of information from there, just the more important parts.

For example, we should use our own start:dev script to accept a different tsconfig.json.



// ./package.json
"start:dev": "nest build --webpack --webpackPath webpack-hmr.config.js --path tsconfig.server.json --watch"

It is important to pass tsconfig.server.json outside of a relative path: ts-loader will look up paths relative to the server executable. If supply it just with the file name it will start searching in parent folders until it finds a matching file.

Next we should tackle Hot Reload in main.ts



// ./src/server/main.ts
import { NestFactory } from  '@nestjs/core';
import { PORT } from  'src/shared/constants/env';
import { AppModule } from  './app.module';

declare const module: any;

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(PORT);

  if (module.hot) {
    module.hot.accept();
    module.hot.dispose(() => app.close());
  }
}
bootstrap();

This implementation will works for us. However make notice of old module disposal: app.close() is called. It causes all modules, services and controller to cease.

Let's check if our server even works after adding HMR. Boot the server and make changes to app.controller.ts. It is important that the server will not reboot at all if you save the file without making any actual changes. If you did indeed make changes to a file it should take ~1-2 seconds to reboot in small projects and no more than ~4-5 seconds in large ones. It is a huge difference that significantly improve developer experience.

If any nest.js modules were using dynamic import/require they may not work after adding Webpack HMR due to bundling. My solution was to boot tsc along with Webpack nest.js executable since dynamically imported files were not frequently changed. You may also need NODE_PATH environment variable to properly resolve dynamic import paths (prepend NODE_PATH=<path> ... to start:dev script).

Caching NEXT.js instance

Still we have not solved the bottleneck we had with new NEXT.js instance for each reboot. This may take up to a few seconds in large projects during initialization and page loads.

We will cache RenderModule from nest-next between module reloads in app.module.ts For this we need nest.js dynamic module initialization.



// ./src/server/app.module.ts
import { DynamicModule, Module } from '@nestjs/common';
import Next from 'next';
import { RenderModule } from 'nest-next';
import { NODE_ENV } from 'src/shared/constants/env';
import { AppController } from './app.controller';
import { AppService } from './app.service';

declare const module: any;

@Module({})
export class AppModule {
    public static initialize(): DynamicModule {
        /* during initialization attempt pulling cached RenderModule
            from persisted data */
        const renderModule =
            module.hot?.data?.renderModule ??
            RenderModule.forRootAsync(Next({ dev: NODE_ENV === 'development' }), {
                viewsDir: null,
            });

        if (module.hot) {
            /* add a handler to cache RenderModule
                before disposing existing module */
            module.hot.dispose((data: any) => {
                data.renderModule = renderModule;
            });
        }

        return {
            module: AppModule,
            imports: [renderModule],
            controllers: [AppController],
            providers: [AppService],
        };
    }
}

Update main.ts to dynamically initialize AppModule.



// ./src/server/main.ts
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

const app = await NestFactory.create(AppModule.initialize());

To make sure everything works we may observe terminal outputs during nest.js reboot. If we succeeded we will not see compiling... message produced when creating a new instance of NEXT.js server. This way we potentially saved hours of developers time.

More SSR techniques

Our original problem was that we did not want to have separate services for nest.js and NEXT.js. It seems that at this point we removed all things that were blocking us from having a decent developer experience with our solution. So how about leveraging our solution to have some advantages?

Earlier we declined the strategy of returning SSR data from controller handlers. We also discovered how we can employ nest.js interceptors to pass data to GSSP in a reusable way. Why don't we use interceptors to pass some common initial data to our client? This data may be user data or tokens/translations/configurations/feature flags and so on.

Creating a configuration

In order to pass some configuration to our client we first need to create some configuration. I will not create a separate module and service for accessing configuration in nest.js (but that is how you would do it in a proper application). A simple file will suffice.

Our configuration will be populated with feature flags. As an example we will use blog_link flag toggling between two link display options.



// ./src/server/config.ts
const CONFIG = {
    features: {
        blog_link: true,
    },
};

export { CONFIG };

Let's create ConfigInterceptor that would put configuration to returned value and include it in @UseInterceptors decorator.

Obviously in a real application you might pull feature flags from a separate service or website. Luckily it's fairly easy to access request information in interceptors: req and res HTTP contexts are available in them so you may execute some middlewares prior to executing interceptors.



// ./src/server/config.interceptor.ts
import {
    Injectable,
    NestInterceptor,
    ExecutionContext,
    CallHandler,
} from '@nestjs/common';
import { Observable } from 'rxjs';
import { map } from 'rxjs/operators';
import { CONFIG } from './config';

@Injectable()
export class ConfigInterceptor implements NestInterceptor {
    intercept(context: ExecutionContext, next: CallHandler): Observable<unknown> {
        return next.handle().pipe(
            map((data) => ({
                ...data,
                config: CONFIG,
            })),
        );
    }
}



// ./src/server/app.controller.ts
import { ConfigInterceptor } from  './config.interceptor';

// ...

@Get('/')
@Render('index')
@UseInterceptors(ParamsInterceptor, ConfigInterceptor)
public home() {
    return {};
}

@Get(':id')
@Render('[id]')
@UseInterceptors(ParamsInterceptor, ConfigInterceptor)
public blogPost() {
    return {};
}

To make sure our interceptor works we may use console.log(ctx.query) in GSSP.

It is important not to include sensitive information in ctx.query: it is also serialized when generating HTML. Therefore it is also not recommended to use ctx.query to pass large items that are unused on the client. As a workaround use ctx.req to access such information: during SSR it will be populated with your Express (or fastify for that matter) req from nest.js.

AOP for GSSP

Now we have reusable code to pass information to the client: we should also add some reusable code to parse this information and do something with it. Let's create buildServerSideProps wrapper for GSSP.



// ./src/client/ssr/buildServerSideProps.ts
import { ParsedUrlQuery } from  'querystring';
import { Config } from  'src/shared/types/config';
import {
    GetServerSideProps,
    GetServerSidePropsContext,
} from  'src/shared/types/next';

type StaticProps = {
    features: Config['features'];
};

type StaticQuery = {
    config: Config;
};

const buildServerSideProps = <P, Q extends ParsedUrlQuery = ParsedUrlQuery>(
    getServerSideProps: (ctx: GetServerSidePropsContext<Q>) => Promise<P>,
): GetServerSideProps<Partial<StaticProps> & P, Partial<StaticQuery> & Q> => {
    return async (ctx) => {
        const { features } = ctx.query.config || {};

        const props = await getServerSideProps(ctx);

        return {
            props: {
                ...props,
                features,
            },
        };
    };
};

export { buildServerSideProps };

If you are paying attention you may notice that buildServerSideProps recieves not a real GSSP but a simplified version that return just the props. This stops you from using redirect and other properties in individual GSSPs. To avoid that use a real GSSP as the param to buildServerSideProps.

In order to have helpful typings we should have a type for Config.



// ./src/shared/types/config.ts
export type Config = {
    features: Record<string, boolean>;
};

// ./src/server/config.ts
import type { Config } from 'src/shared/types/config';

const CONFIG: Config = {
    features: {
        blog_link: true,
    },
};

Also NEXT.js GetServerSideProps does not quite suit us - we can't really override Query type. We will add types of our own.



// ./src/shared/types/next.ts
import {
    GetServerSidePropsResult,
    GetServerSidePropsContext as GetServerSidePropsContextBase,
} from  'next';
import { ParsedUrlQuery } from  'querystring';

export type GetServerSidePropsContext<Q = ParsedUrlQuery> = Omit<
    GetServerSidePropsContextBase,
    'query'
> & { query: Q };

export type GetServerSideProps<P, Q = ParsedUrlQuery> = (
    ctx: GetServerSidePropsContext<Q>,
) => Promise<GetServerSidePropsResult<P>>;

Update pages to use this new wrapper.



// ./src/pages/index.tsx
export const getServerSideProps = buildServerSideProps<THomeProps>(async () => {
    const blogPosts = await fetch('/api/blog-posts');

    return { blogPosts };
});

// ./src/pages/[id].tsx
export const getServerSideProps = buildServerSideProps<TBlogProps, TBlogQuery>(
    async (ctx) => {
        const id = ctx.query.id;

        const post = await fetch(`/api/blog-posts/${id}`);

        return { post };
    },
);

It is once again important not to pass the entire config to your client. It might contain sensitive information like tokens, internal URLs and credentials or just make your payload larger.

Accessing application context

We already can access features property on our pages. But this might not be a very comfortable way of doing so. We have to utilize prop-drilling to pass it to deeply nested components. To avoid that, use global application context.



// ./src/shared/types/app-data.ts
import { Config } from './config';

export type AppData = {
    features: Config['features'];
};

// ./src/client/ssr/appData.ts
import { createContext } from 'react';
import { AppData } from 'src/shared/types/app-data';

const AppDataContext = createContext<AppData>({} as AppData);

export { AppDataContext };

To not repeat application context mounting on every page put this logic in _app.tsx. I will implement it using a class but it is not a requirement.



// ./src/pages/_app.tsx
import NextApp, { AppProps } from 'next/app';
import { AppDataContext } from 'src/client/ssr/appData';
import { AppData } from 'src/shared/types/app-data';

class App extends NextApp<AppProps> {
    appData: AppData;

    constructor(props: AppProps) {
        super(props);

        this.appData = props.pageProps.appData || {};
    }

    render() {
        const { Component, pageProps } = this.props;

        return (
            <AppDataContext.Provider value={this.appData}>
                <Component {...pageProps} />
            </AppDataContext.Provider>
        );
    }
}

export default App;

Update buildServerSideProps slightly.



// ./src/client/ssr/buildServerSideProps.ts
import { AppData } from 'src/shared/types/app-data';

// ...

type StaticProps = {
    appData: Partial<AppData>;
};

// ...

return {
    props: {
        ...props,
        appData: {
            features,
        },
    },
};

And create a hook to use AppDataContext easily.



// ./src/client/ssr/useAppData.ts
import { useContext } from 'react';
import { AppDataContext } from './appData';

const useAppData = () => {
    return useContext(AppDataContext);
};

export { useAppData };

Finally, implement the useFeature hook and utilize it on the index page.



// ./src/client/hooks/useFeature.ts
import { useAppData } from 'src/client/ssr/useAppData';

const useFeature = (feature: string, defaultValue = false) => {
    return useAppData().features[feature] ?? defaultValue;
};

export { useFeature };

// ./src/pages/index.tsx
const Home: FC<THomeProps> = ({ blogPosts }) => {
    const linkFeature = useFeature('blog_link');

    return (
        <div>
            <h1>Home</h1>
            {blogPosts.map(({ title, id }) => (
                <div key={id}>
                    {linkFeature ? (
                        <>
                            {title}
                            <Link href={`/${id}`}> Link</Link>
                        </>
                    ) : (
                        <Link href={`/${id}`}>{title}</Link>
                    )}
                </div>
            ))}
        </div>
    );
};

Validate the changes in the browser. localhost:3000 should display links according to the feature flag we set in the server configuration. Update the configuration and check if after page refresh displayed link changes.

Client-side navigation

After makes such significant changes to our SSR pipelines and GSSP we surely have to check if client-side navigations still work.

Our concerns are justified: browsers will reload the page when making client-side transitions. In the terminal we would see an error: we can't quite serialize appData.features field. It expectedly is undefined when making client-side navigation. Our nest.js controller handlers do not get called when making client-side transitions, remember?

Only apply interceptors or req properties to pass actual initial data. And never expect them to be available in GSSP functions. Such data may be translations, feature flags, CSRF tokens and other configurations.

To solve our current issue let's sanitize buildServerSideProps return values.



// ./src/client/ssr/filterUnserializable.ts
const filterUnserializable = (
    obj: Record<string, unknown>,
    filteredValues: unknown[] = [undefined],
): Record<string, unknown> => {
    return Object.keys(obj).reduce<Record<string, unknown>>((ret, key) => {
        if (typeof obj[key] === 'object' && obj[key] !== null) {
            return {
                ...ret,
                [key]: filterUnserializable(obj[key] as Record<string, unknown>),
            };
        } else if (!filteredValues.includes(obj[key])) {
            return { ...ret, [key]: obj[key] };
        }

        return ret;
    }, {});
};

export { filterUnserializable };

// ./src/client/ssr/buildServerSideProps
import  {  filterUnserializable  }  from  './filterUnserializable';

// ...

return {
    props: {
        ...(await getServerSideProps(ctx)),
        appData: filterUnserializable({ features }) as StaticProps['appData'],
    },
};

This filterUnserializable implementation has some flaws. It will show poor perfomance for large and nested objects. For completeness and consistency between article versions I will leave the code as is to later return in a separate article on how to diagnose and profile such issues.

Let's test client-side transitions once again - should work now.

Having the ability to easily pass arbitary data to your client is one of the primary advantages nest-next offers in my opinion. Data may come from any source including middlewares mutating req properties. You may leverage existing Express solutions in your nest-next project and deliver changes to clients almost immediately.

Subdirectory deployment

In some cases you might want your production application to be deployed in a subdirectory. For example we are working on documentation and our service should be available at /docs. Or with our current application - a blog section on a website with /blog prefix.

What should we do to support such functionality? It seems the only thing blocking us is API requests and client links. Statics will be deployed on a CDN so it will not be a problem. Seems like a fairly simple task at first.

But then we remember that NEXT.js makes requests to an internal endpoint when making client-side transitions. Request handlers executes GSSP on the server and return JSON with data. We are rather helpless when trying to change this. And if we do not use a CDN the entire static will break. This will not do for us.

Luckily in NEXT.js docs we find basePath parameter. It allows NEXT.js to support subdirectory deployments out of the box, adding it to every NEXT.js server endpoint. Great, now we have a plan, let's start.

Development proxy

Before we start coding anything we should provide a way to reproduce our production environment with subdirectory deployment. For this we will use a simple proxy server. Use Docker and nginx to implement and start a proxy. Let's add proxy configuration.



# ./nginx.conf
server {
    listen 8080;

    location /blog/ {
        proxy_pass http://localnode:3000/;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

It is important to understand that nginx will boot in a Docker container with a net of its own. Therefore to properly proxy the request to the nest.js server we would need an address for the host machine. We will use localnode name for it. I have not tested start:proxy script on Windows machines. Unless you are using WSL you may have some issues supplying the container with host machine address.

This configuration will allow proxy to handle requests to localhost:8080/blog/* and proxy them to localhost:3000/*. In package.json add start:proxy script to boot a Docker container with our proxy.



docker run --name nest-next-example-proxy \
    -v $(pwd)/nginx.conf:/etc/nginx/conf.d/default.conf:ro \
    --add-host localnode:$(ifconfig en0 | grep inet | grep -v inet6 | awk '{print $2}') \
    -p 8080:8080 \
    -d nginx

Adding basePath

Add basePath to your server configuration. Update typings and pull data from BASE_PATH environment variable.



// ./src/shared/types/config.ts
export type Config = {
    features: Record<string, boolean>;
    basePath: string;
};

// ./src/server/config.ts
import { Config } from 'src/shared/types/config';

const CONFIG: Config = {
    features: {
        blog_link: true,
    },
    basePath: process.env.BASE_PATH || '',
};

export { CONFIG };

Create next.config.js to configure NEXT.js. Put basePath there as well.



// ./next.config.js
module.exports = {
    basePath: process.env.BASE_PATH,
};

Proxy and basePath

Check website on localhost:8000/blog after booting both server and proxy. The request reaches nest.js server but NEXT.js static requests are unsuccessful. NEXT.js expects that the request comes with the desired basePath in req.url. But in our nginx proxy we cut it off. Add a separate rule to proxy /blog/_next requests without replacing our basePath.



server {
    listen 8080;

    location /blog/_next/ {
        proxy_pass http://localnode:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /blog/ {
        proxy_pass http://localnode:3000/;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Reboot Docker container and check the website in your browser. Unfortunately we are out of luck again.

There was (and still is to some extent) a problem with nest-next. This package applies a filter that forwards unhandled 404s to NEXT.js. But only if the request starts with /_next. Therefore NEXT.js server expects a request that starts with basePath and nest-next proxies only requests starting with /_next.

I submitted a PR to fix this issue and it was merged. However package maintainer Kyle has not published a new version with this fix since then. You may ask him to publish or use a compiled version from my GitHub.



yarn upgrade nest-next@https://github.com/yakovlev-alexey/nest-next/tarball/base-path-dist

Tag base-path-dist only includes the required files.

After bumping npm package check localhost:8080/blog in your browsers. You should finally see a familiar Home page. Client-side navigations also work!

Fetch wrapper

The only thing left is to add basePath to fetch requests. It seems that there is not need for that at the moment. However that is only due to the fact that we call fetch only on the server and never from the client. As soon as you start fetching from the client you start seeing errors.

Let's refactor buildServerSideProps a little. Update typings for AppData and extract appData parsing from ctx.query to a separate method.



// ./src/shared/types/app-data.ts
import { Config } from './config';

export type AppData = Pick<Config, 'basePath' | 'features'>;

// ./src/client/ssr/extractAppData.ts
import { GetServerSidePropsContext } from 'src/shared/types/next';
import { AppData } from 'src/shared/types/app-data';
import { filterUnserializable } from './filterUnserializable';
import { StaticQuery } from './buildServerSideProps';

const extractAppData = (
    ctx: GetServerSidePropsContext<Partial<StaticQuery>>,
) => {
    const { features, basePath } = ctx.query.config || {};

    return filterUnserializable({ features, basePath }) as Partial<AppData>;
};

export { extractAppData };

Use new util in buildServerSideProps.



// ./src/client/ssr/buildServerSideProps.ts
import { extractAppData } from './extractAppData';

// ...

const buildServerSideProps = <P, Q extends ParsedUrlQuery = ParsedUrlQuery>(
    getServerSideProps: (ctx: GetServerSidePropsContext<Q>) => Promise<P>,
): GetServerSideProps<Partial<StaticProps> & P, Partial<StaticQuery> & Q> => {
    return async (ctx) => {
        const props = await getServerSideProps(ctx);

        return {
            props: {
                ...props,
                appData: extractAppData(ctx),
            },
        };
    };
};

export { buildServerSideProps };

Finally we may access basePath on the client. The only thing left is to add this property to the actual fetch wrapper. I will not create an ingenious solution and turn envAwareFetch into a function with side effects.



// ./src/shared/utils/fetch.ts
import { isServer, PORT } from '../constants/env';

type FetchContext = {
    basePath: string;
};

const context: FetchContext = {
    basePath: '',
};

const initializeFetch = (basePath: string) => {
    context.basePath = basePath;
};

const getFetchUrl = (url: string) => {
    if (isServer) {
        // на сервере не нужно добавлять basePath - запрос делается не через proxy
        return url.startsWith('/') ? `http://localhost:${PORT}${url}` : url;
    }

    return url.startsWith('/') ? context.basePath + url : url;
};

const envAwareFetch = (url: string, options?: Partial<RequestInit>) => {
    const fetchUrl = getFetchUrl(url);

    return fetch(fetchUrl, options).then((res) => res.json());
};

export { envAwareFetch as fetch, initializeFetch };

Finally initialize fetch using initializeFetch. It might seem that we should do that in GSSP however it is once again only run on the server. Therefore use _app.tsx as the place to initialize fetch.



// ./src/pages/_app.tsx
constructor(props: AppProps) {
    super(props);

    this.appData = props.pageProps.appData || {};

    initializeFetch(this.appData.basePath);
}

To check that everything works properly you add a fetch request in useEffect on one of the pages.

This way we can deploy nest-next application in subdirectories without loosing any of the features or advantages we had.

Proxy without rewrites

There is also an option to use a proxy without rewrites. Then all requests have to be handled with basePath on the server as well. Proxy configuration would look like this.



server {
    listen 8080;

    location /blog/ {
        proxy_pass http://localnode:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

To prepend basePath to every nest.js server endpoint you may employ something like nest-router and use basePath as path parameter.

It is also important to initialize fetch before making any requests in GSSP. And you would have to pull basePath from an environment variable or directly from config when running GSSP on client-side transition.

Conclusion

nest-next allows you to have very reusable SSR pipelines with all the features of NEXT.js and existing infrastructure you might have with nest.js and Express. Hopefully this series of articles helped you to better understand how you might leverage this technology to help you develop large applications. Skills you might get from completing this tutorial can also be reused for configuring nest.js with other purposes.

Creating a Project with Nest.js + Next.js

Alexey Yakovlev — Wed, 05 Jan 2022 13:38:22 +0000

This is the first part in a series of articles about combining nest.js and NEXT.js. Today you will learn how to setup a project and choose a valid SSR strategy. In the second part you will learn about Hot Module Replacement, more SSR techniques and subdirectory deployments.

Hey, it's October 2022 and Next.js 13 was recently released! It brings lots of changes and the new app directory. While I am working on discovering all the new features and use cases to update this tutorial (and perhaps nest-next), make sure to leave your feedback on nest-next with Next.js 13 in this GitHub issue.

This post is a translation of the original articles by me on Habr. The translation had no input from experienced tech writers or editors. So any feedback towards correcting any mistakes is greatly appreciated.

Table of Contents
Introduction
Before we start
Creating a nest.js application
NEXT.js installation
SSR data preparation
- Using nest.js controller
- Using direct services access
- By sending requests to itself
  - Wrapper for fetch
  - Passing path parameters
Next steps

Introduction

When choosing a framework for web development in 2022 you might be considering NEXT.js. It is a React based framework that supports SSR/SSG, Typescript, great codesplitting and opinionated routing out of the box. It's truly a great choice for solo developers and teams alike.

NEXT.js also offers some backend functionality with serverless functions. However, serverless may not be the way for you and you would like to use something you are more used to. nest.js is a great choice here - it follows MVC architecture familiar to many which makes it very scalable and extendable.

NEXT.js and nest.js are very similar in naming so I will be always typing NEXT.js (the React framework) in uppercase and nest.js (the MVC backend framework) in lowercase to differenciate between them more easily.

But how do you integrate these two? How do you make them work together? The simplest option would be to create a sophisticated (or not so sophisticated) proxy that would forward API requests to nest.js and all others to NEXT.js. But this might not be an option for you if you are limited by the amount of microservices you can use or you might not want a real monorepo in order to share code between services.

Luckily there is an easy option for you to embed Next.js right in your nest.js server. Meet nest-next - a view renderer for nest.js which uses NEXT.js to render pages right from a controller with a single decorator. However this leads to a few caveats in the inner workings of your application. Let's create a simple nest-next application together from the ground up so that we can discover these caveats and I can tell you about some of the best practices I and my colleagues discovered using this technology.

Throughout the article I will try to delve as little as possible into each of the frameworks and will primarily focus on the bridge between the two. When certain knowledge about a framework is needed I will leave links to official documentation.

Before we start

It's highly likely that you or your team opted for NEXT.js before choosing a backend. Here I recommend you to stop temporarily and seriously consider NEXT.js server features - you might not need a real backend for many use cases like a simple backend-of-the-frontend render server. Using two frameworks and maintaining a bridge between them would be an overhead.

You should only consider using nest-next if you are either planning a real node.js backend or you already have certain Express/fastify/nest.js infrastructure you plan to employ.

The article is fairly large since it covers most of the quirky details about the frameworks and it has been split into two parts. In the first one we will create a simple application from the ground up and show solutions the very basic problems you might have with SSR. If you consider yourself an experienced developer in this stack it might be more interesting for you to start on the second part where I talk about some of the more advanced use cases like Hot Module Replacement, SSR techniques and deployment in a subdirectory.

And finally: for those who prefer skipping the article and getting right into code - you can find the source code for this article on my GitHub - https://github.com/yakovlev-alexey/nest-next-example - commit history mostly follows the article.

Creating a nest.js application

First we would need a nest.js application as our base. We will use nest.js CLI to generate a template.



npx @nestjs/cli new nest-next-example

Follow the instructions. I had chosen yarn as my package manager and will leave command snippets for yarn in the article but I assume you are familiar with package managers will be fine using npm in this article.

Upon command completion we will get a mostly empty project which may start instantly. I will remove all the test files (test directory and app.controller.spec.ts) from the project since we are not going to create any tests in this article.

I also recommend using the following directory structure that resembles a monorepo



└── src
    ├── client # client code: hooks, components, etc
    ├── pages # actual NEXT.js pages
    ├── server # nest.js server code
    └── shared # common types, utils, constants etc

Let's make the necessary changes to the nest.js configuration to support our new layout.



// ./nest-cli.json
{
    "collection": "@nestjs/schematics",
    "sourceRoot": "src",
    "entryFile": "server/main"
}

Now if we start the application using yarn start:dev we should see "Hello world" when visiting localhost:3000 in the browser.

Due to certain caveats in nest.js building pipelines you might see an error in your terminal. The error might looks something like this: 'Error: Cannot find module '.../dist/server/main'. In that case you may temporarily set "entryFile" to just "main" - that should solve the issue.

NEXT.js installation

Now let's add NEXT.js to our project.



# NEXT.js and its peers
yarn add next react react-dom
# required types and eslint preset
yarn add -D @types/react @types/react-dom eslint-config-next

Next you should start NEXT.js development server using yarn next dev. Necessary changes to your tsconfig will be made as well as a (few) new files added including next-env.d.ts. NEXT.js will boot successfully. However if we now start nest.js server, we will discover that NEXT.js broke our typescript config. Let's create a separate config for nest.js - I will be reusing existing tsconfig.build.json as tsconfig.server.json with the following contents.



// ./tsconfig.server.json
{
    "extends": "./tsconfig.json",
    "compilerOptions": {
        "noEmit": false
    },
    "include": [
        "./src/server/**/*.ts",
        "./src/shared/**/*.ts",
        "./@types/**/*.d.ts"
    ]
}

Now nest.js works again. Let's update scripts section in our package.json file.



// ./package.json
"scripts": {
    "prebuild": "rimraf dist",
    "build": "yarn build:next && yarn build:nest",
    "build:next": "next build",
    "build:nest": "nest build --path ./tsconfig.server.json",
    "start": "node ./dist/server/main.js",
    "start:next": "next dev",
    "start:dev": "nest start --path ./tsconfig.server.json --watch",
    "start:debug": "nest start --path ./tsconfig.server.json --debug --watch",
    "start:prod": "node dist/main",
    // ... lint/format/test etc
},

Let's add an index page and an App component to src/pages directory.



// ./src/pages/app.tsx
import { FC } from 'react';
import { AppProps } from 'next/app';

const App: FC<AppProps> = ({ Component, pageProps }) => {
    return <Component {...pageProps} />;
};

export default App;



// ./src/pages/index.tsx
import { FC } from 'react';

const Home: FC = () => {
    return <h1>Home</h1>;
};

export default Home;

Now when you start the app using yarn start:next you should the this page on localhost:3000.

You should also add .next folder to your .gitignore - that is where NEXT.js stores its builds.

Making acquaintance between the frameworks

Now we have two separate servers. But what we want is a single nest.js server making use of nest-next: so let's install it.



yarn add nest-next

Next we should initialize the newly installed RenderModule in app.module.ts.



// ./src/server/app.module.ts
import { Module } from '@nestjs/common';
import { RenderModule } from 'nest-next';
import Next from 'next';
import { AppController } from './app.controller';
import { AppService } from './app.service';

@Module({
    /* should pass a NEXT.js server instance
        as the argument to `forRootAsync` */
    imports: [RenderModule.forRootAsync(Next({}))],
    controllers: [AppController],
    providers: [AppService],
})
export class AppModule {}

Now we can make use of @Render decorator exported from nest. So let's create our first page controller in app.controller.ts.



// ./src/server/app.controller.ts
import { Controller, Get, Render } from '@nestjs/common';
import { AppService } from './app.service';

@Controller()
export class AppController {
    constructor(private readonly appService: AppService) {}

    @Get()
    @Render('index')
    home() {
        return {};
    }
}

However when we start the app using yarn start:dev and open the desired page in web browser we will see an error in NEXT.js - the build was not found. Turns out the render server was booted in production mode and expected to see a ready made build of the frontend application. To fix this we should provide dev: true argument when initialising server instance.



// ./src/server/app.module.ts
imports: [
    RenderModule.forRootAsync(Next({ dev: true }))
],

Let's try again. Open localhost:3000 in your browser and you will see a 404. Which is unexpected since we have both a controller and a NEXT.js page. Turns out nest-next is looking in a wrong folder. By default it uses views subdirectory in pages folder rather than the folder itself. I personally do not like this inconsistency with NEXT.js so let's specify viewsDir: null in our RenderModule instance.



// ./src/server/app.module.ts
imports: [
    RenderModule.forRootAsync(
        Next({ dev: true }),
        /* null means that nest-next 
            should look for pages in root dir */
        { viewsDir: null }
    )
],

Adjacent to viewsDir is another dev option - this time for nest-next which enables more specific error serialization. I did not find this option useful but it is there for you if you need it.

Finally, when we open localhost:3000 in the browser we see the page we described earlier in index.tsx.

SSR data preparation

One of the primary NEXT.js advantages is the ability to easily fetch data required to statically or dynamically render a page. Users have a few options to do so. We will use getServerSideProps (GSSP) - the most up-to-date way of fetching dynamic data in NEXT.js. However nest-next properly supports other methods as well.

Time to add another page. Let's imagine that our index is a blog page. And the page we would be creating is a blog post by id.

Add the necessary types and controllers:



// ./src/shared/types/blog-post.ts
export type BlogPost = {
    title: string;
    id: number;
};

// ./src/server/app.controller.ts
import { Controller, Get, Param, Render } from '@nestjs/common';

// ...

@Get(':id')
@Render('[id]')
public blogPost(@Param('id') id: string) {
 return {};
}

Add the new page:



// ./src/pages/[id].tsx
import { GetServerSideProps } from 'next';
import Link from 'next/link';
import { FC } from 'react';
import { BlogPost } from 'src/shared/types/blog-post';

type TBlogProps = {
    post: BlogPost;
};

const Blog: FC<TBlogProps> = ({ post = {} }) => {
    return (
        <div>
            <Link href={'/'}>Home</Link>
            <h1>Blog {post.title}</h1>
        </div>
    );
};

export const getServerSideProps: GetServerSideProps<TBlogProps> = async (
    ctx,
) => {
    return { props: {} };
};

export default Blog;

And refresh our Home page:



// ./src/pages/index.tsx
import { GetServerSideProps } from 'next';
import Link from 'next/link';
import { FC } from 'react';
import { BlogPost } from 'src/shared/types/blog-post';

type THomeProps = {
    blogPosts: BlogPost[];
};

const Home: FC<THomeProps> = ({ blogPosts = [] }) => {
    return (
        <div>
            <h1>Home</h1>
            {blogPosts.map(({ title, id }) => (
                <div key={id}>
                    <Link href={`/${id}`}>{title}</Link>
                </div>
            ))}
        </div>
    );
};

export const getServerSideProps: GetServerSideProps<THomeProps> = async (
    ctx,
) => {
    return { props: {} };
};

export default Home;

Great! Now we have a few pages that need some data. The only thing left is to supply this data. Let's explore the different ways we can do that.

Using nest.js controller

Our home controller in app.controller.ts return an empty object. It turns out that everything that happens to be in that object will be accessible in ctx.query in GSSP.

Let's add some stub data in app.service.ts.



// ./src/server/app.service.ts
import { Injectable } from '@nestjs/common';
import { from } from 'rxjs';

const BLOG_POSTS = [
    { title: 'Lorem Ipsum', id: 1 },
    { title: 'Dolore Sit', id: 2 },
    { title: 'Ame', id: 3 },
];

@Injectable()
export class AppService {
    getBlogPosts() {
        return from(BLOG_POSTS);
    }
}

In controller we may access this service and return the data.



// ./src/server/app.controller.ts
import { map, toArray } from 'rxjs';

// ...

@Get('/')
@Render('index')
home() {
    return this.appService.getBlogPosts().pipe(
        toArray(),
        map((blogPosts) => ({ blogPosts })),
    );
}

Now we will have access to blogPosts property in ctx.query in GSSP. However it seems that this implementation is not very reliable: TypeScript should warn us, that there is in fact no blogPosts in ctx.query. It is typed to be ParsedUrlQuery.

Surely TypeScript is wrong? Let's leave a few console.logs of ctx.query in our index.tsx GSSP. Then open localhost:3000. Check our terminal (that's where the logs would land - GSSP is only run on the server). We indeed see that blogPosts are there. What's wrong then?

Let's open localhost:3000/1 and then click the Home link. Suddenly our terminal logs an empty object. But how is that possible, we clearly return blogPosts property from our controller?

When navigating on the client side NEXT.js fetches an internal endpoint that executes just the GSSP function and returns serialized JSON from it. So our home controller is not called at all and ctx.query is populated purely with path params and search query.

Using direct services access

As stated previously, GSSP is only executed on the server. Therefore we could technically use nest.js services directly from inside GSSP.

This is in fact a very bad idea. You either have to construct every service yourself (then you have lots of repeated code and loose any profits from DI) or expose nest.js application with it's get method.

Still even if you were to bare with the spaghetti facilitated by global application access from different contexts you would end up missing the HTTP contexts when calling services.

By sending requests to itself

Nothing really stops us from making asynchronous request using fetch from GSSP. However we should make a wrapper around fetch to choose which address to call. But before we proceed we have to get information about where the code is executed and what port the server is subscribed to.



// ./src/shared/constants/env.ts
export const isServer = typeof window === 'undefined';

export const isClient = !isServer;

export const NODE_ENV = process.env.NODE_ENV;

export const PORT = process.env.PORT || 3000;

Now update port subscription in main.ts (await app.listen(PORT)) and choose NEXT.js mode depending on environment.



// ./src/server/app.module.ts
RenderModule.forRootAsync(
    Next({ dev: NODE_ENV === 'development' }),
    { viewsDir: null }
)

// ./package.json
"start:dev": "NODE_ENV=development nest start --path ./tsconfig.server.json --watch"

Now that server imports modules from src/shared the structure of compiled nest.js server differs from before. If you previously changed entryFile in nest-cli.json return it to the old value (server/main.ts), clean dist folder and reboot the server.

Wrapper for fetch

Now we can add a wrapper for fetch to choose hostname depending on execution environment.



// ./src/shared/utils/fetch.ts
import { isServer, PORT } from '../constants/env';

const envAwareFetch = (url: string, options?: Record<string, unknown>) => {
    const fetchUrl =
        isServer && url.startsWith('/') ? `http://localhost:${PORT}${url}` : url;

    return fetch(fetchUrl, options).then((res) => res.json());
};

export { envAwareFetch as fetch };

And update app.service.ts.



// ./src/server/app.service.ts
import { Injectable, NotFoundException } from '@nestjs/common';
import { from, of, toArray } from 'rxjs';

const BLOG_POSTS = [
  { title: 'Lorem Ipsum', id: 1 },
  { title: 'Dolore Sit', id: 2 },
  { title: 'Ame', id: 3 },
];

@Injectable()
export class AppService {
  getBlogPosts() {
    return from(BLOG_POSTS).pipe(toArray());
  }

  getBlogPost(postId: number) {
    const blogPost = BLOG_POSTS.find(({ id }) => id === postId);

    if (!blogPost) {
      throw new NotFoundException();
    }

    return of(blogPost);
  }
}

Add new API endpoints to app.controller.ts.



// ./src/server/app.controller.ts
import { Controller, Get, Param, ParseIntPipe, Render } from '@nestjs/common';
import { AppService } from './app.service';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get('/')
  @Render('index')
  home() {
    return {};
  }

  @Get(':id')
  @Render('[id]')
  public blogPost(@Param('id') id: string) {
    return {};
  }

  @Get('/api/blog-posts')
  public listBlogPosts() {
    return this.appService.getBlogPosts();
  }

  @Get('/api/blog-posts/:id')
  public getBlogPostById(@Param('id', new ParseIntPipe()) id: number) {
    return this.appService.getBlogPost(id);
  }
}

Finally let's update GSSP methods.



// ./src/pages/index.tsx
import { fetch } from 'src/shared/utils/fetch';

export const getServerSideProps: GetServerSideProps<THomeProps> = async () => {
    const blogPosts = await fetch('/api/blog-posts');
    return { props: { blogPosts } };
};

// ./src/pages/[id].tsx
import { fetch } from 'src/shared/utils/fetch';

export const getServerSideProps: GetServerSideProps<TBlogProps> = async () => {
    const id = ctx.query.id;
    const post = await fetch(`/api/blog-posts/${id}`);

    return { props: { post } };
};

Visit localhost:3000. Indeed the blog list is available. Let's visit one of the links to a post - everything should work here as well, client side navigation is fine.

However when we update the page on a post page we get an error - there is no such blog post. Everything seemed to work with client side navigation.

As we already discovered nest-next puts controller return value to ctx.query. This means that the actual query is not there and it the responsibility of the user to prepare it.

To fix this issue we will return the id from blogPost controller.



// ./src/server/app.controller.ts
@Get(':id')
@Render('[id]')
public blogPost(@Param('id') id: string) {
    return { id };
}

API endpoint casted the parameter to an integer. In this case it's better to not parse parameters to stay consistent with NEXT.js and keep them as strings.

Now let's refresh the page in the browser - this should have solved our issue.

Passing path parameters

Obviously we are at a terrible situation when we need to manually pass all the parameters in controllers. What if we need to use search parameters? Surely there is a way to fix that?

We will use a snippet of AOP (Aspect-oriented programming) and one of its mechanisms in nest.js: an Interceptor.



// ./src/server/params.interceptor.ts
import {
    Injectable,
    NestInterceptor,
    ExecutionContext,
    CallHandler,
} from '@nestjs/common';
import { Request } from 'express';
import { Observable } from 'rxjs';
import { map } from 'rxjs/operators';

@Injectable()
export class ParamsInterceptor implements NestInterceptor {
    intercept(context: ExecutionContext, next: CallHandler): Observable<unknown> {
        const request = context.switchToHttp().getRequest() as Request;

      /* after executing the handler add missing request query and params */
        return next.handle().pipe(
            map((data) => {
                return {
                    ...request.query,
                    ...request.params,
                    ...data,
                };
            }),
        );
    }
}

According to NEXT.js documentation path parameters take priority over search params. We also will not override the data from our handler.

Decorate page controller handlers with new interceptor.



// ./src/server/app.controller.ts
import { UseInterceptors } from '@nestjs/common';
import { ParamsInterceptor } from './params.interceptor';

// ...

@Get('/')
@Render('index')
@UseInterceptors(ParamsInterceptor)
public home() {
    return {};
}

@Get(':id')
@Render('[id]')
@UseInterceptors(ParamsInterceptor)
public blogPost() {
    return {};
}

It is important that path parameters have the same names in nest.js and NEXT.js. In other words path params in @Get and in @Render should be the same. API endpoints must not have this interceptor - we do not want to return path params back when calling API.

It would be wise to separate API and page controllers. Then we would be able to put @UseInterceptors decorator on the entire controller class. In this article for simplicity purposes API and page controllers are merged.

Let's validate our changes in the browser by refreshing the page. We should still properly see a blog post by id.

Next steps

At this point we have a basic nest-next application capable of rendering pages and supplying them with data. However we are yet to capitalise on some of the real advantages of such a setup. There are also some other quirks you might encounter especially using this combo for enterprise development.

To learn more advanced topics like HMR, more SSR techniques and proxying with nest-next read the second part of this article.

At this moment I hope this article helped you to finally try out those frameworks together with great efficiency despite the scarce info on actual nest-next usage in docs.

How to create usable and maintainable npm packages

Alexey Yakovlev — Mon, 03 Jan 2022 12:37:03 +0000

Open source era is upon us. Many day to day problems that developers meet already have open source solutions on the web. When encountering an issue we instinctively first search the web for ready-made solutions and make attempts at adopting them. However how often do you find existing solutions clunky, poorly documented, unmaintained and unmaintainable?

For me it is a common situation when I end up creating a solution of my own. Lately I have also started making those solutions open source and available to everyone. When doing so I started thinking about ways I can make my solutions more user friendly than others. And I ended up with a list of things I look for when creating a new npm package, a sort of check list of prerequisites to publish a package.

So how can you make your npm packages more maintainable and usable?

As a disclaimer I should say that I am mostly talking about small to medium npm packages that are quickly developed to solve a small-ish problem. Larger packages are a domain of their own and out of scope of this post.

Documentation

It seems very obvious and basic, but how often do you stumble upon a package that has a very basic or outdated README or no README at all? Such packages are mostly useless to users.

As an engineer of a solution it is your responsibility to provide sufficient documentation on how to employ a tool of your creation. But what should be sufficient?

Analysing my usage of npm packages I discovered that it is best to start with a small introduction of your package - what it does, what it helps to achieve. For example, package nest-next starts by saying that it is a "Render Module to add Nextjs support for Nestjs". This is an example of a good short description that would come up in search. Also do not forget to add this description to your VCS hosting (likely GitHub) and package.json so that it comes up in search better. The actual README file could have a more detailed introduction.

Having a Table of Contents helps greatly. Allow readers to quickly scan contents of your README by providing a list of sections with relevant names and anchors. Try not to have a very long list of items and do not overextend your introduction: Table of Contents should be immediately visible when opening a README. When the list becomes very large (more than approximately 10-12 items) it either means that you should reduce the amount of items or that you are dealing with a bigger package and should split up your README in separate docs or a whole website.

Continue with installation instructions. Which peer dependencies should you install? Maybe there are certain prerequisites and engine limitations? State it there and supply readers with installation snippets that are easy to copy.

Finally, instruct users on actual usage. How do you employ your tool? Where do you configure it? What configuration options are available? How to import it's entities? Maybe there are certain features that are not yet implemented or behave unexpectedly? How is your solution different to similar ones? Try to fit the most important things without turning your README into a JSDoc, focus on actual usage and recipes without implementation details. Perhaps leave a link to a project of your own that employs this package as an example.

It would also be nice to tell readers where to leave feedback or submit enhancements. Invite users to leave GitHub Issues or submit a Pull Request with relevant links. This is also the time to acknowledge other packages that might have inspired you.

Do not forget to leave keywords and VCS links in your package.json. And obviously always include README in your published files.

Managing code

Once again, it's fairly obvious to make your code readable. However some of the more popular packages tend to have all the code in a single file with a mixed bag of code styles. Other packages overengineer solutions. Strike a balance between the two.

Employ type-checking to make your code safer to change. You might not even use TypeScript or actual .ts files. You may use JSDoc typings and JS checks to leverage some type-safety without need to compile your code. I used this strategy in one of my packages and found it really useful. And having types through TypeScript annotations or JSDoc is a must for any exported functions.

Never write all the code in a single file. Create separate files for functions, maybe even folders for different types of functions and classes. Try not to add any dependencies to your project unless they are peer or likely to be reused in users node_modules. Use bundlephobia to track your package size.

Do not invest in automatic tests. This might seem counterintuitive but I find spending time on unit tests for small packages wasteful. Unless your package is a unit of itself, a simple function or class with clear inputs and outputs.

While TDD and unit tests are amazing for product development, I find them fairly useless due to nature of small to medium packages that are either never-changing or ever-changing forcing you to endlessly update tests instead of focusing on the solution to the problem at hand. This obviously changes for larger packages and packages with huge user bases but it is not often you create one to solve a day to day problem.

Do not employ powerful and hard-to-configure tools to build and develop your project. Leave a basic .editorconfig file to keep codestyle under control for tiny packages. Do not minify your sources - it will not give a significant enough difference for small packages. It is far better to install a package with less dependencies than a package with minified sources. For most compilation needs Microsoft TypeScript Compiler (tsc) should do the job. Perhaps do not transpile your code at all.

Keep your package.json clean. State all the required snippets in scripts section. Specify a valid engines property. Use a valid main or module field and include only necessary files: dist folder or index file, README and LICENSE files. And perhaps most importantly properly specify package dependencies: peer dependencies should be peer, and no dev deps should be in actual dependency list.

It is also helpful to have a readable commit log. Packages with a single commit in VCS do not look very reliable. Great commit history also helps when discovering code through blame.

Remember, that it is impossible make a solution to all the problems. To conclude this section, your code should do two things: 1) solve the desired problem and no other problem with as little dependencies as possible; 2) be easy to extend or modify so that users can easily alter your solution to suit their goals.

Versioning

Another simple thing that is somehow hard to get right. Always employ Semantic Versioning. Invest in making your commits both human and machine readable. Conventional Commits can help you with that.

It is not uncommon to find a package maintainer who would accept your contribution and then forget to publish a new version of their package. To make sure that it never happens with you create a basic CI workflow that would automatically manage versions and publish your package according to newly pushed commits to VCS. But do not use external services for CI - GitHub Actions and GitLab-CI would suffice.

Luckily such workflows are largely reusable and I have a few public Gists with GitHub Actions workflows for different types of projects. More serious project with tests and linting may employ this multi-stage workflow and smaller packages would be fine with a simple publish-on-push workflow. Both workflows employ bump-package-version-action of my own, check documentation for more details on it.

Be a human

Just that. Respect your users, respond to issues with manners and in a reasonable time, discuss enhancements with contributors and with detailed feedback. Focus not on having a package that you think works, but on a package that solves users problems and respects their opinion.

Remember that your goal is not to have the most popular package. Your goal should be collaborating on creating the best possible tool to solve a problem. Even if someone discovered a better solution as a fork of yours do not be mad at them - ask them if there is a way to integrate their solution into yours.

Conclusion

Let's rewind all the things I stated to a more concise list. When creating a npm package do:

create documentation for the user
provide the user with installation and usage instructions
warn the user of known issues and limitations
leave a link to your VCS
invite users to leave feedback and contributions
type-check your code
provide types for exported entities
keep dependencies up to date
strive to have a smaller package
create a readable and extendable code
employ Semantic Versioning
follow Conventional Commits
automate versioning and publishing
respect and collaborate with users

And do not:

publish a package with an insufficient or without a README
create a JSDoc out of your README
leave users without links to source code
write the code in a single file
overengineer solutions
invest in unit tests (unless your package is a unit of its own)
solve more than one problem at a time
release breaking changes without major versions
add unnecessary dependencies
forget to publish latest versions
argue with your users about functionality

Do you agree with this list? Perhaps you find some points unreasonable or have something to add? Do you even believe in the open source itself and the future being largely open source? Be welcome to discuss in the comments.

Using TypeScript `paths` and `baseUrl` with Parcel

Alexey Yakovlev — Fri, 31 Dec 2021 18:31:54 +0000

You might have heard about the new kid on the block of build tools for web - Parcel. It combines great scalable architecture with amazing zero-configuration developer experience while being lightning fast at the same time. You might have even thought about trying it out and maybe moving one of your TypeScript projects to it.

Unfortunately, if you were leveraging paths and baseUrl TypeScript Compiler options you will likely not have a great time moving - Parcel does not support these properties out of the box. This might be a huge problem for big codebases. You may also prefer TypeScript Compiler's way of handling absolute imports in your project to Parcel's ~ notation. This was the case for me so I decided to try out Parcel plugin ecosystem and create a Resolver Plugin of my own to solve this issue.

parcel-resolver-ts-base-url

The plugin I created is called parcel-resolver-ts-base-url (name parcel-resolver-tspaths was already taken by an outdated resolver). It has 0 dependencies (except for peer Parcel, of course), weighs barely over 1kb and uses memoization to speed up resolution processes.

You can install parcel-resolver-ts-base-url using npm or yarn:

npm i --save-dev parcel-resolver-ts-base-url
# or
yarn add -D parcel-resolver-ts-base-url

Then simply add parcel-resolver-ts-base-url to your .parcelrc:

{
  "extends": "@parcel/config-default",
  "resolvers": ["parcel-resolver-ts-base-url", "..."]
}

At the moment parcel doesn't provide a way I know of to specify tsconfig.json location. Therefore parcel-resolver-ts-base-url will use tsconfig.json from the project root.

From this point usage will be straightforward - just specify desired aliases in paths property and projects baseUrl and start building without ugly relative imports!

Known Issues and Limitations

At the moment this plugin is pretty raw and have not been tested in a large amount of use cases. So I figured it would be important to specify what to look for when using parcel-resolver-ts-base-url.

What will work:

baseUrl imports
paths aliases without and with a single wildcard
paths and baseUrl imports resolving to a npm package with main or module fields
Imports resolving to a folder with an index file
Imports resolving to files with /(c|m)?jsx?/ and /tsx?/ extensions

What will likely not work:

HMR is untested and might not work
Resource imports and imports with specified extensions are not supported
Aliases leading to node_modules will not be resolved as external dependencies
Custom tsconfig.json names and locations

If you are having other issues please consider leaving feedback at GitHub Issues - any feedback would be greatly appreciated. And if you are willing to contribute with solving one of the issues I would be very glad to receive a Pull Request with your solution.

DEV Community: Alexey Yakovlev

How to Cache Variables Between Restarts in Your Node.js Application

Why Not Just Optimize?

Webpack HMR and why it did not work for me

The CommonJS Trick: Keeping Variables Between Restarts

The Hot-Reload Entrypoint

The Variable Cache Module

Forcibly Closing Connections

The Result

Introducing: hot-keeper

Webpack On-Demand Compilation: How to Speed up Dev mode of Legacy Webpack Applications

Introduction

How to introduce on-demand compilation to Webpack

Integrating with webpack-dev-server

Integrating with a custom server

Conclusion

How to create a Keycloak plugin

Introduction

Table of Contents

Initialize the project

Create a test bench

Implement a custom Authenticator

Custom Action Token

Authenticator implementation

Custom resources for plugins

Notify Keycloak of new Providers

Configure test bench

Authenticator configuration

Next steps

Conclusion

Creating an interactive SVG map on the web with D3

Basic zooming and panning

Groups and interactive elements

Window resizes

Minimap

Clickable minimap

Conclusion

Do not read DDD Distilled by Vaughn Vernon

Why you should not read Domain-Driven Design Distilled

What you should read instead

Creating a Miro-like UI overlay layout using CSS

Test bench

Implementing the overlay

Extending the overlay

Conclusion

How I improved a Node web application perfomance multifold

Table of Contents

Introduction

Emulating environment

Containerizing application

Generating load

Narrowing down the issue

Profiling a remote Node.js application

Attaching Node.js inspector

Reading profiles

Benching improvements

Lessons learned

Running SSH in an Alpine Docker Container

Table of Contents

Case study

Configuring SSH

Configuring user

Installing SSH

Running sshd

Troubleshooting

Wrapping up

Advanced Practices for Nest.js + Next.js Projects

Table of Contents

Introduction

CDN deployment

HMR and NEXT.js instance caching

Hot Reload in nest.js

Caching NEXT.js instance

More SSR techniques

Creating a configuration

AOP for GSSP

Accessing application context

Client-side navigation

Subdirectory deployment

Development proxy

Introducing: `hot-keeper`

Integrating with `webpack-dev-server`