DEV Community: ezzabuzaid

Running Untrusted JavaScript Code

ezzabuzaid — Sun, 21 Jul 2024 21:00:00 +0000

IMPORTANT: This is about running JavaScript and TypeScript code only. That being said, the writing might also be the direction to run other code in other languages.

Allowing users to execute their code within your application opens up a world of customization and functionality, yet it also exposes your platform to significant security threats.

Given that it is user code, everything is expected, from halting the servers (it could be infinity loops) to stealing sensitive information.

This article will explore various strategies to mitigate run user code, including Web Workers, static code analysis, and more…

You should care

There are many scenarios where you need to run user-provided code, ranging from collaborative development environments like CodeSandbox and StackBiltz to customizable API platforms like January. Even code playgrounds are susceptible to risks.

Namely, the two essential advantages of safely running user-provided code are:

Gaining your user’s trust: Even if the user is trustworthy, they may execute code copied from other intently bad people.
Secure your environment: the last thing you need is a piece of code halting your server. Think while (true) {}

Define “Sensitive information”

Running user code isn’t harmful until you’re concerned that this might subject some data to be stolen. Whatever data you’re concerned about will be considered sensitive information. For instance, in most cases, JWT is sensitive information (perhaps when used as an authentication mechanism)

What could go wrong

Consider the potential risks of JWT stored in cookies sent with every request. A user could inadvertently trigger a request that sends the JWT to a malicious server, and...

Cross-Site Scripting (XSS).
Denial of Service (DoS) attacks.
Data exfiltration. Without proper safeguards, these threats can compromise the integrity and performance of your application.

Methods

The Evil Eval

The simplest of all, yet the riskiest.

eval('console.log("I am dangerous!")');

When you run this code, it logs that message. Essentially, eval is a JS interpreter capable of accessing the global/window scope.

const res = await eval('fetch(`https://jsonplaceholder.typicode.com/users`)');
const users = await res.json();

This code uses fetch which is defined in the global scope. The interpreter doesn’t know about it, but since eval can access a window, it knows. That implies that running an eval in the browser is different from running it in a server environment or worker.

eval(`document.body`);

How about this...

eval(`while (true) {}`);

This code will halt the browser tab. You might ask why a user would do this to themselves. Well, they might be copying code from the Internet. That's why it's preferred to do static analysis with/or time-box the execution.

You might want to check MDN Docs about eval

Time box execution can be done by running the code in a web worker and using setTimeout to limit the execution time.

async function timebox(code, timeout = 5000) {
  const worker = new Worker('user-runner-worker.js');
  worker.postMessage(code);

  const timerId = setTimeout(() => {
    worker.terminate();
    reject(new Error('Code execution timed out'));
  }, timeout);

  return new Promise((resolve, reject) => {
    worker.onmessage = event => {
      clearTimeout(timerId);
      resolve(event.data);
    };
    worker.onerror = error => {
      clearTimeout(timerId);
      reject(error);
    };
  });
}

await timebox('while (true) {}');

Function Constructor

This is similar to eval but it’s a bit safer since it can’t access the enclosing scope.

const userFunction = new Function('param', 'console.log(param);');
userFunction(2);

This code will log 2.

Note: The second argument is the function body.

The function constructor can’t access the enclosing scope so that the following code will throw an error.

function fnConstructorCannotUseMyScope() {
  let localVar = 'local value';
  const userFunction = new Function('return localVar');
  return userFunction();
}

But it can access the global scope so the fetch example from above works.

WebWorker

You can run “Function Constructor and eval on a WebWorker, which is a bit safer due to the fact that there is no DOM access.

To put more restrictions in place, consider disallowing using global objects like fetch, XMLHttpRequest, sendBeacon Check this writing about how you can do that.

Isolated-VM

Isolated-VM is a library that allows you to run code in a separate VM (v8's Isolate interface)

import ivm from 'isolated-vm';

const code = `count += 5;`;

const isolate = new ivm.Isolate({ memoryLimit: 32 /* MB */ });
const script = isolate.compileScriptSync(code);
const context = isolate.createContextSync();

const jail = context.global;
jail.setSync('log', console.log);

context.evalSync('log("hello world")');

This code will log hello world

WebAssembly

This is an exciting option as it provides a sandboxed environment to run code. One caveat is that you need an environment with Javascript bindings. However, an interesting project called Extism facilitates that. You might want to follow their tutorial.

What is fascinating about it is that you’ll use eval to run the code, but given WebAssembly’s nature, DOM, network, file system, and access to the host environment are not possible (although they might differ based on the wasm runtime).

function evaluate() {
  const { code, input } = JSON.parse(Host.inputString());
  const func = eval(code);
  const result = func(input).toString();
  Host.outputString(result);
}

module.exports = { evaluate };

You'll have to compile the above code first using Extism, which will output a Wasm file that can be run in an environment that has Wasm-runtime (browser or node.js).

const message = {
  input: '1,2,3,4,5',
  code: `
        const sum = (str) => str
          .split(',')
          .reduce((acc, curr) => acc + parseInt(curr), 0);
        module.exports = sum;
`,
};

// continue running the wasm file

Docker

We're now moving to the server-side, Docker is a great option to run code in an isolation from the host machine. (Beware of container escape)

You can use dockerode to run the code in a container.

import Docker from 'dockerode';
const docker = new Docker();

const code = `console.log("hello world")`;
const container = await docker.createContainer({
  Image: 'node:lts',
  Cmd: ['node', '-e', code],
  User: 'node',
  WorkingDir: '/app',
  AttachStdout: true,
  AttachStderr: true,
  OpenStdin: false,
  AttachStdin: false,
  Tty: true,
  NetworkDisabled: true,
  HostConfig: {
    AutoRemove: true,
    ReadonlyPaths: ['/'],
    ReadonlyRootfs: true,
    CapDrop: ['ALL'],
    Memory: 8 * 1024 * 1024,
    SecurityOpt: ['no-new-privileges'],
  },
});

Keep in mind that you need to make sure the server has docker installed and running. I'd recommend having a separate server dedicated only to this that acts as a pure-function server.

Moreover, you might benefit from taking a look at sysbox, a VM-like container runtime that provides a more secure environment. Sysbox is worth it, especially if the main app is running in a container, which means that you'll be running Docker in Docker.

This was the method of choice at January but soon enough, the language capabilities mandated more than passing the code through the container shell. Besides, for some reason, the server memory spikes frequently; we run the code inside self-removable containers on every 1s debounced keystroke. (You can do better!)

Other options

Safest option

I'm particularly fond of Firecracker, but it’s a bit of work to set up, so if you cannot afford the time yet, you want to be on the safe side, do a combination of static analysis and time-boxing execution. You can use esprima to parse the code and check for any malicious act.

How to run TypeScript code?

Well, same story with one (could be optional) extra step: Transpile the code to JavaScript before running it. Simply put, you can use esbuild or typescript compiler, then continue with the above methods.

async function build(userCode: string) {
  const result = await esbuild.build({
    stdin: {
      contents: `${userCode}`,
      loader: 'ts',
      resolveDir: __dirname,
    },
    inject: [
      // In case you want to inject some code
    ],
    platform: 'node',
    write: false,
    treeShaking: false,
    sourcemap: false,
    minify: false,
    drop: ['debugger', 'console'],
    keepNames: true,
    format: 'cjs',
    bundle: true,
    target: 'es2022',
    plugins: [
      nodeExternalsPlugin(), // make all the non-native modules external
    ],
  });
  return result.outputFiles![0].text;
}

Notes:

Rust-based bundlers usually offer a web assembly version, which means you can transpile the code in the browser. Esbuild does have a web assembly version.
Don't include user specified imports into the bundle unless you've allow-listed them.

Additionally, you can avoid transpiling altogether by running the code using Deno or Bun in a docker container since they support TypeScript out of the box.

Conclusion

Running user code is a double-edged sword. It can provide a lot of functionality and customization to your platform, but it also exposes you to significant security risks. It’s essential to understand the risks and take appropriate measures to mitigate them and remember that the more isolated the environment, the safer it is.

References

Migrating From until-destroy To Angular takeUntilDestroyed

ezzabuzaid — Wed, 20 Dec 2023 20:10:43 +0000

Angular has been releasing new things nonstop, one of which Is the DestroyRef token, a utility class that provides an onDestroy hook that is called when a component/directive is destroyed.

@Component({...})
class IAmAHookComponent {
 constructor() {
   inject(DestroyRef).onDestroy(() => {
     // Have something to clean up before I go?
   })
 }
}

We already have this with the OnDestroy interface, but the cool thing about DestroyRef is that it works wonderfully with another utility function named takeUntilDestroyed

Problem

Before takeUntilDestroyed you’ve to do something similar to unsubscribe from an observable stream$

@Component({...})
export class OldSchoolUnsubscribeComponent implements OnInit, OnDestroy {
_destroy: Subject<void> = new Subject<void>();

 ngOnInit(): void {
   stream$
     .pipe(takeUntil(this._destroy))
     .subscribe();
 }

 ngOnDestroy(): void {
   this._destroy.next();
 }
}

With DestroyRef and takeUntilDestroyed you can do the following:

export class NewWayComponent implements OnInit {
  _destroy: DestroyRef = inject(DestroyRef);

  ngOnInit(): void {
    _stream.pipe(takeUntilDestroyed(this._destroy)).subscribe();
  }
}

Pretty cool, isn’t it?

Previously, I’ve used the fantastic until-destroy package that does what the takeUntilDestroyed does but using a decorator.

import { UntilDestroy, untilDestroyed } from '@ngneat/until-destroy';

@UntilDestroy()
@Component({...})
export class UsingPackageComponent {
 ngOnInit() {
   _stream.pipe(untilDestroyed(this)).subscribe();
 }
}

Very similar. However, I always prefer to maximize the framework’s usage as much as possible. As we already have a method now to do it the Angular way, I’d instead remove a package from my node_modules

Solution

You can change them manually, or you can write a script to modify the code for you. In the last blog, we talked about how we can use TypeScript compiler API to do code modification. This case is one of many that the compiler API excels at. That being said, we are going to use the ts-morph package today:

To learn something.
As of today, you cannot instruct the TypeScript AST printer to respect the original code formatting (although you can apply prettier right after code modification to get it back as it was). Hence, ts-morph
ts-morph offers a friendly API to manipulate the source code

Starting with Installing ts-morph
npm i ts-morph

Create a morph project and select Angular-related files

import * as morph from "ts-morph";

const project = new morph.Project({
  // Change it to select the project you want to update.
  tsConfigFilePath: "./tsconfig.json",
});

// Get all the source files that match the angular pattern (e.g., "*.component.ts")
const files = project.getSourceFiles(
  `/**/*.+(component|directive|pipe|service).ts`
);

Keep in mind that you’re only going to migrate the first case of using until-destroy mentioned above. if you’re using checkProperties, arrayName, or blackList, you’ve to improve on the code.

The Code

The plan is to iterate over all files from ts-morph and then loop over all classes in each file.

// Iterate through each component file
for (const file of files) {
  const classes = file.getDescendantsOfKind(morph.SyntaxKind.ClassDeclaration);
  for (const clazz of classes) {
    // Codemod Logic goes here
  }
}

Moving next, you need to filter out the classes that don’t need to be modified. There are four situations where a class is not modifiable:

The class doesn’t have an Angular-related decorator.
The class doesn’t have a UntilDestroy decorator.
The UntilDestroy decorator has options in it.
The untilDestroyed function is not used in the class.

The class doesn’t have an Angular-related decorator.

for (const clazz of classes) {
  {
    // only migrate classes that have @Component, @Directive, @Pipe or @Injectable decorators
    const angularDecorators = ["Component", "Directive", "Pipe", "Injectable"];
    const angularClass = clazz.getDecorator(dec =>
      angularDecorators.includes(dec.getName())
    );
    if (!angularClass) {
      continue;
    }
  }
}

The class doesn’t have a UntilDestroy decorator.

{
  // @UntilDestroy() is our indication that the class needs to be migrated
  const untilDestroyDecorator = clazz.getDecorator(
    dec => dec.getName() === "UntilDestroy"
  );

  if (!untilDestroyDecorator) {
    continue;
  }
}

The UntilDestroy decorator has options in it.

// if options is specified then skip this file
const [optionsArg] = untilDestroyDecorator?.getArguments() ?? [];
const haveOptions = (
  optionsArg as morph.ObjectLiteralExpression
)?.getProperties().length;
if (haveOptions) {
  continue;
}

The untilDestroyed function is not used in the class. You need to get all calls to untilDestroyed via querying the class for CallExpression nodes and check the name of each node to equal untilDestroyed

{
  // migrate untilDestroyed() to takeUntilDestroyed()
  const untilDestroyedCalls = clazz
    .getDescendantsOfKind(morph.SyntaxKind.CallExpression)
    .filter(call => {
      const identifier = call.getLastChildByKind(morph.SyntaxKind.Identifier);
      return identifier?.getText() === "untilDestroyed";
    });

  if (!untilDestroyedCalls.length) {
    continue;
  }
}

With checks in place, you can move to do the code modification. There are 3 places that can contain an untilDestroyed

Within the constructor

@UntilDestroy()
@Injectable()
export class InboxService {
  constructor() {
    interval(1000).pipe(untilDestroyed(this)).subscribe();
  }
}

Within a method

@UntilDestroy()
@Component({})
export class InboxComponent {
  ngOnInit() {
    interval(1000).pipe(untilDestroyed(this)).subscribe();
  }
}

Inlined in the class body

@UntilDestroy()
@Component({})
export class HomeComponent {
  subscription = fromEvent(document, "mousemove")
    .pipe(untilDestroyed(this))
    .subscribe();
}

So you need to loop over untilDestroyedCalls from above and check where each untilDestroyed call is being used and, based on that fact, invoke the modification logic.

// You need destroyRef in case takeUntilDestroyed(this._destroyRef) have been used
let doWeNeedDestroyRef = null;

for (const untilDestroyedCall of untilDestroyedCalls) {
  const withinConstructor = untilDestroyedCall.getFirstAncestorByKind(
    morph.SyntaxKind.Constructor
  );
  const withinMethod = untilDestroyedCall.getFirstAncestorByKind(
    morph.SyntaxKind.MethodDeclaration
  );

  switch (true) {
    case !!withinConstructor:
      // takeUntilDestroyed if used within the constructor can auto-infer the destroyRef
      // from the injection context
      untilDestroyedCall.replaceWithText("takeUntilDestroyed()");
      break;
    case !!withinMethod:
      // takeUntilDestroyed if used within a method needs to be passed the destroyRef
      untilDestroyedCall.replaceWithText(
        "takeUntilDestroyed(this._destroyRef)"
      );

      // set doWeNeedDestroyRef to true so that you can add the _destroyRef property
      doWeNeedDestroyRef ??= true;
      break;
    default:
      // Assuming the observable is declared directly in the class
      // body so you treat it as if it were within the constructor
      untilDestroyedCall.replaceWithText("takeUntilDestroyed()");
      break;
  }
}

You might have noticied the variable doWeNeedDestroyRef that is used to determine if you need to add the _destroyRef property to the class. If you’re using untilDestroyed within a method, you need to add the _destroyRef property to the class.

{
  // add Inject DestroyRef after last public property
  if (doWeNeedDestroyRef) {
    const lastPublicProperty = clazz
      .getInstanceProperties()
      .filter(prop => prop.getScope() === morph.Scope.Public);
    clazz.insertProperty(lastPublicProperty.length, {
      name: "_destroyRef",
      scope: morph.Scope.Private,
      initializer: "inject(DestroyRef)",
    });
  }
}

Alternativly, you can use _destroyRef with takeUntilDestroyed always and avoid this check.

Last but not least, you need to remove the until-destroy import and UntilDestroy decorator.

{
  // ...
  if (!untilDestroyedCalls.length) {
    continue;
  }
  // remove @UntilDestroy() decorator
  untilDestroyDecorator.remove();
}

Remove the until-destroy import

{
  // remove untilDestroyed import
  const untilDestroyedImport = file.getImportDeclaration(
    imp => imp.getModuleSpecifierValue() === "@ngneat/until-destroy"
  );
  untilDestroyedImport?.remove();
}

Finally, you need to ensure the DestroyRef and takeUntilDestroyed imports are set then save the file.

{
  // add imports if needed
  if (fileMigrated) {
    const imports: [string, string[]][] = [
      ["@angular/core", ["inject", "DestroyRef"]],
      ["@angular/core/rxjs-interop", ["takeUntilDestroyed"]],
    ];
    setImports(file, imports);
    file.saveSync();
  }
}

export function setImports(
  sourceFile: morph.SourceFile,
  imports: [string, string[]][]
): void {
  imports.forEach(([moduleSpecifier, namedImports]) => {
    const moduleSpecifierImport =
      sourceFile
        .getImportDeclarations()
        .find(imp => imp.getModuleSpecifierValue() === moduleSpecifier) ??
      sourceFile.addImportDeclaration({
        moduleSpecifier,
      });

    const missingNamedImports = namedImports.filter(
      namedImport =>
        !moduleSpecifierImport
          .getNamedImports()
          .some(imp => imp.getName() === namedImport)
    );

    moduleSpecifierImport.addNamedImports(missingNamedImports);
  });
}

Demo

Demo & Complete Code: To run this code, open the terminal at the bottom and run "npx ts-node ./index.ts"

Navigate to examples folder where you'll see how the code is migrated.

Conclusion

The ts-morph API although simple and to the point it still might confuse you if you're not familiar with the TypeScript compiler API. You can read the Gentle Introduction To Typescript Compiler API to get a better understanding of the TypeScript compiler API.

Make sure to run the script on a clean repository (commit any code) as it will modify the code in place.

References

Gentle Introduction To Typescript Compiler API

ezzabuzaid — Sat, 18 Nov 2023 17:48:38 +0000

TypeScript extends JavaScript by adding types, thereby enhancing code quality and understandability through static type checking which enables developers to catch errors at compile-time rather than runtime.

The TypeScript team has built a compiler tsc to process TypeScript type annotations and emit JavaScript code, however, the compiler is not limited to just compiling TypeScript code to JavaScript, it can also be used to build tools and utilities around TypeScript.

In this article, you'll explore the TypeScript Compiler API, which is an integral part of the TypeScript compiler that exposes various functionalities, enabling you to interact with the compiler programmatically.

The article is organized into different use cases each of which will introduce you to a new aspect of the Compiler API, and by the end of the article, you'll have a thorough understanding of how the Compiler API works and how to use it to build your tools. Keep in mind the use cases are not complete, they're just to demonstrate the concept.

Please note that the article is based on my personal learning and research. If you identify any errors or inaccuracies, please let me know in the comments.

What is A Compiler?

A compiler is a specialized software program that translates source code written in one programming language into another language, usually machine code or an intermediate form. Compilers perform several tasks including lexical analysis, syntax analysis, semantic analysis, code generation, and more.

Compilers come in various forms, serving different needs, to my understanding TypeScript is a Source-to-source compiler, which means it takes TypeScript code and compiles it into JavaScript code.

What is The TypeScript Compiler?

The TypeScript Compiler (tsc) takes TypeScript code (JavaScript and type information) and compiles it into plain JavaScript as the result while in the process it performs type checking to catch errors at compile time rather than at runtime.

For the compilation to happen you need to feed the compiler a TypeScript file/code and the configuration file (tsconfig.json) to guide the compiler on how to behave.

// file.ts
function IAmAwesome() {}

// tsconfig.json
{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "outDir": "./dist",
    "strict": true
  }
}

Run the following command to compile the code:

tsc file.ts --project tsconfig.json

Depending on the configuration, the compiler will generate the following files:

dist
├── file.js
├── file.js.map
├── file.d.ts

JavaScript code: The compiler will output JavaScript code that can be executed later on.
Source map: A source map is a file that maps the code within a compressed file back to its original position in a source file to aid debugging. It is mostly used by the browser to map the code it executes back to its original location in the source file.
Declaration file: A file that provides type information about existing JavaScript code to enable other programs to use the values (functions, variables, ...) defined in the file without having to guess what they are.

The TypeScript compiler generates declaration files for all the code it compiles if enabled in tsconfig.json.

TypeScript Compilation Process: From Source Code to JavaScript, Declaration Files, and Source Maps

What is The TypeScript Compiler API?

The TypeScript Compiler API is an integral part of the TypeScript compiler that exposes various functionalities, enabling you to interact with the compiler programmatically to do stuff like

Manual type checking.
Code generation.
Transform TypeScript code at a granular level.

and more.

TypeScript Compiler API is a lot of interfaces, functions, and classes.

Why would you use the Typescript Compiler API?

Using the TypeScript Compiler API has several benefits, particularly for those interested in building tools around TypeScript. You could utilize the API

Write a Language Service Plugin
To do Static Code Analysis
Or even to build a DSL (Domain Specific Language).
Custom Pre/Post build scripts.
Code Modification/Migration.
Use it as a Front-End for other low-level languages.

Angular recently introduced the Standalone Components, which is a new way to write Angular components without the need to create a module. Angular team created a migration script that does this automatically, and it's using the Typescript Compiler API.

There are a few interesting projects that utilize the Typescript Compiler API, such as:

Use Case: Enforce One Class Per File

You're going to use the Typescript Compiler API to enforce one class per file. This is a common rule that is used in many codebases.

It might be a bit confusing at first, but don't worry, It'll get simpler as you go.

Hint: I strongly recommend you check the code again before going to the next use case.

const tsconfigPath = "./tsconfig.json"; // path to your tsconfig.json
const tsConfigParseResult = parseTsConfig(tsconfigPath);
const program = ts.createProgram({
  options: tsConfigParseResult.options,
  rootNames: tsConfigParseResult.fileNames,
  projectReferences: tsConfigParseResult.projectReferences,
  configFileParsingDiagnostics: tsConfigParseResult.errors,
});

/**
 * Apply class per file rule
 */
function classPerFile(file: ts.SourceFile) {
  const classList: ts.ClassDeclaration[] = [];

  // file.forEachChild is a function that takes a callback and
  // calls it for each direct child of the node

  // Loops over all nodes in the file and push classes to classList
  file.forEachChild(node => {
    if (ts.isClassDeclaration(node)) {
      classList.push(node);
    }
  });

  // If there is more than one class in the file, throw an error
  if (classList.length > 1) {
    throw new Error(`
      Only one class per file is allowed.
      Found ${classList.length} classes in ${file.fileName}
      File: ${file.fileName}
    `);
  }
}

const files = program.getSourceFiles();

// Loops over all files in the program and apply the classPerFile rule
files
  .filter(file => !file.isDeclarationFile)
  .forEach(file => classPerFile(file));

In this code you're doing the following:

Create a program from tsconfig so you've access to all files in the project.
Loop over all files in the program and apply the rule.
The rule is simple, if there is more than one class in the file, throw an error.

Demo for this use case: To run this code, open the terminal at the bottom and run "npx ts-node ./use-cases/enforce-one-class-per-file/"

Let's break it down, there are a few key terms that you need to know:

Program
Source File
Node
Declaration

TypeScript Program

When working with the TypeScript Compiler, one of the central elements you'll encounter is the Program object. This object serves as the starting point for many of the operations you might want to perform, like type checking, emitting output files, or transforming the source code.

The Program is created using the ts.createProgram function, which can accept a variety of configuration options, such as

options: These are the compiler options that guide how the TypeScript Compiler will behave. This could include settings like the target ECMAScript version, module resolution strategy, and whether to include type-checking errors, among others.
rootNames: This property specifies the entry files for the program. It is an array of filenames (.ts) that act as the roots from which the TypeScript Compiler will begin its operations.
projectReferences: If your TypeScript project consists of multiple sub-projects that reference each other, this property is used to manage those relationships.
configFileParsingDiagnostics: This property is an array that will capture any diagnostic information or errors that arise when parsing the tsconfig.json file. More on that later.

const tsconfigPath = "./tsconfig.json"; // path to your tsconfig.json
const tsConfigParseResult = parseTsConfig(tsconfigPath);

const program = ts.createProgram({
  options: tsConfigParseResult.options,
  rootNames: tsConfigParseResult.fileNames,
  projectReferences: tsConfigParseResult.projectReferences,
  configFileParsingDiagnostics: tsConfigParseResult.errors,
});

In this sample, a TypeScript program is created from tsconfig parsing results.

Source File

Writing code is actually writing text, you understand it because you know the language, the semantics, the syntax, etc. But the computer doesn't understand it, it's just a text.

The compiler will take this text and transform it into something that can be utilised. This transformation is called parsing and its output is called Abstract Syntax Tree (AST).

A source file is a representation of a file in your project, it contains information about the file, such as its name, path, and contents.

The AST is a tree-like data structure and as any tree, it has a root node. The root node is Source File.

Abstract Syntax Tree (AST)

The code you write is essentially a text that isn't useful unless it can be parsed. That parsing process produces a tree data structure called AST, it contains a lot of information like the name, kind, and position of the node in the source code.

The AST is used by the compiler to understand the code and perform various operations on it. For example, the compiler uses the AST to perform type-checking.

The following code:

function IAmAwesome() {}

will be transformed into the following AST:

{ // -> Source File Node
  "kind": 308,
  "statements": [ // -> Node[]
    { // -> Function Declaration Node
      "kind": 259,
      "name": { // -> Identifier Node
        "kind": 79,
        "escapedText": "IAmAwesome"
      },
      "parameters": [], // Node[]
      "body": { // Block Node
        "kind": 238,
        "statements": []
      }
    }
  ]

A visual breakdown of a TypeScript Abstract Syntax Tree (AST), illustrating the hierarchical structure of nodes representing a source file with a simple function declaration. Each node is annotated with its role and kind, detailing the organization of code into an AST for compiler processing.

Node

In AST, the fundamental unit is called a Node.

Kind: A numeric value that represents the specific type or category of that node. For instance:

FunctionDeclaration has kind 259
Block has kind 238

These numbers are exported in an enum called SyntaxKind

The node object has more than just these properties but right now we're only interested in a few, nonetheless, two additional important properties you might want to know about are:

Parent: This property points to the node that is the parent of the current node in the AST.

Flags: These are binary attributes stored as flags in the node. They can tell you various properties of the node, such as whether it's a read-only field or it has certain modifiers.

Declaration

Remember the use case you're trying to solve? enforce one class per file. To do that, you need to check if more than one class is declared in the file.

A declaration is a node that declares something, it could be a variable, function, class, etc.

class Test {
  runner: string = "jest";
}

In this example, we have two declarations:

Test -> ClassDeclaration
runner -> PropertyDeclaration

The key difference between a node and a declaration is that

A "Node" is a generic term that refers to any point in the AST, irrespective of what it represents in the source code.
A "Declaration" is a specific type of node that has a semantic role in the program: it introduces a new identifier and provides information about it.

Creating a variable const a = 1; is like saying "Hey compiler, I'm creating VariableDeclaration node with name a and value 1"

Statement

A statement is a node that represents a statement in the source code. A statement is a piece of code that performs some action, for example, a variable declaration is a statement that declares a variable.

let a = 1;

In this example the variable declaration let a = 1; is a statement.

Expression

An expression is a node in the code that evaluates to a value. For example

let a = 1 + 2;

The part 1 + 2 is an expression, specifically a binary expression

Another example

const add = function addFn(a: number, b: number) {};

The function ... is an expression, specifically a function expression.

More advanced example

const first = 1,
  second = 2 + 3,
  third = whatIsThird();

The whole code is a VariableStatement node.
first = 1, second = 2 + 3 and third = whatIsThird() are a VariableDeclaration nodes.
first, second, and third are Identifier nodes.
1, 2 + 3, and whatIsThird() are NumericLiteral, BinaryExpression, and CallExpression nodes respectively.

A diagram illustrating a TypeScript AST segment for a variable statement. It depicts a structure with a variable statement node branching into a variable declaration list, which further separates into individual variable declarations. Each declaration showcases different kinds of assignments: a number literal, a binary expression, and a call expression.

Use Case: No Function Expression

Let's take another example to recap what you've learned so far. You're going to use the Typescript Compiler API to enforce no function expression.

Function Declaration:

function addFn(a: number, b: number) {
  return a + b;
}

Function Expression

let addFnExpression = function addFn(a: number, b: number) {
  return a + b;
};

You need to ensure only the first one is allowed.

type Transformer = (
  file: ts.SourceFile
) => ts.TransformerFactory<ts.SourceFile>;
const transformer: Transformer = file => {
  return function (context) {
    const visit: ts.Visitor = node => {
      if (ts.isVariableDeclaration(node)) {
        if (node.initializer && ts.isFunctionExpression(node.initializer)) {
          throw new Error(`
          No function expression allowed.
          Found function expression: ${node.name.getText(file)}
          File: ${file.fileName}
        `);
        }
      }

      // visit each child in this node (look at the visitor node parameter)
      return ts.visitEachChild(node, visit, context);
    };

    // visit each node in the file
    return node => ts.visitEachChild(node, visit, context);
  };
};

const files = program.getSourceFiles();

files
  .filter(file => !file.isDeclarationFile)
  .forEach(file =>
    ts.transform(file, [transformer(file)], program.getCompilerOptions())
  );

I know this isn't like the first example, but it's similar.

In the first use case, it was enough to loop over the first level nodes in the file, but in this use case, you need to loop over all nodes in the file, the nested ones as well.

The main logic is within the visit function, it checks if the node is a VariableDeclaration and whether its initializer is a FunctionExpression, and if so, then throws an error.

The looping over the files is the same but with a slight difference, you're using ts.transform API.

A few key terms that you need to know:

Transformer
Visitor
Context

Demo for this use case: To run this code, open the terminal at the bottom and run "npx ts-node ./use-cases/no-function-expression"

Transformer

As the name implies, the transformer function can transform the AST (the code) in any way you want. In this example, you're using it to enforce a rule, however, instead of throwing an error, you can transform the code to fix the error (more in the next use case)

Visitor

The visit function is a simpler version of what is called the Visitor Pattern, an essential part of how the TypeScript Compiler API works. Actually, you'll see that design pattern whenever you work with AST, Hey at least I did!

A "visitor" is basically a function you define to be invoked for each node in the AST during the traversal. The function is called with the current node and has few return choices.

Return the node as is (no changes).
Return a new node of the same kind (otherwise might disrupt the AST) to replace it.
Return undefined to remove the node entirely.
Return a visitor ts.visitEachChild(node, visit, context) which will visit the node children if have.

Context

The context -TransformationContext- is an object that contains information about the current transformation and has a few methods that you can use to perform various operations like hoistFunctionDeclaration and startLexicalEnvironment. You'll learn more about it in the next use case.

Use Case: Replace Function Expression With Function Declaration

Same as the previous use case, but instead of throwing an error, you're going to transform function expression to function declaration.

See how the AST for the function expression looks like

const transformer: Transformer = function (file) {
  return function (context) {
    const visit: ts.Visitor = node => {
      if (ts.isVariableStatement(node)) {
        const varList: ts.VariableDeclaration[] = [];
        const functionList: ts.FunctionExpression[] = [];

        // collect function expression and variable declaration
        for (const declaration of node.declarationList.declarations) {
          // the initializer is expression after assignment operator
          if (declaration.initializer) {
            if (ts.isFunctionExpression(declaration.initializer)) {
              functionList.push(declaration.initializer);
            } else {
              varList.push(declaration);
            }
          }
        }

        for (const functionExpression of functionList) {
          // create function declaration out of function expression
          const functionDeclaration = ts.factory.createFunctionDeclaration(
            functionExpression.modifiers,
            functionExpression.asteriskToken,
            functionExpression.name as ts.Identifier,
            functionExpression.typeParameters,
            functionExpression.parameters,
            functionExpression.type,
            functionExpression.body
          );

          // hoist the function declaration to the top of the containing scope (file)
          context.hoistFunctionDeclaration(functionDeclaration);
        }

        // if the varList (non function expression) is same as the original variable statement, return the node as is.
        // it means there is no function expression in the variable statement
        if (varList.length === node.declarationList.declarations.length) {
          return node;
        }

        // if the varList (non function expression) is empty, return undefined to remove the variable statement node
        if (varList.length === 0) {
          return undefined;
        }

        return ts.factory.updateVariableStatement(
          node,
          node.modifiers,
          ts.factory.createVariableDeclarationList(varList)
        );
      }

      return ts.visitEachChild(node, visit, context);
    };

    return node => {
      // Start a new lexical environment when beginning to process the source file.
      context.startLexicalEnvironment();

      // visit each node in the file.
      const updatedNode = ts.visitEachChild(node, visit, context);

      // End the lexical environment and collect any declarations (function declarations, variable declarations, etc) that were added.
      const declarations = context.endLexicalEnvironment() ?? [];
      const statements = [...declarations, ...updatedNode.statements];

      return ts.factory.updateSourceFile(
        node,
        statements,
        node.isDeclarationFile,
        node.referencedFiles,
        node.typeReferenceDirectives,
        node.hasNoDefaultLib,
        node.libReferenceDirectives
      );
    };
  };
};

A few key terms that you need to know:

Factory
Lexical Environment
Hoisting

Demo for this use case: To run this code, open the terminal at the bottom and run "npx ts-node ./use-cases/replace-function-expression-with-declaration"

Factory

The ts.factory object contains a set of factory functions that can be used to create new nodes or update existing ones. You used ts.factory.createFunctionDeclaration to create a new function declaration node using the information from the function expression node and ts.factory.updateSourceFile to update the source file node statements while keeping the other properties intact.

The factory functions are useful to manipulate the AST in a safe way, without worrying about the details of the AST structure.

Lexical Environment

The lexical environment refers to the scope or context in which variables and function declarations are hoisted and managed during the transformation process.

Here are the key points regarding the lexical environment in your transformer function:

Scope Management: The lexical environment helps in managing the scope of variables and function declarations. When you start a new lexical environment with context.startLexicalEnvironment(), you are essentially marking the beginning of a new scope. When you end it with context.endLexicalEnvironment(), you are closing off that scope and collecting any declarations that were hoisted to this scope during the transformation process.
Hoisting: The lexical environment provides the facilities for hoisting function declarations using context.hoistFunctionDeclaration(). Hoisting in this scenario means moving function declarations to the appropriate scope in the source file (the current running scope). For instance, if you have a function declaration inside a function, it will be hoisted to the top of the function, if you have a function declaration inside a for loop, it will be hoisted to the top of the for loop, and so on.

The previous code only handles the function expression at the top level

Declaration Collection: The lexical environment also collects the hoisted declarations through context.endLexicalEnvironment(). This method returns an array of Statement nodes that represent the declarations hoisted to the current lexical environment, which can then be included in the SourceFile node.

These methods ensure that the transformer has a mechanism to correctly manage scope and collect hoisted declarations, which is crucial for accurately transforming code while maintaining correct scoping and semantics.

Use Case: Detect Third-Party Classes Used as Superclasses

The last use case is a bit more complex, you're going to use the Typescript Compiler API to detect third-party classes used as superclasses, not to do anything about it, just detect them which means you don't need to use the transformer function.

const trackThirdPartyClassesUsedAsSuperClass: ts.Visitor = node => {
  if (ts.isClassDeclaration(node)) {
    const superClass = (node.heritageClauses ?? []).find(
      heritageClause => heritageClause.token === ts.SyntaxKind.ExtendsKeyword
    );

    // Not intrested in classes that don't have super class.
    if (!superClass) {
      return node;
    }

    // in case of class declaration, there will always be one heritage clause (extends)
    const superClassType = superClass.types[0].expression;

    // Get the type checker
    const typeChecker = program.getTypeChecker();

    const symbol = typeChecker.getSymbolAtLocation(superClassType);
    if (!symbol) {
      return undefined;
    }

    const superClassDeclaration = (symbol.declarations ?? []).find(
      ts.isClassDeclaration
    );

    if (!superClassDeclaration) {
      // In this case this should never happen (more on that later),
      // but it's here just to satisfy typescript.
      return node;
    }

    const thisSourceCodeFile = node.getSourceFile();
    const sourceCodeInfo = {
      fileName: thisSourceCodeFile.fileName,
      className: node.name?.text,
      ...ts.getLineAndCharacterOfPosition(thisSourceCodeFile, node.pos),
    };
    const thirdPartyCodeInfo = {
      fileName: superClassDeclaration.getSourceFile().fileName,
      className: superClassType.getText(),
    };
    console.log(`
        Class: "${sourceCodeInfo.className}"
        Filename: "${sourceCodeInfo.fileName}"
        SuperClass: "${thirdPartyCodeInfo.className}"
        Filename: "${thirdPartyCodeInfo.fileName}"
        Line: "${sourceCodeInfo.line}"
        Column: "${sourceCodeInfo.character}"
  `);
  }

  // visit each child in this node
  return ts.forEachChild(node, trackThirdPartyClassesUsedAsSuperClass);
};

const files = program.getSourceFiles();

files
  .filter(file => !file.isDeclarationFile)
  .forEach(file => {
    ts.forEachChild(file, trackThirdPartyClassesUsedAsSuperClass);
  });

Key terms that you need to know:

HeritageClauses
Type
Type Checker
Symbol

Demo for this use case: To run this code, open the terminal at the bottom and run "npx ts-node ./use-cases/3rd-party-classes/"

HeritageClauses

When you extend a class or implement an interface, you use a clause called a heritage clause and it's represented by the HeritageClause interface.

interface HeritageClause {
  readonly token: SyntaxKind.ExtendsKeyword | SyntaxKind.ImplementsKeyword;
  readonly types: NodeArray<ExpressionWithTypeArguments>;
}

_ token is the keyword used in the clause, it can be extends or implements._
_ types is an array of ExpressionWithTypeArguments nodes, which represents the types specified in the clause._

types will have a single item if you're extending a class or multiple items if you're implementing interfaces.

In this example, you're only interested in the extends hence const [superClass] = node.heritageClauses

Type

It is the "Type" part in TypeScript 😏, representing the type information of a particular symbol in the AST.

A Type object is associated with a Symbol to represent the type of the symbol (named entity).

interface Type {
  flags: TypeFlags;
  symbol: Symbol;
}

flags: The flags associated with the type which can be used to determine the kind of type -number, string, boolean, etc.-
symbol: The symbol associated with the type (scroll a bit to read about symbols)

Type Checker

It is an essential part of the TypeScript Compiler API, acting as the engine that powers the type system in TypeScript. The Type Checker traverses the AST, examining the Type and Symbol of nodes to ensure they adhere to the type annotations and constraints defined in the code.

The Type Checker can be accessed via the program object, and it provides a rich set of methods to retrieve type information, check types, and obtain symbol information, among other functionalities.

const typeChecker = program.getTypeChecker();

In this use case, you used typeChecker to get the type of the class declaration node and the symbol associated with it, which you then used to get the value declaration of the symbol.

const type = typeChecker.getTypeAtLocation(node);
const symbol = type.symbol;
const valueDeclaration = symbol.valueDeclaration;

TypeChecker has a lot of methods to facilitate working with types, symbols, and nodes.

Symbol

The term "symbol" refers to a named entity in the source code that could represent variables, functions, classes, and so forth.

named entity is a name that is used to identify something. It could be a variable name, function name, class name, etc. In other words any Identifier Node.

A symbol is represented by the Symbol interface, which has the following properties:

interface Symbol {
  flags: SymbolFlags;
  escapedName: __String;
  declarations?: Declaration[];
  valueDeclaration?: Declaration;
  members?: SymbolTable;
  exports?: SymbolTable;
  globalExports?: SymbolTable;
}

flags: The flags associated with the symbol which can be used to determine the type of symbol -variable, function, class, interface, etc.-
escapedName: The name of declarations associated with the symbol.
declarations: List of declarations associated with the symbol, think of function/method override.
valueDeclaration: Points to the declaration that serves as the "primary" or "canonical" declaration of the symbol. In simpler terms, it's the declaration that gives the symbol its "value" or "meaning" within the code. For example, if you have a variable initialized with a function, the valueDeclaration would point to that function expression.
members: Symbol table that contains information about the properties or members of the symbol. For instance, if the symbol represents a class, members would contain symbols for each of the class's methods and properties.
exports: Similar to members, but this is more relevant for modules. It contains the symbols that are exported from the module.
globalExports: This is another symbol table but it's used for global scope exports.

Let's take the following example

class Test {
  runner: string = "jest";
}

The symbol for class "Test" will have the following properties:

const classTestSymbol = {
  flags: 32,
  escapedName: "Test",
  declarations: [ClassDeclaration],
  valueDeclaration: ClassDeclaration,
  members: {
    runner: // Symbol for runner property
  },
};

Symbols let the type checker look up names and then check their declarations to determine types. It also contains a small summary of what kind of declaration it is -- mainly whether it is a value, a type, or a namespace.

Diagnostic

When the TypeScript compiler runs, it performs several checks on the code. If it encounters something that doesn't align with the language rules or the project's configuration, it creates an object that represents identified issues the compiler has found in the code.

Each diagnostic object contains information about the nature of the problem, including:

The file in which the issue was found.
The start and end position of the relevant code.
A message describing the issue.
A diagnostic category (error, warning, suggestion, or message).
A code that uniquely identifies the type of diagnostic.

The issues can range from syntax errors and type mismatches to more subtle semantic issues that might not be immediately apparent. Mainly, there are two kinds of diagnostics:

Syntactic Diagnostics: These are errors that occur when the compiler parses the code and encounters something that doesn't align with the language syntax. For instance, if you have a missing semicolon or a missing closing brace, the compiler will generate a syntactic diagnostic.
Semantic Diagnostics: These are errors that occur when the compiler performs type-checking on the code and encounters something that doesn't align with the type annotations or constraints defined in the code. For instance, if you have a variable of type string and you try to assign a number to it, the compiler will generate a semantic diagnostic.

Here's a simple example of how you might encounter a diagnostic in TypeScript:

let greeting: string = 42;

The TypeScript compiler will generate a diagnostic message for the above code, indicating that the type 'number' is not assignable to the type 'string'.

To obtain semantic diagnostics, you can use the getSemanticDiagnostics method on the program object. This method returns an array of diagnostic objects, each of which contains information about the issue, such as the file in which it was found, the start and end position of the relevant code, and a message describing the issue.

program.getSemanticDiagnostics();

Will return

[
  {
    "start": 0,
    "length": 8,
    "code": 2322,
    "category": 1,
    "messageText": "Type 'number' is not assignable to type 'string'.",
    "relatedInformation": undefined
  }
]

To obtain syntactic diagnostics

// missing expression after assignment operator
let greeting: string =;

// get syntactic diagnostics
program.getSyntacticDiagnostics();

Will return

[
  {
    "start": 0,
    "length": 1,
    "messageText": "Expression expected.",
    "category": 1,
    "code": 1109,
    "reportsUnnecessary": undefined
  }
]

Diagnostics are not just for the compiler; they are also used by IDEs and editors to provide real-time feedback to developers. For example, when you see red squiggly lines under code in Visual Studio Code, that's the editor using TypeScript diagnostics to indicate an issue.

Here's a snippet that demonstrates how to retrieve diagnostics for a given TypeScript program before emitting the output files:

// Retrieve and concatenate all diagnostics
const allDiagnostics = ts.getPreEmitDiagnostics(program);

// Iterate over diagnostics and log them
allDiagnostics.forEach(diagnostic => {
  if (diagnostic.file) {
    let { line, character } = diagnostic.file.getLineAndCharacterOfPosition(
      diagnostic.start!
    );
    let message = ts.flattenDiagnosticMessageText(diagnostic.messageText, "\n");
    console.log(
      `${diagnostic.file.fileName} (${line + 1},${character + 1}): ${message}`
    );
  } else {
    console.log(ts.flattenDiagnosticMessageText(diagnostic.messageText, "\n"));
  }
});

You might have noticed the function says pre-emit getPreEmitDiagnostics which implies there will be diagnostics after the emit process, and you're right, there are.

The post-emit diagnostics are generated after the emit process, and they are used to report issues that might arise during the emit process. For instance, there might be issues with generating source maps or writing output files.

Printer

TypeScript provides a printer API that can be used to generate source code from an AST, it is useful when you want to perform transformations on the AST and then generate the corresponding source code in text format.

function print(file: ts.Node, result: ts.TransformationResult<ts.SourceFile>) {
  const printer = ts.createPrinter({
    newLine: ts.NewLineKind.LineFeed,
  });
  const transformedSource = printer.printNode(
    ts.EmitHint.Unspecified,
    result.transformed[0],
    file
  );
  return transformedSource;
}

In use case 3, you'll need to write the modified AST to the original file back, to do that you'll need to use the printer API to get the AST into text.

Worth mentioning that the printer doesn't preserve the original formatting of the source code.

For this exact reason I don't recommend direct use of the compiler API to modify source code, instead, I use ts-morph which is a wrapper around the compiler API that preserves the original formatting.

How The Compiler Works

This flowchart outlines the TypeScript compilation process, detailing each step from parsing to emitting executable JavaScript code. It highlights phases such as lexical and syntax analysis, semantic analysis (including scope, symbol creation, and type checking), and transformation (converting TypeScript to JavaScript and downleveling to support older JavaScript versions). Diagnostic reporting runs alongside these processes, and during the emitting phase, JavaScript code, source maps, and declaration files are generated, with an option to halt emission on error.

The following is a high-level overview.

Lexical Analysis

In this stage, the compiler takes the source code and breaks it down into a series of tokens, a token is a character or sequence of characters that can be treated as a single unit.

function IAmAwesome() {}

Will be transformed into the following tokens:

<function, keyword>
<IAmAwesome, identifier>
<(, punctuation>
<), punctuation>
<{, punctuation>
<}, punctuation>

The module/function that does this is called the Scanner/Tokenizer which is responsible for scanning the source code and generating the tokens. When the term scanner/tokenizer is used, imagine a while loop that loops over the source code character by character and switch case to determine the token type.

function scan(sourceCode: string) {
  const tokens: Token[] = [];
  let currentChar: string;
  let index = -1;
  do {
    index = index + 1;
    currentChar = sourceCode[index];
  } while (index < sourceCode.length);
  {
    switch (currentChar) {
      case "{":
        // ...
        break;
      case "}":
        // ...
        break;
      // ...
    }
  }
}

Syntax Analysis

In this stage, the compiler takes the tokens generated in the previous stage and uses them to build a tree-like structure called an Abstract Syntax Tree (AST). The AST represents the syntactic structure of the source code.

The part of the compiler that does this is called the Parser which is responsible for parsing the tokens and building the AST that is then used by the semantic analysis stage.

So the Tokensizer generates the tokens and the Parser builds the AST. You may be wondering (are you?) how the parser knows what the AST should look like, well, it's defined in the Language Grammar.

A grammar is a set of rules that define the syntax of a language. Think of English grammar, it defines the rules of the English language, such as how to form a sentence, how to use punctuation, etc.

Grammar is usually represented in a form called Backus-Naur Form (BNF) or Antlr, which is a notation that describes the grammar of a language.

A simple example of grammars:

<identifier> ::= "a" | "b" | "c" | ... | "z"
<function> ::= "function" <identifier>* "(" ")" "{" "}"

The parser will use the grammar to build the AST, in case of writing a function, the parser will look for the function keyword, then the identifier, the (, the ), the {, then the } in case of any of these tokens is missing, the parser will throw an error. Read More on parsing

Another example

 <postal-address> ::= <name-part> <street-address> <zip-part>

    <name-part> ::= <personal-part> <last-name> <opt-suffix-part> <EOL> | <personal-part> <name-part>

  <personal-part> ::= <first-name> | <initial> "."

 <street-address> ::= <house-num> <street-name> <opt-apt-num> <EOL>

       <zip-part> ::= <town-name> "," <state-code> <ZIP-code> <EOL>

<opt-suffix-part> ::= "Sr." | "Jr." | <roman-numeral> | ""
    <opt-apt-num> ::= "Apt" <apt-num> | ""

Semantic Analysis

At this stage, the TypeScript compiler performs semantic analysis on the AST generated during the parsing stage to ensure that the syntactically correct code is also semantically valid. For instance, consider the following TypeScript snippet:

let a = 1;
a();

While the parser confirms this code is free of syntax errors, semantic analysis reveals a type error. This code tries to invoke a as if it were a function, which is semantically incorrect since a is a number.

The semantic analysis phase comprises two key processes: Binding and Type Checking.

Binding: It walks the AST and creates symbols for all declarations it encounters, such as variables, functions, classes, etc.

A Symbol is created for each declaration, capturing its name, type, scope, and additional attributes.
The symbol is then stored in a SymbolTable, which is essentially a map that associates symbols with their names and is scoped to a specific block, function, or module.

The internal representation of a SymbolTable in TypeScript might look something like this:

type SymbolTable = Map<__String, Symbol>;

The function to create such a table could be as follows:

export function createSymbolTable(symbols?: readonly Symbol[]): SymbolTable {
  const result = new Map<__String, Symbol>();
  if (symbols) {
    for (const symbol of symbols) {
      result.set(symbol.escapedName, symbol);
    }
  }
  return result;
}

Emitting

This is where the compiler generates the output files, such as JavaScript files, source maps, and declaration files after the AST has been successfully created and type-checked.

project.emit();

Recall diagnostics? the emit process will fail if there are any errors but that can be overridden by setting noEmitOnError to true.

ESLint

Using ESLint is a better option if you want to enforce rules on your codebase because it's specifically built for that purpose. However, if you want to build a tool that does something more complex, then the Typescript Compiler API is the way to go.

I have written a guide for you to learn more.

Conclusion

The TypeScript Compiler API is a powerful tool that can be used to build tools around TypeScript. It provides a rich set of functionalities that can be used to perform various operations on the source code, such as type checking, code generation, and AST transformation.

It's a bit confusing at first, but once you get the hang of it, you'll be able to build some cool stuff.

You can also generate TypeScript code using the API, think of protobuf to TypeScript code generator.

Next Steps

Practice, practice, practice. The best way to learn is to practice what you've learned here. Try to build a tool that does something useful for you or your team.

Think of manual tasks -typescript-related- that you do often and try to automate them. Stuff that is error-prone, time-consuming, or just boring.

I'm writing a few other blog posts that will help you get started with the Typescript Compiler API, follow me on Twitter to get notified when they're out.

References

Related Projects

Credits

The graph tree was generated using Mermaid JS
The drawing was created using Excalidraw
Image caption were generated using ChatGPT

Real-time OpenAI Response Streaming with Node.js

ezzabuzaid — Wed, 25 Oct 2023 12:36:41 +0000

You know how ChatGPT continuously show the results as it becomes available -generated-, It doesn't wait for the whole response to get together, rather it shows what the model is generating right away.

Smooth user experience: User doesn't have to wait till the whole response is generated.
Easier to read: When ChatGPT first released I liked that it respond as if you're talking to someone, which read different to other form of writing.
Reduce memory usage: This is a benefit of streaming in general, you offload the data without having to buffer it in memory.

In this article, you're going to learn how you can stream OpenAI response just like how ChatGPT does it.

I'm assuming you've basic knowledge in Node.js, and you're familiar with JavaScript features like async/await, for await loop and async iterators.

Problem

There are few ways to stream data from the server:

You're going to use the Fetch API; simple and more suitable for this use case.

Solution

I divided the solution into two parts, the server side and the client side.

The Server Side

You're going to build a tiny Node.js server that uses OpenAI chat completion API.

Let's make simple Node.js server first. No routing framework, no fancy stuff.

import http from "http";

const server = http.createServer();
server.listen(3000);

server.on("request", async (req, res) => {
  switch (req.url) {
    case "/":
      res.write("Hello World");
      res.end();
      break;
    default:
      res.statusCode = 404;
      res.end();
  }
});

The res.write will send data to the client, it can be called many times only before the res.end is called as it will close the connection.

import OpenAI from "openai";
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const config = {
  model: "gpt-4",
  stream: true,
  messages: [
    {
      content: "Once upon a time",
      role: "user",
    },
  ],
};
const completion = await openai.chat.completions.create(config);

The stream option is what you're looking for, it will stream the response body.

The completion object implements the AsyncIterable interface, which means you can use for await loop to iterate over the response body.

for await (const chunk of completion) {
  console.log(chunk);
}

Putting it all together.

import OpenAI from "openai";
import http from "http";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const server = http.createServer();
server.listen(3000);

server.on("request", async (req, res) => {
  switch (req.url) {
    case "/":
      const config = {
        model: "gpt-4",
        stream: true,
        messages: [
          {
            content: "Once upon a time",
            role: "user",
          },
        ],
      };
      const completion = await openai.chat.completions.create(config);

      res.writeHead(200, {
        "Content-Type": "text/plain; charset=utf-8",
      });

      for await (const chunk of completion) {
        const [choice] = chunk.choices;
        const { content } = choice.delta;
        res.write(content);
      }

      res.end();
      break;
    default:
      res.statusCode = 404;
      res.end();
  }
});

In the context of streaming data, a chunk refers to a piece or fragment of data that is handled, processed, or transmitted individually as part of a larger stream of data.

The Client Side

There are few ways to receive data coming from the server while using the Fetch API.

Using for await loop over the body stream.
Manually using response.body!.getReader().
Using 3rd party libraries like RxJS.

Of course, those are not the only ways.

Using Async Iterator

The approach I like the most, it is declarative and to the point.

You might want to check browser compatibility iterating over ReadableStream.
TypeScript also doesn't support ReadableStream as an iterable.

const inputEl = document.querySelector("input");
const resultEl = document.querySelector("p");

inputEl.addEventListener("input", async () => {
  const response = await fetch("http://localhost:3000");

  let total = "";
  const decoder = new TextDecoder();
  for await (const chunk of response.body!) {
    const decodedValue = decoder.decode(chunk);
    total += decodedValue;
    resultEl.textContent = total;
  }
});

Using While Loop

You'll most likely see this approach in the wild, it is a bit imperative and verbose but it is supported by most browsers.

const inputEl = document.querySelector("input");
const resultEl = document.querySelector("p");

inputEl.addEventListener("input", async () => {
  const response = await fetch("http://localhost:3000");

  // Using While Loop
  let total = "";
  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  while (true) {
    const { done, value: chunk } = await reader.read();
    if (done) break;

    const decodedValue = decoder.decode(chunk);
    total += decodedValue;
    resultEl.textContent = total;
  }
});

Using RxJS

const inputEl = document.querySelector("input");
fromEvent(inputEl, "input")
  .pipe(
    switchMap(() => fetch("http://localhost:3000")),
    switchMap(response => response.body.pipeThrough(new TextDecoderStream())),
    scan((acc, chunk) => acc + chunk, "")
  )
  .subscribe(total => {
    resultEl.textContent = total;
  });

RxJS is a great library, but it is a bit overkill for this use case, unless you're already using it in your project.

Decoding The Data

A word on the decoding part, you need to decode the data coming from the server, as it is encoded using Uint8Array by default.

To decode the data, you need to use TextDecoder API.

const decoder = new TextDecoder();

By default it uses utf-8 encoding, which is what you need in most cases.

const decoder = new TextDecoder("utf-8");

The decode method accepts buffer (An ArrayBuffer, a TypedArray, or a DataView) and returns a string.

const decodedValue = decoder.decode(chunk);

Another options is using TextDecoderStream which is a transform stream that takes a stream of Uint8Array chunks as input and emits a stream of strings.

The previous implementation can be rewritten as follows.

let total = "";
const decoderTransformStream = new TextDecoderStream();
const reader = response.body.pipeThrough(decoderTransformStream).getReader();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  total += value;
  resultEl.textContent = total;
}

Canceling The Request

To cancel the request in progress, you need to use the AbortController API.

The Client Side

Cancel the request in progress if you allow the user to generate over the request in progress.

let controller = new AbortController();
let inProgress = false;
inputEl.addEventListener("input", async () => {
  if (inProgress) {
    controller.abort();
    controller = new AbortController();
  }

  inProgress = true;

  const response = await fetch("http://localhost:3000", {
    signal: controller.signal,
  });
  // ... rest of the code

  inProgress = false;
});

Deliberately cancel the request in progress.

const stopGenerationButtonEl = document.querySelector("button");

stopGenerationButtonEl.addEventListener("click", () => {
  if (inProgress) {
    controller.abort();
  }
});

RxJS has a built-in support for canceling the request in progress using switchMap operator.

To deliberately cancel the request in progress, you can use takeUntil operator.

const stopGeneration$ = new Subject();
fromEvent(inputEl, "input").pipe(
  switchMap(() =>
    fetch("http://localhost:3000").pipe(
      takeUntil(fromEvent(stopGenerationButtonEl, "click"))
    )
  )
  // ... rest of the code
);

The Server Side

You need to listent to the close event on the request object which will be emitted when the client closes the connection.

Aborting the request is one of the reasons a connection might close, of other reasons might be the client closed the tab or the browser crashed.

req.on("close", () => {
  completion.controller.abort();
});

The close listener is the place where you want to do clean up, besides aborting OpenAI request you might want to close any database connection or any other resource you're using.

Handling Errors

Following OpenAI API error codes, you need to handle the most likely error you'll get, at least.

Rate Limit Reached - Quota Exceeded (same error code)
Model Is Overloaded (will be in form of a server error)
Token Limit Reached

To commuincate the errors to the client, you're going to prefix the error with ERROR string to simplify handling the error on the client side.

You cannot alter the response status code at this point, it will always be 200 even if the error is in form of a server error.

Server Side

Rate Limit Reached
Model Is Overloaded

try {
  const completion = await openai.chat.completions.create(config);
} catch (error) {
  if (error instanceof RateLimitError) {
    res.end("ERROR:rate_limit_exceeded");
  } else if (error instanceof InternalServerError) {
    res.end("ERROR:internal_server_error");
  }
}

Token Limit Reached

for await (const chunk of completion) {
  const [choice] = chunk.choices;
  const { content } = choice.delta;

  if (choice.finish_reason === "length") {
    res.write("ERROR:token_limit_reached");
  } else {
    res.write(content);
  }
}

Client Side

while (true) {
  const { done, value: chunk } = await reader.read();
  if (done) break;

  const decodedValue = decoder.decode(chunk);

  switch (decodedValue) {
    case "ERROR:rate_limit_exceeded":
      resultEl.textContent = "Rate Limit Exceeded";
      break;
    case "ERROR:internal_server_error":
      resultEl.textContent = "Internal Server Error";
      break;
    case "ERROR:token_limit_reached":
      resultEl.textContent = "Token Limit Reached";
      break;
    default:
      total += decodedValue;
      resultEl.textContent = total;
  }

  total += decodedValue;
  resultEl.textContent = total;
}

Backpressure & Buffering

What if the client is slow to consume the data coming from the server -OpenAI-, you don't want to buffer too much data in the server memory. Things can get pretty bad when you have many concurrent requests.

In Node.js you can check if the client is ready to receive more data by checking the return value of res.write method.

res.write returns false if the buffer is full, that can be known synchronously; to know when the buffer is ready to receive more data, you need to listen to the drain event on the response object.

// ... rest of the code
const bufferFull = res.write(content) === false;
if (bufferFull) {
  await new Promise(resolve => res.once("drain", resolve));
}

That is actually less than a solution, merely a workaround. A better solution would be to store the data somewhere else, like in-memory database, and stream the data from there when the client is ready to receive more.

More On Streams

Beside the solutions you read, stream is vast concept and have many components, mainly categoriesd into the following.

ReadableStream: Represents a source of data, from which you can read. In the client-side, the server can be considered as a readable stream that you access using the Fetch API. In the server-side, the OpenAI API can be considered as a readable stream.
WriteableStream: Opposite to ReadableStream, a WritableStream represents a destination for data, where you can write. In the server-client setup, it represent the response object where you are writing the data to be sent to the client.
TransformStream: A TransformStream is a type of stream that sits between a ReadableStream and a WritableStream, transforming or processing the data as it passes through. This can be utilized for various purposes such as modifying, filtering, or formatting the data before it reaches the client. For instance, if there's a need to format the OpenAI's response before sending it to the client, a TransformStream could be employed.

When combined together, they form a pipeline, where the data flows from the source to the destination, passing through the transform stream. For instance, data from OpenAI's API (ReadableStream) could be passed through a TransformStream for formatting or filtering, and then written to the client (WritableStream).

Conclusion

So I hope you have good idea how to mimic ChatGPT in your project, and how to stream data from the server to the client.

Streaming is fun but you've to keep an eye on the memory usage on both client and server, you don't want to buffer too much data in the memory, try not to hold any data, always stream quick and early.

In case you're using more complex form of Prompt Engineering, like ReAct or Generated Knowledge that relys on the output of a previous LLM request, and have processing within like collecting user info, semantic search, you might then want use Server-Sent Events or WebSockets instead of Fetch API.

Next Steps

Put that in practice if you haven't already.
Use some kind of in-memory database to store the data and stream it from there.
Try to stress load the server and see how it behaves.

References

Node.js Stream
Streams API
An Introduction to Streaming on the Web
https://web.dev/articles/eventsource-basics
https://jakearchibald.com/2016/streams-ftw/#streams-the-fetch-api I have this bookmarked since 2016 😆

How To Build a Typeahead Component Using RxJS

ezzabuzaid — Tue, 19 Sep 2023 19:57:14 +0000

You know when you start typing in a search box and it starts suggesting things to you? That's called typeahead. It's a great way to help users find what they're looking for. In this article, you'll learn how to build a typeahead component using RxJS.

Problem

From end-user perspective, the problem is that they want to search for something and they want to see the results as they type. From developer perspective, the problem is that you need to optimize the search logic to not pressure the server with too many requests.

Solution

To balance the user experience and the performance, you need to make sure that you don't make too many requests. That can be done using RxJS to debounce the user input and only make a request when the user stops typing for a certain amount of time.

Here is how it'll be used in the end:

const search$ = fromEvent(searchInputEl, "input").pipe(
  map(event => event.target.value),
  typeahead({
    minLength: 3,
    debounceTime: 250,
    loadFn: searchTerm => {
      const searchQuery = searchTerm ? `?title_like=^${searchTerm}` : "";
      return fetch(`https://jsonplaceholder.typicode.com/posts${searchTerm}`);
    },
  }),
  // convert the response to json
  switchMap(response => response.json())
);

Getting Started

Before you start, you need to install RxJS.

npm install rxjs

Note: I'm using TypeScript primarily for clarity in showing what options are available through types. You're free to omit them, but if you do want to use types, I'd suggest opting for a framework that has built-in TypeScript support.

Typeahead Operator

You'll write a custom operator that will take an object with the following properties:§

Options Interface

interface ITypeaheadOperatorOptions<Out> {
  /**
   * The minimum length of the allowed search term.
   */
  minLength: number;
  /**
   * The amount of time between key presses before making a request.
   */
  debounceTime: number;
  /**
   * Whether to allow empty string to be treated as a valid search term.
   * Useful for when you want to show defaul results when the user clears the search box
   *
   * @default true
   */
  allowEmptyString?: boolean;

  /**
   * The function that will be called to load the results.
   */
  loadFn: (searchTerm: string) => ObservableInput<Out>;
}

Typeahead Operator

export function typeahead<Out>(
  options: ITypeaheadOperatorOptions<Out>
): OperatorFunction<string, Out> {
  return source => {
    return source.pipe(
      ...operators
      // The implementation goes here
    );
  };
}

The typeahead custom operator accepts options/config object and returns an operator function that takes an observable of "search term" and returns an observable of "result".

The result is represented by the generic type Out which is the type of the result returned by the loadFn function.

Scenario 1: Typical Typeahead

Note: Valid search term is a search term that have been still for a certain amount of time (debounceTime) and satisfies the minimum length (e.g. at least 3 char).

The user types valid search term and the request is sent.
The user types a new valid search term before the current request is finished, the current request is canceled and a new one is sent with the new search term.
The user types a valid search term then before the debounce time is up, the user reverts back to the previous search term, no request is sent.

return source.pipe(
  debounceTime(options.debounceTime),
  filter(value => typeof value === "string"),
  filter(value => {
    if (value === "") {
      return options.allowEmptyString ?? true;
    }
    return value.length >= options.minLength;
  }),
  distinctUntilChanged(),
  switchMap(searchTerm => options.loadFn(searchTerm))
);

debounceTime: Think of it as sliding window. It will wait for a certain amount of time before emitting the last value. If a new value comes in before the time is up, it will reset the timer and the window will start over. This is useful for preventing requests with every keystroke.

filter: The first filter will only pass values that are of type string, it might sound reddundant but it's necessary because the debounceTime operator might emit null when the source observable completes. The second filter will only pass values that are longer than the minimum length or empty string (if allowed).

Note: You'll need empty string to be treated as a valid search term if you want to show default results when the user clears the search box or when the user opens the page/dropdown for the first time.

distinctUntilChanged: It will only emit a value if it's different from the previous one (default behavior). This is useful for preventing requests with same search term (duplicate request). For example, if the user types "rxjs", results are fetched. If the user types "rxjs" again, no need to fetch the results again.

switchMap: Cancels the previous observable and subscribe to the new one. If a user types new search term before the current request is finished, it will cancel the current request and start a new one.

For example, if the user types "Typeahead" and then types "Operator" before the request for "Typeahead" is finished, it will cancel the request for "Typeahead" and send a new one for "Typeahead Operator". Only in the case that the request for "Typeahead" took longer than the debounce time.

Scenario 2: Cache Results

Most of the use cases I've personally seen might benefit from caching the results, especially if the results are not going to change frequently.

You can do that by caching the in-flight observables using the shareReplay operator.

const cache: Record<string, Observable<Out>> = {};
return source.pipe(
  // ... same operators
  switchMap(searchTerm => {
    // Initialize Observable in cache if it doesn't exist
    if (!cache[searchTerm]) {
      cache[searchTerm] = options.loadFn(searchTerm).pipe(
        shareReplay({
          bufferSize: 1,
          refCount: false,
          windowTime: 5000,
        })
      );
    }

    // Return the cached observable
    return cache[searchTerm];
  })
);

shareReplay: It will return the source observable (the request associated with the search term) if it exists in the cache, otherwise it will initialize it and cache it for future use.

bufferSize: presuming you're doing HTTP calls, then it'll be one response.

windowTime: essentially means that the cached observable will be removed from the cache after 5 seconds.

There are two other optimizations techniques that I can think of:

return the cached observable immediately after the user types the search term if it exists in the cache without having to go through the debounce time. I have deliberately decided not to do that so the user can have consistent experience. In my case the debounce time is 1.5s.
Inspired by stale-while-revalidate strategy, you can return the cached observable immediately and then send a new request to update the cache. That can be done using concat operator.

Scenario 3: The Edge Case

In most cases you'll be fine with the previous implementation, but there is an edge case that you might need to handle.

The user types a valid search term (Angular) and the request is sent.
The user types an invalid search term (Ng) before the current request is finished "(Angular) request". Since the search term is invalid, switchMap didn't get the chance to receive it, hence it didn't cancel the current request.
The user typed back (Angular) but the default behaviour of distinctUntilChanged won't allow it to pass through.

let shouldAllowSameValue = false; // -> 1
return source.pipe(
  distinctUntilChanged((prev, current) => {
    if (shouldAllowSameValue /** -> 3 */) {
      shouldAllowSameValue = false;
      return false;
    }
    return prev === current; // -> 4
  }),
  switchMap(searchTerm =>
    // -> 5
    from(options.loadFn(searchTerm)).pipe(
      takeUntil(
        source.pipe(
          tap(() => {
            shouldAllowSameValue = true; // -> 2
          })
        )
      )
    )
  )
);

Let's break it down (follow the numbers in the code comments):

shouldAllowSameValue is a flag that will be used to allow the same value to pass through the distinctUntilChanged operator. It's set to false by default.
shouldAllowSameValue is set to true when the user types an invalid search term before the current request is finished.
If shouldAllowSameValue is true, it means that the user typed an invalid search term before the last request is finished. In that case, we want to allow the same value to pass through the distinctUntilChanged operator.
This is the default behavior of distinctUntilChanged, it will only emit a value if it's different from the previous one.
Converts the loadFn function to an observable, subscribes to it, and cancels it when the source observable emits a new value while the request is still in progress.

distinctUntilChanged: It will only emit a value if it's different from the previous one (default behavior). This is useful for preventing requests with same search term (duplicate request). For example, if the user types "hello", results are fetched. If the user types "hello" again, we don't want to fetch the results again.

Complete code

export function typeahead<Out>(
  options: ITypeaheadOperatorOptions<Out>
): OperatorFunction<string, Out> {
  let shouldAllowSameValue = false;
  return source => {
    return source.pipe(
      debounceTime(options.debounceTime),
      filter(value => typeof value === "string"),
      filter(value => {
        if (value === "") {
          return options.allowEmptyString ?? true;
        }
        return value.length >= options.minLength;
      }),
      distinctUntilChanged((prev, current) => {
        if (shouldAllowSameValue) {
          shouldAllowSameValue = false;
          return false;
        }
        return prev === current;
      }),
      switchMap(searchTerm =>
        from(options.loadFn(searchTerm)).pipe(
          takeUntil(
            source.pipe(
              tap(() => {
                shouldAllowSameValue = true;
              })
            )
          )
        )
      )
    );
  };
}

Example

Framework Agnostic Example

import { fromEvent } from "rxjs";
import {
  debounceTime,
  distinctUntilChanged,
  filter,
  switchMap,
} from "rxjs/operators";

const searchInputEl = document.getElementById("search-input");
const resultsContainerEl = document.getElementById("results-container");

const search$ = fromEvent(searchInputEl, "input").pipe(
  map(event => searchInputEl.value),
  typeahead({
    minLength: 3,
    debounceTime: 1000,
    loadFn: searchTerm => {
      const searchQuery = searchTerm ? `?title_like=^${searchTerm}` : "";
      return fetch(`https://jsonplaceholder.typicode.com/posts${searchTerm}`);
    },
  }),
  // convert the response to json
  switchMap(response => response.json())
);

search$.subscribe(results => {
  resultsContainerEl.innerHTML = results
    .map(result => `<li>${result.title}</li>`)
    .join("");
});

<input type="text" id="search-input" />
<ul id="results-container"></ul>

Keep in mind that it's the practice to use switchMap and not other flattening operators like mergeMap or concatMap in this scenario.

Note: I have deliberately left out the error handling for the sake of simplicity, you might want to implement retry logic or show an error message to the user.

Angular Example

import { Component } from "@angular/core";
import { HttpClient } from "@angular/common/http";
import { FormControl } from "@angular/forms";
import { Observable } from "rxjs";
import {
  debounceTime,
  distinctUntilChanged,
  filter,
  switchMap,
} from "rxjs/operators";

@Component({
  selector: "app-search-bar",
  template: `
    <input type="text" [formControl]="searchControl" />
    <ul>
      <li *ngFor="let result of results$ | async">{{ result.title }}</li>
    </ul>
  `,
})
export class SearchBarComponent {
  searchControl = new FormControl();
  results$: Observable<any[]>;

  constructor(private http: HttpClient) {
    this.results$ = this.searchControl.valueChanges.pipe(
      typeahead({
        minLength: 3,
        debounceTime: 300,
        loadFn: searchTerm => {
          const searchQuery = searchTerm ? `?title_like=^${searchTerm}` : "";
          return this.#http.get<any[]>(
            `https://jsonplaceholder.typicode.com/posts${searchQuery}`
          );
        },
      })
    );
  }
}

Backpressure

Simply put, it's the pressure of too much incoming data that our system can't handle at once. Think of conveyer belt in a factory, if the belt is moving too fast, the workers won't be able to keep up with the incoming products.

In a typeahead scenario, If we let every keystroke from every user hit our server for query processing, we're going to overwhelm it faster than a JavaScript framework becomes outdated. In technical terms, this rapid influx of data can create a bottleneck, leading to increased latency and resource consumption.

This is less of a concern to the frontend developer, as their main focus is usually on user experience rather than backend scalability. However, it's essential to understand that the choices made on the frontend, like how often to trigger server requests, can have a direct impact on backend performance.

Conclusion

In this article, we learned how to build a typeahead component using RxJS. We also learned about backpressure and how it can affect our application's performance. I hope you found this article helpful and that it will help you build better applications in the future.

You can take this further and apply Infinite Scroll along with typeahead to have a unique user experience.

Implementing Infinity Scroll In Angular

ezzabuzaid — Fri, 15 Sep 2023 17:20:57 +0000

We already covered all about how to build Reactive Infinite Scroll in vanilla JavaScript, now let's see how to implement it in Angular.

Directive Or Pipe?

Angular provides two ways to work with DOM elements, Directives and Pipes.

Pipes are used to transform data in the template, think of them as formatting tools that refine how your data is displayed. They're not concerned with the structure of the web page itself; their primary job is to make sure your data looks the way you want it to when it hits the screen.
Directives are used to directly interact with and manipulate the DOM. These can either tweak the characteristics of individual elements (Attribute Directives) or even change the very layout of the DOM by adding or removing elements (Structural Directives).

You're going to implement Infinity Scroll as Directive and Pipe, and see the pros and cons of each approach.

Directive

Modify the InfinityScrollOptions interface to omit the element property because you're going to use the directive's host element.
Add noMoreData$ property to the InfinityScrollOptions interface, this is a user defined Observable that tells if all data had been loaded.

/**
 * Infinity Scroll Options excluding the element
 */
export interface InfinityScrollDirectiveOptions<T>
  extends Omit<InfinityScrollOptions<T>, "element"> {
  /**
   * User defined Observable that
   * tells if all data had been loaded.
   */
  noMoreData$: Observable<any>;
}

The good thing about directives is that you can inject the host element without having to explicitly pass it to the directive.

You need to store the data in a buffer, so that you can accumulate it when the user scrolls, and emit it as a single array and to use it you need to expose the directive instance to the host template.

@Directive({
  selector: "[infinityScroll]",
  // export the directive instance to the host template
  exportAs: "infinityScroll",
  standalone: true,
})
export class InfinityScrollDirective<T> {
  #elementRef = inject<ElementRef<HTMLElement>>(ElementRef);
  /**
   * The data buffer that will be used to accumulate data
   * and emit it as a single array.
   */
  #dataBuffer = new BehaviorSubject<T[]>([]);
  /**
   * The data buffer exposed as an Observable
   */
  data$ = this.#dataBuffer.asObservable();
}

Let's use @Input to pass the InfinityScrollOptions to the directive (you can use InjectionToken if you prefer).

// ... InfinityScrollDirective
export class InfinityScrollDirective<T> {
  #destroy = new Subject<void>();

  @Input({ required: true, alias: "infinityScroll" })
  set options(options: InfinityScrollDirectiveOptions<T[]>) {
    // ensures that previous infinityScroll subscription is unsubscribed
    this.#destroy.next();

    // reset the data buffer
    this.#dataBuffer.next([]);

    infinityScroll({
      ...options,
      element: this.#elementRef.nativeElement,
    })
      .pipe(
        scan((acc, data) => [...acc, ...data], [] as T[]),
        takeUntil(options.noMoreData$),
        takeUntil(this.#destroy)
      )
      .subscribe(data => {
        this.#dataBuffer.next(data);
      });
  }
}

The directive is designed to allow options to be changed at runtime, so it's important to unsubscribe from the previous infinity scroll subscription before creating a new one.

scan operator is used to accumulate the data, and emit it as a single array.
takeUntil, first one, operator is used to stop the infinity scroll when noMoreData$ emits. Second one, is used to stop the infinity scroll when the directive is destroyed.

  ngOnDestroy(): void {
    // Indicate infinity scrolling have to stop
    this.#destroy.next();

    // No more data will be pushed to the data buffer
    this.#dataBuffer.complete();
  }

The ngOnDestroy lifecycle hook is used to unsubscribe from the infinity scroll subscription and complete the data buffer.

<!-- alias to infinity scroll directive instance -->
<div
  [infinityScroll]="infinityScrollOptions"
  #infinityScroll="infinityScroll"
  style="max-width: 15rem;max-height: 10rem; overflow: auto"
>
  <ul>
    <!-- loop over the data source -->
    <li *ngFor="let item of infinityScroll.data$ | async">{{ item.title }}</li>
  </ul>
</div>

<p *ngIf="infinityScrollOptions.loading | async">Loading..</p>

Nothing complex here, loop over the data source and show a loading indicator. The important part is max-height and overflow styles, they're used to make the container scrollable, otherwise there will be no scroll event to listen to.

Configure Infinity Scroll

interface Todo {
  title: string;
}

const PAGE_SIZE = 10;

@Component({
  templateUrl: "./app.component.html",
  standalone: true,
  imports: [CommonModule, InfinityScrollDirective],
})
export class AppComponent {
  #lastBatchLength = new BehaviorSubject<number>(
    PAGE_SIZE /** Default to Page Size */
  );
  /**
   * An observable that signals if all data had been loaded
   *
   * It detects whether all data is loaded or not by comparing
   * the last batch length with the current batch length.
   *
   * I'm assuming that the page length is constant, therefore,
   * if the last batch length is the not same as the current batch length,
   * then we approach the end
   */
  noMoreData$ = this.#lastBatchLength.asObservable().pipe(
    pairwise(),
    filter(([prev, curr]) => prev !== curr)
  );
}

Before configuring the infinity scroll, you need to know when to stop it, it might not be required in your case, but in case you need it, infinty scroll should stop when all data had been loaded otherwise you'd end up making same call over and over again.

// ... AppComponent
export class AppComponent {
  // ... other code
  #http = inject(HttpClient);
  infinityScrollOptions: InfinityScrollDirectiveOptions<Todo[]> = {
    initialPageIndex: 1,
    threshold: 50,
    loading: new BehaviorSubject(false),
    noMoreData$: this.noMoreData$,
    loadFn: (result: InfinityScrollResult) => {
      return this.#http
        .get<Todo[]>(`https://jsonplaceholder.typicode.com/todos`, {
          params: {
            _start: result.pageIndex,
            _limit: PAGE_SIZE,
          },
        })
        .pipe(
          tap(todos => {
            this.#lastBatchLength.next(todos.length);
          })
        );
    },
  };
}

The infinity scroll options are pretty much the same as the ones used in the vanilla implementation, the only difference is that you're using HttpClient to fetch the data.

Pros
- Element reference is available through DI.
Cons
- You have to manage the subscription manually.
- You have to unsubscribe from the previous infinity scroll subscription before creating a new one.
- You have to empty the data buffer before creating a new infinity scroll subscription.

Demo

codesandbox.io

Pipe

@Pipe({
  name: "infinityScroll",
  standalone: true,
})
export class InfinityScrollPipe<T> implements PipeTransform {
  transform(
    options: InfinityScrollDirectiveOptions<T[]>,
    element: HTMLElement
  ): Observable<T[]> {
    return infinityScroll({
      ...options,
      element,
    }).pipe(
      scan((acc, data) => [...acc, ...data], [] as T[]),
      takeUntil(options.noMoreData$)
    );
  }
}

The operators used in the pipe are the same as the ones used in the directive.
The difference is with how we are receiving the element reference, in the directive we are injecting it, but in the pipe we are receiving it as an argument.
Due to pipes nature, you don't need to maintain a seperated data buffer, thereby async will manage the subscription for you.
Use it in a component template

<div
  #infinityScrollPipeEl
  style="max-width: 15rem;max-height: 10rem; overflow: auto"
>
  <ul>
    <li
      class="border"
      *ngFor="
        let item of infinityScrollOptions
          | infinityScroll : infinityScrollPipeEl
          | async
      "
    >
      {{ item.title }}
    </li>
  </ul>
</div>

Pros
- You don't have to manage the subscription manually.
- You don't have to empty the data buffer before creating a new infinity scroll subscription (async will discard the previous data).
Cons
- You have to explicitly pass the element reference to the pipe.

Demo

codesandbox.io

Signals

I wasn't able to round my head around how to implement it using signals due to the need to pass element reference, if you have any idea please let me know in the comments.

Conclusion

You've discovered how to implement Infinity Scroll in Angular using Directive and Pipe, and saw the pros and cons of each approach. Choose the one that suits your needs.
Personally I'd go with the pipe approach, because it's more declarative.

References

Reactive Infinity Scroll

ezzabuzaid — Mon, 11 Sep 2023 10:48:39 +0000

Have you ever experienced slow loading or lag on a webpage while trying to load a large amount of data? If so, you're not alone. An effective solution to improve the experience is to use infinite scrolling, which works similarly to how your Twitter feed continuously loads more tweets as you scroll down.

What is Infinity Scroll

A web design technique where, as the user scrolls down a page, more content automatically and continuously loads at the bottom, eliminating the user’s need to click to the next page.

Scroll down for the result, Or see the complete code

Problem

Infinite scrolling is often used for a few key reasons:

Data Fetching: Loading large data sets all at once can lead to latency issues or even browser crashes.
Mobile Usability: On mobile platforms, scrolling is more intuitive than navigating through multiple pages.
Resource Optimization: Incremental data loading is generally more resource-efficient, reducing the load on both the server and the client, which can lead to faster load times and a better user experience.

Solution

You're going to build a minimal yet efficient function using RxJS. It will include:

Support for vertical scrolling
Horizontal scroll support for both LTR and RTL
A threshold for determining when to fetch more data
Loading state

The writing assumes you have a basic understanding of RxJS. No worries though, I'll explain any special code or RxJS features along the way. So get ready, because you're about to dive into some RxJS operators! 😄

For those already comfortable with RxJS, you can skip the next section or jump to The Code

Well, Let's start!

Getting Started

The only thing you need to get started is RxJS. Install it with this command:

npm i rxjs

RxJS operators

RxJS operators are functions that manipulate and transform observable sequences. These operators can be used to filter, combine, project, or perform other operations on an observable sequence of events.

Common RxJS Operators

There are a lot of them, most used (by me 😆) are tap, map, filter, switchMap, and finalize. You might already know how to use those but lucky you, we're going to learn about other useful operators!

Take a look at the following observable:

const source$ = from([1, 2, 3, 4, 5]);
source$.subscribe(event => console.log(event));

The result would be 1 2 3 4 5. -Each in a new line-

filter

To only log odd numbers

const source$ = from([1, 2, 3, 4, 5]);
source$
  .pipe(filter(event => event % 2))
  .subscribe(event => console.log(event));

Say there's a possibility that the source$ might emit a null value. You can use a filter to stop it from passing through the rest of the sequence.

const source$ = from([1, 2, 3, null, 5]);
source$
  .pipe(filter(event => event !== null))
  .subscribe(event => console.log(event));

map

To change the sequence of events, you can use the map operator.

source$
  .pipe(map(event => (event > 3 ? `Large number` : "Good enough")))
  .subscribe(event => console.log(event));

What if I want to inspect an event without changing the source sequence

tap

source$
  .pipe(
    tap(event => {
      logger.log("log an event in the console");
      // you can perform any operation as well, however return statment are ignore in tap function
    })
  )
  .subscribe(event => console.log(event));

finalize

To monitor the end of an observable's lifecycle, you can use the finalize operator. It gets triggered when the observable completes.

It is usually used to perform some cleanup operations, stop the loading animation, or debug the memory, for example, add a log statement to ensure that the observable is complete and doesn't stuck in the memory 🥲.

debounceTime

Imagine you're building a login form and upon the user typing its password you want to hit the backend server to ensure the password conforms to certain criteria.

condt source$ = fromEvent(passwordInput, 'input').pipe(
  map((event) => passwordInput.value),
  switchMap((password) => checkPasswordValidaity(password))
)
source$.subscribe(event => console.log(event));

This example might work fine with one key caveat; on every keystroke, a request is sent to the backend server, thanks to switchMap it'll cancel previous requests so there might not be as much harm, however, using debounceTime you can ignore input events till the dueTime -argument- is pass.

const source$ = fromEvent(passwordInput, 'input').pipe(
  debounceTime(2000)
  map((event) => passwordInput.value),
  switchMap((password) => checkPasswordValidaity(password))
)
source$.subscribe(event => console.log(event));

Adding debounceTime essentially implies creating 2 seconds between each keystroke, so a user enters "hello" and then before 2 seconds pass enters "world" and only one request will be sent. In other words, each event has to have a 2 seconds distance from the last event.

startWith

An observable might not have value immediately and you need an event readily available for the new source$ subscribers.

const defaultTimezone = '+1'
condt source$ = fromEvent(timezoneInput, 'input').pipe(
  map((event) => timezoneInput.value),
  startWith(defaultTimezone)
)
source$.subscribe(event => console.log(event));

This sample will immediately log "+1" even if timezoneInput value is never entered

fromEvent

You could rewrite the previous example to be as follows

const timezoneInputController = new Subject<string>();
const timezoneInputValue$ = timezoneInputController.asObservable();
timezoneInput.addEventListener("input", () =>
  subject.next(timezoneInputController.value)
);

const source$ = timezoneInputValue$.pipe(
  map(event => event.target.value),
  startWith(defaultTimezone)
);
source$.subscribe(event => console.log(event));

Thanks to RxJS you can use fromEvent that will encapsulate that boilerplate, all you need to do is to say which event to listen to and from what element. Of course fromEvent returns an observable 🙂

takeUntil

I admit this one might be difficult to digest, it was for me. Taking the same previous example, Let's say that you have a form, an input, and a submit button. When the user clicks on the submit button you want to stop listening to the timezoneInput element input event. Yes, takeUntil as it sounds, it lets the subscribers take events until the provided observable emits at least once.

const formSubmission$ = fromEvent(formEl, 'submit')

const defaultTimezone = '+1'
condt source$ = fromEvent(timezoneInput, 'input').pipe(
  map((event) => timezoneInput.value),
  startWith(defaultTimezone)
)
// normally, this subscriber will keep logging the event even if the users clicked on the submit button
source$.subscribe(event => console.log(event));

// Now, once the submit button are clicked the subscriber subscription will be canceled
source$
  .pipe(takeUntil(formSubmission))
  .subscribe(event => console.log(event));

pipe

The pipe function in RxJS is a utility for composing operations on observables. Use it to chain multiple operators together in a readable manner, or to create reusable custom operators. This is crucial when the source sequence is complex to manage.

import { pipe } from "rxjs"; // add it to not to be confused with Observable.pipe

// Create a reusable custom operator using `pipe`
const doubleOddNumbers = pipe<number>(
  filter(n => n % 2 === 1),
  map(n => n * 2)
);

const source$ = from([1, 2, 3, 4, 5]);

source$.pipe(doubleOddNumbers).subscribe(x => console.log(x));
// result: 1, 6, 10

Flattening Operators

Sometimes, you need to fetch some data with every incoming event, say from the backend server. There are a few methods to do this.

switchMap

Like a regular map, the switchMap operator uses a project function that returns an observable -its first argument-, known as the inner observable. When an event occurs, switchMap subscribes to this inner observable, creating a subscription that lasts until the inner observable completes. If a new event arrives before the previous inner observable completes, switchMap cancels the existing subscription and starts a new one. In other words, it switches to a new subscription.

const source$ = from([1, 2, 3, 4, 5]);

function fetchData(id: number) {
  return from(fetch(`https://jsonplaceholder.typicode.com/todos/{id}`));
}

source$
  .pipe(switchMap(event => fetchData(event)))
  .subscribe(event => console.log(event));

In this sample, only the todo with id 5 will be logged because switchMap works by switching the priority to the recent event as explained above. from([...]) will emit the events after each other immediately thereby switchMap will switch (subscribe) to the next event inner observable as soon as it arrives without regard to the previous event. The switch operation essentially means unsubscribing from the previous inner observable and subscribing to the new one.

concatMap

It blocks new events from going through the source sequence unless the inner observable completes. It is particularly useful for database writing operations or animating/moving an element where it's important to complete one action before starting another.

source$
  .pipe(concatMap(event => fetchData(event)))
  .subscribe(event => console.log(event));

This sample will log all todos in order. Essentially what happens is concatMap blocks the source sequance till the inner observable at hand completes.

mergeMap

It doesn't cancel the previous subscription nor blocks the source sequence. mergeMap will subscribe to the inner observable without regard to its completion, so if an event comes through and the previous inner observable hasn't been completed yet that's fine, mergeMap will subscribe to the inner observable anyway.

source$
  .pipe(mergeMap(event => fetchData(event)))
  .subscribe(event => console.log(event));

This sample will log all todos but in uncertain order, for instance, the second request might resolve before the first one and mergeMap doesn't care about the order, If that is important then use concatMap.

exhaustMap

The final one and the most important in this writing is exhaustMap: it is like switchMap but with one key difference; it ignores the recent events in favor of the current inner observable completion in contrary to switchMap which cancels the previous inner observable subscription in favor of a new one.

source$
  .pipe(exhaustMap(event => fetchData(event)))
  .subscribe(event => console.log(event));

This sample will only log the first todo as the first todo request hasn't been completed yet other events came through therefore they've been ignored.

To summarise

switchMap will unsubscribe from the existing subscription (if the previous inner observable hasn't been completed) in favor of a new one when a new event arrives.
concatMap will block the source sequence so the inner observable at hand must be complete before allowing other events to flow.
mergeMap doesn't care about the status of the inner observable so it'll subscribe to the inner observable as events come through.
exhaustMap will ignore any event till the current inner observable is complete.

Okay, that is a lot, isn't it? I understand that if you're new to RxJS you might not be able to digest all this info, Your best bet is to practice and that's what you're trying to do here.

Wow, I really did it, and you did too 😎

Time to talk about some of the Scroll API(s)

Scroll API

You already know the Scroll Bar, it's at the right end of the page 🥸, no really, when the user scroll in any direction the browser emits a few events, like scroll, scrollend, and wheel.

You are going to learn enough that to tackle the problem at hand.

Let's start with scroll and scrollend:

Scroll and ScrollEnd Events

The scroll event fires while an element is being scrolled and scrollend fires when scrolling has completed.

element.addEventListener("scroll", () => {
  console.log(`I'm being scrolled`);
});

element.addEventListener("scrollend", () => {
  console.log(`User stopped scrolling`);
});

Keep in mind that this only works if the element—the one that has the event listener (handler)—is scrollable, not its parent or any ancestor or descendant elements.

Wheel Event

The wheel event fires while an element or any of its children is being scrolled using the mouse/trackpad wheel which means trying to scroll down/up using the keyboard won't trigger it.

Size Properties

For the task at hand, the scroll event will be the primary focus. However, I've also outlined some additional events and properties to give you a well-rounded understanding. Now, let's look at the key size properties you'll need to know:

element.clientWidth: The inner width of the element, excluding borders and scrollbar.
element.scrollWidth: The width of the content, including content not visible on the screen. If the element is not horizontally scrollable then it'd be the same as clientWidth.
element.clientHeight: The inner height of the element, excluding borders and scrollbar.
element.scrollHeight: The height of the content, including content not visible on the screen. If the element is not vertically scrollable then it'd be the same as clientHeight.
element.scrollTop: The number of pixels that the content of an element is scrolled vertically.

Note: When I say "the content," I mean the entirety of what's contained within the HTML element.

The green box is the element while the black box on the left is the width overflow and on the right is the height overflow

Let's take the following example, Calculate the remaining pixels from the user's current scroll position to the end of the scrollable element.

function calculateDistanceFromBottom(element: HTMLElement) {
  const scrollPosition = element.scrollTop;
  const clientHeight = element.clientHeight;
  const totalHeight = element.scrollHeight;
  return totalHeight - (scrollPosition + clientHeight);
}

Take a look at the below image.

scrollPosition indicates to what point the user scrolled

Presuming the totalHeight is 500px, clientHeight 300px, and the scrollPosition is 100px, deducting the sum of scrollPosition and clientHeight from totalHeight would result in 100px which is the remaining distance to reach the bottom of the element.
A similar formula when calculating the remaining distance to the end horizontally

function calculateRemainingDistanceOnXAxis(element: HTMLElement): number {
  const scrollPosition = Math.abs(element.scrollLeft);
  const clientWidth = element.clientWidth;
  const totalWidth = element.scrollWidth;
  return totalWidth - (scrollPosition + clientWidth);
}

Presuming the totalWidth is 750px, clientWidth 500px and the scrollPosition is 150px, deducting the sum of scrollPosition and clientWidth from totalWidth would result in 100px which is the remaining distance to reach the end of the XAxis.
You might have noticed the Math.abs being used and that due to RTL direction where the user has to go in the reverse direction which would make the scrollPosition value to be negative so using Math.abs to unify it in both directions.

Side tip: Using the information you have about the element's sizes, you can also make a function to check if the element can be scrolled or not.

type InfinityScrollDirection = "horizontal" | "vertical";

function isScrollable(
  element: HTMLElement,
  direction: InfinityScrollDirection = "vertical"
) {
  if (direction === "horizontal") {
    return element.scrollWidth > element.clientWidth;
  } else {
    return element.scrollHeight > element.clientHeight;
  }
}

Simply put, if the element scroll's size is the same as its client's size then it isn't scrollable.

The Code

I know you've been looking around to find this section, finally, we'll put all the learnings into action, let's start by creating a function named infinityScroll that accepts options argument

export interface InfinityScrollOptions<T> {
  /**
   * The element that is scrollable.
   */
  element: HTMLElement;
  /**
   * A BehaviorSubject that emits true when loading and false when not loading.
   */
  loading: BehaviorSubject<boolean>;
  /**
   * Indicates how far from the end of the scrollable element the user must be
   * before the loadFn is called.
   */
  threshold: number;
  /**
   * The initial page index to start loading from.
   */
  initialPageIndex: number;
  /**
   * The direction of the scrollable element.
   */
  scrollDirection?: InfinityScrollDirection;
  /**
   * The function that is called when the user scrolls to the end of the
   * scrollable element with respect to the threshold.
   */
  loadFn: (result: InfinityScrollResult) => ObservableInput<T>;
}

function infinityScroll<T extends any[]>(options: InfinityScrollOptions<T>) {
  // Logic
}

As promised, you now can customize the infinite scroll function to your liking. Next, you'll learn how to attach an event listener to the specific element that contains your infinitely scrollable list of items.

function infinityScroll<T extends any[]>(options: InfinityScrollOptions<T>) {
  return fromEvent(options.element, "scroll").pipe(
    startWith(null),
    ensureScrolled,
    fetchData
  );
}

fromEvent listens to scroll event of the scrollable element.
startsWith starts the source sequence to fetch the first batch of data.
ensureScrolled is a chainable operator that confirms the scroll position surpasses the predefined threshold before proceeding.
fetchData is another chainable operator that fetches data based on the pageIndex, more on that later.

ensureScrolled

const ensureScrolled = pipe(
  filter(() => !options.loading.value), // ignore scroll event if already loading
  debounceTime(100), // debounce scroll event to prevent lagginess on heavy scroll pages
  filter(() => {
    const remainingDistance = calculateRemainingDistance(
      options.element,
      options.scrollDirection
    );
    return remainingDistance <= options.threshold;
  })
);

function calculateRemainingDistance(
  element: HTMLElement,
  direction: InfinityScrollDirection = "vertical"
) {
  if (direction === "horizontal") {
    return calculateRemainingDistanceOnXAxis(element);
  } else {
    return calculateRemainingDistanceToBottom(element);
  }
}

filter only passes the scroll events if the element is scrollable, otherwise, it might lead to unexpected behavior.
debounceTime will skip any event, in our case scroll events from flowing the sequence
filter is checking if the remainingDistance either to the bottom (in case of vertical scrolling) or to the end of XAxis in case of horizontal scrolling is less than the threshold. Presuming threshold is 100px then when the scroll position is within 100 pixels of reaching the end (either vertically or horizontally, depending on the configuration), loadMore.next() will be invoked, signaling that more content should be loaded.

fetchData

const fetchData = pipe(
  exhaustMap((_, index) => {
    options.loading.next(true);
    return options.loadFn({
      pageIndex: options.initialPageIndex + index,
    });
  }),
  tap(() => options.loading.next(false)),
  // stop loading if error or explicitly completed (no more data)
  finalize(() => options.loading.next(false))
);

exhaustMap ignores any event till the loadFn completes. If exhaustMap project function (its first argument) has been called, that implies the previous (if any) observable is completed and is ready to accept new events -load more data-.
tap is signaling data loading is finished.
finalize does the same as tap in our case, however, tap won't be called if loadFn -request to the backend server- had responded with an error, and in case of an error, the source observable completes hence finalize. In other words, if the source sequence errored or the user explicitly completed the source then stop the loading.

Notice how exhaustMap indicates the loading state. You might question why not place the loading signal in a tap operator right before exhaustMap. Doing so would cause the loading observable to emit true whenever loadMore triggers. But this doesn't necessarily mean it's time to load more data -the previous inner observable from loadFn hasn't finished yet-. To avoid this, exhaustMap is used to confirm that it's ready to load more data.

The real piece of code; incrementing the page index to fetch the next patch of data

exhaustMap((_, index) => {
  // ...code
  return options.loadFn({
    pageIndex: options.initialPageIndex + index,
    // ...code
  });
});

The exhaustMap project function has two arguments

The event from the source sequence.
The index corresponds to the most recent event (this number signifies the position of the latest event).

In this specific case, you'll be focused on the event position or index. Check out the following example for a clearer understanding of how it operates.

At first time the source loadMore emits the index will be zero.
3 batches of data are loaded, so the next index will be 4.
Assuming initialPageIndex is 1 and the data is to be loaded for the first time then the pageIndex is 1
Assuming initialPageIndex is 1 and the data is to be loaded for the fifth time then the pageIndex is 6
Assuming initialPageIndex is 4 and the data is to be loaded for the first time then the pageIndex is 4

The last case might be off; usualy you might have initialPageIndex 0, but let's say you're scrolling the Twitter feed, and for some reason, the browser reloaded, so instead of loading data from the beginning, you decided to store the pageIndex in some state (URL query string) so in such cases only the data from the last pageIndex will be there so the experience continues as if nothing happened. Prior data needs to be there as well either by loading it till the pageIndex or via implementing an opposite scroll direction data loading 🥲

What about using mergeMap, switchMap, or concatMap? You might have thought about that already!

Given the following scenario: A user scrolled down to the end of the page, but the request to load more data is still pending. The user kept scrolling down to the end of the page but the data was not yet resolved. What do you think would happen?

Note: The following recordings use a slow 3g network speed

Using mergeMap

With each scroll event, mergeMap subscribes to the inner observable without regard to the previous subscription, essentially leading to a new request -loadMode- with each verified scroll event -below the threshold-

Edit in CodePen

Using switchMap

With each scroll event, switchMap will cancel/unsubscribe from the previous subscription and subscribe to the inner observable again, essentially leading to a new request but the previous unresolved one will be canceled so only one request will be pending at a time. That might be okay, however, the event position index will increment each time switchMap subscripes to the inner observable which leads to incorrect data being loaded.

Edit in CodePen

Using concatMap

With each scroll event, concatMap will subscribe to the inner observable, blocking the source sequence till the current subscription completes -loadMore request resolves-, essentially leading to a new request with every verified scroll but holding them onto till it can process a new event. The event position index will increment each time concatMap subscribes to the inner observable which leads to requesting more data than needed. See the recording below and take a good look at what happens in the Network Tap when the user stops scrolling.

Edit in CodePen

Using exaustMap

It is the winner in this scenario because it effectively manages pending requests. When a scroll event triggers a new request, exhaustMap will ignore any subsequent scroll events until the current request (inner observable) is complete. This ensures that only one request is pending at a time, and it prevents the index from incrementing incorrectly.

That being said, a simple workaround would be to explicitly ignore any scroll even while the data is loading.

const fetchData = pipe(
  filter(() => options.loading.value === false),
  // mergeMap, switchMap and concatMap should work now.
  exaustMap((_, index) => {
    // ...
  })
  // ...
);

However, this approach has a limitation. Since options.loading is a user-defined observable, there's a risk that the user might change its value. If that happens, the issue will appear.

Example

Vertical Scrolling

Horizontal Scrolling

RTL Horizontal Scrolling

UX and Accessibility Consideration

Infinite scrolling isn't a magic fix. I know some folks who strongly advise against using it. Here's why:

Bad for Keyboard Users: If you're using a keyboard to get around a website, infinite scrolling can mess that up and get you stuck.Especially if the infinity scrolling is the main way of navigating the website
Hard to Pick Up Where You Left Off: Without page numbers, it's tough to go back to where you were. This makes it hard for users and a headache for developers to implement.
Unreachable Content: Makes certain content like footers hard to reach.
Confusing Screen Readers: If someone's using a screen reader, the constant loading can make the page structure confusing.
Too Much, Too Fast: For some people, like those who get easily distracted, the never-ending flow of content can be overwhelming. This one's just my take, but it's something to think about.

When building an infinite scroll you've to consider important factors such as:

Placing content correctly and making them accessible like footer, and contact information.
Allowing users to return to their previous spot.
Offering the ability to jump ahead.
Ensuring the experience is navigable for users who rely solely on keyboards.

I recognize that these tasks present significant developmental challenges. However, as the saying goes, quality comes at a cost.

This isn't to say that infinite scrolling is bad; instead, the emphasis is on applying it with caution.

Other Pagination Strategies

Traditional Pagination: This approach uses a combination of numbered pagination and 'Previous'/'Next' buttons to offer both specific and sequential page access.
"Load More" Button: Includes a button at the end of the visible content; clicking it appends additional items to the list.
Content Segmentation: Utilizes tabs or filters to categorize content, enabling quick navigation to topic-specific data—e.g., segmenting tweets into categories like Science, Tech, Angular, 2021, etc.

Next Step

In addition to the core functionality, further enhancements can be incorporated

Resume Journey: An option to store the pageIndex to resume the user journey, history API for instance.
Error Recovery: retry loading data when the operation fails. Although I think it shouldn't be part of the infinity scroll function, you can provide it as an option.
Load more data when scrolling up: Imagine you navigate to a profile page and then go back to the feed, like on Twitter. The last page index could be saved in the history API, guiding what to fetch next. But what if you can't load all the earlier data at once? In that case, you can also load more content when the user scrolls up, not just when scrolling down.
Improve performance by integrating Virtual Scrolling to only render visible elements.

Summary

Congrats! You've learned how to implement infinite scrolling and gained a deep understanding of the RxJS operators that power this feature. Alongside the technical side, you've taken a critical look at the potential accessibility challenges that come with infinite scrolling, equipping you with a balanced view of its pros and cons.

This implementation is framework-agnostic, requiring only RxJS as a dependency. While TypeScript is used for type safety, it's not a hard requirement and can be easily omitted.

Stay tuned for an upcoming post on Virtual Scroll. Make sure to follow me to get notified when it's published. Your feedback and opinions are highly valued, so feel free to share them.

Resources

Gentle Introduction To ESLint Rules

ezzabuzaid — Thu, 07 Jul 2022 21:19:50 +0000

Have you ever wondered how your editor can know if you used var instead of let? or that you missed adding alt to the img tag? Yes, that is ESLint ladies and gentlemen.

In this article, we're going to talk about how this is possible and how you can do something similar!

For you who ain't familiar with ESLint, I recommend that you read about it in advance and perhaps try to see it in action. The writing presumes that you have at least little knowledge in configuring ESLint.

All the code snippets are available in Replit - if you can't see the file in the caption, you've to manually navigate to it from the file explorer.

Problems

One day Javascript rolled out new ways to create a variable (using let and const) an after some investigation it was mandatory to adapt to the new way, your team agreed and left the office thinking you're crazy. The next day the team have forgotten and it was like nothing happened yesterday.
New developer joined and was pretty excited to create its first PR, after the onboarding, the new hire created the PR with inline styles everywhere. The team doesn't like inline styles and considers it a bad practice thus the PR was abandoned with one big comment "HTML files are pretty without inline styles, please consider moving them to the styles files".
You're refactoring a legacy class trying to simplify the logic by extracting the master method that does everything based on parameters to a rather set of simple methods. Now the master method is marked as deprecated in favour of the simpler ones.
You're working on React SSR project, you've created a component that utilises the navigator object to get the user agent information, the moment the code runs an error appears from nowhere telling you that navigator is undefined because the component has been executed in the server -because of SSR nature-.
The team members often forgot to unsubscribe from RxJS observable, the memory halted, you blame Angular, and decides to move to React instead.

All these issues can be dealt with using ESLint. ESLint ain't magic that will instantly solve the code bugs and errors, but it can be a very useful tool to warn you about potential bugs.

Throughout this article, we'll write ESLint rules that address a variety of issues, so let's get to the point directly.

How ESLint Can Understand The Code

ESLint has two (of many) major concepts:

Rule: an object that describes what is about to be validated/linted (More information later).
Plugin: Node.js package that exposes a set of rules.

An ESLint rule has a create function with one argument (the context object) and returns a traverse object.

The context object contains additional functionality that is helpful for rules to do their jobs. As the name implies, the context object contains information that is relevant to the context of the rule.

The traverse object keys are AST selectors and the values are functions that take a Node object.

{
  create(context) {
    return {
      VariableDeclaration: (node) => {},
    };
  },
};

Bear with me for a bit, things will get clearer soon! the snippet above is the basis for other definitions.

To know what is the Node object and how ESLint can provide it, we need to talk about CS terms.

Compiler

nope, I mean Compiler phases and not the definition of the Compiler.

Your code (The program) will get smashed down by several layers before it's finally understood by the hardware, these layers are:

Scanning.
Parsing.
There're more but not mandatory to know now.

Would you like to read more about the compiler or read more about the compiler phases, I recommend this book

Scanning

The process of reading the source code a character at a time and chunking them into Tokens

for instance, when passing this code into the scanner

var foo = "bar"

will produce the following tokens

Token { type: 'Keyword', value: 'var', start: 0, end: 3 },
Token { type: 'Identifier', value: 'foo', start: 4, end: 7 },
Token { type: 'Punctuator', value: '=', start: 8, end: 9 },
Token { type: 'String', value: '"bar"', start: 10, end: 15 }

Parsing

The process of reading the output of the Scanning phase (tokens) and outputting a tree data structure called AST or "Abstract syntax tree". go down to learn more about AST

for instance, the above tokens will produce

Node {
  type: 'Program',
  start: 0,
  end: 15,
  body: [
    Node {
      type: 'VariableDeclaration',
      start: 0,
      end: 15,
      declarations: [Array],
      kind: 'var'
    }
  ],
  sourceType: 'script'
}

Finally, we can answer the question what is the Node object in the traverse function? simply put a Node is an object that holds information about a specific part of the program like the location of the variable foo and its initializer in form of a tree structure.

Let's start with a simple example: prevent the developer from using the var keyword, for that, we need to handle nodes of type VariableDeclaration - based on the AST above the selector for the var keyword is VariableDeclaration - and when the handler function is invoked we need to warn the developer that var shouldn't be used.

{
  create(context) {
    return {
      VariableDeclaration: (node) => {
        if (node.kind === "var") {
          context.report({
            node: node,
            message: 'Use "let" instead.',
          });
        }
      },
    };
  },
};

In the shell, type npx eslint use-let.js to see the result for use-let rule.

ESLint Rule

I guess that's enough about how ESLint process your code, now it's the time to get more involved in writing code.

ESLint rule structure as follows

type Rule = {
    meta: {
        hasSuggestion: boolean;
        type: "problem" | "suggestion" | "layout";
        docs: {
            description: string;
            url: string;
            category: string;
            recommended: boolean;
        },
        fixable: "code" | "whitespace";
        schema: [];
    },
    create: Function;
};

Check ESLint rule docs for complete details

We already covered the create function in the above section, so let's talk about meta.

type: If the rule is for a code structure use "layout", if it's to warn about potential bug/error use "problem", otherwise you might want to stick with "suggestion".
fixable: Indicates that the node that represents a segment of the code is fixable (can be fixed automatically) by adding --fix options (More about it later).
schema: In case you want to make the rule configurable (e.g. accept an option to what kind of variable is allowed) you've to add JSON schema.
hasSuggestion: Indicates that the node that represents a segment of the code has suggestions go down.
docs:
1. description: Concise yet meaningful description of what the rule does.
2. url: A link where the user can get more information about the rule (docs or spec).
3. category and recommended out of this article's scope.

Let's modify the rule to let the developer decides which kind of variable to dismiss, the option can be accessed using the context object.

To specify options, we need to write a valid JSON schema. the schema property takes an array of JSON objects. More information about the schema

{
  meta: {
    type: "problem",
    schema: [ // HERE
      {
        enum: ["var", "let"],
      },
    ],
  },
  create(context) {
    const [varKind] = context.options;
    return {
      VariableDeclaration: (node) => {
        if (node.kind === varKind) {
          const allowedVarKind = varKind === "var" ? "let" : "var";
          context.report({
            node: node,
            message: `Use ${allowedVarKind} instead.`,
          });
        }
      },
    };
  },
};

In the shell, type npx eslint no-var-kind.js to see the result for no-var-kind rule.

Let's see how ESLint can fix this code for us...

Rule Fixer

As we saw above the context.report function is what warns the developer about the validity of a segment of code.
this function accepts an object that has the node (that represents the code to be reported), message (meaningful message describing why that code is being reported) and fix function

Note: there're more properties that the report object can have, which we might explain as we go further and others would be out of this article's scope.

The fix function receives a single argument, a fixer object, that you can use to apply a fix, it returns a fixing object (the return value of the fixer object methods) or array of fixing objects

Completing the previous example this is how we can auto-fix the reported code. We need to adjust the meta object to have the fixable option and add the fix function to the report object.

{
  meta: {
    ...,
    fixable: "code"
  }
  create(context) {
    const [varKind] = context.options;
    return {
      VariableDeclaration: (node) => {
        if (node.kind === varKind) {
          const allowedVarKind = varKind === "var" ? "let" : "var";
          context.report({
            node: node,
            message: `Use ${allowedVarKind} instead.`,
            fix(fixer) {
              const start = node.start;
              const end =  node.start + 3;
              const range = [start, end];
              return fixer.replaceTextRange(range, allowedVarKind);
            }
          });
        }
      },
    };
  },
};

Note: you shouldn't determine the node range that way, for the sake of not complicating the code, I've added 3 to the start position, for instance, if var start position is 5 then the end is 8 because var length is 3. Check out the official rule

Make sure you're at no-var-kind.js file first, then in the shell, type npx eslint no-var-kind.js --fix to see the result for no-var-kind rule.

Fixer Object

It has a few methods, each return a fixing object to mutate the source code, you can either pass the token/node or the range (location) of a token/node.

In case you returned an array of fixing objects, make sure that the fixes ain't stepping on each other, otherwise, ESLint will refuse to apply the fixes for that node/token. ESLint can determine the conflicts by inspecting the node/token range of all the fix attempts, if the ranges step on each other ESLint will discard all fixes for that node.

for instance, the fixes in this sample conflicting on the range [4, 5]

fix(fixer) {
  return [
      fixer.replaceTextRange([1, 5], "const"), // changes the var kind to const
      fixer.replaceTextRange([4, 7], "bar") // changes the var name to bar
    ]
}

Be wary to add the fixable option to meta object, otherwise, ESLint will refuse to do any fixes.

Fixer object methods.

insertTextAfter(nodeOrToken, text) - inserts text after the given node or token
insertTextAfterRange(range, text) - inserts text after the given range
insertTextBefore(nodeOrToken, text) - inserts text before the given node or token
insertTextBeforeRange(range, text) - inserts text before the given range
remove(nodeOrToken) - removes the given node or token
removeRange(range) - removes text in the given range
replaceText(nodeOrToken, text) - replaces the text in the given node or token
replaceTextRange(range, text) - replaces the text in the given range

More details

Check the examples to see how the fixer object can be used.

Note: ESLint does not mutate the source code directly rather it does mutate the AST and regenerates the code from it

Rule Suggestions

What if we want to tell the developer that the code is fine but there might better way to do it. In this case, we shouldn't auto-fix the code, instead, you can provide a suggestion!

for instance, we want to suggest that instead of using the ternary operator you should use the normal if statement.

const fooOrBar = true ? 'Foo':'Bar';

First thing first, we need to know what the AST of the above code looks like, for that we'll use AST Explorer.

The ternary operator node type is ConditionalExpression, so we'll use that as an AST selector

Second, the type in meta object needs to be suggestion. You only need hasSuggestions when the rule does suggest a fix or more.

{
  meta: {
    type: "suggestion",
    hasSuggestions: false,
  },
  create(context) {
    return {
      ConditionalExpression: (node) => {
        context.report({
          node: node,
          message: `What about using if instead?`,
        });
      },
    };
  },
};

In the shell, type npx eslint no-ternary.js to see the result for no-ternary rule.

Let's take another example but with real suggestions now. the next rule will suggest that if the var/let is not being reassigned then it should be const instead.

The hasSuggestions is true because the rule suggests fixes, also instead of adding fix function as part of the report object, we added it as part of the suggestion object in the suggest array.
The context object has been used before to get the developer options and now it's used to get the whole source code.
The source code has plenty of useful methods that we are using to get access to the tokens
In a variable declaration, the first token is the variable keyword token and the next one is variable identifier token hence skip the first token.

{
  meta: {
    type: "suggestion",
    hasSuggestions: true,
  },
  create(context) {
    return {
      VariableDeclaration: (node) => {
        if (node.kind !== "const") {
          const sourceCode = context.getSourceCode();
          // the name/identifier of the var/let;
          const varName = sourceCode.getFirstToken(node, { skip: 1 }).value;
          /// continue in the next sample ...
        }
      },
    };
  },
};

We've to get all tokens that reference that variable and check if any of them have equal operator as direct sibling and if so then that variable cannot be const.

const getAllUsageOfThisVar = sourceCode.getTokensAfter(node, {
  filter: (nextToken) => nextToken.value === varName,
});

const reassigned = getAllUsageOfThisVar.some((token) => {
  const directNextToken = sourceCode.getTokenAfter(token);
  return directNextToken.value === "=";
});

In the fix function, get the variable keyword token (we are interested in its exact location in the source code) then replace the text with const.

if (!reassigned) {
  context.report({
    node: node,
    message: `Use const instead.`,
    suggest: [
      {
        desc: "You are not reassigning this variable, would you like to change it to const?",
        fix: function (fixer) {
          const varToken = sourceCode.getFirstToken(node, {
            skip: 0,
          }); // the var/let token
          return fixer.replaceText(varToken, "const");
        },
      },
    ],
  });
}

Make sure you're at prefer-const.js file first, then in the shell, type npx eslint prefer-const.js --fix --fix-type suggestion to see the result for prefer-const rule. you can as well toggle the comment and rerun the command to see the result difference.

Suggestions are very helpful in the editor experience where it'll show you dropdown of fixes that you can choose from.

ESLint Parser

ESLint doesn't process the code into the compiler phases, rather it provides an option to let you specify a Parser. By default, ESLint uses Espree which essentially converts JS source code to AST data structure, so in case you want to write a rule targeting TypeScript source code, you'll need to specify a different parser in your .eslintrc.json configuration file, same applies for different file extension, for HTML you might use this or creating your own parser!

html-parser.js

exports.parseForESLint = function(code, options) {
    return {
        ast: require('parse5').parse(code),
        services: {},
        scopeManager: null,
        visitorKeys: null
    };
};

Parser configuration

Most properly you'll find parsers for all popular extensions - A custom parser is useful in case you want to enrich the generated AST with other properties related to your project or if you're building a custom file extension.

Rule Tester

What about test, I know you like to write test ^^ and that's why we'll talk about it now...

Unit testing rules is easier and explicit than the traditional test cases that you would write.

All you have to do is to prepare the valid and invalid cases and make sure they are always as stated. Let's go back to use-let rule and write test for it.

A sample code that use-let rule will find valid

let foo = "bar";

A sample code that use-let rule will find invalid

var foo = "bar";

What we need to do now is use RuleTester to verify this requirement, it takes an object similar to the one you'd use in the in top level .eslintrc.json.

This class is wrapper over mocha testing library.

The valid and invalid options take an array of cases. The easier way to write a valid test case is by using the static only method, it takes the program to run the test against.
The invalid test case are bit more rich, you've to add the invalid program as well as the errors array that should present, the errors are can be as simple as message or more complex by adding the location and node type and so on

const rule = require("./use-let.rule.js");
const { RuleTester } = require("eslint");

const ruleTester = new RuleTester({
  parserOptions: {
    ecmaVersion: 2020,
  },
});

ruleTester.run("use-let", rule, {
  valid: [RuleTester.only(`let foo = "bar";`)],
  invalid: [
    {
      code: `var foo = "bar";`,
      errors: [{ message: 'Use "let" instead' }],
    },
  ],
});

Plugins Folder Structure

This is an opinionated folder structure and nothing like what ESLint recommend

Will start with folder in the root workspace named "plugins", each folder under this one is a plugin folder with all files related to it. A plugin is essentially an npm package that you can either publish it to a registry or link it locally. All the examples in this article are linked locally.

You do not have to use npm link for that, only add the plugin name and the path to it.

Folder structure tree for ESLint plugins.

plugins
└── <plugin-name>
    ├── rules
    │   ├── <rule-name> # Each rule should have all the required files
    │   │   ├── use-let.rule.js # use-let as an example
    │   │   ├── use-let.test.js
    │   │   └── use-let.docs.md # It's always good to have docs.
    │   └── <rule-name-2> # Each rule should have all the required files
    │       ├── prefer-const.rule.js
    │       ├── prefer-const.test.js # Make sure to test every case!
    │       └── prefer-const.docs.md
    ├── index.js # exports the plugin rules
    └── package.json # look below.

The package.json file should have at least the name "eslint-plugin-'plugin-name'" and main pointing to the rules exports file (index.js).

Examples

Let's write few more examples of ESLint rules and discover their potential.

Image alternative text (img alt)

Because we're validating HTML code we've to use a parser that can transform the HTML code to AST, so for that we'll use eslint-html-parser and add it in the .eslintrc.json file.

This rule will report a message in case img tag doesn't have the alt property.

Here's a sample HTML code

<img src="..." />

The AST is - some properties omitted for brevity.

 {
  type: 'HTMLElement',
  tagName: 'img',
  range: [ 0, 14 ],
  children: [],
  attributes: [
    {
      type: 'HTMLAttribute',
      range: [Array],
      attributeName: [Object],
      attributeValue: [Object],
      value: 'src=""',
    }
  ]
}

In the meta.docs we added url property that give more info about the rule.
The AST selector of an HTML element is HTMLElement.
The if ignores all element that are not img tag.
If the img element have an alt regardless if it does have value or not then we do not report.

{
  meta: {
    type: "problem",
    docs: {
      description: "Img tag should have an alt property",
      url: "https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img#accessibility_concerns",
    }
  },
  create: function (context) {
    return {
      HTMLElement: function (node) {
        if (node.tagName === "img") {
          const hasAltAttribute = node.attributes.some(
            (item) => item.attributeName.value === "alt"
          );
          if (!hasAltAttribute) {
            context.report({
              node: node,
              message: "Where is the alt? consider adding empty alt in case the img doesn't convey any meaning.",
            });
          }
        }
      },
    };
  },
};

In the shell, type npx eslint img-alt.html to see the result for img-alt rule.

Figure Tag

This rule will report a message in case img tag is not within figure tag.

The first if ignores all element that are not img tag.
We walk all the way up till we find figure tag parent.

{
  meta: {
    type: "problem",
    ...other propreties
  },
  create: function (context) {
    return {
      HTMLElement: function (node) {
        if (node.tagName === "img") {
          let parent = node.parent;
          while (parent && parent.tagName !== "figure") {
            parent = parent.parent;
          }
          if (parent?.tagName !== "figure") {
            context.report({
              node: node,
              message: "img tag should be wrapped within figure tag.",
            });
          }
        }
      },
    };
  },
};

In the shell, type npx eslint img-figure.html to see the result for img-figure rule.

Wait! challenge for you, complete the rule to report a message in case figcaption is not within the figure tag.

No Console

This rule will report a message whenever console method is present.

This sample code

console.log("Are you bored yet?");

Will give the following AST - some properties omitted for brevity.

Node {
  type: 'CallExpression',
    callee: {
    type: 'MemberExpression',
      object: {
      type: 'Identifier',
      name: 'console',
    }
    property: {
      type: 'Identifier',
      name: 'log',
    }
  }
}

So in order to check if any method from the console object is used we have to use CallExpression as AST selector and in the handler we need to inspect the callee property to make sure the method is part of the console object and if so we report an error.

{
  meta: {
    type: "problem",
  },
  create(context) {
    return {
      CallExpression: (node) => {
        if (node.callee.object?.name === "console") {
          context.report({
            node: node,
            message: `console.${node.callee.property.name} is cool but some people doesn't like it.`,
          });
        }
      },
    };
  },
};

This of course won't cover all cases, for instance, code like the following won't be discovered by the logic we wrote above because we are no longer calling the console.log directly thus we lost the CallExpression signature. A proper solution is to report whenever a console method present and for that we've to use MemberExpression AST selector.
The why we've not actually did this because I want you to be wary of such cases that can be easily forgotten.

const log = console.log;
log()

In the shell, type npx eslint no-console.js to see the result for no-console rule.

Comments

Let's see how we can leverage the comments and make them useful in different way.

In this example we want to report a message in case a declaration annotated with @private tag is being used in specific directory (make sure to limit this rule to specific files in .eslintrc.json).

/**
 * @private Never use this function.
 */
function _internalFunction() { }

_internalFunction() // an error will be reporated

/**
 * @private and this constant
 */
const nil = null;

if(nil === null){
  // you should see warning in the console ^^
}

So in order to target all nodes we need to use the Program node in the following steps

Extract all tokens from it.
Create a loop that starts from the first token.
Ignore any token that isn't a Keyword (afaik, you can only annotate Keywords "function", "const", "class" and so on).
Ignore any keyword that doesn't have comments.
Ignore in case the comments doesn't have @private annotation.

 {
  meta: {
    type: "problem",
  },
  create(context) {
    const sourceCode = context.getSourceCode();
    return {
      Program: (node) => {
        const tokens = sourceCode.getTokens(node);
        for (let index = 0; index < tokens.length; index++) {
          const token = tokens[index];
          if (token.type !== "Keyword") {
            continue;
          }
          const comments = sourceCode.getCommentsBefore(token);
          if (!comments.length) {
            continue;
          }
          const privateTagMessage = getCommentTagMessage(comments, "@private");
          if (!privateTagMessage) {
            continue;
          }
        }
      },
    };
  },
};

Once we found a keyword with comments that have @private tag we need to find all references to it.

From the previous sample the index is pointing at the keyword token.
Get the keyword Identifier token (the identifier is always the next token after the keyword token).
Create new loop that starts from the token after the keyword identifier to gather the references.
Since we only need the references to the keyword token we need to ignore any token that is not Identifier or doesn't have the same value (identifier name) as the keyword identifier.
Once we found a reference we report the private tag message

const keywordIdentifier = tokens[index + 1];
for (const nextToken of tokens.slice(index + 2)) {
  if (
    nextToken.type === "Identifier" &&
    nextToken.value === keywordIdentifier.value
  ) {
    const message = privateTagMessage;
    context.report({
      node: nextToken,
      message: `${keywordIdentifier.value}: ${message}`,
    });
  }
}

In the shell, type npx eslint private.js to see the result for private rule.

Deprecate a function

What about you creating this to test your understanding, here's how

Create another folder in the plugins directory named deprecated.
Create rules folder inside that directory and within it another folder named deprecate-function.
In the deprecated folder create two files, index.js to export the rule and package.json to make the plugin npm module.
In the deprecate-function folder create two files, deprecate-function.rule.js and deprecate-function.test.js.

The folder structure should look like this

plugins
└── deprecated
    ├── rules
    │   └── deprecate-function
    │       ├── deprecate-function.rule.js # rule logic
    │       └── deprecate-function.test.js # rule unit test
    ├── index.js # exports the plugin rules
    └── package.json # make the plugin npm package.

Now the rule should do the following

Accepts an array of function names to deprecate (scheme option).
If the function name is not defined then ignore (don't report).
use the Identifier selector and check if its name is exist in the functions names array (you need to use context.options).
In case the If statement conditioned then report a message (you can make the message option as well!).

AST selectors

We've already talked about AST selector before so this is a complement to the writing.

You already know the CSS selectors like *, [], >, ...etc, you actually can use those selectors along with the AST selectors, for instance, if you want to find variables with name foo you can use the attribute selector instead of doing an extra if statement.

{
  create(context) {
    "VariableDeclaration[name="foo"]": () => { }
  } 
}

Postface

ESLint rules are not only a way to ensure the developer adheres to a specific style of coding, the way I see it, with ESLint you can build habits in the team members to do the things as routine and not a checklist of things they have to pass.

Following that, you can benefit from ESLint by creating rules that will help new joiners adapt easily to the codebase or folks from other teams who frequently participate in a codebase.

Final thought, ESLint rules can be developer guidelines for a codespace.

Bounce Sections

That's specially for you ^^ to enrich your information!

Abstract Syntax Tree (AST)

We already talked about compiler phase/stages and how each of them is essentially a step towards making the computer understands the code. One of these stages is Parsing which outputs AST.

AST is the source code in tree data structure that represents the relation between the tokens, for instance, this sample code

var foo = "bar"

will output the following

As you can see there's root node Program which is the source code (the variable foo), the body contains list of nodes because usually you've more than one statement 🙂. The VariableDeclaration node is a statement that have the type of declaration and because it's variable and the variable have different kinds, there's kind property.

In the declarations array there's Variable declarator that have information about the var identifier (name) and its initializer.

The node term is an abstract therefore each specific of the node have different properties related to it, for instance BinaryExpression statement (5 + 1) will have left and right properties that have information about the operands an so on.

We've been utilising the AST so far for Static Code Analysis but that's not the only usage of it. You can also create code migration tools using the AST, like changing functions names or removing parameters and more complex usage.

AST Explorer

AST Explorer is an amazing website if you're working with ASTs, it has plenty of parsers for the most of things that you might hear about, for instance it has an option to parse SQL Query and GraphQL and many more.

This this the toolbar

Beside the AST Explorer logo we have 1. Snippet: In case you want to share, reset the state of the page, or save. (New button, will clear the url and Save will let you create new url to share)

JavaScript: A dropdown where you select the language/tool that you want to parse.
espree: A dropdown where you select the a library to parse the selected language/tool.
transform toggle: some languages/tools can be transformed to older version or different syntax. This option will be enabled in case transformation is supported for the selected language/tool.

React Context In Angular

ezzabuzaid — Fri, 10 Sep 2021 15:24:16 +0000

In this article, I'm going to show you how to mimic React Context API in Angular, I'll start by defining React Context, talk about what problem is meant to solve, and a possible implementation in Angular.

I will focus more on implementation and detail it as possible rather than explaining definitions, nonetheless, I'll make sure to explain any irrelevant terms.

If at any point you don't feel interested in reading further, think of this article as a new approach for component communication in Angular.

For clarity about what I’m going to talk about, the project is available to browse through Github. or a Demo if you prefer.

What Is React Context API

From React Documentation

Context provides a way to pass data through the component tree without having to pass props down manually at every level.

props in Angular terms corresponds to Inputs

In other words, context can help you to pass down inputs/props through a components tree without the need to define them at every level/component.

Words 📝 might not be that efficient, a practical example might be.

The Problem

Here's 4 components (AppComponent, Parent, Child, Grandchild), the AppComponent passes a value to the Parent component, the Parent component will pass it to the Child component which forwards it to the Grandchild component.

@Component({
  selector: 'app-root',
  template: '<app-parent [familyName]="familyNameValue"></app-parent>'
})
export class AppComponent {
  familyNameValue = 'The Angulars';
}

@Component({
  selector: 'app-parent',
  template: '<app-child [familyName]="familyName"></app-child>'
})
export class ParentComponent {
  @Input() familyName: string;
}

@Component({
  selector: 'app-child',
  template: '<app-grandchild [familyName]="familyName"></app-grandchild>'
})
export class ChildComponent {
  @Input() familyName: string;
}

@Component({
  selector: 'app-grandchild',
  template: 'Family Name: {{familyName}}'
})
export class GrandchildComponent {
  @Input() familyName: string;
}

As you see, we had to declare the same input at every component starting from the Parent down the Grandchild, in React terms this is called Prop Drilling.

Going to the definition again

Context provides a way to pass data through the component tree without having to pass ~~props~~ Inputs down manually at every level.

Good, let's see the Context way.

The Solution

Hint: I'll explain the implementation later on. keep reading for now.

What if you can remove the inputs and only have a generic one that could be accessed from anywhere in the tree, like this

@Component({
  selector: 'app-root',
  template: `
    <context name="FamilyContext">
      <provider name="FamilyContext" [value]="familyNameValue"> // This part
        <app-grandchild> </app-grandchild>
      </provider>
    </context>
`
})
export class AppComponent { }

And for the component that needs the value

@Component({
  selector: 'app-grandchild',
  template: `
    <consumer name="FamilyContext">
        <ng-template let-value>
           Family Name: {{value}}
        </ng-template>
    </consumer>
`
})
export class GrandchildComponent { }

While this approach seems to work, I do not think a lot of people will agree on this, I myself thought about sandboxing first, maybe that's why there's no like to React Context API in Angular. but again see it as a different way to achieve the same result.

By now it's clear what problem does Context API solves. It's time to see how it works.

How Does React Context API Work

Warning: I'll use React components 😏.

Context API comes with two important components, Provider and Consumer. Provider is the component that will pass a value for decedents consuming components. One provider can have multiple consumers and other providers.

Consumer, as you may have thought, will consume Provider value. React will go up the component tree starting from the Consumer component to find the nearest Provider and provide its value to that Consumer as callback style, if none is found a default value will be used instead. The Consumer will re-render whenever a Provider ancestor value changes.

To create context you simply call createContext passing default value if needed, a context object with Provider and Consumer components attached to it will return.

const MyContext = React.createContext('defaultValue');

The provider have value props that will passed down to the consumers.

function App() {
  return (
    <MyContext.Provider value="valueToBeConsumedByDescendantsConsumer">
      <ComponentThatHaveConsumerAsChild />
    </MyContext.Provider>
  );
}

The consumer takes a function with the Provider value as an argument, the function will be called (re-render 🙃) whenever the Provider value changes.

function ComponentThatHaveConsumerAsChild() {
  return (
    <MyContext.Consumer>
      {(value) => (<h1>{value}</h1>)}
    </MyContext.Consumer>
  );
}

You might want to know that this is not the only way to consume context, there's contextType and useContext, I won't cover them because those are applicable only to React way of doing things.

if you didn't get the whole picture, check the official docs, perhaps it would be more helpful.

Enough talking about React. It's time to code.

Angular implementation

In Angular things are different, so we'll do things in different styles but the same concept and goals remain.

If you start this article from the beginning you saw that we introduced three components

context
provider
consumer

and ended up using them like this

@Component({
  selector: 'app-root',
  template: `
    <context name="FamilyContext"> // (1) -----> The Context Component
      <provider name="FamilyContext" [value]="familyNameValue"> // (2) -----> The Provider Component
        <app-parent> </app-parent>
      </provider>
    </context>
`
})
export class AppComponent { }

@Component({
  selector: 'app-grandchild',
  template: `
    <consumer name="FamilyContext"> // (3) -----> The Consumer Component
        <ng-template let-value>
           Family Name: {{value}}
        </ng-template>
    </consumer>
  `
})
export class GrandchildComponent { }

I'll explain each component in detail soon.

Utility function for strict mode people 😅

export function assertNotNullOrUndefined<T>(value: T, debugLabel: string): asserts value is NonNullable<T> {
    if (value === null || value === undefined) {
        throw new Error(`${ debugLabel } is undefined or null.`);
    }
}

export function assertStringIsNotEmpty(value: any, debugLabel: string): asserts value is string {
    if (typeof value !== 'string') {
        throw new Error(`${ debugLabel } is not string`);
    }
    if (value.trim() === '') {
        throw new Error(`${ debugLabel } cannot be empty`);
    }
}

The Context Component

This component is responsible for declaring a scope for providers and consumers, providers can only be under their context, the same rule applies to consumers.

Unlike React Context API, we don't have reference to a context so in order to ensure the relationship between providers and consumers to a context we need to give the context and its components a name.

A name makes it possible to

Have multiple contexts that can be used without interfering with each other.
The provider and consumer to find their Context easily by looking the name up.
Ensures a provider and a consumer are defined under their context and not in any other place.
Prevents duplicated contexts.

Another thing related to the context component is the defaultValue, if you recall from above if a context doesn't have any provider a default value will be used instead.

In the previous Image, Consumer ( A ) will have the value of the Context because there's no provider above it, and Consumer ( B ) will have the value of Provider ( 1 ).

Initial Implementation

@Component({
  selector: 'context',
  template: '<ng-content></ng-content>' // ----> (1)
})
export class ContextComponent implements OnInit, OnChanges {
  @Input() name!: string; // ----> (2)
  @Input() defaultValue?: any; // ----> (3)

  constructor() { }

  ngOnInit(): void {
    assertStringIsNotEmpty(this.name, 'Context name');  // ----> (4)
  }

  ngOnChanges(changes: SimpleChanges): void {
    const nameChange = changes.name;
    if (nameChange && !nameChange.isFirstChange()) {
      const { currentValue, previousValue } = nameChange;
      throw new Error(`Context name can be initialized only once.\n Original name ${ previousValue }\n New name ${ currentValue }`);
    }
  }

}

ng-content to project the content as is.
Name of the context. reasons above 😁
value that will be provided to the consuming components in case there's no provider for this context.
Ensures the context name is a string and not empty. The same check will be used in the other components.
The name cannot be changed since the code should adhere to the React approach, nevertheless, this is totally up to you. the same check will be used in the other components.

The Provider Component

This component will pass down its value to the consumers hence we need to have an input for that value. Also, you can have zero or more provider components for the same context. consumers will get the value from the nearest one.

In the previous Image, Consumer ( A ) will have the value of the Context, but Consumer ( B ), Consumer ( C ), and Consumer ( E ) will have the value of Provider ( 1 ).
Consumer ( D ) will have the value of Provider ( 2 ) because it is the nearest one.

Initial Implementation

@Component({
  selector: 'provider',
  template: '<ng-content></ng-content>'
})
export class ProviderComponent implements OnInit {
  @Input() name!: string;   // ----> (1)
  @Input() value?: any;   // ----> (2)

  ngOnInit(): void {
    assertStringIsNotEmpty(this.name, 'Provider context name');

    if (this.value === undefined) {   // ----> (3)
      throw new Error(`Provider without value is worthless.`);
    }
  }

  ngOnChanges(changes: SimpleChanges): void {
    const nameChange = changes.name;
    if (nameChange && !nameChange.isFirstChange()) {
      const { currentValue, previousValue } = nameChange;
      throw new Error(`Context name can be initialized only once.\n Original name ${ previousValue }\n New name ${ currentValue }`);
    }
  }

}

Name of the context. The name is needed in order to know to which context it belongs.
value that will be provided to the consuming components.
The provider is valuable as long it holds a value, if at first it doesn't so there's no point in having it, let the consumers rely on a different provider or the default value provided when establishing the context

The Consumer Component

The component will eventually have the value of the nearest provider or the default context value in case no provider is found up in the tree.

before digging into it, let's see the example usage first.

@Component({
  selector: 'app-grandchild',
  template: `
    <consumer name="FamilyContext">
        <ng-template let-value>
           Family Name: {{value}}
        </ng-template>
    </consumer>
`
})
export class GrandchildComponent { }

ng-template will be used as a convenient way to be able to provide the nearest provider value or the context defaultValue using template variable let-value and to have more control over the change detection process. More about this later on.

Initial Implementation

@Component({
  selector: 'consumer',
  template: '<ng-content></ng-content>',
})
export class ConsumerComponent implements OnInit {
  @Input() name!: string;   // ----> (1)
  @ContentChild(TemplateRef, { static: true }) templateRef!: TemplateRef<any>;   // ----> (2)

  ngOnInit(): void {
    assertStringIsNotEmpty(this.name, 'Consumer context name');

    if (this.templateRef === undefined) {   // ----> (3)
      throw new Error(`
        Cannot find <ng-template>, you may forget to put the content in <ng-template>.
        If you do not want to put the content in context then no point in using it.
      `);
    }
  }

  ngOnChanges(changes: SimpleChanges): void {
    const nameChange = changes.name;
    if (nameChange && !nameChange.isFirstChange()) {
      const { currentValue, previousValue } = nameChange;
      throw new Error(`Context name can be initialized only once.\n Original name ${ previousValue }\n New name ${ currentValue }`);
    }
  }

}

Name of the context. The name is needed in order to know to which context it belongs.
The template reference, static: true used to be able to get it in ngOnInit.
ng-template is mandatory. why would you need to use the consumer if you're not making use of it is value?

RECAP: all the code right now only validates the inputs.

The next step is to make sure providers and consumers components are using the correct context.

Hopefully, You know Dependency Injection and how the resolution process works. in nutshell, You inject a dependency and Angular will search for the implementation in several injectors if none is found an error will be all over the browser console 😁.

It is important to understand the resolution process in order to understand the rest of the code. the validation and value resolving logic relying on that mechanism. basically, we'll link each component type with the immediate next one above it, it is like creating a chain of components each has its parent and the final one (first on the tree) will have null. just like the Prototype Chain 😁. take a look at the next image, perhaps it will clear the idea.

Context Validation

Context should be unique, you cannot have multiple contexts with the same name.
Providers and Consumers must have a context.

First, adding a method to ContextComponent that will ensure no other context with the same name is exist.

@Component({
  selector: 'context',
  template: '<ng-content></ng-content>',
})
export class ContextComponent implements OnInit {
  @Input() defaultValue?: any;
  @Input() name!: string;

  constructor(
    @Optional() @SkipSelf() public parentContext: ContextComponent | null   // ----> (1)
  ) { }

  ngOnInit(): void {    
    assertStringIsNotEmpty(this.name, 'Context name'); 
    this.ensureContextUniqueness(this.name); // ----> (2)
  }

  ... code omitted for brevity

  public getContext(contextName: string) {  // ----> (3)
    let context: ContextComponent | null = this;
    while (context !== null) {
      if (context.name === contextName) {
        return context;
      }
      context = context.parentContext;
    }
  return undefined;
  }

  public ensureContextUniqueness(contextName: string) {   // ----> (4)
    let context: ContextComponent | null = this.parentContext;
    while (context !== null) {
      if (context.name === contextName) {
        throw new Error(`Context ${ this.name } already exist.`);
      }
      context = context.parentContext;
    }
  }

}

Inject the parent context component 😲 Check the previous image.

@Optional() is used to implies that this context may be the first context in the tree, therefore no parents will be found.
@SkipSelf() is used to tell the dependency resolution to skip the current component injector and start the process from the parent injector because we already have the current context.

Checks if a context with the same name already exists and if so throws an error.
Find a context by a name, starting from the current context, check if its name is equal to the parameter, if not equal repeat the same step with the parent. In the end, if no context is found return undefined. This method will be needed later on with the other components.
Like point 3, but start with the parent context and not the context itself.

Second, modify the ProviderComponent to grab its context and ensure that it exists.

@Component({
  selector: 'provider',
  template: '<ng-content></ng-content>'
})
export class ProviderComponent implements OnInit {
  @Input() name!: string;
  @Input() value?: any;
  private providerContext!: ContextComponent;

  constructor(
    @Optional() private context: ContextComponent | null,    // ----> (1)
  ) { }

  ngOnInit(): void {
    ... code omitted for brevity

    if (this.context === null) {    // ----> (2)
      throw new Error(
        'Non of provider ancestors is a context component,
         ensure you are using the provider as a context descendant.'
      );
    }

    this.providerContext = this.context.getContext(this.name);  // ----> (3)
    assertNotNullOrUndefined(this.providerContext, `Provider context ${this.name}`);  // ----> (4)
  }

  public getProvider(contextName: string) {  // ----> (5)
    let provider: ProviderComponent | null = this;
    while (provider !== null) {
      if (provider.name === contextName) {
        return provider;
      }
      provider = provider.parentProvider;
    }
    return undefined;
  }

}

Inject the ContextComponent. Angular will search for the nearest context component and inject it, this component will be used to search for another context up in the tree.
Check if there's context at all before searching for the provider context. this might be helpful so you immediately know you missed adding the context.
Get the provider context and assign it to its instance.
Ensure the provider has context.
Find a provider by a context name, starting from the current provider, check if its name is equal to the parameter, if not equal repeat the same step with the parent. In the end, if no provider is found it is okay to return undefined to state that a context doesn't have a provider since it's optional. This method will be needed soon in the consumer component.

Third, modify the ConsumerComponent to grab its context and provider and ensure its context is exists.

@Component({
  selector: 'consumer',
  template: '<ng-content></ng-content>',
})
export class ConsumerComponent implements OnInit {
  @Input() name!: string; 
  @ContentChild(TemplateRef, { static: true }) templateRef!: TemplateRef<any>;
  private consumerContext!: ContextComponent;
  private consumerProvider?: ProviderComponent;

  constructor(
    @Optional() private context: ContextComponent  // ----> (1)
  ) { }

  ngOnInit(): void {
    ... code omitted for brevity

    if (this.context === null) {   // ----> (2)
      throw new Error(
        'Non of consumer ancestors is a context component,
         ensure you are using the consumer as a context descendant.'
      );
    }
    this.consumerContext = this.context.getContext(this.name);  // ----> (3)
    this.consumerProvider = this.provider?.getProvider?.(this.name);  // ----> (4)
    assertNotNullOrUndefined(this.consumerContext, `Consumer context ${this.name}`);  // ----> (5)
  }
}

Inject the ContextComponent. Angular will search for the nearest context and inject it.
Check if there's context at all before searching for the consumer context. this might be helpful so you immediately know you missed adding the context.
Get the consumer context and assign it to its instance.
Ensure the consumer has a context.
Get the consumer nearest provider and assign it to the consumer instance. This will be used next to observe provider value changes.

RECAP: The code validates the inputs and ensures a context exists and only one exists and is correctly used, also it guides the developer on how to use the context and its components.

Now, it's time for getting the value from the context and the nearest provider to the consumer.

Providing the `value`

If you start this article from the beginning you've read that

The Consumer will re-render whenever a Provider ancestor value changes.

That means the ng-template should be updated as well and not just building it the first time.

Providing the value might seem easy at first glance since you just need to build the ng-template and bind a value to it, while that is correct, there're other concerns when it comes to Angular Change Detection, for instance updating the template value in a component that is using OnPush change detection strategy is difficult than the normal component that uses the Default change detection strategy, more information about this soon in sepearted section.

For building, there's ViewContainerRef the create and host the ng-template, also it returns a reference to the ng-template so we can use it to update its value. more examples and information.

@Component({
  selector: 'consumer',
  template: '<ng-content></ng-content>',
})
export class ConsumerComponent implements OnInit, OnDestroy {
  ... code omitted for brevity

  private buildTemplate(initialValue: any) {   // ----> (1)
    this.embeddedView = this.viewContainerRef.createEmbeddedView(this.templateRef, {
      $implicit: initialValue
    });
  }

  private updateTemplate(newValue: string) {   // ----> (2)
    this.embeddedView!.context = {
      $implicit: newValue
    };
    this.embeddedView?.markForCheck();
  }

  private render(value: any) {   // ----> (3)
    if (this.embeddedView) {
      this.updateTemplate(value);
    } else {
      this.buildTemplate(value);
    }
  }

}

Create the template passing it the initial value (which could be its context default value or its nearest provider current value) and stores the ng-template reference for later use.
Update the template value, the let-value, and mark it to be checked in the next change detection cycle.
Wrapper method to either update the template in case it is already there or build it otherwise.

For value changes, normally, the lifecycle that is used to observe @Input changes is OnChanges, but since the value is not passed directly to the consumer component it cannot be used there.

The ProviderComponent will have the ReplaySubject that will emit the new provider value and the ConsumerComponent will subscribe to that subject to update its template.


@Component({
  selector: 'provider',
  template: '<ng-content></ng-content>'
})
export class ProviderComponent implements OnInit, OnDestroy {
  private valueState = new ReplaySubject<any>(1);   // ----> (1)

  ngOnChanges(changes: SimpleChanges): void {   // ----> (2)
    const valueChange = changes.value;
    if (valueChange) {
      this.brodcaseValueChanges(valueChange.currentValue);
    }
  }

  ... code omitted for brevity

  private brodcaseValueChanges(newValue: any) {
    this.valueState.next(newValue);
  }

  public valueChanges() {   // ----> (3)
    return this.valueState.asObservable();
  }

  ngOnDestroy(): void {
    this.valueState.complete();   // ----> (4)
  }

}

Initialize the ReplaySubject with a buffer up to 1 so the new consumers will always be able to access the provider's last value.
Modify the ngOnChanges lifecycle that was used before to ensure the context name doesn't change to have the logic of detecting the provider value changes.
Convert the ReplaySubject to observable for the consumers' components.
On ProviderComponent destroy, complete the ReplaySubject to free up the memory.

Now with the ConsumerComponent part


@Component({
  selector: 'consumer',
  template: '<ng-content></ng-content>',
})
export class ConsumerComponent implements OnInit, OnDestroy {

  private providerValueChangesSubscription?: Subscription;  // ----> (1)

  ngOnInit(): void {
    if (this.consumerProvider) {  // ----> (2)
      this.providerValueChangesSubscription = this.consumerProvider
        .valueChanges()
        .subscribe((providerValue) => {
          this.render(providerValue);  // ----> (3)
        });
    } else {  // ----> (4)
      this.render(this.consumerContext.defaultValue);
    }
  }

  ... code omitted for brevity

  ngOnDestroy(): void {
    this.providerValueChangesSubscription?.unsubscribe();  // ----> (5)
  }

}

A field to hold the provider subscription to unsubscribe on the component destroy.
Check if a provider is defined to subscribe to its value changes.
If there's a provider re-render on its value changes
If there's no provider render only once with the context default value.
Unsubscribe from the provider ReplaySubject on component destroy.

Well, you made it so far, good for you! 😄✌️, now you have React Context in Angular, how great was that?
Let's see the Angular way of sharing data in the component tree.

The Angular Way

Angular does have Dependency Injection framework that provides different approaches to handle a situation where something like React Context API is needed.

In the "The Problem" section, you saw that to pass down a value to the descendants' components you have to declare an @Input at every component even though that a component might merely act as a wrapper for another component. This actually can be changed by providing an InjectionToken to the ancestor component and inject that token at any descendant component to utilize the value.

Change the root component to include the InjectionToken

const FamilyNameToken = new InjectionToken('FamilyName');
@Component({
  selector: 'app-root',
  template: `<app-grandchild> </app-grandchild>`,
  providers: [{provide: FamilyNameToken, useValue: 'The Angulars'}]
})
export class AppComponent { }

And for the component that needs the value to inject the InjectionToken

@Component({
  selector: 'app-grandchild',
  template: `Family Name: {{familyNameValue}}`
})
export class GrandchildComponent {
  constructor(@Inject(FamilyNameToken) public familyNameValue: string) { }
 }

That might look easy and simple at first, but the catch is when you want to update the value you need to have a kind of RxJS Subject because Angular will hard inject the value that corresponds to the InjectionToken into the GrandchildComponent. The other way is to use a class provider to act as a state holder.

class FamilyName {
  private state = new ReplaySubject(1);
  public setName(value: string) {
    this.state.next(value);
   }
  public getName() {
    return this.state.asObservable();
  }
}

The root component will inject the class and sets the value.

@Component({
  selector: 'app-root',
  template: `<app-grandchild> </app-grandchild>`,
  providers: [FamilyName]
})
export class AppComponent {
  constructor(public familyName: FamilyName) {
    $familyNameState = this.familyName.setName('The Angulars');
  }
}

And for the component that needs the value to inject the FamilyName class and subscribe to the changes.

@Component({
  selector: 'app-grandchild',
  template: `Family Name: {{$familyNameState|async}}`
})
export class GrandchildComponent {
  $familyNameState = this.familyName.getName();
  constructor(public familyName: FamilyName) { }
 }

Also, you can re-provide the FamilyName class at any component level so it can act as the ProviderComponent.

With that said, having a way to pass down a value within the component template it self can reduce the amount of class you will need.

Example

To put the implementation in action, I'll use chat components to illustrate the usage of the context.

Follow the demo to see the result.

Chat Message Component
Uses consumer to obtain the message

@Component({
    selector: 'app-chat-message',
    template: `
    <consumer name="ChatContext">
        <ng-template let-value>
            <h4>{{value.message}}</h4>
        </ng-template>
    </consumer>
    `
})
export class ChatMessageComponent { }

Chat Avatar Component
Uses consumer to obtain the avatar. notice the changeDetection is changed to OnPush.

@Component({
    selector: 'app-chat-avatar',
    changeDetection: ChangeDetectionStrategy.OnPush,
    template: `
    <consumer name="ChatContext">
        <ng-template let-value>
            <img width="50" [src]="value.avatar">
        </ng-template>
    </consumer>
    `
})
export class ColorAvatarComponent { }

Chat Container Component
Group the other components and perhaps for styling and aligning. it uses the provider declared in AppComponent for the first chat message and a new provider for the second chat message

@Component({
    selector: 'app-chat-container',
    template: `
    <div style="display: flex;">
        <app-chat-avatar></app-chat-avatar>
        <app-chat-message></app-chat-message> 
        <provider name="ChatContext" [value]="{name:'Nested Provider Value'}">
            <app-chat-message></app-chat-message>
        </provider>
    </div>
    `
})
export class ChatContainerComponent { }

App Component
Declare a context with the name ChatContext with no default value and a provider with initial value chatItem which will be shared to ChatMessageComponent and ChatAvatarComponent.

Clicking on the Change Chat Item button will update the chatItem reference hence updating the consumers to get the new value.

@Component({
  selector: 'app-root',
  template: `
  <context name="ChatContext">
    <provider [value]="chatItem" name="ChatContext">
      <app-chat-container></app-chat-container>
    </provider>
  </context>
  <button (click)="updateChatItem()">Change Chat Item</button>
  `
})
export class AppComponent {
  chatItem = {
    message: 'Initial name',
    avatar: 'https://icon-library.com/images/avatar-icon-images/avatar-icon-images-4.jpg',
  }

  updateChatItem() {
    const randomInt = Math.round(Math.random() * 10);
    this.chatItem = {
      message: `Random ${ randomInt }`,
      avatar: `https://icon-library.com/images/avatar-icon-images/avatar-icon-images-${ randomInt }.jpg`,
    }
  }

}

Bonus Part: The Issue With OnPush

In the Angular Implementation section, there's was an issue when a consumer host component (the component that will be the consumer parent) is using the OnPush change detection strategy so as fix a ReplaySubject used to share the value to the consumer component from its nearest provider.

The thing is that OnPush prevents the component from being auto-checked thus the template of the component won't be updated except on special cases.

One of the components @Input reference changed.
An event handler of the component was triggered.
An observable linked to the component template via the async pipe emits.

Unfortunately, neither of the cases above is applicable on The ConsumerComponent

It doesn't have an @Input for the value because it will be bonded indirectly.
It doesn't have any event handler.
And no observable can be linked to its template since we project the content as is.

Hint: component template implies the template property in the @Component decorator and doesn't refer to ng-template.

The other solution and the initial implementation was to use The DoCheck lifecycle because it is usually used when a component is using OnPush change detection strategy to detect changes to mutable data structures and mark the component for the next change detection check cycle accordingly.

Moreover, the DoCheck lifecycle will be invoked during every change detection run but with OnPush the change detector will ignore the component so it won't be invoked unless it happens manually and again even this is out of the scope because you do not know if the consumer provider value changed or not.

That was just a plus section for the folks who will wonder about that.

Summary

If you didn't use state management libraries before, you might find this handy since it somehow solves the same issue, and if you coming from React background this can be an advantage to have in Angular, nevertheless, Angular can do it on its own with a bit knowledge of dependency injection.

Having such functionality in your app can impart additional value, on the other hand, you have to adapt to the new way of sharing data.

The drawing was made via Excalidraw.

Advance Angular Dynamic Component

ezzabuzaid — Fri, 06 Aug 2021 13:40:33 +0000

In this article, I will show you how you can still use inputs and outputs and support OnChanges lifecycle while creating dynamic components.

If you don't know about dynamic components yet, I recommend this article Dynamically Creating Components with Angular before going forward.

For clarity about what I’m going to talk about, the project is available to browse through Github. or a Demo if you prefer.

UPDATE - 5/8/2023
The component inspection API has been deprecated as of Angular V13, If you're using Angular v14 and above update your implementation to follow "https://github.com/ezzabuzaid/dynamic-component-article/blob/main/src/app/dynamic-component.directive.v14.ts"

The Problem

In order to create a dynamic component, you have to use either ngComponentOutlet directive or ComponentFactoryResolver object, neither provides a way to bind inputs and outputs.

moreover, ngOnChanges won't work, This is because the function that performs inputs checks is generated by the compiler during compilation.

The Solution

To work around the problem we would use a custom directive that could help as little as possible to facilitate the bindings.

We will use ComponentFactoryResolver to create a component factory that holds metadata about the component inputs and outputs. this metadata will be used to ensure correct properties names of inputs and outputs are used.

const factory = componentFactoryResolver.resolveComponentFactory(ComponentType);

factory has two getters that represent the component inputs and outputs.

/**
 * The inputs of the component.
 */
abstract get inputs(): {
    propName: string;
    templateName: string;
}[];
/**
 * The outputs of the component.
 */
abstract get outputs(): {
    propName: string;
    templateName: string;
}[];

Each of which has propName and templateName that corresponds to

@Input(templateName) propName;
@Output(templateName) propName;

templateName defaults to propName if not specifed.

Setup

Our directive would be used like this

<ng-template [dynamic-component]="component" [inputs]="{}" [outputs]="{}"> </ng-template>

Types that will be used in the code

type UserOutputs = Record<string, (event: any) => void>;
type UserInputs = Record<string, any>;
type ComponentInputs = ComponentFactory<any>['inputs'];
type ComponentOutputs = ComponentFactory<any>['outputs'];
type Color = 'red' | 'blue' | 'green';

Utility function for strict mode people 😅

function assertNotNullOrUndefined<T>(value: T): asserts value is NonNullable<T> {
    if (value === null || value === undefined) {
        throw new Error(`cannot be undefined or null.`);
    }
}

The directive

@Directive({
    selector: '[dynamic-component]',
})
export class DynamicComponentDirective implements OnDestroy, OnChanges {
  @Input('dynamic-component') component!: Type<any>;
  @Input() outputs?: UserOutputs = {};
  @Input() inputs?: UserInputs = {};
  ngOnChanges(changes: SimpleChanges) { }
  ngOnDestroy() { }
}

To complete the setup we need to make sure that

outputs/inputs object corresponds to component outputs/inputs, no incorrect names used.
component ngOnChange runs on input change.
outputs EventEmitter are auto unsubscribed from.

I'll show a few functions implementation to better highlight how things are done. You might want to check the complete code while reading the following section.

Validation

Since this is not Angular out-of-the-box solution we can't ensure the correct inputs/outputs names are used hence manual validation is required to avoid hidden issues.

As mentioned above ComponentFactory object will be used to inspect component inputs and outputs,

Inputs

Loop over the user-provided inputs, check if each provided input is declared in the component as Input.
A component input is a field decorated with @Input.

private validateInputs(componentInputs: ComponentInputs, userInputs: UserInputs) {
  const userInputsKeys = Object.keys(userInputs);
  userInputsKeys.forEach(userInputKey => {
      const componentHaveThatInput = componentInputs.some(componentInput => componentInput.templateName === userInputKey);
      if (!componentHaveThatInput) {
          throw new Error(`Input ${ userInputKey } is not ${ this.component.name } input.`);
      }
  });
}

Outputs

Loop over the component outputs, check if each output holds an instance of EventEmitter.
A component output is a field decorated with @Output and has EventEmitter instance as value.

In the other part, we perform a loop over the user-provided outputs, check if each provided output is declared in the component as Output and if the user-provided output is function. that function will be used as EventEmitter handler.

private validateOutputs(componentOutputs: ComponentOutputs, userOutputs: UserOutputs, componentInstance: any) {
  componentOutputs.forEach((output) => {
      if (!(componentInstance[output.propName] instanceof EventEmitter)) {
          throw new Error(`Output ${ output.propName } must be a typeof EventEmitter`);
      }
  });

  const outputsKeys = Object.keys(userOutputs);
  outputsKeys.forEach(key => {
      const componentHaveThatOutput = componentOutputs.some(output => output.templateName === key);
      if (!componentHaveThatOutput) {
          throw new Error(`Output ${ key } is not ${ this.component.name } output.`);
      }
      if (!(userOutputs[key] instanceof Function)) {
          throw new Error(`Output ${ key } must be a function`);
      }
  });
}

Binding

Binding is pretty straightforward now since we won't have incorrect inputs/outputs names.

Inputs

private bindInputs(componentInputs: ComponentInputs, userInputs: UserInputs, componentInstance: any) {
  componentInputs.forEach((input) => {
      const inputValue = userInputs[input.templateName];
      componentInstance[input.propName] = inputValue;
  });
}

Outputs

takeUntil operator used to unsubscribe from the EventEmitter instance later on.
this.subscription is an instance of Subject, which will be declared in the next sections.

private bindOutputs(componentOutputs: ComponentInputs, userOutputs: UserInputs, componentInstance: any) {
  componentOutputs.forEach((output) => {
      (componentInstance[output.propName] as EventEmitter<any>)
          .pipe(takeUntil(this.subscription))
          .subscribe((event) => {
              const handler = userOutputs[output.templateName];
              if (handler) { // in case the output has not been provided at all
                  handler(event);
              }
          });
  });
}

Creating The Component

Creating dynamic components is done using ComponentFactoryResolver and ViewContainerRef.
First, we create a factory using ComponentFactoryResolver, the factory contains the metadata to perform inputs/outputs validation.

Second, we use that factory to create the component using ViewContainerRef, it also takes the injector, which will be declared later on.

private createComponent() {
  this.componentFactory = this.componentFactoryResolver.resolveComponentFactory(this.component);
  this.componentRef = this.viewContainerRef.createComponent<any>(this.componentFactory, 0, this.injector);
}

Cleaning up

To destroy a component we invoke the destroy method defined in ComponentRef, then we clear ViewContainerRef which holds the actual component, doing so will also remove it from the UI.

private destroyComponent() {
  this.componentRef?.destroy();
  this.viewContainerRef.clear();
}

the cleanup will be performed in ngOnDestroy lifecycle, the subscription is as mentioned previously an instance of Subject that we used to unsubscribe from EventEmitter subscriptions.

ngOnDestroy(): void {
  this.destroyComponent();
  this.subscription.next();
  this.subscription.complete();
}

Combine The Functions

Let's call the functions, ngOnChanges lifecycle will be used to create the component whenever the component input or injector input changes, in that case, we destroy the previous component first, then we create the new component.

after that, we perform the validation then bind the inputs and outputs.

private subscription = new Subject();
@Input('dynamic-component') component!: Type<any>;
@Input() outputs?: UserOutputs = {};
@Input() inputs?: UserInputs = {};
@Input() injector?: Injector;

ngOnChanges(changes: SimpleChanges): void {
  // ensure component is defined
  assertNotNullOrUndefined(this.component);

  const shouldCreateNewComponent =
      changes.component?.previousValue !== changes.component?.currentValue
      ||
      changes.injector?.previousValue !== changes.injector?.currentValue;

  if (shouldCreateNewComponent) {
      this.destroyComponent();
      this.createComponent();
  }

  // to make eslint happy ^^
  assertNotNullOrUndefined(this.componentFactory);
  assertNotNullOrUndefined(this.componentRef);

  this.subscription.next(); // to remove old subscription
  this.validateOutputs(this.componentFactory.outputs, this.outputs ?? {}, this.componentRef.instance);
  this.validateInputs(this.componentFactory.inputs, this.inputs ?? {});
  this.bindInputs(this.componentFactory.inputs, this.inputs ?? {}, this.componentRef.instance);
  this.bindOutputs(this.componentFactory.outputs, this.outputs ?? {}, this.componentRef.instance);
}

with that, in place, we have all the required functionality to do what [ngComponentOutlet] can't.

The ngOnChanges

So far we can completely create dynamic components, but we can't use ngOnChanges lifecycle since it doesn't react to @Input changes therefore we have to do this manually.

Another way to do this is to change the @Input field that concerned you to have getter and setter, so you can know when a change happens, but it is not a favorable option so let's stick with ngOnChanges.

Let's start with creating changes object for the component.
Basically, do a loop over new inputs (currentInputs) and compare each input with the previous one, in case of change we add it as changed input to the changes object

private makeComponentChanges(inputsChange: SimpleChange, firstChange: boolean): Record<string, SimpleChange> {
  const previuosInputs = inputsChange?.previousValue ?? {};
  const currentInputs = inputsChange?.currentValue ?? {};
  return Object.keys(currentInputs).reduce((changes, inputName) => {
  const currentInputValue = currentInputs[inputName];
  const previuosInputValue = previuosInputs[inputName];
  if (currentInputValue !== previuosInputValue) {
      changes[inputName] = new SimpleChange(firstChange ? undefined : previuosInputValue, currentInputValue, firstChange);
  }
  return changes;
  }, {} as Record<string, SimpleChange>);
}

Now, we have to manually call the ngOnChanges from the component instance if the component declared it and passes changes as an argument.

Let's modify directive ngOnChanges to have the functionality

ngOnChanges(changes: SimpleChanges): void {
    // ensure component is defined
  assertNotNullOrUndefined(this.component);

  let componentChanges: Record<string, SimpleChange>;
  const shouldCreateNewComponent =
      changes.component?.previousValue !== changes.component?.currentValue
      ||
      changes.injector?.previousValue !== changes.injector?.currentValue;

  if (shouldCreateNewComponent) {
      this.destroyComponent();
      this.createComponent();
      // (1) 
      componentChanges = this.makeComponentChanges(changes.inputs, true);
  }
  // (2)
  componentChanges ??= this.makeComponentChanges(changes.inputs, false);

  assertNotNullOrUndefined(this.componentFactory);
  assertNotNullOrUndefined(this.componentRef);

  this.validateOutputs(this.componentFactory.outputs, this.outputs ?? {}, this.componentRef.instance);
  this.validateInputs(this.componentFactory.inputs, this.inputs ?? {});

  // (3)
  if (changes.inputs) {
      this.bindInputs(this.componentFactory.inputs, this.inputs ?? {}, this.componentRef.instance);
  }

  // (4)
  if (changes.outputs) {
      this.subscription.next(); // to remove old subscription
      this.bindOutputs(this.componentFactory.outputs, this.outputs ?? {}, this.componentRef.instance);
  }

  // (5)
  if ((this.componentRef.instance as OnChanges).ngOnChanges) {
      this.componentRef.instance.ngOnChanges(componentChanges);
  }
}

Create changes object with firstChange as true after creating the component.
In case the component didn't change that means only the inputs or outputs did change so we create changes object with firstChange as false.
Rebind the inputs only if they did change.
Rebind the outputs only if they did change.
Calling component ngOnChanges lifecycle with the possible inputs changes.

Example

Time to try it out. Demo

Here's a simple component that displays a color based on input and emits an event when it changes.

import { Component, EventEmitter, Input, OnChanges, Output, SimpleChanges } from '@angular/core';

@Component({
  selector: 'app-color-box',
  template: `<div style="height: 250px; width: 250px;" [style.background-color]="backgroundColor"></div>`,
})
export class ColorBoxComponent implements OnChanges {
  @Input() backgroundColor: Color = 'red';
  @Output() backgroundColorChanges = new EventEmitter<Color>();

  ngOnChanges(changes: SimpleChanges): void {
    this.backgroundColorChanges.next(changes.backgroundColor);
  }
}

Host component declares <ng-template> with ColorBoxComponent as the dynamic-component with inputs and outputs.
Clicking on Change Color button will invoke ngOnChanges of ColorBoxComponent, just as it should be.

Try to change the input name and you'll see an exception thrown in the console.

A bit about outputs, you'll need to use an arrow function syntax to have this referring to the AppComponent instance.

import { Component } from '@angular/core';
import { ColorBoxComponent } from './color-box.component';

@Component({
  selector: 'app-root',
  template: `
  <ng-template
   [dynamic-component]="component"
   [inputs]="{backgroundColor: backgroundColor}"
   [outputs]="{backgroundColorChanges: onColorChange}">
  </ng-template>
  <button (click)="changeColor()">Change Color</button>
`,
  styleUrls: ['./app.component.css']
})
export class AppComponent {
  component = ColorBoxComponent;
  backgroundColor: Color = 'green';

  onColorChange = (value: Color) => {
    console.log(value, this.backgroundColor);
  }

  changeColor() {
    this.backgroundColor = 'blue';
  }
}

Conclusion

Dynamic component is almost required in every project and having the ability to deal with it in an easy way is important.

Lastly, there's already a package that does all of that and a bit more ng-dynamic-component.

Resources

Using Inline Web Workers To Improve UI Performance

ezzabuzaid — Sun, 23 May 2021 18:26:08 +0000

I'll demonstrate how you can use inline web workers to perform heavy computation without blocking the main thread.

Web Worker

According to the Mozilla Developers Network (MDN) documentation

Web Workers makes it possible to run a script operation in a background thread separate from the main execution thread of a web application. The advantage of this is that laborious processing can be performed in a separate thread, allowing the main (usually the UI) thread to run without being blocked/slowed down.

In other words, Web workers enable you to run JavaScript in a separate thread.

Worth mention that this is not a comprehensive post about web workers, my aim here just to show how you can use inline web workers easily.

Creating Web Worker

There are two kinds of Web Workers, Dedicated Workers, and Shared Workers, the difference is Shared Workers can be accessed by any script that comes from the same domain whilst Dedicated Workers can only be accessed from the same script that created it.

Inline web worker comes under Dedicated Workers kind.

the difference between inline web worker and web worker is that inline web worker could be created using Blob object different that web worker that need standalone file.

we'll use the compute helper function to create web workers.

Compute Helper

compute is generic function that has two parameter, computation and message

it spawns an inline web worker, executes the computation argument passing it the message argument, and returns a promise with either rejection in case of error or resolve when finish.

function fibonacci(num) {
    if (num <= 1) return 1;
    return fibonacci(num - 1) + fibonacci(num - 2);
}

// fibonacci(10) will be running in inline web worker.
const result = await compute(fibonacci, 10);

don't worry about worker termination, it will be terminated immediately after resolving the result.

Let's see the example for better understanding.

For clarity about what I’m going to talk about, the full project is available to browse through Github.

Running without web worker

Here's an image explaining what will happen to the main thread when running CPU intensive tasks.

you should see the battery hanging for at least two seconds.

Running with web worker

Executing the same code but in inline web worker

Additional Resources

MDN: Web Workers

Codepen: UI

html5rocks: The Basics of Web Workers

Twilio: Optimizing JavaScript Application Performance with Web Workers

JavaScript Dart Compute

ezzabuzaid — Fri, 20 Nov 2020 17:06:46 +0000

Dart lang offers an optimistic compute function that can execute a code on a separate isolate(thread) to avoid blocking the main thread or to do heavy computation away from the main thread.

It will "Spawn an isolate, run callback on that isolate, passing it a message, and (eventually) returns the value returned by callback."

In contrast, JavaScript also have threads! or web workers that can
"run scripts in background threads without interfering with the user interface".

hence, we can do compute in JavaScript that will spawn worker, run callback on that worker, passing it a message, and (eventually) returns the value returned by callback."

Example

function factorial(number) {
  if (number <= 0) {
    return 1;
  } else {
    return (number * factorial(number - 1));
  }
};

await compute(factorial, 200);

References
"https://lnkd.in/d5PVjji"
"https://lnkd.in/dqqdBHt"