DEV Community: Craig ☠️💀👻

How does TypeScript's explicit resource management work?

Craig ☠️💀👻 — Mon, 23 Oct 2023 09:30:55 +0000

TypeScript has a fun new using keyword which lets us play with explicit resource management! In this context a "resource" could be a database connection, a file handle, a worker thread - anything that we might need to clean-up in the context of our application.

The release notes provide all the info you need about how consuming using works, but I thought it'd be fun to look at how it desugars to reverse engineer how it works...

Resource management 🤡

Say an application needs to temporarily write some data to disk. It might use fs.openSync to get a handle to the file and then use the handle to actually save the data.



import * as fs from 'node:fs';
import { tmpdir } from 'node:os';
import * as path from 'node:path';

export function tmpData(tmp: unknown): void {
  const tmpPath = path.join(tmpdir(), `${Math.random()}`);
  const file = fs.openSync(tmpPath, 'w+');
  fs.writeSync(file, JSON.stringify(tmp));
}

Before you say anything - yeah, this code doesn't correctly clean up after itself! 🔥

But doing this "correctly" is actually pretty hard, especially if there are multiple resources in use! Even if we're super careful and make sure that all the possible different code paths dispose of the resources, it can be super fragile and susceptible to breaking on future changes.

New shiny ✨

The explicit resource management proposal tries to make it a bit easier for us, by allowing the resource to declare how it should be managed, rather than expecting us to clean everything up when we use the resource. We get a new keyword using to define a variable (rather than const or let), which tells the runtime to clean up the resource at the end of the function.

It looks something like this:



import * as fs from 'node:fs';
import { tmpdir } from 'node:os';
import * as path from 'node:path';

export function tmpData(tmp: unknown): void {
  const tmpPath = path.join(tmpdir(), `${Math.random()}`);
  using file = fs.openSync(tmpPath, 'w+');
  fs.writeSync(file, JSON.stringify(tmp));
}

Doesn't seem like much difference, but if we look at the compiled JavaScript we can see just how much this is doing for us!

Don't feel the need to decipher it all yet, we're gonna break it down I promise 👻



var __addDisposableResource = (this && this.__addDisposableResource) || function (env, value, async) {
  if (value !== null && value !== void 0) {
    if (typeof value !== "object" && typeof value !== "function") throw new TypeError("Object expected.");
    var dispose;
    if (async) {
      if (!Symbol.asyncDispose) throw new TypeError("Symbol.asyncDispose is not defined.");
      dispose = value[Symbol.asyncDispose];
    }
    if (dispose === void 0) {
      if (!Symbol.dispose) throw new TypeError("Symbol.dispose is not defined.");
      dispose = value[Symbol.dispose];
    }
    if (typeof dispose !== "function") throw new TypeError("Object not disposable.");
    env.stack.push({ value: value, dispose: dispose, async: async });
  }
  else if (async) {
    env.stack.push({ async: true });
  }
  return value;
};

var __disposeResources = (this && this.__disposeResources) || (function (SuppressedError) {
  return function (env) {
    function fail(e) {
      env.error = env.hasError ? new SuppressedError(e, env.error, "An error was suppressed during disposal.") : e;
      env.hasError = true;
    }
    function next() {
      while (env.stack.length) {
        var rec = env.stack.pop();
        try {
          var result = rec.dispose && rec.dispose.call(rec.value);
          if (rec.async) return Promise.resolve(result).then(next, function(e) { fail(e); return next(); });
        }
        catch (e) {
          fail(e);
        }
      }
      if (env.hasError) throw env.error;
    }
    return next();
  };
})(typeof SuppressedError === "function" ? SuppressedError : function (error, suppressed, message) {
  var e = new Error(message);
  return e.name = "SuppressedError", e.error = error, e.suppressed = suppressed, e;
});

import * as fs from 'node:fs';
import { tmpdir } from 'node:os';
import * as path from 'node:path';

export function tmpData(tmp) {
  const env_1 = { stack: [], error: void 0, hasError: false };
  try {
    const tmpPath = path.join(tmpdir(), `${Math.random()}`);
    const file = __addDisposableResource(env_1, fs.openSync(tmpPath, 'w+'), false);
    fs.writeSync(file, JSON.stringify(tmp));
  }
  catch (e_1) {
    env_1.error = e_1;
    env_1.hasError = true;
  }
  finally {
    __disposeResources(env_1);
  }
}

Transpiler output 💻:

So that's quite a lot of code that's generated for us by just using (😅) the using keyword! TypeScript has prepended some helper functions (__addDisposableResource and __disposeResources) that do the actual using magic, and then modified the code where we used using. Let's look at those changes first:



// Before:
export function tmpData(tmp: unknown): void {
  const tmpPath = path.join(tmpdir(), `${Math.random()}`);
  using file = fs.openSync(tmpPath, 'w+');
  fs.writeSync(file, JSON.stringify(tmp));
}



// After:
export function tmpData(tmp) {
  const env_1 = { stack: [], error: void 0, hasError: false };
  try {
    const tmpPath = path.join(tmpdir(), `${Math.random()}`);
    const file = __addDisposableResource(env_1, fs.openSync(tmpPath, 'w+'), false);
    fs.writeSync(file, JSON.stringify(tmp));
  }
  catch (e_1) {
    env_1.error = e_1;
    env_1.hasError = true;
  }
  finally {
    __disposeResources(env_1);
  }
}

First, we can see that the transpiler has added a new env_1 variable. It initially has an empty stack array, an error property that starts undefined, and hasError set to false. This seems like a function-level context object for the entire using scope, something like:



export type DisposalScope = {
  stack: Array<unknown>; // Don't know what this is yet
  error: unknown; // You can throw anything in JavaScript
  hasError: boolean;
}

Next, the entire body of the function has been wrapped in try/catch/finally. This guarantees that no matter what we do in the function, the execution will be guarded by the "resource management" functionality.

Question: What happens if we add a return statement before the using declaration?



// Before:
export function tmpData(tmp: unknown): void {
  if (Math.random() > 0.5) {
    return;
  }
  // ...
}



// After:
export async function tmpData(tmp) {
  const env_1 = { stack: [], error: void 0, hasError: false };
  try {
    if (Math.random() > 0.5) {
      return;
    }
  }
  // ...
}

Answer: Nothing different, seems like the try/catch/finally will always wrap the full function body. Makes sense, and seems like the simplest solution!

The catch block is kind of interesting - we can see that it uses env_1 to capture information about the error that was caught. The error is stored so that it can be referenced in the finally block - and rethrown later. You might wonder why error and hasError are both needed - I did! But then I remembered that throw undefined is totally valid in JavaScript, so you can't use hasError = !!error.

We can also see that the declaration expression has been wrapped in __addDisposableResource:



// Before:
using file = fs.openSync(tmpPath, 'w+');



// After:
const file = __addDisposableResource(env_1, fs.openSync(tmpPath, 'w+'), false);

So we can see that using becomes a const, which means we can't later assign a different handle to the variable. Seems reasonable, and like the least magical option, but I can't think of a reason right now that let wouldn't work - let me know if you know why!

__addDisposableResource takes three arguments, firstly the env_1 scope context, then the actual handle to the resource, and lastly a boolean flag (called async) in the code.

Question: How do we change async from true to false?

Answer:
This doesn't change it:



// Before:
using file1 = await fs.openSync(tmpPath, 'w+');



// After:
const file = __addDisposableResource(env_1, await fs.openSync(tmpPath, 'w+'), false);

But this does:



// Before:
await using file1 = fs.openSync(tmpPath, 'w+');



// After:
const file = __addDisposableResource(env_1, await fs.openSync(tmpPath, 'w+'), true);

So we have some additional async semantics that we need to learn more about! I guess this is for cases where you have to do asynchronous clean-up. (I wonder if there is a way to trigger this in a sync function with then?)

Finally (lol), __disposeResources is called in the finally block. It just takes one argument, the env_1 function scope. Since env_1 has access to any thrown errors, __disposeResources must be responsible for handling any errors thrown in the function body.

Helper functions 💪🏻:

`__addDisposableResource` 🗑️:

Let's look at __addDisposableResource first:



var __addDisposableResource = (this && this.__addDisposableResource) || function (env, value, async) {
  if (value !== null && value !== void 0) {
    if (typeof value !== "object" && typeof value !== "function") throw new TypeError("Object expected.");
    var dispose;
    if (async) {
      if (!Symbol.asyncDispose) throw new TypeError("Symbol.asyncDispose is not defined.");
      dispose = value[Symbol.asyncDispose];
    }
    if (dispose === void 0) {
      if (!Symbol.dispose) throw new TypeError("Symbol.dispose is not defined.");
      dispose = value[Symbol.dispose];
    }
    if (typeof dispose !== "function") throw new TypeError("Object not disposable.");
    env.stack.push({ value: value, dispose: dispose, async: async });
  }
  else if (async) {
    env.stack.push({ async: true });
  }
  return value;
};

The first line looks a bit weird:



var __addDisposableResource = (this && this.__addDisposableResource) || function (env, value, async) { 
  // ...
}

This just prevents an existing __addDisposableResource function from being overwritten (like if you've concatenated multiple files containing this transpiler output). So what we're really looking at is:



function __addDisposableResource (env, value, async) {
  if (value !== null && value !== void 0) {
    if (typeof value !== "object" && typeof value !== "function") throw new TypeError("Object expected.");
    var dispose;
    if (async) {
      if (!Symbol.asyncDispose) throw new TypeError("Symbol.asyncDispose is not defined.");
      dispose = value[Symbol.asyncDispose];
    }
    if (dispose === void 0) {
      if (!Symbol.dispose) throw new TypeError("Symbol.dispose is not defined.");
      dispose = value[Symbol.dispose];
    }
    if (typeof dispose !== "function") throw new TypeError("Object not disposable.");
    env.stack.push({ value: value, dispose: dispose, async: async });
  }
  else if (async) {
    env.stack.push({ async: true });
  }
  return value;
}

The outer control flow handles a few different cases:



if (value !== null && value !== void 0) {
  // ...
} else if (async) {
  env.stack.push({ async: true });
}
return value;

So if value is null or undefined (using foo = null), just return value. And if value is null or undefined and async is true (await using foo = null), first push { async: true } to the stack and then return value.

The inner control flow handles validation of value and tries to find the dispose clean-up function.



if (typeof value !== "object" && typeof value !== 
"function") throw new TypeError("Object expected.");
var dispose;
if (async) {
  if (!Symbol.asyncDispose) throw new TypeError("Symbol.asyncDispose is not defined.");
  dispose = value[Symbol.asyncDispose];
}
if (dispose === void 0) {
  if (!Symbol.dispose) throw new TypeError("Symbol.dispose is not defined.");
  dispose = value[Symbol.dispose];
}
if (typeof dispose !== "function") throw new TypeError("Object not disposable.");
env.stack.push({ value: value, dispose: dispose, async: async });

The validation rules are as follows:

value must be an object (or a function which is also an object).
If we're trying to attach an asynchronous clean-up function, the runtime must support the new Symbol.asyncDispose named symbol.
If we're trying to attach a synchronous clean-up function, the runtime must support the new Symbol.dispose named symbol.
If there is a value assigned to value[Symbol.asyncDispose] or value[Symbol.dispose] it must be a function.

Assuming all the validation passes, then the value, the dispose function, and the async flag are all pushed onto the stack as a kind of disposal object. Its shape is something like:



export type Disposal = {
  value: unknown;
  dispose: () => void | () => Promise<void>;
  async: boolean;
}

One interesting thing I noticed is that the disposal object always uses { value: value, dispose: dispose, async: async }, even if the transpile target is ES2022. I'd have thought it could be { value, dispose, async } when targeting modern JS. I wonder how many bytes of JS the would save globally?

So this means that we can figure out the type of a disposable resource. It's something like this:



export type Disposable<T extends object> = T & ({
  [Symbol.asyncDispose](): Promise<void>; 
} | {
  [Symbol.dispose](): void; 
});

So if we try something like this, it should work!



const Resource: Disposable<{}> = {
  [Symbol.dispose]() { return }
}

async function useResource () {
  using foo = Resource;
}

But if I try this in an IDE we get an error:

As usual, TypeScript is right - our object might not have [Symbol.dispose]! If we change it to await using foo = Resource; it works though - a Disposable will definitely have either [Symbol.dispose] or [Symbol.asyncDispose]. So the typings have to be slightly different depending on whether we use using or await using. They should be more like:



export type Disposable = {
  [Symbol.dispose](): void;
};
export type AsyncDisposable = {
  [Symbol.asyncDispose](): Promise<void>; 
};

When using using, only a Disposable is valid. When using await using either a Disposable | AsyncDispoable can be assigned.

This is confirmed by the global types defined in the latest TypeScript node types:



interface SymbolConstructor {
  /**
   * A method that is used to release resources held by an object. Called by the semantics of the `using` statement.
   */
  readonly dispose: unique symbol;

  /**
   * A method that is used to asynchronously release resources held by an object. Called by the semantics of the `await using` statement.
   */
  readonly asyncDispose: unique symbol;
}

interface Disposable {
  [Symbol.dispose](): void;
}

interface AsyncDisposable {
  [Symbol.asyncDispose](): PromiseLike<void>;
}

Note the use of PromiseLike instead of just Promise, which means the dispose function can return an 3rd-party Promise implementation. Also interface over type lol fight me.

Cool, so we can understand how defining a disposable resource works, now onto actually disposing of thing!

`__disposeResources` 🚮:

We know that the __disposeResources() function is called inside the finally block that wraps our using code. The finally block is called at the end of the function execution no matter what, so it can handle errors, and even manipulate the return value of the function.

Here's the full code for __disposeResources again:



var __disposeResources = (this && this.__disposeResources) || (function (SuppressedError) {
  return function (env) {
    function fail(e) {
      env.error = env.hasError ? new SuppressedError(e, env.error, "An error was suppressed during disposal.") : e;
      env.hasError = true;
    }
    function next() {
      while (env.stack.length) {
        var rec = env.stack.pop();
        try {
          var result = rec.dispose && rec.dispose.call(rec.value);
          if (rec.async) return Promise.resolve(result).then(next, function(e) { fail(e); return next(); });
        }
        catch (e) {
          fail(e);
        }
      }
      if (env.hasError) throw env.error;
    }
    return next();
  };
})(typeof SuppressedError === "function" ? SuppressedError : function (error, suppressed, message) {
  var e = new Error(message);
  return e.name = "SuppressedError", e.error = error, e.suppressed = suppressed, e;
});

We can again see the weird overwrite protection:



var __disposeResources = (this && this.__disposeResources) || (function (SuppressedError) { 
  // ... 
});

This time there's another layer of function calls though, and something called SuppressedError:



var __disposeResources = (function (SuppressedError) {
  return function (env) {
    // ...
  };
})(typeof SuppressedError === "function" ? SuppressedError : function (error, suppressed, message) {
  var e = new Error(message);
  return e.name = "SuppressedError", e.error = error, e.suppressed = suppressed, e;
});

So there's an immediately-invoked function expression (IIFE), which expects something called SuppressedError. Again there is some overwrite protection (typeof SuppressedError === "function" ? SuppressedError : function () { /* ... */ }), and then the SuppressedError definition:



function (error, suppressed, message) {
  var e = new Error(message);
  return e.name = "SuppressedError", e.error = error, e.suppressed = suppressed, e;
}

It's not immediately clear what this special error type does, but let's keep moving. Our "real" __disposeResources function body is this bit:



function __disposeResources (env) {
  function fail(e) {
    env.error = env.hasError ? new SuppressedError(e, env.error, "An error was suppressed during disposal.") : e;
    env.hasError = true;
  }
  function next() {
    while (env.stack.length) {
      var rec = env.stack.pop();
      try {
        var result = rec.dispose && rec.dispose.call(rec.value);
        if (rec.async) return Promise.resolve(result).then(next, function(e) { fail(e); return next(); });
      } catch (e) {
        fail(e);
      }
    }
    if (env.hasError) throw env.error;
  }
  return next();
}

Looks like we have some recursion! Fun 🥳! The __disposeResources function has access to the env scope, and seems to loop through the stack, taking each Disposal item from the list, processing it, and then calling next(). So what changes the length of stack?

We remember that inside __addDisposableResource we pushed a new Disposal object to the stack:



env.stack.push({ value: value, dispose: dispose, async: async })`
```

So if we have:

```typescript
// Before:
const tmpPath = path.join(tmpdir(), `${Math.random()}`);
using file = fs.openSync(tmpPath, 'w+');
const tmpPath2 = path.join(tmpdir(), `${Math.random()}`);
using file2 = fs.openSync(tmpPath2, 'w+');
```

Then we get:

```javascript
// After:
const tmpPath = path.join(tmpdir(), `${Math.random()}`);
const file = __addDisposableResource(env_1, fs.openSync(tmpPath, 'w+'), false);
const tmpPath2 = path.join(tmpdir(), `${Math.random()}`);
const file2 = __addDisposableResource(env_1, fs.openSync(tmpPath2, 'w+'), false);
```

Since `env_1` is shared between both `__addDisposableResource` calls, the stack will also be shared, both `Disposal` objects will be used. Note that since `pop` is used, the resources will be cleaned-up in the reverse order that they were created!

The `next()` function contains what happens for each resouce. Let's look at the synchronous flow first:

```javascript
function next() {
  while (env.stack.length) {
    var rec = env.stack.pop();
    try {
      var result = rec.dispose && rec.dispose.call(rec.value);
    } catch (e) {
      fail(e);
    }
  }
  if (env.hasError) throw env.error;
}
```

Get the next `Disposal` object off the stack, try to call the `dispose` function (if it exists), with the `value` set to `this`. If anything goes wrong with the disposal, call `fail()`.

Even if the clean-up was fine, there could still be an error in our original function (`tmpData`), and `env.hasError` could be true! In that case, that error is throw and has to be handled back by whoever called `tmpData()`. But what happens if `disposal` throws an error? Let's look at `fail()`:

```javascript
function fail(e) {
  env.error = env.hasError ? new SuppressedError(e, env.error, "An error was suppressed during disposal.") : e;
  env.hasError = true;
}
```

Now `SuppressedError` makes a bit more sense! If `disposal` fails, but another error already happened, we construct a new `Error` that wraps both the `disposal` error and the original error!. If `env.hasError` is false, then only `disposal` threw, and just the `disposal` error is used. If multiple `disposal` functions fail, then you would get multiple layers of nested errors!

The `async` flow makes things look complicated:

```javascript
if (rec.async) return Promise.resolve(result).then(next, function(e) { fail(e); return next(); });
```

All this is doing is making sure that all the `disposal` functions are called sequentially. Each clean-up function will be completed before the next one starts, regardless of whether it is `async` or not. I guess that helps makes things a bit more predictable.

## Checking the spec 🔎

So we've looked through all the code, and we have a better understanding of the implementation - it's a good time to check out the actual spec proposal and see if we got it right!

> [The Explicit Resource Management proposal](https://github.com/tc39/proposal-explicit-resource-management)

From looking through, it's pretty much what would be expected from the code! There's a few interesting snippets, like this example footgun which would be fixed with the proposal:

```typescript
// Avoiding common footguns when managing multiple resources:
const a = ...;
const b = ...;
try {
  ...
}
finally {
  a.close(); // Oops, issue if `b.close()` depends on `a`.
  b.close(); // Oops, `b` never reached if `a.close()` throws.
}
```

And there's some syntactic structures I didn't even think of:

```typescript
for (await using x of y) ...

for await (await using x of y) ...
```

The proposal even has [example code](https://github.com/tc39/proposal-explicit-resource-management#using-declarations-with-explicit-local-bindings) that approximates the runtime semantics which pretty much lines up with the TS implementation.

All in all, seems about right ✅ ✅ ✅

## Wrap up 🎁

Cool that was fun! I think this spec seems like a good idea, there's a lot of places where this could be used (and lots of existing Web/Server JS APIs that would benefit from it!), so I hope it makes it into the language.

If you liked reading this, reach out and let me know, or [hit me up online](https://twitter.com/phenomnominal) with question/comments/corrections. 🥰🥰🥰

Betterer v5.0.0

Craig ☠️💀👻 — Sat, 13 Nov 2021 00:06:54 +0000

Oof, this one feels like it's been a while coming, but after a whole bunch of work, and a whole bunch of breaking changes I've just released v5.0.0 of Betterer! 🎉

What is Betterer?

Betterer is a test runner that helps make incremental improvements to your code!

The first time Betterer it runs a test, it will take a snapshot of the current state. From that point on, whenever it runs it will compare against that snapshot. It will either throw an error (if the test got worse ❌), or update the snapshot (if the test got better ✅). That's pretty much it!

You can check out the (newly updated!) documentation at https://phenomnomnominal.github.io/betterer/

What happened to v2, v3, v4...?

"But Craig", I hear you say, "The last time you posted about Betterer, it was at v1.0.0!? What's been going on?!". 🔥🔥🔥

That's very astute of you dear reader, and let's just put it this way - I sure do love breaking APIs! One of the interesting things about Betterer is that it is a tool designed for problems that emerge in large and old codebases. That means it has to be able to handle large and old codebases from the get go! So I've had a lot of fun as I've tried to figure out the best workflows and APIs for using Betterer.

Between v1.0.0 and now, I've released a bunch of features, consolidated and simplified APIs, and just generally made Betterer more usable and flexible. I'm pretty happy with where it is at now, so I figured it was about time for an update. I've even been talking about it at a few conferences now that they're coming back! What a world 🌍!

What's in v5.0.0?

Parallel tests:

Performance is hard. Prior to v5, the default Betterer reporter would struggle pretty badly, especially when lots of tests were running and producing lots of issues. That was because the main thread was responsible for updating the reporter output and running all the tests.

To fix this, Betterer will now execute all your tests using Node.js Worker Threads! That frees up the main thread to focus on rendering and also means that multiple tests can run at the same time. Getting this to work required breaking some APIs, so your test definition file needs to change:

Before:

// .betterer.ts
import { BettererTest } from '@betterer/betterer';

export default {
  'my test': new BettererTest({
    // ... test config
  }),
  'my other test': new BettererTest({
    // ... test config
  })
};

After:

// .betterer.ts
import { BettererTest } from '@betterer/betterer';

export default {
  'my test': () =>
    new BettererTest({
      // ... test config
    }),
  'my other test': () =>
    new BettererTest({
      // ... test config
    })
};

But never fear, you can use the betterer upgrade command to do this migration for you! Just running betterer upgrade will show you what the migration will look like, and betterer upgrade --save will actually update your files. Easy ✨. The betterer upgrade command will be used in the future when I (most probably) break more stuff.

Check out the beast of a PR here (and yes, it took me three branches to get it right 😅)

Betterer ❤️ Angular

I've published a new Betterer test for incrementally adding Angular compiler configuration to a project! I'm pretty excited by this, as there are a lot of big Angular codebases out there that don't utilise the full power of the Angular compiler. In particular, I think Betterer could be a good way to introduce the strictTemplates option. You can now do that with the following:

//.betterer.ts
import { angular } from '@betterer/angular';

export default {
  'strict templates': () => angular('./tsconfig.json', {
    strictTemplates: true
  }).include('./src/**/*.ts', './src/**/*.html')
};

Expect to see a full post detailing this in the near future!

Simpler `BettererFileTest`:

The old BettererFileTest API was a bit clunky and confusing due to the BettererFileResolver thing. I've hidden that away in the internals, so now the public API is less clunky and confusing:

Before:

import { BettererFileResolver, BettererFileTest } from '@betterer/betterer';

function myFileTest() {
  const resolver = new BettererFileResolver();
  return new BettererFileTest(
    resolver,
    async (filePaths, fileTestResult) => {
      // test implementation...
    }
  );
}

After:

import { BettererFileTest } from '@betterer/betterer';

function myFileTest() {
  return new BettererFileTest(
    async (filePaths, fileTestResult, resolver) => {
      // test implementation...
    }
  );
}

Smaller public API, less magic, and you only have to use it if you know why, choice! 👍

Improved workflow:

I'm still working on figuring out the ideal Betterer workflow. For now, I recommend running Betterer in Pre-commit mode as a pre-commit hook (perhaps using husky and lint-staged) and in CI mode on your build server.

But one thing about chonky codebases is that they often have lots of contributors! Lots of contributors making changes (and making things better) means that 👻 merge conflicts 👻 in the results file are quite common!

To try to help with resolving merge conflicts, I've introduced the betterer merge command. You can still fix merge conflicts manually, but betterer merge will do it for you! If you're as lazy as me, you can even enable automerge and you'll never have to think about merging the results file ever again (I hope, this could still be buggy 🐛😅.

To enable automerge run:

betterer init --automerge

Improved caching:

Betterer got some cool (ish) caching implemented in v4, but turns out caching is a hard problem (😅), so it's taken a little bit to get right.

It works by passing the --cache flag when running Betterer:

betterer --cache

That will create a file something like this:

{
  "version": 2,
  "testCache": {
    "no hack comments": {
      "packages/angular/src/angular.ts": "b66de728222febdecb3cf11d3aa510b3a8a6ae0e37c0539e37787964573a56ad1b7eb6ee378a9087",
      "packages/angular/src/index.ts": "b66de728222febdecb3cf11d3aa510b3a8a6ae0eb9494122f82a750085fc20d2c3b0f14b34897431",
      "packages/betterer/src/betterer.ts": "b66de728222febdecb3cf11d3aa510b3a8a6ae0e94efcd2f99a4cf14222c400693335ac1b94696bb",
      // ...
    }
  }
}

Betterer will use this cache to only re-test files that have actually changes, so it can be much faster (useful for running on pre-commit!) I suspect there are still issues here, so please try it out and create issues. 🙌

Bug fixes and improvements:

betterer results command to get a quick summary:

Negative filters. I already suspect I'll regret this, but you can now use "!" at the start of a filter to negate it. Now --filter myTest will just run "myTest", and --filter !myTest will run every other test.
Rewrote most of the public API docs. These are now generated from the code, so should hopefully be easier to keep up to date. 🤞
Removed a bunch of stuff from the public API. This means more consistency, and I'll be less likely to accidentally break stuff in the future. 😇

Thanks ❤️

Huge thanks to everyone who has helped me with this stuff, if you've read my rambling, cryptic tweets, opened issues on Github, chatted to me about Betterer at conferences, it's all meant a lot! Maybe I'll print some stickers or something? ☀️

Love 🥰 this? Hate 🤬 this? Go off in the comments, DM me on Twitter, or be the third person to join the Betterer Discord. Catch you on the line 💻!

Incrementally adding Stylelint rules with Betterer

Craig ☠️💀👻 — Mon, 01 Mar 2021 00:41:14 +0000

I just released v4.0.0 of Betterer 🎉 (now with sweet new docs!) and it has a bunch of simplified APIs for writing tests. And just before I shipped it, I got an issue asking how to write a Stylelint test, so let's do it here and explain it line by line:

TL;DR;

Here's the full test:

// stylelint.ts
import { BettererFileResolver, BettererFileTest } from '@betterer/betterer';
import { promises as fs } from 'fs';
import { Configuration, lint } from 'stylelint';

export function stylelint(configOverrides: Partial<Configuration> = {}) {
  const resolver = new BettererFileResolver();
  return new BettererFileTest(resolver, async (filePaths, fileTestResult) => {
    const result = await lint({
      files: [...filePaths],
      configOverrides
    });

    await Promise.all(
      result.results.map(async (result) => {
        const contents = await fs.readFile(result.source, 'utf8');
        const file = fileTestResult.addFile(result.source, contents);
        result.warnings.forEach((warning) => {
          const { line, column, text } = warning;
          file.addIssue(line - 1, column - 1, line - 1, column - 1, text, text);
        });
      })
    );
  });
}

And then using the test:

// .betterer.ts
import { stylelint } from './stylelint'; 

export default {
  'no stylelint issues': stylelint({
    rules: {
      'unit-no-unknown': true
    }
  }).include('./**/*.css')
};

NTL;PR (not that long, please read 😂)

Stylelint

So how does it all work? Let's start with the actual Stylelint part.

Stylelint is pretty easy to set-up. You need a .stylelintrc.json file with configuration:

{
  "extends": "stylelint-config-standard"
}

And then run it on your CSS files:

stylelint "**/*.css"

Running that does the following:

1) searches for the stylelintrc.json configuration file
2) reads the configuration
3) finds the valid files
4) runs the rules
5) returns the results

Stylelint also has a JS API which we're going to use:

import { lint } from 'stylelint';

const result = await lint({
  // ...
});

We could just run the above and it will test the current state of the files with the current configuration in stylelintrc.json. And that's great ✨!

Augmenting the configuration:

For the Betterer test we want to augment the stylelintrc.json configuration with some extra rules... and Stylelint has a really easy way to do that:

import { Configuration, lint } from 'stylelint';

function stylelint(configOverrides: Partial<Configuration> = {}) {
    const result = await lint({
      configOverrides
    });
}

Passing the list of files:

Stylelint also allows us to pass a specific set of files to test:

import { Configuration, lint } from 'stylelint';

function stylelint(configOverrides: Partial<Configuration> = {}, files: Array<string>) {
    const result = await lint({
      files,
      configOverrides
    });
}

So we could call the stylelint function like:

stylelint({
  rules: {
    'unit-no-unknown': true
  }
}, './**/*.css')

And that will run the Stylelint from the stylelinerc.json file, plus the unit-no-unknown rule, on all .css files! Thats most of the tricky stuff sorted ⭐️!

Hooking into Betterer:

This test needs to take advantage of all the snapshotting and diffing magic of Betterer, so we need to wrap it in a test. We want to be able to target individual files, so it specifically needs to be a BettererFileTest.

We first create a BettererFileResolver, which is a little bit of magic that helps work out which file paths are relevant for the test. That is passed as the first argument to BettererFileTest. The second argument is the actual test, which will be an async function that runs the linter.

import { BettererFileResolver, BettererFileTest } from '@betterer/betterer';
import { Configuration, lint } from 'stylelint';

function stylelint(configOverrides: Partial<Configuration> = {}) {
  const resolver = new BettererFileResolver();
  return new BettererFileTest(resolver, async (filePaths) => {
    // ...
  });
}

Each time it runs Betterer will call that function with the relevant set of files, which we will pass along to Stylelint:

import { BettererFileResolver, BettererFileTest } from '@betterer/betterer';
import { Configuration, lint } from 'stylelint';

function stylelint(configOverrides: Partial<Configuration> = {}) {
  const resolver = new BettererFileResolver();
  return new BettererFileTest(resolver, async (filePaths) => {
    const result = await lint({
      files: [...filePaths],
      configOverrides
    });
  });
}

Adding files:

Next thing is telling Betterer about all the files with issues reported by Stylelint. To do this we can use the BettererFileTestResult object, which is the second parameter of the test function:

new BettererFileTest(resolver, async (filePaths, fileTestResult) => {
  // ...
});

The result object from Stylelint contains a list of results. For each item in that list, we need to read the file with Node's fs module, and then call addFile() with the file path (result.source), and the contents of the file. That returns a BettererFile object:

import { promises as fs } from 'fs';

await Promise.all(
  result.results.map(async (result) => {
    const contents = await fs.readFile(result.source, 'utf8');
    const file = fileTestResult.addFile(result.source, contents);
  })
);

Adding issues:

The last thing to do is convert from Stylelint warnings to Betterer issues. To do that we use the addIssue() function! In this case we will use the following overload:

addIssue(startLine: number, startCol: number, endLine: number, endCol: number, message: string, hash?: string):

Stylelint only gives us the line and column for the start of the issue, so we use that as both the start position and the end position. Betterer expects them to be zero-indexed so we subtract 1 from both. This also means that the VS Code extension will add a diagnostic to the whole token with the issue, which is pretty handy! We also pass the text of the issue twice, once as the message, and a second time as the hash. The hash is used by Betterer to track issues as they move around within a file. Stylelint adds specific details to the message so that makes it a good enough hash for our purposes. All up, converting an issue looks like this:

result.warnings.forEach((warning) => {
  const { line, column, text } = warning;
  file.addIssue(line - 1, column - 1, line - 1, column - 1, text, text);
});

The whole test:

Putting that all together and you get this:

// stylelint.ts
import { BettererFileResolver, BettererFileTest } from '@betterer/betterer';
import { promises as fs } from 'fs';
import { Configuration, lint } from 'stylelint';

export function stylelint(configOverrides: Partial<Configuration> = {}) {
  const resolver = new BettererFileResolver();
  return new BettererFileTest(resolver, async (filePaths, fileTestResult) => {
    const result = await lint({
      files: [...filePaths],
      configOverrides
    });

    await Promise.all(
      result.results.map(async (result) => {
        const contents = await fs.readFile(result.source, 'utf8');
        const file = fileTestResult.addFile(result.source, contents);
        result.warnings.forEach((warning) => {
          const { line, column, text } = warning;
          file.addIssue(line - 1, column - 1, line - 1, column - 1, text, text);
        });
      })
    );
  });
}

And then we can use the test like this:

// .betterer.ts
import { stylelint } from './stylelint'; 

export default {
  'no stylelint issues': stylelint({
    rules: {
      'unit-no-unknown': true
    }
  }).include('./**/*.css')
};

And that's about it! The Stylelint API is the real MVP here, nice job to their team! 🔥🔥🔥

Hopefully that makes sense! I'm still pretty excited by Betterer, so hit me up on Twitter if you have thoughts/feelings/ideas, or leave a comment here! ☀️

Over-engineered TypeScript Types - but I learned some stuff!

Craig ☠️💀👻 — Tue, 26 Jan 2021 02:32:35 +0000

Hey team 🌟! This is another long rambling one, to try to document how my brain thought about this problem! My writing soundtrack was "Trust Me" by Sincere Engineer on repeat 🎶:

Hope you like it!

Background

At work we get a few days each month to work on whatever we like, to encourage learning/exploration/curiosity. Last month I decided I would try to solve an interesting JavaScript problem, but ended up uncovering a pretty interesting TypeScript problem 🤓...

My goal was to make the following work:

I wanted to create a function that works like require, but transforms the imported module to run in a separate thread. The implementation of the JavaScript part isn't the point of this post, but here's a simplified version of how I got it to work:

Comlink does most of the heavy lifting (thanks 🥰!) and the real inspiration for this post comes from a little note at the end of the Comlink docs:

While this type (Comlink.Remote<T>) has been battle-tested over some time now, it is implemented on a best-effort basis. There are some nuances that are incredibly hard if not impossible to encode correctly in TypeScript’s type system.

So what are those nuances and can TypeScript handle them? How should the types for workerRequire work?

Incredibly hard nuances

My original use case involves rendering a simple terminal-based UI using Ink. I want to run a task, and update the UI showing the task's progress. Unfortunately, running a CPU intensive task blocks UI from updating until the task is done, which makes the UI feel slow and clunky 😑. It would be much nicer to run the task in a Worker thread and update the UI in the main thread. This is like what you might do in browser-based app!

I want to go from something like this:

// BEFORE:
const { somethingCPUIntensive } = require('./cpu-intensive');
// The main thread is blocked
somethingCPUIntensive();

to something like this:

// AFTER:
const { somethingCPUIntensive } = workerRequire('./cpu-intensive');
// The main thread is no longer blocked
await somethingCPUIntensive();

I have a './cpu-intensive.js' file that exports a function:

// ./cpu-intensive.js
export function somethingCPUIntensive () {
    return fibonacci(50); // plain non-memoized fibonacci function which is pretty slow...
}

The types of this function make sense in isolation. When it is required with workerRequire the function is transformed from synchronous to asynchronous. That means that the types must also be:

// BEFORE:
const { somethingCPUIntensive } = require('./cpu-intensive');
// typeof somethingCPUIntensive is () => number

// AFTER:
const { somethingCPUIntensive } = workerRequire('./cpu-intensive');
// typeof somethingCPUIntensive is () => number

TypeScript can describe this transform succinctly:

type Func = (...args: Array<unknown>) => unknown;
type AsyncFunc<SyncFunc> = SyncFunc extends Func
    ? (...args: Parameters<SyncFunc>) => Promise<ReturnType<SyncFunc>>
    : never;

function fibonacci (n: number): number {
  if (n <= 1) return 1;
  return fibonacci(n - 1) + fibonacci(n - 2);
}

type SyncAdd = typeof fibonacci; // (n: number) => number
type AsyncAdd = AsyncFunc<typeof fibonacci>; // (n: number) => Promise<number>;

This is already quite deep in TypeScript stuff, so if you need a refresher check out Conditional Types, Parameters and ReturnType

Everything in this case works fine, because there is a clear distinction between which code runs in each thread. The fibonacci function runs synchronously in the worker thread, but the main thread receives the result asynchronously.

Unfortunately, the code doesn't actually solve the problem yet. The worker handles the intensive work, but it doesn't update the main thread as the task progresses. There needs to be some kind of callback:

// BEFORE:
const { somethingCPUIntensive } = require('./cpu-intensive');
// The main thread is blocked
somethingCPUIntensive((data) => updateProgress(data));

// AFTER:
const { somethingCPUIntensive } = workerRequire('./cpu-intensive');
await somethingCPUIntensive((data) => updateProgress(data));

In JavaScript land, this Just Works™️. Comlink handles passing arguments from the worker thread to the main thread, and then passing the result back to the worker thread. Seems kind of ✨ magic ✨ right? And it is! But from a types point of view this introduces even more weirdness 🙃...

Imagine that this is the original synchronous implementation of the somethingCPUIntensive function:

// ./cpu-intensive.js
export function somethingCPUIntensive(updateProgress) {
    const progress = doSomeWork();
    updateProgress(progress);
    const moreProgress = doSomeMoreWork();
    updateProgress(moreProgress);
    // ...
}

With a standard require this works as written, everything is synchronous and blocking:

const { somethingCPUIntensive } = require('./cpu-intensive');
somethingCPUIntensive((data) => updateProgress(data));

But with workerRequire everything is different:

const { somethingCPUIntensive } = workerRequire('./cpu-intensive');
await somethingCPUIntensive((data) => updateProgress(data));

The difference can be illustrated by adding types to somethingCPUIntensive:

// ./cpu-intensive.ts
export type Data = {};

export function somethingCPUIntensive(
    updateProgress: (data: Data) => void
) {
    const progress = doSomeWork();
    updateProgress(progress);
    const moreProgress = doSomeMoreWork();
    updateProgress(moreProgress);
    // ...
}

These types are entirely valid when used with require. But they are subtly invalid when used with workerRequire. Comlink will transform each call to updateProgress to being async!

Let me try to make it even more clear with another example. What if somethingCPUIntensive did something more complex, and the callback didn't return void?

// ./cpu-intensive.ts
export type Data = { ... };

export function somethingCPUIntensive(
    updateProgress: (data: Data) => Data
): Data {
    const progress = doSomeWork();
    const data = updateProgress(progress);
    const moreProgress = doSomeMoreWork(data);
    const moreData = updateProgress(moreProgress);
    // ...
}

Now the code is straight wrong! The code says that data and moreData to be of type Data, but when called via Comlink they would actually be Promise<Data> 🤯! The correct typing for this code would be this:

// ./cpu-intensive.ts
export type Data = { ... };

export function somethingCPUIntensive(
    updateProgress: (data: Data) => Promise<Data>
): Promise<Data> {
    const progress = doSomeWork();
    const data = await updateProgress(progress);
    const moreProgress = doSomeMoreWork(data);
    const moreData = await updateProgress(moreProgress);
    // ...
}

Where the callback function is originally declared, it appears to be synchronous. But where it is called it will be asynchronous! This is true if the function is passed from the main thread to the worker thread, or vice versa.

This is the aforementioned nuance and it is indeed quite mind-bending. All synchronous functions that are passed between threads will be transformed to asynchronous! This is true if the function is passed as a callback, or if it is a property on an object that is returned, or if it is an item in a Set Map or Array! The types on the receiving side must expect the function to be asynchronous. It's a pretty tricky mental model to work with.

The types for workerRequire function must encapsulate this confusion and ensure that required modules follow this model!

Typing `workerRequire`

The workerRequire API doesn't give much to work with:

workerRequire('./cpu-intensive');

The first problem is how to go from a string file path (or module name for a node_modules dependency) to the types of the resultant module:

Getting the `Module` type:

TypeScript has special handling for import statements, and also for require statements. But there is no way to tell the compiler that another function should inherit this behaviour.

The only way to go from a string path or module name to the type of that module is with the typeof operator combined with the import() syntax:

type Module = typeof import('./path/to/module');

A function that wraps require with correct types might look like this:

function myRequire<Result>(moduleId: string): Result {
    return require(moduleId);
}
const result = myRequire<typeof import('./path/to/module')>('./path/to/module');

Not exactly elegant, but it works 😅! The next step is to validate the Module type.

Validating the `Module` type:

I want TypeScript to validate that the module that lives in './cpu-intensive' handles asynchronicity correctly. It needs to search through every exported function in the module, and find all the parameters and return values that contain functions, and make sure that they return Promises... 🤔

To simplify things, I assume that the module only exports functions. So a valid module is an object that only has valid functions. And a valid function is a function where none of the arguments contain any synchronous functions, and the return value doesn't contain any synchronous functions. The actual exported function from the module doesn't have to be asynchronous!

That type might begin like this (don't worry if this is scary 😱 or if you don't know all the syntax, I'm going to break it down):

export type WorkerModule<Module> = {
  [Key in keyof Module]: Module[Key] extends Func
    ? WorkerModuleFunction<Module[Key]>
    : never;
};

export type WorkerModuleFunction = // ...

I like to think about about complex types like functions, so what would that look like in JavaScript?

export function validateWorkerModule (module) {
    Object.keys(module).forEach((key) => {
        if (!isValidWorkerFunction(module[key]) {
            throw new Error();
        }
    });
}

function isValidWorkerFunction (func) {
    func.parameters.forEach(parameter => {
        if(!isValidWorkerValue(parameter) {
            throw new Error();
        }
    })
    if (!isValidWorkerValue(func.returnType)) {
        throw new Error();
    }
}

So that kind of makes sense for the first level, but how does isValidWorkerValue work? If I keep going in JavaScript:

function isValidWorkerValue (value) {
    if (isFunction(value)) {
        return isValidWorkerFunction(value);
    }
    // ...
}

This is the first bit of recursion. A function can have a function as an argument, so TypeScript will have to go back up to confirm that the argument is a valid function. Which in turn could have another argument that is a function. But arguments can be other things as well:

function isValidWorkerValue (value) {
    if (isFunction(value)) {
        return isValidWorkerFunction(value);
    }
    if (isObject(value)) {
        return isValidWorkerObject(value);
    }
    if (isArray(value)) {
        return isValidWorkerArray(value);
    }
    if (isSet(value)) {
        return isValidWorkerSet(value);
    }
    if (isMap(value)) {
        return isValidWorkerMap(value);
    }
    if (isPromise(value)) {
        return isValidWorkerPromise(value);
    }
}

The more specific validation functions introduce another level of recursion. There can be deep Objects, or Arrays of Arrays, or Maps of Sets of Promises that return Functions 🙃:

function isValidWorkerObject (obj) {
    Object.keys(obj).forEach(key => {
        const value = obj[key];
        if (!isValidWorkerValue(value) {
            throw new Error();
        }
    });
}

function isValidWorkerArray (arr) {
    arr.forEach(value => {
        if (!isValidWorkerValue(value) {
            throw new Error();
        }
    });
}

// ...

If you've got this far, have a think about how Set, Map, and Promise might work? Let me know in the comments!

There is a nested logic here with multiple layers of recursion and weirdness going on 🤯... How does it translate into types?

There is to be TypeScript type syntax that is equal to the above pseudo-JS:

Typed `if` statements

I've already mentioned Conditional Types once in this post, and I'm going to keep using them here! A conditional type is a type with the form:

type A = B extends C ? D : E;

This is equivalent (and syntactically similar to a JavaScript ternary statement:

const a = b > c ? d : e;

Which is in turn equivalent to the more verbose JavaScript

let a;
if (b > c) {
    a = d;
} else {
    a = e;
}

That means I can write the isValidWorkerValue function as the following type:

export type WorkerValue<Value> = Value extends Obj
  ? Value extends WorkerObject<Value>
    ? Value
    : WorkerObject<Value>
  : Value extends Array<unknown>
  ? Value extends WorkerArray<Value>
    ? Value
    : WorkerArray<Value>
  : Value extends Func
  ? Value extends WorkerFunction<Value>
    ? Value
    : WorkerFunction<Value>
  : Value extends Set<unknown>
  ? Value extends WorkerSet<Value>
    ? Value
    : WorkerSet<Value>
  : Value extends Map<unknown, unknown>
  ? Value extends WorkerMap<Value>
    ? Value
    : WorkerMap<Value>
  : Value extends Promise<unknown>
  ? Value extends WorkerPromise<Value>
    ? Value
    : WorkerPromise<Value>
  : Value;

Which is totally ridiculous, but valid 🤓! But how do the more specific validation types work? There are two parts to this. First the validation needs to check that the given type argument is approximately the right type. I'll use WorkerPromise as an example:

export type WorkerPromise<P extends Promise<unknown>> = 
  P extends Promise<unknown> // Check if it's a Promise
    ? // This is a Promise, keep validating
    : // Not a Promise, bail out

If it isn't a Promise , it should resolve to never so that the WorkerValue type will skip to the next branch of the if/else blocks.

If it is a Promise then it moves onto the second part. The type needs to extract the inner type, which is the type of the resolved value. This can be done with the infer keyword, which is a special power of Conditional Types. Then the inner type can be validated as a WorkerValue - even more recursion!

export type WorkerPromise<P extends Promise<unknown>> = 
  P extends Promise<infer Value>
    ? Value extends WorkerValue<Value>
      ? P
      : SomethingMagic
    : never;

I'll come back to the SomethingMagic in a bit! 😉

Typed loops

The types also need to be able to iterate over each key/value pair in an Object type, or each item in an Array type. TypeScript provides a way to do that as well by using keyof:

export type WorkerObject<O> = {
  [Key in keyof O]: O[Key] extends WorkerValue<O[Key]>
    ? O[Key]
    : SomethingMagic
};

export type WorkerArray<A extends Array<unknown>> = {
  [Index in keyof A]: A[Index] extends WorkerValue<A[Index]>
    ? A[Index]
    : SomethingMagic
};

The keyof syntax is like an iterator, except it iterates over the indices of a type. With an Object-like type, it will give the key, which can then be used to look up the type of the key (with the O[Key] syntax). For an Array type, it will give the numeric indices, which can then be used with the same syntax (A[Index]).

Again, there is more recursion introduced here!

WorkerSet and WorkerMap is implemented similarly to WorkerPromise. Have a crack yourself maybe?

Typed ✨`Errors`✨

The one thing that is missing from the original JavaScript pseudocode is Errors. If you're guessing this is what the SomethingMagic is about, then you're right. But I'll explain why first.

The standard way to show that a type is invalid in TypeScript is the never type.

One example where never could be used would be a function that always throws:

function explode (): never {
    throw new Error('💥');
}

This function never returns anything, so the return type is never. This is also often used with conditional types:

type isPromise<P> = P extends Promise<unknown> ? P : never;
type Yes = isPromise<Promise<boolean>>; // Promise<Boolean>
type No = isPromise<boolean>; // never;

So it kind of makes sense to use never as a "this isn't a valid module" type too, as I hinted in the first type I introduced:

export type WorkerModule<Module> = {
  [Key in keyof Module]: Module[Key] extends Func
    ? WorkerModuleFunction<Module[Key]>
    : never;
};

Hopefully you find this type a little less scary this time?

The WorkerModule type is the type that validates the Module, and is used like this:

type Valid = WorkerModule<typeof import('./path/to/module')>;

If typeof import('./path/to/module') follows the rules, then that is great, the WorkerModule type acts as an identity and resolves to the same input. If the module doesn't follow the rules, then it resolves to never...

That is going to make it basically impossible to figure out why the module isn't valid.

Fortunately, a type doesn't have to resolve to never. It can resolve to an entirely different type, for example a string:

export type WorkerModule<Module> = {
  [Key in keyof Module]: Module[Key] extends Func
    ? WorkerModuleFunction<Module[Key]>
    : 'Module should only export functions'
};

Now using the type, it will explain what the specific issue is:

type MyModule = { foo: 'bar' }
type Valid = WorkerModule<MyModule> // 'Module should only export functions'

But it can be even better:

export type WorkerModuleError<Type, Message, Details = null> = {
  message: Message;
  type: Type;
  details: Details;
};

export type WorkerModule<Module> = {
  [Key in keyof Module]: Module[Key] extends Func
    ? WorkerModuleFunction<Module[Key]>
    : WorkerModuleError<
        Module,
        'Module should only export functions',
        Module[Key]
      >;
};

type MyModule = { foo: 'bar' }
type Valid = WorkerModule<MyModule>
// { 
//   message: 'Module should only export functions'
//   type: { foo: 'bar' }
//   details: 'bar'
// }

And that can continue even further into the validation, like with the WorkerPromise above:

export type WorkerPromise<P extends Promise<unknown>> = 
   P extends Promise<infer Value>
    ? Value extends WorkerValue<Value>
      ? P
      : WorkerModuleError<
          P,
          'Promise value should not contain synchronous functions',
          WorkerValue<Value>
        >
    : never;

Or a more complex example like WorkerFunction:

export type WorkerFunction<F extends Func> = 
  F extends (...args: infer Args) => infer Return
  ? Args extends WorkerArray<Args>
    ? Return extends Promise<infer Value>
      ? Value extends WorkerValue<Value>
        ? F
        : WorkerModuleError<
            F,
            'Function return value should not contain synchronous functions',
            WorkerValue<Value>
          >
      : WorkerModuleError<F, 'Function should return a Promise'>
    : WorkerModuleError<
        F,
        'Function arguments should not contain synchronous functions',
        WorkerArray<Args>
      >
  : never;

This makes it much easier to fix issues in the problematic Module!

Putting it all together:

So, it is possible to use TypeScript to check that a module will behave correctly with Comlink - or my workerRequire function! It can even have nice error messages - I will be using that trick more in the future for sure.

There's a few more little edge-cases that I haven't detailed here, but you can see the full implementation here and check out some of the Type tests here!

The same Type tests are running here in a CodeSandbox if you want to try them out:

Or even better, try it out! See if you can get something running in a separate thread. And then delete everything and use Comlink instead cause this is a Very Bad Idea™️.

Wrapping up:

That's definitely some weird stuff, but I had fun playing with it. And I solved the problem I needed to solve, and I learned a bunch 🎉.

Have I got bugs in my implementation? Yes! Have I encapsulated all the complexity the Comlink authors imagine? Almost definitely not! Have I learned some things? Heck yeah! Have you? Hit me up in the comments or on the bird site.

If you got this far, and you meet me at a conference/meetup in the post-COVID future, tell me, I owe you a drink 🥤!

Thank you!

Lazy debug logging for Node.js

Craig ☠️💀👻 — Wed, 21 Oct 2020 13:29:01 +0000

This may come as a surprise, but I can be very lazy. If I can get good results without much effort, then 🥳! So the idea of adding debug logging to a whole codebase sounds absolutely exhausting to me, and I want to find a better (lazier) way...

The problem:

I have a tool that I've been working on for a while, and debugging it can be kind of a pain - especially when it's running inside VS Code. It'd be nice to have an easy way to get information about what is going on when the tool runs, without having to manually write a bunch of debug logging code and release a new version. That means that the usual approaches are not going to work:

no straight up console.log() statements
no debug(function foo () {}) wrappers
no fancy @debug decorators

I want something I can set and forget, and probably turn on and off via an environment variable. Is this even possible?

Attempt 1 - override `Function.prototype.constructor`:

I know that in the past I've done something where I wrapped every function call in some logging, so I want to try to get that working again. I'm pretty sure I did it by overwriting Function.prototype.constructor, which is a pretty insane hack 😅

JavaScript is a very flexible language, so I can just do stuff like this:



Function.prototype.constructor = (() => {
    const original = Function.prototype.constructor;
    return function (...args) {
        // do stuff...
        return original.call(this, ...args);
    };
})();

I can store a reference to the original constructor function, and replace it with a new function that does some debug logging, and then calls the original function. Seems pretty smart! Job done!

Unfortunately, this doesn't actually work... 😐



function foo () { return 'foo'; } <- Function.prototype.constructor doesn't get called here
foo();

So perhaps I can overwrite Function.prototype.call instead?



Function.prototype.call = (() => {
    const original = Function.prototype.call;
    return function (...args) {
        // do stuff...
        return original.apply(this, args);
    };
})();

function foo () { return 'foo'; }
foo(); // <- Function.prototype.call doesn't get called here either

I know I've done this in the past, so I must just be missing something. Off to Stack Overflow I go!

I found a few relevant questions/answers, like this one, which does exactly what I just tried? I also found an answer for this exact problem that I wrote 7 years ago - thanks past-Craig! 🦄

Apr 9 '13

This is one alternative, not entirely sure how reliable it is though, it feels a bit wrong:

(function () {
  var oldCall = Function.prototype.call;
  var newCall = function(self) {
    Function.prototype.call = oldCall;
    console.log('Function called:', this.name);
    var args = Array.prototype.slice.call(arguments, 1);
    Function.prototype.call = newCall;
    this.apply(self, args);
  }
  Function.prototype.call = newCall;
})();

…

Open Full Answer

It even handles a few edge cases, like handling the correct version of this that is passed in, and swapping back to the original for the actual call to console.log... But I tried it again and still no cigar. Maybe it used to work but doesn't now?

I used n to go back to an older version of Node.js (0.12), and it still doesn't seem to work. Maybe my old answer was just wrong?

Now that I think about it, this approach is probably a bit too much anyway - I would get debug logs for every single function call, including internal Node.js functions! Probably a bit too verbose.

Attempt 2 - Wrap `require` in a `Proxy`?

JavaScript has evolved quite a bit since I answered that question in 2013 - and one of the things that has been added is meta-programming capabilities. There's a special JavaScript object called a Proxy which is designed for this kind of function wrapping. But I still don't want to have to wrap all functions in a call to new Proxy...

One of the cool things about Node.js is that a lot of it is written in JavaScript, which means I can mess with the core functionality, for example require. I can do the same monkey-patching kind of thing from before.



global.require = (() => {
    const original = global.require;
    return function (...args) {
        const result = original.apply(this, args);
        // do stuff...
        return result;
    };
})();

So now I can examine the result of every call to require and mess with it!

I want to look at each of the object on the result (the module.exports object of the module) and find which ones are functions:



function getFunctions (exports) {
    const functions = {};
    Object.keys(exports)
      .filter((functionName) => typeof exports[functionName] === 'function')
      .forEach((functionName) => {
        functions[functionName] = map[functionName];
      });
    return functions;
}

So I can change the // do stuff to something like:



global.require = (() => {
    const original = global.require;
    return function (...args) {
        const exports = original.apply(this, args);
        const functions = getFunctions(exports);
        // wrap functions:
        return exports;
    };
})();

And the wrap functions code will look something like this:



Object.keys(functions).forEach((functionName) => {
    Object.defineProperty(exports, functionName, {
        value: new Proxy(exports[functionName], {
            // ... Proxy options:
        })
    });
});

This is looking pretty promising I think! But right now it is still going to wrap every single exported function from every module, and that's quite a lot of functions. It'd be great to exclude all dependencies, and only instrument the code from my library. So how does require work?



require('fs'); // Built in module
require('express'); // Third-party code in the node_modules folder
require('./some/thing'); // Local code for my library

I only want to intercept the last case. I can just intercept any call to require where the argument starts with a ., that should get ./ and ../ calls.



global.require = (() => {
    const original = global.require;
    return function (id) {
        const exports = original.apply(this, [id]);
        if (shouldWrap(id) {
          const functions = getFunctions(exports);
          // wrap functions:
        }
        return exports;
    };
})();

function shouldWrap (id) {
    return id.startsWith('.');
}

Ah, so slight problem. Third-party code in node_modules will refer to other local files in the same module, which also start with ., so I'm still getting too many functions. I need some better way of filtering the require ids!

I need to dig into require a bit more. It turns out that the global require function actually delegates to the Node.js Module class, which has a require function on its prototype. That function internally calls a _resolveFilename function which turns the require id into a file path. We can call that directly, and check if the path contains node_modules. We can also use the list of builtinModules to filter out things like path and fs:



immport { buildinModules, Module } from 'module';

Module.prototype.require = (() => {
    const original = Module.prototype.require;
    return function (id) {
      const requirePath = Module._resolveFilename(id, this);
      const module = original.apply(this, [id]);
      if (module && shouldWrap(requirePath)) {
        // wrap module:
      }
      return module;
    };
})();

function shouldWrap (requirePath) {
    const isNodeModule = builtinModules.includes(requirePath) || requirePath.includes('node_modules');
    return !isNodeModule;
}

One more little issue is that the require function isn't just a function - it's also an object that has some other stuff on it like require.resolve and require.cache. Right now I am just clobbering everything, so I have to do that a bit better:



immport { buildinModules, Module } from 'module';

const original = Module.prototype.require;
const debugRequire = function (id) {
  // ...
};
Module.prototype.require = Object.assign(debugRequire, original);

Okay, that's sorted, so how will the actual wrapping work? So far I have this:



Object.keys(functions).forEach((functionName) => {
    Object.defineProperty(exports, functionName, {
        value: new Proxy(exports[functionName], {
            // ... Proxy options:
        })
    });
});

The way a Proxy works is that you have a set of traps that describe different operations in JavaScript. I care about functions, so there are two traps that I care about, apply and construct.

Proxy `apply` trap:

The apply trap gives you control to what happens when a function is called - exactly what I want! The basic form of an apply trap handler is like this:



function apply (target, thisArg, argumentsList) {
    return target.apply(thisArg, argumentsList);
}

I can add a bit more stuff to make it more useful:



import { performance } from 'perf_hooks';

function apply (target, thisArg, args) {
    const startTime = performance.now();
    console.log(args);
    const result = target.apply(thisArg, args);
    const endTime = performance.now();
    console.log(result);
    console.log(endTime - startTime);
    return result;
}

One other thing is that result could be a promise, so I need to handle that case as well. Usually I'd prefer to not use the .then() syntax, but using async would make this function always return a Promise, which I don't want!



import { types } from 'util';

function apply (target, thisArg, args) {
    // ...
    const result = target.apply(thisArg, args);
    if (types.isPromise(result)) {
        return result.then((result) => {
            const endTime = performance.now();
            console.log(result);
            console.log(endTime - startTime);
            return result;
        });
      }
    // ...
}

Nice, that's pretty rough but it mostly works!

Proxy `construct` trap:

The construct trap gives you control to what happens when a function is called with new! The basic form of a construct trap handler is like this:



function construct (target, args) {
    return new target(...args);
  }

I'm going to do basically the same thing as the apply trap, but I don't need to worry about Promises, since constructors are synchronous.



import { performance } from 'perf_hooks';

function construct (target, args) {
    const startTime = performance.now();
    console.log(args);
    const instance = new target(...args);
    const endTime = performance.now();
    console.log(endTime - startTime);
    console.log(instance);
    return instance;
}

It'd be great if I could also wrap any methods of the instance too! I can access them via target.prototype and use the getFunctions function again:



function construct (target, args) {
    // ...
    const proto = target.prototype;
    const prototypeFunctions = getFunctions(proto);
    Object.keys(prototypeFunctions).forEach((functionName) => {
      Object.defineProperty(proto, functionName, {
        value: new Proxy(proto[functionName], {
          apply: applyTrapHandler
        })
      });
    });
    // ...
    return instance;
}

Okay so now I have Proxies that wrap Proxies 🤯, but it should still work!

Adding function names:

One thing that would be good to add is for the traps to log the name of the functions. I have access to the name everywhere I need it, but I need the trap to know about it:



function createFunctionCallWrap (name) {
  return function apply (target, thisArg, args) {
      // ...
      console.log(name);
      const result = target.apply(thisArg, args);
      // ...
      return result;
  };
}

function createConstructorCallWrap (name) {
  return function apply (target, thisArg, args) {
      // ...
      console.log(name);
      const instance = new target(...args);
      // ...
      return instance;
  };
}

Nice! Now I get logs for all function calls, all constructor calls, and all instance method calls!

Handling function arguments:

It would also be good to have logging for any function which are passed as arguments (like callback functions). Both traps give me access to the function arguments, so I just need to add another wrapper:



function wrapArgs(args) {
    return args.map((arg) => {
        if (typeof arg !== 'function') {
            return arg;
        }
        return new Proxy(arg, {
            apply: ...
        });
    });
  }

Okay but now I have a little problem. In all the other cases I have access to the name of the function or constructor that is being wrapped, whether it is from the module.exports object, or the target.prototype object. I don't have the names of the arguments to the function, so I can't know the name of any argument that happens to be a function!

I do have access to the function though, which means I can call toString() and get the full text of the function.



function foo (callback) { callback(); };

foo.toString() // "function foo (callback) { callback(); };"

I could do some RegExp magic and read the argument name?



const [argsString] = foo.toString().match(/function .* \((.*?)\)/);
const argNames = argsString.split(', ');

This is going to fall apart very quickly - I'd have to handle async functions, generators, arrow functions, optional arguments, etc. etc.

But I could use a proper parser and it could handle all that for me!



import { parseScript } from 'esprima';

function getFunctionArgumentNames(func) {
    const AST = parseScript(`const f = ${func.toString()}`)
    // find argument names in AST
  }

I can use esprima to get an Abstract Syntax Tree for the function. If func is an anonymous function (like function () {}) the parser will throw, so I need the const f = bit to handle that case.

Once I've got the AST, I can query it for the bits I care about:



import { parseScript } from 'esprima';
import { query } from 'esquery';

function getFunctionArgumentNames(func) {
    const [func] = query(parseScript(`const f = ${func.toString()}`), '[type=/Function/]');
    return func.params.map((param) => {
      const [identifier] = query(param, 'Identifier');
      return identifier.name;
    });
  }

I find the first (and only) function in the AST (which is the representation of the target function), and get its list of params. This could be normal parameters like a, b, c, or complex parameters like d = 6, e = getE(). In either case I want to grab the first Identifier, and return its name. That would be 'a', 'b', 'c', 'd', and 'e' in my above examples.

Now I can finish off wrapArgs



function wrapArgs(argNames, args) {
    return args.map((arg, index) => {
        if (typeof arg !== 'function') {
            return arg;
        }
        return new Proxy(arg, {
            apply: createFunctionCallWrap(argNames[index])
        });
    });
  }

And change createFunctionCallWrap and createConstructorCallWrap:




function createFunctionCallWrap (name) {
  return function apply (target, thisArg, args) {
      // ...
      const result = target.apply(thisArg, wrapArgs(args));
      // ...
      return result;
  };
}

function createConstructorCallWrap (name) {
  return function construct (target, args) {
      // ...
      const instance = new target(...wrapArgs(args));
      // ...
      return instance;
  };
}

Nicer logs:

Okay, so now I have wrapped all functions, constructors, prototype methods, and any of their arguments that happen to be functions. And each time any of those are called I have access to their name, arguments, return value, and start and end times. It would be awesome to render these as a list with each entry and exit on their own line so I can scan it and see how the program ran.

It would also be cool to show somehow how deep in the call stack the program is. I can track that by incrementing a count before executing the real function, and decrementing it once it has run. I'm going to wrap all this logic up into logStart and logEnd functions:



let depth = 0;

function logStart (name, args) {
    depth += 1;
    let debugString = printDepth(name);
    debugString += ` args: ${JSON.stringify(args)}`;
    process.stdout.write(`${debugString}\n`);
}

function logEnd (name, startTime, endTime, result) {
    let debugString = printDepth(name);
    debugString += ` time: ${endTime - startTime}ms`;
    debugString += ` return: ${JSON.stringify(result)}`;
    process.stdout.write(`${debugString}\n`);
    depth -= 1;
}

function printDepth (name) {
    return `${'▸'.repeat(depth)} ${name}`;
}

One issue with this is that if the args or result have any circular references in them, then JSON.stringify will throw. Node.js has already solved that for me though, so I can replace JSON.stringify with util.inspect.

I'm going to wrap that in a little helper so I can have all the options in one place:



import { inspect } from 'util';

function printObject(object) {
    return inspect(object, { getters: true, depth: Infinity }).replace(/\n/g, '');
}

This will print all the details about an object that I need, and then strip any new lines so that each log only uses one line.

So with some dummy code this is the output I get:



▸ init args: []
▸▸ Fixture args: [ 1, [ 'hello' ], [Function] ]
▸▸ Fixture return: Fixture { _a: 1, _b: [ 'hello' ], _c: [Function] }
▸ init return: Fixture { _a: 1, _b: [ 'hello' ], _c: [Function] }
▸ Fixture.func args: []
▸▸ Fixture._func args: [ 1 ]
▸▸▸ _c args: [ 1 ]
▸▸▸ _c return: 1
▸▸ Fixture._func return: [ 'hello' ]
▸ Fixture.func return: undefined

Adding this into my library:

So this is all going great and I want to try to add it to library. Since I want to turn it on and off with a environment variable, I have wrapped it all in a function like this:



function debug () {
    if (process.env.DEBUG) {
        // Overwrite require so it wraps everything.
    }
};

I've added the debug() call inside library and now when I run it nothing happens... The code looks something like this:



import { blah } from './blah';
import { whatever } from './whatever';

export function library () {
    debug();
    // run library
}

And when I put a debugger; inside the overwritten require call, it is never hit... 😐

So there are two things going on here.

1) All the import ... statements are converted into const { blah } = require('./blah'); calls.
2) The whole dependency graph is traversed long before my call to debug() happens.

All the calls to require happen before I've even turned on my debug mode...

To check that I'm right, I move the debug() call to the first line of my index file, so now it looks more like:



debug();

import { blah } from './blah';
import { whatever } from './whatever';

export function betterer () {
    // run Betterer
}

And cool, now it works! 😅

This is probably fine, but it means that the tool is much less configurable, and I can't change settings in between different Betterer runs. Maybe that's good enough?

Wrapping `require.cache`:

I think I can do better than that! As a Node.js application executes, and the dependency graph is traversed, a special cache on the require function is populated. I can access this cache using require.cache, and it contains the complete exports of any given dependency.

When I call debug() I can iterate over that cache and wrap all the functions that have already been required:



Object.keys(require.cache).forEach((requirePath) => {
    const module = require.cache[requirePath];
    if (module && shouldWrap(requirePath)) {
      wrap(module.exports);
    }
});

Boom, just like that. Now I can have my debug() call wherever I want and all past and future requires will be wrapped correctly!

Wrapping up 😎

So that's pretty neat and is working quite well! I'm quite happy with how it ended up, cause it looked pretty unlikely here and there! I've taken my prototype from here, rewritten it in TypeScript, added a bunch of options, and released it on npm at @phenomnomnominal/debug. Maybe that will be of some use to someone else?!

Hit me up with questions/comments/suggestions 🎉

I need to learn about TypeScript Template Literal Types

Craig ☠️💀👻 — Sun, 27 Sep 2020 01:54:21 +0000

It's Sunday in New Zealand and I don't want to get out of bed yet, so instead I'm going to listen to the new Menzingers album and learn about TypeScript Template Literal Types and write down what I found out as I go!

TypeScript string types:

Let's start with what I already know.

TypeScript has a string type. It covers all strings like const hello = "Hello World";, or const myName = `My name is ${name}`;.
You can also use a string literal type, such as type Hello = 'hello', which only matches that specific string.
You can use Union Types to be combine string literal types to be more precise about allowed string inputs. One good example is type Events = 'click' | 'doubleclick' | 'mousedown' | 'mouseup' | ...;

There are limitations to what TypeScript can know. Template strings will cause specific string types to expand out to the generic string type:

type A = 'a';
const a: A = `${'a'}`; // Argument of type 'string' is not assignable to parameter of type '"a"'.

In my experience, once you start typing stuff with specific strings you often end up duplicating a bunch of stuff too. Take the Events example from before:

type EventNames = 'click' | 'doubleclick' | 'mousedown' | 'mouseup';
type Element = {
    onClick(e: Event): void;
    onDoubleclick(e: Event): void;
    onMousedown(e: Event): void;
    onMouseup(e: Event): void;
    addEventListener(eventName: Event): void;
};

If I add a new event name to the EventNames type, I also have to change the Element type! That's probably fine most of the time, but it could cause issues.

Template Literal Types "basics"

(Spoiler: it's not basic at all!)

The PR where Template Literal Types looked cool when I first read it, and people got pretty excited! Then the TypeScript 4.1 Beta release notes came out, so I'm going to go through that first.

TypeScript 4.1 can concatenate strings in types using the same syntax from JavaScript:

type World = "world";

type Greeting = `hello ${World}`;
// same as
//   type Greeting = "hello world";

Using Union Types with concatenation enables combinations:

type VerticalAlignment = "top" | "middle" | "bottom";
type HorizontalAlignment = "left" | "center" | "right";
type Alignment = `${VerticalAlignment}-${HorizontalAlignment}`

declare function setAlignment(value: Alignment): void;

setAlignment("top-left"); // works!
setAlignment("middle-right"); // works!
setAlignment("top-middel"); // error!

There's also some fancy new mapping syntax which means I can change the Element type from before:

type EventNames = 'click' | 'doubleclick' | 'mousedown' | 'mouseup';
type Element = {
  [K in EventNames as `on${Capitalize<EventNames>}`]: (event: Event) => void;
} & {
  addEventListener(eventName: EventNames): void;
};
// same as 
// type Element = {
//  onClick(e: Event): void;
//  onDoubleclick(e: Event): void;
//  onMousedown(e: Event): void;
//  onMouseup(e: Event): void;
//  addEventListener(eventName: Event): void;
//};

That's pretty deep - it takes each of the strings in the EventNames type, passing it to a Capitalize type, and prepending on to each of them! Now if I add a new event name to the EventNames, the Element type will already reflect it!

These new features are obviously really powerful, and people have been making some amazing stuff, e.g.:

// Detect dark theme var iframe = document.getElementById('tweet-1301976526603206657-282'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1301976526603206657&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1304515347932999680-395'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1304515347932999680&theme=dark" }

Grégory Houllier collected some of these examples into one place, so I can see how they work by looking at the implementations!

Type-safe string dot notation:

// Detect dark theme var iframe = document.getElementById('tweet-1309489079378219009-184'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1309489079378219009&theme=dark" }

Full implementation here.

What does it do?

const user = {
  projects: [
    { name: "Cool project!", contributors: 10 },
    { name: "Amazing project!", contributors: 12 },
  ]
};

get(user, "projects.0.contributors"); // <- I want this string to be type-safe!

I thought I was starting with an easy one, but it's still pretty complex! I simplified it a little bit (and probably broke it) but it'll be easier to figure out - my implementation is here.

How does it work?

I'll look at PathValue first.

type PathValue<T, P extends Path<T>> =
  P extends `${infer Key}.${infer Rest}`
  ? Key extends keyof T
    ? Rest extends Path<T[Key]>
      ? PathValue<T[Key], Rest>
      : never
    : never
  : P extends keyof T
    ? T[P]
    : never;

This is the code that take an object and a valid path to an object and returns the type of the value at the end of that path.

Conditional types are really hard to process, so I'm going to rewrite it how I think about it.

PathValue is a generic type so it's kind of like a type function, and it takes two things, T which could be anything, and P which has to be a valid Path for T. PathValue is also a conditional type - it has the shape A extends B ? C : D. In this case it has several nested conditionals! But each of the never bits is a condition that doesn't return a type, so I can simplify it down to the two valid condition paths. That looks something like this:

typefunction PathValue (T, P: Path<T>) {
  if (P extends `${infer Key}.${infer Rest}` && Key extends keyof T && Rest extends Path<T[Key]>) {
    return PathValue<T[Key], Rest>;
  }
  if (P extends keyof T) {
    return T[P];
  }
}

Since the first condition actually calls PathValue again, this is a recursive conditional type 🤯🤯🤯. There are two base conditionals, one continues the recursion, the other ends it. Again I'll look at the "easier" one first.

if (P extends keyof T) {
  return T[P];
}

If P is just a string and it is an exact key of T, then return the type of that key. That means it is the end of the path and it can stop recursing.

The other condition is the magical bit.

if (P extends `${infer Key}.${infer Rest}` && Key extends keyof T && Rest extends Path<T[Key]>) {
  return PathValue<T[Key], Rest>;
}

Here's the fancy new bit:

P extends `${infer Key}.${infer Rest}`

This type says "check if the string contains a '.', and give me the two string literal types either side of the '.'". The equivalent JavaScript would be something like:

const [Key, Rest] = P.split('.');

The next part of the conditional takes the first string literal (Key) and makes sure it is a valid key of T:

Key extends keyof T

The last part of the conditional takes the second string literal (Rest) and makes that it is a valid Path for the type of T[Key].

So in the case of the example:

const user = {
  projects: [
    { name: "Cool project!", contributors: 10 },
    { name: "Amazing project!", contributors: 12 },
  ]
};
get(user, "projects.0.contributors");

If these conditions are all true, then the recursion continues and you go to the level in the object, and the next chunk of the dot-notation string.

That kind of makes sense and I now kind of understand P extends `${infer Key}.${infer Rest}` which seems pretty important. Next up is the Path type:

type Path<T, Key extends keyof T = keyof T> =
  Key extends string
  ? T[Key] extends Record<string, any>
    ? | `${Key}.${Path<T[Key], Exclude<keyof T[Key], keyof Array<any>>> & string}`
      | `${Key}.${Exclude<keyof T[Key], keyof Array<any>> & string}`
      | Key
    : never
  : never;

Again I'm going to write it out in a different way:

typefunction Path<T, Key extends keyof T = keyof T> {
  if (Key extends string && T[Key] extends Record<string, any>) {
    return `${Key}.${Path<T[Key], Exclude<keyof T[Key], keyof Array<any>>> & string}` | `${Key}.${Exclude<keyof T[Key], keyof Array<any>> & string}` | Key;
  }
}

This says that it Key is a string, and the type of the property on type T (Aka T[Key]) is a Record, then return some fancy union. There are three parts to the union:

`${Key}.${Path<T[Key], Exclude<keyof T[Key], keyof Array<any>>> & string}`

`${Key}.${Exclude<keyof T[Key], keyof Array<any>> & string}`

Key;

What does the Exclude<keyof T[Key], keyof Array<any>> bit mean? It uses TypeScript's built-in Exclude type which will remove any types in the second parameter from the first. In this specific case, it is going to remove any valid key for an Array (e.g. push, map, slice). I guess this also includes Object keys, but I'm not super sure how that works off the top of my head. This bit seems to me to be a bit of a nice to have, as it reduces the final set of possible paths a bit, but I can ignore it for now. That gives me:

`${Key}.${Path<T[Key], keyof T[Key]> & string}`

`${Key}.${keyof T[Key] & string}`

Key;

The & string bit is a little trick to reduce keyof T[Key] down to only being a string - I think because you can have symbol keys as well. So I can ignore that too:

So the final union is basically:

`${Key}.${Path<T[Key], keyof T[Key]>}` | `${Key}.${keyof T[Key]}` | Key;

This is another recursive type, where each level of recursion is concatenating the valid key paths like `${Key}.{Path}`, so you get `${Key}.{Path}` | ${Key}.{(`${Key}.{Path})`} | `${Key}.{(`${Key}.{Path})`}` ... etc. That handles all the deeply nested keys. That is combined with the very next layer of keys ${Key}.${keyof T[Key]}, and the current keys Key.

So at a high level there are two recursive types, one with recurses through the valid keys of an object and builds up the whole valid set, using Template Literal Types to concatenate the keys with a ".". The other type splits the concatenated keys and works out the type at each layer of the path. Makes sense I think? Pretty powerful stuff if you hide it away behind a nice API in a library.

Type-safe document.querySelector:

// Detect dark theme var iframe = document.getElementById('tweet-1308472279010025477-178'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1308472279010025477&theme=dark" }

Full implementation here.

What does it do?

This one is a little different, as it doesn't validate that the string is a valid CSS selector (although I'm pretty sure that would be possible with these new types), but it does figure out the best type of the result of the query:

const a = querySelector('div.banner > a.call-to-action') //-> HTMLAnchorElement
const b = querySelector('input, div') //-> HTMLInputElement | HTMLDivElement
const c = querySelector('circle[cx="150"]') //-> SVGCircleElement
const d = querySelector('button#buy-now') //-> HTMLButtonElement
const e = querySelector('section p:first-of-type'); //-> HTMLParagraphElement

How does it work?

Let's look at some of the helper types first:

type Split<S extends string, D extends string> = S extends `${infer T}${D}${infer U}` ? [T, ...Split<U, D>] : [S];
type TakeLast<V> = V extends [] ? never : V extends [string] ? V[0] : V extends [string, ...infer R] ? TakeLast<R> : never;
type TrimLeft<V extends string> = V extends ` ${infer R}` ? TrimLeft<R> : V;
type TrimRight<V extends string> = V extends `${infer R} ` ? TrimRight<R> : V;
type Trim<V extends string> = TrimLeft<TrimRight<V>>;

These are super clever, and seem like they could live alongside Capitalize etc in the base TypeScript types.

Split:

type Split<S extends string, D extends string> = S extends `${infer T}${D}${infer U}` ? [T, ...Split<U, D>] : [S];

Again I'm going to rewrite it:

typefunction Split<S extends string, D extends string> {
    if (S extends `${infer T}${D}${infer U}`) {
        return [T, ...Split<U, D>];
    }
    return [S];
}

So there is another recursive type that takes an input string, and some splitter string D. If the input string contains the splitter string, the part of the string that comes before the splitter is put into an array, and then the second part of the string is passed to the Split type again. The result is splatted (...) which means that the final result will be a single flattened array of strings.

If the input string doesn't contain the splitter, then the whole string is returned. It's wrapped in an array so that the splat works.

TakeLast:

type TakeLast<V> = V extends [] ? never : V extends [string] ? V[0] : V extends [string, ...infer R] ? TakeLast<R> : never;

This one doesn't have anything to do with Template Types particularly but it's still interesting. Rewriting gives me something like this:

typefunction TakeLast<V> {
    if (V extends []) {
        return;
    }
    if (V extends [string]) {
        return V[0];
    }
    if (V extends [string, ...infer R]) {
        return TakeLast<R>;
    }
}

One change I might make to this would be to have type TakeLast<V> be typefunction TakeLast<V extends Array<string>>? That would limit the valid input types and possibly give an easier error message.

Three different paths through here:

1) If the array is empty, return nothing.
2) If the array contains one element, return it.
3) If the array contains more than one element, skip the first element and call TakeLast on the array of remaining elements.

TrimLeft/TrimRight/Trim:

type TrimLeft<V extends string> = V extends ` ${infer R}` ? TrimLeft<R> : V;
type TrimRight<V extends string> = V extends `${infer R} ` ? TrimRight<R> : V;
type Trim<V extends string> = TrimLeft<TrimRight<V>>;

More Template String types here:

Trim is pretty nice, it just calls TrimRight and then TrimLeft.

TrimLeft and TrimRight are basically the same so I'll just rewrite one of them:

typefunction TrimLeft<V extends string> {
    if (V extends ` ${infer R}`) {
        return TrimLeft<R>;
    }
    return V;
}

And I'll actually rewrite this again cause what it's actually doing is:

typefunction TrimLeft<V extends string> {
    if (V.startsWith(' ')) {
        return TrimLeft<R>;
    }
    return V;
}

This type recurses until it finds a string that doesn't start with a space. Makes sense, but still very cool to see it as a type.

TrimRight is pretty much identical but it does an endsWith instead.

StripModifiers

The last bit of Template Type magic I want to look at here is:

type StripModifier<V extends string, M extends string> = V extends `${infer L}${M}${infer A}` ? L : V;
type StripModifiers<V extends string> = StripModifier<StripModifier<StripModifier<StripModifier<V, '.'>, '#'>, '['>, ':'>;

That can be rewritten to be something like this:

typefunction StripModifier<V extends string, M extends string> {
    if (V.contains(M)) {
        const [left, right] = V.split(M);
        return left;
    }
    return V;
}

Then the StripModifiers type just uses the StripModifier type with each of the characters than can follow an element tag name in CSS:

typefunction StripModifiers<V extends string> {
    StripModifier(V, '.');
    StripModifier(V, '#');
    StripModifier(V, '[');
    StripModifier(V, ':');
}

The rest of this example uses these different types to split the CSS selector on relevant characters (' ', '>', and ','), and then select the relevant bit of the remaining selector and returning the correct type.

A lot of the heavy lifting is done by this type:

type ElementByName<V extends string> = 
    V extends keyof HTMLElementTagNameMap 
        ? HTMLElementTagNameMap[V] 
        : V extends keyof SVGElementTagNameMap 
        ? SVGElementTagNameMap[V] 
        : Element;

It maps from a string (such as 'a') to a type (such as HTMLAnchorElement), then checks SVG elements, before falling back to the default Element type.

What next?

The next examples get progressively more bonkers, so I'm not going to write down all my thinking about them - you should check them out though and see if you can see how they work. The JSON parser is probably the best mix of complex and readable.

From this I have a couple thoughts:

1) I should definitely use this for TSQuery
2) TypeScript is probably going to need new syntax for types soon because stuff like:

type ParseJsonObject<State extends string, Memo extends Record<string, any> = {}> =
  string extends State
    ? ParserError<"ParseJsonObject got generic string type">
    : EatWhitespace<State> extends `}${infer State}`
      ? [Memo, State]
      : EatWhitespace<State> extends `"${infer Key}"${infer State}`
        ? EatWhitespace<State> extends `:${infer State}`
          ? ParseJsonValue<State> extends [infer Value, `${infer State}`]
            ? EatWhitespace<State> extends `,${infer State}`
              ? ParseJsonObject<State, AddKeyValue<Memo, Key, Value>>
              : EatWhitespace<State> extends `}${infer State}`
                ? [AddKeyValue<Memo, Key, Value>, State]
                : ParserError<`ParseJsonObject received unexpected token: ${State}`>
            : ParserError<`ParseJsonValue returned unexpected value for: ${State}`>
          : ParserError<`ParseJsonObject received unexpected token: ${State}`>
        : ParserError<`ParseJsonObject received unexpected token: ${State}`>

is pretty tricky 😅.

All in all that was pretty useful for me, I think I get how Template Literal Types work a bit now. I guess I'll see next time I try to use them.

Let me know if this was useful, it was a pretty unfiltered and unedited 🙃

Stricter TypeScript compilation with Betterer

Craig ☠️💀👻 — Wed, 20 May 2020 16:44:52 +0000

This week I announced the new release of ☀️ Betterer, and I'm still really excited about it! I'd like to walk you through a pretty powerful application of Betterer - incrementally adding TypeScript's strict mode.

Betterer TL;DR

Betterer is a test runner that helps make incremental improvements to your code! It is based upon Jest's snapshot testing, but with a twist...

Betterer works in two stages. The first time it runs a test, it will take a snapshot of the result. From then on, it will compare the new result against that snapshot. If the test gets worse, that's a regression and Betterer will throw an error ❌. If the test gets better, we've made some progress and Betterer will update the snapshot ✅. That's pretty much it!

What is strict mode?

TypeScript's strict mode turns on a more aggressive level of type-checking, sometimes jokingly referred to as Typescript's "hard mode"... 🥵

Turning on strict mode enables the following:

noImplicitAny - raise error on expressions and declarations with an implied any type
noImplicitThis - raise error on this expressions with an implied any type
alwaysStrict - parse in strict mode and emit "use strict" for each source file
strictBindCallApply - enable stricter checking of the bind, call, and apply methods on functions
strictNullChecks - in strict null checking mode, the null and undefined values are only assignable to themselves and any
strictFunctionTypes - disable bivariant parameter checking for function types
strictPropertyInitialization - ensure non-undefined class properties are initialized in the constructor

These extra checks give you even more certainty about the behaviour of your code, but if you don't turn them on when you start a project it can be really tricky to enable them later... 🥶 But that's where Betterer comes in!

`@betterer/typescript`

Betterer comes with a package called @betterer/typescript, which enables running the TypeScript compiler with a modified version of an existing configuration:

import { typescriptBetterer } from '@betterer/typescript';

export default {
  'consistent casing in file names': typescriptBetterer('./tsconfig.json', {
    forceConsistentCasingInFileNames: true
  })
};

As you can see, this test works with any of the TypeScript compiler options! But we want to enable hard mode, so we're going to use it like this:

import { typescriptBetterer } from '@betterer/typescript';

export default {
  'stricter compilation': typescriptBetterer('./tsconfig.json', {
    strict: true
  })
};

This test will use Betterer to create a snapshot of the current state, and prevent us from regressing. Then we will fix the issues when we can and incrementally migrate to strict mode!

Cool. Cool cool cool.

Finding a test subject...

So what does it look like with a real project? 🤔 I'm a big fan of strict mode, and I usually add it at the start of my projects! So instead I asked on Twitter if anyone has a project that they wanted to strictify:

// Detect dark theme var iframe = document.getElementById('tweet-1262909583913181184-358'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1262909583913181184&theme=dark" }

And I got a reply 🎉:

// Detect dark theme var iframe = document.getElementById('tweet-1262913787453149184-250'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1262913787453149184&theme=dark" }

Let's go through the process of adding @betterer/typescript to observable-webworker.

Setting up Betterer

First things first, we're going to need to enable Betterer in the project. We can do this by installing the Betterer VS Code Extension, and then running the Initialise Betterer command:

Yes, it's that easy!

Next, we need to add the @betterer/typescript dependency and install everything:

yarn add @betterer/typescript -D

Creating the test

Now we can add our test in the brand new .betterer.ts file:

import { typescriptBetterer } from '@betterer/typescript';

export default {
  'stricter compilation': typescriptBetterer('./tsconfig.json', {
    strict: true
  })
};

As a reminder, this test is going to run the TypeScript compiler with the existing project settings and the additional strict: true setting, and see how many issues there are...

The first time we run it, we get this:

And we get a whole bunch of issues reported in the .betterer.results file:

// BETTERER RESULTS V1.
exports[`stricter compilation`] = {
  timestamp: 1589981710157,
  value: `{
    "projects/observable-webworker/src/lib/from-worker-pool.ts:3842672600": [
      [46, 12, 18, "Type \'Worker\' is not assignable to type \'null\'.", "1459294796"],
      [47, 12, 12, "Type \'true\' is not assignable to type \'false\'.", "4249186060"],
      [49, 22, 13, "Property \'_cachedWorker\' does not exist on type \'LazyWorker\'.", "3268970724"],
      [53, 12, 18, "Object is possibly \'null\'.", "1459294796"]
    ],
    "projects/observable-webworker/src/lib/from-worker.ts:4009652202": [
      [28, 56, 11, "Argument of type \'Input | undefined\' is not assignable to parameter of type \'Input\'.\\n  Type \'undefined\' is not assignable to type \'Input\'.", "1574273654"]
    ],
    "projects/observable-webworker/src/lib/run-worker.ts:1913596105": [
      [4, 65, 7, "Rest parameter \'args\' implicitly has an \'any[]\' type.", "3622679692"],
      [51, 38, 26, "Cannot invoke an object which is possibly \'undefined\'.", "4028913799"],
      [51, 65, 18, "Argument of type \'O | undefined\' is not assignable to parameter of type \'O\'.\\n  Type \'undefined\' is not assignable to type \'O\'.", "3307950093"]
    ],
    ...,
    ...,
    ...,
    "src/readme/worker-pool-hash.worker.ts:2220989309": [
      [0, 21, 8, "Could not find a declaration file for module \'js-md5\'. \'./node_modules/js-md5/src/md5.js\' implicitly has an \'any\' type.\\n  Try \`npm install @types/js-md5\` if it exists or add a new declaration (.d.ts) file containing \`declare module \'js-md5\';\`", "3516231053"]
    ]
  }`
};

That's great! There's quite a few issues there, but now we've got an established baseline and we're not going to introduce any new issues. And thanks to the Betterer VS Code Extension they even show up in the file in the editor:

We don't have time to fix them all right now, but while we're here we could fix a few! 😎

Fixing some issues:

Let's go through and fix a few issues in a file, with the handy VS Code Diagnostics helping us out. And once we're done, we can run Betterer again to update the snapshot!

Now we can make a nice little pull request to the observable-webworker Github Repo and help them on their way to strict mode! 😍

Tada! 🥳🥳🥳

That was a quick little walkthrough of one of the things Betterer can do - what do you reckon? Please let me know in the comments or hit me up on the Twitters! 🦄

Improving Accessibility with Betterer

Craig ☠️💀👻 — Fri, 15 May 2020 16:02:02 +0000

So, yesterday I announced the new release of ☀️ Betterer, thanks if you've checked it out already!

I wanted to write another post describing a different example, this time with a custom test instead of a built-in test! Let's take a look at how we can prevent accessibility regressions (and hopefully encourage improvements!) 👀

Betterer TL;DR

Betterer is a test runner that helps make incremental improvements to your code! It is based upon Jest's snapshot testing, but with a twist...

Betterer works in two stages. The first time it runs a test, it will take a snapshot of the current state. From that point on, whenever it runs it will compare against that snapshot. It will either throw an error (if the test got worse ❌), or update the snapshot (if the test got better ✅). That's pretty much it!

Our first Betterer test

Writing a test with Betterer involves implementing two functions! More formally, we need to implement the BettererTest type:

type BettererTest<ResultType> = {
  test: () => ResultType | Promise<ResultType>;
  constraint: (result: ResultType, expected: ResultType) => ResultType | Promise<ResultType>;
}

So we need to write two functions:

test - the action that needs to happen to get a result,
constraint - the rule to apply to check if the result is better, worse or the same

We can implement these in their own file, or straight in the .betterer.ts file. To keep it simple, we'll do the latter:

// .betterer.ts
export default {
  'improve accessibility': {
    test: ...?,
    constraint: ...?
  }
};

Writing the test

To implement our test, we're going to use Puppeteer and Axe. Puppeteer is a tool that will run a browser and load a page. Axe is a set of accessibility audits that we can run over a web page. We're also going to use Axe Puppeteer which makes it a bit easier to use Axe with Puppeteer.

Lucky for us, we can take the example straight from the Axe Puppeteer documentation! 😍

We launch Puppeteer, get the page that it creates for us and navigate to a website. Then we execute Axe and get the results. Next, we close the page and the browser, before finally returning the number of violations! 🤓

import { AxePuppeteer } from 'axe-puppeteer';
import * as puppeteer from 'puppeteer';

async function improveAccessibility () {
  const browser = await puppeteer.launch();
  const [page] = await browser.pages();

  await page.goto('https://phenomnomnominal.github.io/betterer');
  const results = await new AxePuppeteer(page).analyze();

  await page.close();
  await browser.close();

  return results.violations.length;
}

That's our test sorted!

Writing the constraint

Now what about the constraint? Since our test returns a number, we just need to compare the two results. We want our test to improve when the result is smaller, so the constraint should look something like this:

import { ConstraintResult } from '@betterer/constraint';

function constraint (result: number, expected: number): ConstraintResult {
  if (current === previous) {
    return ConstraintResult.same;
  }
  if (current < previous) {
    return ConstraintResult.better;
  }
  return ConstraintResult.worse;
}

Comparing numbers is fairly common, so we can use the pre-defined smaller or bigger constraints:

import { smaller } from '@betterer/constraint';

So I kind of lied, you can write a test with just one function! 😅

The whole thing

Putting it all together, we have our test:

// .betterer.ts
import { smaller } from '@betterer/constraint';
import { AxePuppeteer } from 'axe-puppeteer';
import * as puppeteer from 'puppeteer';

export default {
  'improve accessibility': {
    async test () {
      const browser = await puppeteer.launch();
      const [page] = await browser.pages();

      await page.goto('https://phenomnomnominal.github.io/betterer');
      const results = await new AxePuppeteer(page).analyze();

      await page.close();
      await browser.close();

      return results.violations.length;
    },
    constraint: smaller
  }
};

How's that look? Not bad I reckon! Betterer will run the test for us and update the test snapshot whenever the results get better. That will help make sure that our audit score only goes in the right direction.

This test isn't perfect - you may noticed that it doesn't matter what the violations are, but how many there are! That's something that we could improve later! For now it will stop us introducing more audit violations, which is a good start ⭐️⭐️⭐️

We could improve this test by keeping track of the specific violations that occurred, so we can have a clearer definition of what better or worse really is, but let's leave that for another day!

That's it!

That's all I got for now. Please let me know what you think in the comments or hit me up on the Twitters! 🦄

Betterer v1.0.0

Craig ☠️💀👻 — Thu, 14 May 2020 17:20:34 +0000

I'm stoked to announce v1.0.0 of Betterer!

I've been locked down in New Zealand for the last little while, and I've used some of that time to smash out what I think is a pretty compelling v1 release of a tool that I'm really excited about!

What is Betterer?

Betterer is a test runner that helps make incremental improvements to your code! It is based upon Jest's snapshot testing, but with a twist...

I'm sure many of us have been in situations where we've seen big changes we'd like to make, or new standards or design decisions that we'd like to encourage, but we just don't have the time to do it.

Usually one of two things happen:

You start a long-lived branch that is awful to maintain and often impossible to merge. It ends up being a time sink ⏱
You and your team make an agreement to make the improvement over time. It gets forgotten about and nothing gets better (in fact usually it gets worse!) 😕

I've seen this happen time and time and again! Sometimes it's introducing a new style rule to a codebase. Other times it's enabling stricter compilation, or decreasing the number of accessibility failures!

How does it work?

To get started, you can run the following from the root of your project:

npx @betterer/cli init

That will give you a brand new .betterer.ts config file which looks something like this:

// .betterer.ts

export default {
  // Add tests here ☀️
};

From here, it's up to you to add some tests!

Let's imagine you're working with a codebase that uses Moment.js. You'd like to migrate away from it for performance reasons.

// src/subtract.js
import * as moment from 'moment';

const now = moment();

console.log(now.subtract(4, 'years'));

Let's also imagine that you're using ESLint in this codebase. One approach to remove Moment.js might be to use the no-restricted-imports ESLint rule, add the eslint-disable-next-line comment all over the place, and cross your fingers that people don't just add more... 🤔

Betterer gives us a better option! We can create a test for that specific rule:

// .betterer.ts
import { eslintBetterer } from '@betterer/eslint';

export default {
  'no import from moment': eslintBetterer('./src/**/*.js', [
    'no-restricted-imports',
    [
      'error',
      {
        name: 'moment',
        message: 'Please use "date-fns" instead.'
      }
    ]
  ])
};

The first time we run the test with Betterer, it will look something like this:

Betterer has now created a snapshot of the current state, stored by default in a .betterer.results file:

// BETTERER RESULTS V1.
exports[`no import from moment`] = {
  timestamp: 1589459511808,
  value: `{
    "src/subtract.js:566118541": [
      [0, 0, 33, "\'moment\' import is restricted from being used. Please use \\"date-fns\\" instead.", "4035178381"]
    ]
  }`
};

The snapshot contains information about the current issues in the code.

The next time we run the test, it will look like this:

Now, someone else on the team comes along and doesn't know about the new rule, and they add a new file that uses Moment.js:

// src/add.js
import * as moment from 'moment';

const now = moment();

console.log(now.add(4, 'years'));

When Betterer runs on their code, they get a nice big error:

Even though a new issue has been introduced, the .betterer.results file doesn't change!

Our teammate reads the helpful error message from ESLint and they update their code to use date-fns...

// src/add.js
import { addYears } from 'date-fns';

console.log(addYears(Date.now(), 4));

... and once again Betterer tells them that the result is the same:

Our teammate has a bit of time on their hands, so they decide to fix up our usage of Moment.js as well! 🎉

This time when they run Betterer, everything is good:

There are now no remaining issues, so this test has met its goal. Since the existing issue has been resolved, it is removed from the snapshot in the .betterer.results file. This means we can move the rule from Betterer over to the normal ESLint configuration, so we don't reintroduce the issues again.

Pretty neat eh! That's an example of the built-in @betterer/eslint test, but there are other built-in tests too. And you can of course write your own tests! Check out the documentation for more details (still a WIP 🚧)!

What's in v1.0.0?

Everything I've mentioned so far has been working for a while! Over the last few months I've really solidified the implementation (basically a whole rewrite to be honest!):

Better error handling and error messages
Better issue comparison. It now understands file renames and issues that around within the same file
The ability to run tests on a single file via the JS API, with betterer.single
A whole bunch more tests!

But I've also added a few key features that are worthy of a 1.0.0 release! 🔥🔥🔥

Force Update (!)

First things first, you can now run Betterer with the --update flag, and the snapshot will be updated even if it got worse! This is handy for when you need to ship something, even if it makes it temporarily worse:

betterer --update

And because this is shamelessly stolen from Jest, you can also use -u.

Watch mode (!!!)

This one is huge! All the changes that I made were building up to this. You can now run Betterer in watch mode and get feedback as you fix up issues:

Same rules apply here, the snapshot will update whenever the test gets better!

There's a bunch of other cool things that could happen with watch mode (gamification much?), so I'm pumped that it's working! 🤩

VS Code extension (!!!!!)

Way to bury the lede! This is probably the coolest bit! Let's just say it was a build up.

Betterer runs entirely in its own world, so the usual ESLint or TypeScript extensions can't report the issues. But everyone loves seeing red squiggly lines in the code editor, so Betterer now has its very own VS Code Extension 🤯!

Initialise in a new project

You can run the betterer.init command in a project! It will generate config files and update your package.json with scripts and dependencies:

See all issues in a file

And when you've got some tests setup, it will show you all the existing issues in a file, and when they were first created. And it will show you any new issues as you make them:

I reckon that's pretty sweet!

So, what's next?

Well, you're going to try out Betterer and open lots of issues and help me make it better(er)! If you'd like to contribute...

There are definitely bugs to fix
The test coverage is pretty good, but there aren't any E2E tests for the VS Code extension yet
The logging and reporting implementation could use some work
There is so much documentation to be written

What a time to be alive! 🤓

In case you can't tell, I'm unreasonably excited about this and I really think this will help with large and legacy codebases. Please let me know what you think in the comments or hit me up on the Twitters!

The new Angular YouTube Player component!

Craig ☠️💀👻 — Sat, 02 Nov 2019 22:36:51 +0000

Easily embed YouTube videos in your Angular applications!

TL;DR; —

There’s a cool new component in the Angular world 🌎! It makes it easier than ever before to add YouTube videos to your Angular applications. Install it today and try it out:

npm i @angular/youtube-player

To find out more details about it, and learn about the full API, read on! 📚

The Angular YouTube Player component 🎉:

One of the big exciting recent changes to Angular (other than Ivy), is the rename of the angular/material repository to angular/component ✨. This rename is meant to capture the fact that the repository is for so much more than only the Angular Material library! The Angular Components repository already also contains the Angular Component Dev Kit, and it is going to be the place where new, high-quality Angular components are built and shared with the community. With the release of the first release candidate of Angular v9, we get our first look at these new components!
One of these brand new components is the @angular/youtube-player. This component is a thin wrapper around the embed Youtube player API. Let’s check out what it looks like:

Documentation is a bit thin on the ground right now, so let’s dig into the code and see how this thing works! First, we need to import and include YouTubePlayerModule:

This gives us access to the <youtube-player> component!

Pretty straightforward!

The Angular YouTube Player API:

Unfortunately documentation for the full API hasn’t written yet. Luckily, we can check out the source code for the component and see what we find…

⚠️ Warning! ⚠️

The YouTube Player component is still pre-release, so this API could change!

Inputs:

[videoId]: string — YouTube Video ID to render. It’s the little hash at the end of the YouTube URL. For example, if your video is found at https://www.youtube.com/watch?v=moSFlvxnbgk, then your videoId is moSFlvxnbgk.
[height]: number — height of video player
[width]: number — width of video player
[startSeconds]: number — the moment when the player is supposed to start playing
[endSeconds]: number — the moment when the player is supposed to stop playing
[suggestedQuality]: SuggestedQuality — the suggested quality of the player. This can take the values 'default' , 'small', 'medium', 'large', 'hd720', 'hd1080', and 'highres'
[showBeforeIframeApiLoads]: boolean — whether the iframe will attempt to load regardless of the status of the API on the page. Set this to true if you don’t want the onYouTubeIframeAPIReady field to be set on the global window

Outputs:

Each output maps to the events fired from the YouTube JS API.

(ready): PlayerEvent — fires whenever a player has finished loading and is ready to begin receiving API calls.
(stateChange): OnStateChangeEvent — fires whenever the player’s state changes
(error): OnErrorEvent — fires if an error occurs in the player
(apiChange): PlayerEvent — fires to indicate that the player has loaded (or unloaded) a module with exposed API methods
(playbackQualityChange): OnPlaybackQualityChangeEvent — fires whenever the video playback quality changes
(playbackRateChange): OnPlaybackRateChangeEvent — fires whenever the video playback rate changes

Here’s a slightly more complete code snippet for rendering and playing a video with the <youtube-player> component:

The end! 📼 🎈

And that’s it! It’s great to see cool new components coming from the Angular team. This one provides a really clear API around a pretty common use-case. I’m sure you’ll all come up with awesome ways to use it. Check it out, file any bugs against the Angular Components repository, and let me know what you’re building!
🦄

Easier TypeScript tooling with TSQuery

Craig ☠️💀👻 — Sun, 27 Oct 2019 17:44:21 +0000

TLDR; 🔥🔥🔥

Need to use the TypeScript APIs for traversing AST nodes? TSQuery might make that a little bit easier. You can use CSS-ish selectors to query the Abstract Syntax Tree, just like you would query the DOM tree:

If you’re not sure what an AST is, or just want to a bit of a reminder, check out this talk I gave at JS Conf AU 2018 for a refresher, or just read on further!

❤️ Love it already? Check out the code on Github.

🦄 Try it out! Have a play in the playground made by Uri Shaked

⁉️ Want to know more? Read on…

What is TSQuery?

TSQuery is a rewrite of ESQuery for TypeScript. ESQuery is a neat little library that allows you to use CSS-like selectors to query a JavaScript AST. That turns out to be a really powerful and flexible way to get information about a piece of JavaScript code! Over the past few months I’ve been writing more TS tooling than JS, and I’ve really missed the power of ESQuery — so I decided to port it over.

A whole range of selectors are supported:

AST node types: ClassDeclaration, Identifier, VariableDeclaration, etc.
Attributes: [name], [left.value=4], [name.name=/^I.*/], [statements.length<3]
:first-child, :last-child, nth-child(6)
node descendant, node > child, node ~ sibling, node + adjacent
and more!

If we look back at our example of a selector from above, we can break it down and explain what’s going on:

tsquery(ast, 'ClassDeclaration[name.name="MyClass"] > Constructor');

Here we start with a query for a specific Node type, a ClassDeclaration. We want to find one where the name property (which is an Identifier node) has a name with the value "MyClass". We then want to find a direct descendent of that which is a Node of type Constructor. We can run this query over the code from above and get an array full on all the matching nodes. In this case, there would only be one result: the node for the constructor!

🌈 Neat!

So, why do we need TSQuery?

I’ve been working with TypeScript a lot lately, and I really love it ❤️! If you went back and told that to the 2014 version of me, I probably wouldn’t believe you, but it’s true!

In my ~5 years of using TypeScript in a fairly large AngularJS/Angular project, I’ve found that adding types to JavaScript makes it easier to build stuff good™, and to maintain that stuff as a codebase grows and evolves. Having types makes it easier to reason about my code, makes refactoring safer, and generally gives me more confidence in the software I create and ship 🚢.

Those reasons alone would be enough to make me really love TypeScript. But the real killer feature is the incredible ecosystem of tooling that has been developed by the TypeScript team and community, including:

The integrations for various IDEs (such as Webstorm, or VS Code) that make refactoring super easy
Specialised tools like TSLint for automated code-style checking
The whole suite of tools that power the Angular CLI for code generation and automatic updating
All these things are built on top of the TypeScript language, and they combine to make a very powerful ecosystem!

But it’s not all roses 🌹🌹🌹…

These tools are all great to use, but writing them can be quite a painful experience. There are lots of barriers to getting started with creating your own tools. To me, the biggest obstacle is getting your head around the idea of an Abstract Syntax Tree (AST), and how you can interrogate and manipulate one.

An AST is a data structure that represents the structure of code in a programming language, without any actual syntax. It describes the ideas that make up a piece of code, without talking about the specific keywords, or specific syntactical tokens.

You can read more about ASTs in this excellent article by Gabriele Petronella

An example of an AST could look something like this:

The “abstract” nature of the data structure is important, as it means that an AST doesn’t necessarily correlate to any particular programming language. It also means that you no longer need to use regular expressions or string manipulation to decipher or modify your source code! Instead, you can refer to parts of the code using the underlying concepts, whether it be the idea of a class, or the fact that a class has a name, or a constructor.

The above AST is a description of the following piece of TypeScript code:

export class MyClass {
    constructor () {

    }
}

Let’s say we want to know something about the constructor of MyClass: Does it take any arguments? Does it actually do anything? We could look at the code directly and find out the answers, but we could also find out by looking at the AST.

TypeScript gives us an easy way to create the AST of a block of code, with the createSourceFile() function. It can be used like this:

Et voilà, we now have a SourceFile object. If we print it out, we can see the same tree structure of the AST as before:

Looking at this we can start to see the parent — child relationships between the nodes. We have a SourceFile (the root node of the AST), which has a series of statements. The first statement is a ClassDeclaration which has a number of members. The first member is the Constructor which has a body, which has its own set of statements 😅 … phew!

Thankfully, we don’t have to memorise all the different names of all the different types of children! TypeScript also gives us an easy way to iterate over all the child nodes of an AST node, with the forEachChild() function.

We can use forEachChild() to loop over the AST, and manually filter out the nodes until we get what we’re after:

That works well, and code just like this powers much of the TSLint project. But it isn’t particularly easy to read, or write, or maintain. And to even get started you have to know about the finer details of TypeScripts’ SourceFile APIs. We can do better!

We have a tree structure that we run queries against to select tree nodes. This is directly analogous to using CSS selectors to query the DOM and select elements!

Let’s look at the TSQuery code for doing the same thing:

That’s a bit better, isn’t it? No more createSourceFile(), and no more forEachChild()!

TSQuery replaces all the manual iterating and filtering of the earlier example with familiar CSS-like selectors. My hope is that, by using a familiar mental model, we can break down some barriers and enable more developers to build really useful tools for the TypeScript ecosystem.

TSQuery also makes it possible to compose, share, and manipulate AST selectors in ways that wouldn’t have really been possible before!

I hope you like it, and I can’t wait to see what people make with it!

What next?

Hopefully I’ve explained why this is a Good Thing™️, and your mind is bursting with great ways to use TSQuery!

I’m going to follow this post up with some examples of how TSQuery can be used, including:

Creating custom TSLint rules
Creating custom Angular Schematics
Finding out interesting stuff about your code base
AND MORE!? ⚡️️️️️️ ⚡️️️️️️ ⚡️️️️️️

Until then, please reach out with any questions, ideas, anything! ❤️

Custom TSLint rules with TSQuery 😍

Craig ☠️💀👻 — Sun, 27 Oct 2019 17:27:31 +0000

(originally posted on Medium)

TL;DR

TSLint already has a fairly straightforward API for adding custom lint rules, but writing the actual logic of the rules can be quite tricky! We can make that a lot easier by using powerful AST node selectors with TSQuery:

If you want to learn more about the “how?” and “why?” of TSQuery, check out this introduction here!

What is a lint rule?

A lint rule is a little piece of code that uses static analysis to automatically detect things that may be wrong with other bits of code. They can be as simple as checking for the presence (or lack of) whitespace, or as complex as avoiding certain functions or patterns in specific circumstances. Lint rules can be used to enforce coding standards throughout a codebase, and to prevent bad code from being released to the world, such as a stray debugger; or console.log();.

There are a number of existing tools for linting your code, such as ESLint, TSLint, HTMLLint, and SASSLint. These tools are each designed for a specific language, come with their own rules, and can be configured to fit your specific needs. ESLint and TSLint even allow you to create custom rules, giving you fine-grained control over the standards of your codebase. We’re going to focus specifically on writing custom rules with TSLint.

If you want to read a bit more about linting (specifically ESLint), check this awesome post by Sam Roberts.

Building a custom lint rule

TSLint comes with a huge set of lint rules out of the box, but sometimes you may want to write a custom lint rule that is specific to your codebase. If you’re frequently seeing the same issue in code reviews, or you’d like to pre-emptively discourage something that could cause future issues, writing a custom lint rule is a great option.

As an example, let’s consider fdescribe and fit, both of which are common utility functions in JavaScript testing libraries. fdescribe gives us a way to run specific groups of tests, and fit gives us a way to run individual tests— very useful! There’s certainly nothing wrong with using either of these functions as part of normal development. However, if they get committed to source control, you could unintentionally end up running only a few tests in your test suite! That wouldn’t be good, so we should write a custom lint rule to prevent it from happening. We’re going to start with a “normal” TSLint rule and then modify it to use TSQuery. To get started, we need to talk about ASTs.

Getting an AST

An Abstract Syntax Tree (AST) is a data structure that contains all the structural meaning of the source code, without using any formal syntax. That means it is perfect for static analysis like linting. Within the TypeScript parser and compiler, the AST is described by an object called a SourceFile. If we want to do any serious analysis of TypeScript code, we need to get our hands on one of them. Thankfully, TSLint makes it very easy! Let’s look at the basic shell of a new TSLint Rule:

A TSLint rule has a Rule class, which extends Rules.AbstractRule from TSLint. It has a single public function called apply, which takes a SourceFile, and returns an array of RuleFailures.

The apply function is the link between the TSLint runner and our custom rule. It is where we get access to the AST via the SourceFile. The apply function is where you set up any configuration that your rule has, and then (typically) call through to applyWithWalker like this:

If you’re still not sure what an AST is, or just want to brush up a bit, check out this ⚡magical⚡ talk I gave at JS Conf AU 2018 for a refresher!

Walking an AST

TSLint’s Walker APIs allow you to walk through each node in the TypeScript AST and define rules that inspect the properties of individual nodes, and the relationships between them.

We start with the SourceFile (the root of the tree) and recursively visit each child of each node that we come across. As we walk through the tree, we can inspect each node and check for structures that break our rule.

In this case we’re looking for any occurrences of code like fdescribe() or fit(). We don’t want our rule to fire if we see a comment (e.g. // fdescribe();) or a standalone variable (e.g. `fit*). This is why AST traversal is preferred over Regular Expressions — we can be very specific about what constitutes a failure.

Exploring an AST

To help understand the structure of our code, we can use something like ASTExplorer to see the different nodes of the AST. Check out this example, and click on the functions in the code. The individual AST nodes should be highlighted:

We can see that when we write something like fdescribe(), TypeScript converts that for us into an AST, with three nodes:

An Expression Statement, which represents a single expression of code.
A Call Expression, which represents a call to a function
And an Identifier, which in this case represents the name of the function that is being called.

ASTExplorer shows you a very rich, detailed representation of the structure of the code, which is very useful but can also be overwhelming. As an alternative, you may want to also check out Uri Shaked’s TSQuery Playground.

Inspecting an AST

Just looking for an Identifier with the value fdescribe on its own would mean we may get false positives. We specifically care about the case where we have an Identifier inside a Call Expression. In code, that looks like something like this:

With the correct structure in place, we now need to test what the text of the Identifier is, and if it matches 'fdescribe', or 'fit’. If we get the right structure and the right Identifier name, we should raise a lint failure:

And boom 💥, we have a working lint rule!

Querying an AST

Our lint rule is pretty cool, and it works great! But the code for the rule isn’t particularly readable, and it could get a lot worse if we had to write a more complex rule. Thankfully, we can use TSQuery, which gives us a powerful way to express these AST traversals, using something like CSS selectors.

We know that we care about a Call Expression if it contains an Identifier, and that Identifier’s name is either fdescribe or fit. This can be expressed with TSQuery with the following query:

CallExpression > Identifier[name=/^f(describe|it)$/]

This query is a bit like CSS with some extra tricks. We can use a child combinator (>) to check for the Identifier inside the Call Expression, and then use an attribute selector ([]) to check for the specific name values with a Regular Expression. We can hook it into our rule like so:

By adding in our TSQuery selector, and running that over the SourceFile, we can just map from our matches to an array of RuleFailures 😎! We no longer need to worry about using applyWithWalker or traversing the tree 🎉!

We can finish off our rule by adding an automatic fix, and an option to only run this rule on files that match specific extensions (no point running the rule on a non-spec file!), and we end up with something like this:

Pretty neat eh?! 🚀

Testing a custom lint rule

While we’re at it, let’s add some tests for our rule. We start by defining what a “failing” bit of code looks like, parsing that code into AST, and then passing the AST to our rule. TSQuery is again useful, as it provides a helper function to turn a string of code into an AST:

We can do the same thing for a “passing” bit of code:

We can even add a test to make sure our “fix” works:

Check out the full .spec.ts file here. There’s no reason why we couldn’t have written these tests first and had a great spec against which to write our custom rule. Let’s write another, more complex rule, and do just that!

Building another custom lint rule

This time, we’re going to kick things up a notch and write a pretty niche rule, based on this great post by Paul Lessing. It turns out that error-handling with @ngrx/effects can be a bit tricky, and there’s a particular case where it can break your whole app:

If you use catchError in an observable chain in an effect, one error can stop the whole chain from running, which is almost definitely what you don’t want. We use @ngrx/effects in our Angular app, and we don’t want this to happen by accident! This basic example becomes the “failing” test case.

Our “passing” test is just some code that doesn’t use catchError:

Creating our query

We need to write a query that only selects calls to catchError() when they occur on the outermost chain in an @Effect. We can start with our failing code and build up from there. Using TSQuery playground to see the structure, we get something like this:

We care about any CallExpression which has the name "catchError", but specifically only at the first level of an Effect chain. An Effect chain is created with a PropertyDeclaration that has a Decorator with the name Effect. We can describe that relationship with the following query:

ClassDeclaration 
> PropertyDeclaration:has(Decorator Identifier[name="Effect"]) 
> CallExpression 
> CallExpression
> Identifier[name="catchError"])

Whew. That looks quite a bit more complex, but it fairly concisely describes the specific structure that we care about. We can now come up with a useful error message, and plug it into a new rule:

Et voilà! Our tests pass, and we have a brand new, super specific lint rule fresh out the oven 🥖! Our codebase is safer than ever before, and we don’t have to remember to look for this in code reviews! This rule could’ve been pretty tricky to write with an AST Walker, but with TSQuery it’s easy.

Wrapping up

There you have it!

TSQuery is particularly suited for this kind of code inspection, so why not have a go at writing your own rule using it. You can see some more examples of lint rules written using TSQuery here and here from Nicholas Jamieson. Please hit me up if you have any comments/questions, and thanks for reading 👋!

DEV Community: Craig ☠️💀👻

How does TypeScript's explicit resource management work?

Resource management 🤡

New shiny ✨

Transpiler output 💻:

Helper functions 💪🏻:

__addDisposableResource 🗑️:

__disposeResources 🚮:

Betterer v5.0.0

What is Betterer?

What happened to v2, v3, v4...?

What's in v5.0.0?

Parallel tests:

Before:

After:

Betterer ❤️ Angular

Simpler BettererFileTest:

Before:

After:

Improved workflow:

Improved caching:

Bug fixes and improvements:

Thanks ❤️

Incrementally adding Stylelint rules with Betterer

TL;DR;

NTL;PR (not that long, please read 😂)

Stylelint

Augmenting the configuration:

Passing the list of files:

Hooking into Betterer:

Adding files:

Adding issues:

The whole test:

Over-engineered TypeScript Types - but I learned some stuff!

Background

Incredibly hard nuances

Typing workerRequire

Getting the Module type:

Validating the Module type:

Typed if statements

Typed loops

Typed ✨Errors✨

Putting it all together:

Wrapping up:

Lazy debug logging for Node.js

The problem:

Attempt 1 - override Function.prototype.constructor:

Attempt 2 - Wrap require in a Proxy?

Proxy apply trap:

Proxy construct trap:

Adding function names:

Handling function arguments:

Nicer logs:

Adding this into my library:

Wrapping require.cache:

Wrapping up 😎

I need to learn about TypeScript Template Literal Types

TypeScript string types:

Template Literal Types "basics"

(Spoiler: it's not basic at all!)

Type-safe string dot notation:

What does it do?

How does it work?

Type-safe document.querySelector:

What does it do?

How does it work?

Split:

TakeLast:

TrimLeft/TrimRight/Trim:

StripModifiers

What next?

Stricter TypeScript compilation with Betterer

Betterer TL;DR

What is strict mode?

@betterer/typescript

Finding a test subject...

Setting up Betterer

Creating the test

Fixing some issues:

Tada! 🥳🥳🥳

`__addDisposableResource` 🗑️:

`__disposeResources` 🚮:

Simpler `BettererFileTest`:

Typing `workerRequire`

Getting the `Module` type:

Validating the `Module` type:

Typed `if` statements

Typed ✨`Errors`✨

Attempt 1 - override `Function.prototype.constructor`:

Attempt 2 - Wrap `require` in a `Proxy`?

Proxy `apply` trap:

Proxy `construct` trap:

Wrapping `require.cache`:

`@betterer/typescript`