My Tooling Wishes for 2020 ✨

Hi, I'm Maël! I've been leading Yarn's development for a few years now, and I've decided to take a short break for the Yarn v2 trunk to talk a bit about the technical issues I've encountered this year and that I hope 2020 will solve 🎄

Given my area of expertise this wishlist contains a lot of specialized items, and I'd be very interested to hear about the problems that other developers encounter, so please share your own stories! 😃

JavaScript

• A proposal to alias Object.prototype.hasOwnProperty.call with a short name. I know that Maps are a superior dataset in general, but checking whether a key exists or not after parsing a JSON dataset shouldn't require to jump through such hoops.

• Weak references! Working with WebAssembly libraries is currently painful because we have no good way to release memory with the same garbage collection semantic as Javascript. Finalization groups can't come soon enough.

---

Node

• A move to make FsEvents become a native package. Everyone and their dog is depending on it somehow, and it's one of the main native package out there. Embedding it would help make projects more stable across Node versions.

• Zip archives becoming first-class citizen for filesystem operations (not tgz, because they have significant random file access costs).

  • Potentially through URL instances that would have a specific protocol? The problem with this is that pretty much every package expects paths to be string nowadays...
  • Amongst other cool things, it would decrease the startup time for single-file CLI applications (such as Yarn), because we would be able to package them into a zip archive whose only the required files would be parsed!
  • More generally, allowing for userland extensions to the fs module would allow for very interesting use cases.

• An up-to-date prototype regarding ESM loaders. Think of it as being able to directly require a TS file if you have the right loader - kinda like ts-node or babel-register, but without having to fight the engine as you currently do.

  • I've been following along the discussions, but it always seemed like the available implementations were very far away from what the API would eventually look like.

• A proposal to deprecate require.resolve and replace it by two new functions that would split the require.resolve semantic into two: one to transform a specifier into a token that can be required, and another that can transform a token into a filesystem path.

  • The original design was fine when modules had a one-one mapping with the filesystem, but as package managers experiment with compressed packages it doesn't hold anymore.
  • This will most likely never happen, since deprecating a function like require.resolve will be very difficult (especially considering that the modules WG is now focused on ESM).

• Additional package managers should be shipped along with Node. That the only officially supported package manager comes from a VC-funded company is frankly embarrassing and somewhat worrying.

  • Note that this isn't about npm itself - they could be the nicest people in the world that I'd still be worried. Remember what happened to SourceForge?

---

TypeScript

• Decorators as expressions support (TypeScript#7342).

  • This would make the Clipanion user experience much nicer, as you could then declare the commands directly within their registration calls.

• Inference improvements so that decorators could contribute to the inference of the class property types (TypeScript#4881).

  • This would fix the one remaining type instability source in Clipanion by ensuring that the properties have a type that matches what the decorator says (whereas we currently rely on the decorator and property geographic proximity to prevent such issues).

• Builtin support for Plug'n'Play. It's by now one of the last remaining major tools to not support it, and sadly it's also one of those whose support is critical to our vision.

  • Note that we have developed a workaround, but it requires significant resources to develop it that could have been spent elsewhere had we received a bit more support.
  • There's been a tentative PR but it didn't pan out. We're now waiting for the second option, which is...

• Plugin support for TypeScript #16607. This would avoid issues we'll go into where innovations end up being conditioned to another project's roadmap. I'm not unfamiliar with the problem - it's even part of why Yarn evolved into a pluggable architecture.

---

Ecosystem

• I hope we'll stop with these postinstall scripts just to print a few messages on the command line. There is now a true good way to do it, which you should adopt instead of postinstall scripts.

  • I've talked about it a lot already, but postinstall scripts have negative effects on the optimizations that package managers can do. Adding them just for the sake of printing a message on the terminal is not a great idea.
  • In fact, I'm considering ignoring the funding field from packages that also list postinstall scripts. Change my mind.

• Open-source sustainability is still a problem, although we seem to rediscover it every few years (more on that later...). I hope we'll find a solution where companies can pay open-source researchers enough to balance the value they extract from their work.

  • Btw, did you know that Yarn currently operates with a budget of, literally, $0? Don't forget to thank my awesome company Datadog (we're hiring!) for letting me work on it part of my time, and my lovely fiancée for the rest of it 😉

---

VSCode

• A fix for the ESLint integration which suddenly stops responding when working with large files or an overused CPU (VSCode-ESLint#600).

• A fix for the ESLint integration so that running Ctrl+S will run as many passes as needed instead of a single one (VSCode-ESLint#154).

• Builtin support for transparently opening files from within Zip archives (VSCode#75559), so that I can access my vendors more easily (particularly when Ctrl+clicking on the symbols).

  • At the very least, allow plugins to add this feature (we can already provide custom filesystems, but they are mostly unused by the regular VSCode).

• VSCode clearly surfacing that a project contains a custom TypeScript SDK and suggesting to use it instead of the default one. I've regularly seen people wondering why VSCode is reporting different results from yarn tsc.

---

GitHub

• A new URL pattern that would allow us to link to source symbols rather than specific lines. This would make it possible to reference particular classes or functions from the documentation without having to pin the link to a specific commit (example where I use this pattern).

• A new concurrency settings in GitHub Actions that would allow us to specify that a single job can run at a time, and in the order they got dispatched exclusively. This would prevent wasteful executions that happen when a merge queue operates in non-atomic mode.

• A way to trigger a workflow when pressing the Merge button instead of doing the regular merge. You can even just connect it to repository_dispatch and let the users take care of the rest.

• Exposing Workflow names to the GitHub API (especially the v4). For some reason only the step names are exposed, but not the workflow names (it's also why Shields cannot use the API).

• An API to access the full list of GitHub Actions via the v4 API. This would make it possible to build our own dashboards.

• An improved search that would allow us to exclude test files from our searches, along with searching file paths and content with regexps.

  • I think it's theoretically possible to filter searched files per their path, but I never managed to make it work.

• Make it possible to trigger a workflow with a specific context when the user posts a comment containing a special tag. For example, posting /autofix would dispatch a new issue_command trigger that workflows could watch (and filter per the tag).

• Add support for custom merge drivers in the .gitattributes file. It seems that even adding support for the the merge=union attribute would be beneficial (GitHub#487).

• An official public GitHub repository with opened/closed issues that we can use to voice feedback? It's honestly a bit awkward to have the GitHub community forums when your community is already used to the tracker workflow - and informations are kinda hard to find, too 🤔

---

Git

• A .gitattributes option to prevent Git from detecting new files as being renamed. In some folders such files are completely different, and marking them as renames is semantically incorrect.

---

Extra

• QEmu being compiled down to WebAssembly. We could then truly run Linux within our browsers! (the currently existing alternatives don't support Node, which makes it a no-go for my use case: interactive Yarn tutorials)

Comments

[Bram Gotink]

Hi Maël,

A lot of these would indeed be great additions to the ecosystem!
I just wanted to respond to one item: the decorators in clipanion.

At work we've enountered a similar problem. Allow me to quickly paint the situation: Angular's component devkit has functions that coerce component input into a specific type, e.g. coerceBoolean for booleans.
This lead to a lot of repitition of the following pattern in our code:

export class FooComponent {
  private _isBar = false;

  @Input()
  public get isBar(): boolean {
    return this._isBar;
  }

  public set isBar(isBar: boolean) {
    this._isBar = coerceBoolean(isBar);
  }
}

To simplify this, we created coercion decorators. The previous code is equivalent to:

export class FooComponent {
  @Input()
  @coerceBooleanProperty()
  public isBar = false;
}

We wanted to ensure typescript enforced the type of the property, otherwise these decorators are a step back with respect to type safety.

Having decorators play into type inference would be a great solution to our problem! In the mean time, we have created types that allow us to write typed decorators. The following stackblitz shows our approach:

This is not as powerful as TypeScript#4881 proposes, e.g. it cannot modify the type of the property, but it does cover our specific use case. I hope this can also help in clipanion.

Kind regards,
Bram

[Maël Nison]

I like that! Basically it's exactly the opposite - instead of saying that the decorator provides the type, you instead let it check that the property name is the same as one of the properties in the class that's compatible with the expected type. Smart!

[Mark Fulton]

You can use a reviver with JSON.parse to actually create Map instances instead of plain objects when parsing JSON so that you can use Map.prototype.has instead of Object.prototype.hasOwnProperty:

JSON.parse('{"a":null,"b":{"c":[0,1]}}', (_, value) =>
  value !== null && typeof value === "object" && !Array.isArray(value)
    ? new Map(Object.entries(value))
    : value
);
// ▸ Map(2) {"a" => null, "b" => Map(1)}

You can then also use a replacer with JSON.stringify to stringify Map instances as plain objects:

JSON.stringify(
  new Map([
    ["a", null],
    ["b", new Map([["c", [0, 1]]])]
  ]),
  (_, value) => (value instanceof Map ? Object.fromEntries(value) : value)
);
// "{"a":null,"b":{"c":[0,1]}}"

[Alex Leung]

Hi Maël,

Have you looked into building a ttypescript plugin to go along with the ts-loader plugin for adding TypeScript Plug-n-Play?

Also, could you ever foresee Yarn integrating with other package registries, for example to include WASM modules thru wapm?