What's new in Piral #4

Piral is an open-source framework for microfrontends based on React. It brings everything to create amazing frontends using a scalable development approach.

This is the fourth blog post about our progress on Piral. We will continue to describe our progress and future plans here on dev.to.

The agenda should be almost identical in each post. We'll use

• Current progress to explain what has been done since the last post
• Future plans to explain what we are currently working on or have in our direct pipe
• Other thoughts to list some of the thoughts for future development

Please reach out to us in case of any feedback, ideas, or criticism. We'd love to improve!

Current Progress

We just released version 0.10 🎉! This is a great release that contains many important changes - mostly under the hood.

Actually, we are already at version 0.10.1 with 0.10.2 around the corner.

In this post I'll want to go into one specific detail: How we generate declarations.

Declaration Generation - Old Version

Previously, we generated the declarations by simply merging different files into a single. The files have been following a certain convention and we made sure to catch referenced files via some regular expressions.

For getting all import references we used:

const importDeclRx = /import\s+((.*?)\s+from\s*)?['"`](.*?)['"`]\s*;?/g;

For getting all export references we used:

const exportDeclRx = /export\s+((.*?)\s+from\s*){1}['"`](.*?)['"`]\s*;?/g;

In the end this results in the following process. We start with a bunch of files that may look as follows:

// a.d.ts
import { Example1 } from "./b";
import { FC } from "react";

export interface Example2 {
  foo: string;
  bar: Example1;
  Component: FC;
}

export { Example1 };

// b.d.ts
export * from "./c";

// c.d.ts
export type Example2 = "foo" | "bar";

export interface Example3 {}

This is now merged together using some app shell name, e.g., for my-app-shell we get:

declare module "my-app-shell" {
  export * from "./my-app-shell/a";
}

declare module "my-app-shell/a" {
  import { Example1 } from "my-app-shell/b";
  import { FC } from "react";

  export interface Example2 {
    foo: string;
    bar: Example1;
    Component: FC;
  }

  export { Example1 };
}

declare module "my-app-shell/b" {
  export * from "./my-app-shell/c";
}

declare module "my-app-shell/c" {
  export type Example2 = "foo" | "bar";

  export interface Example3 {}
}

One of the drawbacks of this approach is that it only works with .d.ts files. At first this seems to be irrelevant, however, since we want to support custom typing declarations, too, this implies that any typings need to be transpiled to a TypeScript declaration first. In many cases this is just unnecessary effort.

Another drawback is that we include types that are not even reached from the root module. This is certainly not what we want. For instance, in c.d.ts we find Example3, which is not exported by module a.d.ts and thus could be left out.

The most significant problem, however, is that the IDE (e.g., VS Code) will recognize all modules and display their names in an import {} from ' auto-completion scenario. This is quite some bloat, to say the least.

What we wanted is a mechanism that:

• Generates a single .d.ts referencing / omitting the externals
• Works with plain .ts files, too
• Can also infer the used API from .js files
• Only exports what can be used in pilets (i.e., do not export what is exclusive to a Piral instance)
• Creates the smallest possible .d.ts

For this, our only choice was to go directly against the TypeScript compiler API.

Declaration Generation - New Version

Piral now uses an internal mechanism that essentially takes

• a set of input files (all .ts, .tsx, .js, and .jsx) from the Piral instance's source directory,
• the typings file referenced in the package.json, if any,
• the file containing the original definition of the PiletApi interface,
• the name of the Piral instance, and
• the package names of the used externals (shared dependencies)

as input arguments.

The declaration generation has three steps:

1. Setup of the visitor context
2. Declaration gathering using the context
3. Creation of the string representation using the context

The declaration gathering itself has two different phases:

1. Assembly of the exposed PiletApi
2. Inclusion of the additionally declared types (starting from the typings field of the package.json)

Both phases essentially boil down to call a function named includeExportedType, which gets the type reference and places it in a container for creating the string representation later. The string representation forms the content of a valid .d.ts file.

The third step involves going over the context. The context has been derived by visiting the different TypeScript nodes. Each node is then queried for meaningful type information. Only if we find something worth keeping we'll descend further.

The context itself is just a plain JS object:

export interface DeclVisitorContext {
  modules: Record<string, TypeRefs>;
  checker: ts.TypeChecker;
  refs: TypeRefs;
  ids: Array<number>;
  usedImports: Array<string>;
  availableImports: Array<string>;
}

export type TypeRefs = Record<string, TypeModel>;

The interesting part is the TypeModel definition. After all, this is the union type of all potentially interesting type definitions.

export type TypeModel =
  | TypeModelString
  | TypeMemberModel
  | TypeModelProp
  | TypeModelBoolean
  | ...
  | TypeModelRef
  | TypeModelAlias;

A single definition may be as simple as only the specification of a kind property, which acts as a discriminator for the union.

export interface TypeModelAny {
  readonly kind: "any";
}

It can also be more complicated. As an example the TypeModelProp which describes a single property of an object has multiple properties and inherits from WithTypeComments:

export interface TypeModelProp extends WithTypeComments {
  readonly name: string;
  readonly optional: boolean;
  readonly kind: "prop";
  readonly valueType: TypeModel;
  readonly id: number;
}

export interface WithTypeComments {
  readonly comment?: string;
}

Declaration Gathering

Back to our two phases for the declaration gathering.

The first part can be covered by a simple visitor that walks the file known to contain the PiletApi interface. Problem solved.

const api = program.getSourceFile(apiPath);

ts.forEachChild(api, node => {
  if (ts.isInterfaceDeclaration(node) && node.name.text === "PiletApi") {
    includeNode(node);
  }
});

The second part is more complicated. Here, we want to include all top-level exports as members of the basic module declaration. Otherwise, we'll take the exports into the declared module.

To illustrate this, let's imagine we have a file containing

export interface Example1 {}

declare module "my-app-shell" {
  export interface Example2 {}
}

declare module "other-module" {
  export interface Example3 {}
}

Assuming the app shell itself is named my-app-shell we end up with two more interfaces exported from my-app-shell. Furthermore, we found another module (other-module) with a single interface.

To achieve this behavior we use a module rotation in the context.

const includeTypings = (node: ts.Node) => {
  context.refs = context.modules[name];

  if (ts.isModuleDeclaration(node)) {
    const moduleName = node.name.text;
    const existing = context.modules[moduleName];
    const before = context.refs;
    context.modules[moduleName] = context.refs = existing || {};

    node.body.forEachChild(subNode => {
      if (isNodeExported(subNode)) {
        includeNode(subNode);
      }
    });

    context.refs = before;
  } else if (isNodeExported(node)) {
    // include exported node in current module
  } else if (ts.isExportDeclaration(node)) {
    // include relevant export nodes
  }
};

Essentially, we change the refs to the selected module. Then we perform the iterative approach again to get all relevant exports. After we obtained the relevant exports we reset the refs to the previously selected module.

Declaration Emission

After the gathering is done we over to return the string representation of the generated model. The declaration has a simple entry point.

return stringifyDeclaration(context);

The stringifyDeclaration function iterates over all contained modules, generating the string representation for each of them.

Our aim is to produce nicely looking generation files, which implies that we also perform some code formatting such as correct indentation levels or line breaks.

export function stringifyModule(name: string, refs: TypeRefs) {
  const content = stringifyExports(refs);
  const formattedContent = content
    .split("\n")
    .map(line => `  ${line}\n`)
    .join("");
  return `declare module "${name}" {\n${formattedContent}}`;
}

export function stringifyDeclaration(context: DeclVisitorContext) {
  const modules = Object.keys(context.modules)
    .map(moduleName => stringifyModule(moduleName, context.modules[moduleName]))
    .join("\n\n");

  const preamble = context.usedImports
    .map(lib => `import * as ${getRefName(lib)} from '${lib}';`)
    .join("\n");
  return `${preamble}\n\n${modules}`;
}

While stringifyDeclaration takes the whole context as input parameter, all other functions are mostly based on the TypeModel type or a related typed. For instance, the stringifyModule function takes the name of a module and all its exported TypeModel references.

Creating the string representation of something like an interface includes iterating over all stored properties. For each property we get its string representation.

function stringifyProp(type: TypeModelProp) {
  const target = type.valueType;
  const comment = stringifyComment(type);
  const isOpt = type.optional ? "?" : "";
  const name = makeIdentifier(type.name);

  if (
    target.kind === "object" &&
    target.calls.length === 1 &&
    target.indices.length === 0 &&
    target.props.length === 0
  ) {
    return `${comment}${name}${isOpt}${stringifySignatures(target.calls[0])}`;
  } else {
    return `${comment}${name}${isOpt}: ${stringifyNode(type.valueType)}`;
  }
}

Again, we take care to not only end up with a valid declaration, but also a readable one.

Summary

The given approach works quite well with the currently implemented types. The complexity of this approach certainly lies in the maintenance field. TypeScript internals need to be respected. The whole concept is certainly sensitive to changes in the TypeScript compiler. More edge cases and types will come up that require additional attention.

We plan to open-source this mechanism to be used in other projects, too. As we depend on TypeScript internals we will potentially ship this package as a single bundle - single flat file.

Future Plans

Right now we work on getting everything in ship shape for a 1.0 release in the first quarter of this year. As usual, we will not stick to this schedule if we find that further work is required.

We still continue on the different work tracks:

• Investigate possibilities for further plugins
• Improve the existing plugins
• Bring in more converters
• Extend the Piral ecosystem
• Enhance our additional SaaS offerings regarding Piral

In the converter space we'll focus on Blazor soon. We already started work there and have a proof of concept (PoC) ready. We'll demonstrate it at some conferences in the near future. This PoC will be taken further into a usable plugin and Visual Studio template for Blazor projects.

Besides Blazor another area that will be investigated by us is React Native. Like with server-side rendering we feel that Piral should be generic enough to support this use case directly. We will see how far we are and what needs to be done to enable using native microapps - at least on a basic PoC level.

In the ecosystem space we are right now dealing with the Chrome / Firefox / and others extension. This dev tools extension will increase the debugging capabilities quite a bit. Furthermore, we look into creating a VS Code extension to get improved capabilities also there - without having to use the CLI or configuring VS Code.

Other Thoughts

Right now the main use case for Piral is distributed application development. Distributed for us means a focus on separated repositories. Nevertheless, we realized that some people also prefer to use Piral in mono repos. This makes sense especially in the beginning of a project when the first modules are developed quite closely to the app shell.

In general we think the development should be able to scale out nicely. This implies a potential start in a mono repo. This also includes that dedicated repositories for some pilets should be possible, too. As a consequence we want to amplify development using this approach, essentially allowing referencing the Piral instance directly in a mono repo, e.g., managed by Lerna or things like Yarn workspaces.

We would love to get some feedback on our work. Is this useful? What would you like to see?

Conclusion

Piral is maturing well. We are happy with the current progress and are positive to reach a great release 1.0 soon. We will certainly invest more time in polishing and enhancing the current solution before elevating.

Including the declaration generation is a big step in the right direction. Our vision is to make the tooling and overall developer experience as smooth as possible.

With some of our clients already running Piral in production, we are sure of its stability and potential. We would love to see more community contributions, interest, and questions. In the end, our goal is not only to provide an outstanding framework but also build a welcoming community around microfrontends in general.