TypeScript Compiler API: Improve API Integrations Using Code Generation

Developing against third-party or unfamiliar web APIs can be painful and slow compared to using native libraries.

For example, making requests with an HTTP client is simple enough but offers no compile-time sanity checks and no way for code suggestion tools like Intellisense to help you navigate. Worst of all, if the API you are consuming introduces breaking changes, you won't find out until runtime.

If you prefer the safety and convenience of strong types when working with web APIs, code generation can be a convenient solution. If the API provides a structured service definition, code generation can ease the development of client code, reduce runtime errors, and alert you to breaking changes before your code is deployed.

For TypeScript and JavaScript apps, the TypeScript Compiler API provides everything you need to build your own code generation tooling.

Examples of Existing Code Generation Tools

There is no sense in building a code generation tool if there's already one available that meets your needs. Several mature code generation tools for web APIs exist already; there's a good chance you've even used one.

The following are fantastic tools for adding safety and sanity to your networked Node.js apps:

• Protobufjs generates input and output types and function signatures for gRPC clients by reading a .protobuf file.
• openapi-generator generates well-defined JavaScript clients for REST APIs based on an OpenAPI spec document.

The above can add a layer of safety to your deployments, especially if the API definition file you're consuming is available over the web and can be fetched as part of your CI process.

However, if you're dealing with a less common API definition format (like a SOAP WSDL file or a custom JSON definition), these tools won't help you at all.

But you're not totally out of luck! Creating code generation tools like these does not require experience writing compilers or even an intimate knowledge of the internals of TypeScript and JavaScript.

Build the Codegen Tooling Yourself

In the many cases when a well-defined API doesn't have existing codegen tooling, such as:

• a home-grown JSON-over-TCP API
• a message queue with specific input and output formats defined in a YAML file
• something dated (and dreaded) like a SOAP API

Then we can take advantage of the TypeScript compiler API to generate types or even complete clients. It simply becomes a problem of coercing one form of structured data into another — and if you're a developer, that's probably your bread and butter.

Abstract Syntax Trees with the TypeScript Compiler API

Part of the TypeScript compiler's job is to construct abstract syntax trees (ASTs) of the input source code when transpiling TypeScript to JavaScript. These trees are structured representations of the source code that are fairly easy to understand, even if you're not used to working with intermediate representations of code (after all, the compiler is emitting plain old JavaScript).

The methods used to construct ASTs from a source code file are exported from the typescript library, allowing us to generate ASTs from scratch. It's easier to get started with a structured API definition.

Example: Generating Input and Return Types for a SOAP API

SOAP APIs are a good example of a messaging protocol with a strongly-typed schema and relatively poor tooling in the JavaScript ecosystem.

If you're unfortunate enough to have to work with a SOAP API in a Node.js project (as I have been several times), you need all the help you can get. So let's work through an example.

A common library for consuming SOAP APIs in JavaScript is soap. This will enable you to make API calls with JavaScript objects as inputs and outputs, but these objects are not type-checked at compile time. All client methods are typed any, and Node won't even know if methods in your code are callable until runtime.

Let's look at an example API definition, how we might currently consume it with the soap package, and some easy-to-miss bugs. Here's a (partial) definition of a stock quote SOAP service:

<definitions
    name="StockQuoteService"
    targetNamespace="http://www.example.com/wsdl/StockQuoteService.wsdl"
    xmlns="http://schemas.xmlsoap.org/wsdl/"
    xmlns:soap = "http://schemas.xmlsoap.org/wsdl/soap/"
    xmlns:qs = "http://www.example.com/wsdl/StockQuoteService.wsdl"
    xmlns:xsd = "http://www.w3.org/2001/XMLSchema"
>
    <message name="FetchQuoteRequest">
        <part name="tickerSymbol" type="xsd:string"/>
        <part name="dateRangeStart" type="xsd:dateTime"/>
        <!-- Optional, defaults to current date -->
        <part name="dateRangeEnd" type="xsd:dateTime" minOccurs="0"/>
    </message>

    <message name="Quote">
        <part name="tickerSymbol" type="xsd:string"/>
        <part name="current" type="xsd:int"/>
        <part name="dateRangeHigh" type="xsd:int"/>
        <part name="dateRangeLow" type="xsd:int"/>
        <part name="dateRangeDelta" type="xsd:string"/>
        <part name="volume" type="xsd:int"/>
    </message>

    <portType name="FetchQuotePortType">
        <operation name="FetchQuote">
            <input message="qs:FetchQuoteRequest"/>
            <output message="qs:Quote"/>
        </operation>
    </portType>

    <binding>
        ...
    </binding>
</definitions>

This WSDL defines a single RPC operation, FetchQuote, and its input and output types, FetchQuoteRequest and Quote. In our client code, these would correspond to a method definition and two object types.

We could jump right in and consume this API with the soap package, with an implementation that's something like this:

import soap from "soap";

// format integer cents as a currency amount string
const formatCents = (cents: number, currencySymbol: string = "$"): string =>
  `${currencySymbol}${(cents / 100).toFixed(2)}`;

// format number as fixed-width percentage string
const formatPercentage = (value: number): string => `${value.toFixed(2)}%`;

async function main() {
  const client = await soap.createClientAsync(
    "http://www.example.com/wsdl/StockQuoteService.wsdl"
  );

  const req = {
    tickerSymbol: "FOO",
    dateRangeStart: "2021-03-10T09:00:00",
    dateRangeEnd: "2021-05-10T18:00:00",
  };

  const res = await client.FetchQuote(req);
  console.log(`Stock Quote for ${res.tickerSymbol}:
    Current price: ${formatCents(res.current)}
    Range: ${req.dateRangeStart} - ${req.dateRangeEnd}
    Range high: ${formatCents(res.dateRangeHigh)}
    Range low: ${formatCents(res.dateRangeLow)}
    Change in range: ${formatPercentage(res.dateRangeDelta)}
    Volume: ${res.volume}
  `);
}

Even with the advantage of a strongly-typed WSDL file, this implementation is open to all kinds of type errors.

There's already a bug in this code that would not be caught until runtime. Notice the WSDL defines Quote.dateRangeDelta as a string, but this code expects a number. The .toFixed() method call will throw a runtime type error, but even using TypeScript as we are in this example will not help us catch this error during compilation.

Accessing the Compiler API

To generate a TypeScript AST, you need the ubiquitous typescript package.

npm install typescript

To create the initial AST, import the compiler and instantiate an in-memory representation of a TypeScript source file:

import ts from "typescript";

const sourceFile = ts.createSourceFile(
  "soap-types.ts", // the output file name
  "", // the text of the source code, not needed for our purposes
  ts.ScriptTarget.Latest, // the target language version for the output file
  false,
  ts.ScriptKind.TS // output script kind. options include JS, TS, JSX, TSX and others
);

Reading the API Definition

We need a structured, programmatically accessible representation of the API schema. In JavaScript and TypeScript, this will usually be an object. Depending on your API definition format, this could be as simple as const schema = require("api.json").

In our case, we'll need a simple library to read the WSDL XML file into something a bit easier to work with. We'll use xml2js.

Since part of our goal is to achieve better type safety, we'll create some types for the API definition as well. For most APIs, we'll have some version of an input and output object, and a named function with the input type as an argument, returning the output type.

At this stage, it's helpful to focus on modeling the schema of the API in an implementation-agnostic way. Don't worry about things like client method signatures or dependency injection if you can help it.

import { Parser } from "xml2js";
import { readFileSync } from "fs";

// Primitive datatypes defined by SOAP (there are more)
type SoapPrimitive =
  | "xsd:boolean"
  | "xsd:double"
  | "xsd:float"
  | "xsd:int"
  | "xsd:short"
  | "xsd:string"
  | "xsd:dateTime";

// SOAP message type
interface IMessage {
  meta: {
    name: string;
  };
  part: {
    meta: {
      name: string;
      type: SoapPrimitive;
      minOccurs?: string; // default = 1, making the field required. 0 notes an optional field
    };
  }[];
}

interface IOperation {
  meta: {
    name: string;
  };
  input: {
    meta: {
      name: string;
    };
  };
  output: {
    meta: {
      name: string;
    };
  };
}

// Top-level WSDL object structure
interface IServiceDefinition {
  definitions: {
    message: IMessage[];
    portType: {
      operation: IOperation[];
    }[];
  };
}

We can then write a function to generate instances of these types. It should read the file into an object and coerce it into the IServiceDefinition interface.

async function readWSDL(filePath: string): Promise<IServiceDefinition> {
  const wsdlData = readFileSync(filePath, { encoding: "utf-8" });

  const xmlParser = new Parser({
    attrkey: "meta", // instructs the parser to pack XML node attributes into a sub-object titled "meta"
  });

  const serviceDefinition = await xmlParser.parseStringPromise(wsdlData);

  // I would recommend a more explicit conversion, but for the sake of brevity, this example just casts the object to the interface type.
  return serviceDefinition as IServiceDefinition;
}

Generating the Input and Output Types

The process of converting a structured API definition into a Javascript object will vary widely based on use case and implementation. It's important to get well-organized information about the types you want to generate. Here, the critical bits are the name, data type, and optionality of the field.

To create TypeScript interfaces to represent the WSDL message types, we'll employ the TypeScript compiler's createInterfaceDeclaration factory (if you prefer class definitions, createClassDeclaration works similarly). We need to provide it with the same information we would provide an interface declaration if we were writing it by hand: attributes, modifiers, and obviously, a name. If you've ever used a compiler API like LLVM, this may feel familiar. In
more specialized use cases, you can even provide generic type parameters and decorators.

For our example, we just need simple interfaces with strongly-typed fields. This will require mapping from WSDL-defined types to TypeScript types.

WSDL defines a lot of types, and your implementation may not need all of them. For this example, I'm excluding some lesser-used ones like hexBinary and unsignedByte and default unrecognized types to string:

const dataTypes: Record<SoapPrimitive, string> = {
  "xsd:boolean": "Boolean",
  "xsd:double": "Number",
  "xsd:float": "Number",
  "xsd:int": "Number",
  "xsd:string": "String",
  "xsd:short": "Number",
  "xsd:dateTime": "Date",
};

// Convert from SOAP primitive type to a Typescript type reference, defaulting to String
function typeFromSOAP(soapType: SoapPrimitive): ts.TypeReferenceNode {
  const typeName = dataTypes[soapType] ?? "String";
  return ts.factory.createTypeReferenceNode(typeName, []);
}

Some input fields may be optional, so we will need a reference to the relevant syntax (a question mark in TypeScript, ignored when importing distributable code into a JavaScript file). To create simple tokens like this (and things like comparison operators, math and logical operators) we can use the createToken factory method.

// used for marking attributes as optional
const optionalFieldMarker = ts.factory.createToken(ts.SyntaxKind.QuestionToken);

It's important we can use these types elsewhere, so we'll need to add the export modifier to the declared types. The ts.ModifierFlags object contains an assortment of modifiers, like class member modifiers (public, protected, static), and the async keyword.

// used for adding `export` directive to generated type
const exportModifier = ts.factory.createModifiersFromModifierFlags(
  ts.ModifierFlags.Export
);

With the basic tools defined, we can write a function to generate the actual AST nodes that will be written to our generated code. The goal is to translate from our familiar structure to TypeScript AST using the factories provided by the compiler to create nodes, then assemble those nodes into interface declarations.

We need an identifier (essentially, the code symbol that names the interface), and a list of properties, each with its own identifier, type, and possibly optionality token (?).

function generateMessageInterface(message: IMessage): ts.InterfaceDeclaration {
  // name of the interface
  const interfaceSymbol = ts.factory.createIdentifier(message.meta.name);

  const interfaceMembers = message.part.map((part) => {
    const propertySymbol = ts.factory.createIdentifier(part.meta.name);

    // WSDL treats fields with no minOccurs as required, and fields with minOccurs > 0 as required
    const required =
      !Boolean(part.meta.minOccurs) || Number(part.meta.minOccurs) > 0;

    // representation of the interface property syntax
    const member = ts.factory.createPropertySignature(
      undefined, // modifiers (not allowed for interfaces)
      propertySymbol, // name of the property
      required ? undefined : optionalFieldMarker,
      typeFromSOAP(part.meta.type) // data type
    );
    return member;
  });

  const messageType = ts.factory.createInterfaceDeclaration(
    undefined, // no decorators
    exportModifier, // modifiers
    interfaceSymbol, // interface name
    undefined, // no generic type parameters
    undefined, // no heritage clauses (extends, implements)
    interfaceMembers // interface attributes
  );

  return messageType;
}

Then to generate all the needed types, it's as simple as mapping over our IServiceDefinition object's message array to this function, placing the resulting interface declaration nodes in a NodeArray, and writing the AST to a TypeScript file.

// read service definition into object
const serviceDefinition = await readWSDL("quote_service.wsdl");

const messageInterfaces = serviceDefinition.message.map(
  generateMessageInterface
);

// add nodes to a node array
const nodeArr = ts.factory.createNodeArray(messageInterfaces);

To write the source tree to disk, we need an instance of the ts.Printer class. Printing the node list to a file is as simple as specifying the file format (usually MultiLine as it is the most readable) and a destination file path.

const printer = ts.createPrinter({ newLine: ts.NewLineKind.LineFeed });
const result = printer.printList(ts.ListFormat.MultiLine, nodeArr, sourceFile);
writeFileSync("soap-types.ts", result, { encoding: "utf-8" });

This will write the following code to a file called soap-types.ts:

export interface FetchQuoteRequest {
  tickerSymbol: String;
  dateRangeStart: Date;
  dateRangeEnd?: Date;
}
export interface Quote {
  tickerSymbol: String;
  current: Number;
  dateRangeHigh: Number;
  dateRangeLow: Number;
  dateRangeDelta: String;
  volume: Number;
}

That's it! Correctly typed input and output that's easy-to-read (for both people and machines). Using these types in place of the Plain Old JavaScript Objects we used in the first example will give us a bit of compile-time safety.

Even standard JavaScript projects can benefit from improved IDE code completion through simple changes to the transpilation target.

We could go further down this path and generate a full type-safe client (likely just a wrapper around a soap.Client), with methods defined for each WSDL operation, using the generated types as inputs and outputs. Generating such a function might look something like this:

// used for adding `async` and `export` modifier to generated function
const asyncExportModifier = ts.factory.createModifiersFromModifierFlags(
  ts.ModifierFlags.Export | ts.ModifierFlags.Async
);

function generateOperationFunction(
  methodName: string,
  inputType: ts.InterfaceDeclaration,
  outputType: ts.InterfaceDeclaration
): ts.FunctionDeclaration {
  const functionIdentifier = ts.factory.createIdentifier(methodName);

  const inputType = ts.factory.createTypeReferenceNode(inputType.name);
  const outputType = ts.factory.createTypeReferenceNode(outputType.name);
  const outputTypePromise = ts.factory.createTypeReferenceNode("Promise", [
    outputType,
  ]);

  const inputParameter = ts.factory.createParameterDeclaration(
    undefined, // decorators,
    undefined, // modifiers,
    undefined, // spread operator
    "input", // name
    undefined, // optional marker,
    inputType, // type
    undefined // initializer
  );

  const fn = ts.factory.createFunctionDeclaration(
    undefined, // decorators
    asyncExportModifier,
    undefined, // asterisk token
    functionIdentifier,
    undefined, // generic type parameters
    [inputParameter], // arguments
    outputTypePromise, // return type
    undefined // function body
  );

  return fn;
}

const operationFunctions =
  serviceDefinition.definitions.portType[0].operation.map((op) =>
    generateOperationFunction(
      op.meta.name,
      messageInterfaces[0],
      messageInterfaces[1]
    )
  );

// add nodes to a node array
const nodeArr = ts.factory.createNodeArray(operationFunctions);

Using this function with the input and output types we just generated, we create the following function stub in soap-types.ts:

export async function FetchQuote(input: FetchQuoteRequest): Promise<Quote>;

To make this really useful, you'd want to inject calls to methods on a soap.Client within these generated functions - but that's beyond the scope of this post.

TypeScript Compiler: A Powerful Codegen Tool

The TypeScript compiler is a powerful code generation tool — any code that you can manually write, you can write programmatically using this API.

It's worth noting that, as in the case of many specialized APIs, the TypeScript compiler API's documentation is incomplete and not particularly friendly to newcomers. This crash-course README is, to my knowledge, the best place to get started, but I feel the best way to get familiar with the API is to play around with it yourself and dive into the code.

There are tons of ways custom code generation can benefit your project beyond generating API types, including instrumenting code at compile time (think performance monitoring, consistent logging, etc.) and generating skeleton code when porting from another similar language.

I hope this article has given you ideas of how to improve your project's reliability or developer experience using code generation. Cheers!

Check out the code from this article in this repo.

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Nate Anderson is a software engineer working on new products at CARD.com, a tech company in the United States providing personalized payment solutions like premium banking. He believes the key to great technology is a close relationship between product and engineering, and is excited about tech like Go, TypeScript, GraphQL, and serverless computing.